WO2023084713A1

WO2023084713A1 - Object sorting system

Info

Publication number: WO2023084713A1
Application number: PCT/JP2021/041598
Authority: WO
Inventors: 裕紀谷崎; 雅信本江; 健李
Original assignee: 株式会社Pfu
Priority date: 2021-11-11
Filing date: 2021-11-11
Publication date: 2023-05-19

Abstract

An object sorting system, wherein: a camera 20 captures a first image of an object group in a transport path along which the object group is transported; an image recognition unit 11 recognizes a second image, which is an image of objects present in the first image, by using a trained model, assigns feature information indicating a feature of the second image to the second image, generates a recognition result in which the feature information for second image is assigned to the first image, and stores the recognition result in a recognition result storage unit 12; an object sorting device 30a sorts a desired object, on the basis of the recognition result, from within the object group being transported along the transport path; and, when an undesired object different than the desired object is erroneously sorted by the object sorting device 30a, a recognition result extraction unit 13a assesses that the recognition result stored in the recognition result storage unit 12 is an erroneous recognition result including an image of the undesired object that was erroneously sorted and extracts the erroneous recognition result from the recognition result storage unit 12.

Description

Object sorting system

The present disclosure relates to an object sorting system.

At the waste disposal site, a large amount of waste flows on a belt conveyor and is processed every day. At sites where waste is processed, sorting of waste is performed manually. While sorting waste is a simple task, it places a heavy burden on workers who sort waste (hereafter sometimes referred to as "sorters"). A device (sometimes referred to hereinafter as a "waste sorter") has been developed to do so.

Patent No. 6854995

When the waste sorting device performs the work that the sorting worker was doing instead of the sorting worker, the waste sorting device recognizes each waste flowing on the belt conveyor, and based on the recognition result, the robot hand It is conceivable to extract the desired waste (hereinafter sometimes referred to as "desired waste") from the waste mass flowing on the belt conveyor using a vacuum cleaner or a suction pad. Therefore, when a plurality of types of waste are mixed and flowed on the belt conveyor, it is necessary for the waste sorting device to identify the type of waste. Recognition using a trained model generated by machine learning is effective for the waste sorting device to recognize various types of waste.

However, since machine learning requires a huge amount of training data, the annotation work performed on image data when generating training data from image data of various types of waste is replaced by image data. If this is done manually by visual observation, it takes a lot of labor to generate training data.

Therefore, this disclosure proposes a technique that can reduce the labor required to generate training data.

The object sorting system of the present disclosure has a camera, a recognition unit, a storage unit, a sorting unit, and an extraction unit. The camera captures a first image, which is an image of the object group, on a transport path along which the object group is transported. The recognition unit recognizes a second image, which is an image of each object existing in the first image, using the trained model, and recognizes feature information, which is information indicating features of the second image, to the second image. to generate a recognition result in which the characteristic information is added to the second image in the first image. The storage unit stores the recognition result. The sorting unit sorts out a desired object from the object group conveyed on the conveying path based on the recognition result. The extracting unit includes an image of the erroneously selected undesired object in the recognition result stored in the storage unit when the selecting unit erroneously selects an undesired object other than the desired object. An erroneous recognition result, which is a recognition result, is determined, and the erroneous recognition result is extracted from the storage unit.

According to the disclosed technology, the effort required to generate training data can be reduced.

FIG. 1 is a diagram illustrating a configuration example of an object sorting system according to Example 1 of the present disclosure. FIG. 2 is a diagram illustrating a configuration example of a control device according to the first embodiment of the present disclosure; FIG. 3 is a diagram illustrating an example of feature information according to Example 1 of the present disclosure. FIG. 4 is a diagram illustrating a trace example of barycentric coordinates according to the first embodiment of the present disclosure. FIG. 5 is a diagram illustrating a trace example of barycentric coordinates according to the first embodiment of the present disclosure. FIG. 6 is a diagram illustrating a trace example of barycentric coordinates according to the first embodiment of the present disclosure. FIG. 7 is a diagram illustrating a trace example of barycentric coordinates according to the first embodiment of the present disclosure. FIG. 8 is a diagram illustrating an example of extraction of desired waste in Example 1 of the present disclosure. FIG. 9 is a diagram illustrating an operation example of the control device according to the first embodiment of the present disclosure; FIG. 10 is a diagram illustrating an operation example of the control device according to the first embodiment of the present disclosure; FIG. 11 is a diagram illustrating an operation example of the control device according to the first embodiment of the present disclosure; FIG. 12 is a diagram illustrating an operation example of the control device according to the first embodiment of the present disclosure; FIG. 13 is a diagram illustrating an operation example of the control device according to the first embodiment of the present disclosure; FIG. 14 is a diagram illustrating an operation example of the control device according to the first embodiment of the present disclosure; FIG. 15 is a diagram illustrating an operation example of the control device according to the first embodiment of the present disclosure; FIG. 16 is a diagram illustrating an example of button color transition according to the first embodiment of the present disclosure. FIG. 17 is a diagram illustrating an example of button color transition according to the first embodiment of the present disclosure. FIG. 18 is a diagram illustrating a configuration example of a control device according to a second embodiment of the present disclosure; FIG. 19 is a diagram illustrating a configuration example of a control device according to a third embodiment of the present disclosure;

Hereinafter, embodiments of the present disclosure will be described based on the drawings. The same symbols are attached to the same configurations in the following embodiments.

[Example 1]
<Configuration of object sorting system>
FIG. 1 is a diagram illustrating a configuration example of an object sorting system according to Example 1 of the present disclosure.

　In FIG. 1, the object sorting system 1 has a control device 10, a camera 20, an object sorting device 30, a belt conveyor 40, a push button 50, a display 60, and an input device 70. Push buttons 50 , display 60 and input device 70 are connected to control device 10 . Examples of the input device 70 include a pointing device such as a mouse and a keyboard. The control device 10, camera 20, and object sorting device 30 are connected to each other via a network.

In the following, a case where the object sorting system 1 shown in FIG. In other words, the case where the objects to be sorted by the object sorting system 1 are waste will be described below as an example. However, the object sorting system 1 may be installed in an assembly factory or the like where a group of parts flows on a belt conveyor. That is, the objects to be sorted by the object sorting system 1 are not limited to waste, and the object sorting system 1 can be used for various objects.

The belt conveyor 40 conveys the waste group placed on the belt conveyor 40 in the conveying direction CD. That is, the belt conveyor 40 forms a transport path along which the waste group is transported in the transport direction CD.

The camera 20 is arranged above the belt conveyor 40 on which the waste group is conveyed, has a predetermined angle of view, and continuously scans a predetermined area on the upper surface of the belt conveyor 40 from above the belt conveyor 40 at a constant frame rate. to shoot. Therefore, the image captured by the camera 20 (hereinafter sometimes referred to as "captured image") is the image of the waste group. A captured image is transmitted from the camera 20 to the control device 10 .

<Configuration of control device>
FIG. 2 is a diagram illustrating a configuration example of a control device according to the first embodiment of the present disclosure; A control device 10a shown in FIG. 2 corresponds to the control device 10 shown in FIG. 1, and an object sorting device 30a shown in FIG. 2 corresponds to the object sorting device 30 shown in FIG. 2, the control device 10a includes an image recognition unit 11, a recognition result storage unit 12, a recognition result extraction unit 13a, a label correction unit 14, a teacher data storage unit 15, a machine learning unit 16, a learned It has a model storage unit 17 and a button color control unit 18 .

The image recognition unit 11 uses the learned model stored in the learned model storage unit 17 to identify each waste image (hereinafter sometimes referred to as “waste image”) present in the captured image. is recognized, and information indicating characteristics of each waste image (hereinafter sometimes referred to as “feature information”) is added to each recognized waste image. The image recognition unit 11 assigns characteristic information to the waste image whose image recognition score is equal to or greater than the threshold TH1. The feature information includes information indicating the type of the waste image (hereinafter sometimes referred to as "type information"), information indicating the contour of the waste image (hereinafter sometimes referred to as "contour information"), and the coordinates of the area centroid of the waste image (hereinafter sometimes referred to as "centroid coordinates"). The image recognition unit 11 recognizes the waste image by, for example, performing instance segmentation on the captured image. The image recognition unit 11 associates the waste image and the feature information already assigned to the waste image with each other, and obtains a recognition result (hereinafter referred to as “image recognition result”), and outputs the generated image recognition result to the recognition result storage unit 12 and the object sorting device 30a. The recognition result storage unit 12 stores the image recognition results sequentially output from the image recognition unit 11 in chronological order. That is, the recognition result storage unit 12 stores captured images in which the waste image and the feature information are associated with each other. Further, the image recognition unit 11 outputs a signal (hereinafter sometimes referred to as a “recognition completion signal”) indicating completion of recognition of the waste image and addition of characteristic information to the waste image. Output to unit 18 .

The object sorting device 30a extracts the desired waste from the waste group conveyed on the belt conveyor 40 based on the image recognition result (that is, waste image and feature information) output from the image recognition unit 11. Select by The object sorting device 30a extracts the desired waste using, for example, a robot hand or a suction pad.

When the push button 50 is pressed, the recognition result extraction unit 13a extracts waste other than the desired waste (hereinafter referred to as “undesired waste”) from the photographed images stored in the recognition result storage unit 12. ), which includes an image of undesired waste erroneously sorted by the object sorting device 30a (hereinafter sometimes referred to as an ``erroneously sorted waste image'') (hereinafter referred to as an ``erroneously sorted waste image''). (sometimes referred to as "object existence image"). Further, the recognition result extracting unit 13a determines that the recognition result including the incorrectly sorted waste presence image is an erroneous recognition result. Then, the recognition result extracting unit 13a extracts the erroneously sorted waste presence image and the characteristic information given to each waste image in the erroneously sorted waste presence image from the recognition result storage unit 12 as the error recognition result. Extract. The recognition result extraction unit 13 a outputs the incorrect recognition result extracted from the recognition result storage unit 12 to the label correction unit 14 and displays it on the display 60 . In addition, the recognition result extraction unit 13a outputs a signal indicating that the extraction of the erroneous recognition result from the recognition result storage unit 12 has been completed (hereinafter sometimes referred to as an “extraction completion signal”) to the button color control unit 18. do.

The push button 50 is pressed by the operator, and the operator presses the push button 50 when the object sorting device 30a erroneously sorts undesired waste. When the push button 50 is pressed, a signal indicating that the push button 50 has been pressed (hereinafter sometimes referred to as a "button pressing signal") is sent from the push button 50 to the recognition result extraction unit 13a and the button color control unit. 18. The recognition result extracting unit 13a extracts an erroneous recognition result from the recognition result storage unit 12 when a button pressing signal is input in response to pressing of the push button 50. FIG. The push button 50 is an example of an interface (hereinafter sometimes referred to as an "erroneous sorting input interface") that receives an input indicating that the object sorting device 30a has erroneously sorted unwanted waste. A software button displayed on a touch panel, for example, can be used as an erroneous sorting input interface other than the push button 50 . That is, the recognition result extracting unit 13a extracts the erroneous recognition result from the recognition result storage unit 12 when an input indicating that the object sorting device 30a has erroneously sorted undesired waste is received in the erroneous sorting input interface.

The input device 70 is operated by the operator, and the operator can use the input device 70 to specify any part of the incorrectly sorted waste existence image displayed on the display 60 . The label correcting unit 14 enables the operator to correct the label information using the input device 70 when the arbitrarily specified portion in the image of the erroneously sorted waste is the display portion of the label information. The label correcting unit 14 outputs a recognition result (hereinafter sometimes referred to as “post-correction recognition result”) after correcting the label information in the misrecognition result to the teacher data storage unit 15 .

The training data storage unit 15 stores in advance many waste images to which feature information is added as training data. Further, the teacher data storage unit 15 newly stores the post-correction recognition result output from the label corrector 14 as additional teacher data.

The machine learning unit 16 performs machine learning using the teacher data stored in the teacher data storage unit 15, and converts the learned model stored in the learned model storage unit 17 to the learned model after machine learning. Update. Therefore, the recognition of the waste image by the image recognition unit 11 is performed using the learned model updated by additional machine learning performed by the machine learning unit 16 using the post-correction recognition result. .

On the other hand, the object sorting device 30a outputs a signal indicating the operating state of the object sorting device 30a (hereinafter sometimes referred to as a "sorting state signal") to the button color control section 18.

The button color control unit 18 changes the color of the push button 50 (hereinafter sometimes referred to as "button color") based on the button depression signal, the recognition completion signal, the extraction completion signal, and the sorting state signal. Let The push button 50 is configured using, for example, an LED (Light Emitting Diode), and the button color can be changed.

<Feature information>
FIG. 3 is a diagram illustrating an example of feature information according to Example 1 of the present disclosure. In the following, empty bottles are assumed as waste, and in the object sorting device 30a, each empty bottle flowing on the belt conveyor 40 is a brown empty bottle (hereinafter sometimes referred to as a "brown bottle") and a brown bottle. It is assumed that the bins are sorted into two types of empty bins, ie, empty bins having colors other than those (hereinafter sometimes referred to as “other colors”) (hereinafter sometimes referred to as “other color bins”). In the following description, it is assumed that the desired waste is a brown bottle and training data for the brown bottle is stored in the training data storage unit 15. FIG.

As shown in FIG. 3, when the captured image includes an image of an empty bin (hereinafter sometimes referred to as an “empty bin image”) BI as a waste image, the image recognition unit 11 stores the learned model storage unit 17 recognizes an empty bin image BI existing in the photographed image using the learned model stored in the . Give the feature information that contains. When the empty bin image BI is an image of a brown bin (hereinafter sometimes referred to as a “brown bin image”), the image recognition unit 11 recognizes the empty bin image as label information LA with the type information “brown bin”. Give to BI. On the other hand, if the empty bin image BI is an image of a bin of another color (hereinafter sometimes referred to as a "bin image of another color"), the image recognition unit 11 labels the type information "bin of another color". The information LA is added to the empty bin image BI. Further, the image recognition unit 11 adds outline information CO to the empty bin image BI in the captured image. The outline information CO is formed by a plurality of coordinate points (x0, y0), (x1, y1), . . Further, the image recognition unit 11 calculates the area barycenter of the empty bin image BI in the captured image, and assigns the barycenter coordinate DG to the empty bin image BI. A barycentric coordinate DG is formed by a single coordinate point (X, Y).

Here, a plurality of coordinate points (x0, y0), (x1, y1), . is a coordinate point in the coordinate system of the captured image, that is, the coordinate system of the camera 20 (hereinafter sometimes referred to as the “camera coordinate system”).

<Trace of barycentric coordinates>
4, 5, 6, and 7 are diagrams showing trace examples of barycentric coordinates according to the first embodiment of the present disclosure.

First, as shown in FIG. 4, the image recognition unit 11 recognizes the brown bin image BO in the captured image acquired at time t11, and calculates the barycentric coordinates DG1 of the brown bin image BO at time t11.

Next, as shown in FIG. 5, the image recognition unit 11 acquires at time t11 based on the frame rate FR [frame/sec] of the camera 20 and the conveying speed CS [mm/sec] of the belt conveyor 40. The barycentric coordinates EG2 of the brown bin image BO in the captured image that will be acquired at time t12 next to the captured image are predicted. The barycentric coordinate EG2 at time t12 is predicted to move CS/FR per frame in the conveying direction CD of the belt conveyor 40 (that is, the X direction in FIG. 1) with respect to the barycentric coordinate DG1 at time t11. Further, the image recognition unit 11 sets a circular allowable range TR of a predetermined size centered on the predicted barycentric coordinate EG2.

Next, as shown in FIG. 6, the image recognition unit 11 recognizes the brown bin image BO in the captured image acquired at time t12, and calculates the barycentric coordinates DG2 of the brown bin image BO at time t12. Further, the image recognition unit 11 determines whether or not the barycentric coordinate DG2 is within the allowable range TR.

When the barycentric coordinate DG2 is within the permissible range TR, the image recognition unit 11 determines that the subject of both the brown bin images BO at time t11 and time t12 is the same brown bin, and as shown in FIG. , a straight trace line TL connecting from the barycentric coordinate DG1 to the barycentric coordinate DG2 is set. On the other hand, when the barycentric coordinate DG2 is not within the allowable range TR, the image recognition unit 11 stops tracing the barycentric coordinate.

The image recognition unit 11 sequentially traces the barycentric coordinates as described above for the captured images that are sequentially acquired at the frame rate FR.

<Extraction of desired waste>
FIG. 8 is a diagram illustrating an example of extraction of desired waste in Example 1 of the present disclosure. In FIG. 8, the frame rate FR of the camera 20 is based on, for example, the conveying speed CS of the belt conveyor 40 and the angle of view of the camera 20, and the same empty bottle conveyed in the conveying direction CD can be photographed three times at maximum. frame rate is set. The object sorting device 30a also uses the suction pad 31 to extract brown bottles, which are desired waste, from the empty bottle group.

In FIG. 8, the image recognition unit 11 recognizes the brown bin image BO in the captured image acquired at time t11, and calculates the barycentric coordinates DG1 (X1, Y1) of the brown bin image BO at time t11.

Next, the image recognition unit 11 recognizes the brown bin image BO in the captured image acquired at time t12, and calculates barycentric coordinates DG2 (X2, Y2) of the brown bin image BO at time t12.

Next, the image recognition unit 11 determines that the subjects of both the brown bin images BO at time t11 and time t12 are the same brown bin, and determines the coordinates DG2 (X2, Y2) from the coordinates DG1 (X1, Y1) to the coordinates DG2 (X2, Y2). A straight trace line TL connecting to is set.

Next, the image recognition unit 11 recognizes the brown bin image BO in the captured image acquired at time t13, and calculates barycentric coordinates DG3 (X3, Y3) of the brown bin image BO at time t13.

Next, the image recognition unit 11 determines that the subjects of both the brown bin images BO at time t12 and time t13 are the same brown bin, and the coordinates of the center of gravity DG1 (X1, Y1) to the coordinates of the center of gravity DG3 (X3, Y3) are determined. A straight trace line TL connecting to is set.

Next, the image recognition unit 11 extends the trace line TL connecting from the barycentric coordinates DG1 (X1, Y1) to the barycentric coordinates DG3 (X3, Y3) to the installation position of the suction pad 31 in the X direction by linear approximation, and By converting the camera coordinate system into the coordinate system of the object sorting device 30a, extraction target coordinates TC(X, Y) of the brown bin are calculated. Based on the transport speed CS, the image recognition unit 11 also determines the times (hereinafter referred to as the Then, tH, which may be called "extraction target time", is calculated. The image recognition unit 11 transmits a control signal including the calculated extraction target coordinates TC(X, Y) and the extraction target time tH to the object sorting device 30a.

In the object sorting device 30a, the suction pad 31 moves to the extraction target coordinates TC(X,Y) according to the extraction target coordinates TC(X,Y) and the extraction target time tH received from the image recognition unit 11, and the extraction target At the time tH, an empty bin positioned at the extraction target coordinates TC(X,Y) is extracted from the empty bin group. As a result, in the object sorting device 30a, brown bottles, which are subjects of the brown bottle images BO recognized by the image recognition unit 11 at times t11, t12, and t13, are extracted from the empty bottle group.

<Operation of the control device>
9 to 15 are diagrams showing operation examples of the control device according to the first embodiment of the present disclosure.

In FIG. 9, the empty bins BOa, BOb, BOc, BOd, BOe, and BOf are sequentially conveyed by the belt conveyor 40 in the conveying direction CD from time ta to time ti. The empty bottle BOa is a brown bottle that is the desired waste, and the empty bottles BOb, BOc, BOd, BOe, and BOf are non-desired waste of other colors.

As the empty bin group is transported, the camera 20 acquires a photographed image CIa that does not include the image of the empty bin at time ta, and includes the brown bin image BIa that is the image of the brown bin BOa at time tb. A photographed image CIb is acquired, at time tc a photographed image CIc including a brown bin image BIa and another color bin image BIb that is an image of the other color bin BOb is acquired, and at time td, the brown bin image BIa and , the other-color bin image BIb and the other-color bin image BIc, which is the image of the other-color bin BOc. A photographed image CIe including the other-color bin image BId, which is the image of the other-color bin BOd, is acquired. At time tf, the camera 20 captures the other-color bin image BIc, the other-color bin image BId, and the other-color bin image BIe, which is the image of the other-color bin BOe, along with the transportation of the empty bin group. A photographed image CIf is acquired, and at time tg, a photographed image CIg including the other-color bin image BId, the other-color bin image BIe, and the other-color bin image BIf, which is the image of the other-color bin BOf, is acquired. At th, a photographed image CIh containing the other color bin image BIe and the other color bin image BIf is acquired, at time ti a photographed image CIi containing the other color bin image BIf is acquired, and at time tj an empty bin is acquired. A captured image CIj that does not include the image of is acquired. Therefore, at the time ti, the captured images CIa, CIb, CIc, CId, CIe, CIf, CIg, CIh, and CIi to which the feature information is assigned are stored in the recognition result storage unit 12 at the respective shooting times ta, tb, They are stored in one-to-one correspondence with tc, td, te, tf, tg, th, and ti.

Here, at time tb, the image recognition unit 11 correctly recognizes the brown bin image BIa as a brown bin image. Therefore, at time tf, the object sorting device 30a uses the suction pad 31 to extract the brown bottle BOa from the empty bottle group, and at time tg, moves the suction pad 31 to the release position and removes the brown bottle BOa from the suction pad 31 at time tg. to release.

On the other hand, at time td, the image recognition unit 11 erroneously recognizes the other-color bin image BIc as a brown bin image. Therefore, at time th, the object sorting device 30a uses the suction pad 31 to extract the other-color bin BOc from the empty bin group, and at time ti, moves the suction pad 31 to the release position to remove the other-color bin BOc from the suction pad 31 at time ti. Release bin BOc. That is, as a result of the image recognition unit 11 erroneously recognizing the other-color bin image BIc as a brown bin image, the object sorting device 30a erroneously sorts the other-color bin BOc as a brown bin.

At the time ti when the other color bottle BOc is released from the suction pad 31, the operator presses the push button 50 when he/she notices that the other color bottle BOc, which is not the brown bottle, has been erroneously sorted by the object sorting device 30a. Therefore, at time ti, a button depression signal is output from the push button 50 to the recognition result extraction unit 13a and the button color control unit 18. FIG.

The recognition result extraction unit 13a determines, at time ti, an erroneous recognition result including the other-color bin image BIc, which is an erroneously sorted waste image, in response to the button press signal. The recognition result extraction unit 13a extracts images stored in the recognition result storage unit 12 based on the frame rate FR, the distance XD in the X direction between the camera 20 and the object sorting device 30a, and the transport speed CS. Of the images CIa, CIb, CIc, CId, CIe, CIf, CIg, CIh, and CIi, the photographed images CId, CIe, and CIf associated with the times td, te, and tf, respectively, are the missorted waste presence images. judge. For example, the recognition result extracting unit 13a determines that the time te before the time ti at which the push button 50 was pressed by XD/CS [sec] is the time te, and ((XD/CS)-(1/ The time FR)) [sec] earlier is determined as time td, and the time earlier than time ti by ((XD/CS)+(1/FR)) [sec] is determined as time tf. Then, the recognition result extracting unit 13a selects the captured images CIa, CIb, CIc, CId, CIe, CIf, CIg, CIh, and CIi stored in the recognition result storage unit 12 at the time ti, at the times td, te, Recognition results including captured images CId, CIe, and CIf corresponding to each time of tf are extracted as erroneous recognition results. The recognition result extraction unit 13a outputs the photographed images CId, CIe, and CIf extracted from the recognition result storage unit 12 together with the feature information to the label correction unit 14, and causes the display 60 to display them together with the outline represented by the outline information.

Since the image recognition unit 11 correctly recognized the brown bin image BIa as a brown bin image at time tb, as shown in FIG. , and correct contour information COa1. Further, since the image recognition unit 11 correctly recognized the other-color bin image BIb as the other-color bin image at time tc, as shown in FIG. and feature information RRb1 including correct label information LAb and correct contour information COb1. On the other hand, since the image recognition unit 11 erroneously recognizes the other-color bin image BIc as a brown bin image at time td, as shown in FIG. Feature information RRc1 including erroneous label information LAc and correct outline information COc1 is provided.

Similarly, as shown in FIG. 11, in the photographed image CIe, the other-color bin image BIb is given feature information RRb2 including correct label information LAb of "other-color bin" and correct contour information COb2. Feature information RRc2 including incorrect label information LAc of "brown bin" and correct contour information COc2 is added to the color bin image BIc. Further, since the image recognition unit 11 correctly recognized the other-color bin image BId as the other-color bin image at time te, as shown in FIG. and feature information RRd1 including correct label information LAd and correct contour information COd1.

Similarly, as shown in FIG. 12, in the photographed image CIf, the other-color bin image BIc is given feature information RRc3 including erroneous label information LAc of "brown bin" and correct contour information COc3. Feature information RRd2 including correct label information LAd of "other color bin" and correct outline information COd2 is added to the color bin image BId. Further, since the image recognition unit 11 correctly recognized the other-color bin image BIe as the other-color bin image at time tf, as shown in FIG. and feature information RRe1 including correct label information LAe and correct contour information COe1.

Fig. 13 shows a display example of misrecognition results including missorted waste images. For example, the display 60 displays the photographed image CI as shown in FIG. 13 extracted from the recognition result storage unit 12 by the recognition result extraction unit 13a. The captured image CI includes empty bin images BI1, BI2, BI3, BI4, and BI5. Empty bin image BI1 is surrounded by a contour represented by contour information CO1, empty bin image BI2 is surrounded by a contour represented by contour information CO2, and empty bin image BI3 is surrounded by a contour represented by contour information CO3. , the empty bin image BI4 is surrounded by the outline represented by the outline information CO4, and the empty bin image BI5 is surrounded by the outline represented by the outline information CO5.

Here, it is assumed that the correct color of empty bin images BI1, BI3, BI4 is brown, and the correct color of empty bin images BI2, BI5 is another color. Therefore, the label information LA1, LA3, LA4 of "brown bottle" given to the empty bin images BI1, BI3, and BI4, respectively, and the label information LA2 of "another color bottle" given to the empty bin image BI2 are correct. Therefore, the label information LA5a of "brown bottle" given to the empty bottle image BI5 is incorrect. That is, the empty bin image BI5, which is the other-color bin image, corresponds to the incorrectly sorted waste image, and the photographed image CI corresponds to the incorrectly sorted waste existence image including the empty bin image BI5, which is the incorrectly sorted waste image.

Therefore, for example, the operator operates the input device 70 to use the pointer PO displayed on the photographed image CI, as shown in FIG. ”, the label information LA5a of the empty bin image BI5 is specified by clicking the display position of the label information LA5a. When the label information LA5a is specified by operating the input device 70, the label correction unit 14 enables correction of the specified label information LA5a. Therefore, the operator uses, for example, a keyboard to correct the label information given to the empty bin image BI5 from "brown bin" to "other color bin" as shown in FIG. According to the correction by the operator, the label correcting unit 14 replaces the incorrect label information LA5a of "brown bottle" attached to the empty bin image BI5 with label information LA5b of "other color bin" indicating the correct type of the empty bin image BI5. to be corrected. In this manner, erroneous label information included in the erroneous recognition result is corrected by the label corrector 14. FIG.

Note that the minimum required storage capacity of the recognition result storage unit 12 can be calculated according to equation (1). In equation (1), "Ts" is the time [sec] required from the time the brown bottle enters the angle of view of the camera 20 until the brown bottle sucked to the suction pad 31 is released from the suction pad 31. , "Tj" is the time [sec] required for the operator to press the push button 50 after the erroneously sorted bins of other colors are released from the suction pad 31, and "FR" is the frame rate of the camera 20. and "DS" is the data size [MB] per photographed image.
Storage capacity of recognition result storage unit 12≧(Ts+Tj)/FR×DS (1)

<Button color transition>
16 and 17 are diagrams illustrating examples of button color transitions according to the first embodiment of the present disclosure. As shown in FIGS. 16 and 17, the button color transitions between green, red, and yellow, for example.

In FIG. 16, when the image recognition unit 11 is in the recognition waiting state from when the image recognition unit 11 outputs the previous recognition completion signal until when it outputs the next recognition completion signal, the button color control unit 18 does not extract the erroneous recognition result. Set the button color to green, which indicates that reception is possible. When the button color is green, pressing the push button 50 is valid.

When a button depression signal is input by pressing the push button 50 in the green state, the button color control unit 18 determines that an instruction to start extraction of erroneous recognition results has been given by the operator, and changes the button color to green. to yellow. When the button color is yellow, pressing the push button 50 is valid.

When a button pressing signal is input by pressing the yellow push button 50, the button color control unit 18 determines that the operator has instructed to stop extracting the erroneous recognition result, and changes the button color to yellow. to green. Alternatively, the button color control unit 18 changes the button color from yellow to green when an extraction completion signal is output from the recognition result extraction unit 13a without the push button 50 being pressed while the button color is yellow. Let

Further, the button color control unit 18 changes the button color from green to red when the recognition completion signal is output from the image recognition unit 11 without the push button 50 being pressed while the button color is green. . When the button color is red, pressing the push button 50 is invalid. After completion of recognition by the image recognition unit 11, the button color control unit 18 keeps the button color red until the movement of the suction pad 31 to the target position corresponding to the extraction target coordinates is completed. The button color control unit 18 determines the state of the suction pad 31 based on the sorting state signal output from the object sorting device 30a.

Further, the button color control unit 18 keeps the button color red until the desired waste reaches the target position after the suction pad 31 has completely moved to the target position corresponding to the extraction target coordinates. Further, when the button color control unit 18 determines based on the sorting state signal that the suction pad 31 has failed to suck the desired waste, it changes the button color from red to green. On the other hand, the button color control unit 18 maintains the button color to red when it is determined based on the sorting state signal that the suction pad 31 has successfully adsorbed the desired waste. The button color control unit 18 keeps the button color red while the suction pad 31 is moving to the desired waste release position.

Then, when the button color control unit 18 determines based on the sorting state signal that the suction pad 31 has completed releasing the desired waste at the release position, it changes the button color from red to green.

FIG. 17 shows the relationship between button color transitions and task transitions in the control device 10a.　　　

In FIG. 17, empty bottles BOf and BOg are non-desired waste bottles of other colors, and empty bottle Boh is a desired waste brown bottle. It is also assumed that the empty bottle BOg is correctly recognized as a bottle of another color, the empty bottle Boh is correctly recognized as a brown bottle, and the empty bottle Bof is erroneously recognized as a brown bottle. Therefore, the control device 10a generates a task TA1 for the empty bin BOf and a task TA2 for the empty bin Boh. A new task is generated when an empty bin whose barycentric coordinates have been successfully traced is out of the angle of view of the camera 20 . Also, the existing task is deleted when the empty bottle is released from the suction pad 31 or when the suction pad 31 fails to suck the empty bottle.

When there is no task at time t21, the button color control unit 18 sets the button color to green.

Next, at time t22, when the task TA1 for the empty bin BOf is generated, the button color control section 18 sets the button color to green. Further, the button color control unit 18 sets the button color to green until time t24 when the task TA2 for the empty bin BOh is generated.

Next, at time t24, when the task TA2 for the empty bin BOh is generated, the button color control section 18 changes the button color from green to red.

Next, when task TA1 is deleted at time t25, the button color control unit 18 changes the button color from red to green.

Next, at time t26 after a predetermined period of time has elapsed since task TA1 was deleted, the button color control unit 18 changes the button color from green to red.

Then, when the task TA2 is deleted at time t27, the button color control unit 18 changes the button color from red to green.

When software buttons displayed on the touch panel are used instead of the push buttons 50 as the misselection input interface, the button color control unit 18 can change the color of the software buttons in the same manner as described above. be.

The first embodiment has been described above.

[Example 2]
<Configuration of control device>
FIG. 18 is a diagram illustrating a configuration example of a control device according to a second embodiment of the present disclosure; A control device 10b shown in FIG. 18 corresponds to the control device 10 shown in FIG. 1, and an object sorting device 30b shown in FIG. 18 corresponds to the object sorting device 30 shown in FIG. However, the push button 50 shown in FIGS. 1 and 2 is not connected to the control device 10b. Further, the object sorting device 30b differs from the object sorting device 30a shown in FIG. 2 in that it does not output a sorting state signal. 18, the control device 10b includes an image recognition unit 11, a recognition result storage unit 12, a recognition result extraction unit 13b, a label correction unit 14, a teacher data storage unit 15, a machine learning unit 16, a learned and a model storage unit 17 .

"Usually, the size of the brown bottle is often smaller than the specified size." Further, in the object sorting device 30b, an empty bottle corresponding to the empty bottle image to which the label information "brown bottle" is added is sorted as the desired waste. Therefore, the recognition result extraction unit 13b extracts empty bin images having an area equal to or larger than the threshold value TH2 among the empty bin images to which the label information “brown bin” is assigned in the photographed images stored in the recognition result storage unit 12. is determined to be a missorted waste image. Then, the recognition result extracting unit 13b extracts the photographed image including the empty bin image having the area equal to or larger than the threshold value TH2 to which the label information "brown bottle" is assigned, from the recognition result storage unit 12 as the false recognition result together with the feature information. Extract.

The second embodiment has been described above.

[Example 3]
<Configuration of control device>
FIG. 19 is a diagram illustrating a configuration example of a control device according to a third embodiment of the present disclosure; A control device 10c shown in FIG. 19 corresponds to the control device 10 shown in FIG. 1, and an object sorting device 30c shown in FIG. 19 corresponds to the object sorting device 30 shown in FIG. However, the push button 50 shown in FIGS. 1 and 2 is not connected to the control device 10c.

19, the control device 10c includes an image recognition unit 11, a recognition result storage unit 12, a recognition result extraction unit 13c, a label correction unit 14, a teacher data storage unit 15, a machine learning unit 16, a learned and a model storage unit 17 . The object sorting device 30 c has a suction pad 31 . The suction pad 31 has a weight sensor for detecting the weight of the sucked empty bin, and the object sorting device 30c receives information indicating the weight of the empty bin sucked by the suction pad 31 (hereinafter referred to as "weight information"). ) is output to the recognition result extraction unit 13c.

Usually, the weight of the brown bottle is often less than the specified weight. Further, in the object sorting device 30c, an empty bottle corresponding to the empty bottle image to which the label information "brown bottle" is added is sorted as the desired waste. Therefore, based on the weight information, the recognition result extracting unit 13c selects empty bin images with the label information of “brown bottle” from among the photographed images stored in the recognition result storage unit 12, and extracts images having weights equal to or greater than the threshold TH3. Empty bin images corresponding to heavy empty bins are determined to be missorted waste images. Then, the recognition result extracting unit 13c recognizes, together with the feature information, the photographed image including the empty bin image corresponding to the empty bin having the label information “brown bottle” and having a weight equal to or greater than the threshold TH3 as the result of misrecognition. Extract from the result storage unit 12 .

The third embodiment has been described above.

[Example 4]
The recognition result storage unit 12, the teacher data storage unit 15, and the learned model storage unit 17 are implemented as hardware, for example, by memory or storage. The image recognition unit 11, the recognition result extraction units 13a, 13b, 13c, the label correction unit 14, the machine learning unit 16, and the button color control unit 18 include hardware such as a CPU (Central Processing Unit), a DSP (Digital signal processor), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit).

The fourth embodiment has been described above.

As described above, the object sorting system of the present disclosure (object sorting system 1 of the embodiment) includes a camera (camera 20 of the embodiment), a recognition unit (image recognition unit 11 of the embodiment), a storage unit ( (recognition result storage unit 12), a selection unit (

object selection devices

30, 30a, 30b, 30c of the embodiment), and an extraction unit (recognition result extraction units 13a, 13b, 13c of the embodiment). The camera captures a first image, which is an image of the object group, on the transport path along which the object group is transported. The recognition unit recognizes a second image that is an image of each object present in the first image using the trained model, adds feature information that is information indicating the feature of the second image to the second image, A recognition result is generated in which feature information is added to the second image in the first image. The storage unit stores recognition results. The sorting unit sorts out a desired object from the group of objects conveyed on the conveying path based on the recognition result. When an undesired object other than a desired object is erroneously sorted by the sorting unit, the extracting unit extracts an erroneous recognition that is a recognition result including an image of the erroneously selected undesired object in the recognition results stored in the storage unit. The result is determined, and an erroneous recognition result is extracted from the storage unit. For example, the feature information includes information indicating the type of each object in the object group and information indicating the outline of each object in the object group.

By doing so, the operator can generate teacher data by correcting the erroneous recognition results based on the erroneous recognition results extracted from the storage unit. can reduce the effort required to generate teacher data compared to the case of performing

Further, the object sorting system of the present disclosure has an interface that receives an input indicating that the undesired object has been erroneously sorted by the sorting unit. to extract

Alternatively, the extracting unit extracts an erroneous recognition result from the storage unit based on the area of the second image.

Alternatively, the extraction unit extracts the recognition error result from the storage unit based on the weight of the desired object sorted by the sorting unit.

By doing this, misrecognition results can be extracted accurately.

In addition, the interface is a button (push button 50 in the embodiment) that accepts input from the operator, and the object sorting system of the present disclosure controls the color of the button to change according to the extraction status of the erroneous recognition result by the extraction unit. have a part.

By doing this, the operator can be notified of the extraction status of the misrecognition results.

In addition, the extraction unit determines an erroneous recognition result based on the frame rate of the camera, the distance between the camera and the sorting unit, and the transportation speed of the object group on the transportation path.

By doing this, it is possible to accurately judge the misrecognition result.

1 object sorting system 10, 10a, 10b, 10c control device 20

cameras

30, 30a, 30b, 30c object sorting device 50 push button 11 image recognition unit 12 recognition result storage unit 13a, 13b, 13c recognition result extraction unit 14 label correction unit 15 Teacher data storage unit 16 Machine learning unit 17 Learned model storage unit 18 Button color control unit

Claims

a camera that captures a first image that is an image of the object group on a transport path along which the object group is transported;
Recognizing a second image that is an image of each object present in the first image using the trained model, adding feature information that is information indicating a feature of the second image to the second image, a recognition unit that generates a recognition result in which the feature information is added to the second image in the first image;
a storage unit that stores the recognition result;
a selection unit that selects a desired object from the object group transported on the transport path based on the recognition result;
When an undesired object other than the desired object is erroneously sorted by the sorting unit, an erroneous recognition result that includes an image of the erroneously selected undesired object in the recognition results stored in the storage unit. an extraction unit that determines a recognition result and extracts the erroneous recognition result from the storage unit;
An object sorting system comprising:
an interface that receives an input indicating that the undesired object has been erroneously sorted by the sorting unit;
The extraction unit extracts the erroneous recognition result from the storage unit when the interface receives the input.
The object sorting system of claim 1.
the interface is a button that accepts the input from an operator;
A control unit that changes the color of the button according to the extraction status of the erroneous recognition result by the extraction unit.
3. An object sorting system according to claim 2.
The extracting unit extracts the erroneous recognition result from the storage unit based on the area of the second image.
The object sorting system of claim 1.
The extraction unit extracts the erroneous recognition result from the storage unit based on the weight of the desired object sorted by the sorting unit.
The object sorting system of claim 1.
The extraction unit determines the false recognition result based on the frame rate of the camera, the distance between the camera and the sorting unit, and the transport speed of the object group on the transport path.
The object sorting system of claim 1.
The feature information includes information indicating the type of each object in the object group and information indicating the contour of each object in the object group,
The object sorting system of claim 1.