JP7455010B2 - Detection device and detection program - Google Patents

Description

One aspect of the present invention relates to a detection device and a detection program.

As a device that detects articles stored in a product box transported by a conveyor or other conveyance means, the device detects an X-ray transmission image of the top surface of the product box, an X-ray transmission image of the side surface of the product box, and the items prepared in advance. A device is known that counts articles stored in a product box based on a reference image of (see, for example, Patent Document 1).

In addition, the presence or absence of overlap between multiple workpieces (articles) transported by a conveyor is detected based on a 3D image of the workpieces being transported taken by a 3D camera and the 3D shape of the reference workpiece. A device is known that does this (for example, see Patent Document 2).

Japanese Patent Application Publication No. 2016-141541 Japanese Patent Application Publication No. 2019-111640

By the way, in a sorter system for sorting products (articles) installed in a distribution center or the like, there are cases where the products to be sorted must be transported one by one by a conveyor. In this case, if multiple products are placed on one tray flowing on the conveyor (so-called double-loading condition) due to a work error, etc., In order to prevent products from being delivered to the wrong destination, it is necessary to promptly detect the tray.

According to the mechanisms described in Patent Documents 1 and 2 mentioned above, there is a possibility that such an abnormal state (double loading state) can be detected, but the device disclosed in Patent Document 1 mentioned above has two An X-ray imaging system (an X-ray generator, an X-ray detection sensor such as a line sensor, etc.) is required to acquire a directional X-ray transmission image. Therefore, there is a problem that the equipment becomes larger and the equipment cost becomes relatively high. The device disclosed in Patent Document 2 also requires a three-dimensional camera to take a three-dimensional image of the article, and therefore has the same problem as the device in Patent Document 1.

One aspect of the present invention is to provide a detection device and a detection program that can detect abnormal conditions related to article transportation with simple equipment.

A detection device according to one aspect of the present invention includes an acquisition unit that acquires a tray image obtained by photographing a tray conveyed from above with a polygonal article placed thereon in a plan view; A recognition unit that detects the vertices of the items included in the tray image by performing image recognition on the tray image, and a recognition unit that detects the vertices of the items included in the tray image, and determines whether multiple items are placed on the tray based on the detection results of the vertices by the recognition unit. and a determination unit that determines.

In the detection device according to one aspect of the present invention, the apex (corner) of the article included in the tray image is detected from the image recognition result of the image of the tray conveyed with the article placed thereon (tray image). be done. Then, based on the detection result of the vertices (for example, the number of vertices), it is determined whether or not a plurality of articles are placed on the tray (so-called double-loading condition). According to the above-mentioned detection device, an abnormal state (the above-mentioned double loading state) can be detected by relatively simple analysis processing of an image photographed from above the tray. That is, abnormal conditions related to article transportation can be detected with simple equipment.

According to one aspect of the present invention, it is possible to provide a detection device and a detection program that can detect abnormal conditions related to article transportation with simple equipment.

1 is a diagram showing the configuration of a detection system including a detection device according to an embodiment. FIG. 3 is a diagram for explaining an example of a tray image acquisition method. It is a figure which shows an example of a tray image and the image recognition result with respect to a tray image. It is a figure which shows the 1st example of a double loading state. It is a figure which shows the 2nd example of a double loading state. It is a figure which shows the 3rd example of a double loading state. It is a figure which shows the 4th example of a double loading state. FIG. 7 is a diagram illustrating an example of a case where a vertex is erroneously detected. It is a flowchart which shows an example of operation of a detection device. 3 is a flowchart illustrating an example of a procedure of a determination process performed by a determination unit. It is a diagram showing an example of the hardware configuration of a detection device. FIG. 3 is a diagram showing the configuration of a detection program stored in a recording medium.

EMBODIMENT OF THE INVENTION Hereinafter, one embodiment of the present invention will be described in detail with reference to the accompanying drawings. In addition, in the description of the drawings, the same or equivalent elements are given the same reference numerals, and redundant description will be omitted.

FIG. 1 is a diagram showing the configuration of a detection system 1 including a detection device 10 according to one embodiment. The detection system 1 is used, for example, in a logistics center. The detection system 1 includes the detection device 10, a sorter system 20, and a camera 30.

The sorter system 20 is a system that sorts products G (articles). Product G has a polygonal shape in plan view. The product G is not limited to a specific product, but may be, for example, a smartphone, a tablet terminal, etc. housed in a package (box). Typically, a package is often formed into a rectangular parallelepiped shape. Further, the outer shape of the product G is similar to the shape of the package. Therefore, in this case, the product G has a rectangular parallelepiped shape and a quadrangular shape in plan view. In the following description, it is assumed that the product G has a rectangular shape in plan view.

Sorter system 20 includes a conveyor 21 and a plurality of trays 22 disposed on and flowing over conveyor 21 . The products G to be sorted are placed on the tray 22. The conveyor 21 conveys the tray 22 with the product G placed thereon. Note that empty trays 22 on which no products G are placed are also flowing on the conveyor 21. The conveyance speed of the tray 22 by the conveyor 21 is, for example, about 85 m/min, which is relatively high.

In the sorter system 20, in order to properly sort the products G, only one product G needs to be placed on one tray 22. However, due to, for example, a work error when placing the product G on the tray 22, an abnormal situation (hereinafter referred to as , "double loading") may occur. If such double loading occurs, there is a risk that the product G will be sorted incorrectly, resulting in a problem (erroneous delivery) in which the product G is delivered to the wrong destination (for example, a store, etc.).

Therefore, the detection system 1 includes a detection device 10 and a camera 30 as a system for detecting the double loading described above. The camera 30 is installed above the conveyor 21, for example, and photographs the tray 22 conveyed by the conveyor 21 from above.

As shown in FIG. 2, the camera 30 is configured to photograph the tray 22 passing through a preset photographing area SA on the conveyor 21. The camera 30 is, for example, a trigger camera. For example, when a sensor (not shown) or the like detects that the entire tray 22 has entered the photographing area SA, the camera 30 photographs the photographing area SA based on a detection signal from the sensor. Through such photographing processing, a tray image IM1 as shown on the left side of FIG. 3 is obtained. The tray image IM1 is a still image that includes the entire tray 22 as a subject. As shown on the left side of FIG. 3, the tray 22 is provided with an ID area in which a tray ID for identifying the tray 22 is written. In the example of FIG. 3, "101" is the tray ID.

Note that the tray image IM1 does not necessarily need to include the entire tray 22 as a subject. The tray image IM1 includes, as a subject, an area on the tray 22 where the product G is scheduled to be placed (for example, if the tray 22 is formed in a box shape, an area inside the box). It's fine as long as you are. Further, the sorter system 20 may include only one camera 30, or may include a plurality of cameras 30 each having a different photographing area SA.

The detection device 10 detects double loading by capturing the tray image IM1 photographed by the camera 30 and analyzing the tray image IM1. As shown in FIG. 1, the detection device 10 includes an acquisition section 11, a recognition section 12, and a determination section 13.

The acquisition unit 11 acquires a tray image IM1 (see the left part of FIG. 3) obtained by photographing the tray 22 conveyed with the product G placed thereon from above. In this embodiment, the acquisition unit 11 acquires the tray image IM1 photographed by the camera 30. The acquisition unit 11 may acquire the tray image IM1 directly from the camera 30, or may acquire the tray image IM1 once recorded on a recording medium different from the camera 30.

The recognition unit 12 detects at least the vertex of the product G included in the tray image IM1 by performing image recognition on the tray image IM1. In the present embodiment, the recognition unit 12 detects, in addition to the vertices of the product G, a product area (article area) and the above-mentioned ID area. As shown on the right side of FIG. 3, the recognition unit 12 detects the ID area, the product area, and the vertices of the product G by, for example, performing object detection processing, edge detection processing, and the like. The result image IM2 shown on the right side of FIG. 3 is a superimposed display of the recognition result by the recognition unit 12 on the tray image IM1. Specifically, the resultant image IM2 is an image in which the ID area A1, product area A2, and vertex area A3 (area including the vertex of the product G) detected by the recognition unit 12 are displayed superimposed on the tray image IM1. .

The ID area A1 is an area that includes a portion of the tray 22 where a tray ID (“101” in the example of FIG. 3) is written. The ID area A1 is set, for example, as a rectangular area.

The product area A2 is an area that includes the whole of one or more products G placed on the tray 22. The product area A2 is set, for example, as a rectangular area. For example, the product area A2 is set as the smallest rectangular area that includes the whole of one or more products G placed on the tray 22. Alternatively, the product area A2 may be set so that each side of the product area A2 and the product G do not touch each other (for example, so that a gap of a predetermined length is created between each side and the product G). However, since the product area A2 is an area determined as a result of image recognition, it is not necessarily set as predetermined as described above. That is, a part of the product G may protrude outside the product area A2. The fact that the area detected by the recognition unit 12 and the actual area may differ in this way also applies to areas other than the product area A2.

The vertex area A3 is an area including the vertices of the product G (in this embodiment, portions corresponding to the four corners of the package of the product G). The vertex area A3 is set, for example, as a rectangular area. The vertex area A3 is set, for example, as a rectangular area of a predetermined size centered on the vertex.

The positions of the ID area A1, product area A2, and vertex area A3 described above are specified, for example, by coordinates corresponding to the four corners of each area A1, A2, and A3. Different class names (classification information) are associated with each of the areas A1, A2, and A3. For example, the ID area A1 is associated with a class name "id" indicating an ID area, the product area A2 is associated with a class name "item" indicating a product area, and the vertex area A3 is associated with a class name indicating a vertex. The name "edge" is associated. Based on such class names, each area A1, A2, A3 can be distinguished internally (that is, in the internal processing of the detection device 10). Note that each area A1, A2, and A3 may be associated with an evaluation value indicating the certainty of the detection result. The recognition unit 12 may acquire only the areas having an evaluation value equal to or greater than a predetermined threshold value from among the detected areas as the final detection results (areas A1, A2, A3).

The recognition unit 12 may execute the object detection process described above by using a trained model that has been trained in advance. The trained model is, for example, a model that has been trained to read the tray image IM1 and detect each area A1, A2, and A3 included in the tray image IM1. Such a trained model may be trained by, for example, deep learning. In this case, the trained model may be configured by a multilayer neural network. However, the machine learning method applied to the trained model is not limited to the above example, and the internal structure of the trained model is determined according to the applied machine learning method.

As the training data for the above learning, for example, the tray image IM1 and information indicating the positions (coordinates) of the ID area, product area, and vertex area in the tray image IM1 (i.e., correct label) are associated with each other. data can be used. Further, learning using the above-mentioned teacher data is performed, for example, as follows. When the tray 22 is empty (first normal example), when one product G is placed on the tray 22 (second normal example), and when two products G are placed on the tray 22. A plurality of training data are prepared for each pattern, such as when the data is loaded (example of double loading (abnormality)). Note that in order to generate a trained model that can handle various variations, for the first normal example, training data is prepared for each of a plurality of different posture patterns of the product G placed in the tray 22. It is preferable that Further, regarding the example of double loading, it is preferable that training data be prepared for various patterns as illustrated in FIGS. 4 to 7, which will be described later. The above-mentioned trained model is obtained by learning using a plurality of pieces of training data prepared in this way.

As shown in the left part of FIG. 3, when the product G has a rectangular shape in plan view, if the image recognition by the recognition unit 12 is properly performed, one ID area A1 and one product Area A2 and four vertex areas A3 will be detected. On the other hand, when the tray 22 is empty and the image recognition by the recognition unit 12 is appropriately executed, only one ID area A1 will be detected. Some examples of detection results by the recognition unit 12 will be described below. In addition, below, attention will be paid only to the area in the tray 22 where the product G is placed, and description of the ID area A1 will be omitted.

(First example of double loading state)
FIG. 4 is a diagram showing a first example of a double loading state. In this example, two products G1 and G2 are placed on the tray 22 while being spaced apart from each other. In this case, as shown in FIG. and four vertex areas A3) of product G2 are detected.

(Second example of double loading)
FIG. 5 is a diagram showing a second example of a double loading state. In this example, two products G1 and G2 are placed on the tray 22, with one product G1 partially overlapping the other product G2. Furthermore, one vertex of the product G1 is covered up by the product G2. In this case, as shown in FIG. and four vertex areas A3) of product G2 are detected. That is, the vertex of the product G1 hidden by the product G2 is not detected, but the vertex area A3 corresponding to the other seven vertices is detected.

Here, the recognition unit 12 may estimate a product occupied area A4 (article occupied area) occupied by the product G within the product area A2. The product occupied area A4 is an area actually occupied by the product G in the product area A2. For example, the recognition unit 12 determines the area occupied by the tray 22 (that is, the area where the product G is not placed) and the area occupied by the product G in the tray image IM1 (see FIG. 3) based on pixel values such as brightness. It may also be divided by binarization processing. Through such binarization processing, a silhouette image in which only the region occupied by the product G is cut out (for example, an image in which the region occupied by the product G is represented in black and the other regions are represented in white) is obtained. The recognition unit 12 may set the product occupation area A4 based on the silhouette image. However, the method for estimating the product occupied area A4 is not limited to the method based on the silhouette image described above. For example, the recognition unit 12 extracts a difference (background difference) by comparing the image of the empty tray 22 photographed in advance and the tray image IM1, and sets the area corresponding to the difference as the product occupied area A4. It's okay.

Alternatively, the recognition unit 12 may extract (estimate) the product occupied area A4 in the tray image IM1 by using, for example, known segmentation processing (eg, semantic segmentation). For example, the recognition unit 12 can detect each of the areas A1 to A4 as shown in FIG. 5 by performing both the object detection process and the segmentation process described above as image recognition for the tray image IM1. However, depending on the degree of light reflection, blur, etc. in the tray image IM1, it may not be possible to accurately estimate the product occupied area A4. In that case, the recognition unit 12 may omit the process of estimating the product occupied area A4.

When the product occupancy area A4 has been estimated, it becomes possible to distinguish and grasp each vertex as an included vertex located inside the product occupancy area A4 (i.e., a vertex in the tray image IM1 that does not come into contact with the area of the tray 22 but comes into contact with the area of another product G) and a circumscribed vertex that comes into contact with the area of the tray 22. The vertex area A31 in FIG. 5 is a vertex area corresponding to an included vertex. In this example, the included vertex is a vertex of product G2 that comes into contact with the area of product G1 (in other words, a vertex that overlaps with product G1). The other vertex area A3 is a vertex area corresponding to a circumscribed vertex.

For example, the recognition unit 12 identifies, as connotative vertices, vertices that are inside the product occupancy area A4 and are separated by a predetermined threshold distance or more from the boundary line that defines the product occupancy area A4 (dotted chain line in FIG. 5). It may be detected as In this case, since vertices that are sufficiently inside the product occupation area A4 can be detected as connotative vertices, erroneous detection of connotative vertices (that is, detecting vertices that do not actually correspond to connotative vertices as connotative vertices) can be avoided. The occurrence can be suppressed.

(Third example of double loading)
FIG. 6 is a diagram showing a third example of a double loading state. The third example differs from the second example (FIG. 5) in that two vertices of the product G1 are covered up by the product G2. In this case, as shown in FIG. and four vertex areas A3) of product G2 are detected. Here, when the above-described product occupation area A4 is estimated by the recognition unit 12, a vertex area A31 corresponding to one connotative vertex (that is, a vertex that contacts the area of product G1 among the vertices of product G2), Vertex areas A3 corresponding to the other five circumscribed vertices are detected separately.

(Fourth example of double loading)
FIG. 7 is a diagram showing a fourth example of a double loading state. In this example, two products G1 and G2 are placed on the tray 22, with product G2, which is slightly smaller than product G1, completely overlapping one product G1. In plan view, the product G2 is arranged inside the product G1 so as not to protrude outside the product G1. In this case, as shown in FIG. The four vertex areas A3 of the product G2 and the four vertex areas A3 of the product G2 are detected. Here, when the above-described product occupation area A4 is estimated by the recognition unit 12, the vertex area A31 corresponding to the four connotative vertices (i.e., each vertex of the product G2) and the four circumscribed vertices (i.e., the product A vertex area A3 corresponding to each vertex of G1) is detected separately.

The determination unit 13 determines whether or not a plurality of products G are placed on the tray 22 (that is, whether or not they are double loaded) based on the apex detection result by the recognition unit 12.

For example, the determination unit 13 may determine whether or not there is a double stacking state based on the number of vertices detected by the recognition unit 12 (that is, the number of vertex areas A3). As an example, when the product G has an N-gon shape, the determination unit 13 determines that the number of vertex areas A3 detected by the recognition unit 12 is greater than or equal to the sum of N and a predetermined threshold value d of 1 or more (i.e., N+d). In some cases, it may be determined that there is double loading. Here, N is an integer of 3 or more, and in this embodiment, N is 4.

As shown in the right part of FIG. 3, when only one product G is placed on the tray 22, the number of vertex areas A3 detected by the recognition unit 12 matches N. Furthermore, when the tray 22 is empty, the number of vertex areas A3 detected by the recognition unit 12 is zero. On the other hand, as in the examples shown in FIGS. 4 to 7, in the double-mounted state, the number of vertex areas A3 detected by the recognition unit 12 is a number larger than N (that is, "N+1" or more). Become. Specifically, in the example of FIG. 4 (first example), the number of detected vertex areas A3 is 8 (=N+4), and in the example of FIG. The number of regions A3 is 7 (=N+3), and in the example of FIG. 6 (third example), the number of detected vertex regions A3 is 6 (=N+2), and in the example of FIG. In example), the number of detected vertex areas A3 is 8 (=N+4). Therefore, according to the above configuration, by comparing the detected number of vertices with "N+d", it is possible to appropriately determine whether or not there is a double loading state.

Here, the threshold value d may be 2. In image recognition by the recognition unit 12, there is a possibility that the vertex area A3 will be erroneously detected. Specifically, an area that does not actually correspond to the apex of the product G may be mistakenly detected as the apex area A3. By setting the threshold value d to 2, it is possible to suppress the erroneous determination of double overlapping that occurs due to such erroneous detection of the vertex area A3. For example, consider a case where, although only one product G is placed on the tray 22, the recognition unit 12 erroneously detects one vertex area A3 corresponding to a vertex that does not actually exist. In this case, the number of vertex areas A3 detected by the recognition unit 12 is five. When the threshold value d is set to 2, the determination unit 13 determines that double loading occurs when the number of vertex areas A3 detected by the recognition unit 12 is six or more. As a result, it is possible to prevent the determining unit 13 from erroneously determining that there is double loading in the above case.

Note that in the double-mounted state, the number of vertex areas A3 detected by the recognition unit 12 may be "N+1". For example, as in the example in FIG. 7, one product G1 is larger than the other product G2, and product G1 covers "N-1" (here, three) vertices of product G2. When the vertex area A3 overlaps the product G2, the number of vertex areas A3 detected by the recognition unit 12 is "N+1" (here, five). However, the possibility of such a situation occurring is considered to be relatively low. Therefore, although setting the threshold value d to 2 may result in failure to determine double loading (not determining that double loading is actually the case), this disadvantage is outweighed by the , it is thought that the advantage of being able to suppress the above-mentioned erroneous determination of double loading (determining that there is double loading when it is not actually double loading) is greater.

Note that if you place more emphasis on recall than accuracy (precision rate) in determining double loading (in other words, if you want to allow a certain amount of false positives to occur and prevent false positives as much as possible), set the threshold value. d may be set to 1. On the other hand, if you place more emphasis on the precision rate than the recall rate for determining double loading (in other words, if you want to allow a certain degree of omission of judgments and prevent erroneous judgments as much as possible), set the threshold value d to a large value. do it. In this way, the threshold value d may be determined depending on the balance between the precision rate and the recall rate of the double loading determination that is desired to be achieved. Moreover, the threshold value d may be determined according to the accuracy of image recognition by the recognition unit 12. For example, if the probability that the vertex area A3 is erroneously detected by the recognition unit 12 is relatively low, the threshold d is set small; if the probability that the recognition unit 12 is erroneously detected by the vertex area A3 is relatively high, then The threshold value d may be increased.

Further, the determination unit 13 excludes the vertex area A32 (see FIG. 8) detected outside the product area A2 from the vertex area A3 detected by the recognition unit 12, and determines whether or not it is a double-loaded state. It may be determined whether For example, as shown in FIG. 8, the area outside the product area A2 (the area corresponding to the tray 22 ), the vertex area A3 may be erroneously detected. As described above, by setting the threshold value d to 2 or more, it is possible to suppress the occurrence of a false determination of double loading when such a vertex area A3 is detected. However, as described above, the determination unit 13 By excluding (invalidating) the vertex area A3 detected outside the product area A2, it is also possible to suppress the occurrence of erroneous determination of double loading. In addition, since erroneous detection of the vertex area A3 may occur inside the product area A2, a process of setting the threshold value d to 2 or more and a process of excluding the vertex area A32 detected outside the product area A2 are performed. and may be used together.

Further, when the recognition unit 12 detects the apex area A3 (that is, the apex area A31 corresponding to the connotative apex) inside the product occupied area A4 (see FIG. 5), the determination unit 13 determines that there is double loading. It may be determined that The vertex area A31 detected inside the product occupied area A4 is a vertex area corresponding to the vertex of another product G that overlaps one product G, if the vertex region A31 is not due to erroneous detection. It turns out. Therefore, according to the above configuration, it is possible to appropriately detect the state of double loading based on the detection of the connotative vertex.

The determination unit 13 may output the determination result described above to an output device such as a display included in the detection device 10, for example. For example, when a double loading state is determined, the determining unit 13 determines that the double loading state is detected in the tray 22 indicated by the tray ID along with the tray ID of the tray 22 for which the double loading state has been determined. Information indicating that this has occurred may be output to a display or the like. Alternatively, the determination unit 13 detects an abnormal state (double loading state) by outputting an alert (warning) from an output device such as a speaker included in the detection device 10 or by lighting up a police lamp installed at the scene. It may also be possible to notify on-site workers, etc. that the detection has been detected.

Next, an example of the operation of the detection device 10 will be described with reference to the flowchart in FIG.

In step S1, the acquisition unit 11 acquires a tray image IM1 (see FIG. 3). The tray image IM1 is an image obtained by the camera 30 photographing the tray 22 passing through the photographing area SA (see FIG. 2).

In step S2, the recognition unit 12 detects the ID area A1, product area A2, and vertex area A3 by performing image recognition on the tray image IM1. Note that when the product G is not placed on the tray 22, only the ID area A1 is detected. Further, the recognition unit 12 may further detect (estimate) the product occupied area A4 (see FIG. 5).

In step S3, the determination unit 13 determines whether or not a plurality of products G are placed on the tray 22 (that is, in a double-loaded state) based on the detection result of the apex (vertex area A3) in step S2. whether or not).

An example of the procedure of the determination process (step S3 in FIG. 9) by the determination unit 13 will be described with reference to the flowchart in FIG. 10.

In step S31, the determination unit 13 determines whether the recognition unit 12 has estimated the product occupation area A4. If the product occupied area A4 is estimated (step S31: YES), the determination unit 13 proceeds to the process of step S32. On the other hand, if the product occupied area A4 is not estimated (step S31: NO), the determination unit 13 proceeds to the process of step S34.

In step S32, the determination unit 13 determines whether the recognition unit 12 detects a vertex inside the product occupied area A4. That is, the determining unit 13 determines whether the vertex area A31 corresponding to the connotative vertex has been detected. If the apex region A31 is detected (step S32: YES), the determination unit 13 determines that the state is double stacked (step S33). On the other hand, if the vertex area A31 is not detected (step S32: NO), the determination unit 13 proceeds to the process of step S34.

In step S34, the determination unit 13 determines whether the recognition unit 12 has detected a vertex (that is, the vertex area A32 (see FIG. 8)) outside the product area A2. If the vertex region A32 is detected (step S34: YES), the determination unit 13 excludes the vertex region A32 (step S35), and proceeds to the process of step S36. On the other hand, if the vertex area A32 is not detected (step S34: NO), the determination unit 13 skips the process of step S35 and proceeds to the process of step S36.

In step S36, the determination unit 13 determines whether the number of vertices (the number of vertex areas A3) detected by the recognition unit 12 is equal to or greater than N+d (step S36). If the number of detected vertices is equal to or greater than N+d (step S36: YES), the determination unit 13 determines that the state is double stacked (step S33). On the other hand, if the number of detected vertices is less than N+d (step S36: NO), the determination unit 13 determines that the state is normal (step S37).

Note that if it is known in advance that the process of estimating the product occupied area A4 (see FIG. 5) by the recognition unit 12 will be omitted (or cannot be performed), steps S31 and S32 in FIG. 10 may be omitted. Good too. In this case, the determination process will start from step S34. Further, the process of excluding the vertex area A32 detected outside the product area A2 is not essential. That is, steps S34 and S35 in FIG. 10 may be omitted.

Further, the determining unit 13 may determine whether or not there is double loading based only on whether or not the vertex area A32 corresponding to the connotative vertex is detected. That is, the determination unit 13 may execute only steps S31 to S33 in FIG. 10 and omit steps S34 to S37 in FIG. In this case, although the determination unit 13 cannot detect double loading in the manner shown in FIG. 4 (the manner in which the two products G1 and G2 do not overlap), Double loading can be detected. Therefore, for example, if the product G is sufficiently large relative to the size of the tray 22 and two products G are placed on the tray 22, the two products G will inevitably overlap each other (i.e., 4), the determination unit 13 can appropriately detect double loading by executing only steps S31 to S33 in FIG. 10.

In the detection device 10 described above, based on the image recognition result for the image of the tray 22 conveyed with the product G placed thereon (tray image IM1), the apex (corner) of the product G included in the tray image IM1 is detected. A vertex area A3 corresponding to is detected. Then, based on the detection result (for example, the number of vertices) of the vertex area A3, it is determined whether or not a plurality of products G are placed on the tray 22 (so-called double-loaded state). According to the detection device 10, an abnormal state (the above-mentioned double loading state) can be detected by a relatively simple analysis process on the tray image IM1 taken from above the tray 22. That is, an abnormal state (double loading state) regarding article transportation (transportation of product G) can be detected with simple equipment.

When it is necessary to sort the products G at high speed, such as in a sorter system 20 in a distribution center, etc., the transport speed of the trays 22 is relatively high. Therefore, it is difficult to capture the three-dimensional shape of the product G in the tray 22 using, for example, an RGB-D sensor. On the other hand, in the detection device 10, the abnormal state (double loading) is detected using the two-dimensional tray image IM1 acquired by the camera 30, without using a special sensor etc. for recognizing the three-dimensional shape as described above. can be detected.

Further, the tray 22 is provided with an ID area in which a tray ID for identifying the tray 22 is written (see FIG. 3). Then, the recognition unit 12 detects not only the vertex area A3 but also the ID area A1. In this case, it becomes possible to easily manage the determination results for each tray 22. For example, the tray 22 on which double loading was detected can be identified by the tray ID.

Note that the object for which double loading is determined by the detection device 10 does not necessarily have to be the product G intended to be sold to consumers, and may be an object other than the product intended for use as a product. Furthermore, although the above embodiment describes the case where "N=4" (that is, the case where the product G has a quadrangular shape in plan view), the product G may have a shape other than a quadrilateral, such as a triangular shape or a pentagonal shape. There may be. Further, the type of tray image IM1 is not particularly limited. The tray image IM1 may be any image as long as it can be subjected to the image analysis described above, and may be, for example, an X-ray image acquired using X-rays.

It should be noted that the block diagram used to explain the above embodiment shows blocks in functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using one physically or logically coupled device, or may be realized using two or more physically or logically separated devices directly or indirectly (e.g. , wired, wireless, etc.) and may be realized using a plurality of these devices. The functional block may be realized by combining software with the one device or the plurality of devices.

Functions include judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, exploration, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, These include, but are not limited to, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assigning. I can't.

For example, the detection device 10 in an embodiment of the present disclosure may function as a computer that performs the communication control method of the present disclosure. FIG. 11 is a diagram illustrating an example of the hardware configuration of the detection device 10 according to an embodiment of the present disclosure. The above-mentioned detection device 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

In addition, in the following description, the word "apparatus" can be read as a circuit, a device, a unit, etc. The hardware configuration of the detection device 10 may be configured to include one or more of each device shown in FIG. 1, or may be configured not to include some of the devices.

Each function in the detection device 10 is implemented by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, so that the processor 1001 performs calculations, controls communication by the communication device 1004, and controls communication by the communication device 1004. This is realized by controlling at least one of data reading and writing in the storage 1003.

The processor 1001, for example, operates an operating system to control the entire computer. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic device, registers, and the like.

Furthermore, the processor 1001 reads programs (program codes), software modules, data, and the like from at least one of the storage 1003 and the communication device 1004 to the memory 1002, and executes various processes in accordance with these. As the program, a program that causes a computer to execute at least part of the operations described in the above embodiments is used. For example, the recognition unit 12 may be realized by a control program stored in the memory 1002 and operated in the processor 1001, and other functional blocks may be similarly realized. Although the various processes described above have been described as being executed by one processor 1001, they may be executed by two or more processors 1001 simultaneously or sequentially. Processor 1001 may be implemented by one or more chips. Note that the program may be transmitted from a network via a telecommunications line.

The memory 1002 is a computer-readable recording medium, and includes at least one of ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), and the like. may be done. Memory 1002 may be called a register, cache, main memory, or the like. The memory 1002 can store executable programs (program codes), software modules, and the like to implement a communication control method according to an embodiment of the present disclosure.

The storage 1003 is a computer-readable recording medium, such as an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (such as a compact disk, a digital versatile disk, or a Blu-ray disk). (registered trademark disk), smart card, flash memory (eg, card, stick, key drive), floppy disk, magnetic strip, etc. Storage 1003 may also be called an auxiliary storage device. The storage medium mentioned above may be, for example, a database including at least one of memory 1002 and storage 1003, a server, or other suitable medium.

The communication device 1004 is hardware (transmission/reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, or the like.

The input device 1005 is an input device (eg, keyboard, mouse, microphone, switch, button, sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that performs output to the outside. Note that the input device 1005 and the output device 1006 may have an integrated configuration (for example, a touch panel).

Further, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses for each device.

The detection device 10 also includes hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). A part or all of each functional block may be realized by the hardware. For example, processor 1001 may be implemented using at least one of these hardwares.

Next, a detection program for causing a computer to function as the detection device 10 of this embodiment will be described. FIG. 12 is a diagram showing the configuration of the detection program P.

The detection program P includes a main module m10 that comprehensively controls the above processing in the detection device 10, an acquisition module m11, a recognition module m12, and a determination module m13. Each of the modules m11 to m13 implements the acquisition unit 11, recognition unit 12, and determination unit 13 in the detection device 10. Note that the detection program P may be transmitted via a transmission medium such as a communication line, or may be stored in a recording medium M as shown in FIG.

Although this embodiment has been described in detail above, it is clear for those skilled in the art that this embodiment is not limited to the embodiment described in this specification. This embodiment can be implemented as modifications and changes without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is for the purpose of illustrative explanation and does not have any restrictive meaning with respect to this embodiment.

The processing procedures, sequences, flowcharts, etc. of each aspect/embodiment described in this disclosure may be rearranged in order as long as there is no contradiction. For example, the methods described in this disclosure use an example order to present elements of the various steps and are not limited to the particular order presented.

The input/output information may be stored in a specific location (eg, memory) or may be managed using a management table. Information etc. to be input/output may be overwritten, updated, or additionally written. The output information etc. may be deleted. The input information etc. may be transmitted to other devices.

Judgment may be made using a value expressed by 1 bit (0 or 1), a truth value (Boolean: true or false), or a comparison of numerical values (for example, a predetermined value). (comparison with a value).

Each aspect/embodiment described in this disclosure may be used alone, may be used in combination, or may be switched and used in accordance with execution. In addition, notification of prescribed information (for example, notification of "X") is not limited to being done explicitly, but may also be done implicitly (for example, not notifying the prescribed information). Good too.

Software includes instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, whether referred to as software, firmware, middleware, microcode, hardware description language, or by any other name. , should be broadly construed to mean an application, software application, software package, routine, subroutine, object, executable, thread of execution, procedure, function, etc.

Additionally, software, instructions, information, etc. may be sent and received via a transmission medium. For example, if the software uses wired technology (coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), etc.) and/or wireless technology (infrared, microwave, etc.) to When transmitted from a server or other remote source, these wired and/or wireless technologies are included within the definition of transmission medium.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc., which may be referred to throughout the above description, may refer to voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may also be represented by a combination of

In addition, the information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a predetermined value, or using other corresponding information. may be expressed.

The names used for the parameters described above are not restrictive in any respect. Furthermore, the mathematical formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure. The various names assigned to these various information elements are not restrictive in any respect, as the various information elements may be identified by any suitable name.

As used in this disclosure, the phrase "based on" does not mean "based solely on" unless explicitly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

As used in this disclosure, any reference to elements using the designations "first," "second," etc. does not generally limit the amount or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Thus, reference to a first and second element does not imply that only two elements may be employed or that the first element must precede the second element in any way.

Where "include", "including" and variations thereof are used in this disclosure, these terms, like the term "comprising," are inclusive. It is intended that Furthermore, the term "or" as used in this disclosure is not intended to be exclusive or.

In this disclosure, when articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are plural.

In this disclosure, the term "A and B are different" may mean "A and B are different from each other." Note that the term may also mean that "A and B are each different from C". Terms such as "separate" and "coupled" may also be interpreted similarly to "different."

DESCRIPTION OF SYMBOLS 10... Detection device, 11... Acquisition unit, 12... Recognition unit, 13... Judgment unit, 22... Tray, A1... ID area, A2... Product area (article area), A3, A31, A32... Vertex area, A4... Product Occupied area (article occupied area), G...product (article), IM1...tray image.

Claims (2)

an acquisition unit that acquires a tray image obtained by photographing a tray conveyed with a polygonal article placed thereon from above when viewed from above;
a recognition unit that detects a vertex of the article included in the tray image by performing image recognition on the tray image;
a determination unit that determines whether a plurality of the articles are placed on the tray based on a detection result of the apex by the recognition unit;
Equipped with
The recognition unit estimates an article occupied area occupied by the article within an article area including the entire article in the tray image,
The determination unit is a detection device that determines that a plurality of the articles are placed on the tray when the recognition unit detects a connotative vertex that is the vertex located inside the article occupation area. . 2. The recognition unit detects, as the connotative vertex, the vertex that is inside the article occupation area and is away from a boundary line defining the article occupation area by a predetermined threshold distance or more. The detection device described.

Images