JP7304070B2 - Information processing method, apparatus, and program for recognizing the arrangement state of a group of articles - Google Patents

Description

The present invention relates to a technique for recognizing the arrangement state of a group of articles such as merchandise.

YOLO (You Only Look Once) (Non-Patent Document 1), SSD (Single Shot multibox Detector) (Non-Patent Document 2), and the like are known as techniques for detecting an individual article in an image. When these techniques are used to detect individual articles, images of individual articles of the type to be detected are learned to detect the type and position of the article in the image to be processed. These techniques basically use a CNN (Convolutional Neural Network).

If YOLO or SSD is used to learn images of individual articles of the type to be detected as usual, it is conceivable that images of groups of articles arranged on shelves in a store, for example, can be similarly processed. It has also been reported that it is not good at detecting individual items contained in a group of closely arranged items (Non-Patent Document 3).

For example, in a state in which a plurality of articles of the same type are densely arranged on a store shelf for each type of article (for example, for each type of content, size, or shape), a robot or the like is used to replenish articles or perform other processes. Assuming that such a thing is performed, automatic recognition of the arrangement state of the article group is required. A store is an example, and the same applies to a warehouse or the like.

J Redmon, et al.”You Only Look Once: Unified, Real-Time Object Detection”arXiv:1506.02640v5 9 May 2016 Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, "SSD: Single shot multibox detector. " in ECCV2016 Goldman, et.al., “Precise Detection in Densely Packed Scenes,” Proc. of CVPR 2019

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention, as one aspect, to provide a novel technique that can be used to recognize the arrangement of a group of aligned articles.

The information processing method according to the present invention includes, in an image of a group of articles arranged in line, an area of the group of articles continuing in the vertical direction parallel to the direction of gravity, an area of the group of articles continuing in the horizontal direction, and an area of the group of articles continuing in the horizontal direction. From the input image, a first region of the vertically continuous article group and a horizontally continuous and a process of detecting a second area of the group of articles that is continuous in the direction of depth and a third area of the group of articles that is continuous in the depth direction.

In the above information processing method, based on the state of overlap of at least two of the first area, the second area, and the third area, in the input image, for each article included in the article group, may further include a process of specifying a fourth region of .

According to one aspect, a new element technology is provided for recognizing the arrangement state of a group of aligned articles.

FIG. 1 is a diagram showing a system outline in an embodiment. FIG. 2 is a diagram showing the definition of directions. FIG. 3 is a diagram for illustrating imaging directions. FIG. 4A is a diagram for explaining machine learning of an arrangement direction detection unit; FIG. 4B is a diagram for explaining the machine learning of the arrangement direction detection unit; FIG. 4C is a diagram for explaining the machine learning of the arrangement direction detection unit; FIG. 5A is a diagram for explaining machine learning of an arrangement direction detection unit; FIG. 5B is a diagram for explaining the machine learning of the arrangement direction detection unit; FIG. 5C is a diagram for explaining the machine learning of the arrangement direction detection unit; FIG. 5D is a diagram for explaining machine learning of the arrangement direction detection unit; FIG. 5E is a diagram for explaining the machine learning of the arrangement direction detection unit; FIG. 6A is a diagram for explaining machine learning of an arrangement direction detection unit; FIG. 6B is a diagram for explaining the machine learning of the arrangement direction detection unit; FIG. 6C is a diagram for explaining machine learning of the arrangement direction detection unit; FIG. 7 is a diagram illustrating a main processing flow according to the embodiment; FIG. 8 is a diagram showing a processing flow of individual article BB detection processing. FIG. 9A is a diagram for explaining the overlay processing in the arrangement direction BB. FIG. 9B is a diagram for explaining the overlay processing in the arrangement direction BB. FIG. 9C is a diagram for explaining the overlay processing in the arrangement direction BB. FIG. 10A is a diagram for explaining the overlay processing in the arrangement direction BB. FIG. 10B is a diagram for explaining the overlay processing in the arrangement direction BB. FIG. 10C is a diagram for explaining the overlay processing in the arrangement direction BB. FIG. 10D is a diagram for explaining the overlay processing in the arrangement direction BB. FIG. 11A is a diagram for explaining processing in the case of partial overlap. FIG. 11B is a diagram for explaining processing in the case of partial overlap. FIG. 11C is a diagram for explaining processing in the case of partial overlap. FIG. 11D is a diagram for explaining processing in the case of partial overlap. FIG. 12A is a diagram showing an example of the detection result of the individual article BB. FIG. 12B is a diagram showing an example of the detection result of the individual article BB. FIG. 13 is a diagram illustrating a processing flow of 3D arrangement pattern generation processing. FIG. 14A is a diagram illustrating an example of a virtual network; FIG. 14B is a diagram illustrating an example of a virtual network; FIG. 14C is a diagram illustrating an example of a virtual network; FIG. 15 is a diagram showing an example of a 3D arrangement pattern. FIG. 16 is a diagram showing an example of a basic 3D arrangement pattern. FIG. 17 is a diagram showing a display example of a missing item detection result. FIG. 18 is a diagram illustrating a processing flow of foreign matter detection processing. FIG. 19 is a diagram for explaining an example of foreign matter detection processing. FIG. 20 is a diagram for explaining an example of foreign matter detection processing. FIG. 21 is a block configuration diagram of a computer device.

FIG. 1 shows an outline of a system according to an embodiment of the invention. The system according to the present embodiment includes a robot 200 having a camera 210 that captures images (including moving images), and an information processing device 100 that can communicate with the robot 200 wirelessly. The robot 200 moves, for example, through the aisles of the store, takes images of shelves in the store, and transmits the data of the taken images to the information processing apparatus 100 .

As shown in FIG. 2, for example, it is assumed that individual articles (in the example of FIG. 2, they are columnar articles, but the shape does not matter) are arranged in a dense array on a store shelf. Since the shelves are photographed from the aisle of the store, the vertical direction is the direction parallel to the direction of gravity, the horizontal direction is the lateral direction of the shelf, and the depth direction is the direction perpendicular to the vertical direction and the horizontal direction of the shelf. Define.

Further, as schematically shown in FIG. 3, the camera 210 moves from a position (1) where the side surface of the group of articles is viewed from the front to a position (2) where the side surface and the upper surface of the group of articles are viewed obliquely from above. It is assumed that an image is taken up to the position (3) where the upper surface of the group of articles is viewed from above. A moving image may be captured from positions (1) to (3), or a still image may be captured at each position.

The information processing apparatus 100 includes an image acquisition unit 101, an image data storage unit 102, an arrangement direction detection unit 103, a first data storage unit 104, an individual article detection unit 105, a second data storage unit 106, and a 3D It has an arrangement pattern generation unit 107 , a third data storage unit 108 , a missing item detection unit 109 , and a foreign object detection unit 110 .

The image acquisition unit 101 receives image data captured by the camera 210 from the robot 200 and stores the data in the image data storage unit 102 . The arrangement direction detection unit 103 detects, for each image frame stored in the image data storage unit 102, an area including a group of articles continuing in the vertical direction, an area including a group of articles continuing in the horizontal direction, and an area including a group of articles continuing in the depth direction. It has a function of detecting a region containing a group of articles that is continuous with the

The arrangement direction detection unit 103 detects, for example, an area that includes a group of articles that are continuous in the vertical direction, an area that includes a group of articles that are continuous in the horizontal direction, and an area that includes a group of items that are continuous in the depth direction. It is a trained model that has been machine-learned as follows.

In object detection using YOLO, SSD, etc., learning is performed to detect individual objects from an input image, but in this embodiment, the mechanism of YOLO, SSD, etc. remains the same, but the learning content is different. Specifically, it learns the arrangement state of the articles. More specifically, instead of recognizing each item individually, in each image frame, an area containing a group of items that are vertically continuous (hereinafter referred to as a vertical BB (bounding box)) and a horizontally continuous A region containing a group of articles (hereinafter referred to as a horizontal BB) and a region containing a group of articles continuing in the depth direction (hereinafter referred to as a depth BB) are learned.

For example, as schematically shown in FIG. 4A, when an image is obtained in which three articles are arranged in the vertical direction and four articles are arranged in the horizontal direction, as schematically shown in FIG. Designate and study 3 horizontal BBs containing 4 items. Similarly, four vertical BBs each containing three items are specified and learned, as shown schematically in FIG. 4C. As a result, when an image such as that shown in FIG. 4A is input, the horizontal BB shown in FIG. 4B and the vertical BB shown in FIG. 4C can be detected.

Also, as schematically shown in FIG. 5A, when an image is obtained in which three articles are aligned in the depth direction, although the horizontal and vertical directions are the same as in FIG. 4A, the image shown in FIG. Three horizontal BBs, each containing four items, are specified and learned, as schematically shown in . Similarly, four vertical BBs, each containing three articles, are specified and learned, as shown schematically in FIG. 5C. Furthermore, as schematically shown in FIG. 5D, four depths BB each containing three articles are designated and learned. In the case of FIG. 5A, three horizontal BBs each containing four articles are further specified and learned, as schematically shown in FIG. 5E. Accordingly, if an image as shown in FIG. 5A is input, a horizontal BB as shown in FIGS. 5B and 5E, a vertical BB as shown in FIG. 5C, and a depth BB as shown in FIG. 5D are generated. be detectable.

Although not shown, if an image is obtained in which only the top surface of the article is included, as shown in FIGS. 4A and 5A, then four depths BB, each containing three articles, are obtained, as shown in FIG. 5D. Specify and learn. Similarly, three horizontal BBs containing four items each are specified and learned, as shown in FIG. 5E. Accordingly, if an image including only the top surface of the article is input, four depth BBs and three horizontal BBs can be detected.

As schematically shown in FIG. 6A, when one item is missing from the state shown in FIG. 5A, a BB different from the BB shown in FIGS. 5B to 5E is specified and learned. . As shown schematically in FIG. 6B, the top horizontal BB (h1) is designated and learned as the horizontal BB containing only two items. Also, the vertical BB (v1) in the second column from the right is also designated as a vertical BB containing only two items and learned. What is learned is a bounding box (BB) containing multiple items.

Further, as schematically shown in FIG. 6C, the frontmost horizontal BB (h2) of the horizontal BBs on the upper surface of the article is designated as the horizontal BB containing only two articles and learned. Similarly, the depth BB (d1) in the second column from the right is also specified and learned as the depth BB containing only two articles. This is for learning a boundary box BB containing multiple items.

If an image as shown in FIG. 6A is input, horizontal BB, vertical BB and depth BB as shown in FIGS. 6B and 6C are detected.

A number of such images are trained for the specified horizontal BB, vertical BB and depth BB as described above. In addition, similar learning shall be performed for various types of goods (including differences in packaging).

Returning to the description of FIG. 1, the arrangement direction detection unit 103 stores data (position, size, etc.) on the horizontal BB, vertical BB, and depth BB, which are detection results, in the first data storage 104 . Note that the data stored in the first data storage unit 104 may be output to an output device such as a display device. The same applies to data stored in the second data storage unit 105 and the third data storage unit 108 .

The individual article detection unit 105 uses the data stored in the first data storage unit 104 and the image data storage unit 102 to detect the area of the individual article (hereinafter referred to as individual article BB), and the detection result is Data (position, size, etc.) about the individual article BB are stored in the second data storage unit 106 . The individual article detection unit 105 basically detects the individual article BB based on the overlapping state of the horizontal BB, vertical BB, and depth BB. (Learning models that have been machine-learned to detect individual items in

Also, the 3D layout pattern generation unit 107 uses the data stored in the first data storage unit 104, the second data storage unit 105, and the image data storage unit 102 to generate a 3D layout pattern of the item group from a plurality of image frames. is specified, and data of the specified 3D arrangement pattern is stored in the third data storage unit 108 .

The missing item detection unit 109 uses the data stored in the third data storage unit 108 to execute processing for detecting missing items, and outputs the processing result. Data stored in the image data storage unit 102, the first data storage unit 104, and the second data storage unit 106 may be used.

Further, the foreign object detection unit 110 uses the data stored in the image data storage unit 102, the first data storage unit 104, and the second data storage unit 106 to detect foreign objects (including different postures) in the object group. Execute the detection process and output the process result.

Next, details of processing in the information processing apparatus 100 will be described with reference to FIGS. 7 to 20. FIG.

The arrangement direction detection unit 103 and the individual article detection unit 105 execute individual article BB detection processing for detecting the individual article BB for each image frame acquired by the image acquisition unit 101 and stored in the image data storage unit 102, and detect the individual article BB. The resulting data on the individual article BB is stored in the second data storage unit 106 (step S1). The individual article BB detection process will be described in detail with reference to FIGS. 8 to 12. FIG.

Next, the 3D layout pattern generation unit 107 uses the data stored in the image data storage unit 102, the first data storage unit 104, and the second data storage unit 106 to generate specific articles included in the plurality of image frames. A 3D arrangement pattern generation process for generating 3D arrangement pattern data for the group is executed, and the processing result is stored in the third data storage unit 108 (step S3). The 3D arrangement pattern generation processing will be described in detail with reference to FIGS. 13 to 15. FIG.

Then, the out-of-item detection unit 109 confirms whether it is set to execute the out-of-item detection process or whether it is instructed by the user (step S5). A product detection process is executed, and the detection result is output (step S7). The missing item detection process will be described later with reference to FIGS. 16 and 17. FIG.

Further, the foreign object detection unit 110 checks whether the foreign object detection process is set to be executed or is instructed by the user (step S9). Execute and output the detection result (step S11). The foreign object detection process will be described later with reference to FIGS. 18 to 20. FIG.

Next, individual article BB detection processing will be described with reference to FIGS. 8 to 12. FIG.

The arrangement direction detection unit 103 detects horizontal BB, vertical BB, and depth BB for each image frame stored in the image data storage unit 102, and stores the detection results in the first data storage unit 104 (step S21). As described above, the arrangement direction detection unit 103 is a learned model that has undergone machine learning to detect horizontal BB, vertical BB, and depth BB, so detection is performed using that function. The horizontal BB, vertical BB, and depth BB are collectively referred to as arrangement direction BB.

Next, the individual article detection unit 105 determines whether there is any overlap in the detected array direction BB based on the data stored in the first data storage unit 104 (step S23). For example, if only one alignment direction BB is detected, there is no overlap, and even if multiple alignment directions BB are detected, they may be detected in an isolated state. No overlap exists. Generally, in the case of a group of aligned articles as shown above, the alignment direction BB is detected such that there is an overlap.

For example, the processing of steps S25 to S33 is executed for the overlapping arrangement direction BB (step S23: Yes route), and the processing for the non-overlapping arrangement direction BB is carried out after step S35 (step S23: No route).

If there is an overlap in the arrangement direction BB, the individual article detection unit 105 executes a process of superimposing the arrangement direction BB, identifies a product area that satisfies a predetermined condition as an individual article BB, and obtains data about the individual article BB. is stored in the second data storage unit 106 (step S25).

Superposition processing is performed according to the following four conditions.
Condition 1. No overlap processing is performed in the same arrangement direction BB.
Condition 2. In the overlap of the horizontal BB and the vertical BB, the intersection of the horizontal center line of the horizontal BB and the vertical center line of the vertical BB is included in the cut out BB.
Condition 3. In the overlap of the horizontal BB and the depth BB, the intersection of the horizontal center line of the horizontal BB and the vertical center line of the depth BB is included in the clipped BB.
Condition 4. In the overlap of the vertical BB and the depth BB, both the vertical center line of the vertical BB and the vertical center line of the depth BB are included in the cut BB. However, a lower limit is set for the height of the clipped region.

For example, when two horizontal BBs containing two articles and two vertical BBs containing two articles are detected as shown in FIG. Not performed. Therefore, horizontal BBs, vertical BBs, and depth BBs are not superimposed. Thus, in FIG. 9A, the narrow longitudinal overlap region e2 occurring between the articles and the narrow lateral overlap region e1 occurring between the articles are ignored.

Also, as shown in FIG. 9B, in the presence of a horizontal BB containing two items and two horizontal BBs containing three items, the overlap of the vertical BB containing the two items on the right end is When considering, according to the conditions 2 and 3, the minute overlapping region e3 is ignored. That is, the overlap region that does not include the intersection of the horizontal centerline of the horizontal BB and the vertical centerline of the vertical BB is ignored. The same is true for the combination of horizontal BB and depth BB.

Furthermore, as shown in FIG. 9C, when only two items are stacked on the frontmost side and three items are stacked on the back two lines, the depth BB (d11) and the vertical BB ( v11) are detected, resulting in their overlap region e4, which according to Condition 4 is rejected as being less than the threshold height.

On the other hand, as shown in FIG. 10A, the product area p12 of the arrangement direction BB in different directions such as the vertical BB (v12) and the horizontal BB (h12) satisfies the above conditions 1 and 2 and is identified as the individual product BB. .

Similarly, as shown in FIG. 10B, the product area p13 of the horizontal BB (h13) and the depth BB (d13) also satisfies the above conditions 1 and 3 and is identified as the individual article BB.

Furthermore, as shown in FIG. 10C, the product area p14 of the depth BB (d14) and the vertical BB (v14) also satisfies the above conditions 1 and 4, so it is identified as the individual article BB.

Also, as shown in FIG. 10D, when a horizontal BB (h15), a depth BB (d15), and a vertical BB (v15) are detected, the horizontal BB (h15), the vertical BB (v15), and the horizontal BB (h15) ) and depth BB (d15) and vertical BB (v15) and depth BB (d15) yield product area p15 for the same article, but satisfying conditions 1 to 4 above, and individual article BB identified as However, if a product area that is substantially the same as the first detected product area is detected according to the superimposition priority, the later detected product area may be excluded.

In order to efficiently and accurately detect the individual article BB, the overlaying process is performed in order, for example, according to the following rules.
1. Preferential processing is performed for superposition of the horizontal BB and the BB in the other arrangement direction, and then superimposition processing of the vertical BB and the depth BB is performed.
2. In the superposition processing of the horizontal BB and the BB in the other arrangement direction, the horizontal BB is selected from bottom to top, from left to right, and from front to back in the image. Also, within the horizontal BB, intersections with the vertical BB or the depth BB are checked from left to right, and superposition processing is performed.
3. For the process of superimposing the vertical BB and the depth BB, the vertical BB is selected from left to right in the image and from the front to the back. Alignment processing is performed.

By performing such a process, detection of individual articles BB is generally performed for portions where there is an overlap in the arrangement direction BB. However, if the arrangement direction BB is coarse and there is little overlap, detection failure of the individual articles BB will occur.

Therefore, the individual article detection unit 105 determines whether or not there is partial overlap in the arrangement direction BB (step S27). This assumes the following situation. For example, as schematically shown in FIG. 11A, a product region p16 is detected with a plurality of overlaps in the arrangement direction BB. area) and the original array direction BB (h16) is larger than one detected product area, it is determined that there is a partial overlap. If no partial overlap is detected, processing returns to the calling processing.

On the other hand, if there is a partial overlap, the individual article detection unit 105 estimates the number of articles for the portion where the individual article BB is not specified based on a predetermined rule, and determines the individual article BB is set, and data on the individual article BB is stored in the second data storage unit 105 (step S29).

More specifically, for example, as shown in FIG. 11A, when the horizontal BB (h16) and the vertical BB (v16) partially overlap, the most individual item BB within the horizontal BB (h16) is Ratio (=L /W) is close to 1 or more, it is assumed that the item is placed to the left (or right) of the leftmost (or right) individual item. In this case, the number of articles is estimated to be the nearest natural number to L/W. When the number of articles can be estimated in this way, one detected individual article BB is evenly arranged by the estimated number while allowing overlap.

Also, as shown in FIG. 11B, when two vertical BBs (v17 and v18) partially overlap one horizontal BB (h17), two product regions (p17 and p18) are detected. be. In such a case, the distance LC between the centers of two adjacent individual products BB (p17 and p18) among the individual products BB within the horizontal BB (h17), and the distance LC between the centers of these two individual products BB (p17 and p18) If the ratio (=LC/W) of the average widths W of the is close to 2 or more, it can be estimated that an article is placed between the two individual articles BB. In this case, the number of items is assumed to be the natural number closest to LC/W-1. When the number of articles is estimated in this way, areas having the average size of the detected individual articles BB are evenly arranged by the estimated number while allowing overlap.

Furthermore, as shown in FIG. 11C , when the horizontal BB (h20) and the vertical BB (v20) partially overlap, the bottom (or top) individual item BB within the vertical BB (v20) The ratio (=L/H) of the distance L between the upper (or lower) side of a certain individual product BB (p20) and the upper (or lower) side of the vertical BB (v20) and the height H of the individual product BB is If it is in the vicinity of 1 or more, it can be estimated that the article is arranged above (or below) the lowest (or above) individual article BB (p20). In this case, the number of articles is assumed to be the natural number closest to L/H. When the number of articles can be estimated in this way, one detected individual article BB is evenly arranged by the estimated number while allowing overlap.

Furthermore, as shown in FIG. 11D, when the horizontal BB (h21 and h22) partially overlaps the depth BB (d21), the frontmost individual product BB ( The ratio of the distance L between the lower side (or upper side) of the individual article BB (p21) and the lower side (or upper side) of the depth BB to the height H of the individual article BB (p21) (=L/ If H) is close to 1 or more, it is estimated that the article is arranged on the front side (or the back side) of the individual article BB that is the closest (or the farthest). In this case, the number of articles is estimated to be the natural number closest to L/H. When the number of articles can be estimated in this way, the individual articles BB used are evenly arranged by the estimated number while allowing overlap.

By setting the individual article BB in the difference area according to such rules, the portion that could not be dealt with only by the product area is filled. In this way, the method for estimating the number of articles for the portion where the individual article BB is not specified from the overlap in the arrangement direction BB is to set the side of the article group to the front ((1) in FIG. 3) or the upper surface of the article group. This is particularly effective for an image at a position viewed from above ((3) in FIG. 3).

Then, the individual article detection unit 105 determines whether or not there is a portion that could not be dealt with according to the predetermined rule (step S31). The rule in step S29 can deal only with a part of the area where the number of articles can be estimated, and in particular, the vertical BB and the depth BB may not be dealt with by the above rule. For example, even in the case of FIGS. 11A and 11B, it is the area other than the individual article BB in the vertical BB, the front row area in FIG. 11D, and the like. In these, an estimate based on the size of the discrete item BB cannot address because the item appears to be a different size than the size of the discrete item BB being detected. Therefore, if there is an area of a predetermined size or larger in which the individual articles BB are not arranged in the arrangement direction BB, it is determined that there is a portion that could not be dealt with according to the predetermined rule. If there is no portion that could not be dealt with by the predetermined rule, the process returns to the calling process.

On the other hand, if there is a portion that could not be dealt with by the predetermined rule, the individual article detection unit 105 performs general object detection in the arrangement direction BB to specify the individual article BB, and stores the result as the second data. Stored in unit 106 (step S33). As described above, general object detection is performed by a machine-learned model that has been machine-learned so that an article to be detected can be detected from an image as before. Then, the process returns to the calling process.

Further, for the arrangement direction BB that does not overlap with any other arrangement direction BB, the individual article detection unit 105 performs general object detection in each arrangement direction BB to detect the individual article BB, as in step S33. Then, the result is stored in the second data storage unit 106 (step S35).

Then, the individual article detection unit 105 determines whether or not there is an area in which the individual article BB could not be detected by general object detection in the arrangement direction BB to be processed in step S35 (step S37). This determination is also basically the same as step S27. That is, if the difference between the sum of the detected individual article BB areas and the original arrangement direction BB is larger than the detected individual article BB, there is an area where the individual article BB cannot be detected. Then judge. If no such region exists, processing returns to the calling processing.

On the other hand, when it is determined that there is an area in which the individual article BB could not be detected, the individual article detection section 105 executes the same processing as in step S29 (step S39). In other words, if the number of articles can be estimated based on the size, this is the process of setting individual articles BB for that number. Then, the process returns to the calling process.

By performing the above processing, the individual articles BB can be detected mainly based on the overlapping state in the arrangement direction BB.

For example, as shown in FIG. 12A, in the case of an image frame viewed from the side of a group of items, individual items BB represented by dotted lines can be presented for each of nine items appearing in the image frame. In addition, as shown in FIG. 12B, in the case of an image frame in which an article group is viewed obliquely from above, an individual article BB represented by a dotted line can be presented for each of the 14 articles appearing in the image frame. becomes.

When performing the 3D arrangement pattern generation process described below, data on how the individual article BB was detected is held. For example, in the case of the product area of the horizontal BB and the vertical BB, the data "horizontal and vertical" are held. When the setting is made from the estimation of the number of articles in the horizontal BB, the data "horizontal" is held.

Next, 3D arrangement pattern generation processing will be described with reference to FIGS. 13 to 15. FIG.

The 3D arrangement pattern generation unit 107 uses the processing results (data stored in the image data storage unit 102, the first data storage unit 104, and the second data storage unit 106) for each image frame for the same product group, A virtual network is generated for each image frame (step S51).

Specifically, one node is provided for each individual article BB, and the above attributes (for example, "horizontal and vertical") of the individual article BB are inherited. If the nodes are related to individual product BBs included in the same horizontal BB, they are horizontally connected by virtual links, and if the nodes are related to individual product BBs contained in the same vertical BB, they are connected vertically by virtual links, Nodes for individual products BB included in the same depth BB are connected by virtual links in the depth direction.

In the example of the image frame shown in FIG. 12A, as shown in FIG. 14A, nodes are set for each of the nine individual articles BB, and the nodes are horizontally and vertically connected by virtual links to form a virtual network. generated. Each node has a "horizontal and vertical" attribute.

In the example of the image frame shown in FIG. 12B, as shown in FIG. 14B, nodes are set for each of the 14 individual articles BB, virtual links are set in the horizontal direction, vertical direction and depth direction, and the virtual links are set as shown in FIG. A different virtual network is created. In FIG. 14B, nodes n1 to n6 have a “horizontal and depth” attribute, nodes n7 and n8 have a “vertical and depth” attribute, and other nodes have “horizontal and vertical” attributes. ” attribute.

In the case of an image frame in which such a group of articles is viewed from directly above, as schematically shown in FIG. Link. Each node has a "horizontal and depth" attribute.

Next, the 3D arrangement pattern generation unit 107 associates the virtual networks generated for each image frame for the same product group, and integrates the virtual networks (step S53).

For example, when a virtual network for FIGS. 14A to 14C is obtained, while confirming that the link structures are at least partially matching, based on the criteria that the similarity of the image features in the individual product BB is a threshold value or more, do the integration.

The partial network of the lower two stages in FIG. 14A has the same link structure as the partial network of the lower two stages in FIG. 14B, including the attributes of the nodes. These correspondences can be identified because the image features in the individual article BB of those parts are also similar. On the other hand, the link structure of the remaining portion does not match, and the image feature amount near the n5 node in FIG. Preferentially adopt node groups. When the virtual network in FIG. 14C and the virtual network in FIG. 14B are compared, the partial networks of the upper two stages in both have the same link structure including node attributes. These correspondences can be identified because the image features in the individual article BB of those parts are also similar. On the other hand, the link structure of the remaining part does not match, and the image feature values of the node immediately below the n5 node in FIG. 14B and the node in the middle of the bottom row in FIG. are preferentially adopted. For details of such processing, see, for example, Norikazu Tsunekawa et al., "Sequential 3D Geometric Modeling from 3D Boundaries of Moving Objects", Information Processing Society of Japan, Research Report, p9-16, 2004.3. See 5, etc.

As described above, the virtual networks are integrated in consideration of the closeness of the image feature amounts while associating the virtual networks with respect to a plurality of image frames. In the case of the examples of FIGS. 14A to 14C, the virtual network shown in FIG. 14B is adopted as a result.

In other words, a node newly appearing when the type of arrangement direction BB does not change due to movement of the viewpoint is assumed to be an object that was out of the field of view, and the new node and the virtual link between the nodes are established. to adopt.

In addition, a new node that appears when the number of types of arrangement directions BB increases due to movement of the viewpoint is regarded as a hidden item, and a virtual link between the new node and the node is adopted. .

Furthermore, a node that disappears when the number of types in the arrangement direction BB increases due to movement of the viewpoint is regarded as having actually no article, and the disappearing node is rejected. Conversely, a node that appears when the number of types of array direction BB is reduced is regarded as erroneous recognition and is not adopted.

The 3D arrangement pattern generation unit 107 then generates a 3D arrangement pattern from the integrated virtual network, and stores data on the 3D arrangement pattern in the third data storage unit 108 (step S55).

For example, in the example of FIG. 14B, the 8 items arranged in the foreground are arranged as they are in the arrangement represented by the virtual network, and the items arranged in the back are arranged as they are in the arrangement represented by the virtual network. The arrangement of the articles arranged on the top is represented by the virtual network, but the other parts are not represented by the virtual network. Therefore, the invisible parts are represented by the virtual network and are arranged in the same arrangement as the visible parts, that is, the number in the horizontal direction, the number in the vertical direction, and the number in the depth direction are the same, and the 3D arrangement pattern is set. Generate.

In the case of FIG. 14B, a 3D arrangement pattern as schematically shown in FIG. 15 is generated. A 3D arrangement pattern is, for example, a cubic representation of one article. In this example, the bottom row has 3×3 items, the second row also has 3×3 items, and the top row has no items in the center, which is the frontmost, but the others. are arranged in the same manner as the bottom and second tiers.

Generating such a 3D arrangement pattern facilitates object recognition by the robot 200 .

Next, missing item detection processing by the missing item detection unit 109 will be described. Various methods are possible for the out-of-stock detection processing. For example, if a 3D arrangement pattern such as that shown in FIG. 15 can be obtained, a missing product can be specified by comparing it with a basic 3D arrangement pattern or past 3D arrangement patterns.

For example, if a past 3D arrangement pattern as shown in FIG. 16 is stored, the product f in FIG. 16 can be identified as being different from the 3D arrangement pattern in FIG. As shown, a missing item may be indicated by showing a frame (dotted line) at a position corresponding to the item f in the image frame.

As another method, the missing item position may be identified by counting the number of individual items BB included in the arrangement direction BB. In the case shown in FIG. 12B, the number of individual articles BB in the four horizontal BBs is three in common, but the number of individual articles BB in the two vertical BBs on both sides is "3", but the vertical BB in the middle The number of individual articles BB in is "2". Similarly, the number of individual articles BB in the two depths BB on both sides is "3", but the number of individual articles BB in the middle depth BB is "2". This identifies the presence of a missing item in the upper portion of the middle vertical BB.

Next, foreign matter detection processing will be described with reference to FIGS. 18 to 20. FIG.

First, the foreign object detection unit 110 extracts specific image frames in which only the vertical BB and horizontal BB and only the horizontal BB and depth BB are detected from a plurality of image frames stored in the image data storage unit 102 (step S61). ). That is, an image frame in which the group of articles is viewed almost from the side and an image frame in which the group of articles is viewed from almost directly above are extracted. For example, assume that an image frame as shown in FIG. 19 is obtained. In addition, although the English character string "CUP" is written on the side of the item, the items other than the central item are arranged so that the "C" can be seen, and the central item is placed so that the "P" can be seen. are placed.

Next, the foreign matter detection unit 110 calculates feature vectors representing the shape and color in the image for each individual article BB specified in the specific image frame (step S63). Various types of such feature vectors are known, and for example, CS-LB descriptors (statistical information of image edges) and hue distribution descriptors are used.

Foreign matter detection unit 110 also calculates the average of the feature vectors for each specific image frame (step S65).

Further, the foreign object detection unit 110 identifies individual articles BB whose distance from the average of the feature vectors is equal to or greater than a threshold for each specific image frame (step S67). As a result, an individual article BB that differs from the average individual article BB in at least one of shape and color, that is, an individual article BB that is a foreign substance (including a different posture) is specified.

In the example of FIG. 19, as indicated by the dotted line frame in FIG. 20, the central article is identified and indicated as a foreign object (or different posture).

If a database with posture data for each object to be detected is available, prepare a learned model that can output the item ID and posture type by performing machine learning on the data accumulated in this database with posture data. Foreign matter detection processing can also be performed by That is, by inputting the image of each individual product BB into this trained model, the product ID and posture type can be obtained. K. Suzuki, Y. Yoshiyasu, A. Gabas, F. Kanehiro, and E. Yoshida, “Toward 6 DOF Object Pose Estimation with Minimum Dataset,” Proc. of the 2019 IEEE/SICE International Symposium on System Integration, pp.462-467. 2019.

Although the embodiment of the present invention has been described above, the present invention is not limited to this. For example, the processing flow is an example, and as long as the processing result does not change, the order of steps may be changed or multiple steps may be executed in parallel. The functional configuration example in FIG. 1 is also an example, and may not match the program module configuration. Further, the information processing apparatus 100 may be implemented by one computer or may be implemented by a plurality of computers. Further, the information processing apparatus 100 may be integrated with the robot 200 or may be provided at a remote location.

The information processing apparatus 100 described above is a computer apparatus, and as shown in FIG. A display control unit 2507 connected to 2509 , a drive device 2513 for a removable disk 2511 , an input device 2515 and a communication control unit 2517 for connecting to a network are connected via a bus 2519 . Note that the HDD may be a storage device such as a solid state drive (SSD). An operating system (OS) and an application program for performing processing in the embodiment of the present invention are stored in the HDD 2505 and read from the HDD 2505 to the memory 2501 when executed by the CPU 2503. . The CPU 2503 controls the display control unit 2507, the communication control unit 2517, and the drive device 2513 according to the processing content of the application program to perform predetermined operations. In addition, data in the middle of processing is mainly stored in the memory 2501, but may be stored in the HDD 2505 as well. In the embodiment of the present technology, an application program for performing the processing described above is stored and distributed in a computer-readable removable disk 2511 and installed from the drive device 2513 to the HDD 2505. It may be installed in the HDD 2505 via a network such as the Internet and the communication control unit 2517 . Such a computer device implements the various functions described above through organic cooperation between hardware such as the CPU 2503 and memory 2501 described above and programs such as the OS and application programs. .

It should be noted that data used by executing the above-described processing is stored in a storage device such as the memory 2501 or the HDD 2505 regardless of whether it is in the middle of processing or processing results.

The embodiments described above are summarized as follows.

The information processing method according to the present embodiment provides an image of a group of articles arranged in line, in which the area of the group of articles continues in the vertical direction, which is parallel to the direction of gravity, and the area of the group of articles continues in the horizontal direction. , and a region of a group of articles continuous in a depth direction orthogonal to the vertical direction and the horizontal direction, from the input image, a first region of the group of items that is continuous in the vertical direction; It includes a process of detecting a second area of the group of articles continuing in the horizontal direction and a third area of the group of articles continuing in the depth direction.

Such a trained model has not been constructed and used so far, and grasping the arrangement direction is an important factor in recognizing the arrangement state of a group of articles arranged in line. By using this processing result, it is also expected to improve the accuracy of detecting individual articles in a group of articles that are aligned.

In the above information processing method, in the input image, a fourth area for each article included in the article group is generated based on the overlapping state of at least two of the first area, the second area, and the third area. You may make it include the process which specifies. There are various ways to utilize the first to third regions, but by paying attention to the overlapping state in this way, it is possible to effectively detect individual articles.

In addition, in the process of specifying the fourth area described above, if there is an area that does not overlap with other areas among the first area, the second area, and the third area, in the product group A second trained model trained to detect regions of individual articles may identify a fourth region for each article for which there is no overlap with the other regions. . If there are not many articles arranged, there may be no overlap, so in that case, general object detection is performed.

Further, in the process of identifying the fourth region described above, a difference region between any one of the first to third regions and the sum region of the fourth region detected in any of the regions is detected. If the difference area is larger than one fourth area, a new fourth area may be identified based on the size of the detected fourth area. If there is not much overlap between regions, estimation using the overlapping portion may be performed.

Furthermore, the information processing method generates a virtual network representing the fourth area with nodes and representing adjacency relationships of the fourth area with links in the vertical, horizontal and depth directions, and generating a virtual network The method may further include a process of determining a three-dimensional arrangement pattern for the group of articles based on . By grasping the pattern of the three-dimensional arrangement in this way, it becomes possible to easily perform a robot operation for a group of articles.

In addition, the information processing method may further include a process of specifying a missing position of the article in the three-dimensional arrangement pattern based on the three-dimensional arrangement pattern. For example, missing items can be easily detected from the difference from the past or reference three-dimensional arrangement pattern.

Note that the above information processing method includes the number of fourth regions included in the first region, the number of fourth regions included in the second region, and the number of fourth regions included in the third region. Based on the above, a process of specifying the missing position of the article may be further included. A part against the regularity of the number becomes a missing position.

Furthermore, the information processing method may further include a process of specifying a fourth area for a foreign object or an article with an abnormal posture in the article group based on the image data in the fourth area. A foreign object or a different posture may be detected from the feature vector of the image data, or the identifier of the article and the identifier of the posture may be extracted from the image data to specify the comparatively small number of fourth regions. good.

A program for causing a computer to execute the information processing method described above can be created, and the program is stored in various storage media.

Further, the information processing apparatus that executes the information processing method as described above may be realized by a single computer, or may be realized by a plurality of computers. shall be called a system or simply a system.

100 Information processing device 101 Image acquisition unit 102 Image data storage unit 103 Arrangement direction detection unit 104 First data storage unit 105 Individual article detection unit 106 Second data storage unit 107 3D arrangement pattern generation unit 108 Third data storage unit 109 Missing item Detection unit 110 Foreign object detection unit

Claims (12)

In the image of the group of articles arranged in line, the area of the group of articles continuing in the vertical direction parallel to the direction of gravity, the area of the group of articles continuing in the horizontal direction, and the area of the group of articles continuing in the horizontal direction are orthogonal to the vertical direction and the horizontal direction. A first area of the vertically continuous article group and a first area of the horizontally continuous article group are detected from the input image by a trained model trained to detect the area of the group of articles continuous in the depth direction. A program for causing a computer to execute a process of detecting a second area of the above and a third area of the article group that is continuous in the depth direction. a fourth area for each article included in the article group in the input image based on the overlapping state of at least two of the first area, the second area, and the third area; 2. The program according to claim 1, further causing said computer to execute specifying processing. In the process of identifying the fourth area,
If there is an area that does not overlap with other areas among the first area, the second area, and the third area, learning is performed to detect the area of each individual article in the article group. 3. The program according to claim 2, wherein the second learned model specifies a fourth area for each article for areas that do not overlap with the other areas. In the process of identifying the fourth area,
If the difference area between any one of the first to third areas and the sum area of the fourth area detected in any of the areas is larger than one detected fourth area, the difference area 4. The program according to claim 2 or 3, wherein a new fourth area is specified based on the size of the detected fourth area. generating a virtual network that represents the fourth area with nodes and represents adjacency relationships of the fourth area with links in the vertical direction, the horizontal direction, and the depth direction;
5. The program according to any one of claims 2 to 4, further causing said computer to execute a process of determining a three-dimensional arrangement pattern of said article group based on said generated virtual network. 6. The program according to claim 5, further causing the computer to execute a process of specifying missing positions of articles in the three-dimensional arrangement pattern based on the three-dimensional arrangement pattern. Based on the number of fourth regions included in the first region, the number of fourth regions included in the second region, and the number of fourth regions included in the third region, the article 5. The program according to any one of claims 2 to 4, further causing said computer to execute a process of identifying missing positions. 5. The computer according to any one of claims 2 to 4, further causing the computer to execute a process of specifying a fourth area for a foreign object or an article with an abnormal posture in the article group based on the image data in the fourth area. One described program. In the image of the group of articles arranged in line, the area of the group of articles continuing in the vertical direction parallel to the direction of gravity, the area of the group of articles continuing in the horizontal direction, and the area of the group of articles continuing in the horizontal direction are orthogonal to the vertical direction and the horizontal direction. A first area of the vertically continuous article group and a first area of the horizontally continuous article group are detected from the input image by a trained model trained to detect the area of the group of articles continuous in the depth direction. and a third area of the group of articles that are continuous in the depth direction. An information processing method in which a computer executes a process. a fourth area for each article included in the article group in the input image based on the overlapping state of at least two of the first area, the second area, and the third area; The information processing method according to claim 9, further comprising identifying. In the image of the group of articles arranged in line, the area of the group of articles continuing in the vertical direction parallel to the direction of gravity, the area of the group of articles continuing in the horizontal direction, and the area of the group of articles continuing in the horizontal direction are orthogonal to the vertical direction and the horizontal direction. A first area of the vertically continuous article group and a first area of the horizontally continuous article group are detected from the input image by a trained model trained to detect the area of the group of articles continuous in the depth direction. and means for detecting a third area of the group of articles continuous in the depth direction. a fourth area for each article included in the article group in the input image based on the overlapping state of at least two of the first area, the second area, and the third area; 12. The information processing system according to claim 11, further comprising identifying means.

Images