JP6151562B2 - Article identification system and its program - Google Patents

Description

  The present invention relates to identification of articles such as bread, and more particularly to a system for identifying the type of an article by image recognition and a program thereof.

  The applicant is considering identifying food such as bread, vegetables, fruits, or tablets packed in a transparent bag by image recognition (for example, Japanese Patent Application Laid-Open No. 2011-170745). One of the problems is that when a plurality of articles are stacked one above the other, it is difficult to separate them into individual articles.

  2 shows prior art relating to separation of articles in contact with each other. Patent Document 2 (Japanese Patent Application Laid-Open No. 2012-198848) proposes that a bend point is extracted from an outline of a pan image and a boundary line candidate between pan and pan is generated so as to connect the bend points. Yes. And in the true boundary line, the boundary line candidate is short, the contour is deeply constricted, the contour is bent at an acute angle around the inflection point, and the circle whose diameter is the boundary line does not protrude from the pan. Yes. Patent Document 2 proposes separating individual breads using candidates having the above-described features as boundaries.

  Non-Patent Document 1 (11th Information Science and Technology Forum 2012 Proceedings H-019) acquires RGB images and distance images for pans that are in contact with each other on a tray, and uses the watershed method from distance images. Proposed to separate individual breads. For example, a point where the distance from the tray becomes maximum (a point where panning is convex when viewed from the camera) is used as a starting point, and an area with a small distance gradient starting from the starting point is merged into one area. The image is divided so that is a boundary. When bread separation is successful, one bread becomes one area. However, no method has been proposed for separating vertically stacked items.

JP2011-170745 JP2012-198848

11th Information Science and Technology Forum 2012 Proceedings H-019

  The subject of this invention is to isolate | separate the several articles | goods which have overlapped up and down, and to enable it to identify each article | item.

An article identification system of the present invention includes an imaging device that acquires a color image and a distance image of an article,
An outline extraction unit for extracting the outline of the article from the distance image;
An edge extraction unit that extracts an edge in a distance image and separates an article that overlaps from the edge inside the contour;
And an individual item identification unit for identifying the type of each separated article.

A program for an article identification system of the present invention includes an imaging device that acquires a color image and a distance image of an article,
A program for an article identification system comprising a computer that processes a color image and a distance image from an imaging device and separates an article that overlaps the top and bottom to identify the type of each article,
The computer,
An outline extraction unit for extracting the outline of the article from the distance image;
An edge extraction unit that extracts an edge in a distance image and separates an article that overlaps from the edge inside the contour;
It functions as an individual item identification unit for identifying the type of each separated article.

  Preferably, the individual item identification unit is configured so that the application condition of the individual item identification data is different from that of the other items with respect to the lower item. In this specification, the description relating to the item identification system also applies to the program for the item identification system as it is.

  When the articles overlap each other, an edge is generated in the distance image (FIG. 5). This edge is characterized in that it is inside the outline when the unseparated article is regarded as a single article, and the lower side of the edge and the background surface of the tray or the like are at a high position. From this, it is possible to separate the vertically stacked articles into individual articles (FIG. 19). Since a part of the lower article is hidden, there is a problem if the identification data for the individual product is applied as it is. For example, the area, aspect ratio, length, etc. of an article cannot be obtained correctly if a part of the article is hidden. On the other hand, color data such as the texture and hue of the article surface is often effective even when a part of the article is hidden. Therefore, it is preferable not to apply the data for identifying the article, which is affected when a part of the article is hidden, to the article under the other article or to change the application condition.

  As the height of the article, the height from a reference plane such as a tray, bucket, belt conveyor or the like is convenient. For this reason, it is preferable to convert the distance image into a height from these reference planes. However, the entire distance image may be converted without converting the entire distance image, but only at a position where a height is required. When the articles overlap one another, the articles are often in contact with each other without overlapping. Therefore, although the separation of the articles in contact will be described, the separation of the articles in contact does not have to be performed.

The figure which shows the use environment of the article | item identification system of an Example. Block diagram of article identification system of embodiment The flowchart which shows the article identification algorithm of an Example The flowchart which shows the article | item identification algorithm of a modification. A diagram showing a model of an article that overlaps vertically The figure which shows the meaning which pays attention to the lightness on the dividing line in an Example Grayscale image of bread on tray Distance image from the bottom of the tray Grayscale image of bread on tray Diagram showing separation results based only on pan color image Distance image from the bottom of the tray The figure which shows the dividing line which connects between the inflection points of a distance image by the valley of a distance image The figure which shows the separation result of a bread | pan when a dividing line is generated so that the inflection points of a distance image may be connected by the valley of a distance image, and the average value of the brightness on a dividing line is deleted more than a threshold Diagram showing Laplace filter Diagram showing Gaussian difference filter A diagram showing a grayscale image and outline of a pan that overlaps vertically The figure which shows the distance image corresponding to FIG. The figure which shows the image which processed the distance image of FIG. 16 with the Laplacian filter, and made it an edge image, and subtracted the edge image from the original distance image. The figure which shows the division | segmentation result at the time of generating a division | segmentation candidate line from the distance image of FIG.

  The optimum embodiment is shown below. The scope of the present invention should be determined according to the understanding of those skilled in the art based on the description of the scope of the claims, taking into account the description of the specification and well-known techniques in this field.

  In FIG. 1, reference numeral 2 denotes an article identification device, which is composed of a personal computer or the like, and constitutes an article identification system together with an imaging device 6 housed in a table 4 and attached to an arm 5. A color camera 8 captures an RGB image. Reference numeral 9 denotes an IR (infrared) projector, and reference numeral 10 denotes an IR (infrared) camera. The image of the grid points projected from the projector 9 onto the tray 11 and the pans 12 and 13 is captured by the IR camera 10, and the distance from the camera 10 is determined. Ask. The imaging device 6 outputs a distance image and a color image such as an RGB image, such as the tray 11 and pans 12 and 13 that are in contact with each other. A container such as a bucket or a belt conveyor may be used instead of the tray 11, and vegetables, fruits, fish, meat, tablets, etc. may be identified instead of bread. In addition, a plurality of articles that overlap each other are identified, but one article may be identified. The table 16 is provided with a POS device 18 and performs accounting processing based on the identification data of individual items from the item identification device 2. The output destination of data from the article identification device 2 is arbitrary. Reference numeral 14 denotes a storage medium which stores an article identification program for causing the article identification apparatus 2 to execute the algorithm shown in FIG. 3 or 4 and to function as shown in FIG.

  FIG. 2 shows the article identification device 2, and FIG. 3 shows its operation. The input is an RGB image and a distance image from the imaging device 6 (step 1), and the HSV converter 24 converts the RGB image into an HSV image (image of hue, saturation, and brightness) (step 2). When only brightness is used for identification of individual items, it is only necessary to convert to an image of brightness instead of an HSV image. The coordinate conversion unit 26 converts the image of the distance from the imaging device 6 into the image of the distance from the bottom surface of the tray 11 (Step 3). For example, when the tray 11 having the same shape is always used, the bottom surface of the tray 11 having no article is imaged, and the equation of the reference plane is obtained from the distance image and stored. Then, using this equation, the distance image is coordinate-transformed so as to represent the distance from the reference plane. The coordinate conversion method is arbitrary.

  The memory 28 stores an HSV image, and the memory 30 stores an image representing a distance from the imaging device 6. The memory 32 stores an image of the distance from the bottom surface of the tray 11, that is, a normalized distance image, and an edge image obtained by extracting an edge from the normalized distance image. Reference numeral 33 denotes a working memory. The contour extraction unit 34 extracts the contour of the article from the normalized distance image, and binarizes the normalized distance image with an appropriate threshold value, for example, by using an edge in the normalized distance image. Extract (step 4). The threshold value may be fixed or variable. The bending point detection unit 35 detects the bending point of the contour (step 5). The condition of the bending point is that the tangential direction or normal direction of the contour changes greatly.

  In step 4, it is assumed that the contour is extracted by binarization instead of the edge. The edge extraction unit 36 extracts an edge by using a Laplacian filter (FIG. 14) or the like in the contour of the article in the normalized distance image, and generates an edge image in which the pixel value becomes + at the edge position. (Step 6). Note that only the extracted edges having a predetermined height or higher may be used, and the image may be smoothed before the edges are extracted.

  When an edge is extracted in step 4, the above processing is executed in step 4. The pixel value in the edge image is subtracted from the pixel value in the normalized distance image and stored in the memory 32 (step 7). However, when the distance normalized by subtraction is negative, that is, lower than the bottom surface of the tray 11, the distance is set to zero. In addition, the subtraction may be performed by multiplying the pixel value of the edge image by an appropriate coefficient and then subtracting after the edge image is processed by a smoothing filter or the like to perform smoothing and multiplication of the coefficient. You may do it. When an edge extraction filter having a pixel value of − at the edge position is used, the pixel value of the edge image is added to the pixel value of the normalized distance image.

  When the articles overlap each other, there is a feature that the lower side of the edge is higher than the reference plane and is inside the contour extracted in step 4. The area inside the contour is divided by the edge, and the fact that it is the lower side of the edge (the lower side of the other article) and the upper side of the edge (the upper side of the other article) is stored in the memory 33 as an attribute of the area. In addition, an attribute indicating that the verification is not necessary is stored in the memory 33 for the inner edge of the contour so that the verification based on the brightness is not performed on the candidate dividing line based on the inner edge of the contour.

  FIG. 5 shows a situation in which the article 48 overlaps the articles 46 and 47 on the tray 11. Here, when the articles 46 to 48 are viewed obliquely from above or from vertically above as indicated by a chain line in the figure, an edge is present in the distance image, and this edge indicates the outline of the article 48 above. When the Laplacian filter is used, the edge appears as a positive contour in the edge image, and when the pixel value of the edge image is subtracted from the pixel value in the normalized distance image, the edge where the upper and lower articles overlap has a short distance from the tray , Changes to valleys in the normalized distance image.

  A line connecting the two inflection points via a normalized valley of the distance image, that is, a route whose height from the tray 11 is a predetermined value or less is set as a division candidate line, and a division candidate line generation unit 37 (Step 8). The division candidate line is short, a circle having the diameter of the division candidate line does not protrude from the outline, and the bending at the bending points at both ends is remarkable, so that the reliability is high. The valley is a region where the height from the tray 11 is lower than the regions on both sides of the valley. For example, the distance gradient in the direction toward both outer sides of the valley (the direction perpendicular to the valley and the outward direction) is +? It may be 0.

  The verification unit 38 deletes the division candidate line when the average value of the brightness on the division candidate line is equal to or greater than the threshold value, and leaves the average value of the brightness on the division candidate line that is less than the threshold value. A plurality of articles in contact are separated into individual articles. However, it is preferable to omit the lightness verification for the division candidate lines based on the edges in the outline, that is, the division candidate lines based on the upper and lower overlaps of the article (step 9). A median value or the like may be used instead of the average value. In addition, when the reliability of the division candidate line is low, the operator may be able to delete, add, or modify the division candidate line (step 10). When separated into individual articles, the distance image and the HSV image or RGB image are read from the memories 30 and 32 (step 11).

  Based on the texture, shape, hue and saturation, brightness, etc. of the article, the type of the article can be identified by the individual item identification unit 42 with reference to the data in the individual item identification database 40 (step 12). Since there is a distance image, the volume of the article and the average and dispersion of the distance from the reference plane can be added to the shape feature, and the degree of unevenness on the article surface can be added to the texture feature. The identification result is output to the POS device 18 or the like (step 13).

  In step 12, when identifying the type of article, the article below the edge is partially hidden. The three elements of shape, texture, and color for identifying the type of article have meaning if a part of the article is hidden, such as the area, volume, aspect ratio, and circumference to area ratio of the article. There is data to decline. On the other hand, color data such as texture and hue are often effective as identification data even if a part of the article is hidden. Therefore, the data whose meaning is reduced when a part of the article is hidden should not be applied to the article under the edge, or only the lower limit is used when identifying the area from the lower limit to the upper limit of the area. Change application conditions. As described above, the application condition of the data in the individual item identification database 40 is changed for the article below the edge.

  The meaning of using the lightness on the division candidate line is shown in FIG. 50 is a constriction of the article, which is a valley of the distance image, 51 is a boundary between the articles, and is also a valley of the distance image. In general, since the resolution of the distance image is lower than the resolution of the RGB image, there is a limit in identifying the constriction 50 and the boundary 51 from only the distance image. In many cases, the bottom surface of the constriction 50 has an area parallel to the bottom surface of the tray 11. In this area, light rays are reflected like a chain line, so that the brightness is increased. On the other hand, the boundary 51 rarely has an area parallel to the bottom surface of the tray 11. For this reason, the RGB image or the gray scale image becomes bright at the constriction 50, and the RGB image or the gray scale image becomes dark at the boundary 51. It should be noted that an edge formed by overlapping articles may be an edge having a high lightness as in the case of the constriction 50, and verification by lightness is preferably omitted.

  The algorithm of the modified example is shown in FIG. 4, the hardware of the article identification device is the same as that in FIG. 2, and the same steps as in FIG. 3 represent the same processing. Steps 6 and 7 in FIG. 3 are omitted, and step 9 is executed. Accordingly, it is possible to generate the division candidate lines so as to separate the articles that are in contact with each other without overlapping vertically. Subsequent to Step 9, Step 6 is executed within the outline of the article, and the edge between the articles that overlap vertically is extracted and used as an edge image. The edge image is subjected to a process such as smoothing, and compared with an appropriate threshold value to obtain an outline in which articles overlap, and this outline is added to the division candidate line (step 21). Further, the vertical relationship with the edge as a boundary is stored in the memory 33 or the like. Then, Steps 10 to 13 in FIG. 3 are executed. In Step 12, the application condition of the article identification data is changed for the article below the edge.

  7 and 8 show examples of breads that are in contact with each other so that they can be easily separated. FIG. 7 shows a gray scale image of the pan on the tray, and FIG. 8 shows a distance image normalized with respect to the tray. From the image of FIG. 7, individual pans can be cut out only from the color image. In the normalized distance image of FIG. 8, the bottom surface of the tray 11 and the pan can be accurately separated.

  9-13 illustrate the separation of the pans when the constricted and complex pans are in contact with each other. FIG. 9 shows a gray scale image of a pan. When a bending point and division candidate lines are generated from the contour of a lightness image as in Patent Document 2, it becomes as shown in FIG. 10, and there are many unnecessary division candidate lines. include.

  A normalized distance image corresponding to FIG. 9 is shown in FIG. 11, and when dividing candidate lines are generated from the contour of the distance image as shown in Patent Document 2, it becomes as shown in FIG. It is included. In an embodiment in which the average value of the brightness on the division candidate line is deleted from the threshold value or more, as shown in FIG. 13, a plurality of pans that are in contact with each other in a complicated shape can be separated.

FIG. 14 shows the value of the Laplacian filter, and when this filter is processed, the pixel value becomes positive at the edge. As the edge extraction filter, the Gaussian difference filter of FIG. 15 or the like can be used, and this filter is a filter obtained by adding a smoothing function to the Laplacian filter. An edge extraction filter obtained by modifying the SUSAN filter can also be used. In this filter, an even power of the difference between the distance d1 from the tray at the target pixel and the distance ds from the tray at the surrounding pixels is equal to Δ = (d1−ds) ^n. Where n is an even number. Then, exp (−Δ · k) is obtained for each surrounding pixel to obtain an added value sum. K is a positive coefficient. If there is no edge, the addition value sum is large, and the addition value sum is small at the edge. Furthermore, the Kenny edge detection algorithm may be modified. For example, for each pixel of interest in the normalized distance image, the absolute value of the gradient and the gradient direction are calculated, and the pixel having the maximum gradient absolute value is connected with priority given to being perpendicular to the gradient direction, and the edge May be extracted. However, the SUSAN filter and Kenny's edge detection algorithm are computationally expensive compared to the Laplacian filter and Gaussian difference filter.

  FIG. 16 shows three donuts that overlap vertically and their outlines, and FIG. 17 shows a normalized distance image. When an edge image obtained by processing the image of FIG. 17 with a Laplacian filter is subtracted from the original distance image, the image of FIG. 18 is obtained. The edge where the upper and lower articles overlap is converted into a valley close to the bottom surface of the tray, there is a bending point at the beginning and end of the overlap at the edge, and the edge becomes a candidate dividing line. Then, when an edge is added to the division candidate line, three donuts can be separated as shown in FIG. It should be noted that it is desirable not to verify the division candidate line based on the edge by brightness. Since the lower donut is partially missing, the type of the three donuts can be correctly identified if elements such as area, volume, aspect ratio, etc. are not used for identification or if application conditions are limited.

  When the algorithm of FIG. 4 is executed, a contour in a state where three donuts are not separated is extracted from the normalized distance image of FIG. 17 in step 4. Thereafter, by extracting edges caused by overlapping donuts in the vertical direction and using the boundaries of the upper and lower donuts as division candidate lines, the edges can be divided as shown in FIG. Also in this case, elements such as area, volume, aspect ratio, etc. are not used for identification or the application conditions are limited for the lower donut.

  In the embodiment, identification of bread is shown. However, articles to be identified such as vegetables, fruits, seafood, meat, and medicine tablets are arbitrary. In the embodiment, the separation of the articles that are in contact with each other and the separation of the articles that are stacked on top and bottom are performed, but the separation of the articles that are in contact with each other may be omitted.

2 Item identification device 4 Table 5 Arm 6 Imaging device 8 Color camera 9 IR projector 10 IR camera 11 Tray 12, 13 Pan 14 Storage medium 16 Table 18 POS device 20 Bus 22 Input unit 24 HSV conversion unit 26 Coordinate conversion units 28-33 Memory 34 Outline extraction unit 35 Bending point detection unit 36 Edge extraction unit 37 Division candidate line generation unit 38 Verification unit 40 Item identification database 42 Item identification unit 46 to 48 Article 50 Neck 51 Boundary

Claims (3)

An imaging device for acquiring a color image and a distance image of the article;
An outline extraction unit for extracting the outline of the article from the distance image;
An edge extraction unit that extracts an edge in a distance image and separates an article that overlaps from the edge inside the contour;
An article identification system comprising: an individual article identification unit that identifies a type of each separated article.   The individual item identification unit is configured to make the application conditions of the identification data for individual items different from those of other items for the items under the other items. The article identification system of claim 1. An imaging device for acquiring a color image and a distance image of the article;
A program for an article identification system comprising a computer that processes a color image and a distance image from an imaging device and separates an article that overlaps the top and bottom to identify the type of each article,
The computer,
An outline extraction unit for extracting the outline of the article from the distance image;
An edge extraction unit that extracts an edge in a distance image and separates an article that overlaps from the edge inside the contour;
A program for an item identification system that functions as an individual item identification unit for identifying the type of each separated item.