JP2014229153A - Article identification system and program for the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately divide individual articles which are in contact with each other.SOLUTION: A color image and distance image of a plurality of articles which are in contact with each other are acquired and required units in the distance image are converted to distance from a reference plane. While outlines of the articles are extracted from the distance from the reference plane and inflection points in the outline are extracted, division candidate lines are generated so that the distances from the reference plane are connected through inflection points by going through small troughs. The plurality of articles which are in contact with each other are divided into individual articles by the division candidate lines. Types of divided individual articles can be discriminated by image recognition.

Description

  The present invention relates to identification of articles such as bread, and more particularly to a system for identifying the type of an article by image recognition and a program thereof.

  The applicant is considering identifying food such as bread, vegetables, fruits, or tablets packed in a transparent bag by image recognition (for example, Japanese Patent Application Laid-Open No. 2011-170745). One problem is that when individual articles, i.e. individual items, are in contact with each other, it is difficult to cut them into pieces on the image, i.e. to extract an image of the individual article.

  Patent Document 2 (Japanese Patent Application Laid-Open No. 2012-198848) proposes that a bend point is extracted from an outline of a pan image and a boundary line candidate between pan and pan is generated so as to connect the bend points. Yes. And in the true boundary line, the boundary line candidate is short, the contour is deeply constricted, the contour is bent at an acute angle around the inflection point, and the circle whose diameter is the boundary line does not protrude from the pan. Yes. Patent Document 2 proposes separating individual breads using candidates having the above-described features as boundaries.

  Non-Patent Document 1 (11th Information Science and Technology Forum 2012 Proceedings H-019) acquires RGB images and distance images for pans that are in contact with each other on a tray, and uses the watershed method from distance images. Proposed to cut individual bread. For example, a point where the distance from the tray becomes maximum (a point where panning is convex when viewed from the camera) is used as a starting point, and an area with a small distance gradient starting from the starting point is merged into one area. The image is divided so that is a boundary. When bread separation is successful, one bread becomes one area.

JP2011-170745 JP2012-198848

11th Information Science and Technology Forum 2012 Proceedings H-019

  In the method of Patent Document 2, when articles having simple shapes are in contact with each other, it is generally possible to correctly separate them. However, when an article having a complicated shape is in contact, it is difficult to accurately divide the article into individual pieces. In the method of Non-Patent Document 1, there is a tendency to cut into areas that are finer than actual articles, and it is difficult to accurately integrate these areas.

  An object of the present invention is to accurately separate articles that are in contact with each other.

The article identification system of this invention is
An imaging device for acquiring a color image and a distance image of a plurality of articles in contact with each other;
A coordinate conversion unit that converts a required portion of the distance image into a distance from a reference plane;
A contour extraction unit that extracts a contour of an article from a distance from a reference plane;
For the contour obtained by the contour extraction unit, a bending candidate point generating unit that extracts a bending point of the contour and generates a dividing candidate line that connects the bending points through a valley having a small distance from the reference plane;
An individual product identification unit that identifies, by image recognition, the types of individual articles cut by the division candidate lines.

The present invention also provides an imaging device that acquires color images and distance images of a plurality of articles that are in contact with each other;
A program for an article identification system comprising a computer for separating a plurality of articles that are in contact with each other into individual articles based on a color image and a distance image from an imaging device, and identifying a type of each article. There,
The computer,
A coordinate conversion unit that converts a required portion of the distance image into a distance from a reference plane;
A contour extraction unit that extracts a contour of an article from a distance from a reference plane;
For the contour obtained by the contour extraction unit, a bending candidate point generating unit that extracts a bending point of the contour and generates a dividing candidate line that connects the bending points through a valley having a small distance from the reference plane;
It functions as an individual product identification unit for identifying the types of individual articles cut by the division candidate lines by image recognition, and as an individual product identification unit for identifying the types of individual articles cut by the division candidate lines by image recognition.

  The required part of the distance image is, for example, the outline of the article or the division candidate line. Even if the entire distance image is converted into an image of the distance from the reference plane (normalized distance image), only the required part is converted into the distance. May be. Preferably, based on the lightness on the division candidate lines in the color image, a plurality of articles in contact with each other are separated from each other by deleting the division candidate lines with high lightness and leaving the division candidate lines with low lightness. It further includes a verification unit that divides the data. Even when the resolution of the distance image is low, it is possible to prevent one object from being separated into two objects by deleting the division candidate lines having high brightness.

  A coordinate conversion unit converts the distance from the imaging device into a distance from a reference plane such as the bottom surface of the tray. As a result, the indentation and the boundary of the article become a valley of distance, and the portion where the article protrudes becomes a distance mountain or ridge. The outline of the article can be extracted by extracting an edge based on the distance from the reference plane or binarizing the surface higher than the reference plane. However, the boundary where the articles are in contact with each other may not be recognized as a contour. In the present invention, an individual product may be cut out using a distance from the reference plane, and an image of the distance from the reference plane may be referred to as a normalized distance image.

The method of Patent Document 2 is applied to the normalized distance image, and the inflection point of the contour is extracted. Next, when dividing candidate lines are generated so as to connect the bending points through a valley having a small distance from the reference plane, the articles in contact with each other can be cut into individual articles. Since the inflection points are connected through positions close to the reference plane, it is possible to accurately extract the division candidate lines. The separated individual articles can be identified by image recognition in the same manner as in the conventional method.
However, if the article has a complicated shape and the resolution of the distance image is insufficient, a false division candidate line may be generated. The false division candidate line corresponds to the constriction of the article and has a feature that the brightness in the color image is high. On the other hand, there is a feature that the brightness of the division candidate line is low at the true article boundary. Therefore, if the candidate dividing lines are verified based on the brightness in the color image, a plurality of articles can be more accurately divided into individual articles.

  The color image may be used as an RGB image or may be converted into an image such as HSV. Further, as the brightness, an average value of R value, G value, and B value in an RGB image may be used. This average value is an approximate value of brightness. In order to easily handle the lightness on the division candidate line, for example, it is preferable to use an average value, a median value, or the like of the lightness on the division candidate line. For the validity of the bending point, the method described in Patent Document 2 may be used as appropriate. The reference plane is not limited to the bottom surface of a container such as a tray, and may be, for example, the top surface of a table on which the tray is placed.

The figure which shows the use environment of the article | item identification system of an Example. Block diagram of article identification system of embodiment The flowchart which shows the article | item identification algorithm of an Example. The figure which shows the meaning which pays attention to the lightness on the dividing line in an Example Grayscale image of bread on tray Distance image from the bottom of the tray Grayscale image of bread on tray The figure which shows the separation result based only on the color image of bread Distance image from the bottom of the tray The figure which shows the dividing line which connects between the inflection points of a distance image by the valley of a distance image The figure which shows the result of carving the pan when the dividing line is generated so that the inflection points of the distance image are connected by the valley of the distance image, and the average value of the brightness on the dividing line is deleted above the threshold value Figure showing the result of panning when the watershed method is applied to a distance image The figure which shows the result of carving the bread by extracting the edge from the distance image

  The optimum embodiment is shown below. The scope of the present invention should be determined according to the understanding of those skilled in the art based on the description of the scope of the claims, taking into account the description of the specification and well-known techniques in this field.

  In FIG. 1, reference numeral 2 denotes an article identification device, which is composed of a personal computer or the like, and constitutes an article identification system together with an imaging device 6 accommodated in, for example, a table 4 and attached to an arm 5. The imaging device 6 used in the embodiment is inexpensive and has a low resolution of the distance image. 8 picks up an RGB image with a color camera. Reference numeral 9 denotes an IR (infrared) projector, and reference numeral 10 denotes an IR (infrared) camera. The image of the grid points projected from the projector 9 onto the tray 11 and the pans 12 and 13 is captured by the IR camera 10, and the distance from the camera 10 is determined. Ask. The imaging device 6 outputs a distance image and a color image such as an RGB image, such as the tray 11 and pans 12 and 13 that are in contact with each other. A container such as a bucket, a belt conveyor, or the like may be used instead of the tray 11, and vegetables, fruits, fish, meat, tablets, or the like may be recognized instead of bread. Further, although a plurality of articles that are in contact with each other are identified, one article may be identified. The table 16 is provided with a POS device 18 and performs accounting processing based on the identification data of individual items from the item identification device 2. The output destination of data from the article identification device 2 is arbitrary. Reference numeral 14 denotes a storage medium which stores an article identification program for causing the article identification apparatus 2 to execute the algorithm shown in FIG. 3 and to function as shown in FIG.

  FIG. 2 shows the article identification device 2, and FIG. 3 shows its operation. The input is an RGB image and a distance image from the imaging device 6 (step 1), and the HSV converter 24 converts the RGB image into an HSV image (image of hue, saturation, and brightness) (step 2). When only brightness is used for identification of individual items, it is only necessary to convert to an image of brightness instead of an HSV image. The coordinate conversion unit 26 converts the image of the distance from the imaging device 6 into the image of the distance from the bottom surface of the tray 11 (Step 3). For example, when the tray 11 having the same shape is always used, the bottom surface of the tray 11 having no article is imaged, and the equation of the reference plane is obtained from the distance image and stored. Then, using this equation, the distance image is coordinate-transformed so as to represent the distance from the reference plane. The coordinate conversion method is arbitrary.

  The memory 28 stores an HSV image, the memory 30 stores an image representing a distance from the imaging device 6, and the memory 32 stores an image of a distance from the bottom surface of the tray 11, that is, a normalized distance image. The contour extraction unit 34 extracts the contour of the article from the normalized distance image, and extracts the contour by binarizing the normalized distance image with an appropriate threshold, for example, using an edge in the normalized distance image ( Step 4). The threshold value may be fixed or variable.

  The bending point detection unit 35 detects the bending point of the contour (step 5). The condition of the bending point is that the tangential direction or normal direction of the contour changes greatly. A line connecting two valley points via a valley of a distance image, that is, a route having a height from the tray 11 of a predetermined value or less is set as a division candidate line, and is generated by the division candidate line generation unit 36 (step) 6). The first condition is that the candidate division line passes through the valley, and other than this, the circle whose diameter is the candidate division line does not protrude from the outline, and the bend at the bending points at both ends is remarkable. High nature. The valley means that the height from the tray is lower than the area on both sides of the valley, and for example, the height gradient from the valley to the both outsides may be + or 0.

  The verification unit 38 deletes the division candidate line when the average value of the brightness on the division candidate line is equal to or greater than the threshold value, and leaves the average value of the brightness on the division candidate line that is less than the threshold value. A plurality of articles in contact are cut into individual articles (step 7). A median value or the like may be used instead of the average value. If the reliability of the division candidate line is low, the operator may delete the division candidate line, add it, or modify it (step 8). When divided into individual articles, the distance image and the HSV image or RGB image are read from the memories 30 and 32 (step 9). Based on the texture, shape, hue and saturation, brightness, etc. of the article, the type of the article can be identified by the individual item identification unit 42 with reference to the data in the individual item identification database 40 (step 10). In particular, since there is a distance image, the volume of the article and the average and dispersion of the distance from the reference plane can be added to the shape feature, and the degree of unevenness on the article surface can be added to the texture feature. The recognition result is output to the POS device 18 or the like (step 11).

  The meaning of using the lightness on the division candidate line is shown in FIG. 44 is a constriction of the article, which is a valley of the distance image, 45 is a boundary between the articles, and is also a valley of the distance image. In general, the resolution of the distance image is lower than the resolution of the RGB image, so that there is a limit in identifying the constriction 44 and the boundary 45 from only the distance image. In many cases, the bottom surface of the constriction 44 has an area parallel to the bottom surface of the tray 11. In this area, light rays are reflected like a chain line, so that the brightness is increased. On the other hand, the boundary 45 rarely has an area parallel to the bottom surface of the tray 11. For this reason, the RGB image or grayscale image becomes bright at the constriction 44, and the RGB image or grayscale image becomes dark at the boundary 45.

  FIG. 5 shows a gray scale image of a pan on the tray, and FIG. 6 shows a distance image normalized with respect to the tray. From the image of FIG. 5, individual pans can be cut out only from the color image. In the normalized distance image of FIG. 6, the bottom surface of the tray 11 and the pan can be accurately separated.

  FIG. 7 shows a grayscale image where constricted and complex shaped breads are in contact with each other. When a bending point and a division candidate line are generated from the contour of a brightness image as in Patent Document 2, it is as shown in FIG. 8, and many unnecessary division candidate lines are included.

  A normalized distance image corresponding to FIG. 7 is shown in FIG. 9, and a bending point is extracted from the contour of the distance image as in Patent Document 2. Although there are many inflection point pairs, a pair of inflection points that can be connected by a trough in the normalized distance image is extracted, and a trough in the normalized distance image at the extracted inflection point pair is used as a division candidate line.

  The candidate division lines are as shown in FIG. 10 and include unnecessary candidate division lines. This is because the resolution of the distance image is insufficient and the contact portion of the article is imaged higher than the actual from the reference plane. Increasing the resolution of the range image can solve this problem, but the imaging device becomes expensive. Therefore, if the average value of the brightness on the division candidate line is deleted, a plurality of pans that are in contact with each other in a complicated shape can be separated as shown in FIG.

  FIG. 12 shows an example in which the Watershed method in Non-Patent Document 1 is not effective, using a color image and a normalized distance image, and a point where the distance is maximum from the bottom of the tray, that is, a convex portion on the article is a marker. The initial value of the area. If the area is enlarged by merging an area with a small distance gradient in this area, the result will be in the lower right, and it is difficult to integrate the areas, so the articles cannot be accurately separated. When the Watershed method is applied to an RGB image, the result on the lower left is obtained, and the article cannot be separated accurately.

  FIG. 13 shows an example of failure when an article is cut out from an edge in a normalized distance image, and an outline with incomplete pan separation can be obtained with only the edge. Panning can be correctly separated by detecting a bending point, generating a candidate dividing line, and deleting a candidate dividing line with high brightness. If an edge in a color image, for example, a lightness edge is used, the lower right donut in the input image is recognized as two articles.

  In the embodiment, identification of bread is shown. However, articles to be identified such as vegetables, fruits, seafood, meat, and medicine tablets are arbitrary. Range image acquisition is not limited to using an IR projector and IR camera.

2 Item identification device 4 Table 5 Arm 6 Imaging device 8 Color camera 9 IR projector 10 IR camera 11 Tray 12, 13 Pan 14 Storage medium 16 Table 18 POS device 20 Bus 22 Input unit 24 HSV conversion unit 26 Coordinate conversion units 28-32 Memory 34 Outline extraction unit 35 Bending point detection unit 36 Division candidate line generation unit 38 Verification unit 40 Item identification database 42 Item identification unit 44 Constriction 45 Boundary

Claims (3)

An imaging device for acquiring a color image and a distance image of a plurality of articles in contact with each other;
A coordinate conversion unit that converts a required portion of the distance image into a distance from a reference plane;
A contour extraction unit that extracts a contour of an article from a distance from a reference plane;
For the contour obtained by the contour extraction unit, a bending candidate point generating unit that extracts a bending point of the contour and generates a dividing candidate line that connects the bending points through a valley having a small distance from the reference plane;
An article identification system comprising: an individual article identification unit that identifies, by image recognition, the types of individual articles cut out by division candidate lines.   Based on the lightness on the division candidate lines in the color image, the division candidate lines with high lightness are deleted and the division candidate lines with low lightness are left, so that a plurality of articles that are in contact with each other are cut into individual articles. The article identification system according to claim 1, further comprising a section. An imaging device for acquiring a color image and a distance image of a plurality of articles in contact with each other;
A program for an article identification system comprising a computer for separating a plurality of articles that are in contact with each other into individual articles based on a color image and a distance image from an imaging device, and identifying a type of each article. There,
The computer,
A coordinate conversion unit that converts a required portion of the distance image into a distance from a reference plane;
A contour extraction unit that extracts a contour of an article from a distance from a reference plane;
For the contour obtained by the contour extraction unit, a bending candidate point generating unit that extracts a bending point of the contour and generates a dividing candidate line that connects the bending points through a valley having a small distance from the reference plane;
A program that functions as an individual product identification unit that identifies, by image recognition, an individual product type that is identified by an image recognition, and an individual product identification unit that identifies, by image recognition, an individual product type that has been segmented by a verification unit.