JPH09293142A - Method for recognizing article - Google Patents

Abstract

PROBLEM TO BE SOLVED: To securely identify the color of an article even if the color of the center part of the article cannot be identified and to improve the precision of article recognition by means of adding it to the identification of an outline. SOLUTION: A partial outline side is expansion-processed (S31) and statistics on the AND picture of the saturation picture of tableware and the expanded side is taken (S32). When a low saturation part is not less than a threshold, it is judged to be an achromatic part (S33) and the AND picture of the lightness picture of tableware and the expanded partial outline side is generated (S34). Then, it is divided into a low lightness part and a high lightness part with the appropriate threshold, and statistics on the number of picture elements is taken (S34). When the low lightness part is not less than the threshold, the color of the partial outline side is made into black (S35 and S36). When the low lightness part does not exist for not less than the threshold, the color of the partial outline side is made into white (S37). When the low saturation part does not exist for not less than the threshold, it is judged to be the high saturation part (S33), and the AND picture of the color picture of tableware and the expanded partial outline side is generated (S38). It is color-divided by the appropriate threshold and statistics on the number of the picture elements in the respective color divisions is taken. Then, the division with the largest number of the picture elements is set to be the color of the partial outline side (S39).

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of recognizing an article such as tableware, and more particularly to a method of recognizing an article by identifying the contour color of the article and the color of the edge.

[0002]

2. Description of the Related Art For example, in an employee cafeteria, various dishes with rice, miso soup, side dishes, etc. are placed on a tray and handed to employees. In such an employee cafeteria, various dishes with different shapes and sizes are served. A device is known that automatically determines the price of various dishes on the tray by deciding the dishes to be served and the fee, and imaging the patterns of various tableware placed on the tray with a camera to identify them. There is.

[0003]

However, if the shape and size of the tableware are changed for each dish and the number of kinds of dishes increases, it is necessary to prepare various types of tableware having different shapes and sizes accordingly. Not only that, the management of tableware becomes troublesome.
Therefore, it is possible to deal with different dishes by changing the color of the tableware even if they have the same shape and size, but in the case of tableware, the dishes are arranged inside the tableware, so the entire tableware is simply used. If the image of is taken, the color of the dish is also taken, and the color of the tableware cannot be identified.

Therefore, according to the first aspect of the present invention, even when the color of the central portion of the article cannot be identified such as when dishes are served on the tableware, the color of the article can be surely identified, and Provided is a method for recognizing an article, which can improve the accuracy of article recognition by identifying a color and an outline.

According to the second aspect of the present invention, the color of each article can be surely identified even when a plurality of articles whose central portions cannot be identified are in contact with each other. Provided is a method of recognizing an article, which can improve the accuracy of recognizing an article contacted by identification.

[0006]

According to the first aspect of the present invention,
In a pattern recognition method of recognizing an article by extracting an image of the article with an imaging camera and recognizing the contour of the article, a logical product of an image created by expanding the extracted contour and an image including color information of the article An image is created, and the color of the edge of the article is identified by taking statistics of the logical product image, thereby recognizing the article.

According to a second aspect of the present invention, in a pattern recognition method in which a plurality of articles are picked up by an image pickup camera, the contours of the picked-up articles are extracted, and each article is recognized, the extracted contours are defined as partial contour sides. Divide and create a logical product image of the image created by expanding each partial contour side and the image containing the color information of the article, and by taking the statistics of each logical product image, the edge of each article The purpose is to identify the colors and thereby identify each item.

[0008]

Embodiments of the present invention will be described below with reference to the drawings. In addition, this Embodiment describes what applied this invention to the automatic charge settlement device in a cafeteria. FIG. 1 is a block diagram showing the overall configuration. Various tableware 2, 2, ... Mounted on a tray 1 is imaged by an imaging camera 3, and an image signal from the imaging camera 3 is imaged by a tableware recognition device 4
To supply.

The tableware recognizing device 4 includes a tableware image capturing section 5, a tableware image separating section 6, a figure recognizing section 7, and a charge settlement section 8.
When the tableware image capturing section 5 captures an image signal from the image pickup camera 3, the tableware image capturing section 5 supplies the image signal to the next tableware image separating section 6. The tableware image separation unit 6 removes the background color from the image, separates only the tableware image, and supplies it to the figure recognition unit 7 in the next stage. At this time, an image having information on the saturation, hue, and brightness of the tableware is saved. The figure recognition unit 7 recognizes the shape, size, and color of the figure from the pattern of the tableware image, identifies which tableware each figure is, and supplies it to the fee settlement unit 8. The price of the dish is set in advance in the fee settlement unit 8 according to the shape, size, and color of the tableware. The fee settlement unit 8 uses the tray 1 based on the tableware identification result of the figure recognition unit 7. The total price of the dishes on the tableware 2 is calculated and displayed on the display. The customer pays the money based on the total amount displayed on the display and pays the amount.

The figure recognizing section 7 constitutes a main part and is shown in FIG.
Processing based on the flowchart shown in FIG. In this process, first, in S1, the separated tableware image is binarized and the contour is extracted. Then, in S2, decomposition into straight lines and circular arcs is performed. That is, the coordinates of the individual points forming the contour are extracted by vectorizing the extracted contour. For example, if the contour is extracted as shown in FIG. 3, the contour is decomposed into partial contour sides. As shown in FIG. 4, the decomposition into the partial contour sides is performed by searching the contour with two adjacent small vectors a and b, and if the angle formed by the vectors is equal to or larger than a threshold value, the interior angle of the rectangular tableware, or This is done by determining that the tableware is in contact with each other and deleting it. As a result, FIG.
As shown in, the partial contour sides T1, T2, T3, T4, T5
Get.

At this time, the inside angle of the quadrangle is detected, and the coordinates of the inside angle are obtained. As shown in FIG. 6, the counterclockwise direction viewed from the vector a is defined as +, that is, the angle formed by the vectors a and b is defined as +, and the clockwise direction viewed from the vector a is −, that is, the vectors a and c. The angle formed by is defined as −.

The inner angle of the quadrangle is, for example, when the contour is traced counterclockwise as shown in FIG.
Seen from above, the vector b is +90 degrees, and is −90 degrees at the portions where the quadrangles are in contact with each other. If the contour is traced clockwise, the opposite is true. In the same way, even if the directions of the vectors of a and b are made opposite to each other and traced on the contour, the relationship of the directions of the vectors becomes opposite at the contact portion of the interior angle and the quadrangle. Further, since the corners of the tableware are strictly round, the inner corner and the contact portion can be detected more reliably by opening a little between the vector a and the vector b.

The coordinates of the interior angle that meet these conditions are determined. It should be noted that the points on the contour detected by such a method may be used as the coordinates of the interior angle, but considering that the corners of the tableware are strictly round, the vector a is detected from the point detected as the interior angle.
The vector b may be separated from each other, and the equation of the straight line of the vectors a and b may be obtained and the intersection of the two straight lines may be used as the coordinates of the interior angle.
Also, a similar investigation may be performed without separating the two vectors. When the interior angle is detected by separating the vector a and the vector b in the interior angle detection, the vector a and the vector b at that time can be used as they are.
The method of obtaining the coordinates of the interior angle is shown in FIG.

As described above, the decomposition into the partial contour side, the detection of the interior angle, and the coordinates of the interior angle can be determined. Whether the partial contour side is a circular arc or a straight line is determined by, for example, creating two vectors with the same start point but different end points from the vector data of the sides, and if there is almost no angle between the vectors For example, it can be detected as an arc. Then, S3
At, the color of each separated partial contour side is obtained. This performs the processing based on the flowchart shown in FIG.

First, in S31, the decomposed partial contour side is expanded. For example, considering the partial contour side T1, the partial contour side T1 shown in FIG. 10 (a) is expanded to be thickened as shown in FIG. 10 (b). Then, S3
At 2, the statistics of the logical product image of the saturation image of the tableware 2 and the inflated side are taken. That is, the logical product image is created by performing the processes shown in (c) to (d) of FIG. 10, and the chromatic part (high-saturation part) and the achromatic part (low-saturation part) are further divided by an appropriate threshold value. Then, the statistics of the number of pixels are taken.
This corresponds to the case where there are two color sections shown in (e) of FIG.

Subsequently, in S33, it is determined whether the low saturation portion is above a certain threshold value, and if the low saturation portion is above a certain threshold value, it is determined to be an achromatic portion, and then in S34. An AND image of the brightness image of the tableware 2 and the inflated partial contour side is created. Then, it is divided into a low lightness portion and a high lightness portion by an appropriate lightness threshold value, and statistics of the number of pixels of each are taken. Subsequently, in S35, it is determined whether or not the low lightness portion is above a certain threshold value. If the low lightness portion is above a certain threshold value, it is determined in S36 that the color of the partial contour side is black,
If there is no low-brightness part above a certain threshold, S37
Then, it is determined that the color of the partial contour side is white.

If it is determined in S33 that the low-saturation portion does not exceed a certain threshold, it is determined to be a high-color portion. And
In S38, a logical product image of the hue image of the tableware 2 and the inflated partial contour side is created. Then, it is divided into several color sections by an appropriate threshold value, and the statistics of the number of pixels included in each color section are obtained. Then, in S39, the hue section having the largest number of pixels is set as the color of the partial contour side.

By performing the above-described processing of S31 to S39 for all the individual partial contour sides, the color of each partial contour side can be accurately identified, and the edge color of each tableware 2 can be obtained. That is, even if a dish is placed in the tableware 2, the tableware 2
The color of the tableware 2 can be accurately obtained by accurately identifying the color of the edge of the tableware 2. Using the straight-line side data, interior angle data, and color information of the partial contour side decomposed as described above, a quadrangle is determined in S4.

The determination of this quadrangle is made based on FIG. The processing of FIG. 11 is processing when the contour is traced leftward, and the sides shorter than a certain threshold are excluded from the determination target. That is, in S41, what angles are present at both ends of the straight side is detected and registered as data.

Subsequently, in S42, two mutually perpendicular sides forming the same figure are detected. Here, the vertical side that constitutes the same figure existing on the right side of the target side is detected. As shown in (a) of FIG. 12, first, the vector b of the side Q must be +90 degrees with respect to the vector a of the side P of interest. This principle is the same theory as when the internal angle is detected. Furthermore, it is a condition that the distance between the side P and the side Q is within a certain threshold.

However, reliable detection cannot be performed under these conditions. For example, in the case shown in FIG. 12B, the condition that the vector b of the side Q must be +90 degrees with respect to the vector a of the side P of interest is satisfied, but the sides P and Q are obviously Is not the vertical and horizontal sides of the same figure. Therefore, a vector c connecting the tip of the side P and the origin of the side Q is created, and the angle between the vector a and the vector c is −
If it is NG, if +, it is OK. In the case of (b) of FIG. 12, the angle formed is −, and therefore NG, which is not detected as two mutually perpendicular sides forming the same figure.

When the interior angle has already been detected on the right side of the side P of interest, the angle formed by the vector c and the vector a connecting the interior angle and the origin of the side Q (anywhere on Q) is 90 degrees. Although it must be, in the case of (a) of FIG.
The vector b of the side Q is + the vector a of the side P of interest
Although the condition that it has to be 90 degrees is satisfied, two quadrangles are in contact with each other, and the side P and the side Q are considered to be sides belonging to different figures, so they are set to NG. On the other hand, in the case of FIG. 13B, the angle formed by the vector a and the vector c connecting the interior angle and the origin of the side Q is 90 degrees, so that it is OK. In the figure, ○ indicates +90 that has been detected.
Indicates the interior angle of degrees.

If there is a detected interior angle of +90 degrees on the left side of the side Q, the side of the same figure is defined by the angle between the tip of the side P and the vector c on which the interior angle is formed and the vector a on the side P. Or not. In the case of FIG. 14, the angle formed by the vector c and the vector a is not 90 degrees but NG. Further, when different inner angles exist on both the right side of the side P and the left side of the side Q, it becomes NG without creating a vector connecting the angles. This is because the same figure should share the same interior angle.

As described above, by verifying the relationship between sides, the relationship between sides and interior angles, and the relationship between interior angles and interior angles,
It can be seen whether the two sides are the vertical and horizontal sides of the same figure.

Then, in S43, two parallel sides forming the same figure are detected. First, as shown in FIG. 15, the side P of interest and the side Q of the detection target have an overlapping portion, and the angle between the vector a of the side P and the vector b of the side Q must be 180 degrees. . However, this alone does not tell whether or not parallel sides that form the same figure. It is verified whether two parallel sides are the same figure by using the information of the interior angle and the vertical side detected so far.

The side P of interest and the side Q to be detected have an overlapping portion, and the angle between the vector a of the side P and the vector b of the side Q must be 180 degrees. If the interior angle exists on the right side, it is necessary to satisfy the condition that the angle between the vector c connecting the interior angle and the origin of the side Q and the vector a of the side P must be +90 degrees or more. It is considered that this happens when the quadrangles are in contact. Therefore, in the case of the figure shown in FIG. 16 (a), the angle formed by the vector c of the side P and the vector c connecting the interior angle (the circled portion in the figure) and the origin of the side Q is +90.
Since it is below the degree, it will be NG.

The side P of interest and the side Q of the detection target have an overlapping portion, and the vector a of the side P and the vector b of the side Q are included.
In addition to the condition that the angle formed by must be 180 degrees, and if there is a vertical side R that constitutes the same figure that has already been detected on the right side of the side P, the origin of the vertical side R (anywhere on R is acceptable. ) And the vector c connecting the origin of the side Q and the vector a of the side P must meet the condition that the angle must be +90 degrees or more. Therefore, FIG.
In the case of the figure shown in (b), the angle between the vector c connecting the origin of the vertical side R and the origin of the side Q and the vector a of the side P is +
Since it is below 90 degrees, it will be NG.

Under the same principle, the side P of interest and the side Q to be detected have an overlapping portion, and the vector a of the side P and the side Q
In addition to satisfying the condition that the angle formed by the vector b of 180 degrees must be 180 degrees, there is an interior angle on the right side of the side P, and the side Q
If there is an interior angle also on the left side of, the condition that the angle between the vector c connecting the interior angles and the vector a of the side P must be +90 degrees or more must be satisfied. Therefore, in the case of the figure shown in FIG. 16 (c), the angle between the vector c connecting the interior angles and the vector a of the side P is +90 degrees or less, so that the figure is NG.

Such verification is also performed for the inner corner on the left side of the side P and the vertical side. Further, the accuracy can be further improved by replacing the positions of the side P and the side Q and performing the same verification. That is, when the positions of the side P and the side Q are replaced, an inner angle exists on the left side of the side Q, and the angle between the vector c connecting the tip of the side P and the inner angle and the vector a of the side P is +90 degrees or less. In the case of the figure shown in (d) of FIG. 16, the condition that it must be satisfied must be satisfied, but since the angle between the vector c and the vector a of the side P exceeds +90 degrees, it becomes NG.

As described above, the interior angle, the vertical side and the parallel side forming the same figure are detected. continue,
In S44, the sides included in the same figure are put together and a figure number is given. Then, in S45, each figure is identified based on the determination condition of the quadrangle. That is, the size of the quadrangle is determined using the information obtained in S42 and S43.

The conditions for determining the size of the rectangle are shown in FIG.
When parallel sides P and Q exist among the sides registered as the same figure, the distance d between the parallel sides is set to the size of a quadrangle. If there is no parallel side,
As shown in FIG. 8, when the vertical side Q exists on one side of the side P of interest and the interior angle Y exists on the other side, whether the distance d between the intersection point of the sides P and Q and the interior angle Y is the size of a quadrangle. , FIG. 19
As shown in, when the inside angles Y and Z are present at both ends of the side P of interest, the distance d between the inside angles Y and Z is set to the size of a quadrangle.

When one side P of interest has a vertical side Q and the other side has a vertical side R, the distance between the intersection of sides P and Q and the intersection of sides P and R is determined. The size of the rectangle can be calculated. Further, in this case, since the side Q and the side R are parallel sides, the size of the quadrangle can be obtained even if the parallel sides exist.

If the above conditions are not met, that is, in S46, the number of figures that do not meet the conditions is 1
If any exist, replace the dishes 2 on the tray 1. As described above, by using the information on the parallel sides, the vertical sides, and the inside angles, even if a plurality of square tableware 2 are in contact with each other or the circular tableware 2 overlaps with the circular tableware 2, The pattern and size can be recognized accurately and accurately.

When the determination of the quadrangle in S4 is completed, the determination of the circle is subsequently performed in S5. In this circular judgment, the center and the radius of the arc are identified by the least squares method as in the conventional method, and the circular tableware is judged by assuming that the arc and the arc side close to the center constitute the same tableware. As a result, circular tableware can be recognized accurately.

In this way, after binarizing the image of the tableware 2 and extracting the contours, the contours are decomposed into partial contour sides, the decomposed individual partial contour sides are expanded, and the expanded side and the tableware are expanded. A chromatic portion and an achromatic portion are separated by taking a statistic of an AND image with an image with a saturation of 2 and comparing it with a threshold value. Then, for the achromatic portion, it is further determined whether the partial contour edge is white or black by obtaining the statistics of the logical product image of the image of the brightness of the tableware 2 and the expanded partial contour edge and comparing it with a threshold value. For the chromatic portion, the statistics of the image of the logical product of the hue image of the tableware 2 and the inflated side of the contour are taken, divided into several color categories by an appropriate threshold, and the number of pixels included in each color category is calculated. Since the color of the partial contour side is determined by taking the statistic of 1, the color of the edge of the tableware 2 can be accurately identified and the color of the tableware can be accurately obtained even if the dish is placed on the tableware 2. Further, even if tableware having different colors are in contact with each other, the color of the edge of each tableware can be accurately identified and the color of each tableware can be accurately obtained.

Therefore, even if the dishes have the same shape and size, but different colors, they can be judged as different dishes.
Even if tableware having the same shape and the same size but different colors are in contact with each other, the color of each tableware can be accurately identified, so that the accuracy of tableware recognition can be improved. Therefore, since different dishes having the same shape and size and different colors can be used for different dishes, the number of dishes having different shapes and sizes can be minimized, in other words, many dishes can be dealt with with a small number of dishes, The tableware can be easily managed. In addition, although this embodiment has described the case where the present invention is applied to a charge automatic settlement device in a cafeteria to recognize tableware, the present invention is not necessarily limited to this, and is also applied to recognition of articles other than tableware. it can.

[0037]

As described above, according to the first aspect of the present invention, the color of the article can be surely identified even when the color of the central portion of the article cannot be identified such as when dishes are served on a table. Therefore, the accuracy of the article recognition can be improved by identifying the color and the contour of the article. According to the invention of claim 2, the color of each article can be surely identified even when a plurality of articles whose central color cannot be identified are in contact with each other. Therefore, the accuracy of recognizing the contacted article can be improved.

[Brief description of drawings]

FIG. 1 is a block diagram of an automatic charge settlement apparatus showing an embodiment of the present invention.

FIG. 2 is a flowchart showing a graphic recognition process of a graphic recognition unit in the embodiment.

FIG. 3 is a diagram showing an example of an image contour for explaining the graphic recognition processing in FIG.

FIG. 4 is a diagram for explaining a decomposition process into partial contour sides in the image contour example of FIG. 3;

FIG. 5 is a diagram showing an image contour example in which the image contour example of FIG. 3 is decomposed into partial contour sides.

6A and 6B are views for explaining the definition of an angle when the interior angle of a quadrangle is detected in the decomposition processing into the partial contour sides in FIG.

7A and 7B are views for explaining a method of detecting an inner corner of a quadrangle and a corner of a contact portion in the decomposition processing into the partial contour sides in FIG.

8A and 8B are views for explaining how to obtain the coordinates of the inside angle of the quadrangle in the decomposition processing into the partial contour sides in FIG.

FIG. 9 is a flowchart showing in detail the processing for obtaining a side color in the graphic recognition processing of FIG.

FIG. 10 is a schematic diagram for explaining a process for obtaining a side color in FIG.

FIG. 11 is a flowchart showing in detail the quadrilateral determination processing in the figure recognition processing of FIG.

FIG. 12 is a diagram for explaining OK / NG determination conditions when a vertical side is detected in the square determination process of FIG. 11;

FIG. 13 is a diagram for explaining OK / NG determination conditions when a vertical side is detected in the square determination process of FIG. 11;

14 is a diagram for explaining NG determination conditions when a vertical side is detected in the square determination process of FIG.

FIG. 15 is a diagram for explaining a determination condition when detecting parallel sides in the determination process of the quadrangle in FIG. 11;

16 is a diagram for explaining NG determination conditions when parallel sides are detected in the square determination process of FIG.

FIG. 17 is a diagram for explaining the determination of the size of a quadrangle when there are parallel sides in the quadrangle determination process of FIG. 11.

18 is a diagram for explaining determination of the size of a quadrangle when there are no parallel sides in the quadrangle determination process of FIG. 11.

19 is a diagram for explaining determination of the size of a quadrangle when there is no parallel side in the quadrangle determination process of FIG. 11.

[Explanation of symbols]

 2 ... tableware 3 ... imaging camera 4 ... tableware recognition device 7 ... figure recognition unit

Claims (2)

[Claims] 1. A method of recognizing an article by imaging the article with an imaging camera and extracting the contour of the imaged article, wherein an image created by expanding the extracted contour and an image including color information of the article are provided. A method for recognizing an article, characterized in that the color of the edge of the article is identified by creating a logical AND image of the above and the statistics of this logical image are taken, and the article is recognized by this. 2. A method of recognizing each article by imaging a plurality of articles with an imaging camera and extracting the contours of each of the captured articles, wherein the extracted contours are divided into partial contour sides, and each of these partial contour sides And the image containing the color information of the article is created, and the color of the edge of each article is identified by taking the statistics of each of the AND images. A method of recognizing an article, which comprises recognizing an article.