JP2001216571A - Method and device for adjusting charge - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and device for adjusting a charge capable of automatically adjusting the charge by recognizing merchandise itself. SOLUTION: Based on images picked up by image pickup parts 11-1 and 11-2, a height recognition part 13a recognizes the height of the merchandise, a shape recognition part 13b recognizes the shape of the merchandise, a color recognition part 13c recognizes the color of the merchandise, and an identification part 14 recognizes the merchandise directly.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for paying a fee, and more particularly, to a method and apparatus for automatically paying a fee in a cafeteria-style restaurant or the like.

[0002]

2. Description of the Related Art In a cafeteria-type restaurant, customers (purchasers) freely select products to purchase and place them on trays and the like, and a clerk checks the products at a checkout station to settle charges. In such a cafeteria-type restaurant, automation of fee settlement is delayed due to its characteristics as compared with a restaurant with a fixed menu.

[0003] As a method of automating the bill settlement in the cafeteria system, for example, it is conceivable to apply an "automatic billing apparatus for food and drink" described in JP-A-10-69579. This "food and beverage automatic payment device"
The tableware placed on the tray is irradiated with slit light and scanned, and three-dimensional information of the tableware is obtained from an image obtained by imaging the tableware with a television camera, based on the obtained three-dimensional information of the tableware. To identify the fee.

[0004]

However, since the above-mentioned "food and beverage automatic payment apparatus" does not identify the product itself, it is necessary to put the product on the tableware according to the price, and in particular, the tableware is required. It is difficult to use it in fast food restaurants and the like that do not.

Accordingly, an object of the present invention is to provide a fee settlement method and a device capable of recognizing a product itself and performing automatic payment of the fee.

[0006]

According to a first aspect of the present invention, there is provided a charge settlement method for identifying a product placed on a tray and performing a payment for the identified product. First and second image capturing means arranged at intervals of 1 to obtain images of the products placed on the tray, and the first image and the first image obtained by the first image capturing means. The second image acquired by the second imaging means
The height information of the product is acquired based on the image of the first image, the shape information and the color information of the product are calculated based on at least one of the first image and the second image, and the calculated height is calculated. The method is characterized in that a product is identified based on the information, the shape information, and the color information, and the fee is settled.

According to a second aspect of the present invention, in the first aspect of the present invention, the identification is performed based on the height information at a predetermined height from at least one of the first image and the second image. A plurality of distance images indicating only the products are created, and shape information and color information are calculated and performed for each of the created plurality of distance images.

According to a third aspect of the present invention, in the first or second aspect, the shape information includes information indicating a size of the product.

According to a fourth aspect of the present invention, there is provided a charge settlement device for identifying a commodity placed on a tray and performing settlement of the identified commodity, wherein the first image pickup means arranged at a predetermined interval. And second imaging means, and height information of the product is obtained based on the first image obtained by the first imaging means and the second image obtained by the second imaging means. Height recognition means; shape recognition means for acquiring shape information of the product based on at least one of the first image and the second image; and at least one of the first image and the second image. Color recognition means for obtaining the color information of the product based on one of the height information, the height information obtained by the height recognition means, the shape information obtained by the shape recognition means, based on the color information obtained by the color recognition means Product identification means for identifying the product Characterized in that it.

According to a fifth aspect of the present invention, in the fourth aspect of the present invention, the height recognizing means determines at least one of the first image and the second image based on the acquired height information. A plurality of distance images indicating only products of a predetermined height are generated, and the shape recognizing unit and the color recognizing unit are configured so that, for each of the plurality of distance images generated by the height recognizing unit, shape information and color information are respectively provided. Is obtained.

According to a sixth aspect of the present invention, in the fourth or fifth aspect, the shape recognizing means includes information indicating the size of the product in the acquired shape information.

[0012]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, an embodiment of a fee settlement method and apparatus according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram showing an appearance of a fee settlement apparatus according to the present invention. The charge settlement device 10 includes a tray 20
The merchandise is identified based on image data obtained by imaging the merchandise placed thereon, and the fee is settled.

FIG. 2 is a block diagram showing the structure of the fee settlement apparatus 10. As shown in FIG. As shown in FIG.
Are two imaging units 11-1 and 11-2 and a storage unit 1
2. Recognition unit 13, identification unit 14, settlement unit 15, display unit 1.
6, an input unit 17, and a data transfer unit 18. The recognition unit 13 includes a height recognition unit 13a, a shape recognition unit 13b, and a color recognition unit 13c.

The imaging units 11-1 and 11-2 capture the images of the tray 20 and the products placed on the tray 20, respectively. The storage unit 12 stores image data captured by the imaging units 11-1 and 11-2, and the recognition unit 13 recognizes an image of each product based on the image data. In the recognition unit 13, the height recognition unit 13a recognizes the height of each product from the two pieces of image data, the shape recognition unit 13b recognizes the shape of each product from any of the two pieces of image data, and the color recognition unit 13c Recognizes the color of each product from one of the two image data.

The identification unit 14 identifies the product based on the height, shape, and color information of each product obtained by the recognition unit 13, and the settlement unit 15 adjusts the fee based on the identification result. Do. At this time, if there is a product that cannot be identified by the identification unit 14, the fact is displayed on the display unit 16, and a clerk receiving the identification identifies the product, and the input unit 17.
The product name (type) is further input, and the settlement unit 15 performs settlement. The charge data settled by the settlement unit 15 is transferred from the data transfer unit 18 to a POS system or the like.

Here, a method of acquiring each information in the recognition unit 13 will be described. In the recognizing unit 13, first, the height recognizing unit 13a acquires the height information of the product. The acquisition of the height information is performed by the cameras 101-1 and 101-2 as shown in FIG.
(Corresponding to the imaging means 11-1 and 11-2, 102-1, 1
If the distance between the light sources 02-2 is A, the focal length is F, and the parallax described later is D, the distance Z from the camera, that is, the height information of the product can be obtained from Expression 1.

[0018]

(Equation 1)

Here, the parallax D will be described. Parallax D
Is obtained from two images captured by the cameras 101-1 and 101-2. For example, FIGS. 4A and 4B
, The tray 20 (image) is extracted in the X direction with reference to the edge of the tray 20. The distance from the right end of the extracted tray 20 to the image is ΔX1,
Let it be ΔX2.

Here, utilizing the fact that the extracted tray 20 is equidistant from the cameras 101-1 and 101-2, a density change in the X direction (one-dimensional information, indicated by an arrow in FIG. 4A) FIG. 4 (c) shows the density change on the broken line.
FIG. 4D shows the density change on the line indicated by the middle arrow.
Shown in ) Are associated with each other by DP matching (dynamic programming), and the disparity is calculated. However, when performing DP matching, both ends are fixed (the length of the tray 20 is constant),
The condition is that the path width limit is M (the maximum width that does not deviate any more, M <| ΔX1−ΔX2 |), and there is no backtracking (the order of the mapping does not reverse).

Thus, the known ΔX1, ΔX
When 2 is used, the parallax Dx of each point is as shown in Expression 2.

[0022]

(Equation 2) However, dx is the parallax obtained by associating a certain line in the left and right images (images by the cameras 101-1 and 101-2) cut out based on the end of the tray 20. For the distance calculation (cost function) of DP, color information (R, G, B) or density information ((R
+ G + B) / 3) is used, and is defined as in Expression 3 to associate each pixel. FIG. 5 shows the correspondence. Note that the second term on the right side of Equation 3 represents the normalized correlation.

As a condition for associating two left and right images, when xyz coordinates are taken in a three-dimensional space as shown in FIG. 6, the image plane of each image is , X
-Be on the same plane parallel to the y-plane, and make the scanning line direction parallel to the x-axis.

[0024]

(Equation 3) As described above, by obtaining the parallax D, the camera 101-
1, the distance from 101-2 is calculated, and a depth image in the depth direction is obtained. That is, the cameras 101-1 and 101-1 are provided for each pixel.
It is possible to obtain a distance image having the distance information from 1-2, and obtain the height information of the product from this distance image.

Since the accuracy of the height information is limited by the size of the image obtained by the cameras 101-1 and 101-2, there is a limit to the accuracy due to the shift amount of one pixel. In order to improve this accuracy, as shown in FIG. 7, adjacent pixels in the X direction, the average value N of n and n + 1
Is obtained, and the distance is corrected to the required accuracy by performing the correction using Expression 4.

[0026]

(Equation 4)

Next, the recognizing unit 13 processes the image (one of the left and right images) based on the distance image generated by the height recognizing unit 13a. The processing here is to extract a set of points (pixels) according to the distance from each image, and thereby to extract a region of a product at a height within a predetermined range.

For example, as shown in FIG. 8A, a cup 21, a coffee fresh 22, a hamburger 23, a double burger 24, an apple pie 25,
From the image in which each product of chicken 26 and potato 27 exists,
When a product having a height of 10 cm or more is extracted, an image having only the cup 21 and the double burger 24 can be obtained as shown in FIG. Similarly, when an image of a product having a height of 5 cm or more and less than 10 cm is extracted, FIG.
As shown in the figure, an image in which only the hamburger 23, the chicken 26, and the potato 27 are present can be obtained, and the height is 5c.
m (exactly 2 cm or more and less than 5 cm to facilitate discrimination from the tray 20 and exclude papers such as napkins), as shown in FIG. An image in which only the fresh 22 and the apple pie 25 exist can be obtained.

After obtaining the images classified for each height of the product in this way, the recognition unit 13 recognizes the shape of the product by the shape recognition unit 13b and recognizes the color of the product by the color recognition unit 13c. .

The shape recognizing unit 13b recognizes the shape and size of the product from the images classified according to the height of the product.
Recognized shapes are rectangles, squares, rectangles, circles (including roundness), and the like. When the target product is fast food, predetermined packaging is applied, and the characteristics of the packaging ( Boxes can be classified into rectangles, cups can be circular, etc.).

The color recognition unit 13c converts the color space of the image of each product from RGB (R: red, G: green, B: blue) to HLS.
(H: hue, L: lightness, S: saturation) and recognize the color of the product. Equation 5 shows a conversion equation from RGB to HLS.

[0032]

(Equation 5) Color recognition is basically performed by checking the distribution of color components using H and S. For example, when the specific color I of the wrapping paper of a certain product A is α1 <I <α2 on the H plane and β1 <I <β2 on the S plane, the number of pixels is a certain number or more. The product can be recognized as A.

If the color determination is difficult only with the blue system H and S, the lightness L is used for the determination as necessary.

In this way, the identification unit 14 identifies a product based on the height, shape, and color of the product recognized by the recognition unit 13, and this identification refers to a table as shown in FIG. Do it. At this time, non-billing items such as the coffee fresh 22 are excluded from the target of payment. Note that the goods and their prices shown in FIG.
5 or include the price in the table of FIG.

Even if the products are placed on the tray 20 in contact with each other, if the heights of the products are different, the products are separated as separate products by the height recognizing unit 13a.
If the colors are different, they are separated as separate products by the color recognition unit 13c. Further, even if the height and the color are the same, since the size of the product is recognized by the shape recognition unit 13b, it can be recognized as a separate product. However, this does not apply when the products are stacked vertically.

Next, the flow of the fee settlement using the fee settlement device 10 will be described. FIG. 10 is a diagram showing a flow of the fee settlement.

First, the customer places an arbitrary product on the tray 20, and sets the tray 20 at a predetermined position of the fee settlement device 10 (step 301). Then, the fee settlement apparatus 10 recognizes the product from the image obtained by imaging the product (step 302), and if there is a product that cannot be recognized, displays an inquiry for an unknown object (step 30).
3). In response, the clerk visually recognizes the product and performs an operation of selecting the type of the unknown product (step 304). In addition,
If the charge settlement apparatus 10 has recognized all the commodities, the processing of steps 303 and 304 is omitted.

When the charge settlement apparatus 10 recognizes all the commodities, or when an operation for selecting an unknown commodity is performed by the clerk, the charge settlement apparatus 10 displays the amount of money (step 305) and confirms it with the clerk. Is pressed (step 306), the settlement data is input to the POS system or the like (step 307).
The settlement process ends.

Next, the operation flow of the fee settlement apparatus 10 will be described. FIG. 11 is a flowchart showing the flow of the operation of the fee settlement apparatus 10.

When the tray 20 is set at a predetermined position, the fee settlement device 10 starts operating, and the image pickup unit 11-1.
11-2 captures an image and obtains an image of the tray 20 and the placed product (step 401). Subsequently, the acquired images (two images) are stored in the storage unit 12, and the recognizing unit 13 generates a distance image from the two images stored in the storage unit 12 (Step 402). The image area of the product is extracted (step 403). When the recognizing unit 13 extracts the area, the recognizing unit 14 identifies the product (Step 404).

Next, the settlement unit 15 adjusts the fee based on the identification result of the identification unit 14. If there is an unknown product in the identification result (YE at step 405).
S), the settlement section 15 displays on the display section 16 a prompt for input of data on the unknown product (step 406), and waits for data input from the input section 17 by the clerk (step 4).
07 and NO).

Thereafter, if data is input by the clerk (YES in step 407), or
If there is no unknown product in the identification result (NO in step 405), the settlement unit 15 displays the prices of all the products on the display unit 16 and the sales clerk indicates “O” as the meaning of confirmation.
Wait for the input of the "K key" (NO in step 409).

Subsequently, when the "clerk" inputs the "OK key" to the input unit 17 (YES in step 409), the settlement unit 15 transfers the settlement data from the data transfer unit 18 to a POS system or the like (step 410). ), And terminate the process.

In the above description, the product to be settled is described as fast food. However, the target product is not limited to fast food, but may be applied to the settlement of products sold in a self-picking system such as a cafeteria style. Can be.

[0045]

As described above, according to the present invention, the product is directly identified based on the height, shape, color, etc. of the product, and the settlement is performed. This makes it possible to automate the settlement without preparing such items and the like, and the product can be sold efficiently.

[Brief description of the drawings]

FIG. 1 is a diagram showing an appearance of a fee settlement device according to the present invention.

FIG. 2 is a block diagram showing a configuration of the fee settlement device 10.

FIG. 3 is a diagram (1) for explaining a method for acquiring height information;
It is.

FIG. 4 is a diagram (2) for explaining a method for acquiring height information;
It is.

FIG. 5 is a diagram showing association of cost functions.

FIG. 6 is a diagram for explaining conditions for associating two left and right images.

FIG. 7 is a diagram for explaining a method for improving distance accuracy.

FIG. 8 is a diagram showing a distance image.

FIG. 9 is a diagram showing a product identification table.

FIG. 10 is a diagram showing a flow of fee settlement.

FIG. 11 is a flowchart showing a flow of the operation of the fee settlement apparatus 10.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 Charge settlement apparatus 11-1, 11-2 Imaging part 12 Storage part 13 Recognition part 13a Height recognition part 13b Shape recognition part 13c Color recognition part 14 Identification part 15 Settlement part 16 Display part 17 Input part 18 Data transfer part 20 Tray 21 cup 22 coffee fresh 23 hamburger 24 double burger 25 apple pie 26 chicken 27 potato 101-1, 101-2 camera 102-1, 102-2 light source

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 3E042 AA04 BA17 CC10 5B057 AA02 BA19 CA01 CA12 CA16 CB01 CB12 CB16 CE08 CE09 DB02 DB06 DC09 5L096 AA02 CA02 EA27 EA35 FA66 FA69

Claims (6)

[Claims] 1. A charge settlement method for identifying a product placed on a tray and performing payment for the identified product, comprising: a first imaging unit and a second imaging unit disposed at predetermined intervals; Obtaining an image of a product placed on the tray; and obtaining the product based on a first image obtained by the first imaging unit and a second image obtained by the second imaging unit. And height information, shape information, and color information of the product are calculated based on at least one of the first image and the second image. A charge settlement method comprising: identifying a product based on a contract and adjusting a charge. 2. The method according to claim 1, wherein, based on the height information, a plurality of distance images indicating only products of a predetermined height are created from at least one of the first image and the second image. 2. The method according to claim 1, wherein shape information and color information are calculated for each of the plurality of created distance images. 3. The method according to claim 1, wherein the shape information includes information indicating a size of the product. 4. A charge settlement device for identifying a product placed on a tray and performing payment for the identified product, comprising: a first imaging unit and a second imaging unit disposed at predetermined intervals; A height recognition unit that obtains height information of the product based on a first image obtained by the first imaging unit and a second image obtained by the second imaging unit; Shape recognition means for obtaining shape information of the product based on at least one of a first image and the second image; and a color of the product based on at least one of the first image and the second image Color recognition means for obtaining information; height information obtained by the height recognition means; shape information obtained by the shape recognition means; and product identification means for identifying a product based on the color information obtained by the color recognition means. Characterized by having Gold settlement device. 5. The height recognizing means, based on the acquired height information, generates a plurality of distance images indicating only products of a predetermined height from at least one of the first image and the second image. 5. The charge according to claim 4, wherein the shape recognizing unit and the color recognizing unit obtain shape information and color information for each of the plurality of distance images generated by the height recognizing unit. 6. Checkout device. 6. The fee settlement apparatus according to claim 4, wherein the shape recognizing means includes information indicating the size of the product in the acquired shape information.