JP2009176209A - Automatic adjustment device for restaurant - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an automatic adjustment device for a restaurant, capable of reliably recognizing various types of tableware and precisely adjusting price regardless of orientation of the tableware. <P>SOLUTION: The tableware is recognized using three-dimensional information by stereo measurement using three cameras 2L, 2R, 2V arranged on each vertex of a triangle. The most reliable information is chosen among the plurality of pieces of three-dimensional information acquired by combination of three cameras. As a result, the stable tableware three-dimensional information can be measured, and precise price adjustment is attained. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an automatic settlement apparatus for a cafeteria, and in particular, in a settlement apparatus installed in a cafeteria-style cafeteria and automatically settles a fee according to the type and number of food and drink selected by a user. It relates to improvement means.

  2. Description of the Related Art Conventionally, several automatic payment apparatuses that can be applied to a cafeteria-style dining room where a user freely selects a desired food from a variety of foods provided by the cafeteria have been devised. This type of automatic payment apparatus is configured to automatically set the price of food and drink selected by the user by setting a price in advance for the tableware to be used and determining the tableware.

  For example, the automatic settlement apparatus disclosed in Patent Document 1 captures tableware placed on a normal tray with a single TV camera from the top, discriminates the tableware based on the planar shape of the tableware, and settles the fee. I do. However, according to such a configuration, since there are severe restrictions on the types of tableware that can be used, it is difficult to provide a wide variety of food and drink on a wide variety of tableware.

In order to improve this, the automatic settlement apparatus disclosed in Patent Document 2 acquires product height information by using two television cameras, and takes into account product height information, shape information, and color information. To calculate the charge.
JP-A-4-304597 JP 2001-216571 A

  However, since the technology disclosed in Patent Document 2 uses only two TV cameras, there is a case where the height information of the product cannot be obtained accurately, and thus there is a problem that the product cannot be settled accurately. .

  This point will be specifically described with reference to FIG. FIG. 2 shows a case where the square tableware 8b placed on the table 7 is recognized by the two cameras 2L and 2R. A height recognition means (not shown) obtains height information by calculation using a parallax corresponding to the center distance between the two cameras 2L and 2R. The parallax intersects an epipolar line that is a projection of a straight line connecting the camera 2L and the point of interest on the tableware 8b onto the camera 2R and a projection of a straight line connecting the camera 2R and the point of interest on the tableware 8b onto the camera 2L. Required by direction. However, as shown in FIG. 2, when the side of the square tableware 8b is parallel to the epipolar line, the corresponding point cannot be specified, so that the height information on the side cannot be obtained. As described above, according to the technique disclosed in Patent Document 2, the recognition accuracy may be extremely deteriorated depending on the direction in which the tableware is placed, and the product (food or drink) cannot be settled accurately.

  The present invention has been made to solve such deficiencies in the prior art, and the object thereof is to reliably recognize various types of tableware regardless of the direction in which the tableware is placed. An object of the present invention is to provide an automatic checkout apparatus for a cafeteria that can perform the checkout of a restaurant with high accuracy.

  In order to solve the above-described problems, the present invention firstly includes a table on which tableware selected by the user is placed, an imaging unit for imaging the tableware placed on the table, and image data captured from the imaging unit. Image data storage means for storing the feature data, teaching data storage means in which feature quantities of a plurality of types of tableware are stored in advance, image data stored in the image data storage means, and the tableware stored in the teaching data storage means Recognition means for recognizing the type of tableware imaged by the image pickup means, price storage means for pre-registered prices of food and drink set for each type of tableware, and the recognition Based on the type and number of tableware recognized by the means and the price registered in the price storage means, the total amount of food and drink provided and served on the tableware placed on the table is calculated. In an automatic settlement apparatus comprising a calculation unit, three or more imaging units are installed opposite to the table, the three or more imaging units are arranged at each vertex position of a triangle, and the recognition unit is The height of the tableware from among a plurality of three-dimensional information obtained from image data captured by one of the three or more imaging units and image data captured by the other imaging unit. One or more types of three-dimensional information including information are selected and the type of the tableware is recognized.

  For example, when three imaging means (cameras) are arranged as described above, as shown in FIG. 3, three epipolar lines that are not parallel to each other, that is, LR epipolar line and RV epipolar line, for one point on the tableware 8b. , VL epipolar line exists. The LR epipolar line is a projection onto the camera 2R that is a straight line connecting the camera 2L and the attention point on the tableware 8b (projection onto the camera 2L is a straight line connecting the camera 2R and the attention point on the tableware 8b). The RV epipolar line is a projection of a straight line connecting the camera 2R and the point of interest on the tableware 8b onto the camera 2V (projection of a straight line connecting the camera 2V and the point of interest on the tableware 8b onto the camera 2R). The line is a projection of a straight line connecting the camera 2V and the attention point on the tableware 8b onto the camera 2L (projection of a straight line connecting the camera 2L and the attention point on the tableware 8b onto the camera 2V). For this reason, even if one side of the tableware 8b is parallel to one epipolar line (LR epipolar line in FIG. 3), the other two epipolar lines are parallel to one side of the tableware 8b. Instead, it is possible to obtain height information by parallax.

  This will be further described with reference to FIG. In FIG. 4, in order to make the explanation easier to understand, the camera 2L is arranged at a right angle portion of a right triangle, the camera 2R is arranged in a direction parallel to the long side 8b-1 of the tableware 8b, and the camera is arranged at the other vertex. The captured image of each camera when 2V is arranged is shown. The camera R images 2R-1 and 2R-2, and the camera V images 2V-1 and 2V-2 are the same images shown in different places. As is clear from FIGS. 4A and 4B, since the long side 8b-1 of the tableware 8b is parallel to the LR epipolar line, no parallax can be obtained from the camera L image and the camera R image. . On the other hand, as apparent from FIGS. 4A and 4C, if the camera V image 2V-1 and the camera image 2L-1 are used, there is a VL parallax, so the long side 8b-1 It is possible to calculate the height information. Similarly, as is clear from FIG. 4A and FIG. 4E, the short side 8b-2 of the tableware 8b is parallel to the VL epipolar line, so that there is no parallax from the camera L image and the camera V image. I can't get it. On the other hand, as apparent from FIGS. 4A and 4D, if the camera R image 2R-2 and the camera image 2L-1 are used, there is an LR parallax, so the long side 8b-1 It is possible to calculate the height information.

  Therefore, even if one side of the tableware 8b is parallel to one epipolar line, the height information of the side can be obtained using another camera image, so that the recognition accuracy of the tableware 8b is increased. It is possible to pay for food and drink accurately.

  Secondly, in the first automatic settlement apparatus, the first recognition unit selects the three-dimensional information having the highest epipolar certainty from the plurality of three-dimensional information. The tableware type is recognized based on the selected three-dimensional information and the feature amount of the tableware stored in the teaching data storage means.

  The epipolar confidence is the XY plane of each line segment obtained by performing edge extraction, segmentation, and linear approximation of edges in each segment of two camera images captured by two cameras arranged in the XY direction. The angle between the projection and the epipolar line is 90 degrees, and the larger the index, the higher the reliability of the three-dimensional position in the three-dimensional information obtained from the two camera images. Therefore, the three-dimensional information having the highest epipolar certainty of the line segment is selected from the three or more types of three-dimensional information obtained from the combination of the camera images captured by three or more cameras. Recognizing the type of tableware based on the three-dimensional information makes it possible to recognize the other types of tableware with the highest accuracy, so that the payment for the food and drink can be performed with higher accuracy.

  Thirdly, according to the present invention, in the first automatic settlement apparatus, the three imaging units are arranged at each vertex position of an equilateral triangle in a plane parallel to the table, and two of the three imaging units are arranged. Is arranged in parallel with one side of the table.

  According to such a configuration, since various types of tableware can be recognized with high accuracy by the three imaging means, the cost increase due to the increase in the imaging means can be minimized. In addition, since the three image pickup means are equally arranged in a plane parallel to the table, it is easy to associate points on the image data picked up by each image pickup means, and the program configuration of the recognition means is simplified. The settlement process can be speeded up.

  According to the present invention, three or more imaging units are installed opposite to the table, and these three or more imaging units are arranged at the respective vertex positions of the triangle, and the recognition unit is one of the three or more imaging units. One or more types of three-dimensional information including table height information from among a plurality of three-dimensional information obtained from image data captured by one imaging means and image data captured by another one imaging means Is selected and the type of tableware is recognized, so that the tableware on the table can be accurately recognized regardless of the shape and posture of the tableware placed on the table, and high-accuracy fee settlement is possible.

  Hereinafter, an embodiment of an automatic settlement apparatus according to the present invention will be described with reference to the drawings.

  FIG. 1 is an overall configuration diagram of the present invention. In this figure, reference numeral 1 denotes a control device that recognizes tableware 8a, 8b, 8c arranged on the tray 6 and placed on the table 7 and settles the fee. 2L, 2R, and 2V are television cameras, and image the dishes 8a, 8b, and 8c. In the present embodiment, these three television cameras 2L, 2R, 2V are arranged at the vertex positions of equilateral triangles in a plane parallel to the table 7. An illumination device 3 irradiates the tableware 8a, 8b, 8c, which is a subject, with light for photographing. Reference numeral 4 denotes a non-contact card reader, which exchanges necessary settlement information including ID information with an ID card possessed by a restaurant user. A touch panel 5 displays the recognition results of the dishes 8a, 8b, and 8c and the total price, and the dining room user operates the touch switch set on the screen to make necessary inputs.

  When a cafeteria user uses this automatic checkout device, the cafeteria user arranges dishes 8a, 8b, and 8c on which desired food and drink are arranged from various menus provided by the cafeteria on the tray 6, and this tray. After placing 6 on the table 7, the ID card owned by itself is held over the non-contact card reader 4. When the ID card is detected by the contactless card reader 4, tableware recognition and fee settlement are performed, and the result is displayed on the touch panel 5. The cafeteria user confirms the payment details and, if the result is correct, operates a key such as “END” displayed on the touch panel 5. This completes the settlement.

  Next, the internal configuration of the control device 1 will be described. FIG. 5 is a hardware configuration diagram of the control device 1. Reference numeral 10 denotes a bus, which connects each device described below. Reference numeral 11 denotes a CPU which performs all calculations in this apparatus. A ROM 12 stores an initialization program at power-on. Reference numeral 13 denotes a RAM, which stores a program executed by the apparatus and an intermediate result of the calculation, image data taken from three television cameras, and the like. Reference numeral 14 denotes a serial I / O, through which the CPU 11 exchanges information between the card reader 4 and the touch panel 5. Reference numeral 15 denotes a camera interface. When the imaging command from the CPU 11 is issued, the camera interface 15 performs imaging control of the television cameras 2L, 2R, and 2V, and transfers the captured image data to the RAM 13 via the bus 10. Reference numeral 16 denotes an illumination interface that turns on and off the illumination device 3. Reference numeral 17 denotes a DISK interface, which is an interface with the hard disk 18. The hard disk 18 stores a program executed by the apparatus, tableware teaching data, a tableware amount table, and the like. A program to be executed by this apparatus is transferred to the RAM 13 by a program stored in the ROM 12 when the power is turned on, and is executed by the CPU 11. Reference numeral 19 denotes a LAN interface, which exchanges data with a host computer. In this embodiment, the settled information is transferred to the host computer, and the actual fee is settled by payroll deduction or credit deduction.

  FIG. 6 is an overall process flow of a program executed by the CPU 11. When the power is turned on, initialization is performed in step 101. In initialization, the program, teaching data, amount table, and the like stored in the hard disk 18 are transferred to the RAM 14. It also initializes each I / O and LAN interface. Next, in step 102, the input is checked. If there is an end key input from the touch panel 5, the process ends. On the other hand, if there is an ID card reading input from the card reader 4, the process proceeds to step 103. In step 103, the tableware is recognized. Details of the tableware recognition process will be described later. The recognized tableware number is returned from the tableware recognition. In step 104, the fee for the tableware recognized in step 103 is calculated. In the fee calculation, a table of registered tableware numbers, tableware names, and amounts is used. Calculate the total price using the recognized tableware number. In step 105, the amount is displayed on the touch panel 5. The displayed contents are the name of the tableware recognized as the total amount and the respective amounts. When the process ends, the process returns to step 102. At this time, the ID card number and the total amount are written to the hard disk 18 as settlement information. The settlement information written in the hard disk 18 is transferred to the host computer at the request of the host computer.

  Hereinafter, the processing flow of the tableware recognition processing in step 103 will be described with reference to FIG. First, in step 201, the tableware 8a, 8b, 8c placed on the table 7 is imaged using the three cameras 2L, 2R, 2V. The captured image information is stored in the RAM 13. Next, in step 202, three-dimensional measurement of the tableware 8a, 8b, 8c is performed based on the image data from the three cameras 2L, 2R, 2V. The processed result is output as a set of line segments approximating a three-dimensional shape. Each line segment has information on the three-dimensional position regarding the start point and the end point. Details of the processing contents will be described later. In step 203, the obtained line segments are grouped. In the grouping, the ones with the close start point or end point of each line segment are made the same group. By grouping, the edges of each tableware become one group. In step 204, feature quantities are calculated for each grouped group. One group corresponds to one tableware. In the present embodiment, the average height, area, and perimeter of the group are calculated as feature amounts. At this time, a group having a height of 10 mm or less is determined to be a shadow of chopsticks or tableware, and the group itself is deleted.

  Next, in step 205, the feature quantity of the previously obtained three-dimensional information is compared with the teaching data stored in the hard disk 18. The teaching data is registered in advance as a table. In the teaching data table, the tableware number, tableware name, average height of the edge, area, and perimeter are registered. The calculated feature quantity is compared with each value in this table, and a search is made for a difference that is less than a certain threshold value. If there is a match, the table number is registered as a recognition result in step 206. If there is no match, go to step 207. In step 207, it is checked whether the processing has been completed for all groups. If not, the process returns to step 205. If completed, the process is terminated. In this way, recognition of all tableware 8a, 8b, 8c on the tray is performed.

  FIG. 8 is a three-dimensional measurement process flow showing details of step 202. In this processing flow, first, in step 301, measurement processing is performed using the image data of the left camera 2L and the image data of the right camera 2R. Next, in step 302, measurement processing is performed using the image data of the right camera 2R and the image data of the vertical camera 2V. Further, in step 303, measurement processing is performed using the image data of the vertical camera 2V and the image data of the left camera 2L. In step 304, the optimal three-dimensional information obtained in steps 301 to 303 is selected from the three-dimensional information. In this example, the most suitable epipolar certainty described below is selected.

  FIG. 9 is a processing flow of measurement processing showing details of step 301, step 302, and step 303. However, in this processing flow, the case where the left camera image and the right camera image in step 301 are input will be described for the sake of convenience.

First, in step 401, the two images are differentiated, and an edge in which the difference in shading in the image is significant is extracted. In step 402, segmentation is performed using the connected state in the edge image of the left camera image. Next, in step 403, the edge is linearly approximated within the segment connected to the left camera. The start point of a line segment approximated by a straight line is set to (XLS, YLS), and the end point is set to (XLE, YLE). In step 404, corresponding points in the right camera image are searched for the start point and end point of all line segments. An edge is searched for along the LR epipolar line in the image, and a found point is set as a corresponding point. The corresponding points for the start point (XRS, YRS) and the corresponding points for the end point are (XRE, YRE). In step 405, a three-dimensional calculation is performed for each corresponding point. The three-dimensional coordinates of the corresponding point are set as (x, y, z) and calculated by the following formula.

In the above formula, a is the distance between the cameras. In addition, the Y axis of each camera is assumed to be parallel to the epipolar line. If they are not parallel, coordinate transformation is performed in advance using a rotation matrix. Next, at 406, an epipolar certainty factor K is obtained. The epipolar confidence K is defined by the angle formed by the projection of each line segment on the XY plane and the epipolar line, and is determined by the following equation so as to be maximum at 90 degrees.

  In the above equation, θ is an angle formed by the projection of the line segment onto the XY plane and the epipolar line. This K is obtained for each line segment. Three line segments of the same part can be obtained by combining two cameras each. Of these, the one with the largest K has the highest reliability, and is adopted as the final three-dimensional position. If no corresponding point is found and three are obtained, the most reliable one is adopted. This makes it possible to always detect a reliable three-dimensional position regardless of the direction of the edge of the tableware.

  Since the automatic settlement apparatus for a cafeteria according to the present embodiment has the above-described configuration, the following effects can be obtained.

  1. Since the tableware 8a, 8b, 8c to be settled is recognized based on highly reliable three-dimensional information, there are few restrictions on the tableware to be used, and a variety of menus can be provided. In this embodiment, the height discrimination performance is adjusted to about 5 mm, and the height of the tableware is usually about 20 mm to 100 mm. Therefore, the number of menus is about 18 times that of the case where only the planar shape is used. Can be set.

  2. Since only the three-dimensional shape of the tableware is used as a clue, it is not necessary to add information such as an IC tag to the tableware, and the tableware used conventionally can be used as it is.

  3. Since the tableware is recognized by imaging from above the tableware, it is not necessary to use a special tray or the like.

  4). Since recognition is performed by paying attention to the outer shape of the tableware, there is no influence of the dishes accumulated in the tableware, and any checkout process can be performed before or after the meal.

  5). There is no need to handle cash because payment is done by card.

  6). Since only those having a height of 10 mm or more are detected using the height image, chopsticks, spoons, etc. can be excluded from the adjustment target without special processing or setting.

  7. Since the number of cameras is three, the high cost due to the increase in the number of cameras can be minimized. In addition, since the three cameras 2L, 2R, and 2V are equally arranged in a plane parallel to the table 7, it is easy to associate points on the image data captured by the cameras 2L, 2R, and 2V. The program configuration of the recognition means can be simplified and the settlement process can be speeded up.

  In the above-described embodiment, the payment is performed using the ID card. However, the payment may be performed using a prepaid card instead of the configuration.

  In the embodiment, the number of cameras is three. However, the gist of the present invention is not limited to this, and the number of cameras can be any number of three or more.

  Further, in the embodiment, the type of tableware is determined by selecting the one having the highest epipolar certainty from the plurality of three-dimensional information, but it is not always necessary to select the one having the highest epipolar certainty. It is sufficient to select at least one type of three-dimensional information including table height information and determine the type of tableware.

It is a whole block diagram of the automatic adjustment apparatus for dining rooms which concerns on embodiment. It is an optical system arrangement | positioning figure of the automatic adjustment apparatus for dining rooms which concerns on a prior art example. It is an optical system arrangement | positioning figure of the automatic adjustment apparatus for dining rooms which concerns on embodiment. It is a figure which shows the example of the captured image of FIG. It is a hardware block diagram of the automatic adjustment apparatus for dining rooms which concerns on embodiment. It is the whole processing flow of the automatic adjustment apparatus for restaurants which concerns on embodiment. It is the tableware recognition processing flow of the automatic adjustment apparatus for dining rooms which concerns on embodiment. It is a three-dimensional recognition process flow of the automatic adjustment apparatus for dining rooms which concerns on embodiment. It is a measurement recognition process flow of the automatic adjustment apparatus for dining rooms which concerns on embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Control apparatus 2L, 2R, 2V Television camera 3 Illumination apparatus 4 Non-contact card reader 5 Touch panel 6 Tray 8 Tableware

Claims (3)

Table for placing tableware selected by a user, imaging means for imaging tableware placed on the table, image data storage means for storing image data captured from the imaging means, and characteristics of a plurality of types of tableware Based on the teaching data storage means in which the amount is stored in advance, the image data stored in the image data storage means, and the feature amount of the tableware stored in the teaching data storage means, the image is taken by the imaging means Recognition means for recognizing the type of tableware, price storage means for pre-registered prices of food and drink set for each type of tableware, type and number of tableware recognized by the recognition means, and the price storage means An automatic checkout device comprising a calculation means for calculating the total amount of food and drink provided on the tableware placed on the table based on the price registered in the table,
Three or more of the imaging units are installed opposite to the table, and the three or more imaging units are arranged at each vertex position of the triangle, and the recognition unit is one of the three or more imaging units. One or more types of three-dimensional information including the height information of the tableware out of a plurality of pieces of three-dimensional information obtained from image data captured by one imaging unit and image data captured by another one imaging unit. An automatic settlement apparatus for a cafeteria characterized by selecting and recognizing the type of the tableware.   The recognizing unit selects three-dimensional information having the highest epipolar certainty of the line segment from the plurality of three-dimensional information, and the tableware stored in the selected three-dimensional information and the teaching data storage unit The automatic settlement apparatus for canteens according to claim 1, wherein the type of tableware is recognized based on the feature amount of the restaurant.   In the plane parallel to the table, the three image pickup means are arranged at each vertex position of an equilateral triangle, and two of the three image pickup means are arranged parallel to one side of the table. The automatic settlement apparatus for cafeterias according to claim 1.

Images