JP2016105228A - Commodity ordering apparatus, commodity ordering method, commodity price output device and commodity price output method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a commodity ordering apparatus capable of automatically ordering desired commodity to a kitchen or the like.SOLUTION: A commodity ordering apparatus 1 includes: a storage unit that stores commodities to be provided to a customer together with sub-articles and the sub-articles while associating them with each other, for each sub-article to be provided to the customer together with a commodity; and a CPU that, when an object picked up by an imaging apparatus 12 is recognized as a sub-article to be provided to a customer together with a commodity based on the image information of the object, outputs a piece of ordering information of the commodity stored in the storage means while being associated with the sub-article.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a product ordering device, a product ordering method, a product price output device, and a product price output method using object recognition.

  In the self-type udon shop, the customer takes desired chikuwa or tempura from the display stand using tongs and places them on a plate on the tray and receives payment at the POS terminal (register). Here, POS is an abbreviation for point of sale system. In such udon shops, cooking products such as chikuwa and tempura are often displayed on the display stand as they are. As a result, the freshly prepared food can be provided to the customer without sacrificing the taste and texture as it is, and the number of steps to put on the plate can be omitted. On the other hand, there is a problem that it is difficult to electronically manage the cooked product by not placing the cooked product on the plate.

There are many different types of cooked products, and even if they are the same type, the appearance is not always the same. Udon shop clerk needs to manually identify the type of food on the tray and manually input it to the POS terminal. Therefore, a predetermined training period for the store clerk is necessary.
For this reason, in recent years, POS terminals have appeared that automate product registration by photographing food on a tray with a camera and automatically recognizing the type and quantity of food on the tray from this image information. Thereby, the training period of the store clerk can be shortened.

  The problem of Patent Document 1 is described as “Accurately identify the type of pan by image recognition.” The configuration includes “A color image of pan is divided into a color image of an inner region and a color image of an outer region. From the pan image, the feature amount related to the outline of the pan, the feature amount related to the texture of the pan, the feature amount in the color space of the inner region of the pan, and the feature amount in the color space of the outer region of the pan. And identify the type of bread. "

JP 2011-170745 A

  The udon shop is configured so that customers can eat and drink in the store. In such a store, the customer uses the tongs to take the desired food from the food display stand to the tray and order a drink from the store clerk. At this time, the store clerk needs to order (order) the drink requested by the customer from the kitchen, the backyard or the like, create the drink, and register the sales.

  Drinks are liquid and do not have an inherent form by themselves. In addition, a common drink container is provided regardless of the type of product. Therefore, it is difficult to identify the type of product based on the appearance of the drink or its container. Further, when taking a picture with a camera, a drink container having a predetermined height cannot be focused together with a short cooked product, and there is a possibility that it cannot be taken clearly. Furthermore, even when a drink is recognized as an object and registered for sales, it is not possible to save the trouble of ordering this drink to the kitchen or backyard.

  Therefore, an object of the present invention is to provide a product ordering device, a product ordering method, a product price output device, and a product price output method that can reduce the labor involved in outputting product information.

In order to achieve the above object, the present invention
For each sub-provided provided to the customer together with the product, storage means for storing the product to be provided to the customer together with the sub-provided,
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Control means for outputting the order information of the product stored in
A product ordering device characterized by comprising:

To achieve the above objective,
For each sub-provided provided to the customer together with the product, the step of storing the product to be provided to the customer together with the sub-provided in advance in the storage means;
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Outputting the order information of the product stored in
A product ordering method characterized by comprising:

To achieve the above objective,
For each sub-provided provided to the customer together with the product, storage means for storing the product to be provided to the customer together with the sub-provided,
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Control means for outputting the price of the product stored in
A product price output device characterized by comprising:

To achieve the above objective,
For each sub-provided provided to the customer together with the product, the step of storing the product to be provided to the customer together with the sub-provided in advance in the storage means;
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Outputting the price information of the product stored in
A product price output method characterized by comprising:

  According to the present invention, it is possible to provide a product ordering device, a product ordering method, a product price output device, and a product price output method that can reduce the labor involved in outputting product information.

It is a perspective view which shows the goods ordering apparatus in this embodiment. It is a schematic block diagram which shows a goods ordering apparatus. It is a logical block diagram which shows a goods ordering apparatus. It is a figure which shows the example of a screen transition. It is an activity diagram which shows operation | movement of a goods ordering apparatus. It is a figure which shows an example of the data for object recognition. It is a figure which shows an example of merchandise content data. It is a flowchart which shows recognition processing, such as goods. It is a sequence diagram which shows registration processing of goods etc.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. In addition, in several drawing, the same code | symbol is provided to the same element and the overlapping description is abbreviate | omitted suitably.
FIG. 1 is a perspective view showing product ordering apparatuses 1-1 and 1-2 in the present embodiment.
As shown in FIG. 1, the product ordering device 1-1 includes a customer display 11, a photographing device 12, and a lighting device 13, and is installed in a store lane 15. The merchandise ordering apparatus 1-2 is configured in the same manner as the merchandise ordering apparatus 1-1, and is installed in the lane 15. Hereinafter, when the product ordering apparatuses 1-1 and 1-2 are not particularly distinguished, they are simply referred to as the product ordering apparatus 1. The store clerk stands on the back side of the drawing of the lane 15 when serving customers. The customer stands near the left end of the lane 15 on the near side of the drawing, places the tray 3 on the lane 15 and moves from left to right.
A plurality of types of coasters 5a are placed on the lane 15, and a product ordering device 1-1 is installed on the right side thereof. In the coaster 5a or in the vicinity thereof, coffee that is provided by the customer together with the coaster 5a and its type are described. The merchandise ordering apparatus 1-1 recognizes the coaster 5a placed on the tray 3 by the customer and orders a drink corresponding to the coaster 5a.
A straw 5b is placed on the right side of the product ordering apparatus 1-1, and a product ordering apparatus 1-2 is installed on the right side thereof. In the straw 5b or in the vicinity thereof, juice that is provided by the customer together with the straw 5b and its type are described. The merchandise ordering apparatus 1-2 recognizes the straw 5b placed on the tray 3 by the customer and orders a drink corresponding to the straw 5b. The merchandise ordering apparatus 1-2 recognizes the straw 5b, but does not recognize the coaster 5a, and does not output a coffee order corresponding thereto. Thereby, the downstream product ordering apparatus 1-2 operates so as not to order the same drink repeatedly.
The coffee corresponding to the coaster 5a of the present embodiment has a longer creation period than the juice corresponding to the straw 5b. Therefore, more preparation periods can be assigned to coffee having a relatively long preparation period. As a result, the merchandise registration system 7 (see FIG. 2) is less likely to wait for customers and can receive orders from more customers.
Upon receiving the drink order information, the merchandise ordering apparatus 1 prints the order using the kitchen printer 4 (see FIG. 2) installed in the kitchen. This operation will be described in detail with reference to FIGS.

  The customer display 11 is a liquid crystal display device, for example, and faces the front side of the drawing, which is the customer side. The customer display 11 displays information (product name, amount of money, etc.) related to the handled product to the customer.

The photographing device 12 photographs the tray 3 placed in the lane 15 and the handling product or / and the side offer on the tray 3 from directly above. Examples of supplementary products include coasters 5a, straws 5b, disposable chopsticks 5c (see FIG. 4), spoons, forks, knives, sugar bags, gum syrup containers, milk containers, napkins, etc. It is to provide. Hereinafter, handled products and sub-provided items may be referred to as “handled products”. The shooting area 121 is an area on the lane 15 that can be shot by the shooting apparatus 12.
The illumination device 13 is an LED (Light Emitting Diode) light source, for example, and is installed adjacent to the photographing device 12. The illuminating device 13 illuminates the tray 3 placed in the lane 15 and the handling goods or / and sub-provided items on the tray 3 from directly above. As a result, the photographing device 12 can photograph the handled product on the tray 3 more clearly. When the imaging device 12 takes an image, the coaster 5a on the tray 3 is irradiated with illumination light from the illumination device 13. The color of the tray 3 is preferably white or light. Furthermore, it is desirable that the tray 3 has a fine matte process on its upper surface. Thereby, reflection of illumination light and environmental light can be suppressed.
The customer places the coaster 5a, which is a sub-provided item, on the tray 3, and slides the tray 3 to the photographing area 121. In the example of FIG. 1, one coaster 5 a is placed on the tray 3. The store clerk operates the POS terminal 2 (see FIG. 2) on the right side of the product ordering device 1-2. The clerk pays the customer's order with the POS terminal 2 and receives a drink as a handling product from the kitchen and places it on the coaster 5a.

  The merchandise ordering apparatus 1 specifies the handled merchandise on the tray 3 by processing the image photographed by the photographing apparatus 12. The merchandise ordering apparatus 1 further determines the type of drink based on the side offer such as the coaster 5a, the straw 5b, the spoon, the chopsticks, the sugar bag, the gum syrup container, the milk container, etc. The order information is printed by the kitchen printer 4. Thereby, the goods ordering apparatus 1 can notify the order of drinks to the outside kitchen side. Further, the merchandise ordering apparatus 1 transmits information such as the merchandise name, quantity, and price of the drink to the external POS terminal 2, and registers the drink for sales. That is, the product ordering apparatus 1 functions as a product price output device that outputs price information of handled products.

FIG. 2 is a schematic configuration diagram showing the product ordering apparatuses 1-1 and 1-2 and the product registration system 7 including them.
In addition to the configuration shown in FIG. 1, the merchandise ordering apparatus 1-1 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, a storage unit 104, a communication Part 14. Each component is connected to be communicable with each other via an internal bus and input / output circuits (not shown). The merchandise ordering apparatus 1-1 is connected to the kitchen printer 4 and the POS terminal 2 via a network. Similarly, the merchandise ordering apparatus 1-2 is connected to the kitchen printer 4 and the POS terminal 2 via a network.

The CPU 101 is a central control device and is a control unit that controls the entire product ordering device 1.
The RAM 102 is temporary storage means used by the CPU 101, and temporarily stores image data and various variables related to a program executed by the CPU 101.
The ROM 103 is a non-volatile storage unit and stores a program executed by the CPU 101.

  The customer display 11 is a display unit that is controlled by the CPU 101 and displays information (product name, amount, etc.) regarding a photographed image of the handled product and the accounting of the handled product to the customer.

  The storage unit 104 is configured by, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like, and is a storage unit that stores various programs and various files. Various programs and various files stored in the storage unit 104 are all or partially copied to the RAM 102 and executed by the CPU 101 when the merchandise ordering apparatus 1 is activated. The storage unit 104 stores various data.

  The photographing device 12 includes a color CCD (Charge Coupled Device) image sensor, a color CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the like. The imaging device 12 is an imaging unit that performs imaging under the control of the CPU 101. The imaging device 12 captures a moving image of 30 fps (frame per second), for example. Frame images (captured images) sequentially captured by the imaging device 12 at a predetermined frame rate are stored in the RAM 102.

  Under the control of the CPU 101, the illuminating device 13 emits light downward from the tray 3 so that the photographic image becomes clearer when the photographic device 12 shoots the tray 3 or a handled product on the tray 3. Irradiate. Thereby, the precision of image processing can be improved. The timing of light irradiation by the illumination device 13 may be matched with the timing of shooting of the imaging device 12, or may be always, for example.

The communication unit 14 is a network interface controller, for example, and is connected to the kitchen printer 4 and the POS terminal 2. The CPU 101 causes the kitchen printer 4 to print the order content via the communication unit 14 and causes the POS terminal 2 to register the product.
The POS terminal 2 is a terminal for registering accounting. The POS terminal 2 receives identification information such as chikuwa and tempura, which are handled products, and beverage identification information from the product ordering devices 1-1 and 1-2, and displays the total price, sales management and sales results. Performs calculation and input / output such as management.

FIG. 3 is a logical block diagram showing the product ordering apparatus 1.
The CPU 101 (see FIG. 2) of the product ordering apparatus 1 embodies the processing unit 8 by executing a program (not shown) stored in the ROM 103 (see FIG. 2). The processing unit 8 includes an image storage unit 81, an object recognition processing unit 82, a confirmation notification unit 83, an order printing unit 84, and an information output unit 85. The object recognition processing unit 82 includes an object detection unit 821, a similarity calculation unit 822, and a similarity determination unit 823.
The processing unit 8 refers to the object recognition data 105, the product content data 106, and the sales master 107 stored in the storage unit 104. The processing unit 8 recognizes whether or not the object photographed by the photographing device 12 is a handling product based on the image information of the object. In addition, when the processing unit 8 recognizes that the object photographed by the photographing device 12 is a sub-provided item, the processing unit 8 displays the order information of the handling product stored in association with the recognized sub-provided item. The data is output to the kitchen printer 4 (see FIG. 2).
Note that the processing unit 8 refers to the object recognition data 105, the product content data 106, and the sales master 107 stored in the storage unit 104.

In the object recognition data 105, template information that combines feature amounts modeled for each type of cooked product or sub-provided item is registered in advance. That is, the object recognition data 105 is a data file in which an association between a product name or a product ID and a feature quantity of the handled product is set for each of the handled products displayed and sold in the store. It functions as a dictionary for recognizing products. In the object recognition data 105, for each sub-provided product provided to the customer together with the handled product, the handled product to be provided to the customer together with the sub-provided product or a combination thereof is stored in association with each other.
Here, in the object recognition data 105 stored in each merchandise ordering apparatus 1, information related to the cooked product and sub-provided items to be recognized by each apparatus is stored. As a result, since only the cooked goods and sub-provided items to be ordered by each product ordering apparatus 1 are ordered from the kitchen, the ordering of products does not overlap.

The product content data 106 is a data file in which content information of handled products is set. In the product content data 106, a product ID (IDentifier), a product name, a unit price, discount information, and the like are set as the content information of the handled product.
The sales master 107 is a data file for recording sales registration of handled products, and records a product ID sold to a customer, a corresponding product classification, a product name, a unit price, a sales quantity, and the like.
The image storage unit 81 sequentially captures and stores frame images (color digital images) captured and captured by the image capturing device 12.

  The object detection unit 821 uses a technique such as edge detection to separate a candidate image such as a handling product in the captured frame image from the background, that is, to cut out and detect only the object to be identified. Specifically, the processing unit 8 always images the imaging region 121 on the lane 15 with a predetermined period by the imaging device 12. The object detection unit 821 binarizes the captured frame image and extracts a contour line. Next, the object detection unit 821 compares the contour line extracted from the previous frame image with the contour line extracted from the current frame image, separates the image into regions, and detects an object.

The similarity calculation unit 822 identifies the type of each handled product based on the detected separated image of each object. The similarity calculation unit 822 calculates the state of the surface such as the size, shape, hue, surface unevenness, etc. as a feature amount for each separated image. The similarity calculation unit 822 further compares each feature value of each separated image with each feature value of the handling product or the like recorded in the object recognition data 105 to obtain each separated image and the object recognition data 105. The degree of similarity with each recorded product is calculated.
That is, the similarity refers to the degree of similarity between the separated images when the feature quantity assumed by the handling product recorded in the object recognition data 105 is assumed to be 100% similarity. Is shown. That is, the similarity indicates to what extent the separated image of the object is similar to the reference image assumed for the handled product. Note that, when there are a plurality of types of feature amounts, the similarity calculation unit 822 performs a comprehensive evaluation based on the plurality of feature amounts. At that time, the similarity calculation unit 822 may weight each feature amount.

Recognizing an object included in an image in this way is called generic object recognition. In "Current status and future of general object recognition" written by Keiji Yanai below, a survey of general object recognition research is added to methods, data sets and evaluation benchmarks are performed, and the future is prospected.
Keiji Yanai, “Current Status and Future of General Object Recognition”, [online], Journal of Information Processing Society of Japan, November 15, 2007, Vol. 48, No. SIG16, pp. 1-24, [October 31, 2014 Day search], Internet <URL: http: //mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

A technique for performing general object recognition by dividing an image into regions for each object (object) is described in the following document.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, [October 31, 2014 search], Internet <URL: http: // citeseerx .ist.psu.edu / viewdoc / download? doi = 10.1.1.145.3036 & rep = rep1 & type = pdf>

  It should be noted that the method for calculating the degree of similarity between the feature amount of the photographed product image and the feature amount of the product image of the handled product recorded in the object recognition data 105 is not particularly limited. For example, the similarity between the feature quantity of the photographed product image and the feature quantity of each handled product recorded in the object recognition data 105 may be calculated as an absolute evaluation or as a relative evaluation. Also good.

When the similarity is calculated as an absolute evaluation, the feature amount of the separated image is compared with the feature amount of the handled product stored as the object recognition data 105 on a one-to-one basis, and is calculated as a result of this comparison. The degree of similarity (0 to 100%) may be adopted as it is.
Further, when calculating the similarity as a relative evaluation, the calculation is performed so that the sum of the similarities with each handled product is 1.0 (100%). For example, it is assumed that the feature quantities of the products A and B and the secondary offers M and N are stored as the object recognition data 105. At this time, for the separated image, for example, the similarity is 0.65 for the product A, the similarity is 0.2 for the product B, the similarity is 0.1 for the sub-provided M, and the secondary offer. For the object N, the degree of similarity is calculated as 0.05.

  The similarity determination unit 823 uniquely determines a corresponding handled product or the like based on the similarity calculated by the similarity calculation unit 822 for each separated image of the object.

The confirmation notification unit 83 displays, on the customer display 11, an audio output, etc., that the handled product etc. is uniquely decided for the object of the separated image for which the similarity determination unit 823 has uniquely decided the handled product etc. To inform customers.
More specifically, the confirmation notifying unit 83 displays the separated image in which the similarity determination unit 823 uniquely determines the handled product and the like on the customer display 11 together with the green frame line, so that the handling corresponding to the separated image is performed. Indicates that a product has been uniquely determined.

  The order printing unit 84 causes the kitchen printer 4 to print a drink order. This allows the customer to communicate drink order information to the kitchen without verbally communicating the drink order to the store clerk. The merchandise ordering apparatus 1 may be configured to display on the customer display 11 that a drink has been ordered.

  The information output unit 85 refers to the product content data 106 based on the product ID of the confirmed product or the like confirmed as described above, and acquires the product classification, product name, unit price, number of sales, and the like corresponding thereto. The information output unit 85 displays the acquired information on the customer display 11 and transmits it to the POS terminal 2.

  4A to 4C are diagrams illustrating examples of screen transition. This screen is displayed on the customer display 11 (see FIG. 1).

4A is a diagram showing an example of a screen when the tray 3 on which the black coaster 5a, the chikuwa 6 and the disposable chopsticks 5c are moved to the shooting area 121 on the lane 15. FIG. The black coaster 5a is a side offer provided to the customer together with hot coffee, and is an index indicating the order of the hot coffee.
On the customer display 11, a through image by the photographing device 12 is always displayed. The customer places the chikuwa 6, the coaster 5 a and the disposable chopsticks 5 c on the tray 3, and moves the tray 3 to the photographing area 121. Thereby, the screen of FIG. 4B is displayed on the customer display 11.
In this screen, the coaster 5a is provided with a green outline, indicating that the secondary provision corresponding to the separated image of the coaster 5a has been uniquely determined. On the coaster 5a, a window 51 of “hot coffee S • 100 yen” is displayed, showing the contents of the corresponding product.

Chikuwa 6 is the product itself. In this screen, the chikuwa 6 is given a green outline to indicate that the handling product corresponding to the separated image of the chikuwa 6 has been uniquely determined. On the chikuwa 6 is displayed a window 61 of “chikuwa 180 yen” showing the contents of the corresponding product.
The disposable chopsticks 5c are not an index indicating the products handled. Since the disposable chopsticks 5c are not recognized by the commodity ordering apparatus 1, no outline or window is displayed.
At the top of the screen, the message “I will order in the kitchen in 5 seconds” is displayed. Here, 5 seconds is the time until the order information of the product is output. This message is displayed in countdown on the customer display 11 in real time. When 5 seconds have elapsed, the screen transitions to the screen of FIG.

FIG. 4B is a diagram illustrating an example of a screen when a drink is ordered in the kitchen.
In the example of FIG. 4B, the tray 3 has moved to the right side of the imaging area 121. At the top of the screen, the message “I ordered coffee in the kitchen” is displayed. This message is deleted after a predetermined time. After that, the customer receives coffee / S size which is a handling product from the store clerk.
The merchandise ordering apparatus 1 recognizes the tray 3 for a predetermined time, thereby confirming the order of the drink and printing the order information on the kitchen printer 4. Thereby, the merchandise ordering apparatus 1 can determine the merchandise order of the customer without contact, and is hygienic. When the customer cancels the order, the tray 3 or the black coaster 5a may be removed from the shooting area 121 of the product ordering apparatus 1 within a predetermined time. Thereby, the merchandise ordering apparatus 1 can cancel the merchandise order of the customer without contact.

FIG. 5 is an activity diagram showing the operation of the product ordering apparatus 1.
When the power is turned on, the merchandise ordering apparatus 1 transitions to a tray recognition process in mode M10.
Mode M10 is a tray recognition processing mode. At this time, the merchandise ordering apparatus 1 performs object recognition processing on the photographed image by the photographing apparatus 12 and recognizes a new tray. When the tray recognition process ends in mode M10, the merchandise ordering apparatus 1 transitions to mode M11.
Mode M11 is a through image display mode. At this time, the merchandise ordering apparatus 1 displays the through image of the photographing apparatus 12 on the customer display 11 as it is, and transitions to the mode M12. Specifically, this through image display is the screen illustrated in FIG.
Mode M12 is a determination in the through image display mode. At this time, the merchandise ordering apparatus 1 transitions to mode M13 if a new tray is detected in the captured image, and returns to mode M10 in other cases and repeats tray recognition processing.

Mode M13 is a product etc. recognition processing mode. At this time, the merchandise ordering apparatus 1 performs object recognition processing on the photographed image by the photographing apparatus 12 and recognizes a handled commodity or the like. When the product etc. recognition process ends in mode M13, the product ordering apparatus 1 transitions to mode M14.
Mode M14 is a recognition result display mode. At this time, the merchandise ordering apparatus 1 displays the result of the merchandise etc. recognition process on the customer display 11. Specifically, the display of the result of the product etc. recognition process is the screen illustrated in FIG.
Mode M15 is a determination in the recognition result display mode. At this time, the merchandise ordering apparatus 1 transitions to mode M16 if the predetermined time has elapsed, and returns to mode M13 if it is within the predetermined time, and repeats the merchandise recognition processing.

Mode M16 is a determination after a predetermined time has elapsed. At this time, the merchandise ordering apparatus 1 transitions to mode M17 if it detects an indicator such as a coaster and a handled product, and transitions to mode M18 if it detects only the handled product and does not detect the index. (Non-detection), the operation of FIG.
Mode M17 is an order print processing mode for the kitchen. At this time, the merchandise ordering apparatus 1 prints an order for handling merchandise corresponding to an index such as a coaster by the kitchen printer 4.
Mode M18 is a mode for notifying the POS terminal 2. At this time, the merchandise ordering apparatus 1 notifies the POS terminal 2 of the order of the handling merchandise recognized by the photographing apparatus 12 and the order of the handling merchandise corresponding to an index such as a coaster.
Mode M19 is an ordered mode. At this time, the merchandise ordering apparatus 1 displays that the order has been placed. Specifically, the display indicating that the order has been made is the screen illustrated in FIG. When the ordered mode of mode M19 ends, the operation of FIG. 5 ends.

FIG. 6 is a diagram illustrating an example of the object recognition data 105.
The object recognition data 105 includes a feature amount column 105a related to a handled product, an image link column 105b, a type column 105c, a size column 105d, and a product name column 105e.
The feature amount column 105a stores a feature amount modeled for each type related to the handled product.

  The image link column 105b stores a link of a reference image related to the handled product. The feature quantity stored in the feature quantity column 105a is information indicating the shape, hue, surface state, etc. of this reference image. The similarity determination unit 823 can determine with which handling product or the like the object related to the separated image is high by referring to the feature amount column 105a.

  The type column 105c stores the type of whether the handled product or the like is the handled product itself or an index indicating the handled product. In the present embodiment, the index indicating the handled product is a side offer provided to the customer accompanying the handled product, such as a coaster, a straw, a spoon, a chopstick, a sugar bag, a gum syrup container, a milk container, a napkin, and the like. .

  The size column 105d stores size information about the handled product. The confirmation notifying unit 83 refers to the size field 105d, so that, for example, depending on the size of a coaster, a straw, a spoon, or the like that is a sub-provided product, different handling products or different types of products It becomes possible to place an order.

  The product name column 105e stores the product name of the handled product. When “product” is stored in the type column 105c, the product name of the handled product is stored in the product name column 105e. When “index” is stored in the type column 105c, the product name corresponding to this index is stored in the product name column 105e. The confirmation notifying unit 83 determines that the product is a sub-provided item when the type field 105c relating to the uniquely handled product or the like is “indicator”, and determines the product name field 105e corresponding to the above-described index in the kitchen printer 4. Print the order.

FIG. 7 is a diagram illustrating an example of the product content data 106.
The product content data 106 includes a product name field 106a, a product image link field 106b, and a unit price field 106c. The product content data 106 is not limited to these columns, and may include an arbitrary information column related to the handled product such as discount information.
The product name column 106a stores the product name of the handled product.
The product image link column 106b stores image links of handled products.
The unit price of the handled product is stored in the unit price column 106c. The confirmation notifying unit 83 of the product ordering apparatus 1 displays the unit price on the customer display 11 with reference to the unit price column 106c.

FIG. 8 is a flowchart showing the product etc. recognition process. This will be described with reference to FIGS. 1 to 3 as appropriate.
When the customer places the tray 3 on the lane 15 and moves it to the imaging area 121, the processing unit 8 recognizes the tray 3 and starts a series of processes.
In step S30, the processing unit 8 sets a timer. This timer measures the time from when the tray 3 is recognized until a drink corresponding to a coaster or the like on the tray 3 is determined and ordered from the kitchen. The value of this timer is referred to in the process of step S41 described later.
In step S <b> 31, the processing unit 8 outputs a shooting-on signal to the shooting device 12 and starts capturing (shooting) a product image. The image storage unit 81 stores a frame image (captured image) captured by the imaging device 12 in the RAM 102.
In step S <b> 32, the object recognition processing unit 82 performs a series of object recognition processes on the frame image stored in the image storage unit 81. In this processing, the object detection unit 821 attempts to recognize (detect) a separated image of an object that is a commodity handled. The similarity calculation unit 822 reads a feature amount from the detected separated image. Further, the similarity determination unit 823 determines whether or not a handled product or the like whose feature amount similarity is equal to or greater than a predetermined threshold exists in the object recognition data 105.

  Specifically, when the chikuwa and the black coaster are placed on the tray 3, the object detection unit 821 detects the separated image of the chikuwa and the separated image of the black coaster. The similarity calculation unit 822 reads each feature amount from the separated image of Chikuwa and the separated image of the black coaster. The similarity determination unit 823 searches the object recognition data 105 based on the feature amount of the separated image of the chikuwa, that is, the similarity between the feature amount of the separated image of the chikuwa and each feature amount of the object recognition data 105. By determining the degree, it is determined that this object is a chikuwa. Further, the similarity determination unit 823 searches the object recognition data 105 based on the feature amount of the black coaster separation image, that is, the feature amount of the black coaster separation image and each feature amount of the object recognition data 105. This object is determined to be a black coaster.

In steps S33 to S39, the processing unit 8 repeats the process for the recognized object. By repeating this process, the order to the kitchen printer 4 and the product registration of the POS terminal 2 are performed.
In step S34, the similarity determination unit 823 determines whether or not the object is an index newly placed on the tray. The similarity determination unit 823 proceeds to the process of step S35 if this object is a newly placed index (Yes), and if this object is not a newly placed index (No) ), The process proceeds to step S37. More specifically, when the product ordering apparatus 1 repeatedly executes the product etc. recognition process, if the customer places a new subsidiary product on the tray, the processes of steps S35 and S36 are performed only once. Done. Here, the index refers to the recognized secondary offering.

  In step S36, the similarity determination unit 823 searches the object recognition data 105 (see FIG. 6) to determine the product name and product ID (not shown), searches the product content data 106, and the product of this product. Determine the unit price. Thereby, the merchandise ordering apparatus 1 can determine the handling merchandise.

In step S <b> 37, the similarity determination unit 823 determines whether this object is a handled product newly placed on the tray. If the object is a newly placed handling product (Yes), the similarity determination unit 823 proceeds to the process of step S38, and if the object is not a newly placed handling product. (No), the process proceeds to step S39.
In step S <b> 38, the similarity determination unit 823 searches the object recognition data 105 to identify the handled product, searches the product content data 106, and determines the product unit price of the handled product.

In step S39, the processing unit 8 determines whether the processing has been repeated for all recognized objects. If the processing unit 8 has not repeated the process for all recognized objects (No), the process unit 8 returns to the process of step S33. If the process has been repeated for all recognized objects (Yes), the process unit 8 proceeds to the process of step S40. .
In step S <b> 40, the confirmation notification unit 83 displays the product name and the unit price on the customer display 11. At this time, the product ordering apparatus 1 functions as a product price output device that outputs the price of the product.
In step S41, the processing unit 8 determines whether or not a predetermined time has elapsed since the process of step S30. If it is determined that the predetermined time has not elapsed (No), the processing unit 8 returns to the process of step S31. If it is determined that the predetermined time has elapsed (Yes), the processing of FIG.
By processing in this way, the product ordering apparatus 1 indicates that the customer provides the subsidiary product such as the coaster 5a on the tray 3 and moves the tray 3 to the photographing area 121, thereby indicating the subsidiary product. Drink order information can be confirmed. The confirmed order information is printed by the kitchen printer 4.

FIG. 9 is a sequence diagram illustrating registration processing of products and the like.
When the customer holds the tray 3 and puts the handled product or the auxiliary product on the tray 3 and passes the position of the product ordering apparatus 1-1, the operation of the sequence Q10 starts.
In sequence Q10, the merchandise ordering apparatus 1-1 detects a new tray 3 from the photographed image, and recognizes the handled merchandise and the sub-provided items on the tray 3. The merchandise ordering apparatus 1-1 identifies the handled merchandise stored in the storage unit 104 in association with the sub-provided product.
In sequence Q <b> 11, the product ordering apparatus 1-1 outputs order information of the specified handled product to the kitchen printer 4.
In sequence Q12, the kitchen printer 4 prints the received order information of the handling product on a predetermined sheet. Thereby, the store clerk on the kitchen side can know the handling products to be provided to the customer together with the subsidiary offer selected by the customer. Moreover, since the merchandise ordering apparatus 1-1 can transmit the order information of the customer to the kitchen earliest, it is possible to suitably order the merchandise that takes the most preparation time.
In sequence Q <b> 13, the product ordering apparatus 1-1 outputs information such as the price of handled products to the POS terminal 2.
In sequence Q14, the POS terminal 2 temporarily registers the handled product based on information such as the price of the handled product received from the product ordering apparatus 1-1.

Further, when the customer places a handling product or a subsidiary product on the tray 3 and passes the position of the product ordering apparatus 1-2, the operation of the sequence Q20 starts.
In sequence Q20, the merchandise ordering apparatus 1-2 detects the new tray 3 from the captured image, and recognizes the handled merchandise and the sub-provided items on the tray 3. The merchandise ordering apparatus 1-2 specifies the handling merchandise stored in the storage unit 104 in association with the sub-provided product.
In sequence Q21, the merchandise ordering apparatus 1-2 outputs order information of the specified handled merchandise to the kitchen printer 4. The specified handling product is newly recognized by the product ordering device 1-2.
In sequence Q22, the kitchen printer 4 prints the received order information of the handled product. Thereby, the store clerk on the kitchen side can know the handling products to be provided to the customer together with the sub-offer selected by the customer.
In sequence Q23, the product ordering apparatus 1-2 outputs information such as the price of the handled product to the POS terminal 2.
In sequence Q24, the POS terminal 2 additionally provisionally registers the handled product based on the received information such as the price of the handled product.

Further, when the customer places a commodity or a secondary supply on the tray 3 and passes the position of the POS terminal 2, the operation of the sequence Q30 starts.
In sequence Q30, the clerk of the POS terminal 2 receives the ordered product from the kitchen and places the ordered product on the tray 3 and delivers it.
In sequence Q31, the POS terminal 2 performs a checkout process for the ordered product relating to the customer, and ends the process of FIG.

(Modification)
The present invention is not limited to the above-described embodiment, and can be modified without departing from the spirit of the present invention. For example, there are the following (a) to (d).

(A) The product ordering apparatus operates to recognize foods other than cooked products such as chikuwa and tempura, such as bread, vegetables such as cabbage and carrots, and fruits such as apples and pears. May be. That is, a product / product that is not completely the same form but has a similar form can be a recognition target of the commodity ordering apparatus using object recognition.
(B) In the above-described embodiment, as an example, it is described that a product registration system is used for ordering products handled in a udon shop. However, the present invention is not limited to this, and the product registration system may be applied to a bakery that can be eaten and consumed in a store, and a fast food restaurant that provides hamburgers, chickens, donuts, sandwiches, and the like.
(C) In the above embodiment, drinks that are handled products are ordered by coasters, straws, spoons, sugar bags, gum syrup containers, milk containers, and the like. However, the present invention is not limited to this, and the handled product may be other than food or food and drink. Moreover, as a sub-provided product, for example, a fork, a knife, a lotus, or a paper apron provided together with a handling product may be used, and is not limited.
(D) In the above-described embodiment, the merchandise ordering apparatus 1 determines the order when it recognizes the tray 3 or the secondary offer for a predetermined time and recognizes the order and provides it to the customer together with the secondary offer. Product order information is printed by the kitchen printer 4. However, the present invention is not limited thereto, and the merchandise ordering apparatus 1 may determine the order of merchandise to be handled when, for example, the tray 3 passes the right end of the imaging region 121. Further, the merchandise ordering apparatus 1 may provide a proximity sensor at the right end of the lane 15 and the imaging region 121, for example, and when the proximity sensor detects the tray 3, the merchandise ordering apparatus 1 may determine the order of the handled merchandise. Thereby, the merchandise ordering apparatus 1 can determine the customer's order without contact.

The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
For each sub-provided provided to the customer together with the product, storage means for storing the product to be provided to the customer together with the sub-provided,
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Control means for outputting the order information of the product stored in
A product ordering device characterized by comprising:
<Claim 2>
The control means outputs the order information of the product after elapse of a predetermined time after recognizing the sub-provided,
The product ordering apparatus according to claim 1, wherein:
<Claim 3>
A display means for displaying a countdown time until the order information is output;
The product ordering apparatus according to claim 2, wherein
<Claim 4>
For each sub-provided provided to the customer together with the product, the step of storing the product to be provided to the customer together with the sub-provided in advance in the storage means;
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Outputting the order information of the product stored in
A method for ordering merchandise, comprising:
<Claim 5>
For each sub-provided provided to the customer together with the product, storage means for storing the product to be provided to the customer together with the sub-provided,
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Control means for outputting the price of the product stored in
A product price output device characterized by comprising:
<Claim 6>
For each sub-provided provided to the customer together with the product, the step of storing the product to be provided to the customer together with the sub-provided in advance in the storage means;
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Outputting the price information of the product stored in
A product price output method characterized by comprising:

1,1-1,1-2 Product ordering device (Product price output device)
11 Display for customers (display means)
12 Shooting device (Shooting means)
121 imaging region 13 illumination device 14 communication unit 15 lane 101 CPU (control means)
102 RAM
103 ROM
104 storage unit (storage means)
105 Object Recognition Data 106 Product Content Data 107 Sales Master 2 POS Terminal 3 Tray 4 Kitchen Printer 5a Coaster (Sub-Supply)
5b straw (sub-provided)
51 Window 6 Chikuwa 61 Window 7 Product Registration System 8 Processing Unit (Control Unit)
81 Image storage unit 82 Object recognition processing unit 821 Object detection unit 822 Similarity calculation unit 823 Similarity determination unit 83 Confirmation notification unit 84 Order printing unit 85 Information output unit

Claims (6)

For each sub-provided provided to the customer together with the product, storage means for storing the product to be provided to the customer together with the sub-provided,
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Control means for outputting the order information of the product stored in
A product ordering device characterized by comprising: The control means outputs the order information of the product after elapse of a predetermined time after recognizing the sub-provided,
The product ordering apparatus according to claim 1, wherein: A display means for displaying a countdown time until the order information is output;
The product ordering apparatus according to claim 2, wherein For each sub-provided provided to the customer together with the product, the step of storing the product to be provided to the customer together with the sub-provided in advance in the storage means;
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Outputting the order information of the product stored in
A method for ordering merchandise, comprising: For each sub-provided provided to the customer together with the product, storage means for storing the product to be provided to the customer together with the sub-provided,
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Control means for outputting the price of the product stored in
A product price output device characterized by comprising: For each sub-provided provided to the customer together with the product, the step of storing the product to be provided to the customer together with the sub-provided in advance in the storage means;
When the object photographed by the photographing means is recognized as a sub-provided provided to the customer together with the product based on the image information of the object, the storage means is associated with the recognized sub-provided. Outputting the price information of the product stored in
A product price output method characterized by comprising:

Images