JP2016066384A - Commodity recognition device and commodity recognition program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize a commodity from a taken image.SOLUTION: A commodity recognition device displays a frame for enclosing a commodity on an image taken by an imaging unit. The commodity recognition device recognizes a candidate of the commodity displayed in the frame on the basis of the feature amount of the image in an area enclosed by the frame to output a piece of information relevant to a commodity of the most likelihood candidate. When receiving a candidate change instruction, the commodity recognition device outputs a piece of information relevant to a commodity other than the most likelihood candidate selected from the candidates.SELECTED DRAWING: Figure 14

Description

  Embodiments described herein relate generally to a product recognition apparatus and a product recognition program for causing a computer to function as the product recognition apparatus.

  In a cafeteria-style cafeteria facility, the user prepares dishes prepared in advance on a tray, or asks a waiter to serve them, and finally performs accounting. The accounting is usually performed by registering the dishes on the tray with the cashier one by one in the cash registering device. For this reason, cafeteria-style cafeteria facilities have a problem of requiring time for accounting, and various proposals have been made to solve this problem. However, a conventionally proposed recognition device recognizes a container in which products (dishes) are arranged. For this reason, a container is restrict | limited to what can be recognized with a recognition apparatus. In this way, it is difficult to operate because there is no degree of freedom in the container.

JP-A-8-180099 JP 2001-216571 A

  A problem to be solved by an embodiment of the present invention is to provide a product recognition apparatus and a product recognition program that can be efficiently operated and can be easily corrected with a simple operation even when there is an error in the recognition result. It is.

  In one embodiment, the product recognition apparatus includes image display means, frame display means, recognition means, first output means, change acceptance means, and second output means. The image display means displays an image photographed by the imaging unit on the display unit. The frame display means displays a frame for enclosing the product at any one or a plurality of locations on the image displayed on the display unit. The recognizing unit recognizes the candidate for the product displayed in the frame from the feature amount of the image in the area surrounded by the frame. The first output means outputs information relating to the most likely candidate product among the product candidates recognized by the recognition means. The change accepting unit accepts a candidate change instruction. When the second output means receives the candidate change instruction, the second output means outputs information relating to a product other than the most promising candidate selected from the candidates.

The external view which shows one Embodiment of a goods recognition apparatus. The block diagram which shows the principal part structure of the goods recognition apparatus. The schematic diagram which shows the data structure of a recognition dictionary file. The functional block diagram of the goods recognition apparatus. The schematic diagram which shows the main memory areas formed in RAM of the goods recognition apparatus. The flowchart which shows the principal part of the information processing procedure which CPU of the goods recognition apparatus performs according to a goods recognition program. 7 is a flowchart specifically showing a procedure of image recognition processing in FIG. The flowchart which shows the information processing procedure after being judged as YES in judgment block Act13 of FIG. The flowchart which shows the information processing procedure after being judged as YES in judgment block Act14 of FIG. The flowchart which shows the information processing procedure after being judged as YES in judgment block Act7 of FIG. The schematic diagram which shows an example of the screen displayed on the panel display part of a goods recognition apparatus. The schematic diagram which shows the example of a screen after a dish item is touched from the screen shown in FIG. The schematic diagram which shows the example of a screen after a cooking item is recognized from the screen shown in FIG. The schematic diagram which shows the example of a screen after another cooking item is touched from the screen shown in FIG. The schematic diagram which shows the example of a screen after a cooking item is recognized from the screen shown in FIG. The schematic diagram which shows the example of a screen after the inside of a frame is touched from the screen shown in FIG. The schematic diagram which shows the example of a screen after a cooking item is recognized from the screen shown in FIG.

Hereinafter, an embodiment of a product recognition device will be described.
This embodiment is a case where it is installed in a cafeteria-style cafeteria facility such as an employee cafeteria or a student cafeteria.

First, the product recognition apparatus of the present embodiment employs a generic object recognition technique. General object recognition is a technique for recognizing the type of an article or the like from image data obtained by capturing the object (object) as a target. The computer extracts the appearance feature amount of the article included in the image from the image data. The computer compares the feature amount data of the reference image registered in the recognition dictionary file to calculate the similarity, and recognizes the type of the article based on the similarity. Techniques for recognizing articles included in images are described in the following documents.
Keiji Yanai, “Current Status and Future of General Object Recognition”, IPSJ Journal, Vol. 48, no. SIG16 [Search August 10, 2010], Internet <URL: http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>
Further, techniques for performing general object recognition by dividing an image into regions for each object are described in the following documents.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, [Search August 10, 2010], Internet <URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1. 1.145.3036 & rep = repl & type = pdf>
FIG. 1 is an external view of a product recognition apparatus 1 according to this embodiment. The product recognition device 1 installed on the desk surface 2A of the accounting counter 2 includes a plate-like body portion 11A and a collar portion 11B. The body portion 11A is erected along the edge on the back side of the table surface 2A so that the front surface faces the front side. The flange portion 11B protrudes from a front upper end portion of the body portion 11A in a direction substantially orthogonal to the front surface. For this reason, the lower surface of the collar part 11B opposes the table surface 2A.

  The collar unit 11 </ b> B incorporates the imaging unit 12. Further, the flange portion 11B forms a reading window (not shown) on the lower surface. The imaging unit 12 includes a CCD (Charge Coupled Device) imaging device that is an area image sensor and its drive circuit, and an imaging lens for forming an image of the imaging region on the CCD imaging device. The imaging region refers to a region of an image that forms an image on the area of the CCD imaging device from the reading window through the imaging lens. The imaging unit 12 outputs an image of the imaging region formed on the CCD imaging device through the imaging lens. Note that the imaging unit 12 is not limited to the one using a CCD imaging device, and may be, for example, a CMOS (complementary metal oxide semiconductor) image sensor.

  The merchandise recognition apparatus 1 further includes an operation panel 13 and an IC card reader / writer (hereinafter abbreviated as a card RW) 18. The operation panel 13 is attached to the top of the body portion 11A. The operation panel 13 uses the front side of the body portion 11A as an operation surface. And the operation panel 13 arrange | positions the keyboard 14, the touchscreen 15, and the receipt issuing port 16 on the said operation surface. The receipt issuing port 16 issues a receipt printed by a receipt printer 17 incorporated in the operation panel 13.

  The card RW 18 is attached to one side surface of the operation panel 13. The card RW 18 reads and writes data with respect to the IC card. The IC card stores a unique user ID and electronic money data. That is, the IC card has a user ID function and an electronic money function. Users (employees, students, etc.) who use the cafeteria facility have their own IC cards.

  A region 2B on the front side of the body portion 11A on the table surface 2A is a space for placing a container 3 on which a dish serving as a product is placed. This area 2B is included in the imaging area of the imaging unit 12. That is, the imaging unit 12 photographs the container 3 in which dishes are arranged. In FIG. 1, an elliptical tray is illustrated as the container 3, but the shape and number of the containers 3 are not particularly limited. The point is that it is a tableware or tray that a user of the cafeteria facility can prepare dishes prepared in advance and bring them to the accounting counter 2 and can be placed within the area 2B.

  FIG. 2 is a block diagram showing a main configuration of the product recognition apparatus 1. The commodity recognition apparatus 1 includes a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, and an auxiliary storage device 24. The merchandise recognition apparatus 1 connects the CPU 21 to the ROM 22, the RAM 23, and the auxiliary storage device 24 via a bus line 25 such as an address bus or a data bus.

  The CPU 21 corresponds to the central part of the computer. CPU21 controls each part in order to implement | achieve the various functions as the goods recognition apparatus 1 according to an operating system or an application program.

  The ROM 22 corresponds to a main storage part of the computer. The ROM 22 stores the above operating system and application programs. The ROM 22 may store data necessary for the CPU 21 to execute various processes.

  The RAM 23 corresponds to a main storage part of the computer. The RAM 23 stores data necessary for the CPU 21 to execute various processes as necessary. The RAM 23 is also used as a work area when the CPU 21 performs various processes.

  The auxiliary storage device 24 corresponds to the auxiliary storage portion of the computer. The auxiliary storage device 24 is, for example, an EEPROM (electrically erasable programmable read-only memory), a hard disk drive, or an SSD (solid state drive). The auxiliary storage device 24 stores data used when the CPU 21 performs various processes and data generated by the processes performed by the CPU 21. The auxiliary storage device 11 may store the above application program.

  The merchandise recognition device 1 connects a communication interface 26 to the bus line 25. Then, the merchandise recognition apparatus 1 accesses the database server 41 via the communication interface 26. The database server 41 includes a user database 40. The user database 40 stores personal information such as sex and age, information related to the use history of the cafeteria facility, and the like in association with the user ID of each user.

  The merchandise recognition device 1 connects the imaging unit 12, the keyboard 14, the touch panel 15, the receipt printer 17, and the card RW 18 via an input / output circuit (not shown). The touch panel 15 includes a panel display unit 151 and a touch panel sensor 152 that is arranged on the screen of the panel display unit 151.

  The product recognition apparatus 1 having such a configuration has a recognition dictionary file 50. The recognition dictionary file 50 is stored in the auxiliary storage device 24. Note that the storage destination of the recognition dictionary file 50 is not limited to the auxiliary storage device 24. For example, the RAM 23 may be used. Alternatively, it may be stored in a storage device of an external device connected via the communication interface 26.

  FIG. 3 is a schematic diagram showing the data structure of the recognition dictionary file 50. The recognition dictionary file 50 stores recognition dictionary data for each cooking item provided to the user at the cafeteria facility. The recognition dictionary data includes items of menu ID, menu name, price, amount of heat, and appearance feature parameter. The menu ID is a code for individually identifying the cooking item. The menu name is a unique name of the cooking item identified by the corresponding menu ID. The price is the selling price (yen) per unit amount of the cooking item identified by the corresponding menu ID. The amount of heat is a parameter derived for the standard amount of the cooking item identified by the corresponding menu ID. The appearance feature parameter is a parameterization of appearance features such as the standard shape, surface color, pattern, and unevenness of the cooking item identified by the corresponding menu ID.

  As shown in FIG. 4, the product recognition apparatus 1 having such a configuration includes a capture unit 61, an image display unit 62, a frame display unit 63, a recognition unit 64, a first output unit 65, a change reception unit 66, and a second reception unit. An output unit 67, an input reception unit 68, and a determination unit 69 are included.

  The capturing unit 61 captures an image captured by the imaging unit 12. The image display unit 62 displays the image captured by the capture unit 61 on the touch panel 15 that is a display unit. The frame display means 63 displays a frame for enclosing a product (dish) at any one or a plurality of locations on the image displayed on the touch panel 15. The recognizing means 64 recognizes the product candidates displayed in the frame from the feature amount of the image in the area surrounded by the frame. The first output means 65 outputs information related to the most likely candidate product among the product candidates recognized by the recognition means 64. The change accepting unit 66 accepts a product candidate change instruction output by the first output unit 65. When the second output unit 67 receives an instruction to change a product candidate from the change receiving unit 66, the second output unit 67 outputs information related to a product other than the most promising candidate selected from the product candidates. The input receiving unit 68 receives an instruction input to an arbitrary location on the image displayed on the touch panel 15. The determination unit 69 determines whether or not the place where the instruction input is received by the input reception unit 68 is inside the frame displayed by the frame display unit 63. When it is determined that the frame is outside the frame, the frame display unit 63 operates. When it is determined that the frame is inside the frame, the change receiving unit 66 operates.

  In order to realize such a function, the product recognition apparatus 1 stores a product recognition program in the ROM 22 or the auxiliary storage device 24. Further, as shown in FIG. 5, the merchandise recognition apparatus 1 forms a product number counter 71 with a count value n, a candidate memory 72, and a detail memory 73 in the RAM 23.

  The candidate memory 72 has an area 721 for storing menu IDs of menu items recognized from the first candidate to the fourth candidate by an image recognition process to be described later, and an area 722 for storing the candidate number m, in the order of frame numbers. .

  The detail memory 73 includes an area 731 for storing a detail record including deletion mark coordinates (X0, Y0), menu ID, menu name, price, and heat amount in the order of frame numbers, and an area 732 for storing the total price and the total heat amount. Have Each item will be described later.

  When the merchandise recognition program is activated, the CPU 21 starts processing according to the procedure shown in the flowcharts of FIGS. The contents of the processes illustrated in FIGS. 6 to 10 and described below are merely examples, and various processes capable of obtaining similar results can be used as appropriate.

  The CPU 21 waits for the IC card to be read by the card RW18 (Act1). Until the IC card is read (NO in Act 1), the process does not proceed to Act 2. When the IC card is read (YES in Act1), the CPU 21 proceeds to the process of Act2. That is, the CPU 21 resets the product number counter 71 to “0” (Act 2).

  The CPU 21 captures an image (frame image) captured by the imaging unit 12 (Act3: capture means 61). And CPU21 displays this picked-up image on the panel display part 151 of the touch panel 15 (Act4: Image display means 62).

  FIG. 11 is an example of one screen of the panel display unit 151 on which the captured image 81 is displayed. As shown in the figure, in addition to the captured image 81, a total amount column 82, a total calorie column 83, a confirmation button 84, and a cancel button 85 are displayed.

  When the captured image 81 is displayed on the screen of the panel display unit 151, the CPU 21 checks whether or not the screen is touched (Act5). When the screen is not touched (NO in Act 5), the CPU 21 checks the product counter 71 (Act 6). If the product counter 71 has been reset to “0” (NO in Act 6), the CPU 21 returns to the processing in Act 3. That is, the CPU 21 captures the next captured image from the imaging unit 12 (Act 3) and displays it on the panel display unit 151 (Act 4).

  The CPU 21 repeats capturing and displaying of the captured image until the screen is touched. When the screen is touched (YES at Act5), the CPU 21 checks which area of the screen is touched (Act7, Act8). If the area is not the area of the confirm button 84 or the cancel button 85 (NO in Act 7) and is not the area of the captured image 81 (NO in Act 8), the CPU 21 returns to the process of Act 5. That is, touch input is ignored.

  When the area of the captured image 81 is touched (YES in Act 8), the CPU 21 detects the coordinates (X, Y) of the touch position (Act 9: input receiving means 68). For coordinates (X, Y), the lower left corner of the reference point, for example, the area of the photographed image 81 is the origin (0, 0) of the two-dimensional coordinates, and the right direction from this origin (0, 0) is the X direction and the upper direction is the upper direction. It is defined as the Y direction. Then, the CPU 21 obtains the distance in the X direction and the distance in the Y direction from the origin (0, 0) to the touch position, and converts them into coordinates (X, Y).

  The CPU 21 checks the product number counter 71 (Act10). When the product counter 71 is reset to “0” (NO in Act 10), the CPU 21 determines a recognition area of the image, and displays a frame 91 surrounding the area in the area of the captured image 81. That is, as shown in FIG. 12, the CPU 21 displays a rectangular frame 91 having a vertical length a and a horizontal length b with the touch position coordinate (X, Y) as the center (Act11: frame display means). 63). The color of the frame 91 at this time is a default color, for example, black.

  The CPU 21 executes image recognition processing (Act 12). FIG. 7 is a flowchart showing a specific procedure of image recognition processing. First, the CPU 21 counts up the product counter 71 by “1” (Act 21). Further, the CPU 21 extracts appearance feature amounts such as a contour shape, a surface shade, a pattern, and an uneven state from the image surrounded by the frame 91 (Act 22). Then, the CPU 21 calculates, for all menu item records registered in the recognition dictionary file 50, a similarity indicating how much the feature amount is similar to the appearance feature parameter of the record (Act23).

  If the similarity is calculated for all menu items, the CPU 21 selects the menu items from the first to the fourth in descending order of similarity. The menu IDs of the first to fourth menu items are sequentially stored in the area 721 from the first candidate to the fourth candidate of the frame number n (n is the count value of the product counter 71) in the candidate memory 72. (Act24: recognition means 64). That is, the menu ID of the menu item with the highest similarity is stored in the first candidate area, and the menu ID of the second highest menu item is stored in the second candidate area. Hereinafter, the menu ID of the third highest menu item is stored in the third candidate area, and the menu ID of the fourth highest menu item is stored in the fourth candidate area.

  The CPU 21 sets the candidate number m of the frame number n in the candidate memory 72 to “1” (Act 25). Then, the CPU 21 acquires information (menu ID, menu name, price, and heat amount) of the menu item specified by the menu ID of the mth candidate (m is a candidate number) from the recognition dictionary file 50 (Act26). The CPU 21 calculates the coordinates of the upper right corner of the frame 91 as the deletion mark coordinates (X0, Y0) (Act27).

  The CPU 21 sets the deletion mark coordinates (X0, Y0) and the menu item information in the detail record of the frame number n in the detail memory 73 (Act 28). Next, as shown in FIG. 13, the CPU 21 displays a deletion mark 92 at the position of the deletion mark coordinates (X0, Y0). Further, the CPU 21 displays the menu name, price, and heat amount of the menu item information in the frame 91 (Act 29: first output means 65). Further, the CPU 21 calculates the total amount and the total heat amount from the data in the detail memory 73, and displays them in the total amount column 82 and the total calorie column 83 on the screen (Act 30). Thus, the image recognition process ends.

  When the image recognition process ends, the CPU 21 returns to the process of Act5. That is, the CPU 21 checks whether or not the screen of the panel display unit 151 is touched (Act 5). If the screen is not touched (NO in Act 5), the CPU 21 checks the product counter 71 (Act 6).

  At this time, since the product counter 71 has been counted up (YES in Act 6), the CPU 21 returns to the processing of Act 5. Therefore, the CPU 21 does not capture a captured image. As a result, the previously captured image is displayed as a still image on the panel display 151 as it is.

  When the screen is touched (YES at Act5), the CPU 21 checks which area of the screen is touched (Act7, Act8). If the area of the captured image 81 is touched again (YES in Act 8), the CPU 21 detects the coordinates (X, Y) of the touch position (Act 9). At this time, the product number counter 71 is incremented. Therefore, the process proceeds to “YES” in the process of Act10. That is, the CPU 21 determines whether the deletion mark 92 has been touched, whether the inside of the recognition area frame 91 has been touched, or the outside of the frame 91 has been touched (Act 13, 14: determination means 69).

  Specifically, the CPU 21 collates the touch position coordinates (X, Y) with the deletion mark coordinates (X0, Y0) of each detail record stored in the detail memory 73. If there is a deletion mark coordinate (X0, Y0) that substantially matches the touch position coordinate (X, Y), it is determined that the deletion mark 92 has been touched. If the touch position coordinates (X, Y) match the coordinates in the area of any of the frames 91, it is determined that the inside of the frame 91 has been touched. If the touch position coordinates (X, Y) are not any of the above, it is determined that the outside of the frame 91 has been touched.

  If it is determined that the outside of the frame 91 has been touched (NO in Acts 13 and 14), the CPU 21 proceeds to the process of Act11. That is, as shown in FIG. 14, the CPU 21 displays a rectangular frame 91a having a vertical length a and a horizontal length b with the touch position coordinate (X, Y) as the center (Act11: frame display). Means 63). Then, the CPU 21 executes image recognition processing within the frame 91a. As a result, the product number counter n is further counted up. In the area 721 from the first candidate to the fourth candidate of the frame number n in the candidate memory 72, the menu IDs of the menu items from the first place to the fourth place are stored in descending order of similarity. In addition, in the detail record with the frame number n in the detail memory 73, the information of the menu item of the first candidate (first similarity) is set together with the deletion mark coordinates (X0, Y0). And as shown in FIG. 15, the menu name of the said menu item information, a price, and a calorie | heat amount are displayed in the frame 91a. Further, the total amount and the total amount of heat are updated.

  If it is determined that delete mark 92 has been touched (YES in Act 13), CPU 21 proceeds to the process shown in the flowchart of FIG. That is, the CPU 21 detects the frame number P from the detail record including the deletion mark coordinates (X0, Y0) that substantially match the touch position coordinates (X, Y) (Act 41). Then, the CPU 21 erases the frame 91 identified by the frame number P from the screen (Act 42: erasing means 68). Further, the CPU 21 deletes the record of the frame number P from the detail memory 73 (Act 43). Further, the CPU 21 recalculates the total amount and the total amount of heat from the data in the detailed memory 73, and updates the display of the total amount column 82 and the total calorie column 83 (Act 44).

  Thereafter, the CPU 21 counts down the product counter 71 by “1” (Act 45). Then, the CPU 21 checks whether or not the count value n of the product counter 71 is “0” (Act 46). When the count value n is not “0” (NO in Act 46), the CPU 21 returns to the process of Act 5. When count value n becomes “0” (YES in Act 46), CPU 21 returns to the processing in Act 3.

  Therefore, when the count value n is not “0”, the CPU 21 does not capture a new captured image. For this reason, the captured image displayed on the panel display unit 151 does not change. However, when the count value n becomes “0”, the CPU 21 resumes capturing of the captured image. Therefore, the captured image displayed on the panel display unit 151 is updated to the latest image.

  If it is determined that the inside of the frame 91 has been touched (NO in Act 13, YES in Act 14: change accepting means 66), the process proceeds to the process shown in the flowchart of FIG. That is, the CPU 21 detects the frame number Q from the detail record including the deletion mark coordinates (X0, Y0) that form the frame 91 in which the touch position coordinates (X, Y) exist (Act 51). Then, the CPU 21 adds “1” to the candidate number m corresponding to the frame number Q in the candidate memory 72 (Act 52).

  The CPU 21 determines whether or not the candidate number m has exceeded the maximum number M of candidates (“4” in this embodiment) (Act 53). If candidate number m exceeds maximum value M (YES in Act 53), CPU 21 returns candidate number m to “1” (Act 54). If candidate number m does not exceed maximum value M (NO in Act 53), candidate number m is not changed.

  The CPU 21 acquires information (menu ID, menu name, price, and heat amount) of the menu item specified by the menu ID of the m-th candidate (m is a candidate number) from the recognition dictionary file 50 (Act 55). Then, the CPU 21 rewrites the menu item information in the detail record of the frame number n in the detail memory 73 to the current menu item information (Act 56: second output means 67). Further, the CPU 21 changes the menu name, price, and amount of heat displayed in the frame 91 of the frame number Q to the current menu item information (Act 57). Then, the CPU 21 recalculates the total amount and the total amount of heat from the data in the detailed memory 73, and updates the values in the total amount column 82 and the total calorie column 83 (Act 58). Further, the CPU 21 changes the color of the frame 91 with the frame number Q to a color corresponding to the m-th candidate (Act 59). That is, in the present embodiment, at least four types of colors are prepared for the frame 91, and the color of the frame 91 is changed according to the menu item candidate rank displayed in the frame 91. Thereafter, the CPU 21 returns to the processing of Act 21.

  Therefore, each time the user touches the inside of the frame 91, the menu item information in the frame is switched to the information of the lower menu item in order from the first candidate ranking. When the inside of the frame in which the menu item information with the lowest candidate ranking is displayed is further touched, the menu item information in the frame returns to the information of the menu item with the highest candidate ranking.

  When the button image is touched (YES in Act 7), the CPU 21 confirms which of the confirm button 84 or the cancel button 85 is touched (Act 61). When the confirm button 84 is touched (“Confirm” in Act 31), the CPU 21 executes each process of Act 62 to Act 64.

  In Act 62, the CPU 21 executes settlement processing using electronic money. That is, the CPU 21 subtracts the data for the total price from the electronic money data of the IC card via the card RW18. In Act 63, the CPU 21 edits the receipt data based on the data in the detail memory 73. Then, the CPU 21 outputs receipt data to the printer 17 and controls the issuance of receipts. In Act 64, the CPU 21 edits the user history information with the data in the detailed memory 73 and the IC card user ID. Then, the CPU 21 transmits user history information to the database server 41 via the communication interface 26. In the database server 41, user history information is stored in the user database 40 for each user ID.

  When the processing of Act 62 to Act 64 is completed, the CPU 21 initializes the screen of the panel display unit 151 (Act 65). Further, the CPU 21 clears the candidate memory 72 and the detail memory 73 (Act 66).

  On the other hand, when the cancel button 85 is touched (“Cancel” in Act 61), the CPU 21 does not execute the processes of Act62 to Act64. The CPU 21 initializes the screen of the panel display unit 151 (Act 65). Further, the CPU 21 clears the candidate memory 72 and the detail memory 73 (Act 66). This completes the process of the product recognition program for one user.

  In the cafeteria-style cafeteria facility in which the product recognition apparatus 1 of the present embodiment is installed, the user places the container 3 on which the dish is placed in the area 2B of the accounting counter 2. Then, the user causes the card RW 18 to read the data of his / her IC card. Then, as shown in FIG. 11, since the captured image 81 is displayed on the screen of the touch panel 15, the user touches the dish images arranged on the container 3 one by one.

  Now, suppose that the user touches the cooking image displayed on the upper left of the container 3. Then, a frame 91 centered on the touch position coordinates (X, Y) is displayed on the captured image 81 as shown in FIG. In the product recognition apparatus 1, the dish recognition process in the frame 91 is executed, and the menu items from the first candidate rank to the fourth rank are selected in descending order of similarity. Then, as shown in FIG. 13, the name, price, and heat quantity of the menu item with the highest candidate ranking are displayed in a frame 91. In addition, a delete mark 92 is displayed in the frame 91. The total amount and total calories of recognized dishes are also displayed.

  Next, it is assumed that the user touches an image of a dish placed on the upper center side of the container 3. Then, as shown in FIG. 14, a frame 91 a centered on the touch position coordinates (X, Y) is displayed on the captured image 81. In the product recognition apparatus 1, the dish recognition process in the frame 91a is executed, and the menu items from the candidate rank 1 to the 4th rank are selected in descending order of similarity. Then, as shown in FIG. 15, the name, price, and amount of heat of the menu item ranked first in the candidate ranking are displayed in the frame 91a. In addition, a deletion mark 92 is displayed in the frame 91a. The total amount of food recognized and the total calories are also combined.

  Here, it is assumed that the user notices that the information on the menu item displayed in the frame 91 is not the information on the dish displayed on the upper left of the container 3. In that case, there are a method of touching the deletion mark 92 in the frame 91 and redoing the image recognition process, and a method of touching the inside of the frame 91 and displaying information on the menu item having a lower candidate order.

  When the deletion mark 92 is touched, the frame 91 is deleted. Therefore, the user touches another part of the dish image and repeats the recognition process. Then, the frame 91 centered on the touch position coordinates (X, Y) is displayed again on the captured image 81. Then, the product recognition apparatus 1 executes the recognition process of the dishes in the frame 91, selects the menu items from the first candidate rank to the fourth rank in order of similarity, and names the menu items ranked first in the candidate rank. The price and the amount of heat are displayed in the frame 91. Since the area of the recognition image differs depending on the touch position, the menu item is highly likely to be correctly recognized from the dish image by the re-recognition process.

  On the other hand, as shown in FIG. 16, when the inside of the frame 91 is touched, as shown in FIG. 17, the display content in the frame 91 is switched to the information of the menu item with the lower candidate rank. In addition, the total amount and total calories of the recognized dishes are also updated. Accordingly, when the correct menu item is recognized at the second or lower candidate rank, the information in the frame 91 can be corrected to the correct menu item information by a simple operation.

  The user repeats the touch operation described above until all dishes arranged in the container 3 are correctly recognized. When all the dishes are recognized, the user touches the confirmation button 84. Then, a payment process using electronic money is performed on the dish arranged in the container 3, and a receipt is issued from the receipt issuing port 16. Further, the user history information is transmitted from the product recognition apparatus 1 to the database server 41, and the usage history information of the user stored in the database 40 is updated.

  Thus, the merchandise recognition apparatus 1 does not recognize the container 3 in which dishes are arranged, but recognizes the dishes arranged in the containers 3 one by one. Therefore, since there is no need for a restriction such as a one-to-one correspondence between the container 3 and the dish or a special information medium attached to the container 3, a cafeteria-style cafeteria facility is efficiently operated.

  Moreover, the merchandise recognition apparatus 1 not only recognizes the dishes stacked in the container, but also calculates the total amount by summing the prices of the dishes. The merchandise recognition apparatus 1 automatically settles the price of the recognized dish using the electronic money data of the IC card read by the card RW18. Therefore, cafeteria-style cafeteria facilities are operated efficiently, including not only recognition of dishes but also payment of prices.

  Moreover, even if there is an error in the recognition result of the merchandise recognition device 1, it can be easily corrected by a simple operation.

The present invention is not limited to the above embodiment.
For example, in the above-described embodiment, when a candidate change instruction is received by the change accepting unit 66, information related to the menu item one level lower than that is displayed, but information related to all menu items lower than the current level is displayed and output. Then, information appropriately selected by the user may be displayed and output in the frame 91.

  In the above embodiment, the merchandise recognition apparatus 1 corresponding to electronic money payment is exemplified, but the payment method is not particularly limited, and may be applicable to credit card payment or cash payment. Alternatively, the settlement function may be omitted from the merchandise recognition device 1. In this case, the merchandise recognition apparatus 1 connects a payment terminal such as a POS (Point Of Sales) terminal via a communication line. And the merchandise recognition apparatus 1 should just transmit and output the information of the recognized cooking item to a POS terminal.

  In the embodiment, the recognition dictionary file 50 is stored in the auxiliary storage device 24 of the product recognition apparatus 1. However, the storage location of the recognition dictionary file 50 is not limited to the auxiliary storage device 24. For example, the recognition dictionary file 50 may be stored in a storage device externally attached to the merchandise recognition device 1, and the data in the recognition dictionary file 50 may be retrieved by the CPU 21 accessing the storage device as necessary.

  Moreover, in the said embodiment, although the frame 91 was made into the quadrangle whose vertical length is a and horizontal length is b, the size and shape of the frame 91 are not specifically limited. A function for changing the size and size of the frame displayed on the panel display unit 151 may be added. By providing this function, the dish image can be correctly surrounded by a frame, so that an effect of increasing the recognition rate can be expected.

  Moreover, in the said embodiment, it programmed so that a menu item might be recognized from the image whenever it touches, but it is not necessarily limited to this. For example, the coordinates of the touched position may be sequentially stored, and if there is an operation input for starting recognition, the menu item may be sequentially recognized from the image in the area specified by each coordinate. By doing so, image recognition is performed after the user touches a plurality of dishes.

  Moreover, in the said embodiment, although the case where the dish arranged in the container was recognized was shown, goods, such as bread other than a dish, may be made into recognition object. Moreover, in the said embodiment, although the calorie | heat amount was illustrated as a parameter derived | led-out with respect to the standard amount of goods, a parameter is not limited to a calorie | heat amount. For example, nutrient components such as proteins, lipids, and calcium may be used.

  In the above-described embodiment, the control program for realizing the functions of the invention is recorded in advance in the ROM 22 or the auxiliary storage device 24 which is a program storage unit in the apparatus. However, the present invention is not limited to this, and a similar program may be downloaded from the network to the apparatus. Alternatively, a similar program recorded on the recording medium may be installed in the apparatus. The recording medium may be in any form as long as it can store a program such as a CD-ROM or a memory card and can be read by the apparatus. Further, the function obtained by installing or downloading the program may be realized in cooperation with an OS (operating system) in the apparatus. Moreover, the program of this embodiment may be incorporated in a portable information terminal such as a mobile phone having a communication function or a so-called PDA to realize the function.

  In addition, although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 1 ... Information recognition apparatus, 12 ... Imaging part, 13 ... Operation panel, 15 ... Touch panel, 17 ... Receipt issue port, 18 ... Card RW, 21 ... CPU, 50 ... Recognition dictionary file, 71 ... Item number counter, 72 ... Candidate memory 3 ... Detail memory.

Claims (6)

Image display means for displaying an image captured by the imaging unit on the display unit;
Frame display means for displaying a frame for enclosing the product at any one or a plurality of locations on the image displayed on the display unit;
Recognizing means for recognizing product candidates displayed in the frame from the feature amount of the image in the area surrounded by the frame;
A first output means for outputting information related to the most likely candidate product among the candidate products recognized by the recognition means;
Change accepting means for accepting change instructions for candidates output by the first output means;
And a second output unit that outputs information relating to a product other than the most promising candidate selected from the candidates when the change instruction unit accepts a product candidate change instruction. Product recognition device. The recognition means calculates the similarity between the parameter indicating the appearance feature amount of each product and the feature amount of the image in the region surrounded by the frame line for each product, and recognizes candidates in descending order of similarity.
The first output means outputs information related to the product with the product having the highest similarity as the most likely candidate product,
The second output unit sequentially selects a lower candidate product having a low similarity every time a change instruction is received by the change receiving unit, and outputs information related to the product. The product recognition apparatus according to 1. Input receiving means for receiving an instruction input to an arbitrary location on the image displayed on the display unit;
The product recognition apparatus according to claim 1, wherein the frame display unit displays the frame around a place where an instruction input is received by the input reception unit. A determination means for determining whether or not the place where the instruction input is received by the input reception means is inside the frame displayed by the frame display means;
4. The product recognition apparatus according to claim 3, wherein the frame display means operates when it is determined to be outside the frame, and the change reception means operates when it is determined to be inside the frame. . Each of the first and second output means is means for displaying and outputting information relating to a product in the frame; and
The said 2nd output means makes the display form of the said frame differ from the frame in which the information which concerns on the said product is displayed and output by the said 1st output means inside any one of Claim 1 thru | or 4 characterized by the above-mentioned. The product recognition apparatus according to claim 1. To a computer that connects the imaging unit and the display unit,
A function of displaying an image captured by the imaging unit on the display unit;
A function of displaying a frame for enclosing a product at any one or a plurality of locations on the image displayed on the display unit;
A function of recognizing product candidates displayed in the frame from the feature amount of the image in the area surrounded by the frame;
A function to output information on the most likely candidate product among the recognized products;
The ability to accept candidate change instructions;
A product recognition program for realizing a function of outputting information relating to a product other than the most promising candidate selected from the candidates when receiving the candidate change instruction.