JP2013229058A - Store system and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a store system and program according to which, even if a problem occurs in object recognition, it is possible to easily solve the problem.SOLUTION: A store system comprises: image output means 1613 for outputting an image photographed by photographing means; object recognition means 611 for recognizing a specific object by reading a feature amount of the output image; confirmation screen display means 615 for displaying at least an image about the recognized object on a display section; and problem solution means 616 for, when a problem occurs in object recognition, receiving an instruction indicating the problem and solving the problem according to a content of the received instruction.

Description

  Embodiments described herein relate generally to a store system and a program.

  Conventionally, in a supermarket or the like, a code symbol such as a barcode or a two-dimensional code (for example, QR code (registered trademark)) attached to a product is imaged by a digital camera such as a CCD image sensor, and the code detected from the captured image 2. Description of the Related Art A product code reader that reads a product code by detecting and decoding a symbol is used.

  However, in the prior art, since a code symbol such as a barcode or a two-dimensional code must be attached to a product, a package or the like is separately prepared for a fresh food product or bread, and the code symbol is attached. Or a code symbol table is prepared.

  The store system according to the embodiment includes an image output unit that outputs an image captured by the imaging unit, and an object recognition unit that recognizes a specific object by reading a feature amount of the output image. Further, the store system of the embodiment accepts an instruction indicating the problem when there is a problem in the confirmation image display means for displaying at least an image of the recognized object on the display unit and the recognition of the object, Problem solving means for solving the problem according to the received instruction content.

  The store system according to the embodiment includes an imaging unit, a display unit, an image output unit that outputs an image captured by the imaging unit, and an object recognition that recognizes a specific object by reading a feature amount of the output image Means, a confirmation image display means for displaying at least an image of the recognized object on the display unit, and if there is a problem in the recognition of the object, an instruction indicating the problem is received and the received instruction content is changed to And a problem solving means for solving the problem accordingly.

  The program according to the embodiment causes a computer to function as an object recognition unit that recognizes a specific object by reading a feature amount of an image captured by the imaging unit. In addition, the program according to the embodiment accepts an instruction indicating a problem when there is a problem with the confirmation image display unit that displays at least an image of the recognized object on the display unit and the recognition of the object. And function as problem solving means for solving the problem according to the received instruction content.

FIG. 1 is a perspective view showing a checkout system according to the present embodiment. FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal and the commodity reading apparatus. FIG. 3 is a conceptual diagram illustrating the data structure of a PLU file. FIG. 4 is a block diagram illustrating a functional configuration of the CPU. FIG. 5 is a flowchart illustrating an example of processing operation of the checkout system. FIG. 6 is a conceptual diagram illustrating an example of a reading area in the reading window. FIG. 7 is a conceptual diagram showing a display example by the display device. FIG. 8 is a conceptual diagram showing a display example by the display device. FIG. 9 is a conceptual diagram illustrating a display example by the display device when a product for which recognition is not performed occurs. FIG. 10 is a conceptual diagram showing a display example by the display device when one is originally recognized as one. FIG. 11 is a conceptual diagram showing a display example by a display device when a plurality of identical products are recognized as one because they overlap each other. FIG. 12 is a conceptual diagram showing a display example by a display device when a return of merchandise is requested after recognition. FIG. 13 is a conceptual diagram showing a method for allowing an operator (store clerk) to specify an image area that does not rise as a candidate. FIG. 14 is a conceptual diagram illustrating a method for allowing an operator (store clerk) to specify an image area that does not rise as a candidate. FIG. 15 is a conceptual diagram showing a method of dividing an image area by an operator (a sales clerk) when one is originally recognized as one. FIG. 16 is a conceptual diagram showing a method for causing an operator (store clerk) to specify a product to be returned. FIG. 17 is a conceptual diagram showing a display example by the customer display device. FIG. 18 is an explanatory diagram schematically showing a specific example of processing.

  Below, the store system and program concerning this embodiment are explained with reference to drawings for a checkout system as an example. The store system is a checkout system (POS system) provided with a POS terminal that performs registration and settlement of commodities related to one transaction. This embodiment is an application example to a checkout system introduced in a store such as a supermarket, a pastry shop, or a donut shop.

  FIG. 1 is a perspective view showing a checkout system 1. As shown in FIG. 1, the checkout system 1 includes a POS terminal 11 that performs registration and settlement of commodities related to one transaction. The POS terminal 11 is placed on the upper surface of the drawer 21 on the checkout table 51. The drawer 21 is controlled by the POS terminal 11 for the opening operation. On the upper surface of the POS terminal 11, a keyboard 22 having various keys as an input unit pressed by an operator (store clerk) is arranged. A display device 23 that displays information toward the operator is provided behind the keyboard 22 when viewed from the operator who operates the keyboard 22. The display device 23 displays information on the display surface 23a. A touch panel 26 as an input unit is stacked on the display surface 23a. A customer display device 24 is erected so as to be rotatable further behind the display device 23. The customer display device 24 displays information on the display surface 24a. The customer display device 24 shown in FIG. 1 has the display surface 24a facing the front side in FIG. 1, but the customer display device 24 is rotated so that the display surface 24a faces the back side in FIG. Thus, the customer display device 24 displays information to the customer.

  A display case 151 is arranged so as to form an L shape with the checkout table 51 on which the POS terminal 11 is placed. A cargo receiving surface 152 is formed on the upper surface of the display case 151. A paper box 153 for storing the product A is placed on the cargo receiving surface 152. Note that the paper box 153 may be a tray used in a pastry store, a bakery store, or the like.

  A commodity reading apparatus 101 connected to the POS terminal 11 so as to be able to transmit and receive data is installed on the receiving surface 152 of the display case 151. The commodity reading apparatus 101 includes a thin rectangular housing 102. A reading window 103 is disposed in front of the housing 102. A display / operation unit 104 is attached to the upper portion of the housing 102. The display / operation unit 104 is provided with a display device 106 on which a touch panel 105 is laminated. A keyboard 107 is disposed on the right side of the display device 106. A card reading groove 108 of a card reader (not shown) is provided on the right side of the keyboard 107. A customer display device 109 for providing information to the customer is installed on the back left side of the display / operation unit 104 as viewed from the operator.

  Such a product reading apparatus 101 includes a product reading unit 110 (see FIG. 2). The merchandise reading unit 110 has an imaging unit 164 (see FIG. 2) disposed behind the reading window 103.

  In pastry shops and donut shops, the customer selects the product A in the display case 151, and the store clerk takes it on a tray (not shown). Thereafter, the product A is moved from the tray to the paper box 153 by a store clerk. After all the products are moved to the paper box 153, the paper box 153 is directed to the reading window 103 of the product reading apparatus 101. At this time, the imaging unit 164 (see FIG. 2) arranged on the back side of the reading window 103 images all the products A in the paper box 153. The merchandise reading apparatus 101 detects all or part of the merchandise A included in the image captured by the imaging unit 164. When the product reading apparatus 101 detects that the image captured by the imaging unit 164 includes all or part of the product A, the captured image is output to the POS terminal 11. In the POS terminal 11, for each of the products A, the product reading unit is referred to with reference to a PLU file F1 (details will be described later, see FIG. 3) in which information related to sales registration of the product A and an image of the product A are associated with each other. The product A to be registered for sales is identified by recognizing the product A, which is a specific object, from all or part of the images of the product A captured by the imaging unit 164 of 110, and the product ID of the identified product A The sales registration, such as product classification, product name, unit price, etc., is recorded in a sales master file (not shown).

  FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal 11 and the commodity reading apparatus 101. The POS terminal 11 includes a microcomputer 60 as an information processing unit that executes information processing. The microcomputer 60 is configured by connecting a ROM 62 (Read Only Memory) and a RAM 63 (Random Access Memory) to a CPU 61 (Central Processing Unit) that executes various arithmetic processes and controls each unit.

  The drawer 61, the keyboard 22, the display device 23, the touch panel 26, and the customer display device 24 are all connected to the CPU 61 of the POS terminal 11 through various input / output circuits (all not shown). . These are controlled by the CPU 61.

  The keyboard 22 includes a numeric key 22d, a temporary fastening key 22e, and a fastening key 22f on which numbers such as “1”, “2”, “3”,...

  An HDD 64 (Hard Disk Drive) is connected to the CPU 61 of the POS terminal 11. The HDD 64 stores programs and various files. All or a part of the programs and various files stored in the HDD 64 are copied to the RAM 63 and sequentially executed by the CPU 61 when the POS terminal 11 is activated. An example of a program stored in the HDD 64 is a product sales data processing program PR. An example of a file stored in the HDD 64 is a PLU file F1 distributed and stored from the store computer SC.

  The PLU file F1 is a file in which an association between information related to sales registration of the product A and an image of the product A is set for each of the products A displayed and sold in the store. FIG. 3 is a conceptual diagram illustrating the data configuration of the PLU file F1. As shown in FIG. 3, for each product A, the PLU file F <b> 1 includes a product ID uniquely assigned to each product A, product information to which the product A belongs, product information such as product name, unit price, and product image obtained by capturing the product. And a threshold value of “similarity: 0.XX”. Although the details will be described later, the threshold value of “similarity: 0.XX” is the product of the product that is stored in the PLU file F1 in advance when the product A is a donut or the like and is fried / too deep By comparing with the image, it is possible to determine that the article is different from the normal state.

  Returning to FIG. 2, a communication interface 25 for executing data communication with the store computer SC is connected to the CPU 61 of the POS terminal 11 via an input / output circuit (not shown). The store computer SC is installed in a store backyard or the like. A PLU file F1 distributed to the POS terminal 11 is stored in the HDD (not shown) of the store computer SC.

  Further, the CPU 61 of the POS terminal 11 is connected to a connection interface 65 that enables data transmission / reception with the commodity reading apparatus 101. The product reading apparatus 101 is connected to the connection interface 65. The CPU 61 of the POS terminal 11 is connected to a printer 66 that prints on receipts and the like. The POS terminal 11 prints the transaction content of one transaction on the receipt under the control of the CPU 61.

  The commodity reading apparatus 101 also includes a microcomputer 160. The microcomputer 160 is configured by connecting a ROM 162 and a RAM 163 to a CPU 161 via a bus. The ROM 162 stores a program executed by the CPU 161. The CPU 161 is connected to an imaging unit 164 and an audio output unit 165 via various input / output circuits (none of which are shown). The operations of the imaging unit 164 and the audio output unit 165 are controlled by the CPU 161. The display / operation unit 104 is connected to the POS terminal 11 via the connection interface 176. The operation of the display / operation unit 104 is controlled by the CPU 61 of the POS terminal 11.

  The imaging unit 164 is a color CCD image sensor, a color COMS image sensor, or the like, and is an imaging unit that performs imaging from the reading window 103 under the control of the CPU 161. For example, the imaging unit 164 captures a 30 fps moving image. Frame images sequentially captured by the imaging unit 164 at a predetermined frame rate are stored in the RAM 163.

  The audio output unit 165 is an audio circuit and a speaker for generating a preset warning sound or the like. The sound output unit 165 performs notification by sound such as a warning sound under the control of the CPU 161.

  Furthermore, a connection interface 175 that connects to the connection interface 65 of the POS terminal 11 and enables data transmission / reception with the POS terminal 11 is connected to the CPU 161. Under the control of the CPU 161, an image (an image including all or part of the product A) imaged by the imaging unit 164 of the product reading apparatus 101 is output via the connection interface 175, and the POS terminal via the connection interface 65. 11 is input.

  Next, functional units of the CPU 161 and the CPU 61 that are realized by the CPU 161 and the CPU 61 sequentially executing programs will be described with reference to FIG. FIG. 4 is a block diagram illustrating the functional configuration of the CPU 161 and the CPU 61. As illustrated in FIG. 4, the CPU 161 has functions as a captured image capturing unit 1611, a product detection unit 1612, and an image output unit 1613 that is an image output unit by sequentially executing a program. Similarly, the CPU 61 is an object recognition unit 611 that is an object recognition unit, a product registration unit 612, a similarity determination unit 613, a defective product notification unit 614, a confirmation image display unit 615 that is a confirmation image display unit, and a problem solving unit. A function as a problem solving unit 616 is provided.

  The captured image capturing unit 1611 outputs an imaging on signal to the imaging unit 164 and causes the imaging unit 164 to start an imaging operation. The captured image capturing unit 1611 sequentially captures frame images captured by the imaging unit 164 and stored in the RAM 163 after the start of the imaging operation. Captured frame images by the captured image capturing unit 1611 are performed in the order stored in the RAM 163.

  The product detection unit 1612 detects all or part of the product A included in the frame image captured by the captured image capture unit 1611 using a pattern matching technique or the like. Specifically, an outline or the like is extracted from an image obtained by binarizing the captured frame image. Next, the contour line extracted from the latest frame image is compared with the contour line extracted from the current frame image, and the part that has changed, that is, the product directed to the reading window 103 for sales registration Detects reflections. As another method for detecting a product, the presence or absence of a skin color region is detected from a captured frame image. Next, when the skin color area is detected, that is, when the reflection of the clerk's hand is detected, the contour line is detected as described above, so that the clerk's hand is assumed to be grasped by the hand. Attempt contour extraction. At this time, when the contour indicating the shape of the hand and the other contour are detected, the reflection of the product is detected because the clerk's hand is holding the product.

  The image output unit 1613 outputs the frame image captured by the captured image capturing unit 1611 to the POS terminal 11 via the connection interface 175. The image output unit 1613 may sequentially output the frame image captured by the captured image capturing unit 1611 to the POS terminal 11, but in the present embodiment, all or part of the product A is detected by the product detection unit 1612. Assume that a frame image is output to the POS terminal 11. In this way, by outputting a frame image in which all or part of the product A is detected by the product detection unit 1612 to the POS terminal 11, the PLU file F <b> 1 is obtained from the frame image that does not include all or part of the product A. It is possible to prevent the POS terminal 11 from performing object recognition performed with reference to FIG. Since recognition processing of a specific object requires processing time, by preventing processing with a frame image that does not include recognition of the specific object and does not include all or part of the product A, processing time can be reduced. Shortening can be achieved.

  The object recognizing unit 611 refers to the product image of the PLU file F1, and from the whole or a part of the image of the product A imaged by the imaging unit 164 of the product reading device 101, the color, surface unevenness, etc. The product A is recognized as a specific object by reading the state of the surface. Note that the object recognition unit 611 does not consider the outline or size of the product A in order to shorten the processing time. In the POS terminal 11, the product read by the product reading device 101 can be identified from the products registered in advance in the PLU file F1 based on the recognition result of the object recognition unit 611.

  The similarity determination unit 613 calculates the similarity for the product A recognized by the object recognition unit 611, and determines whether or not the similarity exceeds a predetermined threshold value in the PLU file F1.

Recognizing an object contained in an image in this way is called generic object recognition. Regarding such general object recognition, various recognition techniques are described in the following documents.
Keiji Yanai, “Current Status and Future of General Object Recognition”, IPSJ Journal, Vol. 48, no. SIG16 [Search August 10, 2010], Internet <URL: http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

Further, techniques for performing general object recognition by dividing an image into regions for each object are described in the following documents.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, [August 10, 2010 search], Internet <URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1. 1.145.3036 & rep = rep1 & type = pdf>

  The product registration unit 612 is information related to sales registration related to the product image recognized by the object recognition unit 611, that is, the product ID, product classification, product name, and product name of the product identified as the product read by the product reading device 101. Record unit price etc. in sales master file etc. and register sales.

  Next, the operation of the checkout system 1 will be described in detail. FIG. 5 is a flowchart showing an example of the operation of the checkout system 1 according to the present embodiment.

  First, the operation on the commodity reading apparatus 101 side will be described. As illustrated in FIG. 5, when processing is started in response to the start of product registration by the POS terminal 11, the captured image capturing unit 1611 outputs an imaging on signal to the imaging unit 164 and performs imaging by the imaging unit 164. Is started (step S1). Next, the captured image capturing unit 1611 captures the frame image (captured image) captured by the image capturing unit 164 and stored in the RAM 163 (step S2). Next, the product detection unit 1612 detects all or a part of all the products A in the paper box 153 from the frame image captured by the captured image capture unit 1611 (step S3). Next, the image output unit 1613 outputs to the POS terminal 11 a frame image in which all or part of all the products A in the paper box 153 are detected by the product detection unit 1612 (step S4).

  FIG. 6 is a conceptual diagram illustrating an example of the reading region R in the reading window 103. Specifically, FIG. 6 is a conceptual diagram illustrating the reading region R when the product A is read. As shown in FIG. 6, when the product A in the paper box 153 is reflected in the reading area R in the above-described movement process of the product A, all the contents in the paper box 153 are captured from the frame image obtained by capturing the reading area R in step S3. All or part of the product A is detected. By detecting all or part of all the products A in the paper box 153, a frame image obtained by imaging the reading region R is output to the POS terminal 11 in step S4.

  Next, the CPU 161 determines the presence / absence of business end from the POS terminal 11 by a product registration end notification or the like (step S5). When business is continued (step S5: No), the CPU 161 returns the process to step S2 to continue the process. When the work is to be ended (step S5: Yes), the captured image capturing unit 1611 outputs an imaging off signal to the imaging unit 164, ends the imaging by the imaging unit 164 (step S6), and ends the process.

  Next, the operation on the POS terminal 11 side will be described. As shown in FIG. 5, when processing is started in response to the start of product registration in response to an operation instruction on the keyboard 22, the CPU 61 outputs all products A in the paper box 153 output from the product reading device 101. Alternatively, a frame image in which a part is detected is received (step S11). Next, the object recognizing unit 611 refers to the product image of the PLU file F1, and selects the product A as a specific object from all or some images of all the products A in the paper box 153 output from the product reading device 101. Recognize in order (step S12).

  Here, as a result of the recognition in step S12, a plurality of candidates for the product A may appear. Therefore, when there are a plurality of candidates for the product A (step S13: Yes), the CPU 61 notifies the store clerk by screen display on the display device 23 as shown in FIG. 7 (step S14). FIG. 7 is a conceptual diagram showing a display example by the display device 23. As shown in FIG. 7, on the display screen of the display device 23, product names and unit prices of a plurality of product A candidates as a result of recognition in step S12 are acquired from the PLU file F1 and displayed. In the example shown in FIG. 7, “plain donut” and “sugar donut” are displayed.

  Thereafter, when the CPU 61 determines that the operator (clerk) has selected the correct candidate as the product A via the touch panel 26 (step S15: Yes), the CPU 61 proceeds to step S16.

  In addition, when there are not a plurality of candidates for the product A (step S13: No), the process directly proceeds to step S16.

  Next, the similarity determination unit 613 calculates the similarity for the product A recognized by the object recognition unit 611 (step S16). As for the similarity, when the product image of each product stored in the PLU file F1 is 100% = “similarity: 1.0”, all or a part of the image of the product A output from the product reading apparatus 101 is displayed. It shows how similar they are. As described above, the degree of similarity is calculated according to the state of the surface such as the hue and the surface roughness. For example, the weight may be changed according to the color tone and the surface unevenness.

  Next, the similarity determination unit 613 determines whether or not the similarity calculated in step S16 exceeds a predetermined threshold (“similarity: 0.XX”) (step S17).

  When the similarity determination unit 613 determines that the similarity calculated in step S16 exceeds a predetermined threshold (“similarity: 0.XX”) (step S17: Yes), in step S18, the product The registration unit 612 registers the recognition result of the product A, which is a specific object, and the sales of the product A that is selected from the recognition result and specified as one.

  Next, the CPU 61 determines whether or not all objects have been recognized (step S19). If the CPU 61 determines that all objects have been recognized (step S19: Yes), the process proceeds to step S20. If it is determined that all objects have not been recognized (step S19: No), the process proceeds to step S12. To continue the recognition process for the next object.

  In step S20, the problem solving unit 616 notifies the operator (clerk) that the product A has been recognized by displaying the screen on the display device 23 as shown in FIG. In the example illustrated in FIG. 8, the problem solving unit 616 displays an image of the product A in the paper box 153, and also displays a frame line ( The first line (Y) is also displayed.

  However, as shown in FIG. 9, there are cases where unrecognized products are generated due to the influence of the color of the paper box 153 (tray color), the shadow of the operator, etc., even though all objects have been recognized. is there. In such a case, it is necessary to perform recognition processing again by designating an image area that does not rise as a candidate.

  In addition, as shown in FIG. 10, there may be a case where another is originally recognized as one. In such a case, it is necessary to perform recognition processing again by designating an area that has been combined and recognized. Note that in the example shown in FIG. 10, since another object is originally recognized as one, it is recognized as a completely different object (jam pan).

  Furthermore, as shown in FIG. 11, a plurality of identical products may be recognized as one because they overlap each other. In such a case, it is necessary to re-input only the number.

  Therefore, the problem solving unit 616 includes an instruction unit 30 and a determination key 40 for solving the various problems described above in the screen display on the display device 23. As shown in FIGS. 8 to 12, the instruction unit 30 displayed on the display device 23 includes a division instruction key 31, an addition instruction key 32, a number addition instruction key 33, and a deletion key 35.

  As shown in FIG. 9, even though all objects have been recognized, if an unrecognized product is generated due to the color of the paper box 153 (tray color) or the shadow of the operator, the operator The (clerk) presses the additional instruction key 32 via the touch panel 26. When the problem solving unit 616 determines that the operator (clerk) has pressed the add instruction key 32 via the touch panel 26, the problem solving unit 616 causes the operator (clerk) to designate an image area that does not rise as a candidate. It should be noted that the method of declaring the additional implementation of recognition when a product for which recognition has not been performed occurs is not limited to the method of pressing the additional instruction key 32 displayed on the display device 23 via the touch panel 26. Alternatively, for example, an additional instruction key may be provided on the keyboard 22 and the addition of recognition may be declared by pressing the additional instruction key.

Here, a method for causing the operator (clerk) to designate an image area that does not rise as a candidate will be described. FIG. 13 is a conceptual diagram showing a method for allowing an operator (store clerk) to specify an image area that does not rise as a candidate. As shown in FIG. 13, as a method for designating an image area that does not rise as a candidate through the touch panel 26, there is a method as described below.
(A) Method of designating the center of the area (see (a) of FIG. 13)
(B) Method of designating by the diagonal of the area (see (b) of FIG. 13)
(C) Method of specifying by enclosing with circle (spline) (see (c) of FIG. 13)

As shown in FIG. 13B, in the method of designating by the diagonal of the area, the problem solving unit 616 has a frame line whose aspect is different from the frame line (first line) Y indicating that it has been recognized. (Second line) Z1 designates an image area that does not rise as a candidate.
Further, as shown in FIG. 13C, in the method of specifying by enclosing with circles (splines), the problem solving unit 616 has a different aspect from the frame line (first line) Y indicating that it is recognized. An image area that does not rise as a candidate is designated by the frame line (second line) Z2.

  The problem solving unit 616 causes the operator (store clerk) to designate an image area that does not rise as a candidate by any of the methods described above, and then the operator (store clerk) changes the display device 23 as shown in FIG. If it is determined that the displayed re-recognition key 34 has been pressed, it is assumed that there has been an additional instruction (step S21: Yes), and the process returns to step S12 to execute recognition processing for an object in the image area that does not rise as a candidate. .

  Note that the method of causing the operator (store clerk) to specify an image area that does not rise as a candidate is not limited to the method of specifying via the touch panel 26, but may be a voice instruction (up / down / left / right), An unanalyzed area may be designated with the keyboard 22. For example, as an example of designating an unanalyzed area with the keyboard 22, as shown in FIG. 14, the screen displayed on the display device 23 is divided into nine parts and displayed as an overlay, and an approximate area is displayed by an operator with 1-9 of the numeric keypad 22d. Let the (clerk) specify. It should be noted that, including keys other than the numeric keypad 22d, keys corresponding to screen division (4 × 4 16 divisions, 5 × 5 25 divisions) may be displayed in an overlay manner.

  Further, as shown in FIG. 10, when a case where another one is originally recognized is generated, the operator (clerk) presses the division instruction key 31 via the touch panel 26. If the problem solving unit 616 determines that the operator (clerk) has pressed the division instruction key 31 via the touch panel 26, the problem solving unit 616 causes the operator (clerk) to divide the image area via the touch panel 26. As a method for dividing the image area by the operator (clerk), the area is divided by a straight line (spline) via the touch panel 26 as shown in FIG.

  The problem solving unit 616 divides the image area into operators (clerks), and then the operator (clerk) presses the re-recognition key 34 displayed on the display device 23 as shown in FIG. If it is determined that there is an instruction for division (step S21: Yes), the process returns to step S12 to execute recognition processing of the object in each divided image area.

  Also, as shown in FIG. 11, when a plurality of identical products are recognized as being overlapped, the operator (clerk) presses the number addition instruction key 33 via the touch panel 26. If the problem solving unit 616 determines that the operator (clerk) has pressed the number addition instruction key 33 via the touch panel 26, the problem solving unit 616 causes the operator (clerk) to input the number from the numeric keypad 22d of the keyboard 22.

  In addition, as shown in FIG. 12, when all the objects have been recognized and the product is requested to be returned after the product is recognized, the operator (clerk) presses the delete key 35 via the touch panel 26. To do. When the problem solving unit 616 determines that the operator (clerk) has pressed the delete key 35 via the touch panel 26, the problem solving unit 616 causes the operator (clerk) to specify a product to be returned (deleted).

As a method for causing the operator (store clerk) to specify a product to be returned (deleted), a method similar to the method for causing the operator (store clerk) to designate an image area that does not appear as a candidate as described above can be considered. FIG. 16 is a conceptual diagram showing a method for causing an operator (store clerk) to specify a product to be returned (deleted). As shown in FIG. 16, as a method for designating a product to be returned (deleted) via the touch panel 26, there is a method as described below.
(A) A method of directly specifying a product to be returned (see (a) of FIG. 16)
(B) A method of designating a product to be returned by the diagonal of the area (see (b) of FIG. 16)
(C) A method for specifying a product to be returned by surrounding it with a circle (spline) (see (c) of FIG. 16).

  Note that the method of causing the operator (clerk) to specify the product to be returned (deleted) is not limited to the method of specifying via the touch panel 26, and may be a method of instructing by voice (up / down / left / right). The unanalyzed area may be designated with the keyboard 22. For example, as an example of designating an unanalyzed area with the keyboard 22, as described with reference to FIG. 14, the screen displayed on the display device 23 is divided into nine parts and displayed as an overlay, and an approximate area is displayed with 1-9 of the numeric keypad 22d. Let the operator (clerk) specify it. It should be noted that, including keys other than the numeric keypad 22d, keys corresponding to screen division (4 × 4 16 divisions, 5 × 5 25 divisions) may be displayed in an overlay manner.

  The problem solving unit 616 causes the operator (store clerk) to specify the item to be returned (deleted), and then the operator (store clerk) displays the re-recognition key displayed on the display device 23 as shown in FIG. If it is determined that 34 is pressed, it is assumed that there is a return (deletion) instruction (step S22: Yes), the process returns to step S18, and the sales registration of the product A whose number is reduced is executed in the product registration unit 612. Let

  If the problem solving unit 616 determines that the operator (clerk) has input the number via the numeric keypad 22d or the like of the keyboard 22 (step S23: Yes), the process returns to step S18, and the number increases in the product registration unit 612. The sales registration of the product A that has been reduced or decreased is executed.

  On the other hand, after notifying the operator (clerk) that the product A has been recognized by displaying the screen on the display device 23 as shown in FIG. 8, there is no problem as shown in FIGS. The operator (store clerk) depresses the confirmation key 40 via the touch panel 26. If the problem solving unit 616 determines that the operator (clerk) has pressed the confirmation key 40 via the touch panel 26 (step S24: Yes), the process proceeds to step S25.

  Thus, according to the checkout system 1 of the present embodiment, even when there is a problem in object recognition, the problem can be easily solved.

  In step S25, the confirmation image display unit 615 causes the customer to confirm the purchased product by the screen display on the customer display device 24 as shown in FIG. In the example shown in FIG. 17, the image of the paper box 153 packed with the product A is displayed in the display area G1, and the product information such as the product name and unit price of the product A stored in advance, the number, the total price, etc. The history of purchased products is also displayed in the display area G2. The product name of the product A displayed in the display area G1 is also appended. The CPU 61 of the POS terminal 11 displays the purchase product history in the display area G <b> 2 based on information related to the purchase product stacked in the RAM 63. With such a screen display, it can be confirmed to the customer whether there is a mistake with the desired product. Conventionally, in pastry shops, donut shops, etc., a paper box 153 packed with product A is presented to the customer through the display case 151 to confirm that there is no error, and then registration and closing operations for sales transactions, etc. In this system, the purchased product is displayed on the product reader 101 and information such as the product name and price of all products is read in a batch, so the customer completes the POS operation in a short time after confirmation. It becomes possible to do.

  Here, the screen display on the customer display device 24 as shown in FIG. 17 is different from the read image shown in FIG. The screen display on the customer display device 24 as shown in FIG. 17 is the state seen from the customer side during face-to-face sales, and the read image shown in FIG. 6 is the state seen from the operator (clerk) side. In other words, the confirmation image display unit 615 displays the image as it is viewed from the customer side by inverting the read image vertically and horizontally. The confirmation image display unit 615 also displays captions such as product name candidates to be displayed over the image in a direction that can be read in accordance with the inverted image.

  Next, the CPU 61 determines whether or not there is an end of business due to the end of sales registration by an operation instruction of the keyboard 22 (step S26). When the business is to be ended (step S26: Yes), the CPU 61 ends the screen display on the customer display device 24 and ends the process.

  On the other hand, when the similarity determination unit 613 determines that the similarity calculated in step S16 does not exceed the predetermined threshold (“similarity: 0.XX”) (step S17: No), the defective product notification The unit 614 notifies the store clerk that the product is not recognized as a regular product (step S27), and prompts the store side to take measures such as sales stop or discount sales. Specifically, the defective product notification unit 614 displays a screen on the display device 106 via the display device 23 and the connection interfaces 65 and 176, and outputs a warning sound by the audio output unit 165 via the connection interfaces 65 and 175. Inform the store clerk about the error. By performing this notification, the checkout system 1 can prompt the store clerk to perform an appropriate operation such as reading another product A.

  If the CPU 61 determines that there has been an instruction for discount sales based on an operation instruction on the keyboard 22 (step S28: Yes), in step S18, the product registration unit 612 causes the recognition result of the product A, which is a specific object, The sales of the product A selected from the recognition results and specified as one is discounted and registered. If it is determined that there has been an instruction to cancel the sale by operating the keyboard 22 or the like (step S28: No), the process returns to step S12.

  As shown in FIG. 18, when the product A recognized by the object recognizing unit 611 is “plain donut”, the threshold is stored in the PLU file F1 as “similarity: 0.50”, which is indicated by (a). Since the degree of similarity is 0.717 for the product, it is determined that the calculated degree of similarity exceeds a predetermined threshold. On the other hand, for the product shown in (b), since the similarity is 0.252, it is determined that the calculated similarity does not exceed a predetermined threshold. Accordingly, the product shown in (a) is registered as “plain donut” as usual. On the other hand, regarding the product shown in (b), although it is temporarily recognized as "plain donut", it has become old, discolored, damaged, deformed, that is, the color and surface irregularities are not normal, Sales are not registered as defective. In particular, in the case of donuts, breads, etc., if a product image based on the normal baking condition or fried condition is stored in the PLU file F1, and if the baking shortage / overheating or shortage / overage falls outside the threshold, it is a defective product. It can be judged.

  As described above, according to the checkout system 1 of the present embodiment, when a donut or bread is old, discolored, damaged, deformed, or has a different color due to poor cooking, the PLU file By comparing with the product image stored in F1, it is possible to determine that a product out of the standard is a defective product.

  Compare with regular products by numerical judgment, especially in busy stores such as stores that have limited time to check the freshness of products and stores that have many store staff and individual differences in cooking donuts and bread. Is possible.

  In addition, according to the checkout system 1 of the present embodiment as described above, the product information such as the product name and unit price can be confirmed together with the product image, so that mistakes can be reduced. . In addition, since the products on the tray and the products in the paper box 153 can be read at once and the product information can be called at once, it is possible to conduct transactions related to buying and selling in a short time.

  Furthermore, according to the checkout system 1 of the present embodiment as described above, in a store where there are many similar products and a plurality of candidates for the product A appear, all the candidates for the product A are displayed and selected. It is possible to prevent errors.

  In the present embodiment, the object to be recognized is a sales product stored in the PLU file F1, but is not limited to this, and may be a sales promotion product used for sales promotion.

  Programs executed by the POS terminal 11 and the commodity reading apparatus 101 according to the present embodiment are provided by being incorporated in advance in a ROM or the like. The programs executed by the POS terminal 11 and the product reading apparatus 101 according to the present embodiment are files that can be installed or executed, and are CD-ROM, flexible disk (FD), CD-R, and DVD (Digital Versatile Disk). ) Or the like may be recorded and provided on a computer-readable recording medium.

  Further, the program executed by the POS terminal 11 and the commodity reading apparatus 101 according to the present embodiment may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. good. Further, the program executed by the POS terminal 11 and the commodity reading apparatus 101 according to the present embodiment may be configured to be provided or distributed via a network such as the Internet.

  The program executed by the POS terminal 11 of the present embodiment includes the above-described units (the object recognition unit 611, the product registration unit 612, the similarity determination unit 613, the defective product notification unit 614, the confirmation image display unit 615, and the problem solving unit 616). ), And the actual hardware is loaded by the CPU (processor) by reading the program from the ROM and executing the program on the main memory, the object recognition unit 611, and the product registration A unit 612, a similarity determination unit 613, a defective product notification unit 614, a confirmation image display unit 615, and a problem solving unit 616 are generated on the main storage device.

  The program executed by the product reading apparatus 101 according to the present embodiment has a module configuration including the above-described units (the captured image capturing unit 1611, the product detection unit 1612, and the image output unit 1613). The CPU (processor) reads the program from the ROM and executes it, so that the above-described units are loaded onto the main storage device, and the captured image capturing unit 1611, the product detection unit 1612, and the image output unit 1613 are loaded onto the main storage device. It is to be generated.

  In the present embodiment, an object recognition unit 611, a product registration unit 612, a similarity determination unit 613, a defective product notification unit 614, a confirmation image display unit 615, and a problem solving unit 616 are provided on the POS terminal 11 side. However, the present invention is not limited to this, and the product recognition device 611, the product registration unit 612, the similarity determination unit 613, the defective product notification unit 614, the confirmation image display unit 615, and the problem solving unit 616 are provided on the product reading apparatus 101 side. Anyway. In this case, the commodity reading apparatus 101 is a store system.

  Further, in the present embodiment, the captured image capturing unit 1611, the product detection unit 1612, and the image output unit 1613 are provided on the product reading apparatus 101 side. However, the present invention is not limited to this, and the captured image is captured on the POS terminal 11 side. An image capturing unit 1611, a product detection unit 1612, and an image output unit 1613 may be provided. In this case, the POS terminal 11 is a store system.

  Further, the product reading device 101 side includes a captured image capturing unit 1611, a product detection unit 1612, and an image output unit 1613, and sales registration as performed in the POS terminal 11 may be performed on the product reading device 101 side. Good.

DESCRIPTION OF SYMBOLS 1 Store system 22 Input part, Keyboard 24 Display part 26 Input part, Touch panel 164 Imaging means 611 Object recognition means 615 Confirmation image display means 616 Problem solving means 1613 Image output means

Japanese Patent Laid-Open No. 06-223271

The store system according to the embodiment includes an image output unit that outputs an image captured by the imaging unit, and an object recognition unit that recognizes a specific object by reading a feature amount of the output image. The store system according to the embodiment includes a confirmation image display unit that displays at least an image of the recognized object on the display unit, and an object to be returned when there is a problem in which a return is offered after the object is recognized. accepting an indication, and a problem solving means for solving the problems in accordance with instruction contents accepted.

The store system according to the embodiment includes an imaging unit, a display unit, an image output unit that outputs an image captured by the imaging unit, and an object recognition that recognizes a specific object by reading a feature amount of the output image Means, and at least a confirmation image display means for displaying an image of the recognized object on the display unit, and if there is a problem that a return can be offered after the object is recognized, an instruction indicating the object to be returned is received. , and a problem solving means for solving the problems in accordance with instruction contents accepted.

The program according to the embodiment causes a computer to function as an object recognition unit that recognizes a specific object by reading a feature amount of an image captured by the imaging unit. The program of the embodiment, computer, if there is at least the image and check the image display means for displaying on the display unit of the recognized object; problem to be offered to return after recognition of the object, to return It receives an instruction indicating an object, to function as a problem solving means for solving the problems in accordance with instruction contents accepted.

Note that the method of causing the operator (clerk) to specify the product to be returned (deleted) is not limited to the method of specifying via the touch panel 26, and may be a method of instructing by voice (up / down / left / right). The area where the product to be returned (deleted) is present may be specified using the keyboard 22. For example, as an example of designating an area where a product to be returned (deleted) exists with the keyboard 22, as shown in FIG. 14, the screen displayed on the display device 23 is divided into nine and displayed in an overlay, and the numeric keypad 22d is displayed. In 1-9, the operator (store clerk) designates an approximate area. It should be noted that, including keys other than the numeric keypad 22d, keys corresponding to screen division (4 × 4 16 divisions, 5 × 5 25 divisions) may be displayed in an overlay manner.

Claims (11)

Image output means for outputting an image captured by the imaging means;
Object recognition means for recognizing a specific object by reading the feature amount of the output image;
Confirmation image display means for displaying at least an image of the recognized object on the display unit;
If there is a problem in the recognition of the object, a problem solving means for receiving an instruction indicating the problem and solving the problem according to the received instruction content;
A store system comprising: When there is an object that has not been recognized, the problem solving means receives an instruction to designate an area where an object that has not been recognized exists, and causes the object recognition means to execute object recognition for the area. ,
The store system according to claim 1. The problem solving means receives an instruction to divide an area where another object is originally recognized as one object when another object is originally recognized as one object, and the object for each divided area Causing the object recognition means to perform recognition,
The store system according to claim 1. The problem solving means accepts the number of overlapping objects when a plurality of identical objects are recognized as being overlapped,
The store system according to claim 1. It further includes an input unit that receives input from the user,
The problem solving means receives an instruction indicating the problem when there is a problem in the recognition of the object via the input unit, and responds to the instruction content via the input unit for the image displayed on the display unit To solve the problem,
The store system according to claim 1. The input unit is a touch panel laminated on the display unit,
The problem solving means receives an instruction indicating the problem when there is a problem in the recognition of the object via the touch panel, and according to the instruction content via the touch panel for the image displayed on the display unit Solve a problem,
The store system according to claim 5. The input unit is a keyboard having various keys,
The problem solving means receives an instruction indicating the problem when there is a problem in recognizing the object through an operation on a key of the keyboard corresponding to various processes for solving the problem, and displays on the display unit Solving the problem according to the instruction content corresponding to the key operation of the keyboard on the displayed image;
The store system according to claim 5. In the case where the instruction for the image displayed on the display unit is area specification, the problem solving means is any of the specification of the center of the area, the specification of the area diagonally, and the specification by surrounding with a spline. Accept instructions for the image displayed on the display unit,
The store system according to claim 5. The problem solving means displays the recognized object with a first line, and displays the image displayed on the display unit as a process for solving the problem when the object is recognized. A second line for instructing the area designation for is displayed in a different form from the first line,
The store system according to claim 1. An imaging unit;
A display unit;
Image output means for outputting an image captured by the imaging unit;
Object recognition means for recognizing a specific object by reading the feature amount of the output image;
Confirmation image display means for displaying at least an image of the recognized object on the display unit;
If there is a problem in the recognition of the object, a problem solving means for receiving an instruction indicating the problem and solving the problem according to the received instruction content;
A store system comprising: Computer
Object recognition means for recognizing a specific object by reading the feature quantity of the image captured by the imaging means;
Confirmation image display means for displaying at least an image of the recognized object on the display unit;
If there is a problem in the recognition of the object, a problem solving means for receiving an instruction indicating the problem and solving the problem according to the received instruction content;
A program characterized by functioning as

Images