JP5784170B2 - Store accounting system and store accounting program - Google Patents

Description

  Embodiments of the present invention relate to a store accounting system using an imaging unit and a store accounting program for causing a computer to function as the store accounting system.

  There is a merchandise reading apparatus that recognizes merchandise from an image taken by an imaging means such as a CCD camera. Such a commodity reading apparatus includes an imaging unit in the apparatus and a reading window in a housing of the apparatus. Then, the imaging means sets an imaging area from the reading window through the imaging lens. For this reason, for example, when a product is recognized using such a product reading device, the operator holds the product over the reading window of the device so that the entire appearance of the product is imaged by the imaging means.

  An operator who is familiar with the operation of the product reading apparatus is not so burdened to hold the product over the reading window so that the entire appearance of the product is imaged by the imaging means. However, some unfamiliar operators may hide the product by hand, and the product may not be recognized.

JP 2011-165139 A

  The problem to be solved by the present invention is to provide a store accounting system and a store accounting program capable of notifying a user that a product is in a state where it cannot be recognized because it is hidden by hand, for example.

In one embodiment, the store accounting system includes a reading unit, a display unit, a first storage unit, a second storage unit, a recognition unit, a display unit, and a notification unit. The first storage means stores data representing the appearance of the product to be sales processed. The second storage means stores data representing the appearance of the hand. The recognizing unit recognizes the object held over the reading unit based on the data stored in the first and second storage units. When the display unit recognizes that the object is the product to be sold, the display unit displays information on the product on the display unit. When recognizing that the hand is held over the reading unit, the notifying unit notifies that the product is not recognized without displaying the information on the product to be sold on the display unit.

1 is an external view of a store accounting system according to an embodiment. The block diagram which shows the hardware constitutions of the goods reading apparatus and POS terminal which are integrated in the same system. The schematic diagram which shows the data structure of the goods data file used with the system. The schematic diagram which shows the data structure of the foreign material recognition dictionary file used with the system. The schematic diagram which shows the function structure which CPU implement | achieves in the goods reading apparatus of the system. The flowchart which shows the principal part procedure of the information processing which CPU performs according to a goods reading program in the goods reading apparatus of the system. The top view which shows an example of the image recognition screen displayed on a touchscreen in the goods reading apparatus of the same system. The figure which shows the example of a display of the image recognition screen when the goods are concealed with the hand in the goods reading apparatus of the system. The figure which shows the example of a display when a hand is recognized as a foreign material in the state shown in FIG.

  Hereinafter, an embodiment of a store accounting system will be described with reference to the drawings. This embodiment is a case where the present invention is applied to a store accounting system including a vertical product reading device standing on an accounting counter provided in a store such as a supermarket.

  FIG. 1 is an external view of a store accounting system according to an embodiment. This system includes a commodity reading device 1 as a registration unit and a POS (Point Of Sales) terminal 2 as a settlement unit. The commodity reading device 1 is attached on the accounting counter 3. The POS terminal 2 is installed on the cash register 4 via the drawer 5. The product reading device 1 and the POS terminal 2 are electrically connected by a communication cable (not shown).

  The product reading apparatus 1 includes a keyboard 11, a touch panel 12, and a customer display 13 as devices necessary for product registration. These display / operation devices (keyboard 11, touch panel 12, customer display 13) are attached to a thin rectangular housing 1 </ b> A constituting the main body of the commodity reading apparatus 1.

  An imaging unit 14 is built in the housing 1A. A rectangular reading window 1B is formed in front of the housing 1A. The imaging unit 14 includes a CCD (Charge Coupled Device) imaging device that is an area image sensor and its drive circuit, and an imaging lens for forming an image of the imaging region on the CCD imaging device. The imaging region refers to a region of a frame image that forms an image on the area of the CCD imaging device from the reading window 1B through the imaging lens. The imaging unit 14 outputs an image of the imaging region formed on the CCD imaging device through the imaging lens. Here, the imaging unit 14 functions as an imaging unit that captures an image including a product held over the reading window 1B.

  The POS terminal 2 includes a keyboard 21, an operator display 22, a customer display 23, and a receipt printer 24 as devices necessary for payment.

  The checkout counter 3 has an elongated shape along the customer passage on the back side. The cash register 4 is placed substantially perpendicular to the transaction counter 3 on the front side of the end of the transaction counter 3 on the downstream side with respect to the moving direction of the customer moving along the transaction counter 3. The front side of the accounting counter 3 and the front side of the cash register 4 serve as a space for so-called cashiers in charge of accounting.

  In the approximate center of the accounting counter 3, the housing 1A of the commodity reading apparatus 1 is erected with the keyboard 11, the touch panel 12, and the reading window 1B facing the cashier side on the front side. The customer display 13 of the commodity reading apparatus 1 is attached to the housing 1A so as to face the customer passage.

  The receiving surface on the upstream side of the customer movement direction across the commodity reading device 1 of the accounting counter 3 is a space for placing a shopping basket 6 in which unregistered commodity M to be purchased by the shopper is placed. In addition, the downstream receiving surface is a space for placing a shopping basket 7 into which the product M registered by the product reading device 1 is placed.

  FIG. 2 is a block diagram illustrating a hardware configuration of the commodity reading device 1 and the POS terminal 2. The commodity reading apparatus 1 includes a CPU (Central Processing Unit) 101 as a control unit main body. The merchandise reading apparatus 1 connects a ROM 103 and a RAM 104 to the CPU 101 via a bus line 102 such as an address bus or a data bus. The ROM 103 stores a program executed by the CPU 101. The RAM 104 stores a foreign object recognition dictionary file 9 described later.

  The commodity reading apparatus 1 includes the imaging unit 14 and the audio output unit 105. The imaging unit 14 and the audio output unit 105 are also connected to the CPU 101 via the bus line 102, respectively. The voice output unit 105 functions as a voice output unit that outputs voice guidance.

  Further, a keyboard 11, a touch panel 12 and a customer display 13 which are display / operation devices are connected to the bus line 102 via a connection interface 106 and a connection interface 107. The touch panel 12 includes a panel display unit 121 using, for example, a liquid crystal display, and a touch panel sensor 122 arranged on the screen of the panel display unit 121.

  The POS terminal 2 also includes a CPU 201 as a control unit main body. The POS terminal 2 connects a ROM 203, a RAM 204, an HDD (Hard Disk Drive) device 205, a communication interface 206, and a connection interface 207 to the CPU 201 via the bus line 202. The keyboard 21, operator display 22, customer display 23, printer 24, and drawer 5 are also connected to the CPU 201 via the bus line 202.

  The communication interface 206 is connected to a store computer SC serving as the center of the store via a network such as a LAN (Local Area Network). With this connection, the POS terminal 2 can transmit / receive data to / from the store computer SC. For example, the POS terminal 2 receives the product data file 8 from the store computer SC. The received product data file 8 is stored in the HDD device 205.

  The connection interface 207 is connected to both connection interfaces 106 and 107 of the commodity reading apparatus 1 via a communication cable. With this connection, the POS terminal 2 can transmit and receive data to and from the commodity reading device 1. For example, the POS terminal 2 receives product information read by the product reading device 1. The POS terminal 2 transmits the data in the product data file 8 to the product reading device 1.

  FIG. 3 is a schematic diagram showing the data structure of the product data file 8. As shown in FIG. 3, the product data file 8 stores the product attribute (product classification), name (product name), unit price, appearance feature parameter, etc. in association with the product code for identifying each product. . The appearance feature parameter is obtained by quantifying the standard appearance feature amount of a product such as the shape, surface color, pattern, unevenness, etc. of the product for each product.

  FIG. 4 is a schematic diagram showing the data structure of the foreign object recognition dictionary file 9. As shown in FIG. 4, the foreign object recognition dictionary file 9 stores appearance characteristic parameters of various foreign objects and guidance data corresponding to the respective foreign objects. Here, the foreign object is a product other than a product that is held by the reading window 1B that is a reading unit and is not processed by the POS terminal 2 that is a settlement unit even when captured by the imaging unit 14 that is an imaging unit. Typical examples of the object include an operator's hand when the product is held over the reading window 1B, and a packaging container such as a tray or a net for packaging the product. On the other hand, the guidance data is data that is output as a voice when a corresponding foreign object is recognized. For example, guidance data such as “hand is recognized” is set for a hand that is a foreign object. .

  In the present embodiment, external feature quantities such as the shape of the left and right hand backs and fingers of each cashier that can be an operator of the commodity reading device 1, color, unevenness, and the like are quantified, and the foreign object recognition dictionary is used as foreign object appearance feature data. Saved in file 9. The foreign object recognition dictionary file 9 is created in advance by the store computer SC and stored in the RAM 104 of the commodity reading apparatus 1 via the POS terminal 2. Here, even if the foreign object recognition dictionary file 9 is held over the reading window 1B (reading unit) and imaged by the imaging unit 14 (imaging means), the POS terminal 2 (settlement unit) does not perform sales processing. It functions as storage means for storing data representing the appearance of objects other than products.

  As shown in FIG. 5, the CPU 101 of the commodity reading apparatus 1 recognizes the object held over the reading window 1B from the image picked up by the image pickup unit 14, and the recognizing means 31 recognizes the object. If the detected object is a product for which sales processing is performed at the POS terminal 2 (settlement unit), the output unit 32 that outputs information on the product to the POS terminal 2 and the object recognized by the recognition unit 31 are sold at the POS terminal 2. Informing means 33 for informing that the object is an object other than an unprocessed product. The CPU 101 further includes a determination unit 34 that determines whether the object recognized by the recognition unit 31 is a product that is sales processed at the POS terminal 2 or an object other than a product that is not sales processed. When the determination means 34 determines that an object other than the product has been recognized, a notification is made. Further, the CPU 101 includes a display unit 35 that displays an image captured by the imaging unit on the touch panel 12, and the notification unit 33 indicates an object other than the product recognized by the recognition unit 31 in the image displayed on the touch panel 12. I will inform you. In addition, the notification unit 33 displays an object (foreign matter) other than the product recognized by the recognition unit 31 in the image displayed on the touch panel 12, and then from the voice output unit 105 to the foreign matter recognition dictionary after a predetermined time has elapsed. The voice guidance of guidance data set corresponding to the foreign object in the file 9 is output.

  Each function of the CPU 101 will be described with reference to the flowchart of FIG. This flowchart shows the main procedure of the information processing executed by the CPU 101 in accordance with the product reading program.

  This process is started, for example, when a command for instructing the start of product registration is input from the POS terminal 2 via the connection interface 106. First, the CPU 101 displays the image recognition screen 50 on the panel display unit 121 of the touch panel 12 (ST1).

  An example of the image recognition screen 50 is shown in FIG. As shown in the figure, the image recognition screen 50 includes an image display region R and is displayed in a pop-up manner on the registration screen 60. The registration screen 60 has a current area 61 and a detail area 62. The current area 61 is an area for displaying the product name, unit price, quantity, amount, etc. of the latest registered product. The detail area 62 is an area for displaying the product name, unit price, quantity, amount, etc. of each product registered in one commercial transaction.

  Next, the CPU 101 outputs an imaging on signal to the imaging unit 14 (ST2). In response to this imaging on signal, the imaging unit 14 starts imaging of the imaging region. Frame images of the imaging area captured by the imaging unit 14 are sequentially stored in the RAM 104.

  CPU 101 captures the frame image stored in RAM 104 (ST3). Then, the CPU 101 displays this frame image in the image display area R of the image recognition screen 50 (ST4).

  Next, the CPU 101 recognizes an object included in the image from the frame image displayed in the image display region R (ST5). Specifically, the CPU 101 first extracts a contour line or the like from an image obtained by binarizing the frame image. Thereby, the CPU 101 tries to extract the contour of the object displayed in the frame image. When the contour of the object is extracted, the CPU 101 reads feature amounts such as the shape of the object, the color of the surface, the pattern, and the uneven state from the image within the contour. Then, the CPU 101 collates the read feature data with each appearance feature parameter stored in the foreign object recognition dictionary file 9 to determine whether or not an appearance feature parameter similar to the feature data is detected ( ST6).

  As a result, when an appearance feature parameter similar to the feature amount data stored in foreign object recognition dictionary file 9 is not detected, CPU 101 recognizes that product M has been detected (YES in ST6). On the other hand, when an appearance feature parameter similar to the feature amount data stored in the foreign object recognition dictionary file 9 is detected, the CPU 101 recognizes that a foreign object N other than a product has been detected (NO in ST6). .

Such a technique for recognizing an object included in an image is called generic object recognition. Regarding this general object recognition technique, various recognition techniques are described in the following documents.
Keiji Yanai, “Current Status and Future of General Object Recognition”, IPSJ Journal, Vol. 48, no. SIG16 [Search August 10, 2010], Internet <URL: http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>
Further, a technique for performing general object recognition by dividing an image into regions for each object is described in the following document.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, [Search August 10, 2010], Internet <URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1. 1.145.3036 & rep = repl & type = pdf>
The CPU 101 repeats the recognition process of the frame image displayed on the image recognition screen 50 until the product M or the foreign object N other than the product is recognized.

  For example, as shown in FIG. 7, when the operator holds the product M over the reading window 1B so that the contour of the product M is extracted, the contour of the product M is extracted from the frame image and the product M is recognized. When product M is recognized (YES in ST6), CPU 101 outputs feature amount data of recognized product M to POS terminal 2 (ST7). At this time, the CPU 101 displays a frame in the image display region R so as to surround the object (product M) recognized from the frame image.

  When the CPU 201 of the POS terminal 2 inputs the feature value data of the product M from the product reading device 1, the feature value data is collated with the appearance characteristic parameters for each product stored in the product data file 8. Then, the product having the appearance feature parameter most similar to the feature amount of the product M is determined as the product M.

  When the product M is determined, the CPU 201 reads the product code, product name, unit price, etc. of the product M from the product data file 8. In addition, the sales amount is calculated by setting the number of sales points to “1”. The product sales data including the product code, the product name, the unit price, the number of sales, the sales price, etc. is processed in the RAM 204. The product sales data is output to the product reading device 1 via the connection interface 207 as confirmed product information.

  CPU101 which output the feature-value data of the goods M to the POS terminal 2 in step ST7 waits for fixed goods information (ST8). If the confirmed product information is received from the POS terminal 2 via the connection interface 106 (YES in ST8), the CPU 101 displays the product code, the product name, the unit price of the confirmed product information in the current area of the registration screen 60, The number of sales points, the sales amount, etc. are displayed (ST9).

  Thereafter, CPU 101 determines whether or not a command for instructing the end of product registration has been input (ST10). When the end command is not input (NO in ST10), CPU 101 captures the next frame image from RAM 104 (ST3).

  The termination command command is output from the POS terminal 2 to the commodity reading device 1 in response to an input operation of, for example, the [Total] key arranged on the keyboard 11. That is, the CPU 201 of the POS terminal 2 outputs an end command to the product reading device 1 when the key signal of the [Total] key is input from the product reading device 1. Further, based on the product sales data registered in the RAM 204, a registration closing process for one commercial transaction is executed.

  When an end command is input via connection interface 106 (YES in ST10), CPU 101 outputs an imaging off signal to imaging unit 14 (ST11). Further, the CPU 101 deletes the image recognition screen 50 from the panel display unit 121 (ST12). In response to the imaging off signal, the imaging unit 14 ends imaging of the imaging region.

  On the other hand, as shown in FIG. 8, for example, when the operator hides the product M with the back and fingers of the hand and holds it over the reading window 1B, the contour of the product M is not extracted, but the contour of the hand is extracted. For this reason, the product M is not recognized, and the operator's hand is recognized as a foreign object N other than the product. When foreign object N other than the product is recognized (NO in ST6), CPU 101 causes image display region R to have a rectangular frame 51 so as to surround the hand recognized as foreign object N, as shown in FIG. (ST13). The frame 51 at this time is made different in color, shape, and the like from the frame for enclosing the recognized product M, for example.

  In addition, the CPU 101 displays an image of the confirmation button 52 on the image recognition screen 50. Then, it waits for the touch operation of the image of the confirmation button 52 (ST14). During this standby period, the CPU 101 determines whether or not a predetermined time (for example, 5 seconds) has elapsed (ST15). If the predetermined time has elapsed without touching the confirmation button 52 image (NO in ST14) (YES in ST15), the CPU 101 utters a predetermined voice guidance, for example, “Please hold the product”. Output from the output unit 105 (ST16). Thereafter, the CPU 101 waits for the touch operation of the image of the confirmation button 52 again (ST14).

  When it is detected by a signal from the touch panel sensor 122 that the image of the confirmation button 52 has been touched (YES in ST14), the CPU 101 captures the next frame image from the RAM 104 (ST3). And CPU101 performs the process after step ST3 again.

  As described above, in the commodity reading apparatus 1, the display unit 35 is configured by the process of step ST1 of the information processing procedure executed by the CPU 101 according to the commodity reading program, the recognition unit 31 is configured by the process of step ST5, and step ST6. The process of step ST7 constitutes the determination means 34, the process of step ST7 constitutes the output means 32, and the process of steps ST13 to ST16 constitutes the notification means 33. In other words, when an object other than a product is recognized as a foreign object N from the captured image, the notification unit 33 notifies the foreign object N displayed on the captured image displayed on the touch panel 12 by surrounding the frame 51, and then a confirmation button. When a certain period of time elapses without touching the image 52, a voice guidance is output from the voice output unit 105 to notify the user.

  When a store cashier in which the store accounting system of this embodiment is introduced takes out an unregistered product M from the shopping basket 6, the store cashier holds the product M over the reading window 1 </ b> B of the product reading device 1. Then, the image captured by the imaging unit 14 is displayed in the image display area R of the image recognition screen 50 displayed on the touch panel 12.

  At this time, as shown in FIG. 7, when the product is held over the reading window 1 </ b> B so that the cashier can see the appearance of the product M, the product reading apparatus 1 recognizes the product M. As a result, in the POS terminal 2, sales data of the product M is processed for sales.

  However, as shown in FIG. 8, if the cashier holds the product M over the reading window 1 </ b> B with the hand hidden, the product M is not recognized. In the commodity reading apparatus 1, the cashier's hand is recognized as the foreign object N. At this time, as shown in FIG. 9, a rectangular frame 51 is displayed on the image recognition screen 50 so as to surround the cashier's hand as an object currently recognized by the recognition means 31 of the CPU 101. Therefore, the cashier can easily confirm that the hand, not the product M, has been recognized by displaying the frame 51 on the hand image.

  In addition, after the frame 51 is displayed, when a certain time has elapsed without touching the image of the confirmation button 52, it is notified that the product M is not recognized by voice guidance. Therefore, even if the display of the frame 51 is not noticed, the cashier immediately notices that the product M is not recognized, so that a quick response can be taken.

The present invention is not limited to the above embodiment.
For example, in the embodiment, when an object other than a product is recognized, a frame 51 is displayed so as to surround the recognized object in the captured image displayed on the touch panel 12, and when a predetermined time elapses, a predetermined voice guidance is displayed. Is output to notify that the product M is not recognized, but the audio output may be omitted and the notification may be performed only by displaying the frame 51. Or you may alert | report only by audio | voice output conversely. Further, the content of the notification is not limited to the display of the frame 51 or the voice output of the guidance. In short, it is only necessary to notify the operator that the product is not recognized.

  In the above embodiment, the storage location of the foreign object recognition dictionary file 9 is the RAM 104 of the commodity reading apparatus 1, but the storage location is not limited to this. For example, it may be stored in the HDD device 205 of the POS terminal 2. Further, in the above embodiment, it is determined whether the recognized object is a foreign object other than the product first, and if it is not a foreign object, the product M is specified, but it is first determined whether the recognized object is a product, If the identification object cannot be specified, it may be determined that the recognized object is a foreign object. In this case, the foreign object recognition dictionary file 9 is not necessarily required.

  Moreover, although the said embodiment was applied to the goods reading apparatus 1 of the accounting counter which a cashier respond | corresponds, an application example is not limited to this. For example, the present invention can be applied to a product reading device of a self-checkout system in which a customer himself performs accounting. In addition, the commodity reading device 1 may be integrated with the POS terminal 2. In short, the present invention can be applied to a checkout system including the commodity reading device 1 and the POS terminal 2.

  In the above embodiment, a product reading program for realizing the functions of the invention is recorded in advance in the ROM 103 which is a program storage unit in the apparatus. However, the present invention is not limited to this, and a similar program may be downloaded from the network to the apparatus. Alternatively, a similar program recorded on the recording medium may be installed in the apparatus. The recording medium may be in any form as long as it can store a program such as a CD-ROM or a memory card and can be read by the apparatus. Further, the function obtained by installing or downloading the program may be realized in cooperation with an OS (operating system) in the apparatus.

In addition, although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.
Hereinafter, the invention described in the scope of claims of the present application will be appended.
[1] Recognizing means for recognizing an object held by the reading unit from an image captured by the imaging unit, and information on the product when the object recognized by the recognizing unit is a product for which sales processing is performed by the settlement unit Output means for outputting to the settlement unit, and notification means for notifying that the object recognized by the recognition unit is an object other than a product that is not processed by the settlement unit. apparatus.
[2] The supplementary note, further comprising: a determining unit that determines whether the object recognized by the recognizing unit is a product that is processed by the settlement unit or an object other than a product that is not processed for sales. [1] The product reading device according to [1].
[3] A storage unit that stores data representing an appearance of an object other than a product that is held by the reading unit and is not captured by the settlement unit even when captured by the imaging unit. The product reading apparatus according to appendix [2], wherein the determination unit makes a determination based on data stored in the storage unit.
[4] It further includes display means for displaying an image picked up by the image pickup means, and the notification means displays an object other than the product recognized by the recognition means in the image displayed by the display means. The commodity reading apparatus according to any one of the supplementary notes [1] to [3], characterized in that notification is provided.
[5] Voice output means for outputting voice guidance is further provided, and the notification means shows an object other than the product recognized by the recognition means in the image displayed by the display means, and then is constant. When the time has elapsed, the product reading apparatus according to the additional note [4], which is notified by outputting a predetermined voice guidance by the voice output unit.
[6] A recognition function for causing a computer to recognize an object held by the reading unit from an image captured by the imaging unit, and an object recognized by the recognition function to be processed by the settlement unit. Product reading for realizing an output function for outputting product information to the settlement unit and a notification function for notifying that an object recognized by the recognition function is an object other than a product that is not sales processed by the settlement unit program.

  DESCRIPTION OF SYMBOLS 1 ... Product reading apparatus, 2 ... POS terminal, 8 ... Product data file, 9 ... Foreign object recognition dictionary file, 12 ... Touch panel, 14 ... Imaging part, 50 ... Image recognition screen, 51 ... Frame, 101, 201 ... CPU, 105 ... sound output unit, R ... image display area.

Claims (5)

A reading unit;
A display unit;
First storage means for storing data representing the appearance of a product to be sold;
Second storage means for storing data representing the appearance of the hand;
Recognizing means for recognizing an object held over the reading unit based on data stored in the first and second storage means;
When recognizing that the object is the product to be sales processed, display means for displaying information on the product on the display unit;
Recognizing that a hand is held over the reading unit, notifying means for notifying that a product is not recognized without displaying information on the product to be sold on the display unit;
A store accounting system characterized by comprising: The store accounting system according to claim 1, wherein the notifying unit notifies that the product is not recognized by displaying an image indicating that the hand is recognized on the display unit. The store accounting system according to claim 2, wherein the notifying unit notifies that a product is not recognized by showing a hand in an image displayed on the display unit. Voice output means for outputting voice guidance;
Further comprising
The notification means displays an image indicating that the hand has been recognized on the display unit, and after a predetermined time has elapsed, the voice output means outputs a predetermined voice guidance so that the product is not recognized. store accounting system according to claim 2, characterized in that for informing. On the computer,
First storage means for storing data representing the appearance of the product to be sales processing and, based on the data stored in the second storage means for storing data representing the appearance of the hand, held over the read isolation portion A function to recognize the object,
When recognizing that the object is the product to be sales processed, a function of displaying information on the product on a display unit;
When recognizing that a hand is held over the reading unit, a function for notifying that a product is not recognized without displaying information on the product to be sold on the display unit;
Store accounting program to realize.