JP5748880B2 - Information processing apparatus, store system, and program - Google Patents

Description

  Embodiments described herein relate generally to an information processing apparatus, a store system, and a program.

  Conventionally, a feature amount of a target article is extracted from image data obtained by imaging an object (object) as a target using an image sensor device, and the degree of similarity is compared with comparison data (feature quantity) in a dictionary prepared in advance. There is a technology related to general object recognition that calculates and recognizes (detects) the type or the like of the article according to the calculated similarity. Further, there has been proposed a store system that uses the technology related to general object recognition to identify products such as fruits and vegetables and registers sales of products recognized as objects.

  By the way, for example, when it is desired to cancel data registered as a product sale by object recognition or to cancel a product extracted as a product candidate, it is necessary to temporarily stop the object recognition operation and cancel it by key input. For this reason, the operator sometimes has to put the target article that has been held over the image sensor device for object recognition once, so that the operability is poor.

The information processing apparatus according to the embodiment includes a capturing unit that captures a frame image including an image of an object captured by the imaging unit, an image of the object included in the captured frame image, and each registered product registered in the dictionary. Similarity calculation means for calculating a similarity indicating how similar the reference image is, a product candidate extraction means for extracting a candidate for the registered product corresponding to the object based on the similarity, Display means for displaying the candidate extracted by the commodity candidate extraction means on the display device as a provisionally confirmed commodity, and before confirming the provisionally confirmed commodity as the registered commodity corresponding to the object, the color attribute of the frame image is predetermined. Special processing execution means for executing a predetermined special processing for the provisionally confirmed product when it is determined that the reference value is exceeded.

The store system according to the embodiment includes a capturing unit that captures a frame image including an image of an object captured by the imaging unit, an image of the object included in the captured frame image, and a reference for each registered product registered in the dictionary. Similarity calculating means for calculating the degree of similarity between the images, product candidate extracting means for extracting a candidate for the registered product corresponding to the object based on the similarity, and the product Display means for displaying the candidate extracted by the candidate extraction means on the display device as a provisionally confirmed product, and before confirming the provisionally confirmed product as the registered product corresponding to the object, the color attribute of the frame image is a predetermined value. When it is determined that the reference value has been exceeded, special processing execution means for executing predetermined special processing for the provisionally confirmed product, and the registration determined by the determination means It includes goods and a sales registration processing means for sales registration processing, the.

The program according to the embodiment includes a computer that captures a frame image including an image of an object captured by the imaging unit, an image of the object included in the captured frame image, and each registered product registered in the dictionary. Similarity calculating means for calculating the degree of similarity between the reference image and the product candidate extracting means for extracting candidates for the registered goods corresponding to the object based on the similarity; Display means for displaying the candidate extracted by the commodity candidate extraction means on a display device as a provisionally confirmed commodity, and before confirming the provisionally confirmed commodity as the registered commodity corresponding to the object, the color attribute of the frame image is When it is determined that the predetermined reference value is exceeded, the special function executing means is configured to execute a special process predetermined for the provisionally confirmed product.

FIG. 1 is a perspective view illustrating an example of a checkout system (store system) according to the embodiment. FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal and the commodity reading apparatus. FIG. 3 is a conceptual diagram illustrating the data structure of a PLU file. FIG. 4 is a block diagram illustrating functional configurations of the POS terminal and the commodity reading apparatus. FIG. 5 is a diagram illustrating an example of a frame image captured by the image capturing unit. FIG. 6 is a diagram illustrating an example of a provisional confirmation screen. FIG. 7 is a diagram illustrating an example of a confirmation screen. FIG. 8 is a diagram illustrating an example of a screen on which illustration images of product candidates are displayed. FIG. 9 is a flowchart illustrating a procedure of product identification processing executed by the product reading device. FIG. 10 is a flowchart showing a procedure of sales registration processing executed by the POS terminal. FIG. 11 is an external perspective view showing the configuration of the self-POS. FIG. 12 is a block diagram showing a hardware configuration of the self-POS.

  Hereinafter, an information processing apparatus, a store system, and a program according to the present embodiment will be described with reference to the drawings, taking a checkout system as an example. The store system is a checkout system (POS system) provided with a POS terminal that performs registration and settlement of commodities related to one transaction. This embodiment is an application example to a checkout system introduced in a store such as a supermarket.

  FIG. 1 is a perspective view showing an example of a checkout system 1. As shown in FIG. 1, the checkout system 1 includes a product reading device 101 that reads information about a product, and a POS terminal 11 that performs registration and settlement of the product related to one transaction. Below, the example which applies the goods reading apparatus 101 as an information processing apparatus concerning this embodiment is demonstrated.

  The POS terminal 11 is placed on the upper surface of the drawer 21 on the checkout table 41. The drawer 21 is controlled by the POS terminal 11 for the opening operation. On the upper surface of the POS terminal 11, a keyboard 22 that is pressed down by an operator (a store clerk) is arranged. A display device 23 that displays information toward the operator is provided behind the keyboard 22 when viewed from the operator who operates the keyboard 22. The display device 23 displays information on the display surface 23a. A touch panel 26 is laminated on the display surface 23a. A customer display device 24 is erected so as to be rotatable further back than the display device 23. The customer display device 24 displays information on the display surface 24a. The customer display device 24 shown in FIG. 1 has the display surface 24a facing the front side in FIG. 1, but the customer display device 24 is rotated so that the display surface 24a faces the back side in FIG. Thus, the customer display device 24 displays information to the customer.

  A horizontally long table-shaped counter table 151 is arranged so as to form an L shape with the checkout table 41 on which the POS terminal 11 is placed. A load receiving surface 152 is formed on the upper surface of the counter table 151. A shopping basket 153 for storing the product G is placed on the cargo receiving surface 152. The shopping basket 153 may be divided into a first shopping basket 153a brought in by a customer and a second shopping basket 153b placed at a position sandwiching the commodity reading device 101 from the first shopping basket 153a. it can. The shopping basket 153 is not limited to a so-called basket shape, and may be a tray or the like. The shopping basket 153 (second shopping basket 153b) is not limited to a so-called basket shape, and may be a box shape, a bag shape, or the like.

  On the receiving surface 152 of the counter stand 151, the commodity reading apparatus 101 connected to the POS terminal 11 so as to be able to transmit and receive data is installed. The commodity reading apparatus 101 includes a thin rectangular housing 102. A reading window 103 is disposed in front of the housing 102. A display / operation unit 104 is attached to the upper portion of the housing 102. The display / operation unit 104 is provided with a display device 106 on which a touch panel 105 is laminated. A keyboard 107 is disposed on the right side of the display device 106. A card reading groove 108 of a card reader (not shown) is provided on the right side of the keyboard 107. A customer display device 109 for providing information to the customer is installed on the back left side of the display / operation unit 104 as viewed from the operator.

  Such a product reading apparatus 101 includes a product reading unit 110 (see FIG. 2). The merchandise reading unit 110 has an imaging unit 164 (see FIG. 2) disposed behind the reading window 103.

  In the first shopping basket 153a brought in by the customer, a product G related to one transaction is stored. The commodity G in the first shopping basket 153a is moved to the second shopping basket 153b by an operator who operates the commodity reading apparatus 101. In this movement process, the product G is directed to the reading window 103 of the product reading apparatus 101. At this time, the imaging unit 164 (see FIG. 2) arranged in the reading window 103 images the product G.

  The merchandise reading apparatus 101 displays a screen for designating which merchandise G included in the image captured by the imaging unit 164 corresponds to which merchandise registered in a PLU file F1 (see FIG. 3) described later. Displayed on the operation unit 104 and notifies the POS terminal 11 of the product ID of the specified product. In the POS terminal 11, based on the product ID notified from the product reading device 101, information related to sales registration such as the product classification, product name, and unit price of the product corresponding to the product ID is stored in a sales master file (not shown) or the like. And record sales.

  FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal 11 and the commodity reading apparatus 101. The POS terminal 11 includes a microcomputer 60 as an information processing unit that executes information processing. The microcomputer 60 is configured by connecting a ROM (Read Only Memory) 62 and a RAM (Random Access Memory) 63 to a CPU (Central Processing Unit) 61 that executes various arithmetic processes and controls each unit.

  The drawer 61, the keyboard 22, the display device 23, the touch panel 26, and the customer display device 24 are all connected to the CPU 61 of the POS terminal 11 through various input / output circuits (all not shown). . These are controlled by the CPU 61.

  The keyboard 22 includes a numeric key 22d, a temporary fastening key 22e, and a fastening key 22f on which numbers such as “1”, “2”, “3”... Including.

  An HDD (Hard Disk Drive) 64 is connected to the CPU 61 of the POS terminal 11. The HDD 64 stores programs and various files. All or a part of the programs and various files stored in the HDD 64 are copied to the RAM 63 and executed by the CPU 61 when the POS terminal 11 is activated. An example of a program stored in the HDD 64 is a product sales data processing program PR. An example of a file stored in the HDD 64 is a PLU file F1 distributed and stored from the store computer SC.

  The PLU file F1 is a product file in which an association between information related to sales registration of the product G and an image of the product G is set for each product G to be displayed and sold in the store.

  FIG. 3 is a conceptual diagram illustrating the data configuration of the PLU file F1. As shown in FIG. 3, the PLU file F1 includes, for each product G, a product ID uniquely assigned to the product, information on the product such as a product classification to which the product G belongs, a product name, a unit price, and a product image obtained by capturing the product This is a file for storing (reference image) and an illustration image showing the product as product information of the product G. The product image is a captured image (photograph) of the product used for determining the similarity. The PLU file F1 is configured to be readable by the commodity reading apparatus 101 via a connection interface 65 described later.

  Note that the data configuration of the PLU file F1 is not limited to the example of FIG. 3, and for example, feature quantities such as color and surface irregularities read from a product image may be stored for each product.

  Returning to FIG. 2, a communication interface 25 for executing data communication with the store computer SC is connected to the CPU 61 of the POS terminal 11 via an input / output circuit (not shown). The store computer SC is installed in a store backyard or the like. A PLU file F1 distributed to the POS terminal 11 is stored in the HDD (not shown) of the store computer SC.

  Further, the CPU 61 of the POS terminal 11 is connected to a connection interface 65 that enables data transmission / reception with the commodity reading apparatus 101. The product reading apparatus 101 is connected to the connection interface 65. The CPU 61 of the POS terminal 11 is connected to a printer 66 that prints on receipts. The POS terminal 11 prints the transaction content of one transaction on the receipt under the control of the CPU 61.

  The commodity reading apparatus 101 also includes a microcomputer 160. The microcomputer 160 is configured by connecting a ROM 162 and a RAM 163 to a CPU 161 via a bus. The ROM 162 stores a program executed by the CPU 161. The CPU 161 is connected to an imaging unit 164 and an audio output unit 165 via various input / output circuits (none of which are shown). The operations of the imaging unit 164 and the audio output unit 165 are controlled by the CPU 161. The display / operation unit 104 is connected to the product reading unit 110 and the POS terminal 11 via the connection interface 176. The operation of the display / operation unit 104 is controlled by the CPU 161 of the product reading unit 110 and the CPU 61 of the POS terminal 11.

  The imaging unit 164 includes an image sensor (not shown) such as a color CCD image sensor or a color CMOS image sensor, and a light source (not shown) such as an LED, and is controlled from the reading window 103 under the control of the CPU 161. It is an imaging means for performing imaging. For example, the imaging unit 164 captures a 30 fps moving image. Frame images (captured images) sequentially captured at a predetermined frame rate by the imaging unit 164 are stored in the RAM 163.

  The audio output unit 165 is an audio circuit and a speaker for generating a preset warning sound or the like. The audio output unit 165 performs warning sound or audio notification under the control of the CPU 161.

  Furthermore, a connection interface 175 that connects to the connection interface 65 of the POS terminal 11 and enables data transmission / reception with the POS terminal 11 is connected to the CPU 161. In addition, the CPU 161 transmits and receives data to and from the display / operation unit 104 via the connection interface 175.

  Next, functional configurations of the CPU 161 and the CPU 61 that are realized by the CPU 161 and the CPU 61 executing programs will be described with reference to FIG.

  FIG. 4 is a block diagram illustrating functional configurations of the POS terminal 11 and the commodity reading apparatus 101. As illustrated in FIG. 4, the CPU 161 of the commodity reading apparatus 101 executes a program stored in the ROM 162, whereby a background capturing unit, an image capturing unit 51 serving as a capturing unit, and a commodity detecting unit 52 serving as a detecting unit. A similarity calculation unit 53 as a similarity calculation unit, a similarity determination unit 54 as a product candidate extraction unit, a confirmation notification unit 55, a product candidate presentation unit 56, an input reception unit 57, an information output unit 58, and a determination unit. Functions of the determination unit 59 and the special process execution unit 50 that is a special process execution unit are provided. Similarly, the CPU 61 of the POS terminal 11 has a function as the sales registration unit 611 by executing the program PR.

  The image capturing unit 51 outputs an imaging on signal to the imaging unit 164 and causes the imaging unit 164 to start an imaging operation. The image capturing unit 51 sequentially captures frame images captured by the image capturing unit 164 and stored in the RAM 163 after the start of the image capturing operation. The capturing of the frame image by the image capturing unit 51 is performed in the order stored in the RAM 163.

  FIG. 5 is a diagram illustrating an example of a frame image captured by the image capturing unit 51. As shown in FIG. 5, when the operator holds the product G over the reading window 103, all or part of the product G is photographed in the reading region R of the imaging unit 164.

  By the way, when the product G (object) is not located near the light source of the imaging unit 164, that is, near the reading window 103, the photographed image by the imaging unit 164 is in a state where the brightness is 0 or close thereto. Then, when the product G (object) to be recognized is brought close to the reading window 103, the light from the light source is reflected on the product G (object), so that the brightness of the image captured by the imaging unit 164 changes brightly.

  The image capturing unit 51 detects such brightness change (change in color attribute), and performs object recognition processing (product detection unit 52, product detection unit 52, when the brightness exceeds a predetermined reference value). The similarity calculation unit 53, the similarity determination unit 54, the confirmation notification unit 55, the product candidate presentation unit 56, the input reception unit 57, the determination unit 59, and the special process execution unit 50) are activated. Thereby, it is possible to suppress the CPU load when the object recognition process described later is unnecessary.

  Note that, depending on the installation environment of the product reading apparatus 101, there may be variations in the captured image in the absence of the product G (object). Therefore, an object recognition engine (product detection unit 52, similarity calculation unit 53, similarity determination unit 54, confirmation notification unit 55, so as not to increase the CPU load due to unnecessary object recognition processing due to the influence of unnecessary disturbances, It is necessary to control and set the lightness reference value for operating the product candidate presentation unit 56, the input reception unit 57, the determination unit 59, and the special process execution unit 50).

  Note that as the color attribute, in addition to the lightness representing the brightness of the color, a saturation representing the intensity of the color may be used.

  The product detection unit 52 detects all or part of the product G included in the frame image captured by the image capture unit 51 using a pattern matching technique or the like. Specifically, a contour line or the like is extracted from an image obtained by binarizing the captured frame image. Next, based on the difference between the contour line extracted from the previous frame image and the contour line extracted from the current frame image, a product directed to the reading window 103 for sales registration is detected.

  The previous frame image is, for example, an object recognition engine (product detection unit 52, similarity calculation unit 53, similarity determination unit 54, confirmation notification unit 55, product candidate presentation unit 56, input reception unit 57, confirmation unit. 59, a background frame image captured by the image capturing unit 51 before starting the special processing execution unit 50), that is, before starting the object recognition processing. The automatic acquisition of the background frame image as the previous frame image before the activation of the object recognition application is performed in an environment where the background is relatively unchanged (for example, the imaging unit 164 receives the load on the upper surface of the counter table 151). This is useful for use in a case where the vertical downward direction is attached to the surface 152 as the reading direction. The environment includes the state of lighting (the surroundings and lighting of the commodity reading apparatus 101 itself), the clothes of the user, and the like.

  Further, as a previous frame image, a background frame image captured by the image capturing unit 51 immediately after execution of the previous object recognition process may be used. This is useful for use in environments where the background is relatively changeable, or where the customer's clothes and distance from the product reading device 101 are expected to change variously, such as self-checkout. It is.

  Furthermore, the user may acquire the previous frame image as a background at an arbitrary timing. There are various cases in which the previous frame image is acquired as a background at an arbitrary timing, but this is effective when, for example, the PLU file F1, which is a dictionary, is created.

  As another method for detecting a product, the presence or absence of a skin color region is detected from a captured frame image. Next, when a skin color area is detected, that is, when a clerk's hand is detected, it is assumed that the clerk's hand is grasped by detecting the above-described contour line in the vicinity of the skin color area. Try to extract the outline of the product. At this time, when the contour indicating the shape of the hand and the contour of the other object are detected in the vicinity of the contour of the hand, the product is detected from the contour of the object.

  The similarity calculation unit 53 reads the surface state such as the hue of the product G and the unevenness of the surface as a feature amount from all or a part of the product G imaged by the imaging unit 164. The similarity calculation unit 53 does not consider the outline or size of the product G in order to shorten the processing time.

  In addition, the similarity calculation unit 53 reads the surface state such as the color of the registered product and the unevenness of the surface from the product image of each product (hereinafter referred to as registered product) registered in the PLU file F1 as a feature amount. The similarity between the product G and the product registered in the PLU file F1 is calculated by comparing with the feature amount of the product G. Here, the similarity is the degree of similarity of all or part of the product G when the product image of each product stored in the PLU file F1 is 100% = “similarity: 1.0”. It shows whether you are doing. Note that, for example, the similarity may be calculated by changing the weight between the color tone and the surface roughness.

Recognizing an object contained in an image in this way is called generic object recognition. Regarding such general object recognition, various recognition techniques are described in the following documents.
Keiji Yanai, “Current Status and Future of General Object Recognition”, IPSJ Journal, Vol. 48, no. SIG16 [Search August 10, 2010], Internet <URL: http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

Further, techniques for performing general object recognition by dividing an image into regions for each object are described in the following documents.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, [searched August 10, 2010], Internet <URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1. 1.145.3036 & rep = rep1 & type = pdf>

  Note that the method of calculating the similarity between the captured image of the product G and the product image of the registered product registered in the PLU file F1 is not particularly limited. For example, the similarity between the captured image of the product G and each registered product registered in the PLU file F1 may be calculated as an absolute evaluation or may be calculated as a relative evaluation.

  When calculating the similarity as an absolute evaluation, the captured image of the product G and each registered product registered in the PLU file F1 are compared on a one-to-one basis, and the similarity derived as a result of this comparison is used as it is. do it. Further, when calculating the similarity as a relative evaluation, if five registered products (products GA, GB, GC, GD, GE) are registered in the PLU file F1, the captured product G is The similarity is 0.6, the similarity is 0.1 for the product GB, the similarity is 0.1 for the product GC, the similarity is 0.1 for the product GD, and the product GE On the other hand, the similarity may be calculated such that the similarity is 0.1, and the total similarity with each registered product is 1.0 (100%).

  The similarity determination unit 54 compares the similarity between the image of the product G and the product image registered in the PLU file F1 for each frame image captured by the image capturing unit 51. In the present embodiment, a plurality of conditions for the similarity between the product image of the registered product and the image of the product G are provided in stages, and the similarity determination unit 54 determines the registered product according to the satisfied condition or Select candidates for registered products. Although the conditions regarding the degree of similarity are not particularly limited, a case where conditions a to c are used will be described below.

  Here, the condition a and the condition b are the first conditions according to the present embodiment, and the product G captured by the imaging unit 164 is determined as one product among the registered products registered in the PLU file F1. It is a condition. The condition c is a second condition according to the present embodiment, and is a condition for extracting a candidate for the product G imaged by the imaging unit 164 from the registered products registered in the PLU file F1.

  The similarity determination unit 54 determines (determines) a registered product that satisfies the condition a or the condition b as a product that corresponds to the product G captured by the imaging unit 164 on a one-to-one basis (hereinafter referred to as a confirmed product). Further, the similarity determination unit 54 determines that the registered product satisfying the condition c is not a confirmed product but a candidate for the product G imaged by the imaging unit 164 (hereinafter referred to as a product candidate). Then, product candidates for the product G are extracted by extracting registered products satisfying the condition c from the plurality of registered products registered in the PLU file F1.

  The details of the conditions a to c are not particularly limited as long as the conditions a to c are set stepwise according to the degree of similarity, but as an example, the conditions a to c can be provided by a plurality of preset threshold values. Here, the case where the conditions a to c are set by the first threshold value to the third threshold value will be described. Note that the magnitude relationship between the first to third thresholds is first threshold> second threshold> third threshold.

  The similarity determination unit 54 counts the number of times that the degree of similarity with the registered product is equal to or higher than a predetermined first threshold (for example, 90%), and the condition a is satisfied when the number of times exceeds a predetermined number. It is determined that When the first threshold is set sufficiently high so that no erroneous determination is made, the condition a may be determined by setting the predetermined number of times as one.

  The similarity determination unit 54 determines that the condition b is satisfied when the similarity to the registered product is less than the first threshold (for example, 90%) and equal to or greater than the second threshold (for example, 75%) that is smaller than the first threshold. Determined to be satisfied. Then, it is determined that the registered product that satisfies the condition b is a confirmed product, but requires a confirmation operation by the operator. Note that the number of times the similarity with the registered product is less than the first threshold (for example, 90%) and the second threshold value (for example, 75%) is less than the first threshold is counted, It may be determined that the condition b is satisfied.

  Furthermore, when the similarity with the registered product is less than the second threshold (for example, 75%) and greater than or equal to the third threshold (for example, 10%) smaller than the second threshold, the similarity determination unit 54 determines that the condition c is Determined to be satisfied. Furthermore, the number of times that the degree of similarity with the registered product is less than the second threshold value (for example, 75%) and the third threshold value (for example, 10%) that is less than the second threshold value is counted. You may determine with the condition c being satisfied when it becomes above.

  Each condition a to c can be appropriately set according to the degree of similarity, the number of determinations, and the like, and is not limited to the above-described example. Moreover, the predetermined number of times used for the determination of the conditions a to c may be different for each condition.

  The confirmation notification unit 55 informs an operator or customer that the product imaged by the imaging unit 164 is uniquely provisionally confirmed as a registered product satisfying the condition a or the condition b by image output or audio output. Inform.

  More specifically, the confirmation notification unit 55 indicates that the registered product satisfying the condition a is provisionally confirmed as a product (provisionally confirmed product) captured by the imaging unit 164 (see FIG. 5). 6) is displayed on the display device 106.

  FIG. 6 is a diagram illustrating an example of the provisional confirmation screen 71. When there is a registered product satisfying the condition a, the confirmation notification unit 55 stops displaying the captured image in the reading region R (see FIG. 5), and displays the illustration image G1 corresponding to the provisionally confirmed product and the product name “carrot”. The data is read from the PLU file F1 and displayed on the provisional confirmation screen 71. Further, the confirmation notification unit 55 displays the product name and product price (unit price) of the provisionally confirmed product read from the PLU file F1 in the product name display area 81 and the price display area 82, respectively. The confirmation notification unit 55 may display a product image (photo) read from the PLU file F1 instead of the illustration image G1. Further, the confirmation notification unit 55 displays a confirmation key 85 on the temporary confirmation screen 71. The confirmation key 85 is used to cause the operator to operate when declaring the confirmation of the provisionally confirmed product.

  The confirmation unit 59 confirms the provisionally confirmed product as a registered product corresponding to the object when there is an operation via the touch panel 105 on the confirmation key 85 on the provisional confirmation screen 71.

  Further, the confirmation notifying unit 55 sounds the same sound as that of the conventional barcode reading, assuming that the object recognition is confirmed with sufficient reliability in accordance with the operation on the confirmation key 85 of the temporary confirmation screen 71. This informs that the recognition has been completed normally and the data has been confirmed.

  In addition, the confirmation notification unit 55 outputs information related to the confirmed product such as the product name read from the PLU file F1 to the voice output unit 165 in accordance with the operation on the confirmation key 85 on the provisional confirmation screen 71. The voice output unit 165 notifies the operator or customer of information indicating the confirmed product by outputting the input information.

  The confirmation notification unit 55 also receives a confirmation screen 72 for receiving a final confirmation operation as to whether or not the registered product (provisionally confirmed product) satisfying the condition b is the product G imaged by the imaging unit 164 (see FIG. 7). Is displayed on the display device 106.

  FIG. 7 is a diagram illustrating an example of the confirmation screen 72. When there is a registered product satisfying the condition b, the confirmation notification unit 55 reads the illustration image G1 corresponding to the provisionally confirmed product from the PLU file F1 and displays it on the confirmation screen 72. In addition, the confirmation notification unit 55 uses a product name of the provisionally confirmed product read from the PLU file F1, and a message asking whether the read product G is a product of the illustration image G1 is “carrot?” And so on. Further, the confirmation screen 72 is provided with buttons such as “Yes / No” so as to be selectable by a touch operation on the touch panel 105.

  As described above, the confirmation screen 72 shows the product name and product image of the registered product (provisionally confirmed product) that is uniquely selected for one product G as a result of the similarity determination, and the product G and the registered product. Are displayed in a one-to-one relationship. Therefore, the confirmation screen 72 is a screen for notifying that the registered product satisfying the condition b is uniquely determined as the product G imaged by the imaging unit 164.

  In the present embodiment, when the product imaged in this way is uniquely determined, an illustration image of the temporarily confirmed product is displayed instead of the captured image. Accordingly, the operator can intuitively recognize that the identification of the product has been confirmed, and can recognize at a glance which product the identified product is.

  Note that the confirmation notification unit 55 may cause the audio output unit 165 to output the information on the provisionally fixed product and the information for prompting confirmation in synchronization with the timing at which the confirmation screen 72 is displayed. May not be performed. The presence or absence of audio output on the temporary confirmation screen 71 and the confirmation screen 72 can be set as appropriate.

  The product candidate presentation unit 56 causes the display device 106 to display information on the registered product that satisfies the condition c as a product candidate. More specifically, the product candidate presentation unit 56 reads the illustration image and product name of the registered product satisfying the condition c from the PLU file F1, and uses the display device 106 from the one with the high similarity calculated by the similarity calculation unit 53. Are output sequentially. The display device 106 displays the product candidate illustration images and product names in descending order of similarity in the product candidate presentation area 83 (see FIG. 8).

  FIG. 8 is a diagram illustrating an example of a screen on which illustration images G1, G2, and G3 of product candidates are displayed. As illustrated in FIG. 8, product candidate illustration images G1, G2, and G3 and product names are displayed in the product candidate presentation area 83 in order from registered products with a high degree of similarity. These illustration images G1, G2, and G3 are configured to be selectable in accordance with a selection operation on the touch panel 105. A selection button 84 for selecting a product from the product list is provided below the product candidate presentation area 83, and the product selected from the product list is processed as the provisionally confirmed product described above. Although FIG. 8 shows an example in which three product candidates corresponding to the illustration images G1 to G3 are displayed, the number of product candidates and the display method are not particularly limited. In addition, a product image (photo) may be displayed instead of the illustration image.

  Even if the product candidate illustration images G <b> 1 to G <b> 3 are displayed on the display device 106, when the selection operation for these product candidates is not accepted, the image capturing process by the image capturing unit 51, the product detecting unit 52 The detection process and the similarity calculation process by the similarity calculation unit 53 are continued. Therefore, while the product candidate is not selected, the captured image of the reading region R is displayed on the screen of the display device 106.

  The input receiving unit 57 receives various input operations corresponding to the display on the display device 106 via the touch panel 105 or the keyboard 107. For example, the input receiving unit 57 performs an input operation (confirmation operation) indicating that the product of the displayed illustration image G1 is finally confirmed based on the selection operation on the confirmation screen 72 (see FIG. 7). Accept. The confirmation notification unit 55 displays the temporary confirmation screen 71 on the display device 106 when the input reception unit 57 receives a confirmation operation.

  The input receiving unit 57 receives a selection operation for any one of the illustration images G1 to G3 (see FIG. 8) of the product candidates displayed on the display device 106. The input receiving unit 57 receives the registered product of the selected illustration image as a provisionally confirmed product for the product G. When the product detection unit 52 can detect a plurality of products G, the input receiving unit 57 may receive a selection operation of a plurality of product candidates from the product candidates. When the input receiving unit 57 accepts the selection operation, the confirmation notifying unit 55 causes the display device 106 to display the provisional confirmation screen 71 that displays the accepted product candidate as the provisionally confirmed product.

  The information output unit 58 outputs information indicating the product (for example, product ID, product name, etc.) to the POS terminal 11 via the connection interface 175 for the confirmed product confirmed as described above.

  Note that the information output unit 58 may output the sales quantity separately input via the touch panel 105 or the keyboard 107 to the POS terminal 11 together with the product ID and the like. Further, as information output from the information output unit 58 to the POS terminal 11, the product ID read by the information output unit 58 from the PLU file F1 may be directly notified, or a product name that can specify the product ID. Alternatively, the file names of product images and illustration images may be notified, and the storage location of the product ID (storage address in the PLU file F1) may be notified to the POS terminal 11.

  When the special process execution unit 50 determines that the color attribute (for example, lightness) of the frame image exceeds a predetermined reference value before the determination by the determination unit 59, or the amount of change per unit time of the color attribute When it is determined that (for example, the amount of change in brightness per unit time) exceeds a predetermined reference value, a special process determined in advance is executed for the provisionally confirmed product. In the present embodiment, the special process execution unit 50 executes a confirmation cancellation process for canceling the confirmation of the provisionally confirmed product as a special process.

  The sales registration unit 611 of the POS terminal 11 performs sales registration of the corresponding product based on the product ID and the sales quantity output from the information output unit 58. Specifically, the sales registration unit 611 refers to the PLU file F1, and records the notified product ID and the product classification, product name, unit price, etc. corresponding to the product ID together with the number of sales in a sales master file or the like. To register sales.

  The product reading apparatus 101 also includes a known code symbol reading unit (not shown) that optically reads a code symbol such as a barcode or a two-dimensional code as information about the product. The merchandise reading apparatus 101 transmits a merchandise ID, which is a number obtained by reading the code symbol by the code symbol reading section, to the POS terminal 11. The product reading apparatus 101 includes an object recognition processing mode that reads information about a product by object recognition processing by the product reading unit 110 as described above, and a code symbol reading mode that reads information about a product by code symbol reading processing by a code symbol reading unit. Can be switched by a key operation of the keyboard 107 or the like.

  The operation of the checkout system 1 when the commodity reading apparatus 101 is set to the code symbol reading mode will be briefly described. When the product reading apparatus 101 is set to the code symbol reading mode, the POS terminal 11 determines that the number obtained by reading the code symbol in the code symbol reading unit matches the product ID stored in the PLU file F1. Then, a signal indicating that and information related to sales registration such as the product classification, product name, and unit price of the product corresponding to the product ID are transmitted to the product reading apparatus 101. On the other hand, when the product reading apparatus 101 receives a signal indicating that it matches the product ID and information related to sales registration of the product corresponding to the product ID, for example, the product reading device 101 emits a beeping sound due to a short buzzer sound. At the same time, information related to sales registration is displayed on the display device 106.

  Based on the product ID notified from the product reading device 101, the POS terminal 11 acquires information related to sales registration such as product classification, product name, unit price, etc. of the product corresponding to the product ID from the PLU file F1, and stores the sales master. Record sales in a file (not shown) and register sales.

  Next, the operation of the checkout system 1 in the object recognition processing mode will be described in detail. First, the operation of the commodity reading apparatus 101 will be described. FIG. 9 is a flowchart illustrating a procedure of product identification processing executed by the product reading apparatus 101. In FIG. 9, the first threshold is set to 90% (0.90) similarity, the second threshold is set to 75% (0.75), and the third threshold is set to 10% (0.10) similarity. Although described, each threshold value is not limited to these.

  When processing is started in response to the start of product registration by the POS terminal 11, the image capturing unit 51 outputs an imaging on signal to the imaging unit 164 and starts imaging by the imaging unit 164 (step S11). The image capturing unit 164 captures the background image data captured and stored in the RAM 163 (step S12).

  As described above, the object recognition engine (the product detection unit 52, the similarity calculation unit 53, the similarity determination unit 54, the confirmation notification unit 55, the product candidate presentation unit 56, the input reception unit 57, the determination unit 59, and the special process execution unit 50). By acquiring a frame image of the background before starting up, it is possible to acquire a background including the current state of illumination (the surroundings and the illumination of the product reading apparatus 101 itself), the clothes of the user, and the like.

  Next, the image capturing unit 51 captures a frame image (captured image) captured by the image capturing unit 164 and stored in the RAM 163 (step S13).

  Next, the image capturing unit 51 acquires the color attribute (for example, brightness) of the frame image captured by the image capturing unit 51 (step S14), and the color attribute (for example, brightness) has a predetermined reference value. When it exceeds (step S15: Yes), it is determined that the product G is directed to the reading window 103 for sales registration, and the object recognition engine (product detection unit 52, similarity calculation unit 53) that executes object recognition processing is performed. The similarity determination unit 54, the confirmation notification unit 55, the product candidate presentation unit 56, the input reception unit 57, the confirmation unit 59, and the special process execution unit 50) are activated (step S16), and the process proceeds to step S17.

  Note that the reference value of the color attribute (for example, brightness) may be dynamically changed. For example, when the product G is left in front of the product reading apparatus 101, the color attribute (for example, brightness) does not change, so that a predetermined time has elapsed (for example, after 2 seconds). If the reference value is set to be equal to or higher than the current brightness, the recognition process for the product G that has been left is not executed. When the reference value is changed in this way, it may be returned to the original reference value by using a change in brightness as a result of imaging the next object.

  Next, the product detection unit 52 detects all or part of the product G from the frame image captured by the image capture unit 51 based on the difference between the background frame image and the frame image captured by the image capture unit 51. Is performed (step S17).

  Subsequently, the similarity calculation unit 53 reads the feature amount of the product G from all or part of the image of the product G, and compares it with the feature amount of each product image registered in the PLU file F1 to determine whether the product is registered. The similarity is calculated (step S18).

  The similarity determination unit 54 determines whether there is a registered product with a similarity of 90% or more (step S19). If there is no registered product that is 90% or more (step S19: No), the process proceeds to step S21. When there is a registered product that is 90% or more (step S19: Yes), it is determined whether or not the number of times the similarity to the same registered product is 90% or more has reached a predetermined number (for example, 3 times). (Step S20). When the predetermined number is not reached (step S20: No), the process returns to step S12, and the image capturing unit 51 captures a new frame image captured by the image capturing unit 164.

  When the predetermined number of times has been reached (step S20: Yes), the confirmation notification unit 55 tentatively confirms the product, and displays a provisional confirmation screen 71 (see FIG. 6) including an illustration image of the temporarily confirmed product. Is notified of the provisionally confirmed product by voice notification of the first voice and the product name of the provisionally confirmed product (step S27). The first sound is a sound similar to a notification sound indicating that the product ID coincides with the product ID “beep” generated by, for example, a short buzzer sound when the product reading apparatus 101 is set to the code symbol reading mode. In this way, by making the sound similar to the notification sound indicating that it matches the product ID in the code symbol reading mode, it is easy to understand that it has been determined from the long-standing practice. Even if only the object recognition processing mode is not provided without the code symbol reading mode, the first sound is, for example, “beep” when the product reading apparatus 101 is set to the code symbol reading mode. It is better to use a sound similar to a notification sound indicating that the product ID matches.

  As shown in FIG. 9, the confirmation notifying unit 55 waits for an operation of the confirmation key 85 provided on the temporary confirmation screen 71 (step S <b> 28) or whiteout detection (step S <b> 29).

  Here, the whiteout detection process will be described in detail. In the whiteout detection process, the frame image (captured image) captured by the imaging unit 164 and stored in the RAM 163 is captured again, the color attribute (for example, brightness) of the captured frame image is acquired, and the color attribute ( For example, it is assumed that overexposure is detected when the brightness is over the upper limit.

  That is, the upper limit value of the color attribute (for example, lightness) is a threshold value for recognizing overexposure, which is a phenomenon in which, for example, the gradation of a bright part is lost and the subject is completely white.

  When the confirmation key 85 provided on the provisional confirmation screen 71 is operated (step S28: Yes), the confirmation unit 59 confirms the provisionally confirmed product as a registered product corresponding to the object, and the information output unit 58 Outputs the product ID of the registered product confirmed as the registered product to the POS terminal 11 together with the number of sales separately input via the keyboard 107 or the like (step S30), and then proceeds to the process of step S31.

  On the other hand, when the overexposure is detected (step S29: Yes), the special process execution unit 50 executes the cancel process (step S32), and thereafter proceeds to the process of step S31. The canceling process in step S32 is a process of executing a confirmation canceling process for a provisionally confirmed product that has been in a provisionally confirmed state after the object recognition process (here, steps S17 to S20). Such a definite cancellation process corresponds to a correction process for a conventional product registered product.

  For example, overexposure can be generated in a frame image captured by the imaging unit 164 by bringing a hand holding the product G close to (in close contact with) the reading window 103 of the product reading apparatus 101. That is, the brightness (color attribute) at which whiteout occurs is set as a reference value, and when the brightness (color attribute) exceeds the reference value, whiteout is detected (step S29: Yes). to decide.

  That is, in the present embodiment, after provisional confirmation of a product, a hand holding the product G or the like is brought close to (contacted to) the reading window 103 to give an input that is not in a normal object recognition state, and overexposure is performed. It is possible to generate and cancel the confirmation of the product.

  By doing so, it becomes possible to execute final confirmation cancellation of the product by an operation similar to the operation of holding the product G over the reading window 103, and the operability is greatly improved.

  When canceling product registration, an item for cancellation (for example, a white plate) that can cause overexposure may be held over the reading window 103.

  In the present embodiment, the detection of overexposure is used as a trigger for canceling product registration. However, the present invention is not limited to this, and an operation that causes an extreme change in brightness (change in color attribute) (for example, reading) An extreme change in brightness (change in color attribute) due to the operation of putting a hand or an item to be canceled in or out of the reading area R of the imaging unit 164 immediately before the window 103 may be used as a trigger for the cancellation process. That is, when the amount of change in brightness (color attribute) per unit time exceeds a predetermined reference value, it is determined that an extreme change in brightness (change in color attribute) has occurred (step S29: Yes).

  Step S19: In the case of No, the similarity determination unit 54 determines whether there is a registered product having a similarity of 75% or more and less than 90% (Step S21). If there is a registered product whose similarity is within this range (step S21: Yes), it is determined that the registered product is a confirmed product that requires operator confirmation, and a confirmation screen 72 is displayed on the display device 106. Then, the information about the product and the information for prompting confirmation are notified by voice (step S22). The input receiving unit 57 determines whether or not “yes” is selected on the confirmation screen 72 (see FIG. 7) (step S23). When “Yes” is selected (step S23: Yes), the process proceeds to step S27, where the confirmation notification unit 55 provisionally confirms the product, displays the provisional confirmation screen 71, and sets the product name of the provisionally confirmed product. Make a voice notification. On the other hand, when “No” is selected (step S23: No), the process returns to step S12.

  On the other hand, when there is no registered product with a similarity of 75% or more and less than 90% (step S21: No), the similarity determination unit 54 determines whether there is a registered product with a similarity of 10% or more and less than 75%. And the registered product whose similarity is within this range is extracted as a product candidate for the product G (step S24). If there is no registered product within this range (step S24: No), the process returns to step S12. When there is a registered product with a similarity of 10% or more and less than 75% (step S24: Yes), the similarity determination unit 54 determines that the corresponding registered product is a product candidate for the product G. Then, the product candidate presenting unit 56 sorts the illustration images and product names of the registered products determined as the product candidates in descending order of similarity and displays them in the product candidate presentation area 83, and indicates that there are a plurality of candidates. Voice notification is made with the second voice (step S25). The second sound is a sound different from the first sound described above, and is, for example, a “buzzy” sound due to the sound of a slightly long buzzer sound or a sound prompting selection.

  The input receiving unit 57 determines whether a selection operation for the product image of the registered product has been received (step S26). When the selection is accepted (step S26: Yes), the process proceeds to step S27, and the confirmation notification unit 55 temporarily confirms the product and displays the provisional confirmation screen 71. On the other hand, when selection is not accepted (step S26: No), it transfers to step S12.

  In step S <b> 31, the CPU 161 determines the presence / absence of business end from the POS terminal 11 based on a product registration end notification. When business is continued (step S31: No), the CPU 161 returns the process to step S12, and the image capturing unit 51 captures the background image data captured by the image capturing unit 164 and stored in the RAM 163, and continues the process. Let

  In the present embodiment, the background image data stored in the RAM 163 is the image capture unit 51 after the product is identified by the product reading device 101 and the product ID is output to the POS terminal 11 in step S30. Since new background image data is taken in, the latest image data is updated.

  Even when “Yes” is not selected on the confirmation screen 72 (see FIG. 7) in Step S23 or when the selection operation is not accepted in Step S26, the process returns to Step S12 to create a new background. Since the image data is captured, the background image data stored in the RAM 163 is updated to the latest one.

  When the job is to be ended (step S31: Yes), the image capturing unit 51 outputs an imaging off signal to the imaging unit 164, ends imaging by the imaging unit 164 (step S33), and ends the process.

  Next, the operation of the POS terminal 11 will be described. FIG. 10 is a flowchart showing a procedure of sales registration processing executed by the POS terminal 11.

  First, when processing is started in response to the start of product registration in response to an operation instruction on the keyboard 22, the CPU 61 determines the product ID of the confirmed product and the number of items sold by the product reading device 101 in step S30 of FIG. Is received (step S41). Next, the sales registration unit 611 reads the product type, unit price, and the like from the PLU file F1 based on the product ID and the sales quantity received in step S41, and stores the sales of the product G read by the product reading device 101 as a sales master file. (Step S42). Subsequently, the CPU 61 determines whether or not there is an end of business due to the end of sales registration or the like by an operation instruction of the keyboard 22 (step S43). When the business is continued (step S43: No), the CPU 61 returns to step S41 again and continues the process. When the business is to be ended (step S43: Yes), the CPU 61 ends the process.

  As described above, according to the present embodiment, when it is determined that the color attribute of the frame image exceeds a predetermined reference value before the provisionally confirmed product is confirmed as the registered product corresponding to the object, A special process determined in advance is executed. Alternatively, if it is determined that the amount of change per unit time of the color attribute of the frame image has exceeded a predetermined reference value before the provisionally confirmed product is confirmed as a registered product corresponding to the object, A special process determined in advance was executed. Thus, for example, when a confirmation cancellation process for canceling the confirmation of the provisionally confirmed product is assigned as a special process, a normal object recognition is performed by bringing the hand holding the product G close to (in close contact with) the reading window 103. By giving an input that is not a state and generating whiteout, it is possible to execute final confirmation of the product, so that the operability is greatly improved.

  In the present embodiment, for example, the final confirmation of the product is executed by detecting overexposure. However, the present invention is not limited to this, and various processes can be assigned. For example, a confirmation process for determining that the same product is continuously confirmed, a key display process for displaying a numeric key for inputting the quantity of the product on the display device 106 of the product reading apparatus 101, and the like may be assigned. Therefore, in the case of applying a confirmation process that considers that the same product is continuously confirmed, for example, by repeatedly detecting overexposure, it is possible to determine the same number of repetitions of the same product.

  As mentioned above, although embodiment of this invention was described, the said embodiment was shown as an example and is not intending limiting the range of invention. The above embodiment can be implemented in various other forms, and various omissions, replacements, changes, additions, and the like can be made without departing from the scope of the invention. Moreover, the said embodiment and its deformation | transformation are included in the range of the invention, the summary, and the invention described in the claim, and its equal range.

  For example, in the above-described embodiment, the product reading apparatus 101 captures one product G at a time. However, the number of products G captured at a time is not particularly limited, and may be plural. When imaging a plurality of products G at a time, the similarity between the plurality of products G and the images (G1, G2, G3,...) Of the registered products is calculated, By comparing these, the confirmed product and the candidate product may be selected.

  Further, the configuration of each display screen described above is not limited to the examples of FIGS. 5 to 8, and a display area and operation buttons for displaying other elements may be provided.

  In the above embodiment, the POS terminal 11 includes the PLU file F1. However, the present invention is not limited thereto, and the product reading apparatus 101 may include the PLU file F1, or the POS terminal 11 and the product reading apparatus 101. The external device that can be accessed may be provided with the PLU file F1.

  In the above embodiment, the product reading apparatus 101 has the functions of the similarity calculation unit 53 and the similarity determination unit 54. However, the present invention is not limited thereto, and the POS terminal 11 includes the similarity calculation unit 53, the similarity determination unit. It is good also as a form which provides the function of 54 and outputs a calculation result and a determination result to the goods reading apparatus 101. FIG.

  In the above description, the example in which the similarity is determined in four stages using the three threshold values of the first to third thresholds as the plurality of threshold values has been described. However, the embodiment is not limited thereto. It may be determined in more stages using three or more threshold values, or may be determined using two or less threshold values. Each threshold value may be provided so as to be changeable by the user. Moreover, although the example which made the 1st-3rd threshold value and conditions ac common to all goods was given in the above-mentioned, it is not limited to this, Even if each threshold value and conditions are set for every registered goods, respectively. Good.

  In the above description, the first condition is divided into two stages: the case where the confirmation screen 72 is displayed (condition b) and the case where the confirmation screen 72 is not displayed (condition a). One condition does not have to be divided into multiple stages. That is, when another example is shown along with the above-described example, the product may be determined without displaying the confirmation screen 72 when any of the conditions a and b is satisfied. It may be possible to set whether or not to display.

  Moreover, in the said embodiment, although the checkout system 1 comprised by the POS terminal 11 and the goods reading apparatus 101 was applied as a store system, it is not restricted to this, The function of the POS terminal 11 and the goods reading apparatus 101 is provided. Alternatively, the present invention may be applied to a single-unit apparatus. As a single device having the functions of the POS terminal 11 and the commodity reading device 101, there is a self-checkout device (hereinafter simply referred to as self-POS) installed and used in a store such as a supermarket.

  11 is an external perspective view showing the configuration of the self-POS 200, and FIG. 12 is a block diagram showing the hardware configuration of the self-POS 200. In addition, below, the same code | symbol is attached | subjected and shown about the same structure shown by FIG. 1 and FIG. 2, and the duplicate description is abbreviate | omitted. As shown in FIGS. 11 and 12, the main body 202 of the self-POS 200 includes a display device 106 having a touch panel 105 disposed on the surface thereof, and a product reading unit that reads product images for recognizing (detecting) product types and the like. 110 is provided.

  For example, a liquid crystal display is used as the display device 106. The display device 106 is a guidance screen for notifying the customer of the operation method of the self-POS 200, various input screens, a registration screen for displaying product information read by the product reading unit 110, a total price of goods, a deposit price, and change. Display the amount, etc. and display the payment screen for selecting the payment method.

  The product reading unit 110 reads a product image by the imaging unit 164 by a customer holding the code symbol attached to the product over the reading window 103 of the product reading unit 110.

  Further, on the right side of the main body 202, a product placement table 203 for placing unsettled products in a basket is provided, and on the left side of the main body 202, a product placement table 204 for placing settled products is provided. There are provided a bag hook 205 for hanging a bag for putting a product already settled, and a temporary placing table 206 for temporarily placing the settled product before putting it in the bag. The product placement tables 203 and 204 are provided with weighing machines 207 and 208, respectively, and have a function of confirming that the product weight is the same before and after settlement.

  In addition, the main body 202 of the self-POS 200 is provided with a changer 201 for depositing banknotes for payment and receiving fishing banknotes.

  When the self-POS 200 having such a configuration is applied to a store system, the self-POS 200 functions as an information processing apparatus.

  In addition, the program executed by each device of the above embodiment is provided by being incorporated in advance in a storage medium (ROM or storage unit) included in each device, but is not limited thereto, and can be installed in a form or executable. Various types of files may be recorded and provided on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk). Furthermore, the storage medium is not limited to a medium independent of a computer or an embedded system, but also includes a storage medium in which a program transmitted via a LAN, the Internet, or the like is downloaded and stored or temporarily stored.

  Further, the program executed by each device of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network, or via a network such as the Internet. It may be configured to be provided or distributed.

DESCRIPTION OF SYMBOLS 1 Store system 50 Special process execution means 51 Capture means 53 Similarity calculation means 54 Goods candidate extraction means 59 Determination means 101,200 Information processing apparatus 164 Imaging means 611 Sales registration processing means F1 Dictionary

JP 2003-173369 A

Claims (6)

Capture means for capturing a frame image including an image of the object imaged by the imaging means;
Similarity calculating means for calculating a similarity indicating how similar the image of the object included in the captured frame image and the reference image of each registered product registered in the dictionary;
Product candidate extraction means for extracting candidates for the registered product corresponding to the object based on the similarity;
Display means for displaying on the display device the candidate extracted by the product candidate extraction means as a provisionally confirmed product;
When it is determined that the color attribute of the frame image exceeds a predetermined reference value before the provisionally confirmed product is confirmed as the registered product corresponding to the object, a special predetermined for the provisionally confirmed product is determined. Special processing execution means for executing processing;
An information processing apparatus comprising: The special process execution means executes a registration cancel process for canceling the confirmation of the provisionally confirmed product as the special process.
The information processing apparatus according to claim 1. The special process execution means executes a confirmation process that considers that the same product is continuously confirmed as the special process.
The information processing apparatus according to claim 1. The special process execution means executes a key display process for displaying a numeric key for inputting the quantity of the provisionally confirmed product as the special process.
The information processing apparatus according to claim 1. Capture means for capturing a frame image including an image of the object imaged by the imaging means;
Similarity calculating means for calculating a similarity indicating how similar the image of the object included in the captured frame image and the reference image of each registered product registered in the dictionary;
Product candidate extraction means for extracting candidates for the registered product corresponding to the object based on the similarity;
Display means for displaying on the display device the candidate extracted by the product candidate extraction means as a provisionally confirmed product;
When it is determined that the color attribute of the frame image exceeds a predetermined reference value before the provisionally confirmed product is confirmed as the registered product corresponding to the object, a special predetermined for the provisionally confirmed product is determined. Special processing execution means for executing processing;
Sales registration processing means for performing sales registration processing on the registered product confirmed by the confirmation means;
Store system equipped with. Computer
Capture means for capturing a frame image including an image of the object imaged by the imaging means;
Similarity calculating means for calculating a similarity indicating how similar the image of the object included in the captured frame image and the reference image of each registered product registered in the dictionary;
Product candidate extraction means for extracting candidates for the registered product corresponding to the object based on the similarity;
Display means for displaying on the display device the candidate extracted by the product candidate extraction means as a provisionally confirmed product;
When it is determined that the color attribute of the frame image exceeds a predetermined reference value before the provisionally confirmed product is confirmed as the registered product corresponding to the object, a special predetermined for the provisionally confirmed product is determined. Special processing execution means for executing processing;
Program to function as.

Images