JP6306775B2 - Information processing apparatus and program - Google Patents

Description

Embodiments described herein relate generally to an information processing apparatus and a program.

  Conventionally, the type or the like of an article is recognized according to the degree of similarity obtained by comparing the feature quantity of a target article extracted from image data captured by an image sensor device with matching data (feature quantity) in a dictionary prepared in advance. There is a technique related to (detection) general object recognition (object recognition). In addition, a store system has been proposed in which the technology related to general object recognition is used to identify products such as fruits and vegetables, and sales of the identified products are registered.

  In the dictionary used for general object recognition described above, collation data (features) is registered in advance. However, if the matching data (feature value) registered in the dictionary is incomplete, the degree of similarity becomes low and the product is difficult to recognize.

  In such a case, recognition accuracy is improved by eliminating deficiencies in the matching data (features) registered in the dictionary by additional learning or the like.

  However, it is difficult to determine whether or not the additional learning of the matching data (features) registered in the dictionary is necessary because no criterion is indicated.

The problem to be solved by the present invention is to provide an information processing apparatus and program capable of informing whether or not additional learning of collation data (feature amount) in object recognition is necessary.

The information processing apparatus of the embodiment includes an imaging unit, a selection unit, a first display unit, an extracting unit, a similarity calculating unit, and a notification unit. The imaging unit images an article directed to the window. It said selection means, from a plurality of articles matching data for identifying an article which the imaging unit has captured is stored, to select the articles accuracy checked. When the article is selected by the selection means, the first display means displays a message indicating that the article is held over the window. The extraction means, the image pickup unit to extract a feature quantity of the articles contained in the image captured. The similarity calculation means compares the with the collation data, the feature quantity extracted by the extracting unit, the similarity is calculated for each article the reference data is stored. The notification means, in similarity for each article the similarity calculation means has calculated, if the similarity of the matching data of the article selected by the selecting means is not high, is selected by said selection means wherein notifying the updated reference data of the article was.

FIG. 1 is a perspective view illustrating an example of a checkout system according to the embodiment. FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal and the commodity reading apparatus. FIG. 3 is a conceptual diagram illustrating the data structure of a PLU file. FIG. 4 is a block diagram showing a functional configuration of the POS terminal. FIG. 5 is a diagram illustrating an example of a frame image captured by the image capturing unit. FIG. 6 is a diagram illustrating an example of the confirmation screen. FIG. 7 is a diagram illustrating an example of a confirmation screen. FIG. 8 is a diagram illustrating an example of a screen on which illustration images of product candidates are displayed. FIG. 9 is a flowchart showing the flow of the product recognition accuracy check process executed by the checkout system. FIG. 10 is an explanatory diagram illustrating an example of the accuracy check screen. FIG. 11 is an explanatory diagram illustrating an example of a captured image. FIG. 12 is an explanatory diagram illustrating an example of a cancellation execution screen. FIG. 13 is an explanatory diagram illustrating an example of a high recognition result screen. FIG. 14 is an explanatory diagram illustrating an example of a redetermination screen. FIG. 15 is an explanatory diagram illustrating an example of the middle recognition result screen. FIG. 16 is an explanatory diagram illustrating an example of a low recognition result screen. FIG. 17 is an explanatory diagram illustrating an example of the additional learning preparation screen. FIG. 18 is an external perspective view showing the configuration of the self-POS. FIG. 19 is a block diagram showing a hardware configuration of the self-POS.

  Hereinafter, an information processing apparatus, a store system, and a program according to the present embodiment will be described with reference to the drawings, taking a checkout system as an example. The store system is a checkout system (POS system) provided with a POS terminal that performs registration and settlement of commodities related to one transaction. This embodiment is an application example to a checkout system introduced in a store such as a supermarket.

  FIG. 1 is a perspective view illustrating an example of a checkout system 1 according to the embodiment. As shown in FIG. 1, the checkout system 1 includes a product reading device 101 that reads information about products, and a POS terminal 11 that performs registration and settlement of products related to one transaction. Below, the example which applies the POS terminal 11 as an information processing apparatus concerning this embodiment is demonstrated.

  The POS terminal 11 is placed on the upper surface of the drawer 21 on the checkout table 41. The drawer 21 is controlled by the POS terminal 11 for the opening operation. On the upper surface of the POS terminal 11, a keyboard 22 that is pressed by an operator (a store clerk) is arranged. A display device 23 that displays information toward the operator is provided behind the keyboard 22 when viewed from the operator who operates the keyboard 22. The display device 23 displays information on the display surface 23a. A touch panel 26 is laminated on the display surface 23a. A customer display device 24 is erected so as to be rotatable further behind the display device 23. The customer display device 24 displays information on the display surface 24a. The customer display device 24 shown in FIG. 1 has the display surface 24a facing the front side in FIG. 1, but the customer display device 24 is rotated so that the display surface 24a faces the back side in FIG. Thus, the customer display device 24 displays information to the customer.

  A horizontally long table-shaped counter table 151 is arranged so as to form an L shape with the checkout table 41 on which the POS terminal 11 is placed. A load receiving surface 152 is formed on the upper surface of the counter table 151. A shopping basket 153 for storing the product A is placed on the cargo receiving surface 152. The shopping basket 153 may be divided into a first shopping basket 153a brought in by a customer and a second shopping basket 153b placed at a position sandwiching the commodity reading device 101 from the first shopping basket 153a. it can. The shopping basket 153 is not limited to a so-called basket shape, and may be a tray or the like. The shopping basket 153 (second shopping basket 153b) is not limited to a so-called basket shape, and may be a box shape, a bag shape, or the like.

  On the receiving surface 152 of the counter stand 151, the commodity reading apparatus 101 connected to the POS terminal 11 so as to be able to transmit and receive data is installed. The commodity reading apparatus 101 includes a thin rectangular housing 102. A reading window 103 is disposed in front of the housing 102. A display / operation unit 104 is attached to the upper portion of the housing 102. The display / operation unit 104 is provided with a display device 106 which is a display unit having a touch panel 105 laminated on the surface thereof. A keyboard 107 is disposed on the right side of the display device 106. A card reading groove 108 of a card reader (not shown) is provided on the right side of the keyboard 107. A customer display device 109 for providing information to the customer is installed on the left rear side of the display / operation unit 104 as viewed from the operator.

  Such a product reading apparatus 101 includes a product reading unit 110 (see FIG. 2). The product reading unit 110 arranges an imaging unit 164 (see FIG. 2) on the back side of the reading window 103.

  In the first shopping basket 153a brought in by the customer, the commodity A related to one transaction is stored. The product A in the first shopping basket 153a is moved to the second shopping basket 153b by an operator who operates the product reading apparatus 101. In this movement process, the product A is directed to the reading window 103 of the product reading apparatus 101. At this time, the imaging unit 164 (see FIG. 2) arranged in the reading window 103 images the product A.

  The merchandise reading apparatus 101 displays a screen for designating which merchandise A included in the image captured by the imaging unit 164 corresponds to which merchandise registered in a PLU file F1 (see FIG. 3) described later. Displayed on the operation unit 104 and notifies the POS terminal 11 of the product ID of the specified product. In the POS terminal 11, based on the product ID notified from the product reading device 101, information related to sales registration such as the product classification, product name, and unit price of the product corresponding to the product ID is stored in a sales master file (not shown) or the like. And record sales.

  FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal 11 and the commodity reading apparatus 101. The POS terminal 11 includes a microcomputer 60 as an information processing unit that executes information processing. The microcomputer 60 is configured by connecting a ROM (Read Only Memory) 62 and a RAM (Random Access Memory) 63 to a CPU (Central Processing Unit) 61 that executes various arithmetic processes and controls each unit.

  The drawer 61, the keyboard 22, the display device 23, the touch panel 26, and the customer display device 24 are all connected to the CPU 61 of the POS terminal 11 through various input / output circuits (all not shown). These are controlled by the CPU 61.

  The keyboard 22 includes a numeric key 22d, a temporary fastening key 22e, and a fastening key 22f on which numbers such as “1”, “2”, “3”... Including.

  An HDD 64 (Hard Disk Drive) is connected to the CPU 61 of the POS terminal 11. The HDD 64 stores programs and various files. All or a part of the programs and various files stored in the HDD 64 are copied to the RAM 63 and executed by the CPU 61 when the POS terminal 11 is activated. An example of programs stored in the HDD 64 is a product sales data processing program PR1 and a product recognition accuracy check program PR2. An example of a file stored in the HDD 64 is a PLU file F1 distributed and stored from the store computer SC.

  The PLU file F1 is a product file that stores information related to sales registration of the product A for each of the products A that are displayed and sold in the store. In the following description, the PLU file F1 is used as a dictionary, but the dictionary may be a file different from the PLU file F1. The dictionary stores collation data (features) for identifying the product A extracted from the captured image data for a plurality of products. When the dictionary is a file different from the PLU file F1, the collation data (feature amount) stored in the dictionary and the information (identification information) of the PLU file F1 are linked. The feature amount is a parameterization of appearance features such as the standard shape, surface color, pattern, and unevenness of the product.

  FIG. 3 is a conceptual diagram illustrating the data configuration of the PLU file F1. As shown in FIG. 3, the PLU file F1 includes, for each product A, a product ID that is uniquely assigned identification information, product information to which the product A belongs, product information such as product name, product type, unit price, and the like. It is a file that stores an illustration image showing the product A and feature quantities such as hues and surface irregularities read from the captured product image as product information of the product A. Note that the feature amount is data for collation used for determination of similarity described later. The PLU file F1 is configured to be readable by the product reading apparatus 101 via a connection interface 65 described later.

  Returning to FIG. 2, the communication interface 25 for executing data communication with the store computer SC is connected to the CPU 61 of the POS terminal 11 via an input / output circuit (not shown). The store computer SC is installed in a store backyard or the like. A PLU file F1 distributed to the POS terminal 11 is stored in the HDD (not shown) of the store computer SC.

  Further, the CPU 61 of the POS terminal 11 is connected to a connection interface 65 that enables data transmission / reception with the commodity reading apparatus 101. The product reading apparatus 101 is connected to the connection interface 65. A printer 66 that prints on receipts and the like is connected to the CPU 61 of the POS terminal 11. The POS terminal 11 prints the transaction content of one transaction on the receipt under the control of the CPU 61.

  The commodity reading apparatus 101 also includes a microcomputer 160. The microcomputer 160 is configured by connecting a ROM 162 and a RAM 163 to a CPU 161 via a bus. The ROM 162 stores a program executed by the CPU 161. An imaging unit 164 and an audio output unit 165 are connected to the CPU 161 via various input / output circuits (all not shown). The operations of the imaging unit 164 and the audio output unit 165 are controlled by the CPU 161. The display / operation unit 104 is connected to the product reading unit 110 and the POS terminal 11 via the connection interface 176. The operation of the display / operation unit 104 is controlled by the CPU 161 of the product reading unit 110 and the CPU 61 of the POS terminal 11.

  The imaging unit 164 is a color CCD image sensor, a color CMOS image sensor, or the like, and is an imaging unit that performs imaging from the reading window 103 under the control of the CPU 161. For example, the imaging unit 164 captures a moving image of 30 fps (Flame Per Second). Frame images (captured images) sequentially captured at a predetermined frame rate by the imaging unit 164 are stored in the RAM 163.

  The audio output unit 165 is an audio circuit and a speaker for generating a preset warning sound or the like. The audio output unit 165 performs warning sound or audio notification under the control of the CPU 161.

  Further, a connection interface 175 that connects to the connection interface 65 of the POS terminal 11 and enables data transmission / reception with the POS terminal 11 is connected to the CPU 161. In addition, the CPU 161 transmits and receives data to and from the display / operation unit 104 via the connection interface 175.

  Next, a functional configuration of the CPU 61 that is realized when the CPU 61 executes programs (a product sales data processing program PR1 and a product recognition accuracy check program PR2) will be described. FIG. 4 is a block diagram showing a functional configuration of the POS terminal 11.

  FIG. 4 is a block diagram showing a functional configuration of the POS terminal 11. As illustrated in FIG. 4, the CPU 61 of the POS terminal 11 executes the product sales data processing program PR1 and the product recognition accuracy check program PR2, thereby performing an image capturing unit 51, a product detection unit 52, and a similarity calculation unit. 53, a similarity determination unit 54, a product presentation unit 55, an input reception unit 57, an information input unit 58, a sales registration unit 59 as a sales registration processing means, an article designation unit 91, a notification unit 92, and an additional learning unit 93. It has a function.

(Product registration process and sales registration process)
First, general object recognition (object recognition) by the image capture unit 51, the product detection unit 52, the similarity calculation unit 53, the similarity determination unit 54, the product presentation unit 55, the input reception unit 57, and the information input unit 58 of the POS terminal 11. The outline of the merchandise registration process according to) and the sales registration process by the sales registration unit 59 will be described.

  The image capturing unit 51 outputs an imaging on signal to the imaging unit 164 and causes the imaging unit 164 to start an imaging operation. The image capturing unit 51 sequentially captures frame images captured by the image capturing unit 164 and stored in the RAM 163 after the start of the image capturing operation. The capturing of the frame image by the image capturing unit 51 is performed in the order stored in the RAM 163.

  FIG. 5 is a diagram illustrating an example of a frame image captured by the image capturing unit 51. As shown in FIG. 5, when the operator holds the product A over the reading window 103, all or part of the product A is photographed and displayed on the display device 106 of the product reading apparatus 101 in the reading area R of the imaging unit 164. The

  The product detection unit 52 functions as an extraction unit. The product detection unit 52 detects the product A included in the frame image captured by the image capturing unit 164 and captured by the image capturing unit 51, and extracts a feature amount. The product detection unit 52 extracts the feature amount of the captured product A by detecting all or part of the product A included in the frame image using a pattern matching technique or the like. Specifically, a contour line or the like is extracted from an image obtained by binarizing the captured frame image. Next, the contour line extracted from the previous frame image is compared with the contour line extracted from the current frame image to detect the product A directed to the reading window 103 for sales registration.

  As another method for detecting the product A, the presence or absence of a skin color region is detected from the captured frame image. Next, when a skin color area is detected, that is, when a clerk's hand is detected, it is assumed that the clerk's hand is grasped by detecting the above-described contour line in the vicinity of the skin color area. Attempt to extract the contour of product A. At this time, when the contour indicating the shape of the hand and the contour of the other object in the vicinity of the contour of the hand are detected, the product detection unit 52 detects the product A from the contour of the object.

  Further, the product detection unit 52 reads the state of the surface such as the hue of the product A and the surface unevenness from the whole or a part of the detected product A as a feature amount. Note that the product detection unit 52 does not consider the outline or size of the product A in order to shorten the processing time.

  The similarity calculation unit 53 functions as a similarity calculation unit, and includes surface colors such as the color of product images of each product registered in the PLU file F1 (hereinafter referred to as registered product) and surface irregularities. The feature amount that is in the state and the product detection unit 52 detect the product A and compare the extracted feature amounts, thereby calculating the similarity between the product A and the registered product registered in the PLU file F1. Here, when the product image at the time of product registration of each product stored in the PLU file F1 is 100% = “similarity: 1.0”, the similarity is the total or a part of the product A. It shows how much is similar. Note that, for example, the similarity may be calculated by changing the weight between the color tone and the surface roughness.

Recognizing an object contained in an image in this way is called generic object recognition. Regarding such general object recognition, various recognition techniques are described in the following documents.
Keiji Yanai, “Current Status and Future of General Object Recognition”, IPSJ Journal, Vol. 48, no. SIG16 [Search August 10, 2010], Internet <URL: http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

Further, techniques for performing general object recognition by dividing an image into regions for each object are described in the following documents.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, [August 10, 2010 search], Internet <URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1. 1.145.3036 & rep = rep1 & type = pdf>

  Note that the method for calculating the similarity between the captured image of the product A and the registered product registered in the PLU file F1 is not particularly limited. For example, the similarity between the captured image of the product A and each registered product registered in the PLU file F1 may be calculated as an absolute evaluation or may be calculated as a relative evaluation.

  When calculating the similarity as an absolute evaluation, the captured image of the product A and each registered product registered in the PLU file F1 are compared on a one-to-one basis, and the similarity derived as a result of this comparison is used as it is. do it. Further, when the similarity is calculated as a relative evaluation, if five registered products (products AA, AB, AC, AD, and AE) are registered in the PLU file F1, the captured product A is compared with the product AA. The similarity is 0.6, the similarity is 0.1 for the product AB, the similarity is 0.1 for the product AC, the similarity is 0.1 for the product AD, and the product AE On the other hand, the similarity may be calculated such that the similarity is 0.1, and the total similarity with each registered product is 1.0 (100%).

  The similarity determination unit 54 compares the similarity between the image of the product A and the registered product registered in the PLU file F1 for each frame image captured by the image capturing unit 51. In the present embodiment, a plurality of conditions for the similarity between the product image of the registered product and the image of the product A are provided in stages, and the similarity determination unit 54 determines the registered product according to the satisfied condition or A candidate for product A is selected. Although the conditions regarding the degree of similarity are not particularly limited, the case where conditions a to d are used will be described below.

  Here, the condition a and the condition b are the first condition according to the present embodiment, and are for determining the product A captured by the imaging unit 164 as one product among the registered products registered in the PLU file F1. It is a condition. The condition c is a second condition according to the present embodiment. When the registered products registered in the PLU file F1 do not include a plurality of articles of different varieties belonging to the same category (product), This is a condition for extracting a candidate for the product A imaged by the imaging unit 164. Furthermore, the condition d is the third condition according to the present embodiment, and when a plurality of articles of different varieties belonging to the same category (product) are included in the candidates for the goods A that satisfy the condition c, This is a condition for extracting a candidate for the product A imaged by the imaging unit 164.

  The similarity determination unit 54 determines (determines) a registered product that satisfies the condition a or the condition b as a product that corresponds to the product A captured by the imaging unit 164 on a one-to-one basis (hereinafter referred to as a confirmed product). The similarity determination unit 54 determines that the registered product satisfying the condition c is not a confirmed product but a candidate for the product A imaged by the imaging unit 164 (hereinafter referred to as a product candidate). Then, product candidates for the product A are extracted by extracting registered products satisfying the condition c from a plurality of registered products registered in the PLU file F1.

  The similarity determination unit 54 also regards the registered product that satisfies the condition d (articles of different varieties belonging to the same category (product)) as candidates for the product A captured by the image capturing unit 164 and not a confirmed product. judge. Then, a product candidate for the product A is extracted by extracting a registered product satisfying the condition d from a plurality of registered products registered in the PLU file F1.

  The details of the conditions a to c are not particularly limited as long as the conditions a to c are set stepwise according to the degree of similarity, but as an example, the conditions a to c can be provided by a plurality of preset threshold values. Here, the case where the conditions a to c are set by the first threshold value to the third threshold value will be described. Note that the magnitude relationship between the first to third thresholds is first threshold> second threshold> third threshold.

  The similarity determination unit 54 counts the number of times that the degree of similarity with the registered product is equal to or higher than a predetermined first threshold (for example, 90%), and the condition a is satisfied when the number of times exceeds a predetermined number. It is determined that When the first threshold is set sufficiently high so that no erroneous determination is made, the condition a may be determined by setting the predetermined number of times as one.

  The similarity determination unit 54 determines that the condition b is satisfied when the similarity to the registered product is less than the first threshold (for example, 90%) and equal to or greater than the second threshold (for example, 75%) that is smaller than the first threshold. Determined to be satisfied. Then, it is determined that the registered product that satisfies the condition b is a confirmed product, but requires a confirmation operation by the operator. Note that the number of times the similarity with the registered product is less than the first threshold (for example, 90%) and the second threshold value (for example, 75%) is less than the first threshold is counted, It may be determined that the condition b is satisfied.

  Furthermore, the similarity determination unit 54 determines that the condition c is equal to or less than the second threshold (for example, 75%) and the third threshold (for example, 10%) that is smaller than the second threshold, when the similarity to the registered product is less than the second threshold (for example, 75%). Determined to be satisfied. Furthermore, the number of times that the degree of similarity with the registered product is less than the second threshold value (for example, 75%) and the third threshold value (for example, 10%) that is less than the second threshold value is counted, and this number is a predetermined number of times. You may determine with the condition c being satisfied when it becomes above.

  Each condition a to c can be appropriately set according to the degree of similarity, the number of determinations, and the like, and is not limited to the above-described example. Moreover, the predetermined number of times used for the determination of the conditions a to c may be different for each condition.

  In addition, when the registered product satisfying the condition c includes a plurality of articles of different varieties belonging to the same category (product), the similarity determination unit 54 adds the similarities of the plurality of varieties. When the similarity of the category (product) obtained by adding up the similarities of a plurality of varieties is equal to or higher than a predetermined second threshold (for example, 75%), it is determined that the condition d is satisfied.

  Further, the similarity determination unit 54 creates a ranking in which the registered products are ranked according to the degree of similarity.

  The product presentation unit 55 informs an operator or customer that the product A imaged by the imaging unit 164 is uniquely determined as a registered product satisfying the condition a or the condition b by image output or audio output. Inform.

  More specifically, the product presentation unit 55 displays a confirmation screen 71 (see FIG. 6) indicating that the registered product satisfying the condition a is uniquely confirmed as a product (confirmed product) imaged by the imaging unit 164. Is displayed on the display device 106.

  FIG. 6 is a diagram illustrating an example of the confirmation screen 71. When there is a registered product satisfying the condition a, the product presentation unit 55 stops displaying the captured image in the reading region R (see FIG. 5), and displays the illustration image G1 corresponding to the confirmed product and the product name “carrot” as PLU. The data is read from the file F1 and displayed on the confirmation screen 71. Further, the product presentation unit 55 displays the product name and product price (unit price) of the confirmed product read from the PLU file F1 in the product name display area 81 and the price display area 82, respectively. The product presentation unit 55 may display a product image (photograph) read from the PLU file F1 instead of the illustration image G1. The present invention is not limited to this, and the product name may be displayed on the confirmation screen 71 without displaying the illustration or the product image. In this case, the voice output unit 165 notifies that the product has been confirmed. The voice may be a product name registered in advance linked to a product, or may be an electronic sound such as “beep”.

  Further, the product presentation unit 55 displays a confirmation screen 72 (see FIG. 7) for receiving a final confirmation operation as to whether or not the registered product (determined product) that satisfies the condition b is the product A imaged by the imaging unit 164. It is displayed on the display device 106.

  FIG. 7 is a diagram illustrating an example of the confirmation screen 72. When there is a registered product satisfying the condition b, the product presentation unit 55 reads the illustration image G1 corresponding to the confirmed product from the PLU file F1 and displays it on the confirmation screen 72. In addition, the product presentation unit 55 uses a product name of the confirmed product read from the PLU file F1, and a message asking whether the read product A is a product of the illustration image G1 is “carrot?” Or the like. Is displayed. Further, the confirmation screen 72 is provided with buttons such as “Yes / No” so as to be selectable by a touch operation on the touch panel 105.

  Thus, on the confirmation screen 72, as a result of the similarity determination, the product name and product image of the registered product (confirmed product) uniquely selected for one product A are shown, and the product A and the registered product are displayed. Are displayed in a one-to-one relationship. Therefore, the confirmation screen 72 is a screen for notifying that the registered product satisfying the condition b is uniquely determined as the product A imaged by the imaging unit 164.

  In addition, the product presentation unit 55 causes the display device 106 to display information on the registered product that satisfies the condition c as a product candidate. More specifically, the product presentation unit 55 reads the illustration image and product name of the registered product that satisfies the condition c from the PLU file F1, and the display unit 106 determines whether the similarity calculated by the similarity calculation unit 53 is high. Output sequentially. The display device 106 displays the product candidate illustration images and product names in descending order of similarity in the product candidate presentation area 83 (see FIG. 8).

  FIG. 8 is a diagram illustrating an example of a screen on which illustration images G1, G2, G3, and G4 of product candidates are displayed. As illustrated in FIG. 8, in the product candidate presentation area 83, product candidate illustration images G1, G2, G3, and G4 and product names are displayed in order from registered products with a high degree of similarity. These illustration images G1, G2, G3, and G4 are configured to be selectable according to a selection operation on the touch panel 105. In addition, a selection button 84 for selecting the product A from the product list is provided below the product candidate presentation area 83, and the product selected from the product list is processed as the above-described confirmed product. Although FIG. 8 shows an example in which four product candidates corresponding to the illustration images G1 to G4 are displayed, the number of product candidates and the display method are not particularly limited. Further, instead of the illustration image, a product image (photo) may be displayed as a product candidate.

  In addition, the product presentation unit 55 causes the display device 106 to display information about registered products that are products of different varieties belonging to the same category (product) that satisfies the condition d as product candidates. More specifically, the product presentation unit 55 reads from the PLU file F1 illustration images and product names of registered products (articles of different varieties belonging to the same category (product)) that satisfy the condition d, and the similarity calculation unit 53 Are sequentially output to the display device 106 from those with the high similarity.

  The input receiving unit 57 receives various input operations corresponding to the display on the display device 106 via the touch panel 105 or the keyboard 107. For example, the input receiving unit 57 receives an input operation (confirmation operation) indicating that the product of the displayed illustration image G1 is finally confirmed based on a selection operation on the confirmation screen 72 (see FIG. 7). . The product presentation unit 55 displays the above-described confirmation screen 71 on the display device 106 when the input reception unit 57 receives a confirmation operation.

  The input receiving unit 57 receives a selection operation for any one of the illustration images G1 to G4 (see FIG. 8) of the product candidates displayed on the display device 106. The input receiving unit 57 receives the registered product of the selected illustration image as a confirmed product for the product A. When the product detection unit 52 can detect a plurality of products A, the input reception unit 57 may receive a selection operation of a plurality of product candidates from the product candidates. When the input receiving unit 57 receives the selection operation, the product presentation unit 55 causes the display device 106 to display a confirmation screen 71 that displays the accepted product candidate as a confirmed product.

  The information input unit 58 inputs information indicating the product A (for example, product ID, product name, etc.) via the connection interface 175 for the confirmed product confirmed as described above.

  The information input unit 58 may input the number of sales separately input via the touch panel 105 or the keyboard 107 together with the product ID and the like.

  The sales registration unit 59 registers the sales of the corresponding product A based on the product ID and the sales quantity input from the information input unit 58. Specifically, the sales registration unit 59 refers to the PLU file F1 and records the notified product ID and the product classification, product name, unit price, etc. corresponding to the product ID in the sales master file etc. together with the number of sales. To register sales.

(Product recognition accuracy check process for object recognition)
Next, the product recognition accuracy at the time of object recognition by the image capturing unit 51, the product detection unit 52, the similarity calculation unit 53, the similarity determination unit 54, the article designation unit 91, the notification unit 92, and the additional learning unit 93 of the POS terminal 11. The check process will be described.

  As described above, the POS terminal 11 uses the feature amount of the target article (object) extracted from the image data captured by the imaging unit 164 and the matching data (features) in the PLU file F1 that is a dictionary prepared in advance. General object recognition (object recognition) that recognizes (detects) the type or the like of the article according to the degree of similarity compared with the amount) is employed.

  In the PLU file F1, which is a dictionary used for such general object recognition, collation data (features) is registered in advance. However, if the matching data (features) registered in the PLU file F1, which is a dictionary, is incomplete, the degree of similarity is low even when the product A is held over appropriately, and the product A is recognized. It becomes difficult to be done.

  Therefore, the POS terminal 11 has an additional learning function for updating data for collation of the PLU file F1, which is a dictionary. However, if you notice a deficiency in the verification data (features) during the business hours of the store, you will have to temporarily suspend your work and learn more, which is very time consuming.

  However, as to whether or not the additional learning of the collation data (feature value) is necessary, a determination criterion is not shown, and it is difficult to determine whether or not to perform additional learning.

  Therefore, the POS terminal 11 according to the present embodiment can notify whether or not the additional learning of the verification data (feature amount) in the object recognition is necessary. That is, the CPU 61 of the POS terminal 11 executes the product recognition accuracy check program PR2, and as shown in FIG. 4, the image capturing unit 51, the product detection unit 52, the similarity calculation unit 53, and the similarity determination unit. 54, functions as an article designation unit 91, a notification unit 92, and an additional learning unit 93. In the following, each unit in the product recognition accuracy check process for object recognition will be described.

  The article specifying unit 91 functions as a selection unit, and selects a target product for a product recognition accuracy check from a PLU file F1 which is a dictionary prepared in advance. The article designating unit 91 designates the target product for the product recognition accuracy check when the operator selects the product name of the product for the product recognition accuracy check from the list of registered products stored in the PLU file F1.

  The notification unit 92 notifies the result of the product recognition accuracy check process. When the similarity of the target product specified by the article specifying unit 91 is not high in the ranking of the similarity of the selected registered product among the products registered in the PLU file F1 used as the dictionary, the notification unit 92 In addition, it informs about additional learning of the verification data (feature value) of the selected registered product. In that case, the alerting | reporting part 92 alert | reports in three steps according to the similarity degree of object goods. The notification stage may be two stages or four or more stages.

  Next, details of the three stages notified by the notification unit 92 will be described.

  First, a description will be given of notification when the ranking of the ranking of registered products selected as a target of the product recognition accuracy check process is high. When the ranking of the similarity of the registered product selected as the target of the product recognition accuracy check process is high, the notification unit 92 displays and notifies a screen indicating that there is no need to update by additional learning. Here, the ranking ranking high in the similarity of the selected registered products indicates that the ranking ranking satisfies the conditions of at least one of the first ranking and the confirmed products.

  Next, a description will be given of notification when the ranking of the ranking of the similarity of the registered products selected as the target of the product recognition accuracy check process is medium. When the ranking ranking in the similarity of the registered product selected as the target of the product recognition accuracy check process is medium, the notification unit 92 displays and notifies a screen indicating that updating by additional learning is recommended. Here, the ranking of the ranking in the similarity of the selected registered products indicates that the product is a candidate product excluding the first ranking. That is, the ranking rank is the second rank to the fourth rank. In addition, the candidate products excluding the first ranking are not limited to the second to fourth, but may include the fifth or lower.

  Next, the notification when the ranking of the ranking of the similarity of the registered products selected as the target of the product recognition accuracy check process is low will be described. When the ranking of the similarity of the registered products selected as the target of the product recognition accuracy check process is low, the notification unit 92 displays a screen indicating that additional learning is necessary for notification. Here, the low ranking ranking in the similarity of the selected registered product indicates that the similarity is lower than that of the candidate product. That is, the ranking is a ranking such as 5th or lower. Note that the notification method is not limited to the screen display and may be notified by voice.

  The additional learning unit 93 additionally learns (additional registration) the data (feature amount) for matching the products registered in advance in the PLU file F1. Specifically, the additional learning unit 93 re-takes and updates an image that becomes data for comparison (feature amount) and captures it again. The matching data (feature amount) can be stored (stored) in a certain amount. When the matching data (feature amount) reaches a certain amount, the data is deleted from the old matching data (feature amount) or the matching data (feature amount) is deleted at random.

  Next, the operation of the checkout system 1 in the product recognition accuracy check process will be described in detail. FIG. 9 is a flowchart showing the flow of the product recognition accuracy check process executed by the checkout system 1. As a precondition, it is assumed that the product recognition accuracy check program PR2 is activated.

  As shown in FIG. 9, the CPU 61 (article specifying unit 91) of the POS terminal 11 displays a product selection screen (not shown) (step S11). The product selection screen is a screen for displaying a list of registered products registered in the dictionary and selecting a product to be subjected to product recognition accuracy check processing from the registered products. The operator can select the product A to be subjected to the product recognition accuracy check process by pressing the product button displayed on the product selection screen.

  Next, the CPU 61 (article specifying unit 91) of the POS terminal 11 determines whether or not a press of a product button is detected (step S12). When the CPU 61 (article specifying unit 91) of the POS terminal 11 does not detect pressing of the product button (step S12; No), the CPU 61 (article specifying unit 91) of the POS terminal 11 stands by in step S12.

  When pressing of the product button is detected (step S12; Yes), the CPU 61 of the POS terminal 11 displays the accuracy check screen G5 (see FIG. 10) (step S13).

  FIG. 10 is an explanatory diagram showing an example of the accuracy check screen G5. The accuracy check screen G5 is a screen that displays the status of the product recognition accuracy check process and the determination result. The accuracy check screen G5 roughly includes a product display area R51, a normal recognition determination area R52, a real-time recognition determination area R53, and a stop button B51.

  The product display area R <b> 51 is an area for displaying an image captured by the image capturing unit 51 by the image capturing unit 164 capturing a target product for the product recognition accuracy check process. The product display area R51 includes a product name area R511 and an imaging area R512. The product name area R511 is an area in which the product name of a target product for which a pre-selected product recognition accuracy check is performed is displayed. The imaging region R512 is a region for displaying an image captured by the imaging unit 164. By displaying the image in the imaging region R512, the operator can confirm that the imaging unit 164 is imaging the product A.

  The normal recognition determination area R52 is an area in which the determination result of the product recognition accuracy check process is displayed. The normal recognition determination area R52 includes a determination result area R521 and a message display area R522. The determination result area R521 is an area for displaying the determination result. The message display area R522 is an area for displaying a message in the product recognition accuracy check process. Therefore, the message display area R522 displays a message “Please hold the product” in the initial state.

  The real-time recognition determination area R53 is an area for displaying the recognition result in real time by displaying the recognition result in a single frame. The cancel button B51 is a button to be pressed when canceling the product recognition accuracy check process.

  Next, the CPU 61 of the POS terminal 11 displays the captured image G51 captured by the image capturing unit 164 and captured by the image capturing unit 51 in the product display region R51 (step S14).

  FIG. 11 is an explanatory diagram illustrating an example of the captured image G51. The captured image G51 is an image captured by the image capturing unit 164 and captured by the image capturing unit 51. The captured image G51 displays the product A surrounded by a frame line. Further, the captured image G51 displays a message indicating that the product A cannot be read when the imaging method is incomplete. FIG. 11 shows a state where the image of cabbage is captured and the message “reading NG” is displayed.

  Next, the CPU 61 of the POS terminal 11 determines whether or not the pressing of the stop button B51 provided in the accuracy check screen G5 is detected (step S15). When pressing of the cancel button B51 is detected (step S15; Yes), the CPU 61 of the POS terminal 11 displays a cancel execution screen G6 (step S16).

  FIG. 12 is an explanatory diagram showing an example of the cancellation execution screen G6. The cancel execution screen G6 is a screen for confirming that the product recognition accuracy check process is canceled when the cancel button B51 is pressed. Therefore, the cancel execution screen G6 displays a message “Are you sure you want to cancel the accuracy check?”. The cancel execution screen G6 includes a yes button B61 and a no button B62. The Yes button B61 is a button to be pressed when canceling the product recognition accuracy check process. The No button B62 is a button that is pressed when canceling the cancellation of the product recognition accuracy check process.

  Next, the CPU 61 of the POS terminal 11 determines whether or not the pressing of the Yes button B61 provided on the cancellation execution screen G6 is detected (Step S17). When pressing of the Yes button B61 is detected (step S17; Yes), the CPU 61 of the POS terminal 11 ends the product recognition accuracy check process.

  On the other hand, when the pressing of the Yes button B61 is not detected (step S17; No), the CPU 61 of the POS terminal 11 determines whether or not the pressing of the No button B62 provided in the stop execution screen G6 is detected ( Step S18).

  When pressing of the No button B62 is not detected (step S18; No), the CPU 61 of the POS terminal 11 returns to step S17. On the other hand, when pressing of the No button B62 is detected (step S18; Yes), the CPU 61 of the POS terminal 11 returns to step S13.

  When pressing of the stop button B51 provided in the accuracy check screen G5 is not detected (step S15; No), the CPU 61 (product detection unit 52) of the POS terminal 11 determines whether or not the extraction of the feature amount of the product A is completed. Is determined (step S19). When the feature amount extraction of the product A has not been completed (step S19; No), the CPU 61 (the product detection unit 52, the similarity calculation unit 53, and the similarity determination unit 54) of the POS terminal 11 returns to step S14. .

  On the other hand, when the feature amount extraction of the product A is completed (step S19; Yes), the CPU 61 (similarity calculation unit 53) of the POS terminal 11 calculates the similarity for the registered product (step S20). Next, the CPU 61 (similarity determination unit 54) of the POS terminal 11 creates a ranking in which the registered products are ranked according to the degree of similarity (step S21).

  Next, the CPU 61 of the POS terminal 11 determines whether or not the ranking rank in the similarity of the registered product selected as the target of the product recognition accuracy check process is high (step S22). Here, the high ranking ranking means that the ranking ranking is the first in the similarity level of the selected registered product, or that it satisfies at least one of the conditions of being a confirmed product with a high similarity level. is there.

  When the ranking in the similarity of the selected registered product is high (step S22; Yes), the CPU 61 (notification unit 92) of the POS terminal 11 displays the high recognition result screen G52 in the normal recognition determination region R52. (Step S23).

  FIG. 13 is an explanatory diagram showing an example of the high recognition result screen G52. The high recognition result screen G52 is a screen that displays a confirmed product with a high degree of similarity and does not require additional learning. Here, the additional learning is to add and store product matching data (feature amount) in a dictionary in which matching data (feature amount) is registered. In the high recognition result screen G52, a mark indicating that the degree of similarity is high is displayed in the determination result region R521. The high recognition result screen G52 displays a message “No additional learning is necessary (automatically determinable product)” in the message display area R522.

  Next, the CPU 61 of the POS terminal 11 determines whether or not the product A is out of frame from the captured image G51 captured by the image capturing unit 164 and captured by the image capturing unit 51 (step S24). When the product A is not out of frame (step S24; No), the CPU 61 of the POS terminal 11 determines whether or not pressing of the stop button B51 is detected (step S27). On the other hand, when the product A is out of frame (step S24; Yes), the CPU 61 of the POS terminal 11 displays the redetermination screen G55 in the imaging region R512 (step S25).

  FIG. 14 is an explanatory diagram showing an example of the redetermination screen G55. The redetermination screen G55 is a screen for asking the operator whether or not to perform the accuracy check again. The redetermination screen G55 is displayed in the imaging region R512 when the product A is out of frame after the accuracy check. The re-determination screen G55 displays a message “If re-determination, press the [Re-determination] button and hold the product over the scanner”. The redetermination screen G55 includes a redetermination button B54. The redetermination button B54 is a button that is pressed to return to the initial state and perform the determination in the product recognition accuracy check process again.

  Next, the CPU 61 of the POS terminal 11 determines whether or not pressing of the redetermination button B54 is detected (step S26). When pressing of the redetermination button B54 is detected (step S26; Yes), the CPU 61 of the POS terminal 11 returns to step S13 which is an initial state. On the other hand, when pressing of the redetermination button B54 is not detected (step S26; No), the CPU 61 of the POS terminal 11 determines whether pressing of the stop button B51 is detected (step S27).

  When pressing of the cancel button B51 is detected (step S27; Yes), the CPU 61 of the POS terminal 11 proceeds to step S16 and displays a cancel execution screen G6. On the other hand, when pressing of the stop button B51 is not detected (step S27; No), the CPU 61 of the POS terminal 11 returns to step S24.

  When the ranking order in the similarity of the selected registered product is not high (step S22; No), the CPU 61 of the POS terminal 11 ranks the ranking in the similarity of the registered product selected as the target of the product recognition accuracy check process. Is determined to be medium (step S28). Here, the ranking of the ranking means that it is a commodity candidate excluding the first ranking. Specifically, the ranking of the similarity of the selected registered products is from 2nd to 4th.

  When the ranking in the similarity of the selected registered product is medium (step S28; Yes), the CPU 61 (notification unit 92) of the POS terminal 11 displays the medium recognition result screen G53 in the normal recognition determination region R52. (Step S29).

  FIG. 15 is an explanatory diagram showing an example of the middle recognition result screen G53. The middle recognition result screen G53 is a screen that displays that the product candidate has a similarity other than the first and recommends additional learning.

  In the middle recognition result screen G53, a mark indicating that the similarity is within the allowable range is displayed in the determination result region R521. Then, the middle recognition result screen G53 displays a message “please learn more to increase the determination result” in the message display region R522. Further, the middle recognition result screen G53 displays an additional learning execution button B52a in the message display region R522. The additional learning execution button B52a is a button (operator) to be pressed when starting the additional learning process of the feature amount of the target product for the product recognition accuracy check.

  When the ranking order in the similarity of the selected registered product is not medium (step S28; No), the CPU 61 (notification unit 92) of the POS terminal 11 has a low ranking in the similarity of the selected registered product. If there is, the low recognition result screen G54 is displayed in the normal recognition determination region R52 (step S30). Here, the low ranking ranking in the similarity of the selected registered product indicates that the similarity is lower than the product candidate. Specifically, the ranking of the ranking of the similarity of the selected registered products is 5th or lower.

  FIG. 16 is an explanatory diagram illustrating an example of the low recognition result screen G54. The low recognition result screen G54 is a screen that displays that the similarity is lower than the product candidate and that additional learning is necessary. The low recognition result screen G54 is displayed when the product A cannot be detected. The case where the product A cannot be detected is a case where the whole or a part of the product A is not imaged, or a case where the product A is hidden by a hand holding the product A.

  In the low recognition result screen G54, a mark indicating that the degree of similarity is low is displayed in the determination result region R521. Then, the low recognition result screen G54 displays a message “additional learning is necessary” in the message display region R522. The low recognition result screen G54 displays an additional learning execution button B52b in the message display region R522. The additional learning execution button B52b is a button to be pressed when performing additional learning.

  Next, the CPU 61 of the POS terminal 11 performs the same processing (step S31 to step S34) as the processing (step S24 to step S27) that is executed when the ranking ranking in the similarity of the selected registered product is high. . The difference from the process executed when the ranking rank is high is in determining whether or not pressing of the additional learning execution buttons B52a and B52b for performing additional learning is detected.

  Specifically, when the pressing of the cancel button B51 is not detected (step S34; No), the CPU 61 of the POS terminal 11 determines whether or not the additional learning execution buttons B52a and B52b are pressed (step S35). .

  When pressing of the additional learning execution buttons B52a and B52b is not detected (step S35; No), the CPU 61 of the POS terminal 11 returns to step S31. On the other hand, when pressing of the additional learning execution buttons B52a and B52b is detected (step S35; Yes), the CPU 61 (additional learning unit 93) of the POS terminal 11 displays the additional learning preparation screen G7 (step S36).

  FIG. 17 is an explanatory diagram showing an example of the additional learning preparation screen G7. The additional learning preparation screen G7 is a screen that prompts the operator to prepare for starting the additional learning processing. Therefore, the additional learning preparation screen G7 displays a message “Please prepare three product samples to learn”. Further, the additional learning preparation screen G7 displays the product A to be prepared. FIG. 17 displays cabbage as the product A to be prepared. Further, the additional learning preparation screen G7 includes a cancel button B71 and an OK button B72. The cancel button B71 is a button that is pressed when canceling additional learning. The OK button B72 is a button to be pressed when preparation is completed and additional learning is started.

  Next, the CPU 61 of the POS terminal 11 determines whether or not the pressing of the OK button B72 is detected (step S37). When pressing of the OK button B72 is not detected (step S37; No), the CPU 61 of the POS terminal 11 determines whether pressing of the cancel button B71 is detected (step S38).

  When pressing of the cancel button B71 is not detected (step S38; No), the CPU 61 of the POS terminal 11 returns to step S37. On the other hand, when pressing of the cancel button B71 is detected (step S38; Yes), the CPU 61 of the POS terminal 11 returns to step S28 and displays a screen corresponding to the similarity.

  On the other hand, when pressing of the OK button B72 is detected (step S37; Yes), the CPU 61 (additional learning unit 93) of the POS terminal 11 starts additional learning (step S39) and ends the process.

  As described above, according to the present embodiment, the article designating unit 91 causes the target product for the product recognition accuracy check to be selected from the PLU file F1. The commodity detection unit 52 detects the commodity A included in the frame image captured by the image capturing unit 164 and captured by the image capturing unit 51, and extracts a feature amount. Then, the similarity calculation unit 53 compares the feature amount of each product registered in the PLU file F1 with the feature amount extracted by the product detection unit 52, thereby registering the registered product registered in the PLU file F1. The similarity is calculated for each. When the similarity of the product A selected by the article specifying unit 91 is not high in the PLU file F1, the notification unit 92 updates the matching data (feature amount) stored in the PLU file F1 by additional learning. Notify about what to do. Therefore, this embodiment can alert | report about the necessity of the additional learning of the data for collation (feature amount).

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  In the above embodiment, the POS terminal 11 is described as an example of the information processing apparatus. However, the information processing apparatus is not limited to the POS terminal 11. For example, a personal computer or a tablet terminal may be used. In this case, an imaging device such as a scanner may be connected to a personal computer or tablet terminal.

  Moreover, in the said embodiment, CPU61 of the POS terminal 11 which is information processing apparatus is the image capture part 51, the goods detection part 52, the similarity calculation part 53, the similarity determination part 54, the goods presentation part 55, an input reception part. 57, the information input unit 58, the sales registration unit 59, the article designation unit 91, the notification unit 92, and the additional learning unit 93 have been described. However, these functions may be provided other than the CPU 61 of the POS terminal 11. For example, the CPU 161 of the commodity reading apparatus 101 may be provided with all or part of the functions described above.

  Moreover, although the said embodiment demonstrated the data for collation as a feature-value, the data for collation are good also as the imaged product image (reference | standard image).

  Moreover, in the said embodiment, although the checkout system 1 comprised by the POS terminal 11 and the goods reading apparatus 101 was applied as a store system, it is not restricted to this, The function of the POS terminal 11 and the goods reading apparatus 101 is provided. Alternatively, the present invention may be applied to a single-unit apparatus. As a single device having the functions of the POS terminal 11 and the commodity reading device 101, there is a self-checkout device (hereinafter simply referred to as self-POS) installed and used in a store such as a supermarket.

  Here, FIG. 18 is an external perspective view showing the configuration of the self-POS 200, and FIG. 19 is a block diagram showing the hardware configuration of the self-POS 200. In addition, below, the same code | symbol is attached | subjected and shown about the same structure shown by FIG. 1 and FIG. 2, and the duplicate description is abbreviate | omitted. As shown in FIGS. 18 and 19, the main body 202 of the self-POS 200 is a product reading that reads a product image for recognizing (detecting) the display device 106 on which the touch panel 105 is disposed, the type of the product A, and the like. Part 110 is provided.

  For example, a liquid crystal display is used as the display device 106. The display device 106 includes a guidance screen for notifying the customer of the operation method of the self-POS 200, various input screens, a registration screen for displaying product information read by the product reading unit 110, a total price of the product A, a deposit price, The amount of change is displayed, and a payment screen for selecting a payment method is displayed.

  The product reading unit 110 is configured to read a product image by the imaging unit 164 by a customer holding the code symbol attached to the product A over the reading window 103 of the product reading unit 110.

  Further, a product placement table 203 is provided on the right side of the main body 202 for placing an unsettled product A in a basket, and a product placement table 204 is provided on the left side of the main body 202 for placing a product A that has been settled. In addition, a bag hook 205 for hanging a bag for putting a product A that has already been settled, and a temporary placing table 206 for temporarily placing the product A that has already been settled before being put in the bag are provided. The product placement tables 203 and 204 are provided with measuring devices 207 and 208, respectively, and have a function of confirming that the weight of the product A is the same before and after settlement.

  In addition, the main body 202 of the self-POS 200 is provided with a changer 201 for depositing banknotes for payment and receiving fishing banknotes.

  When the self-POS 200 having such a configuration is applied to a store system, the self-POS 200 functions as an information processing apparatus.

  In addition, the program executed by each device of the above embodiment is provided by being incorporated in advance in a storage medium (ROM or storage unit) included in each device, but is not limited thereto, and can be installed in a form or executable. Various types of files may be recorded and provided on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk). Furthermore, the storage medium is not limited to a medium independent of a computer or an embedded system, but also includes a storage medium in which a program transmitted via a LAN, the Internet, or the like is downloaded and stored or temporarily stored.

  Further, the program executed by each device of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network, or via a network such as the Internet. It may be configured to be provided or distributed.

DESCRIPTION OF SYMBOLS 11 POS terminal 52 Product detection part 53 Similarity calculation part 91 Article designation | designated part 92 Notification part 164 Imaging part F1 PLU file PR2 Product recognition accuracy check program G5 Accuracy check screen

JP 2003-173369 A

Claims (6)

An imaging unit for imaging an article directed to the window;
A selection means for selecting an accuracy check target article from a plurality of articles in which verification data for identifying the article imaged by the imaging unit is stored ;
When the article is selected by the selection means, a first display means for displaying a message indicating that the article is held over the window;
An extraction means for said imaging unit extracts a feature quantity of the articles contained in the image captured,
Wherein the verification data is compared with the feature amount extracted by the extracting unit, the similarity calculation means for calculating a similarity for each article the reference data is stored,
In similarity for each article the similarity calculation means has calculated, if the similarity of the matching data of the article selected by the selecting means is not high, matching the articles selected by said selection means An informing means for informing about the update of the business data ;
An information processing apparatus comprising: A display unit for displaying an image captured by the imaging unit;
If the imaging method is incomplete, a message indicating that the article cannot be read is displayed on the display unit.
The information processing apparatus according to claim 1. When an article is out of frame from an image captured by the imaging unit after the notification unit notifies, a screen asking whether to perform the accuracy check again on the display unit is displayed.
The information processing apparatus according to claim 2 . When the accuracy check is performed again on the screen, a message indicating that the article is held over the window is displayed.
The information processing apparatus according to claim 3 . An input means for receiving an input for instructing additional learning to register the data for verification extracted by the extraction means in addition to the data for verification of the accuracy check target when the notification means notifies;
Second display means for displaying a screen for prompting preparation of an article for additional learning with a specified number when the input means receives an instruction for additional learning;
The information processing apparatus according to claim 1, further comprising: A computer including an imaging unit that images an article directed to a window ;
A selection means for selecting an accuracy check target article from a plurality of articles in which verification data for identifying the article imaged by the imaging unit is stored ;
When the article is selected by the selection means, a first display means for displaying a message indicating that the article is held over the window;
An extraction means for said imaging unit extracts a feature quantity of the articles contained in the image captured,
Wherein the verification data is compared with the feature amount extracted by the extracting unit, the similarity calculation means for calculating a similarity for each article the reference data is stored,
In similarity for each article the similarity calculation means has calculated, if the similarity of the matching data of the article selected by the selecting means is not high, matching the articles selected by said selection means An informing means for informing about the update of the business data ;
Program to function as.

Images