JP5567606B2 - Information processing apparatus and program - Google Patents

Description

  Embodiments described herein relate generally to an information processing apparatus and a program.

  Conventionally, it relates to general object recognition in which feature quantities of an article are extracted from image data obtained by imaging the article, and compared with data (feature quantities) prepared in advance for recognition (detection) of the type of the article. There is a technology for recognizing products to be registered for sales by performing object recognition for identifying products such as fruits and vegetables using technology.

  By the way, in object recognition, when a plurality of products having similar feature quantities are included in the data for verification, a similar product different from the actual product may be recognized. However, in the above-described conventional technology, when another similar product is recognized by object recognition, an actual product is selected from a plurality of product lists by a key operation different from the operation for performing object recognition. There was a need. For this reason, for example, when another similar product is recognized by object recognition, it is necessary to release the product in hand to perform object recognition and perform a key operation to select the actual product. The nature was bad.

  In order to solve the above-described problem, an information processing apparatus according to an embodiment includes a capturing unit that sequentially captures images captured by an imaging unit, an article detecting unit that detects an article included in the captured image, and the article. Is included in the image based on the recognition means for recognizing the detected article from a plurality of articles of different types by reading the feature amount of the detected image and the sequentially captured images. Movement detecting means for detecting movement of the recognized article, and operation receiving means for receiving an operation instruction corresponding to the movement of the recognized article.

  Further, the program according to the embodiment includes a computer that controls the information processing apparatus, a capturing unit that sequentially captures images captured by the imaging unit, an article detecting unit that detects an article included in the captured image, and the article. Is included in the image based on the recognition means for recognizing the detected article from a plurality of articles of different types by reading the feature amount of the detected image and the sequentially captured images. It is made to function as a movement detecting means for detecting the movement of the recognized article and an operation receiving means for receiving an operation instruction corresponding to the movement of the recognized article.

FIG. 1 is a perspective view illustrating an example of a checkout system (store system) according to the embodiment. FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal and the commodity reading apparatus. FIG. 3 is a conceptual diagram illustrating the data structure of a PLU file. FIG. 4 is a block diagram illustrating functional configurations of the POS terminal and the commodity reading apparatus. FIG. 5 is a conceptual diagram illustrating an example of commodity detection. FIG. 6 is a flowchart illustrating an example of the operation of the commodity reading apparatus. FIG. 7 is a conceptual diagram illustrating a display example of the display device. FIG. 8 is a conceptual diagram illustrating an example of an operation instruction by moving a product. FIG. 9 is a flowchart showing an example of the operation of the POS terminal.

  Hereinafter, embodiments of an information processing apparatus and a program will be described in detail with reference to the accompanying drawings. The embodiment described below is an embodiment of the information processing apparatus and program, and does not limit the configuration, specifications, or the like. The present embodiment is an application example to a checkout system (store system) provided with a POS terminal that is introduced into a store such as a supermarket and performs registration / settlement of products for one transaction.

  FIG. 1 is a perspective view showing an example of a checkout system 1. As shown in FIG. 1, the checkout system 1 includes a product reading device 101 that reads information about a product, and a POS terminal 11 that performs registration and settlement of the product related to one transaction. Below, the example which applies the goods reading apparatus 101 as an information processing apparatus concerning this embodiment is demonstrated. In addition, below, the same code | symbol is attached | subjected and shown about the same structure shown by several drawing, The duplication description may be abbreviate | omitted.

  The POS terminal 11 is placed on the upper surface of the drawer 21 on the checkout table 41. The drawer 21 is controlled by the POS terminal 11 for the opening operation. On the upper surface of the POS terminal 11, a keyboard 22 that is pressed down by an operator (a store clerk) is arranged. A display device 23 that displays information toward the operator is provided behind the keyboard 22 when viewed from the operator who operates the keyboard 22. The display device 23 displays information on the display surface 23a. A touch panel 26 is laminated on the display surface 23a. A customer display device 24 is erected so as to be rotatable further back than the display device 23. The customer display device 24 displays information on the display surface 24a. The customer display device 24 shown in FIG. 1 has the display surface 24a facing the front side in FIG. 1, but the customer display device 24 is rotated so that the display surface 24a faces the back side in FIG. Thus, the customer display device 24 displays information to the customer.

  A horizontally long table-shaped counter table 151 is arranged so as to form an L shape with the checkout table 41 on which the POS terminal 11 is placed. A cargo receiving surface 152 is formed on the upper surface of the counter table 151. A shopping basket 153 for storing the product G is placed on the cargo receiving surface 152. The shopping basket 153 may be divided into a first shopping basket 153a brought in by a customer and a second shopping basket 153b placed at a position sandwiching the commodity reading device 101 from the first shopping basket 153a. it can. The shopping basket 153 is not limited to a so-called basket shape, and may be a tray or the like. The shopping basket 153 (second shopping basket 153b) is not limited to a so-called basket shape, and may be a box shape, a bag shape, or the like.

  On the receiving surface 152 of the counter table 151, the commodity reading apparatus 101 connected to the POS terminal 11 so as to be able to transmit and receive data is installed. The commodity reading apparatus 101 includes a thin rectangular housing 102. A reading window 103 is disposed in front of the housing 102. A display / operation unit 104 is attached to the upper portion of the housing 102. The display / operation unit 104 is provided with a display device 106 on which a touch panel 105 is laminated. A keyboard 107 is disposed on the right side of the display device 106. A card reading groove 108 of a card reader (not shown) is provided on the right side of the keyboard 107. A customer display device 109 for providing information to the customer is installed on the back left side of the display / operation unit 104 as viewed from the operator.

  Such a product reading apparatus 101 includes a product reading unit 110 (see FIG. 2). The merchandise reading unit 110 has an imaging unit 164 (see FIG. 2) disposed behind the reading window 103.

  In the first shopping basket 153a brought in by the customer, a product G related to one transaction is stored. The commodity G in the first shopping basket 153a is moved to the second shopping basket 153b by an operator who operates the commodity reading apparatus 101. In this movement process, the product G is directed to the reading window 103 of the product reading apparatus 101. At this time, the imaging unit 164 (see FIG. 2) arranged in the reading window 103 images the product G.

  In the commodity reading apparatus 101, referring to the PLU file F1 (see FIG. 3), the commodity G included in the image captured by the image capturing unit 164 is displayed as a barcode or QR code (registered trademark) attached to the commodity G. It is specified by reading code information or general object recognition (details will be described later). Next, the product reading apparatus 101 displays a screen for accepting confirmation of the identified product G, input of the number, etc., on the display / operation unit 104, and the product ID of the product G for which confirmation input has been accepted, Such information is notified to the POS terminal 11. In the POS terminal 11, based on information such as the product ID and the number notified from the product reading device 101, information related to sales registration such as product classification, product name, and unit price of the product corresponding to the product ID is stored in the sales master file ( (Not shown) and the like to record sales.

  FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal 11 and the commodity reading apparatus 101. The POS terminal 11 includes a microcomputer 60 as an information processing unit that executes information processing. The microcomputer 60 is configured by connecting a ROM (Read Only Memory) 62 and a RAM (Random Access Memory) 63 to a CPU (Central Processing Unit) 61 that executes various arithmetic processes and controls each unit.

  The drawer 61, the keyboard 22, the display device 23, the touch panel 26, and the customer display device 24 are all connected to the CPU 61 of the POS terminal 11 through various input / output circuits (all not shown). . These are controlled by the CPU 61.

  The keyboard 22 includes a numeric key 22d, a temporary fastening key 22e, and a fastening key 22f on which numbers such as “1”, “2”, “3”,...

  An HDD 64 (Hard Disk Drive) is connected to the CPU 61 of the POS terminal 11. The HDD 64 stores programs and various files. All or a part of the programs and various files stored in the HDD 64 are copied to the RAM 63 and executed by the CPU 61 when the POS terminal 11 is activated. An example of a program stored in the HDD 64 is a product sales data processing program PR. An example of a file stored in the HDD 64 is a PLU file F1 distributed and stored from the store computer SC.

  The PLU file F1 is a product file in which an association between information related to sales registration of the product G and an image of the product G is set for each product G to be displayed and sold in the store.

  FIG. 3 is a conceptual diagram illustrating the data configuration of the PLU file F1. As shown in FIG. 3, the PLU file F1 includes, for each product G, a product ID uniquely assigned to the product, information on the product such as a product classification to which the product G belongs, a product name, a unit price, and a product image obtained by capturing the product. (Reference image) is a file that stores product information of the product G. The product image is a captured image (photograph) of the product used for determining the similarity. The PLU file F1 is configured to be readable by the commodity reading apparatus 101 via a connection interface 65 described later.

  Note that the data configuration of the PLU file F1 is not limited to the example of FIG. 3, and for example, feature quantities such as color and surface irregularities read from a product image may be stored for each product.

  Returning to FIG. 2, a communication interface 25 for executing data communication with the store computer SC is connected to the CPU 61 of the POS terminal 11 via an input / output circuit (not shown). The store computer SC is installed in a store backyard or the like. A PLU file F1 distributed to the POS terminal 11 is stored in the HDD (not shown) of the store computer SC.

  Further, the CPU 61 of the POS terminal 11 is connected to a connection interface 65 that enables data transmission / reception with the commodity reading apparatus 101. The product reading apparatus 101 is connected to the connection interface 65. The CPU 61 of the POS terminal 11 is connected to a printer 66 that prints on receipts. The POS terminal 11 prints the transaction content of one transaction on the receipt under the control of the CPU 61.

  The commodity reading apparatus 101 also includes a microcomputer 160. The microcomputer 160 is configured by connecting a ROM 162 and a RAM 163 to a CPU 161 via a bus. The ROM 162 stores a program executed by the CPU 161. The CPU 161 is connected to an imaging unit 164 and an audio output unit 165 via various input / output circuits (none of which are shown). The operations of the imaging unit 164 and the audio output unit 165 are controlled by the CPU 161. The display / operation unit 104 is connected to the product reading unit 110 and the POS terminal 11 via the connection interface 176. The operation of the display / operation unit 104 is controlled by the CPU 161 of the product reading unit 110 and the CPU 61 of the POS terminal 11.

  The imaging unit 164 is a color image sensor such as a CCD (Charged Coupled Device) imaging device or a CMOS (Complementary Metal Oxide Semiconductor) imaging device, and is an imaging unit that performs imaging from the reading window 103 in color under the control of the CPU 161. is there. For example, the imaging unit 164 captures a 30 fps moving image. Frame images (captured images) sequentially captured at a predetermined frame rate by the imaging unit 164 are stored in the RAM 163.

  The audio output unit 165 is an audio circuit and a speaker for generating a preset warning sound or the like. The audio output unit 165 performs warning sound or audio notification under the control of the CPU 161.

  Furthermore, a connection interface 175 that connects to the connection interface 65 of the POS terminal 11 and enables data transmission / reception with the POS terminal 11 is connected to the CPU 161. In addition, the CPU 161 transmits and receives data to and from the display / operation unit 104 via the connection interface 175.

  Next, functional configurations of the CPU 161 and the CPU 61 that are realized by the CPU 161 and the CPU 61 executing programs will be described with reference to FIG.

  FIG. 4 is a block diagram illustrating functional configurations of the POS terminal 11 and the commodity reading apparatus 101. As illustrated in FIG. 4, the CPU 161 of the commodity reading apparatus 101 executes a program stored in the ROM 162, thereby causing an image capturing unit 51, a commodity detecting unit 52, a code reading unit 53, an object recognizing unit 54, and object movement detection. The function as the part 55, the input reception part 56, and the information output part 57 is provided. Similarly, the CPU 61 of the POS terminal 11 has a function as the sales registration unit 611 by executing the program PR.

  The image capturing unit 51 outputs an imaging on signal to the imaging unit 164 and causes the imaging unit 164 to start an imaging operation. The image capturing unit 51 sequentially captures color frame images captured by the image capturing unit 164 and stored in the RAM 163 after the start of the image capturing operation. The capturing of the frame image by the image capturing unit 51 is performed in the order stored in the RAM 163.

  The product detection unit 52 detects all or part of the product G included in the frame image captured by the image capture unit 51 using a pattern matching technique or the like. Specifically, an outline or the like is extracted from an image obtained by binarizing the captured frame image. Next, the contour line extracted from the latest frame image is compared with the contour line extracted from the current frame image, and the part that has changed, that is, the product directed to the reading window 103 for sales registration G reflection is detected. As another method for detecting the product G, the presence or absence of a skin color region is detected from the captured frame image. Next, when the skin color area is detected, that is, when the reflection of the clerk's hand is detected, the product G that is assumed to be gripped by the clerk's hand by detecting the above-described contour line. Attempt to extract contours. At this time, when the contour indicating the shape of the hand and the other contours are detected, since the store clerk's hand holds the product G, the reflection of the product G is detected.

  FIG. 5 is a conceptual diagram illustrating an example of detection of the product G. As shown in FIG. 5, the image capturing unit 51 compares the latest frame images G10 and G20 with each other, the frame image G10 in which nothing is captured from the difference in luminance gradation, and the product G or the clerk's hand H. The frame image G20 having a reflection of some object such as is identified. Next, the image capturing unit 51 detects the product G by performing image processing such as the above-described extraction of the contour line and detection of the skin color region on the frame image G <b> 20 in which the reflection of some object is identified.

  The code reading unit 53 is based on the frame image captured by the image capturing unit 51, and includes a barcode or a two-dimensional code (for example, QR code (registered trademark)) attached to the product G captured by the imaging unit 164. ) And the like are decoded and read. Based on the read code information, the product G captured by referring to the PLU file F1 is specified. Specifically, the code reading unit 53 reads a frame image stored in the RAM 163 and detects a barcode by a pattern matching technique or a two-dimensional code from a predetermined finder pattern. Next, the code reading unit 53 reads product information of the product G such as a product code (product ID) from the detected barcode or two-dimensional code. Next, the code reading unit 53 specifies the product G corresponding to the product information (for example, product code) read with reference to the PLU file F1.

  Based on the frame image captured by the image capturing unit 51, the object recognition unit 54 reads the surface state such as the hue of the product G captured by the image capturing unit 164 and the surface roughness as a feature amount. By referring to the PLU file F1 based on the read feature amount, the product G picked up from the plurality of products G of different types set in the PLU file F1 is recognized. Specifically, the object recognition unit 54 reads the frame image stored in the RAM 163 and detects all or part of the product G using a pattern matching technique or the like. For example, an outline or the like is extracted from an image obtained by binarizing the captured frame image. Next, the contour line extracted from the previous frame image is compared with the contour line extracted from the current frame image, and the product G directed to the reading window 103 for sales registration is recognized.

  As another method for detecting a product, the presence or absence of a skin color region is detected from a captured frame image. Next, when a skin color area is detected, that is, when a clerk's hand is detected, it is assumed that the clerk's hand is grasped by detecting the above-described contour line in the vicinity of the skin color area. Try to extract the outline of the product. At this time, when the contour indicating the shape of the hand and the contour of the other object in the vicinity of the contour of the hand are detected, the product is recognized from the contour of the object.

  Next, the object recognizing unit 54 reads the surface state such as the hue of the product G and the surface unevenness from the whole or a part of the product G imaged by the imaging unit 164 as a feature amount. Note that the object recognition unit 54 does not consider the outline or size of the product G in order to shorten the processing time.

  In addition, the object recognition unit 54 reads the surface state such as the color of the registered product and the surface unevenness from the image of each product registered in the PLU file F1 (hereinafter referred to as registered product) as a feature amount. The degree of similarity between the product G and the registered product registered in the PLU file F1 is calculated by comparing with each of the G feature values. Here, the similarity is the degree of similarity of all or part of the product G when the image of each product stored in the PLU file F1 is 100% = “similarity: 1.0”. It indicates whether or not Note that, for example, the similarity may be calculated by changing the weight between the color tone and the surface roughness. Next, the object recognizing unit 54 identifies the product with the highest calculated similarity as the captured product G among the products in the PLU file F1.

Such object recognition for recognizing an object included in an image is called generic object recognition. Regarding such general object recognition, various recognition techniques are described in the following documents.
Keiji Yanai, “Current Status and Future of General Object Recognition”, IPSJ Journal, Vol. 48, no. SIG16 [Search August 10, 2010], Internet <URL: http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

Further, techniques for performing general object recognition by dividing an image into regions for each object are described in the following documents.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, [searched August 10, 2010], Internet <URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1. 1.145.3036 & rep = rep1 & type = pdf>

  It should be noted that the method of calculating the degree of similarity between the captured image of the product G and the registered product image registered in the PLU file F1 is not particularly limited. For example, the similarity between the captured image of the product G and each registered product registered in the PLU file F1 may be calculated as an absolute evaluation or may be calculated as a relative evaluation.

  When calculating the similarity as an absolute evaluation, the captured image of the product G and each registered product registered in the PLU file F1 are compared on a one-to-one basis, and the similarity derived as a result of this comparison is used as it is. do it. Further, when calculating the similarity as a relative evaluation, if five registered products (products GA, GB, GC, GD, GE) are registered in the PLU file F1, the captured product G is The similarity is 0.6, the similarity is 0.1 for the product GB, the similarity is 0.1 for the product GC, the similarity is 0.1 for the product GD, and the product GE On the other hand, the similarity may be calculated such that the similarity is 0.1, and the total similarity with each registered product is 1.0 (100%).

  Here, the case where the product G can be identified by the object recognition of the object recognition unit 54 refers to the case where a product whose similarity exceeds the predetermined threshold is found from the PLU file F1. For example, when the similarity is calculated as a relative evaluation, it is assumed that five registered products (products GA, GB, GC, GD, GE) are registered in the PLU file F1. The threshold for determining that object recognition has been completed is 0.5. The similarity of the product GA to the imaged product G is 0.6, the similarity of the product GB is 0.1, the similarity of the product GC is 0.1, the similarity of the product GD is 0.1, the similarity of the product GE When the degree is 0.1 or the like, it is assumed that the product G can be identified by object recognition by the object recognition unit 54 because the similarity of the product GA exceeds the threshold. Further, the similarity of the product GA to the imaged product G is 0.4, the similarity of the product GB is 0.2, the similarity of the product GC is 0.2, the similarity of the product GD is 0.1, and the product GE If the similarity of the product GA is 0.1 or the like, it is assumed that the product G cannot be identified by object recognition by the object recognition unit 54 because the similarity of the product GA does not exceed the threshold. However, in the object recognition processing by the object recognition unit 54, a list of products (candidate list) similar to the product G is extracted from the PLU file F1 even if the similarity does not exceed a predetermined threshold. Therefore, when the product G cannot be specified, the product G can be recognized, but the product G cannot be specified.

  Further, the object recognition unit 54 displays the recognition result by the object recognition, that is, the recognized product G on the display screen (operation screen) of the display device 106 and notifies the operator (see FIG. 7). The notification of the recognition result may be voice notification in which the recognition result is read out and notified by voice output by the voice output unit 165. The object recognition unit 54 may notify the product having the highest similarity as the recognition result when the product G has been recognized but the product G has not been specified.

  The object movement detection unit 55 recognizes the product G recognized by the object recognition unit 54 based on the frame images sequentially captured by the image capture unit 51 (including the case where the above-described product G has not been specified). Detecting movement. Specifically, the object movement detection unit 55 compares the frame images that are temporally forward and backward from the sequentially captured frame images, and detects the difference, thereby detecting the product G recognized by the object recognition unit 54. Detecting movement. For example, by detecting the movement of the product G recognized by the object recognition unit 54 to the upper, lower, right, and left from the difference between the frame images that move forward and backward in time, The movement of the product G in the downward and left-right directions can be detected. Further, by detecting a change in the size of the product G recognized by the object recognizing unit 54 from the difference between the frame images moving back and forth in time, the movement of the product G in the direction approaching or moving away from the reading window 103 is detected. it can.

  The input receiving unit 56 receives various input operations corresponding to the display on the display device 106 via the touch panel 105 or the keyboard 107. For example, the information output unit 57 displays a confirmation screen for the product G specified by the code reading unit 53 or the object recognition unit 54 on the display device 106, and the sales of the specified product G based on a selection operation on the confirmation screen. An input operation (confirmation operation) for registration is accepted by the keyboard 107. The confirmation operation accepted by the information output unit 57 is notified to the POS terminal 11 by the information output unit 57. In addition, after the confirmation operation for registering sales of the product G, the input receiving unit 56 displays an operation screen for receiving an input of the number of products G to be registered for sales on the display device 106, and the number input operation is performed by the keyboard. Accept at 107. The number received by the input receiving unit 56 is notified to the POS terminal 11 by the information output unit 57. The POS terminal 11 receives the confirmation operation and the number notification, and performs sales registration in the sales registration unit 611.

  In addition to the input operation of the keyboard 107 and the touch panel 105, the input reception unit 56 receives an operation instruction corresponding to the movement of the product G detected by the object movement detection unit 55. Specifically, the movement of the product G in the upper or lower direction or the left and right direction on the parallel surface of the reading window 103 is detected, and for example, an upward operation instruction corresponding to the upward movement in the vertical direction, the downward movement in the vertical direction is supported. The downward operation instruction and the left / right operation instruction corresponding to the left / right movement are received. Thereby, in the product reading apparatus 101, the product G recognized in front of the reading window 103 is moved up, down, right, and left, so that an operation instruction corresponding to the cursor key for instructing the up / down / left / right direction on the keyboard 107 is performed. It can be performed. The input receiving unit 56 may receive a selection instruction, a cancellation instruction, or the like corresponding to the movement of the product G in a direction approaching or moving away from the reading window 103. In this case, by moving the product G recognized in front of the reading window 103 in a direction approaching or moving away from the reading window 103, an operation instruction corresponding to a selection key or a cancel key on the keyboard 107 can be performed.

  In addition, the input receiving unit 56 causes the display device 106 to display an operation screen on which the frame image sequentially captured by the image capturing unit 51 and a guide image for guiding an operation instruction corresponding to the movement of the product G are displayed. An operation instruction corresponding to the movement of the product G on the operation screen may be received. Specifically, a guide image showing an operation content (a movement position of the product G when the product G itself is recognized as a cursor pointer) accepted when the cursor pointer or the like is moved to a predetermined area on the operation screen is displayed. It is displayed on the operation screen (see FIG. 7), and the operator is prompted to perform an operation instruction corresponding to the movement of the product G. The operator can input a desired operation instruction without operating the keyboard 107 or the like by checking the operation screen and moving the product G.

  Further, the input receiving unit 56 superimposes operation items (for example, operation menus and selection items) corresponding to predetermined operation instructions, which are the above-described guidance images, and frame images sequentially captured by the image capturing unit 51. The displayed operation screen (see FIG. 8) is displayed on the display device 106. The input receiving unit 56 receives an operation instruction corresponding to the operation item when the object movement detection unit 55 detects the movement of the product G to a position overlapping the operation item. That is, the input reception unit 56 receives an operation instruction using the product G itself as a cursor pointer. In this case, the operator can input an operation instruction corresponding to the operation item without operating the keyboard 107 or the like by checking the operation screen and moving the product G to the operation item.

  The information output unit 57 sends information (for example, product ID and product name) and the number indicating the product to the POS terminal 11 via the connection interface 175 for the product G for which the confirmation operation has been performed as described above. Output. As information output to the POS terminal 11 by the information output unit 57, the product ID read from the PLU file F1 by the information output unit 57 based on the product G specified by the code reading unit 53 or the object recognition unit 54 is directly used. The product name that can specify the product ID or the file name of the product image may be notified, and the storage location (storage address in the PLU file F1) of the product ID may be notified. You may notify to the POS terminal 11.

  The sales registration unit 611 of the POS terminal 11 performs sales registration of the corresponding product based on the product ID output from the information output unit 57 and the number of registered sales. Specifically, the sales registration unit 611 records the notified product ID and the product classification, the product name, the unit price, etc. corresponding to the product ID together with the number in the sales master file etc. with reference to the PLU file F1. Register sales.

  Next, the operation of the checkout system 1 will be described in detail. First, the operation of the commodity reading apparatus 101 will be described. FIG. 6 is a flowchart illustrating an example of the operation of the commodity reading apparatus 101.

  As illustrated in FIG. 6, when processing is started in response to the start of product registration by the POS terminal 11, the image capturing unit 51 outputs an imaging on signal to the imaging unit 164 and performs imaging by the imaging unit 164. Start (S11). Next, the image capturing unit 51 starts capturing the frame image captured by the image capturing unit 164 and stored in the RAM 163 (S12).

  Next, the CPU 161 determines whether or not the product detection unit 52 has detected the product G from the frame image captured by the image capture unit 51 (S13). Here, when the commodity G is not detected (S13: NO), the CPU 161 stands by for processing. When the product G is detected from the frame image captured by the image capturing unit 51 (S13: YES), the CPU 161 determines whether the code information is decoded by the code reading unit 53 and the code has been read. (S14). If the code is read (S14: YES), the product G can be identified from the read code, and the process proceeds to S19.

  When the code is not read (S14: NO), the object recognition unit 54 performs object recognition of the product G detected by the product detection unit 52 (S15). Therefore, in the commodity reading apparatus 101, when the code is not read, the commodity G is specified by object recognition.

  Here, the object recognition unit 54 notifies the operator by displaying the recognition result of the object recognition on the operation screen of the display device 106, and accepts an operation instruction corresponding to the movement of the product G, thereby performing object recognition. It is determined whether or not to display the obtained candidate list of products with high similarity (S16).

  FIG. 7 is a conceptual diagram illustrating a display example of the display device 106. As shown in FIG. 7, on the operation screen G30 of the display device 106, a guidance image G31 for guiding the display of the candidate list and a notification image G32 indicating the recognition result by the object recognition unit 54 are displayed. Therefore, the operator can grasp the recognition result of the product G by the object recognition unit 54 by confirming the notification image G32. Here, when the product G is erroneously recognized, the operator moves the product G held by the hand H according to the guide image G31 so that the product recognition unit 54 recognizes that the degree of similarity is high. A list can be displayed.

  In S16, when the candidate list is not displayed (S16: NO), since the operator confirms that the recognition result notified by the notification image G32 is correct, the process proceeds to S19.

  When displaying the candidate list in S16, that is, when the product G is moved according to the guide image G31 (S16: YES), the input accepting unit 56 recognizes that the similarity is high by the object recognizing unit 54. The list is displayed on the operation screen G30 of the display device 106 (S17). Next, the input receiving unit 56 receives a selection of a product from the candidate list by receiving an operation instruction corresponding to the movement of the product G (S18).

  FIG. 8 is a conceptual diagram illustrating an example of an operation instruction by moving the product G. As shown in FIG. 8, the recognition result of the notification image G32 is “apple” for the product G, which is a so-called “carrot”, and therefore the operator superimposes the product G on the guide image G31 according to the guide image G31. Move to do. In response to the movement of the product G to the guide image G31, the input receiving unit 56 displays a candidate list for receiving the selection of the product on the operation screen G30. Specifically, the item images G33 and G34 for selecting a product recognized as having high similarity by the object recognition unit 54, such as “apple” and “carrot”, and the item image G35 for canceling the selection of the product are operated. It is displayed on the screen G30. The operator can input the desired product from the candidate list of products recognized as having high similarity by the object recognition unit 54 by moving the product G so as to be superimposed on the desired item images G33 and G34. it can. For example, as shown in the figure below, the recognition result of the object recognition unit 54 is changed from “apple” to “carrot” by moving the product G so as to be superimposed on the item image G34 corresponding to “carrot”. be able to. The candidate list display on the operation screen G30 may display, for example, a product name corresponding to “apple” based on the product image in the PLU file F1 together with the display of “apple” as the product name. .

  In S19, the input receiving unit 56 receives the confirmation of the product G to be registered for sales and the input of the number from the keyboard 107 (S19). Needless to say, the confirmation of the product G and the input of the number thereof may be accepted by the operation instruction corresponding to the movement of the product G described above. Next, the information output unit 57 outputs the product ID for the product G confirmed in S19 and the number received in S19 to the POS terminal 11 (S20).

  Next, the CPU 161 determines the presence / absence of business end from the POS terminal 11 based on a product registration end notification or the like (S21). When the business is continued (S21: NO), the CPU 161 returns the process to S13 and continues the process. When the work is to be ended (S21: YES), the image capturing unit 51 outputs an imaging off signal to the imaging unit 164, ends the imaging by the imaging unit 164 (S22), and ends the process.

  Next, the operation of the POS terminal 11 will be described. FIG. 9 is a flowchart showing the procedure of sales registration processing executed by the POS terminal 11.

  First, when processing is started in response to the start of product registration in response to an operation instruction on the keyboard 22, the CPU 61 receives the product ID and the number of products G output from the product reading device 101 in S20 of FIG. (S31). Next, the sales registration unit 611 reads the product classification, unit price, etc. from the PLU file F1 based on the product ID and the number received in S31, and registers the sales of the product G read by the product reading device 101 in the sales master file. (S32). Subsequently, the CPU 61 determines whether or not there is an end of business due to the end of sales registration or the like by an operation instruction of the keyboard 22 (S33). When the business is continued (S33: NO), the CPU 61 returns to S31 and continues the process. When the business is to be ended (S33: YES), the CPU 61 ends the process.

  Note that the program executed by the commodity reading apparatus 101 of the present embodiment is provided by being incorporated in advance in a ROM or the like. A program executed by the commodity reading apparatus 101 according to the present embodiment is a file in an installable format or an executable format, and is a computer such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk). The information may be provided by being recorded on a recording medium that can be read by the user.

  Furthermore, the program executed by the commodity reading apparatus 101 according to the present embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, the program executed by the product reading apparatus 101 of the present embodiment may be configured to be provided or distributed via a network such as the Internet.

  The program executed by the product reading apparatus 101 of the present embodiment includes the above-described units (image capturing unit 51, product detection unit 52, code reading unit 53, object recognition unit 54, object movement detection unit 55, and input reception unit 56). , And an information output unit 57). As actual hardware, a CPU (processor) reads out the program from the ROM and executes the program, so that each unit is loaded onto the main storage device. It is generated on the device.

  DESCRIPTION OF SYMBOLS 1 ... Checkout system, 11 ... POS terminal, 21 ... Drawer, 22 ... Keyboard, 22d ... Numeric keypad, 22e ... Temporary fastening key, 22f ... Fastening key, 23 ... Display device, 23a ... Display surface, 24 ... Display device for customers , 24a ... display surface, 25 ... communication interface, 26 ... touch panel, 41 ... checkout table, 51 ... image capture unit, 52 ... product detection unit, 53 ... code reading unit, 54 ... object recognition unit, 55 ... object movement Detection unit, 56 ... input reception unit, 57 ... information output unit, 60 ... microcomputer, 61 ... CPU, 62 ... ROM, 63 ... RAM, 64 ... HDD, 65 ... connection interface, 66 ... printer, 101 ... commodity reading device DESCRIPTION OF SYMBOLS 102 ... Housing 103 ... Reading window 104 ... Display / operation part 105 ... Touch panel 10 DESCRIPTION OF SYMBOLS ... Display device, 107 ... Keyboard, 108 ... Card reading groove, 109 ... Customer display device, 110 ... Goods reading part, 151 ... Counter stand, 152 ... Receipt surface, 153 ... Shopping basket, 153a ... First shopping basket, 153b ... second shopping basket, 160 ... microcomputer, 161 ... CPU, 162 ... ROM, 163 ... RAM, 164 ... imaging unit, 165 ... audio output unit, 175,176 ... connection interface, 611 ... sales registration unit, G ... Product, G10, G20 ... Frame image, G30 ... Operation screen, G31 ... Guidance image, G32 ... Notification image, G33-G35 ... Item image, H ... Hand, F1 ... PLU file, PR ... Program, SC ... Store computer

JP 2008-97172 A

Claims (6)

Capture means for sequentially capturing images captured by the imaging means;
Article detection means for detecting an article contained in the captured image;
Recognizing means for recognizing the detected article as an unspecified article from a plurality of articles of different types by reading a feature amount of an image in which the article is detected;
A movement detecting means for detecting movement of the recognized article included in the image based on the sequentially captured images;
Operation accepting means for accepting an operation instruction corresponding to the movement of the recognized article and identifying the recognized article ;
An information processing apparatus comprising: The operation accepting means accepts a change to an article different from the recognized article from a plurality of articles of different types based on the operation instruction.
The information processing apparatus according to claim 1. A notification means for notifying the recognized article;
The operation accepting unit accepts a change to an article different from the recognized article based on a predetermined operation instruction corresponding to the movement of the recognized article after the notification of the recognized article. Start,
The information processing apparatus according to claim 2. The image processing apparatus further includes display means for displaying the sequentially captured image and a guidance image for guiding an operation instruction corresponding to the movement of the recognized article.
The information processing apparatus according to any one of claims 1 to 3. The display means superimposes and displays operation items corresponding to predetermined operation instructions on the sequentially captured images,
The operation accepting unit accepts an operation instruction corresponding to the operation item in accordance with the movement of the recognized article included in the sequentially captured images to the operation item.
The information processing apparatus according to claim 4. A computer for controlling the information processing apparatus;
Capture means for sequentially capturing images captured by the imaging means;
Article detection means for detecting an article contained in the captured image;
Recognizing means for recognizing the detected article as an unspecified article from a plurality of articles of different types by reading a feature amount of an image in which the article is detected;
A movement detecting means for detecting movement of the recognized article included in the image based on the sequentially captured images;
Operation accepting means for accepting an operation instruction corresponding to the movement of the recognized article and identifying the recognized article ;
Program to make it function.

Images