JP2015130116A - Information processor and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor and a program capable of increasing work efficiency relevant to recognition of commodities.SOLUTION: The information processor includes: taking-in means that takes in an image which is taken by imaging means; detection means that detects the commodity image taken in by the taking-in means; recognition means that performs a recognition processing to compare a feature quantity of the commodity detected by the detection means and a feature quantity of each of reference commodities as a reference and to recognize a reference commodity which has a similarity threshold value or more as a commodity candidate; commodity candidate display means that displays the commodity candidates in a selectable manner; and guidance display means that displays a guidance screen for prompting a transition to another processing at a predetermined timing during the recognition means executing the recognition processing.

Description

  Embodiments described herein relate generally to an information processing apparatus and a program.

  2. Description of the Related Art Conventionally, there is a technique for recognizing the type or the like of an article by extracting the feature quantity of the article from image data obtained by imaging the article and comparing it with data indicating the feature quantity of a matching article prepared in advance. In addition, a store system that registers sales of recognized products by using this technology for recognition (identification) of products such as fruits and vegetables has been proposed. In such a store system, when there are a plurality of matching products (product candidates) that are product candidates, these product candidates are displayed in a selectable manner, and a selection from the user is accepted.

  By the way, in the above technique, it is known that the recognition accuracy changes depending on the way of imaging. For this reason, depending on how the image is captured, it may take time until the product candidate is displayed, or it may take time until the product candidate is stably displayed. In such a case, since the operator tends to continue imaging until a desired product is recognized, there is a possibility that the time required for imaging becomes longer and the work efficiency (productivity) related to the recognition of the product is reduced. there were.

  The problem to be solved by the present invention is to provide an information processing apparatus and program capable of improving work efficiency related to product recognition.

  The information processing apparatus according to the embodiment includes capture means, detection means, recognition means, product candidate display means, and guidance display means. The capturing unit captures an image captured by the imaging unit. The detection means detects a product from the image captured by the capture means. The recognition unit compares the feature amount of the product detected by the detection unit with the feature amount of each reference product as a reference, and executes a recognition process for recognizing a reference product whose similarity is equal to or greater than a threshold as a product candidate. To do. The product candidate display means displays the product candidates in a selectable manner. The guidance display means displays a guidance screen that prompts a transition to another process at a predetermined timing while the recognition means executes the recognition process.

FIG. 1 is a perspective view showing an external configuration of a checkout system according to the embodiment. FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal and the commodity reading apparatus illustrated in FIG. FIG. 3 is a diagram schematically showing an example of the data structure of the PLU file shown in FIG. FIG. 4 is a block diagram illustrating a functional configuration of the POS terminal and the commodity reading apparatus illustrated in FIG. FIG. 5 is a diagram illustrating an example of a product candidate screen. FIG. 6 is a diagram illustrating an example of a guidance screen displayed when a product candidate is recognized. FIG. 7 is a diagram illustrating an example of a guidance screen displayed when a product candidate is not recognized. FIG. 8 is a flowchart illustrating a procedure of product recognition processing executed by the product reading apparatus. FIG. 9 is a flowchart illustrating a procedure of guidance display processing executed by the product reading apparatus. FIG. 10 is a flowchart showing a procedure of sales registration processing executed by the POS terminal. FIG. 11 is a perspective view showing an external configuration of the self-POS according to the embodiment. FIG. 12 is a block diagram showing a hardware configuration of the self-POS shown in FIG.

  Hereinafter, an information processing apparatus and a program according to the present embodiment will be described with reference to the drawings, taking a checkout system as an example. The store system is a checkout system (POS system) provided with a POS terminal that performs registration and settlement of commodities related to one transaction. This embodiment is an application example to a checkout system introduced in a store such as a supermarket.

  FIG. 1 is a perspective view showing an external configuration of the checkout system 1. As shown in FIG. 1, the checkout system 1 includes a POS terminal 11 and a commodity reading device 101 as an information processing device.

  The POS terminal 11 is placed on the upper surface of the drawer 21 on the checkout table 51. The drawer 21 is controlled by the POS terminal 11 for the opening operation. The keyboard 22 pressed by an operator (such as a store clerk) is disposed on the upper surface of the POS terminal 11. A display device 23 that displays information toward the operator is provided behind the keyboard 22 as viewed from the operator who operates the keyboard 22. The display device 23 displays information on the display surface 23a. The touch panel 26 is stacked on the display surface 23a. The customer display device 24 is erected so as to be rotatable further back than the display device 23. The customer display device 24 displays information on the display surface 24a.

  The customer display device 24 shown in FIG. 1 is in a state where the display surface 24a faces the front side in FIG. 1, but the customer display device 24 is rotated so that the display surface 24a faces the back side in FIG. By doing so, the customer display device 24 displays information for the customer.

  A horizontally long counter-shaped counter table 151 is arranged so as to form an L shape with the checkout table 51 on which the POS terminal 11 is placed. A load receiving surface 152 is formed on the upper surface of the counter table 151. A shopping basket 153 for storing the product G is placed on the cargo receiving surface 152. The shopping basket 153 may be divided into a first shopping basket 153a brought in by a customer and a second shopping basket 153b placed at a position sandwiching the commodity reading device 101 from the first shopping basket 153a. it can.

  On the receiving surface 152 of the counter stand 151, the commodity reading apparatus 101 connected to the POS terminal 11 so as to be able to transmit and receive data is installed. The commodity reading apparatus 101 includes a thin rectangular housing 102.

  A reading window 103 is disposed in front of the housing 102. A display / operation unit 104 is attached to the upper portion of the housing 102. The display / operation unit 104 is provided with a display device 106 such as a liquid crystal display on which a touch panel 105 is laminated. A keyboard 107 is disposed on the right side of the display device 106. A card reading groove 108 of a card reader (not shown) is provided on the right side of the keyboard 107. A customer display device 109 for providing information to the customer is installed on the left rear side of the display / operation unit 104 as viewed from the operator.

  In the first shopping basket 153a brought in by the customer, the commodity G related to one transaction is stored. The commodity G in the first shopping basket 153a is moved to the second shopping basket 153b by an operator who operates the commodity reading apparatus 101. In this movement process, the product G is directed to the reading window 103 of the product reading apparatus 101. At this time, the imaging unit 164 (see FIG. 2) arranged in the reading window 103 images the product G.

  FIG. 2 is a block diagram illustrating a hardware configuration of the POS terminal 11 and the commodity reading apparatus 101. The POS terminal 11 includes a microcomputer 60 as an information processing unit that executes information processing. The microcomputer 60 includes a CPU (Central Processing Unit) 61 that executes various arithmetic processes and controls each unit, a ROM (Read Only Memory) 62, and a RAM (Random Access Memory) 63. The ROM 62 stores a program executed by the CPU 61. Further, the microcomputer 60 includes a time measuring unit (not shown) such as an RTC (Real Time Clock) for measuring time.

  The CPU 61 of the POS terminal 11 includes a drawer 21, a keyboard 22, a display device 23, a customer display device 24, a communication interface 25, a touch panel 26, an HDD (Hard Disk Drive) 64, a connection interface 65, and a printer 66. They are connected via an input / output circuit (both not shown).

  The keyboard 22 includes a numeric key 22d, a temporary fastening key 22e, and a fastening key 22f on which numbers such as “1”, “2”, “3”,...

  The HDD 64 stores various programs and various files. The CPU 61 copies all or a part of the program stored in the HDD 64 to the RAM 63 and executes it when the POS terminal 11 is activated.

  The HDD 64 stores data files such as the PLU file F1. The PLU file F1 can be read from the product reading apparatus 101 via the connection interface 65.

  The PLU file F1 is a data file in which an association between a product G sold in a store and information related to sales registration of the product G is set. FIG. 3 is a diagram schematically illustrating an example of the data configuration of the PLU file F1. As shown in the figure, in the PLU file F1, for each product G, a product ID uniquely assigned, information about the product such as product name and unit price, and a product image obtained by capturing the product G are associated with each other. be registered. Further, in the PLU file F1, for each product G, the feature amount of the product G (reference product feature amount data) is associated and registered.

  The product image is an image obtained by capturing each of the reference products to be compared in the similarity determination described later. The merchandise image is displayed as an image representing the merchandise candidate when the merchandise candidate to be described later is displayed. In addition, the feature amount of the product G extracted in advance from the captured image (for example, product image) of each product G is registered in association with the corresponding product ID. Here, the feature amount is information representing the characteristics of the product G, such as the color and pattern of the surface of the product G, the uneven state, and the shape.

  In the present embodiment, the feature amount of each product G is registered in advance in the PLU file F1, but the present invention is not limited to this, and a feature amount may not be registered. In this case, a feature amount may be acquired each time by extracting a feature amount from each product image by a feature amount extraction unit 1613 described later. In addition, a display image may be registered instead of the product image. Hereinafter, the product registered in the PLU file F1 is referred to as “registered product”.

  Returning to FIG. 2, the communication interface 25 for executing data communication with the store computer SC is connected to the CPU 61 of the POS terminal 11 via an input / output circuit (not shown). The store computer SC is installed in a store backyard or the like. The HDD (not shown) of the store computer SC stores a PLU file F1 distributed to the POS terminal 11, an inventory management file for managing the inventory status of each registered product registered in the PLU file F1, and the like.

  The connection interface 65 enables data transmission / reception with the commodity reading apparatus 101. The product reading apparatus 101 is connected to the connection interface 65. The printer 66 prints on a receipt or the like. The POS terminal 11 prints the transaction content of one transaction on a receipt by the printer 66 under the control of the CPU 61.

  The product reading apparatus 101 includes a product reading unit 110 and a display / operation unit 104. The product reading unit 110 includes a microcomputer 160. The microcomputer 160 includes a CPU 161, a ROM 162, and a RAM 163. The ROM 162 stores a program executed by the CPU 161. Further, the microcomputer 160 includes a time measuring unit (not shown) such as an RTC for measuring time.

  The CPU 161 is connected to an imaging unit 164, an audio output unit 165, and a connection interface 175 via various input / output circuits (all not shown). The operations of the imaging unit 164, the audio output unit 165, and the connection interface 175 are controlled by the CPU 161.

  The imaging unit 164 is a color CCD sensor, a color CMOS sensor, or the like, and is an imaging unit that performs imaging from the reading window 103. For example, the imaging unit 164 captures a 30 fps moving image. Frame images (captured images) sequentially captured at a predetermined frame rate by the imaging unit 164 are stored in the RAM 163. The audio output unit 165 is an audio circuit and a speaker for generating a preset warning sound or the like. The sound output unit 165 performs notification by sound such as a warning sound under the control of the CPU 161.

  The display / operation unit 104 includes a touch panel 105, a display device 106, a keyboard 107, a customer display device 109, and a connection interface 176. The connection interface 175 of the commodity reading unit 110 is connected to the connection interface 65 of the POS terminal 11 to enable data transmission / reception with the POS terminal 11. The connection interface 175 is connected to the display / operation unit 104 via the connection interface 176, and the CPU 161 performs data transmission / reception between the product reading unit 110 and the display / operation unit 104 via the connection interface 175.

  Next, functional configurations of the CPU 161 and the CPU 61 that are realized by the CPU 161 and the CPU 61 executing programs will be described with reference to FIG.

  FIG. 4 is a block diagram illustrating functional configurations of the POS terminal 11 and the commodity reading apparatus 101. As illustrated in FIG. 4, the CPU 161 of the product reading apparatus 101 sequentially executes the program, thereby causing an image capturing unit 1611, a product detection unit 1612, a feature amount extraction unit 1613, a similarity determination unit 1614, a product candidate display unit. 1615, a guidance display unit 1616, an input reception unit 1617, and an information output unit 1618 are provided.

  The image capturing unit 1611 is a functional unit corresponding to the capturing unit. The image capturing unit 1611 outputs an imaging on signal to the imaging unit 164 and causes the imaging unit 164 to start an imaging operation. The image capturing unit 1611 sequentially captures captured images captured by the image capturing unit 164 and stored in the RAM 163 after the start of the image capturing operation. Captured images are captured by the image capture unit 1611 in the order stored in the RAM 163.

  The product detection unit 1612 is a functional unit corresponding to the detection unit. The product detection unit 1612 detects all or part of the outline of the product G included in the captured image captured by the image capturing unit 1611 using a known pattern matching technique or the like. Next, the contour line extracted from the previous captured image (frame image) is compared with the contour line extracted from the current frame image, and the product G directed to the changed portion, that is, the reading window 103 is compared. Detect the reflection area.

  As another method for detecting the product G, it is determined whether or not a skin color area is detected from the captured image, and when the skin color area is detected, that is, when the reflection of the clerk's hand is detected, this skin color is detected. By detecting the above-described contour line in the vicinity of the region, the contour extraction of the product G that is assumed to be held by the clerk's hand is attempted. At this time, when the contour indicating the shape of the hand and the contour of the other object are detected in the vicinity of the contour of the hand, the product G is detected from the contour of the object.

  The feature amount extraction unit 1613 is a functional unit corresponding to the extraction unit. The feature amount extraction unit 1613 uses, as a feature amount, the surface state (surface color, pattern, unevenness, shape, etc.) of the product G detected by the product detection unit 1612 from the captured image captured by the image capture unit 1611. Extract.

  The similarity determination unit 1614 is a functional unit corresponding to the calculation unit and the recognition unit. The similarity determination unit 1614 compares the feature amount of each registered product registered in the PLU file F1 with the feature amount extracted by the feature amount extraction unit 1613, and calculates the similarity. The similarity determination unit 1614 recognizes a registered product (product ID) whose value is equal to or greater than a predetermined threshold among the calculated similarities as a candidate for the product G (product candidate) imaged by the imaging unit 164.

  In addition, the similarity determination unit 1614, when there is one registered product in which the similarity value is very higher than the other among the registered products having a similarity equal to or higher than a threshold, the corresponding registration It has an automatic confirmation function that makes a product a product candidate. It should be noted that the similarity comparison criterion can be arbitrarily set. For example, one registered product having a similarity of 20% or more may be automatically determined by comparing with the highest similarity calculated for other product candidates.

  Also, the similarity is a value (similarity) indicating how similar the feature quantities of the product G are compared with the standard feature quantities of the product registered in the PLU file F1. That's fine. Note that the concept of similarity is not limited to this example, but is a value indicating the degree of coincidence with the feature quantity of each registered product registered in the PLU file F1, and the feature quantity of the product G and each registered in the PLU file F1. It may be a value indicating how much the feature quantity of the registered product correlates.

Recognizing an object included in an image as described above is called generic object recognition. Regarding such general object recognition, various recognition techniques are described in the following documents.
Keiji Yanai, “Current Status and Future of General Object Recognition”, IPSJ Journal, Vol. 48, no. SIG16 [December 12, 2013 search], Internet <URL: http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

Further, the following document describes a technique for performing general object recognition by dividing an image into regions for each article (object).
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, [December 12, 2013 search], Internet <URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1. 1.145.3036 & rep = rep1 & type = pdf>

  Note that the method of calculating the similarity is not particularly limited. For example, the similarity may be calculated as an absolute evaluation or may be calculated as a relative evaluation. When calculating the similarity as an absolute evaluation, the captured image of the product G and the registered product are compared on a one-to-one basis, and the similarity derived as a result of this comparison may be employed as it is. Further, when calculating the similarity as a relative evaluation, it is only necessary to calculate the sum of the similarities between the captured product G and each registered product to be 1.0 (100%).

  The product candidate display unit 1615 is a functional unit corresponding to the selection unit and the display control unit. The product candidate display unit 1615 displays information on the registered product recognized as the product candidate by the similarity determination unit 1614 on the display device 106. More specifically, the merchandise candidate display unit 1615 reads a record of registered merchandise recognized as a merchandise candidate from the PLU file F1 and displays it in the merchandise candidate screen of the display device 106.

  FIG. 5 is a diagram illustrating an example of a product candidate screen. The product candidate screen A1 displayed on the display device 106 includes a captured image area A11 and a product candidate area A12.

  The captured image area A11 is an area for displaying a captured image captured by the image capturing unit 1611. A manual registration button B1 for manually recognizing (registering) the product G from a department code, a product list, or the like is provided below the captured image area A11. The CPU 161 displays a manual registration screen (not shown) for manually recognizing (registering) the product G on the display device 106 in accordance with the operation of the manual registration button B1. Note that the product specified via the manual registration screen is processed as a confirmed product to be described later.

  The product candidate area A12 is an area for displaying information related to registered products recognized as product candidates. A display area A12a is arranged in the commodity candidate area A12. FIG. 5 shows an example in which four display areas A12a are set in the commodity candidate area A12, but the number of display areas A12a is not particularly limited. Moreover, it is good also as a form which increases or decreases the number of display area A12a dynamically according to the number of goods candidates.

  The commodity candidate display unit 1615 arranges and displays information (G1 to G4) such as commodity images and commodity names of commodity candidates in the display area A12a. The product candidates displayed in the product candidate area A12 are configured to be selectable in response to a touch operation on the touch panel 105. Here, the arrangement order of the product candidates is not particularly limited. For example, it is good also as a form arrange | positioned sequentially from display area A12a of the upper left of drawing to display area A12a of the lower right in order of the product candidate with a high similarity. With this configuration, the operator of the commodity reading apparatus 101 can determine the commodity G by selecting the commodity candidate displayed in the commodity candidate area A12.

  By the way, it is known that the recognition accuracy changes depending on how the product G is held over the reading window 103. For this reason, depending on how the product G is held up, it may take time until the product candidate is displayed, it may take time until the product candidate is displayed stably, or it may take time until automatic confirmation is performed. Sometimes. In such a case, since the operator tends to continue imaging until a desired product is recognized, there is a possibility that the time required for imaging becomes longer and the work efficiency (productivity) related to the recognition of the product is reduced. there were.

  Therefore, the guidance display unit 1616 displays a guidance screen including information that prompts the transition to another action (processing) at a predetermined timing while the similarity determination unit 1614 recognizes the product candidate. 106.

  Specifically, the guidance display unit 1616 starts timing at the timing when the product detection unit 1612 detects the product G, and displays a guidance screen when a predetermined time (for example, 1 second) elapses. In addition, when displaying the guidance screen, the guidance display unit 1616 switches the content of the guidance screen depending on whether or not the similarity determination unit 1614 recognizes the product candidate. In addition, it is preferable that the similarity determination unit 1614 stops the process related to the recognition of the product candidate at the timing when the guidance screen is displayed.

  FIG. 6 is a diagram illustrating an example of a guidance screen displayed when a product candidate is recognized. As shown in FIG. 6, the first guidance screen A2a is displayed on the product candidate screen A1 (captured image area A11). The first guidance screen A2a includes a message M1 that prompts selection of a product candidate displayed in the product candidate area A12, a message M2 that prompts re-recognition of the product candidate, and a re-recognition button B2 for instructing re-recognition of the product candidate. And have.

  When selecting the product candidate, the operator of the product reading apparatus 101 selects a product candidate corresponding to the product G from the product candidates displayed in the product candidate area A12 according to the message M1. If there is no product candidate displayed in the product candidate area A12 that corresponds to the product G, the operator of the product reading apparatus 101 operates the re-recognition button B2 in accordance with the message M2. When the guidance display unit 1616 accepts the operation of the re-recognition button B2, the guidance display unit 1616 deletes the first guidance screen A2a, and causes the similarity determination unit 1614 to resume the process related to the recognition of the product candidate. When manual registration is performed, a manual registration screen (not shown) is displayed by operating the manual registration button B1 on the product candidate screen A1.

  Moreover, FIG. 7 is a figure which shows an example of the guidance screen displayed when a goods candidate is not recognized. As shown in FIG. 7, the second guidance screen A2b is displayed superimposed on the product candidate screen A1. The second guidance screen A2b includes a message M3 indicating that no product candidate is found, a message M4 for prompting re-recognition or manual registration of the product candidate, a re-recognition button B3, and a close button B4.

  The operator of the product reading apparatus 101 operates the re-recognition button B3 according to the message M4 when it is desired to continue the product candidate recognition process. When the guidance display unit 1616 accepts the operation of the re-recognition button B3, the guidance display unit 1616 deletes the second guidance screen A2b, and causes the similarity determination unit 1614 to resume the processing related to the recognition of the product candidate.

  Moreover, the operator of the product reading apparatus 101 operates the close button B4 in accordance with the message M4 when the product candidate recognition process is to be continued. When the guidance display unit 1616 receives the operation of the close button B4, the guidance display unit 1616 deletes the second guidance screen A2b. Thereafter, the operator of the product reading apparatus 101 operates a manual registration button B1 on the product candidate screen A1, thereby displaying a manual registration screen (not shown).

  In addition, you may comprise so that it can transfer to a manual registration screen (not shown) from a guidance screen by providing the button which has a function equivalent to manual registration button B1 in guidance screen (A2a, A2b).

  As described above, the product reading apparatus 101 according to the present embodiment displays a guidance screen that prompts the user to shift to another process at a predetermined timing while the similarity determination unit 1614 executes the process related to product candidate recognition. indicate. Thereby, when it takes time until the commodity candidate is displayed, it is possible to prompt the operator for a registration method other than the object recognition. Therefore, since it is possible to prevent the product image from being inadvertently continued, work efficiency (productivity) related to product recognition can be improved.

  Further, the product reading device 101 provides a guidance screen with different contents depending on whether or not a product candidate is recognized. Accordingly, it is possible to promote the shift to another process corresponding to the recognition status of the product candidate, and it is possible to improve convenience.

  In addition, the commodity reading apparatus 101 provides the guidance screen including an operation element (re-recognition buttons B2 and B3) for instructing resumption of the recognition process. Thereby, since the registration method by object recognition can be continued according to the operation instruction from an operator, the convenience can be improved.

  In the present embodiment, the base point related to the display of the guidance screen is the time when the product detection unit 1612 detects the product G. However, the present invention is not limited to this, and can be arbitrarily set. For example, when the image capturing unit 1611 first captures a captured image for the product G, the guidance screen (the first guidance screen A2a or the second guidance screen A2b) is displayed after a predetermined time has elapsed from the base point. Also good. Alternatively, the product candidate display unit 1615 may display the first guidance screen A2a after a predetermined time has elapsed from the base point when the product candidate is first displayed for the product G. As described above, in the commodity reading apparatus 101 according to the present embodiment, since the timing for displaying the guidance screen can be arbitrarily set, the guidance screen can be displayed at the timing according to the use environment.

  Returning to FIG. 4, the input receiving unit 1617 receives various input operations corresponding to the display on the display device 106 via the touch panel 105 or the keyboard 107. For example, the input receiving unit 1617 receives a selection operation for any one of the product candidates displayed from the product candidates displayed on the display device 106. The input receiving unit 1617 receives the selected product candidate as a product (determined product) corresponding to the product G imaged by the imaging unit 164. When the product detection unit 1612 can detect a plurality of products G, the input reception unit 1617 may receive a selection operation for a plurality of product candidates from the product candidates.

  The information output unit 1618 outputs information (for example, product ID, product name, etc.) indicating the product to the POS terminal 11 via the connection interface 175 for the confirmed product confirmed as described above.

  The information output unit 1618 may output the number of sales separately input via the touch panel 105 or the keyboard 107 to the POS terminal 11 together with the product ID and the like. Further, as information output from the information output unit 1618 to the POS terminal 11, a product ID read by the information output unit 1618 from the PLU file F1 may be directly notified, or a product name for which the product ID can be specified. The file name of the product image may be notified, or the storage location (storage address in the PLU file F1) of the product ID may be notified.

  On the other hand, the CPU 61 of the POS terminal 11 has a function as the sales registration unit 611 by executing a program. The sales registration unit 611 performs sales registration of the corresponding product based on the product ID and the number of sales output from the information output unit 1618 of the product reading apparatus 101. Specifically, the sales registration unit 611 refers to the PLU file F1 and records the notified product ID and the product classification, product name, unit price, etc. corresponding to the product ID together with the number of sales in a sales master file or the like. Register sales.

  Hereinafter, the operation of the checkout system 1 will be described. First, the operation of the commodity reading apparatus 101 will be described with reference to FIG. Here, FIG. 8 is a flowchart showing the procedure of the merchandise recognition process executed by the merchandise reading apparatus 101. As a premise of this processing, it is assumed that the product candidate screen A1 is displayed on the display device 106.

  When this process is started in response to the start of product registration by the POS terminal 11, the image capturing unit 1611 outputs an imaging on signal to the imaging unit 164 and starts imaging by the imaging unit 164 (step S11). .

  The image capturing unit 1611 captures a frame image (captured image) captured by the image capturing unit 164 and stored in the RAM 163 (step S12). Next, the product detection unit 1612 detects all or part of the product G from the captured image captured in step S12 (step S13). Here, when the commodity G is not detected (step S13; No), the process returns to step S12.

  When the product G is detected in step S13 (step S13; Yes), the guidance display unit 1616 starts the guidance display process in the background of the product recognition process (step S14). The guidance display process will be described later.

  The feature amount extraction unit 1613 extracts the feature amount of the product G detected in step S13 from the captured image captured in step S12 (step S15). Next, the similarity determination unit 1614 compares the feature quantity extracted in step S15 with the feature quantity of each registered product registered in the PLU file F1, and calculates the similarity (step S16).

  The similarity determination unit 1614 determines whether or not there is a registered product whose value is equal to or greater than a predetermined threshold among the calculated similarities (step S17). Here, when it is determined that there is no registered product equal to or greater than the threshold (step S17; No), the process returns to step S12.

  On the other hand, when it is determined in step S17 that there is a registered product equal to or greater than the threshold (step S17; Yes), the similarity determination unit 1614 recognizes the product ID of the registered product as a product candidate.

  The commodity candidate display unit 1615 reads the commodity candidate record recognized in step S17 from the PLU file F1, and displays it in the display area A12a of the commodity candidate area A12 (step S18). Subsequently, the input receiving unit 1617 determines whether the selection of a product candidate has been received via the touch panel 105 or the keyboard 107 (step S19). When the selection operation is performed (step S19; Yes), the input reception unit 1617 sets the selected product candidate as a confirmed product corresponding to the product G, and proceeds to step S20. In Step S19, when selection of a product candidate is not accepted (Step S19; No), the process returns to Step S12 again.

  In step S20, the information output unit 1618 outputs information such as a product ID indicating a confirmed product to the POS terminal 11 (step S20), and proceeds to step S21. If the sales quantity is separately input via the touch panel 105 or the keyboard 107, the sales quantity is output to the POS terminal 11 together with information indicating the confirmed product in step S21. If the sales quantity is not input, the sales quantity “1” may be output as a default value.

  In the subsequent step S21, the CPU 161 determines whether or not there is an end of business from the POS terminal 11 based on a product registration end notification or the like (step S21). When the business is continued (step S21; No), the CPU 161 returns the process to step S12 to continue the process. When the job is to be ended (step S21; Yes), the image capturing unit 1611 outputs an imaging off signal to the imaging unit 164, thereby ending imaging by the imaging unit 164 (step S22), and product recognition processing. Exit. In step S21, when the guidance display unit 1616 is measuring the elapsed time, it is preferable to reset (stop) the measurement of the elapsed time.

  Next, the guidance display process described above will be described. FIG. 9 is a flowchart showing a procedure of guidance display processing executed by the product reading apparatus 101.

  First, the guidance display unit 1616 determines whether or not the time is being measured (step S31). If it is determined that the time is being measured (step S31; Yes), the guidance display unit 1616 proceeds to step S33. When it is determined that the time is not being measured (step S31; No), the guidance display unit 1616 starts the time measurement (step S32), and then proceeds to step S33.

  Next, the guidance display unit 1616 determines whether or not the elapsed time since the start of timekeeping has reached a predetermined time (step S33). If it is determined that the predetermined time has not been reached (step S33; No), the guidance display unit 1616 returns to step S31.

  When it is determined that the predetermined time has been reached (timeout) (step S33; Yes), the guidance display unit 1616 stops timing (step S34). Next, the guidance display unit 1616 stops the operation of the similarity determination unit 1614 (step S35), and then proceeds to step S36. In step S36, the guidance display unit 1616 determines whether a product candidate is recognized in the above-described product recognition process (step S36).

  When it is determined that the commodity candidate is recognized (step S36; Yes), the guidance display unit 1616 displays the first guidance screen A2a described in FIG. 6 (step S37). The guidance display unit 1616 stands by until any button is operated on the first guidance screen A2a (step S38; No).

  When the operation of the re-recognition button B2 is accepted on the first guidance screen A2a (step S38; Yes), the guidance display unit 1616 deletes the first guidance screen A2a (step S39). And the guidance display part 1616 complete | finishes a process, after restarting operation | movement of the similarity determination part 1614 (step S43). Note that the first guidance screen A2a is also deleted when a product candidate is selected or the manual registration button B1 is operated on the product candidate screen A1 in a superimposed relationship.

  On the other hand, when it is determined in step S36 that the product candidate is not recognized (step S36; No), the guidance display unit 1616 displays the second guidance screen A2b described in FIG. 7 (step S40). Then, the guidance display unit 1616 waits until any button is operated on the second guidance screen A2b (Step S41; No → Step S44; No).

  When the operation of the re-recognition button B3 is accepted on the second guidance screen A2b (step S41; Yes), the guidance display unit 1616 deletes the second guidance screen A2b (step S42). And the guidance display part 1616 complete | finishes a process, after restarting operation | movement of the similarity determination part 1614 (step S43).

  When the operation of the close button B4 is accepted on the second guidance screen A2b (Step S41; No → Step S44; Yes), the guidance display unit 1616 deletes the second guidance screen A2b (Step S45). Then, the guidance display unit 1616 ends the process while the similarity determination unit 1614 is stopped.

  Next, the processing operation of the POS terminal 11 will be described. FIG. 10 is a flowchart showing a procedure of sales registration processing executed by the POS terminal 11.

  First, when processing is started in accordance with the start of product registration in response to an operation instruction on the keyboard 22, the CPU 61 outputs the product ID of the confirmed product output by the product reading device 101 in step S21 in FIG. Are received (step S51). Next, the sales registration unit 611 reads the product type, unit price, and the like from the PLU file F1 based on the product ID and the sales quantity received in step S51, and stores the sales of the product G read by the product reading device 101 as a sales master file. (Step S52).

  Subsequently, the CPU 61 determines whether or not the business has ended due to the end of sales registration or the like by an operation instruction of the keyboard 22 (step S53). When business is continued (step S53; No), the CPU 61 returns to step S51 and continues the process. When the business is finished (step S53; Yes), the CPU 61 finishes the sales registration process.

  As described above, according to the present embodiment, the guidance screen that prompts the user to shift to another process is displayed at a predetermined timing while the similarity determination unit 1614 executes the process related to the recognition of the product candidate. . Thereby, since it can suppress that the imaging of goods is continued carelessly, the work efficiency (productivity) concerning recognition of goods can be improved.

  As mentioned above, although embodiment of this invention was described, the said embodiment was shown as an example and is not intending limiting the range of invention. The above embodiment can be implemented in various other forms, and various omissions, replacements, changes, additions, and the like can be made without departing from the scope of the invention. Moreover, the said embodiment and its deformation | transformation are included in the range of the invention, the summary, and the invention described in the claim, and its equal range.

  For example, in the above embodiment, the operation (function) of the similarity determination unit 1614 is stopped along with the display of the guidance screen. However, the operation may be continued. When this form is adopted, while the guidance screen is displayed, it may be controlled not to display the newly recognized product candidate in the product candidate area A12, or the operation of the re-recognition buttons B2 and B3 You may control to display with.

  In the above embodiment, the POS terminal 11 includes the PLU file F1. However, the present invention is not limited to this, and the commodity reading apparatus 101 may include a part or all of the PLU file F1.

  In the above embodiment, the commodity reading apparatus 101 recognizes commodity candidates. However, the POS terminal 11 may have a part or all of the functional units included in the commodity reading apparatus 101.

  For example, the POS terminal 11 includes a feature amount extraction unit 1613 and a similarity determination unit 1614, and the product reading device 101 includes an image capture unit 1611, a product detection unit 1612, a product candidate display unit 1615, a guidance display unit 1616, and an input. It is good also as a form provided with the reception part 1617 and the information output part 1618. FIG. In this case, on the product reading apparatus 101 side, a captured image that is captured by the image capturing unit 1611 and detected by the product detection unit 1612 is transmitted to the POS terminal 11. The product reading apparatus 101 receives the result of the product (registered product) recognized on the POS terminal 11 side, and the product candidate display unit 1615 displays the received result as a product candidate. In addition, when the POS terminal 11 includes all the functional units of the product reading device 101, the product reading device 101 functions as an imaging device, and the POS terminal 11 uses a captured image transmitted from the product reading device 101 as a product candidate or the like. Is displayed.

  In the above embodiment, an example using a stationary scanner device (product reading device 101) has been described. However, the present invention is not limited to this, and a so-called handy type scanner device is connected to the POS terminal 11. Also good.

  Further, in the above embodiment, the embodiment of the present invention is applied to the product reading device 101 out of the checkout system 1 including the POS terminal 11 and the product reading device 101. However, the present invention is not limited to this. Applied to a single-unit apparatus having the functions of the terminal 11 and the commodity reading apparatus 101 or a checkout system configured by connecting the commodity reading apparatus 101 and the POS terminal 11 shown in FIG. You may make it do. As a single device having the functions of the POS terminal 11 and the commodity reading device 101, there is a self-checkout device (hereinafter simply referred to as self-POS) installed and used in a store such as a supermarket.

  Here, FIG. 11 is a perspective view showing an external configuration of the self-POS 200, and FIG. 12 is a block diagram showing a hardware configuration of the self-POS 200. In addition, below, the same code | symbol is attached | subjected and shown about the same structure shown by FIG.1 and FIG.2, and the duplicate description is abbreviate | omitted.

  As shown in FIGS. 11 and 12, the main body 202 of the self-POS 200 includes a display device 106 having a touch panel 105 disposed on the surface thereof, and a product reading unit that reads product images for recognizing (detecting) product types and the like. 110.

  For example, a liquid crystal display is used as the display device 106. The display device 106 is a guidance screen for notifying the customer of the operation method of the self-POS 200, various input screens, a registration screen for displaying product information read by the product reading unit 110, a total price of goods, a deposit price, and change. Display the amount, etc. and display the payment screen for selecting the payment method.

  The product reading unit 110 reads a product image by the imaging unit 164 by a customer holding the code symbol attached to the product over the reading window 103 of the product reading unit 110.

  In addition, a product placement table 203 is provided on the right side of the main body 202 for placing an unsettled product in a basket, and a product placement table 204 is provided on the left side of the main body 202 for placing a settled product. Moreover, a bag hook 205 for hanging a bag for putting a product already settled, and a temporary placing table 206 for temporarily placing the settled product before putting it in the bag are provided. The product placement tables 203 and 204 are provided with weighing devices 207 and 208, respectively, and have a function of confirming that the product weight is the same before and after settlement.

  Further, the main body 202 of the self-POS 200 is provided with a changer 201 for depositing banknotes for settlement and receiving fishing bills.

  When the embodiment of the present invention is applied to the self-POS 200 having such a configuration, the self-POS 200 functions as an information processing apparatus. Note that the one-device apparatus having the functions of the POS terminal 11 and the commodity reading apparatus 101 is not limited to the self-POS 200 having the above-described structure, and may be an apparatus configured without the measuring instruments 207 and 208. .

  In addition, the program executed by each device of the above-described embodiment is provided by being incorporated in advance in a storage medium (ROM or storage unit) included in each device, but is not limited thereto. For example, an installable or executable file is recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk). Also good. Furthermore, the storage medium is not limited to a medium independent of a computer or an embedded system, but also includes a storage medium in which a program transmitted via a LAN, the Internet, or the like is downloaded and stored or temporarily stored.

  Further, the program executed by each device of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, the program executed by each device of the above embodiment may be distributed via a network such as the Internet.

  Moreover, the program of the said embodiment is good also as a form which incorporates in portable information terminals, such as a mobile telephone, a smart phone, and PDA (Personal Digital Assistant) which have a communication function, and implement | achieves the function.

DESCRIPTION OF SYMBOLS 1 Checkout system 11 POS terminal 611 Sales registration part 101 Goods reading apparatus 1611 Image capture part 1612 Goods detection part 1613 Feature amount extraction part 1614 Similarity determination part 1615 Goods candidate display part 1616 Guidance display part 1617 Input reception part 1618 Information output F1 PLU file SC store computer

JP2013-89085A

Claims (6)

Capture means for capturing an image captured by the imaging means;
Detecting means for detecting a product from the image captured by the capturing means;
A recognition unit that compares the feature amount of the product detected by the detection unit with the feature amount of each reference product that serves as a reference, and executes a recognition process for recognizing a reference product whose similarity is equal to or greater than a threshold as a product candidate; ,
Product candidate display means for displaying the product candidates in a selectable manner;
Guidance display means for displaying a guidance screen that prompts a transition to another process at a predetermined timing while the recognition means executes the recognition process;
An information processing apparatus comprising:   The information processing apparatus according to claim 1, wherein the guidance display unit switches contents displayed on the guidance screen according to whether or not the recognition unit has recognized the commodity candidate.   The guidance display means is based on the time when the detection means detects the product or the time when the product candidate display means starts displaying the product candidate, and at the timing when a predetermined time has elapsed from the base point, the guidance screen The information processing apparatus according to claim 1 or 2, wherein the information is displayed.   The information processing apparatus according to any one of claims 1 to 3, wherein the recognition unit stops the recognition process with the display of the guidance screen.   The information processing apparatus according to claim 4, wherein the guidance display unit displays the guidance screen including an operation element for instructing resumption of the recognition process. Computer
Capture means for capturing an image captured by the imaging means;
Detecting means for detecting a product from the image captured by the capturing means;
A recognition unit that compares the feature amount of the product detected by the detection unit with the feature amount of each reference product that serves as a reference, and executes a recognition process for recognizing a reference product whose similarity is equal to or greater than a threshold as a product candidate; ,
Product candidate display means for displaying the product candidates in a selectable manner;
Guidance display means for displaying a guidance screen that prompts a transition to another process at a predetermined timing while the recognition means executes the recognition process;
Program to make it function.

Images