JP2013089087A - Commodity reader and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a commodity reader capable of recognizing an image at a timing intended by an operator and a program.SOLUTION: The commodity reader includes: imaging means for imaging temporally successive multiple images; image recognition means for recognizing an image of a commodity included in the images and deriving candidate commodities of the commodity; display means; display control means for causing the display means to display candidate commodities; operation reception means for receiving a selection operation of one of candidate commodities displayed in the display means; action detection means for detecting an action of a user from the multiple images; action determination means for determining whether the action detected by the detection means corresponds to a prescribed action pattern; and control means for controlling operations of the image recognition means and the display control means on the basis of a result of the determination by the action determination means.

Description

  Embodiments described herein relate generally to a commodity reading apparatus and a program.

  Conventionally, in a store such as a supermarket, it is possible to register a product by reading a barcode attached to the product with a scanner device or the like. In addition, products that are not attached with barcodes are manually registered by an operator such as a store clerk performing a key operation, but there is a problem that the key operation is complicated.

  Therefore, conventionally, there has been proposed an apparatus for recognizing a product from an image obtained by capturing the product by using an image recognition technology and registering a product selected by an operator among one or more candidate products. Yes.

  However, the conventional technology has a problem that reading efficiency is low because the product needs to be stationary for several seconds when the product is imaged. In addition, when the operator recognizes the image while moving the product, if the image is recognized at an inappropriate timing not intended by the operator, the recognition efficiency may decrease due to the product being out of the image recognizable range. There was sex.

  The commodity reading apparatus according to the embodiment includes an imaging unit, an image recognition unit, a display unit, a display control unit, an operation reception unit, an operation detection unit, an operation determination unit, and a control unit. The imaging means captures a plurality of temporally continuous images. The image recognition means recognizes a product included in the image and derives a candidate product that is a candidate for the product. The display control means displays each candidate product on the display means. The operation accepting unit accepts an operation for selecting a candidate product displayed on the display unit. The motion detection means detects a user motion from a plurality of images. The motion determination means determines whether or not the motion detected by the motion detection means corresponds to a predetermined motion pattern. The control unit controls the operations of the image recognition unit and the display control unit based on the determination result of the operation determination unit.

FIG. 1 is a perspective view showing an example of a checkout system including a commodity reading apparatus according to the first embodiment. FIG. 2 is a diagram schematically illustrating a hardware configuration of the product reading device and the product checkout device according to the first embodiment. FIG. 3 is a diagram schematically illustrating an example of a data configuration of an operation database according to the first embodiment. FIG. 4 is a diagram illustrating an operation example performed by an operator. FIG. 5 is a diagram schematically showing an operation example of the operator. FIG. 6 is a diagram schematically illustrating an operation example of an operator. FIG. 7 is a diagram schematically illustrating an operation example of an operator. FIG. 8 is a diagram schematically illustrating an operation example of an operator. FIG. 9 is a diagram schematically illustrating an example of a data configuration of a product database. FIG. 10 is a diagram schematically illustrating an example of the data configuration of a PLU file. FIG. 11 is a diagram illustrating a functional configuration of the commodity reading apparatus according to the first embodiment. FIG. 12 is a diagram for explaining the operation of the operation determination unit. FIG. 13 is a diagram for explaining the operation of the operation determination unit. FIG. 14 is a diagram for explaining the operation of the operation determination unit. FIG. 15 is a diagram for explaining the operation of the image recognition unit. FIG. 16 is a flowchart showing a procedure of first product reading processing executed by the product reading apparatus according to the first embodiment. FIG. 17 is a flowchart showing a procedure of the image recognition process shown in FIG. FIG. 18 is a diagram illustrating an example of a product image displayed on the display / operation unit. FIG. 19 is a diagram schematically illustrating a hardware configuration of the commodity reading apparatus according to the second embodiment. FIG. 20 is a diagram schematically illustrating an example of a data configuration of an operation database according to the second embodiment. FIG. 21 is a diagram illustrating a functional configuration of a commodity reading apparatus according to the second embodiment. FIG. 22 is a flowchart showing a procedure of second product reading processing executed by the product reading apparatus according to the second embodiment. FIG. 23 is a diagram illustrating an example of a confirmation image displayed on the display / operation unit. FIG. 24 is a diagram for explaining an example in which two imaging units are configured.

  Below, the goods reading apparatus and program which concern on this embodiment are demonstrated with reference to drawings for the example of the checkout system introduced in stores, such as a supermarket.

[First Embodiment]
FIG. 1 is a perspective view showing an example of a checkout system 1 including a commodity reading apparatus 10 according to the first embodiment. As shown in FIG. 1, the checkout system 1 includes a product reading device 10 and a product checkout device 20.

  The commodity reading apparatus 10 is disposed substantially at the center of the counter table 31 and includes a housing 11 and a display / operation unit 12. The housing 11 is formed in a rectangular shape, and a reading window 13 is provided substantially at the center of the surface on the operation side of the operator. An imaging unit 54 (see FIG. 2), which will be described later, is provided on the back side of the reading window 13.

  The display / operation unit 12 is provided on the upper portion of the housing 11. The display / operation unit 12 displays characters, images, and the like under the control of a display / operation control unit 75 (see FIG. 11), which will be described later, and receives an operation input from an operator who operates the product reading apparatus 10. The display / operation unit 12 is realized using, for example, a touch panel display.

  A load receiving surface 311 is formed on the upper surface of the counter table 31. On the cargo receiving surface 311, a shopping basket 41 for storing the product G to be purchased is placed. The shopping basket 41 is divided into a first shopping basket 41a brought in by a customer and a second shopping basket 41b placed at a position sandwiching the commodity reading device 10 from the first shopping basket 41a. The shopping basket 41 is not limited to a so-called basket shape, and may be a tray shape, a box shape, a bag shape, or the like.

  In the first shopping basket 41a, a commodity G related to one transaction is stored. The commodity G in the first shopping basket 41a is moved to the second shopping basket 41b by an operator who operates the commodity reading apparatus 10. In this movement process, the product G is presented on the reading window 13 of the product reading device 10. At this time, the imaging unit 54 (see FIG. 2) in the reading window 13 images the product G held by a user (hereinafter referred to as an operator including the operator of the apparatus).

  The merchandise reading apparatus 10 performs two types of merchandise registration operations by using an image obtained by capturing the merchandise G. As the first product registration operation, the product reading device 10 reads a code symbol such as a barcode attached to the product G from the image of the product G captured by the imaging unit 54, and is included in the code symbol. The product code is registered as the product code by outputting it to the product checkout device 20 (POS terminal 21). Here, the product code is identification information for identifying each product sold in the store, and a unique product code is assigned in advance for each product type. In the present embodiment, it is assumed that a code symbol is attached to some products sold in the store.

  As the second product registration operation, the product reading device 10 detects the operation of the operator holding the product G from a plurality of temporally continuous images captured by the imaging unit 54, and the detected operation is determined in advance. In a case corresponding to the predetermined operation pattern, the candidate product is presented to the display / operation unit 12 by recognizing the product G in the image. Then, the product reading device 10 registers the product by outputting the product code of the product corresponding to the candidate selected by the operator to the product settlement apparatus 20 (POS terminal 21). Here, as will be described later, the candidate product is displayed on the display / operation unit 12 as a product image representing the product with characters or images (photos, illustrations, icons, etc.).

  On the other hand, the commodity checkout apparatus 20 is provided on a checkout table 32 that is arranged so as to form an L shape with the counter table 31. The product checkout apparatus 20 includes a POS terminal 21 and a drawer 22, and the POS terminal 21 is placed on the drawer 22. The drawer 22 stores banknotes and coins, and is controlled by the POS terminal 21 for opening operation.

  On the upper surface of the POS terminal 21, a keyboard 23 that is pressed by an operator is disposed. The keyboard 23 has numeric keys such as “1”, “2”, “3”... And a multiplication key “x” displayed on the top surface, a numeric keypad, a temporary tightening key, a tightening key, etc. A).

  A display / operation unit 24 such as a touch panel display for displaying information to the operator is provided on the back side of the keyboard 23 as viewed from the operator who operates the keyboard 23. On the far side of the display / operation unit 24, a customer display unit 25 is provided so as to be rotatable. The customer display unit 25 displays information on the display surface. In addition, although the display part 25 for customers shown in FIG. 1 has the display surface facing the front side in FIG. 1, by rotating the display part 25 for customers so that the display surface faces the back side in FIG. The customer display unit 25 displays information for the customer.

  The POS terminal 21 registers information such as the product classification, product name, and unit price of the product corresponding to the product code in a sales master file (not shown) based on the product code input from the product reading device 10. Further, the POS terminal 21 calculates the total amount for one transaction from the unit price and the number of sales points of each product related to one transaction, and performs settlement based on the total amount.

  FIG. 2 is a diagram schematically illustrating a hardware configuration of the checkout system 1 (the product reading device 10 and the product checkout device 20).

  The commodity reading apparatus 10 includes a microcomputer 50 as an information processing unit that executes information processing. The microcomputer 50 is configured by connecting a ROM (Read Only Memory) 52 and a RAM (Random Access Memory) 53 to a CPU (Central Processing Unit) 51 that executes various arithmetic processes and controls each unit.

  The CPU 51 executes a program stored in the ROM 52 or the storage unit 56 to realize a functional unit (see FIG. 11) described later. The ROM 52 stores various programs executed by the CPU 51. The RAM 53 functions as a work area for the CPU 51.

  In addition to the display / operation unit 12 described above, the imaging unit 54, the communication unit 55, and the storage unit 56 are connected to the CPU 51 via various input / output circuits (none of which are shown).

  The imaging unit 54 is an image sensor such as a color CCD or color COMS, and performs imaging from the reading window 13 under the control of the CPU 51. For example, the imaging unit 54 captures an image at 30 fps. The imaging direction of the imaging unit 54 is adjusted so that the product G presented in front of the reading window 13 is within the imaging range.

  The communication unit 55 is a communication interface for connecting to a later-described first communication unit 64 of the POS terminal 21, and transmits / receives various data to / from the POS terminal 21.

  The storage unit 56 is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs and data executed by the CPU 51. In addition, the storage unit 56 includes an operation database 561 that associates a predetermined operation pattern with a command corresponding to the operation pattern, and a barcode as data related to a first commodity reading process (see FIG. 16) described later. Stored is a product database 562 in which feature information indicating the characteristics of each product that has not been set, a product code of the product, and a product image are associated with each other.

  FIG. 3 is a diagram schematically illustrating an example of the data configuration of the operation database 561. As shown in the figure, the operation database 561 stores one or a plurality of predetermined operation patterns in association with identification numbers for identifying the operation patterns and commands corresponding to the operation patterns. The command column stores a “recognition presentation command” for instructing execution of image recognition, which will be described later, and presentation of a candidate product in the image recognition. In addition, the “recognition presentation command” is an operation corresponding to another operation (hereinafter referred to as an image recognition operation) other than an operation (hereinafter referred to as a code scanning operation) performed at the time of reading a code symbol in the commodity reading apparatus 10 Patterns are associated and registered. Hereinafter, the operation pattern will be described with reference to FIGS.

  4 to 8 are diagrams schematically illustrating an operation example of the operator. In the case of the system configuration described with reference to FIG. 1, the code scanning operation mainly includes, for example, an operation of passing the product G from the right to the left in front of the reading window 13 of the product reading apparatus 10 as illustrated in FIG. 4. If the barcode cannot be read well, an operation is performed to bring the product G closer to the reading window 13 of the product reading apparatus 10 as shown in FIG. In such a case, the “recognition presentation command” is associated with a predetermined operation pattern other than the operations shown in FIGS.

  In the operation database 561 in FIG. 3, an operation pattern representing rotational movement is stored in association with the identification number 1 as an image recognition operation. The operation pattern of the identification number 1 corresponds to the case where the product G is moved so as to draw a circle in front of the reading window 13 of the product reading device 10 as shown in FIG. Further, the motion database 561 stores a motion pattern indicating a motion of shaking up and down (shake) in association with the identification number 2. As shown in FIG. 7, the operation pattern of the identification number 2 corresponds to a case where an operation of shaking the product G up and down (vertical direction) is performed in front of the reading window 13 of the product reading apparatus 10. Further, the motion database 561 stores a motion pattern representing rotation in association with the identification number 3. As shown in FIG. 8, the operation pattern of the identification number 3 is an operation of rotating the product G around the rotation axis AX orthogonal to the surface including the reading window 13 in front of the reading window 13 of the product reading device 10. Corresponds to the case where

  The “recognition presentation command” is preferably associated with an operation pattern of the image recognition operation that easily distinguishes the difference from the code scanning operation and does not place an excessive load on the operator. Since the code scanning operations shown in FIGS. 4 and 5 are operations in which the locus is drawn on a substantially horizontal plane, the image recognition operations intersect as shown in FIGS. 6, 7, and 8. It is easy to discriminate if the operation includes a lot of movement. More specifically, the movement component includes a vertical component. Note that the operation pattern of the image recognition operation registered in the operation database 561 is not limited to the example in FIG. 3, and an arbitrary operation pattern can be registered.

  FIG. 9 is a diagram schematically illustrating an example of the data configuration of the product database 562. As shown in the figure, in the product database 562, feature information indicating features of each product that is not attached with a barcode among products sold in a store, a product code of the product, and a product image are stored. It is stored in association.

  Here, the feature information is information that characterizes the product such as the shape, surface color, and pattern of the product, and an image obtained by capturing a part or the whole of the product, a feature vector that will be described later, and the like can be used. The product image is a product image captured in advance, an icon representing the product, a character string indicating the product name, or the like.

  Returning to FIG. 2, the commodity settlement apparatus 20 also includes a microcomputer 60 as an information processing unit that executes information processing. The microcomputer 60 is configured by connecting a ROM 62 and a RAM 63 to a CPU 61 via a bus.

  The drawer 61, the keyboard 23, the display / operation unit 24, and the customer display unit 25 are all connected to the CPU 61 via various input / output circuits (not shown). These are controlled by the CPU 61.

  The CPU 61 is connected to a first communication unit 64, a second communication unit 65, a printer 66 and a storage unit 67.

  The first communication unit 64 is a communication interface that enables data transmission / reception with the communication unit 55 of the commodity reading apparatus 10. The second communication unit 65 is a communication interface that enables data transmission / reception with the store computer SC. Here, the store computer SC is installed in a store backyard or the like, and holds a PLU file 671 distributed to the POS terminal 21. The printer 66 is a printing device such as a thermal printer, and prints transaction details of one transaction on a receipt under the control of the CPU 61.

  The storage unit 67 is a storage device such as an HDD or an SSD, and stores various programs executed by the CPU 61 and various files. The storage unit 67 stores a PLU file 671 distributed from the store computer SC. The PLU file 671 is a file that stores the product name, unit price, etc. of each product sold at the store.

  FIG. 10 is a diagram schematically illustrating an example of the data configuration of the PLU file 671. As shown in FIG. 10, in the PLU file 671, for each product, a product code of the product and information about the product such as the product classification, product name, and unit price to which the product belongs are stored as product information. When the product code is input from the product reading device 10, the CPU 61 reads product information corresponding to the product code from the PLU file 671 and registers it in a sales master file (not shown).

  Next, each functional unit realized by the CPU 51 of the commodity reading apparatus 10 executing a program will be described.

  FIG. 11 is a diagram illustrating a functional configuration of the commodity reading apparatus 10. As shown in the figure, the CPU 51 of the commodity reading apparatus 10 executes a program to thereby execute an image acquisition unit 71, a code symbol reading unit 72, an operation determination unit 73, an image recognition unit 74, a display / operation. A control unit 75 and a function as the control unit 76 are provided.

  The image acquisition unit 71 sequentially acquires images captured by the imaging unit 54 in the RAM 53 and the like. The code symbol reading unit 72 reads the code symbol attached to the product G from the image acquired by the image acquisition unit 71. Further, the code symbol reading unit 72 acquires the product code included in the code symbol by decoding the read code symbol, and outputs the acquired product code to the control unit 76.

  The operation determination unit 73 detects the operation of the operator holding the product G from the plurality of temporally continuous images acquired by the image acquisition unit 71. Here, the operation to be detected indicates an operation composed of a series of movements of the product G within the field of view of the imaging unit 54. As an operation detection method, for example, an optical flow 81 is first derived from the correspondence between two temporally continuous images as shown in FIG. 12, and an average of the derived optical flow 81 is obtained. As shown in FIG. 13, a main motion vector 82 is acquired.

The motion determination unit 73 performs the above-described detection of the motion vector 82 every time a new image is acquired by the image acquisition unit 71, and obtains a motion trajectory by sequentially connecting the obtained motion vectors 82. The motion determination unit 73 compares the obtained trajectory with each motion pattern stored in the motion database 561 to determine whether it corresponds to any motion pattern. When the obtained movement trajectory corresponds to any one of the movement patterns stored in the movement database 561, the movement determination unit 73 sends a command associated with the movement pattern (in the case of this embodiment, a recognition presentation command). It outputs to the control part 76. The detection of the operation can be performed using a known technique, and can be realized by using a method described in the following document, for example.
Ross Cutler and Matthew Turk, “View-based interpretation of real-time optical flow for gesture recognition”, Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, pp.416-421, 1998.

  In addition to the above-described motion determination, the motion determination unit 73 selects an image suitable for image recognition from the series of images used for the determination, and outputs the image to the control unit 76 together with the command. This is performed by, for example, a method of selecting an image at the time when the trajectory passes through the center of the field of view of the imaging unit 54 or selecting an image corresponding to a specific part of the trajectory. The number of images to be selected is not limited to one, and a plurality of images may be selected. Further, as shown in FIG. 14, a portion of an image in which the size of the optical flow 81 exceeds a predetermined threshold is extracted as a subject region 83 and only the subject region 83 is cut out as an image suitable for image recognition. An image may be output.

  Although an example in which a two-dimensional trajectory is obtained from an optical flow has been described above, the present invention is not limited to this, and the motion of a subject in a three-dimensional space can also be obtained by using a motion stereo technique. Hereinafter, an outline of a method for obtaining the relative movement of the subject with respect to the imaging unit 54 from the correspondence between the points in the two images will be described.

  When the point in the three-dimensional space and the projected image on the image plane are expressed by the homogeneous coordinates x and X, respectively, the projection relationship by the imaging unit 54 is expressed by the following formula (1) using the projection matrix P.

  Here, the projection matrix P is expressed by the following equation (2) using the 3 × 3 upper triangular matrix K representing the internal parameters of the imaging unit 54, the 3 × 3 rotation matrix R, and the three-dimensional translation vector t. Can be represented.

Here, K is specifically a matrix such as the following formula (3), f _u and f _v are focal lengths, u ₀ and v ₀ are coordinates of the intersection (image center) between the optical axis and the image plane. It is assumed that these parameters are obtained in advance.

Now, when two images are given, the respective projection matrices P ₁ and P ₂ are defined as in the following equations (4) and (5). Here, I is a 3 × 3 unit matrix, and 0 is a three-dimensional zero vector.

Points represented by homogeneous coordinates X, when the homogeneous coordinates on the two images and x _1, x _2, uses the projection matrix P _1, P ₂ between the X and x _1, x ₂ Thus, the following relationships (6) and (7) are established.

If z ₁ and z ₂ are defined as in the following formulas (8) and (9), a constraint condition of the following formula (10) is established between z ₁ and z ₂ .

However, in the above equation (10), E is called an essential matrix, and is represented by the following equation (11) using the rotation matrix R and the parallel movement vector t. Here, t ₁ , t ₂ , and t ₃ are elements of the parallel movement vector t.

  When the correspondence between the two images is obtained by the optical flow, the constraint condition expression of the above equation (10) is obtained for each set of correspondences. The basic matrix E of the above formula (11) that satisfies the many constraint conditions obtained in this way is obtained, and the rotation matrix R and the parallel movement vector t are obtained from the basic matrix E, whereby the relative relationship between the imaging unit 54 and the subject is obtained. Movement can be obtained.

Therefore, the motion determination is not limited to using a two-dimensional trajectory, and a trajectory in a three-dimensional space, a rotation angle of a subject, or the like may be used. In addition, as a method for obtaining the motion of the subject in the three-dimensional space from the motion on the image, for example, a method described in the following document can be used. If the technique described in the following document is used, the relative movement between the imaging unit 54 and the subject can be obtained if the correspondence between at least five pairs of points is known.
David Nister, “An Efficient Solution to the Five-Point Relative Pose Problem”, IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol.26, No. 6, June 2004, pp. 756-770.

Returning to FIG. 11, the image recognition unit 74 performs image recognition on the image input from the operation determination unit 73 via the control unit 76, thereby selecting a product that is a candidate for the product G included in the image. Derived as a candidate. For the image recognition processing, for example, a method described in the following document can be used, but the method is not limited to this, and other image recognition techniques may be used. Below, the procedure at the time of performing image recognition using the method described in the following literature is demonstrated.
Tomohiro Nakai, Satoshi Ito, Tomoyuki Shibata, Susumu Kubota, “Additional Learning Object Recognition System Using Co-occurrence Features and k-Nearest Method”, Image Recognition and Understanding Symposium (MIRU2011), pp.1701-1702, 2011.

First, the image recognition unit 74 divides an image into a plurality of blocks BL as shown in FIG. 15, and derives a feature vector representing the features of the partial image corresponding to the block BL for each block BL. Here, for example, a technique described in the following document can be used to derive the feature vector.
Satoshi Ito and Susumu Kubota, “Object classification using heterogeneous co-occurrence features”, Proceedings of the 11th European Conference of Computer Vision, vol.2, pp.209-222, 2010.

  Next, the image recognizing unit 74 searches the product database 562 for k learning samples closest to the feature vector for each of the derived feature vectors by the k-nearest neighbor method. In this case, the learning sample is a feature vector (feature information) of each product registered in the product database 562. Since k learning samples are obtained for all the blocks BL, for example, when there are N blocks BL, kN learning samples are obtained. Therefore, the image recognition unit 74 aggregates kN learning samples for each product name and uses a value normalized by dividing by kN as the similarity. Then, the image recognizing unit 74 selects a learning sample product whose calculated similarity exceeds a predetermined threshold value, and outputs the product code of the product to the control unit 76.

  The display / operation control unit 75 displays characters, images, and the like on the display screen of the display / operation unit 12 according to the control of the control unit 76 and controls the operation contents operated on the touch panel of the display / operation unit 12. To the unit 76.

  Specifically, the display / operation control unit 75 causes the display / operation unit 12 to display the product image input from the control unit 76 in a selectable manner. When the display / operation control unit 75 receives an operation for selecting any one of the product images displayed on the display / operation unit 12, the display / operation control unit 75 displays information indicating the selected product image. Output to 76.

  The control unit 76 comprehensively controls the operation of each functional unit of the commodity reading apparatus 10. When receiving the input of the product code as information indicating the successful reading of the code symbol from the code symbol reading unit 72, the control unit 76 gives priority to the input product code regardless of the progress of other processes. By outputting to the terminal 21, the product corresponding to the product code is registered. In addition, when the product code is input from the code symbol reading unit 72 and the processing is being executed by the operation determination unit 73 or the image recognition unit 74, the control unit 76 cancels the processing, It is possible to prevent duplicate product codes from being input from the image recognition unit 74.

  Further, when the control unit 76 receives an input of a command and an image as a determination result of the operator operation from the operation determination unit 73, the control unit 76 controls the image recognition unit 74 and the display / operation control unit 75 according to the contents instructed by the command. By controlling, a product image of a candidate product is presented on the display / operation unit 12.

  Specifically, the control unit 76 causes the image recognition unit 74 to perform image recognition by outputting the image received from the operation determination unit 73 to the image recognition unit 74. When image recognition succeeds in the image recognition unit 74 and a product code of one or a plurality of products is input as a candidate, the control unit 76 displays a product image associated with each of the input product codes from the product database 562. By reading and outputting to the display / operation control unit 75, the display / operation unit 12 is caused to display. When the control unit 76 receives the selection of the product image via the display / operation control unit 75, the control unit 76 outputs the product code of the product corresponding to the selected product image to the POS terminal 21. Register.

  Next, a product reading operation in the product reading apparatus 10 will be described. Here, FIG. 16 is a flowchart illustrating a procedure of a product reading process (first product reading process) executed by the product reading apparatus 10.

  First, when the display / operation unit 12 or the POS terminal 21 instructs the start of product registration related to one transaction, the control unit 76 outputs an imaging on signal to the imaging unit 54 to start imaging (steps). S11). Subsequently, the control unit 76 operates the code symbol reading unit 72 to start the code symbol reading process (step S12), and operates the operation determining unit 73 to start the operator operation determination process (step S12). Step S15).

  Here, when the product G is presented to the reading window 13 by the operation of the operator, the code symbol reading unit 72 tries to read the code symbol from the images sequentially acquired by the image acquiring unit 71 (step S13; No). When the code symbol is read (step S13; Yes), the code symbol reading unit 72 decodes the read code symbol, and outputs the product code included in the code symbol to the control unit 76.

  When the product code is input from the code symbol reading unit 72, the control unit 76 cancels other processes in steps S16 to S19 described later during the parallel operation (step S14), and converts the input product code into the POS terminal 21. To register the product (step S20). The control unit 76 cancels the processes in steps S16 to S19, and then returns the operation determination unit 73 to the process in step S16 described later.

  On the other hand, when a plurality of temporally continuous images are sequentially acquired by the image acquisition unit 71, the operation determination unit 73 sequentially detects an operator's operation from the plurality of images and is registered in the operation database 561. Wait until an operation corresponding to any one of the operation patterns is detected (step S16; No). If it is determined that an operation corresponding to any one of the operation patterns has been detected (step S16; Yes), the operation determination unit 73 obtains a command associated with the operation pattern and one or a plurality of images used for the determination. It outputs to the control part 76.

  When receiving the command and image input from the operation determination unit 73, the control unit 76 outputs the input image to the image recognition unit 74 in accordance with the contents of the command, thereby causing the image recognition unit 74 to execute image recognition processing. (Step S17).

  Here, FIG. 17 is a flowchart showing the procedure of the image recognition processing in step S17. First, the image recognition unit 74 recognizes an image input from the control unit 76, and compares the feature information of the product G included in the image with the feature information of each product registered in the product database 562. Thus, the similarity to the product is calculated (step S171).

  Next, the image recognizing unit 74 determines whether or not there is a product exceeding a predetermined threshold among the similarities with the products calculated in step S171 (step S172). Here, when the calculated similarities are all equal to or less than the threshold value, the image recognition unit 74 determines that the recognition has failed (step S172; No), and returns to step S16 in FIG. If there is a product exceeding a predetermined threshold in the similarity to each product calculated in step S171, the image recognition unit 74 determines that the recognition is successful (step S172; Yes), and the product of the corresponding product. After the code is output as a candidate to the control unit 76 (step S173), the process proceeds to step S18 in FIG.

  Returning to FIG. 16, upon receiving the product code output from the image recognition unit 74 in the image recognition processing in step S17, the control unit 76 reads the product image corresponding to each of the product codes from the product database 562, and displays / displays the product image. The product image is displayed (presented) on the display / operation unit 12 by outputting to the operation control unit 75 (step S18).

  Here, FIG. 18 is a diagram illustrating an example of a product image displayed on the display / operation unit 12. In the figure, an area A11 is a display area for displaying an image picked up by the image pickup unit 54. Under the control of the control unit 76, an image being picked up by the image pickup unit 54 or an operation determination unit 73 is displayed. The input image is displayed. The area A12 is a display area for displaying a product image of each product that is a candidate. The figure shows an example in which six product images G11 to G16 are displayed.

  Returning to FIG. 16, the display / operation control unit 75 waits until any one product image is selected from the product images displayed on the display / operation unit 12 (step S19; No), and the specific product image. Is selected (step S19; Yes), information indicating the selected product image is output to the control unit 76.

  When a specific product image is instructed from the display / operation control unit 75, the control unit 76 reads out a product code corresponding to the instructed product image from the product database 562 and outputs the product code to the POS terminal 21. Then, the product is registered (step S20). Note that the operator can cancel without selecting the product image, and in this case, the process returns to step S16 again without registering the product.

  Subsequently, the control unit 76 determines whether or not the display / operation unit 12 or the POS terminal 21 has instructed the end of the product registration related to one transaction (step S21). Here, when the end of the product registration is not instructed (step S21; No), the control unit 76 returns the processing related to the code symbol reading unit 72 and the operation determination unit 73 to step S13 and step S16, respectively, so that Preparing for the reading of products.

  In step S21, when an instruction to end product registration is given (step S21; Yes), the control unit 76 outputs an imaging off signal to the imaging unit 54 to stop imaging by the imaging unit 54 ( Step S22), the process is terminated.

  As described above, according to the commodity reading apparatus 10 of the present embodiment, image recognition of a commodity is started with a trigger of an operator holding the commodity operating in a predetermined operation pattern. As a result, it is possible to perform image recognition at a timing intended by the operator, so that recognition efficiency can be improved, and efficiency and convenience at the time of reading a product can be improved.

  In addition, since the image recognition operation other than the code scanning operation is an operation pattern for instructing image recognition and candidate presentation, the candidate for the product is displayed on the display / operation unit 12 when the code scanning operation is performed. It can be suppressed. Thereby, for example, when a code scanning operation is performed in a store where a product with a code symbol and a product without a code symbol are mixed and sold, a candidate is selected by mistake. Therefore, the efficiency and convenience at the time of reading a product can be improved.

  In the present embodiment, the image recognition unit 74 is operated when a command and an image are input from the operation determination unit 73. However, the present invention is not limited to this, and the operation determination unit 73 starts detecting the operation. At this point, the image recognition unit 74 may be operated.

  Specifically, when detection of an operator action is started in the action determination unit 73, the control unit 76 starts image recognition by outputting the image acquired by the image acquisition unit 71 to the image recognition unit 74, When it is determined that the motion determination unit 73 corresponds to the motion pattern, the candidate product images obtained by the image acquisition unit 71 are displayed on the display / operation unit 12. As a result, the image recognition unit 74 can be operated without waiting for the determination result in the operation determination unit 73, so that the product images of each candidate product can be displayed more quickly.

[Second Embodiment]
Next, a commodity reading apparatus according to the second embodiment will be described. In the second embodiment, a description will be given of a mode in which a code scanning operation is detected in addition to the detection of an image recognition operation, and the candidate presentation method is changed according to the operation. In addition, about the member similar to 1st Embodiment, the same code | symbol is provided and description is abbreviate | omitted.

  FIG. 19 is a diagram schematically illustrating a hardware configuration of the commodity reading apparatus 10a according to the second embodiment. The commodity reading apparatus 10 a includes a CPU 51, a ROM 52, and a RAM 53 as a microcomputer 50 as an information processing unit that executes information processing. Further, the display / operation unit 12, the imaging unit 54, the communication unit 55, and the storage unit 56 a are connected to the CPU 51 via various input / output circuits (all not shown).

  The storage unit 56 a is a storage device similar to the storage unit 56 described above, and stores various programs and data executed by the CPU 51. The storage unit 56a includes, as data related to a second product reading process (see FIG. 22) described later, an operation database 561a in which a predetermined operation pattern and a command corresponding to the operation pattern are associated, and a product database 562. Is remembered.

  FIG. 20 is a diagram schematically illustrating an example of the data configuration of the operation database 561a. As shown in the figure, the operation database 561a stores one or more predetermined operation patterns in association with an identification number for identifying the operation pattern and a command corresponding to the operation pattern. Here, the action database 561a has a “recognition command” for instructing execution of image recognition in addition to the above-described “recognition presentation command” as a command stored in the command column. Different from 561.

  Here, the “recognition command” is registered in association with an operation pattern corresponding to a code symbol reading operation in the commodity reading apparatus 10a. In the case of the system configuration illustrated in FIG. 1, for example, the operation of passing the product G from the right to the left in front of the reading window 13 of the product reading device 10 illustrated in FIG. 4, or the product reading device 10 illustrated in FIG. The operation of bringing the product G closer to the reading window 13 is the code symbol reading operation. Therefore, the operation database 561a shown in FIG. 20 stores an operation pattern corresponding to the code symbol reading operation shown in FIG. Further, an operation pattern corresponding to the code symbol reading operation shown in FIG. 5 is stored in association with the identification number 12. Note that the operation pattern of the image recognition operation registered in the operation database 561a is not limited to the example of FIG. 20, and an arbitrary operation pattern can be registered.

  Next, each functional unit realized by the CPU 51 of the commodity reading apparatus 10a executing a program will be described. FIG. 21 is a diagram illustrating a functional configuration of the commodity reading apparatus 10a according to the present embodiment. As shown in the figure, the CPU 51 of the commodity reading apparatus 10a executes a program, thereby executing an image acquisition unit 71, a code symbol reading unit 72, an operation determination unit 73, an image recognition unit 74, and a display / operation. The controller 75 and the function as the controller 76a are provided.

  The control unit 76a corresponds to the control unit 76 of the first embodiment, and comprehensively controls the operation of each functional unit of the product reading device 10a. When receiving the input of the product code as the information indicating the successful reading of the code symbol from the code symbol reading unit 72, the control unit 76a gives priority to the input product code regardless of the progress of other processes. By outputting to the terminal 21, the product corresponding to the product code is registered. When the product code is input from the code symbol reading unit 72 and the process is being executed by the operation determination unit 73 or the image recognition unit 74, the control unit 76a cancels these processes, It is possible to prevent duplicate product codes from being input from the image recognition unit 74.

  When the control unit 76a receives an input of a command and an image as a determination result of the operator's motion from the motion determination unit 73, the image recognition unit 74 and the display / operation control unit 75 according to the contents instructed by the command. By controlling, a product image of a candidate product is presented on the display / operation unit 12.

  Specifically, when a recognition presentation command is input from the motion determination unit 73, the control unit 76a outputs an image received from the motion determination unit 73 to the image recognition unit 74, thereby causing the image recognition unit 74 to perform image recognition. To do. When image recognition succeeds in the image recognition unit 74 and one or more products are input as candidates, the control unit 76a reads the product image associated with each of the input product codes from the product database 562 and displays it. By outputting to the operation control unit 75, the display / operation unit 12 is displayed. When the control unit 76a receives the selection of the product image via the display / operation control unit 75, the control unit 76a outputs the product code of the product corresponding to the selected product image to the POS terminal 21 to thereby display the product code. Register.

  In addition, when a recognition command is input from the operation determination unit 73, the control unit 76a outputs the image received from the operation determination unit 73 to the image recognition unit 74, thereby causing the image recognition unit 74 to perform image recognition. . When image recognition succeeds in the image recognition unit 74 and one or more products are input as candidates, the control unit 76a holds the input product code in the RAM 53 or the like and controls the display / operation control unit 75. As a result, an image for confirming whether or not to present a candidate is displayed on the display / operation unit 12. When receiving the candidate presentation instruction via the display / operation control unit 75, the control unit 76a reads out the product image associated with each of the held product codes from the product database 562, and displays the operation / control unit. By outputting to 75, it is displayed on the display / operation unit 12. When the control unit 76a receives the selection of the product image via the display / operation control unit 75, the control unit 76a outputs the product code of the product corresponding to the selected product image to the POS terminal 21 to thereby display the product code. Register.

  Next, a product reading operation in the product reading apparatus 10a will be described. Here, FIG. 22 is a flowchart showing a procedure of a product reading process (second product reading process) executed by the product reading apparatus 10a.

  First, when the display / operation unit 12 or the POS terminal 21 instructs the start of product registration related to one transaction, the control unit 76a outputs an imaging-on signal to the imaging unit 54 to start imaging (steps). S31). Subsequently, the control unit 76a starts the code symbol reading process by operating the code symbol reading unit 72 (step S32), and starts the operator action determination process by operating the operation determining unit 73 (step S32). Step S35).

  Here, when the product G is presented to the reading window 13 by the operation of the operator, the code symbol reading unit 72 tries to read the code symbol from the images sequentially acquired by the image acquiring unit 71 (step S33; No). When the code symbol can be read (step S33; Yes), the code symbol reading unit 72 decodes the read code symbol and outputs the product code included in the code symbol to the control unit 76a.

  When the product code is input from the code symbol reading unit 72, the control unit 76a cancels other processes in steps S36 to S43 described later during the parallel operation (step S34), and the input product code is converted to the POS terminal 21. To register the product (step S44). The control unit 76a cancels the processes in steps S36 to S43, and then returns the operation determination unit 73 to the process in step S36 described later.

  On the other hand, when a plurality of temporally continuous images are sequentially acquired by the image acquisition unit 71, the operation determination unit 73 sequentially detects the operator's operation from the plurality of images and is registered in the operation database 561a. Wait until an operation corresponding to one of the operation patterns is detected (step S36; No). When it is determined that an operation corresponding to any one of the operation patterns is detected (step S36; Yes), the operation determination unit 73 obtains a command associated with the operation pattern and one or a plurality of images used for the determination. It outputs to the control part 76a.

  When receiving the command and the image input from the operation determination unit 73, the control unit 76a determines whether the command is a recognition presentation command or a recognition command (step S37). Here, when the input command is a recognition presentation command (step S37; Yes), the control unit 76a outputs the input image to the image recognition unit 74 in accordance with the contents of the command, thereby the image recognition unit. 74 is caused to execute image recognition processing (step S38). The image recognition process in step S38 is the same as the image recognition process described with reference to FIG.

  When the control unit 76a receives the product code output from the image recognition unit 74 in the image recognition processing in step S38, the control unit 76a reads the product image corresponding to each of the product codes from the product database 562, and displays it on the display / operation control unit 75. By outputting, the product image is displayed (presented) on the display / operation unit 12 (step S42).

  On the other hand, when the command input from the operation determination unit 73 is a recognition command in step S37 (step S37; No), the control unit 76a outputs the input image to the image recognition unit 74 according to the contents of the command. Thus, the image recognition unit 74 is caused to execute image recognition processing (step S39). The image recognition process in step S39 is the same as the image recognition process described with reference to FIG.

  When the control unit 76a receives the product code output from the image recognition unit 74 in the image recognition processing in step S39, the control unit 76a holds the product code in the RAM 53 or the like and controls the display / operation control unit 75, thereby A confirmation image for confirming whether or not to present is displayed on the display / operation unit 12 (step S40).

  Here, FIG. 23 is a diagram illustrating an example of a confirmation image displayed on the display / operation unit 12. In the drawing, an area A11 is a display area for displaying an image captured by the imaging unit 54 described above, and an image being captured by the imaging unit 54 is displayed under the control of the control unit 76a. Further, in the area A13, a confirmation image for confirming whether or not to present a candidate is displayed, and a button B11 for instructing the candidate presentation is displayed together with a message for prompting confirmation. . In FIG. 23, it is assumed that a candidate is instructed when the button B11 is selected.

  Returning to FIG. 22, the display / operation control unit 75 waits until the display of the candidate is instructed from the display / operation unit 12 (step S <b> 41; No), and when the display of the candidate is instructed (step S <b> 41; Yes). The display / operation control unit 75 outputs information instructing candidate presentation to the control unit 76a. When the display / operation control unit 75 instructs the presentation of the candidate, the control unit 76a proceeds to step S42, reads the product image associated with each of the stored product codes from the product database 562, and displays it. -By outputting to the operation control part 75, it is displayed on the display / operation part 12 (step S42).

  Subsequently, the display / operation control unit 75 waits until any one product image is selected from the product images displayed on the display / operation unit 12 (step S43; No), and a specific product image is selected. If so (step S43; Yes), information indicating the selected product image is output to the control unit 76a.

  When a specific product image is instructed from the display / operation control unit 75, the control unit 76a reads a product code corresponding to the instructed product image from the product database 562, and outputs the product code to the POS terminal 21. Then, the product is registered (step S44). Note that the operator can cancel without selecting the product image, and in this case, the process returns to step S36 again without registering the product.

  Subsequently, the control unit 76a determines whether or not the display / operation unit 12 or the POS terminal 21 has instructed the end of product registration related to one transaction (step S45). Here, when the end of the product registration is not instructed (step S45; No), the control unit 76 returns the processing related to the code symbol reading unit 72 and the operation determination unit 73 to step S33 and step S36, respectively. Preparing for the reading of products.

  In step S45, when an instruction to end product registration is given (step S45; Yes), the control unit 76a outputs an imaging off signal to the imaging unit 54 to stop imaging by the imaging unit 54 ( Step S46), the process is terminated.

  As described above, according to the commodity reading apparatus 10a of the present embodiment, image recognition of a commodity is started with a trigger that an operator holding the commodity operates in a predetermined operation pattern. As a result, it is possible to perform image recognition at a timing intended by the operator, so that recognition efficiency can be improved, and efficiency and convenience at the time of reading a product can be improved.

  Also, candidates are presented immediately during the image recognition operation, and candidates are presented according to instructions from the operator during the code scanning operation. Thereby, when the code scanning operation for presenting the code symbol attached to the product to the reading window 13 is performed, since the display of the product candidate on the display / operation unit 12 can be suppressed, the candidate is erroneously displayed. It is possible to prevent the occurrence of mistakes such as selection. In addition, by presenting candidates in response to a request from the operator, the code symbol can be supplementarily used when reading is unsuccessful, so that the efficiency and convenience at the time of reading a product can be improved.

  As mentioned above, although embodiment of this invention was described, the said embodiment was shown as an example and is not intending limiting the range of invention. The above embodiment can be implemented in various other forms, and various omissions, replacements, changes, additions, and the like can be made without departing from the scope of the invention. Moreover, the said embodiment and its deformation | transformation are included in the range of the invention, the summary, and the invention described in the claim, and its equal range.

  In the above embodiment, the code symbol reading by the code symbol reading unit 72 and the detection of the operator action by the operation determining unit 73 are performed using the image captured by the single imaging unit 54. It is not limited to this. For example, a dedicated imaging unit 54 is provided for each of the code symbol reading unit 72 and the operation determination unit 73, and code symbol reading and detection of an operator's operation are performed using images captured by each of these imaging units 54. It is good also as a form to perform. In this case, the imaging range of each imaging unit 54 is preferably adjusted so as to overlap at a position where the product G is presented in front of the reading window 13. Hereinafter, a configuration example using two imaging units 54 will be described as an example in the case of using a plurality of imaging units 54.

  FIG. 24 is a diagram for explaining an example in which two imaging units 54 are configured, and schematically shows a cross section at the reading window 13 portion of the commodity reading apparatus 10 (or 10a). As shown in the figure, in the reading window 13 of the commodity reading apparatus 10, two imaging units 54 (54a, 54b) are provided.

  In this example, it is assumed that the imaging unit 54a captures an image used for code symbol reading by the code symbol reading unit 72, and the imaging unit 54b captures an image used for detection of an operator motion by the motion determination unit 73. doing.

  Here, the imaging direction of the imaging unit 54a is adjusted so that the product G presented in front of the reading window 13 is within the imaging range A21. The imaging direction of the imaging unit 54b is adjusted so that the product G presented in front of the reading window 13 is within the imaging range A22. Furthermore, it is assumed that the imaging range A21 of the imaging unit 54a and the imaging range A22 of the imaging unit 54b are adjusted so that part or all of them overlap at the position where the product G is presented in front of the reading window 13. .

  As described above, by configuring the imaging ranges of the imaging units 54a and 54b to overlap at the position where the product G is presented, the operations and effects of the above-described embodiment can be enjoyed efficiently. Note that a scanner device such as a laser scanner may be used for reading the code symbol. However, in this case as well, the scanning range of the scanner device and the imaging range of the imaging unit 54 at the position where the product G is presented. It is preferable to adjust so that they overlap.

  Moreover, in the said embodiment, although the operation | movement database 561 and the goods database 562 were set as the form with which the goods reading apparatus 10 is provided, not only this but the operation | movement database 561 and the goods database 562 which POS terminal 21 and another apparatus hold | maintain are used. The product reading device 10 may be referred to.

  Moreover, in the said embodiment, although it was set as 2 units | sets, the goods reading apparatus 10 and the goods settlement apparatus 20, it is not restricted to this, The apparatus of 1 unit provided with the function of the goods reading apparatus 10 and the goods settlement apparatus 20 ( For example, it may be a self-register.

  In addition, the program executed by the product reading device 10 (10a) of the above embodiment is provided by being incorporated in advance in a storage medium (ROM 52 or storage unit 56 (56a)) provided in the product reading device 10 (10a). However, the present invention is not limited to this, and can be recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), etc. in a file in an installable or executable format. And may be configured to be provided. Furthermore, the storage medium is not limited to a medium independent of a computer or an embedded system, but also includes a storage medium in which a program transmitted via a LAN, the Internet, or the like is downloaded and stored or temporarily stored.

  Further, the program executed by the commodity reading apparatus 10 (10a) of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. You may comprise so that it may provide or distribute via networks, such as the internet.

DESCRIPTION OF SYMBOLS 1 Checkout system 10, 10a Goods reading apparatus 11 Housing 12 Display / operation part 13 Reading window 20 Goods settlement apparatus 21 POS terminal 22 Drawer 51 CPU
52 ROM
53 RAM
54 Imaging unit 55 Communication unit 56 Storage unit 561, 561a Operation database 562 Product database 71 Image acquisition unit 72 Code symbol reading unit 73 Operation determination unit 74 Image recognition unit 75 Display / operation control unit 76, 76a Control unit G Product

Japanese Patent No. 3223064

Claims (9)

Imaging means for imaging a plurality of temporally continuous images;
Image recognition means for recognizing products included in the image and deriving candidate products that are candidates for the products;
Display means;
Display control means for displaying each of the candidate products on the display means;
Operation accepting means for accepting an operation for selecting a candidate product displayed on the display means;
Operation detecting means for detecting a user's operation from the plurality of images;
An operation determination unit for determining whether the operation detected by the operation detection unit corresponds to a predetermined operation pattern;
Control means for controlling the operation of the image recognition means and the display control means based on the determination result of the action determination means;
A commodity reading apparatus. Reading means for reading a code symbol attached to the product;
Output means for outputting product information included in the code symbol read by the reading means or product information corresponding to a candidate product received by the operation receiving means;
The product reading apparatus according to claim 1, comprising: The operation pattern includes a first operation pattern corresponding to a predetermined operation other than an operation when the reading unit reads a code symbol attached to the product,
The control means causes the image recognition means to perform the image recognition and causes the display control means to perform the image recognition when the action determination means determines that the user's action corresponds to the first action pattern. The commodity reading device according to claim 2, wherein the candidate commodity derived in step (a) is displayed. The operation pattern has a second operation pattern corresponding to an operation when the reading unit reads a code symbol attached to the product,
The control means causes the image recognition means to perform the image recognition when the action determination means determines that the user's action corresponds to the second action pattern, and then via the operation acceptance means. The product reading apparatus according to claim 2 or 3, wherein the display control unit waits for display of the candidate image derived by the image recognition until display is instructed.   5. The commodity reading apparatus according to claim 3, wherein the first motion pattern includes a motion component that intersects with a main motion direction constituting the second motion pattern.   The product reading device according to any one of claims 3 to 5, wherein the first operation pattern represents a rotation or a shake operation.   7. The control unit according to claim 2, wherein when the code symbol is successfully read by the reading unit, the control unit cancels the processing being executed by the image recognition unit, the motion detection unit, and the motion determination unit. The commodity reading apparatus according to one item.   The control means causes the image recognition means to start the image recognition when detection of the user's action is started in the action detection means, and that the action of the operator corresponds to the action pattern in the action determination means. The product reading apparatus according to any one of claims 1 to 7, wherein when it is determined, the display control unit displays a candidate image derived by the image recognition. A computer provided with an imaging means and a display means,
Image recognition means for recognizing a product included in the image captured by the imaging means and deriving a candidate product that is a candidate for the product;
Display control means for displaying each of the candidate products on the display means;
Operation accepting means for accepting an operation for selecting a candidate product displayed on the display means;
An operation detecting means for detecting an operation of an operator holding the product from a plurality of temporally continuous images imaged by the imaging means;
An operation determination unit for determining whether the operation detected by the operation detection unit corresponds to a predetermined operation pattern;
Control means for controlling the operation of the image recognition means and the display control means based on the determination result of the action determination means;
Program to make it function.

Images