JP6330115B1 - Product management server, automatic cash register system, product management program, and product management method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve a product identification rate of a product that is taken in and out of a shopping cart while suppressing a data communication amount and a calculation processing amount, and automatically managing and paying out a product management server, an automatic cash register system, and a product Provide management program and product management method. A merchandise management server 1 for managing merchandise to be taken in and out of a shopping cart, a hand position data obtaining unit 41 for obtaining hand position data from hand position detecting means, and a hand from outside the shopping cart. A hand movement determination unit 42 that determines whether the user has moved inward or from the inside to the outside, a hand image group data acquisition unit 43 that acquires hand image group data when the hand moves in and out of the shopping basket, and the inside When the product is held by the moved hand, the product is registered in the shopping basket data storage unit 33. When the product is held by the hand moved to the outside, the purchased product management unit deletes the registered product. 46. [Selection] Figure 1

Description

  The present invention relates to a merchandise management server, an automatic cash register system, a merchandise management program, and a merchandise management method for automatically managing and paying off merchandise that a purchaser puts in and out of a shopping cart when shopping at a store. is there.

  In recent years, an automatic cash register system has been developed that automatically settles a product stored in a shopping basket. For example, JP-A-2006-57813 discloses a merchandise management system having a server for managing merchandise information of merchandise and a shopping cart in which merchandise is input or taken out, and a communication device provided in the shopping basket includes: An image capturing unit that captures a moving image in the shopping basket, and a wireless communication unit that transmits the moving image captured by the image capturing unit to the server, and the server recognizes an object of a product put in the shopping cart based on the moving image. And an object recognition unit that tracks the position of the product, and an image recognition unit that performs image recognition of the product and identifies the product based on the moving image. A product management system that keeps track of the position of a product until it is identified is disclosed (Patent Document 1).

JP, 2006-57813, A

  However, in Patent Document 1 described above, it is necessary to continue to transmit the moving image in the shopping basket to the server while shopping is being performed. In addition, the server needs to perform object recognition processing and image recognition processing for all received moving images. For this reason, the amount of data communication and the amount of calculation processing are enormous, and there is a problem that a large load is applied to the server, so there is no practicality.

  Moreover, in the said patent document 1, it is necessary to keep track of the position of goods until it is identified after goods are thrown into a shopping basket. For this reason, there is a new problem that products cannot be identified when products are successively added to the shopping basket and the products being tracked are hidden behind other products.

  The present invention has been made to solve such problems, and improves the identification rate of goods to be taken in and out of a shopping cart while suppressing the amount of data communication and the amount of calculation processing. An object of the present invention is to provide a product management server, an automatic cash register system, a product management program, and a product management method that can be managed and settled.

  In order to solve the problem that the product management server according to the present invention improves the identification rate of products that are put in and out of the shopping cart and automatically manages while suppressing the amount of data communication and calculation processing, A product management server for managing products to be taken in and out of a shopping cart, wherein hand position data for acquiring hand position data indicating the position of the hand from hand position detection means provided in the vicinity of the opening of the shopping cart An acquisition unit; a hand action determination unit that determines whether the hand has moved from outside to inside of the shopping cart or from inside to outside based on the hand position data; and A hand image group data acquisition unit for acquiring hand image group data consisting of the hand image group from an imaging means provided in the vicinity of the opening of the shopping cart when moving from outside to inside or from inside to outside; Based on the hand image group data, when the product is held by a hand moved from the outside to the inside of the shopping cart, the product is registered in the shopping cart data storage unit, and from the inside of the shopping cart to the outside A purchase product management unit that deletes the product registered in the shopping basket data storage unit when the product is held by the moved hand.

  Further, as one aspect of the present invention, in order to solve the problem of extracting only the product area corresponding to the product with high accuracy and reducing the misperception rate of the product, based on the optical flow of the hand image group data, You may have the goods area extraction part which extracts the area | region other than the said hand as a goods area corresponded to the said goods among the said vertically moved areas while extracting the vertically moving area which moves to an up-down direction.

  Furthermore, as one aspect of the present invention, in order to solve the problem of accurately specifying an area other than the hand among the up and down movement areas, the product area extraction unit includes the shape of the hand among the up and down movement areas. The product region may be extracted by excluding a region having a shape similar to and / or a region having color information similar to the color information of the hand.

Moreover, as one aspect of the present invention, in order to solve the problem of further improving the product identification rate, the product has a product identification unit that identifies the product based on the product region, The product may be identified using one or both of the following modes (a) and (b).
(A) an identification code mode for identifying the product based on an identification code included in the product area;
(B) A deep learning mode in which the product is identified based on an output value when numerical data related to the product region is input to each of the product discriminants for each product obtained by deep learning.

  Furthermore, as one aspect of the present invention, the problem of automatically and accurately determining whether a shopper's hand has moved from the outside of the shopping cart to the inside or from the inside of the shopping cart has been solved. For this purpose, the hand position detecting means is provided in two upper and lower stages in the vicinity of the opening of the shopping basket, and is composed of an upper detection surface and a lower detection surface for detecting a light transmission state or a light shielding state. When the upper detection surface changes from a light-transmitting state to a light-blocking state and the lower detection surface changes from a light-transmitting state to a light-blocking state, the operation determining unit determines that the hand has moved from the outside to the inside of the shopping cart. If the upper detection surface changes from the light shielding state to the light transmission state after the lower detection surface changes from the light shielding state to the light transmission state, it is determined that the hand has moved from the inside of the shopping cart to the outside. May be.

  Further, as one aspect of the present invention, in order to solve the problem of acquiring the same number of hand image group data from a moving image and extracting a region corresponding to a vertically moving hand with high accuracy, The determination unit determines the distance between the upper detection surface and the lower detection surface, and until the lower detection surface changes from the light passing state to the light shielding state after the upper detection surface changes from the light passing state to the light shielding state. On the basis of the time, the downward movement speed when the hand moves from the outside to the inside of the shopping cart is calculated, and the distance between the upper detection surface and the lower detection surface and the lower detection surface are shielded from light. The upward movement speed when the hand moves from the inside of the shopping basket to the outside is calculated based on the time until the upper detection surface changes from the light shielding state to the light passing state after changing from the state to the light passing state. May be.

  In order to solve the problem that the automatic cash register system according to the present invention improves the identification rate of products that are put in and out of the shopping basket while automatically reducing the amount of data communication and the amount of calculation processing, The merchandise management server according to any one of the aspects described above, a shopping cart provided with the hand position detecting means and the imaging means, and a cash register device that settles the merchandise registered in the shopping cart data storage unit.

  In order to solve the problem that the product management program according to the present invention improves the identification rate of products that are put in and out of the shopping basket and automatically manages while suppressing the amount of data communication and calculation processing, A product management program for managing products to be taken in and out of a shopping basket, wherein hand position data for acquiring hand position data indicating the position of the hand is obtained from hand position detection means provided in the vicinity of the opening of the shopping cart An acquisition unit; a hand action determination unit that determines whether the hand has moved from outside to inside of the shopping cart or from inside to outside based on the hand position data; and Hand image group data for acquiring hand image group data consisting of the hand image group from an imaging means provided near the opening of the shopping basket when moving from outside to inside or from inside to outside Based on the hand image group data, when the product is held by a hand moved from the outside to the inside of the shopping cart, the product is registered in the shopping cart data storage unit, and the shopping cart When the product is held by a hand moved from the inside to the outside, the computer functions as a purchased product management unit that deletes the product registered in the shopping cart data storage unit.

  In order to solve the problem that the product management method according to the present invention improves the identification rate of products that are put in and out of the shopping basket and automatically manages while suppressing the amount of data communication and calculation processing, A merchandise management method for managing merchandise to be taken in and out of a shopping basket, wherein hand position data for obtaining hand position data indicating a hand position is obtained from hand position detection means provided in the vicinity of the opening of the shopping cart. An acquisition step; a hand action determination step for determining whether the hand has moved from outside to inside of the shopping cart or from inside to outside based on the hand position data; and Hand image group data for acquiring hand image group data consisting of the hand image group from an imaging means provided near the opening of the shopping basket when moving from outside to inside or from inside to outside Based on the obtaining step and the hand image group data, if the product is held by a hand moved from the outside to the inside of the shopping cart, the product is registered in the shopping cart data storage unit, and the shopping cart A purchase merchandise management step of deleting the merchandise registered in the shopping cart data storage unit when the merchandise is held by a hand moved from the inside to the outside.

  ADVANTAGE OF THE INVENTION According to this invention, the identification rate of the goods withdrawn / inserted with respect to a shopping cart can be improved, controlling and adjusting automatically, suppressing the data communication amount and the amount of calculation processes.

1 is an overall configuration diagram showing an embodiment of an automatic cash register system including a product management server according to the present invention. It is a hand position detection means of this embodiment, Comprising: (a) Arrangement | positioning of an upper stage detection surface and a lower stage detection surface, (b) It is a figure which shows arrangement | positioning of a beam sensor and a reflecting plate. It is a figure which shows an example of the product data in this embodiment. It is a figure which shows an example of the shopping basket data in this embodiment. In this embodiment, it is a figure which shows the operation | movement from which the (a) hand moves from the inside of a shopping cart to the inside, and the (b) operation | movement to which the hand moves from the inside of a shopping cart to the outside. In this embodiment, it is a figure which shows an example of the optical flow at the time of (a) moving a shopping cart, (b) putting goods into a shopping cart, and (c) putting out goods from a shopping cart. In this embodiment, it is a figure which shows a mode that a goods area | region is extracted from an up-and-down movement area | region. It is a flowchart which shows the merchandise management method performed by the merchandise management server and merchandise management program of this embodiment. In this embodiment, it is a flowchart which shows the process which determines operation | movement of a hand.

  Hereinafter, an embodiment of a product management server, an automatic cash register system, a product management program, and a product management method according to the present invention will be described with reference to the drawings.

  The automatic cash register system 100 according to the present embodiment automatically settles the products put in the shopping cart 10. As shown in FIG. 1, the shopping cart 11 that moves the shopping cart 10 and the shopping cart 10. The merchandise management server 1 manages the merchandise to be taken in and out, and the cash register apparatus 12 which settles the merchandise in the shopping basket 10. Hereinafter, each configuration will be described in detail.

  The cash register device 12 is constituted by a POS (Point Of Sale) cash register, and settles the products put in the shopping basket 10. In the present embodiment, the cash register device 12 has a communication function for receiving the product name / price / quantity, etc. of the product from the product management server 1, a checkout function for summing up and summing up the total amount of purchased products, and the product name and total. It has a display function that displays the amount of money.

  The shopping cart 11 places and moves the shopping cart 10. In the present embodiment, as shown in FIG. 1, the shopping cart 11 includes a communication unit 13 for communicating with the product management server 1 via a wireless LAN or the like, and a hand position provided in the vicinity of the opening of the shopping basket 10. It has detection means 14 and imaging means 15. In the present embodiment, the shopping cart 11 and the shopping cart 10 are configured separately. However, the present invention is not limited to this configuration, and a part of the shopping cart 11 may be the shopping cart 10.

  The hand position detection means 14 detects the position of the hand in the vicinity of the opening of the shopping basket 10. In the present embodiment, as shown in FIG. 2A, the hand position detection means 14 is provided in two upper and lower stages near the opening of the shopping basket 10, and detects the light transmission state or the light shielding state. It consists of a surface 14a and a lower detection surface 14b.

  As shown in FIG. 2B, the upper detection surface 14a and the lower detection surface 14b each emit a beam sensor 141 in which a light emitting element and a light receiving element are integrated, and infrared light output from the light emitting element. It is comprised from the reflecting plate 142 reflected to an element. The beam sensors 141 are arranged in two upper and lower rows along one side of the upper end edge of the shopping cart 11, and a plurality of reflectors are arranged along the upper end edge of the shopping cart 11 at positions facing each beam sensor 141. 142 is provided. In FIG. 2B, only one line is shown for easy viewing.

  As a result, the infrared light is reciprocated in a planar shape so as to cover the opening of the shopping basket 10, and functions as the upper detection surface 14a and the lower detection surface 14b. That is, the upper detection surface 14a and the lower detection surface 14b detect a light-transmitting state if there is nothing in the opening of the shopping basket 10, while detecting an opaque state when an object passes through the opening of the shopping basket 10. To do. And the information of a light transmission state or a light-shielding state is acquired by the hand position data acquisition part 41 mentioned later as hand position data which shows the position of a hand.

  In the present embodiment, the hand position detecting means 14 is composed of the upper detection surface 14a and the lower detection surface 14b made up of the beam sensor 141 and the reflecting plate 142, but any device that can detect the position of the hand. However, the present invention is not limited to this configuration. For example, an input device including two infrared cameras and an infrared LED may be used as the hand position detection unit 14 such as Leap Motion (registered trademark) that operates a computer by a hand gesture.

  The imaging means 15 is for imaging a hand or a product. In the present embodiment, as shown in FIG. 1, the imaging means 15 includes a video camera 15a and a strobe light source 15b, and captures a moving image in the vicinity of the opening of the shopping basket 10 from a substantially horizontal direction. is set up. Each time the hand passes through the opening of the shopping basket 10 up and down, the hand image group data acquisition unit 43 described later cuts out a short moving image and a still image at the moment when the strobe light source 15b is emitted from the moving image. Are extracted and acquired as hand image group data consisting of hand image groups.

  In the present embodiment, the imaging unit 15 includes the video camera 15a and the strobe light source 15b, but is not limited to this configuration. That is, a high-sensitivity camera capable of continuous shooting at a high speed may be used as long as it can acquire hand movements as an image group. In this case, a series of still image groups obtained by continuously shooting the hand at a high speed is acquired as hand image group data.

  The product management server 1 automatically manages the products to be purchased. In the present embodiment, the merchandise management server 1 is configured by a computer such as a server, and as shown in FIG. 1, a communication unit 2 that mainly communicates with the shopping cart 11 and the cash register device 12, and a book The product management program 1a of the embodiment and various data are stored, the storage unit 3 that functions as a working area when the arithmetic processing unit 4 performs various arithmetic processes, and the product management program 1a installed in the storage unit 3 It is comprised from the arithmetic processing means 4 which performs various arithmetic processing by performing. Hereinafter, each component will be described in detail.

  The communication means 2 is composed of a wireless communication module or the like, and is for implementing a communication function in the merchandise management server 1. In the present embodiment, the communication means 2 receives the above-described hand position data and hand image group data from the shopping cart 11 by wireless communication such as wireless LAN and Bluetooth (registered trademark), and various data relating to purchased products. Is transmitted to the cash register apparatus 12.

  The storage means 3 is composed of a hard disk, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, etc., and stores various data and a working area when the arithmetic processing means 4 performs various processes. It functions as. In the present embodiment, the storage unit 3 includes a program storage unit 31, a product data storage unit 32, a shopping basket data storage unit 33, and a product discriminant storage unit 34, as shown in FIG. .

  The program storage unit 31 is installed with a product management program 1a for controlling the product management server 1 of the present embodiment. Then, the arithmetic processing means 4 executes the product management program 1a to cause the computer as the product management server 1 to function as each component described later.

  The usage form of the product management program 1a is not limited to the above configuration. For example, the product management program 1a may be stored in a computer-readable non-transitory recording medium such as a CD-ROM or a USB memory, and directly read from the recording medium and executed. Moreover, you may utilize by a cloud computing system, an ASP (Application Service Provider) system, etc. from an external server.

  The product data storage unit 32 stores product data, which is various data related to all products handled in the store. In the present embodiment, as the product data, as shown in FIG. 3, a product name, a price, and a product reference value are stored in association with a product ID for identifying each product. The product reference value is used when identifying a product in the deep learning mode described later, and is a threshold value that each product should satisfy at least.

  The shopping basket data storage unit 33 stores shopping basket data that is various data relating to the products put in the shopping basket 10. In the present embodiment, as the shopping cart data, as shown in FIG. 4, the product name, price, and quantity are stored for each of the products put in the shopping cart 10.

  The product discriminant storage unit 34 stores a product discriminant used when identifying a product by image processing. In the present embodiment, the product discriminant storage unit 34 stores a product discriminant for each product obtained by deep learning for all products handled in the store. And in deep learning mode mentioned later, it is used when identifying goods.

  Next, the arithmetic processing means 4 is constituted by a CPU (Central Processing Unit) and the like, and by executing the commodity management program 1a installed in the storage means 3, as shown in FIG. Unit 41, hand movement determination unit 42, hand image group data acquisition unit 43, product region extraction unit 44, product identification unit 45, purchased product management unit 46, and purchased product transmission unit 47. It has become. Hereinafter, each component will be described in more detail.

  The hand position data acquisition unit 41 acquires hand position data indicating the position of the hand. In the present embodiment, the hand position data acquisition unit 41 is in a light-transmitting state or a light-blocking state from each of the upper detection surface 14a and the lower detection surface 14b as the hand position detection means 14 provided in the shopping cart 11. These are acquired as hand position data and sequentially provided to the hand movement determination unit 42.

  The hand movement determination unit 42 determines whether the hand has moved from the outside to the inside of the shopping basket 10 or whether the hand has moved from the inside to the outside based on the hand position data. In the present embodiment, the hand movement determination unit 42 constantly monitors each hand position data of the upper detection surface 14a and the lower detection surface 14b acquired by the hand position data acquisition unit 41. When there is a change in the light transmission state or the light shielding state, the hand movement is determined based on the change.

  Specifically, as shown in FIG. 5A, the manual operation determination unit 42 changes the lower detection surface 14b from the light passing state to the light blocking state after the upper detection surface 14a has changed from the light passing state to the light blocking state. If it has changed, it is determined that the hand has moved from the outside to the inside of the shopping basket 10. Thereby, the operation | movement when putting goods into the shopping basket 10 and the operation | movement when putting a hand in the shopping basket 10 in order to return goods are detected.

  On the other hand, as shown in FIG. 5B, the manual motion determination unit 42, when the lower detection surface 14b changes from the light shielding state to the light passing state and then the upper detection surface 14a changes from the light shielding state to the light passing state. It is determined that the hand has moved from the inside of the shopping basket 10 to the outside. Thereby, the operation | movement when putting out a hand after putting goods into shopping basket 10 and the operation when taking out goods from shopping basket 10 are detected.

  Moreover, in this embodiment, the hand movement determination unit 42 has a downward movement speed when the hand moves from the outside to the inside of the shopping basket 10 and an upward movement speed when the hand moves from the inside to the outside of the shopping basket 10. And are to be calculated.

Specifically, the hand movement determination unit 42 records the following four times t1, t2, t3, and t4.
Time at which the upper detection surface 14a changes from the light passing state to the light blocking state (t1)
Time at which the lower detection surface 14b changes from the light passing state to the light blocking state (t2)
Time at which the lower detection surface 14b changes from the light shielding state to the light passing state (t3)
Time when the upper detection surface 14a changes from the light shielding state to the light passing state (t4)

Then, the manual motion determination unit 42 determines that the distance S between the upper detection surface 14a and the lower detection surface 14b and the lower detection surface 14b are in the light-transmitting state after the upper detection surface 14a is changed from the light-transmitting state to the light-shielding state. The downward movement speed Vd is calculated using the following equation (1) based on the time (t2−t1) from when the light changes to the light shielding state.
Vd = S / (t2-t1) Formula (1)

Further, the manual motion determination unit 42 determines that the distance S between the upper detection surface 14a and the lower detection surface 14b and the upper detection surface 14a from the light shielding state after the lower detection surface 14b changes from the light shielding state to the light passing state. The upward moving speed Vu is calculated using the following equation (2) based on the time (t4-t3) until the light passing state is changed.
Vu = S / (t4-t3) (2)

  In the present embodiment, the manual motion determination unit 42 is configured such that when the upper detection surface 14a is in a light-transmitting state and the lower detection surface 14b is in a light-shielding state, a product in the shopping basket 10 shields the lower detection surface 14b. The error signal is output because there is a possibility that the hand position data is output or incorrect hand position data is output.

  The hand image group data acquisition unit 43 acquires hand image group data composed of hand image groups. In the present embodiment, the hand image group data acquisition unit 43 acquires a short-time moving image from the imaging unit 15 each time the hand passes through the opening of the shopping basket 10, and acquires hand image group data from the moving image. Extract. Specifically, when the hand moves from the outside of the shopping cart 10 to the inside, a moving image of about several seconds is cut out from the time t2, and when the hand moves from the inside of the shopping cart 10 to the outside, the time t4 A moving image of about several seconds is cut out.

  In the present embodiment, the length of the moving image cut out by the hand image group data acquisition unit 43 is determined according to the moving speed of the hand calculated by the hand motion determination unit 42. That is, when the upward movement speed or the downward movement speed is large, the moving image cut-out time is shortened, and when the upward movement speed or the downward movement speed is small, the moving image cut-out time is lengthened. As a result, since almost the same number of still images are extracted as hand image group data from any moving image, an optical flow described later is appropriately calculated.

  The product area extraction unit 44 extracts a product area corresponding to a product from the hand image group data. In the present embodiment, the product region extraction unit 44 first calculates an optical flow in which the motion of each corresponding point in continuous hand image group data is represented by a vector, as shown in FIG. FIG. 6 is an image diagram in which the optical flow in the vicinity immediately above the opening of the shopping basket 10 is calculated in the horizontal viewing direction, and the direction and size at the position of the arrow indicate the moving direction and moving speed at each point. ing.

  Next, the product region extraction unit 44 extracts a vertically moving region that moves in the vertical direction based on the optical flow of the hand image group data. Specifically, the product region extraction unit 44 refers to the optical flow at each point of the hand image group data, and extracts a region whose main component in the movement direction is upward or downward as a vertical movement region. At this time, the product region extraction unit 44 moves up and down only when the movement direction and movement speed of the optical flow are substantially the same as the direction and speed (upward movement speed or downward movement speed) calculated by the manual motion determination unit 42. Extract as a region. Thereby, the area | region corresponding to the hand which moves up and down is extracted with high precision.

  Subsequently, the product region extraction unit 44 extracts a region other than the hand from the extracted vertical movement region as a product region. Specifically, as shown in FIG. 7, the product region extraction unit 44 has a shape that is similar to the shape of the hand, and has color information that is similar to the color information of the hand, in the vertical movement region. Is identified as a hand region, and the hand region is excluded from the vertical movement region. If an area other than the hand area remains, the area is extracted as a product area.

  As a method for specifying an area having a shape similar to the shape of a hand, for example, as a shape of a hand to be put in and out of the shopping cart 10, several patterns such as a state in which a product is held or an open state are previously set as a reference. Store it as data. Then, the reference data and the vertical movement area are compared, and an area having a similarity higher than a predetermined threshold may be specified as a hand area corresponding to a hand.

  In the present embodiment, the region having a shape similar to the shape of the hand and having color information similar to the color information of the hand is estimated as the hand region corresponding to the hand. However, the present embodiment is limited to this configuration. Instead, a region having a shape similar to the shape of the hand or a region having color information similar to the color information of the hand may be estimated as the hand region. In this case, for example, the certainty factor of the estimation based on the shape and the certainty factor of the estimation based on the color information may be calculated, and the higher certainty factor may be used as the hand region.

The product identification unit 45 identifies a product based on the product area. In the present embodiment, the merchandise item identification unit 45 identifies merchandise while properly using both the following modes (a) and (b).
(A) Identification code mode: Mode for identifying a product based on an identification code included in the product area (b) Deep learning mode: Product for each product discriminant for each product category obtained by deep learning A mode for identifying products based on output values when numerical data related to an area is input

  Specifically, when an identification code such as a barcode is detected from the product area, the product identification unit 45 must identify the product in the identification code mode and detect an identification code such as a barcode from the product area. For example, products are identified in the deep learning mode.

  In the identification code mode, the product identification unit 45 specifies the product ID by reading the identification code from the product area. Then, the product ID is identified by referring to the product data registered in the product data storage unit 32 for the product ID.

  The identification code is not limited to a visible barcode such as a JAN code, and may be any code that can identify a product. For example, a so-called invisible barcode printed with a special ink that reacts to black light (ultraviolet light), a digital watermark barcode, or the like may be used. With such a barcode, a large number of items can be printed over the entire surface of the product, and the detection rate of the identification code is improved.

  On the other hand, in the deep learning mode, the product identification unit 45 identifies products using a multilayer neural network. In this embodiment, the neural network is a multi-stage convolutional neural network including a convolutional layer and a pooling layer and constructed using a library, and image data that is known in advance as to which product is classified is used as teacher data. It was deep learning.

  Specifically, first, a large number of images cut out for each product are prepared, and the number of pixels is unified in all images. Then, normalization is performed so that the average value of the gray value at each pixel is 0 and the standard deviation is 1, and the normalized numerical data group of each pixel in the normalized image is used for learning, and the product discriminant is specified. To do.

In addition, as a linear approximation formula used for a product discriminant, there exists a linear discriminant function used by multivariate analysis as represented, for example by following formula (3).
z ₁ = a ₁₁ x ₁ + a ₁₂ x ₂ + a ₁₃ x ₃ + a ₁₄ x ₄ + b ₁ Formula (3)
However, each symbol represents the following.
z ₁ : Output variable x ₁ , x ₂ , x ₃ , x ₄ : Input variable a ₁₁ , a ₁₂ , a ₁₃ , a ₁₄ : Weight b ₁ : Arbitrary variable

  In the above formula (3), when an image of 2 × 2 pixels is input, since the number of pixels is four, the output value when the gray value of each pixel is used as the input value represents the product of the image The work of correcting each weight so as to be a value is “learning”. For each weight, after using a data set used in general image processing to improve identification accuracy, teacher data and test data are created from actual product images, and learning and test experiments are repeated. Is derived by

  In addition, the neural network has a structure in which many units are connected in a mesh pattern, and each unit has a nonlinear activation function or the like, so that the whole image becomes extremely complicated. However, if one unit is expressed by a simple linear approximation expression, a large number of the above formulas (3) are arranged as a product discriminant to constitute one stage, and the output value from each unit at an arbitrary stage is set to the next stage. Many output values are obtained at the final stage by connecting the input values in a mesh. Then, the product discriminant product with the maximum output value is obtained as the identification result.

  As described above, in the deep learning mode, the product identification unit 45 first adds the numerical data group (the density value of each pixel) related to the product region to each product discriminant for each product stored in the product discriminant storage unit 34. Etc.) and the output value is calculated. Then, the product having the maximum output value is specified as the product included in the product area.

  Further, in the present embodiment, in order to prevent erroneous recognition of a product, a threshold value of an output value that should be satisfied at a minimum for each product is set in the product data storage unit 32 as a product reference value. Thereby, even if the product identification unit 45 is the product having the maximum output value, if the output value is less than the product reference value, the product identification unit 45 determines that the corresponding product does not exist, and performs error processing described later. It is supposed to be executed.

  In addition, as the error processing, for example, an extra moving image in a time zone before and after the moving image cut out by the hand image group data acquisition unit 43 is recorded, and the above-described processing is repeated for the moving image to obtain a product. You may identify. Alternatively, it is possible to warn that identification of a product has failed from a speaker or a display (not shown) attached to the shopping cart 11 and to re-execute the identification process or to manually input product information. Good.

  The purchased product management unit 46 manages products to be purchased. In the present embodiment, the purchased merchandise management unit 46, based on the hand image group data, if the merchandise is gripped by a hand moved from the outside to the inside of the shopping cart 10, the merchandise is put in the shopping cart 10. It determines with a thing, and the said goods are registered into the shopping basket data storage part 33. FIG. On the other hand, based on the hand image group data, the purchased merchandise management unit 46 determines that the merchandise has been taken out of the shopping cart 10 when the merchandise is held by a hand moved from the inside of the shopping cart 10 to the outside. Then, the product registered in the shopping basket data storage unit 33 is deleted.

  Specifically, the purchased product management unit 46 refers to the determination result by the manual motion determination unit 42 when the product is identified by the product identification unit 45, and is identified if the movement direction of the hand is the inward direction. The product name and price of the product are read from the product data storage unit 32 and registered together with the quantity in the shopping basket data storage unit 33. On the other hand, if the moving direction of the hand is the outward direction, the data relating to the identified product is deleted from the shopping basket data storage unit 33. Thereby, the merchandise stored in the shopping basket 10 and the shopping basket data are updated so as to always correspond.

  The purchased merchandise transmission unit 47 transmits data of merchandise to be purchased to the cash register apparatus 12. In the present embodiment, the purchased product transmission unit 47 transmits the product name, price, and quantity to all of the products stored in the shopping basket data storage unit 33 to the cash register device 12 via the communication unit 2. ing.

  Next, operations of the product management server 1, the automatic cash register system 100, the product management program 1a, and the product management method of the present embodiment will be described with reference to FIG.

  When goods are automatically settled using the automatic cash register system 100 of the present embodiment, the goods are taken in and out of the shopping cart 10 placed on the shopping cart 11. At this time, when the hand position data acquisition unit 41 acquires the hand position data from the hand position detection means 14 (step S1), the hand movement determination unit 42 determines the hand movement based on the hand position data (step S1). S2). Hereinafter, the manual motion determination process according to step S2 will be described with reference to FIG.

  First, since the upper detection surface 14a and the lower detection surface 14b are in a light-transmitting state when the hand is away from the shopping basket 10, the manual operation determination unit 42 changes the upper detection surface 14a from the light-transmitting state to the light-shielding state. It is determined whether or not (step S21). As a result of the determination, when the upper detection surface 14a changes from the light passing state to the light blocking state (step S21: YES), the manual motion determination unit 42 records the changed time (t1) (step S22).

  Subsequently, the manual motion determination unit 42 further determines whether or not the lower detection surface 14b has changed from the light passing state to the light blocking state (step S23). As a result of the determination, when the lower detection surface 14b also changes from the light-transmitting state to the light-blocking state (step S23: YES), the manual motion determination unit 42 records the changed time (t2) (step S24), It is determined that the hand has moved from the outside to the inside of the shopping basket 10 (step S25). Thereafter, the hand movement determination unit 42 calculates the downward movement speed of the hand using the above equation (1) (step S26), and then ends this process.

  On the other hand, when the result of determination in step S21 is that the upper detection surface 14a does not change from the light passing state to the light blocking state (step S21: NO), or as a result of determination in step S23, the lower detection surface 14b is moved from the light passing state. If the light shielding state does not change (step S23: NO), the hand is in the shopping basket 10, and the upper detection surface 14a and the lower detection surface 14b may be in the light shielding state.

  Therefore, the manual motion determination unit 42 determines whether or not the lower detection surface 14b has changed from the light shielding state to the light passing state (step S27). As a result of the determination, when the lower detection surface 14b changes from the light shielding state to the light passing state (step S27: YES), the manual motion determination unit 42 records the changed time (t3) (step S28).

  Subsequently, the manual motion determination unit 42 further determines whether or not the upper detection surface 14a has changed from the light shielding state to the light passing state (step S29). As a result of the determination, when the upper detection surface 14a also changes from the light shielding state to the light passing state (step S29: YES), the manual motion determination unit 42 records the changed time (t4) (step S30), It is determined that the hand has moved from the inside of the shopping basket 10 to the outside (step S31). Thereafter, the hand movement determination unit 42 calculates the upward movement speed of the hand using the above equation (2) (step S32), and then ends this process.

  On the other hand, if the result of determination in step S27 is that the lower detection surface 14b does not change from the light shielding state to the light transmission state (step S27: NO), or the result of determination in step S29 is that the upper detection surface 14a passes from the light shielding state. If the light state does not change (step S29: NO), it is determined that there is no hand movement, and this process is terminated. By the manual operation determination process as described above, it is automatically and accurately determined whether the shopper's hand has moved from the outside of the shopping basket 10 to the inside or from the inside of the shopping basket 10 to the outside.

  Returning to FIG. 8, when the hand movement determination unit 42 determines that the hand has moved from the outside of the shopping cart 10 to the inside, or when the hand has moved from the inside of the shopping cart 10 to the outside ( (Step S3: YES), the hand image group data acquisition unit 43 cuts out a moving image from the imaging means 15 and acquires hand image group data (step S4). Thereby, hand image group data when putting a hand in and out of the shopping basket 10 is reliably acquired. In addition, useless image data in which a hand is not reflected is not acquired, so that the data communication amount and the calculation processing amount are suppressed.

  In the present embodiment, the hand image group data acquisition unit 43 cuts out a moving image with a length corresponding to the upward movement speed or the downward movement speed calculated by the manual motion determination unit 42. For this reason, since almost the same number of hand image group data is extracted from any moving image, the optical flow is appropriately calculated.

  On the other hand, if the result of determination in step S3 is that the hand has not moved inward or outward with respect to the shopping basket 10 (step S3: NO), the process returns to step S1 unless a settlement instruction is given (step S12: NO). . Thereby, the movement of the hand with respect to the shopping basket 10 is always monitored.

  Next, based on the optical flow of the hand image group data acquired in step S4, the product area extraction unit 44 extracts a vertical movement area that moves in the vertical direction (step S5). Thereby, the area | region of the hand which moves upwards or downwards with respect to the shopping basket 10 is extracted. In the present embodiment, the product area extraction unit 44 extracts the vertical movement area only when the movement direction and movement speed of the optical flow are substantially the same as the direction and speed calculated by the manual motion determination unit 42. Thereby, the area | region corresponding to the hand which moves up and down is extracted with high precision.

  Subsequently, the product region extraction unit 44 excludes the hand region from the vertical movement region extracted in step S5 and determines whether there is a region other than the hand (step S6). At this time, in this embodiment, since the region having the shape similar to the shape of the hand and having the color information similar to the color information of the hand among the vertically moving regions is specified as the hand region, The area is accurately identified.

  If the result of determination in step S6 is that there is an area other than the hand (step S6: YES), the product area extraction unit 44 extracts the area as a product area (step S7). As a result, only the product area corresponding to the product is extracted with high accuracy, and the misperception rate of the product is reduced. On the other hand, if there is no area other than the hand (step S6: NO), the process returns to step S1 unless there is an instruction for payment (step S12: NO). Thereby, since the product identification process is not executed for the hand image group data not holding the product, the calculation processing amount is reduced.

  Subsequently, the product identification unit 45 identifies a product based on the product region extracted in step S7 (step S8). At this time, in this embodiment, the product identification unit 45 identifies the product in the identification code mode if an identification code is detected from the product region, and the product in the deep learning mode if no identification code is detected from the product region. Therefore, the product identification rate is improved.

  Next, based on the movement direction of the hand determined by the manual motion determination unit 42, the purchased product management unit 46 is identified by the product identification unit 45 if it is a movement from the outside to the inside (step S9: inward direction). The merchandise is registered in the shopping basket data storage unit 33 (step S10). On the other hand, if the movement direction of the hand is from the inside to the outside (step S9: outward direction), the purchased product management unit 46 deletes the product identified by the product identifying unit 45 from the shopping basket data storage unit 33. (Step S11). Thereby, the goods put in and out of the shopping basket 10 are reflected in the shopping basket data in the shopping basket data storage unit 33 in almost real time, and the goods to be purchased are automatically managed.

  On the other hand, if the product cannot be identified as a result of the identification process in step S8 (step S8: NO), various error processes are executed (step S12). Thereby, since accurate shopping basket data is registered about all goods, it becomes possible to settle correctly.

  Thereafter, as long as there is no settlement instruction (step S13: NO), the above-described processing from step S1 to step S12 is repeated. On the other hand, when payment is instructed, the purchased product transmission unit 47 transmits the product name, price, and quantity to all of the products stored in the shopping basket data storage unit 33 to the cash register device 12. As a result, the cash register device 12 can calculate the total amount of all the products without confirming the products put in the shopping basket 10 one by one, and can automatically and accurately settle the products.

According to the product management server 1, the automatic cash register system 100, the product management program 1a, and the product management method of the present embodiment as described above, the following effects can be obtained.
1. While suppressing the amount of data communication and the amount of calculation processing, it is possible to improve and automatically manage the identification rate of goods that are put in and out of the shopping cart 10.
2. Only the product area corresponding to the product can be extracted with high accuracy, and the misperception rate of the product can be reduced.
3. Of the up and down movement areas, areas other than the hand can be accurately identified, and the product area can be extracted with high accuracy.
4). By using a plurality of modes for identifying products, the product identification rate can be further improved.
5. It is possible to automatically and accurately determine whether the shopper's hand has moved from the outside to the inside of the shopping basket 10 or from the inside to the outside.
6). By using the upward moving speed or the downward moving speed of the hand, it is possible to acquire approximately the same number of hand image group data from the moving image and to extract a region corresponding to the vertically moving hand with high accuracy.
7). While suppressing the amount of data communication and the amount of calculation processing, it is possible to improve the identification rate of goods that are put in and out of the shopping basket 10 and to automatically pay out.

  The product management server 1, the automatic cash register system 100, the product management program 1a, and the product management method according to the present invention are not limited to the above-described embodiments, and can be changed as appropriate.

  For example, in the above-described embodiment, the hand position detection unit 14 is configured by the upper detection surface 14a and the lower detection surface 14b including the beam sensor 141 and the reflection plate 142, but as described above, two infrared rays are detected. An input device composed of a camera and an infrared LED may be used as the hand position detecting means 14. In this case, the hand illuminated by the infrared LED is photographed by two infrared cameras, and the three-dimensional position of the hand obtained by image analysis is output as hand position data. Then, by using the time series data of the three-dimensional position, the moving direction and moving speed of the hand are specified.

  In the case of the above configuration, a virtual plane that covers the opening of the shopping basket 10 is set three-dimensionally as a reference time (the above-described times t2 and t4) for cutting out a moving image, and more than the virtual plane. The time when the position of the hand reaches the lower side may be the time t2, and the time when the position of the hand reaches the upper side of the virtual plane may be the time t4.

  Further, in the case of the above configuration, an infrared camera constituting the hand position detecting unit 14 may be used as the imaging unit 15. In this case, the hand image group data acquisition unit 43 acquires infrared image data from the infrared camera as hand image group data.

DESCRIPTION OF SYMBOLS 1 Commodity management server 1a Commodity management program 2 Communication means 3 Storage means 4 Arithmetic processing means 10 Shopping basket 11 Shopping cart 12 Cash register device 13 Communication unit 14 Hand position detection means 14a Upper detection surface 14b Lower detection surface 15a Video camera 15b Strobe light source 15 Imaging means 31 Program storage unit 32 Product data storage unit 33 Shopping basket data storage unit 34 Product discriminant storage unit 41 Hand position data acquisition unit 42 Hand motion determination unit 43 Hand image group data acquisition unit 44 Product region extraction unit 45 Product identification unit 46 Purchased product management unit 47 Purchased product transmission unit 100 Automatic cash register system 141 Beam sensor 142 Reflector

Claims (9)

A product management server for managing products to be taken in and out of a shopping basket,
A hand position data acquisition unit for acquiring hand position data indicating the position of the hand from the hand position detection means provided near the opening of the shopping cart;
Based on the hand position data, a hand action determination unit that determines whether the hand has moved from the outside to the inside of the shopping cart or from the inside to the outside,
When the hand moves from the outside to the inside of the shopping cart or from the inside to the outside, the hand image group data including the image group of the hand is acquired from the imaging means provided near the opening of the shopping cart. A hand image group data acquisition unit;
Based on the hand image group data, when the product is held by a hand moved from the outside to the inside of the shopping cart, the product is registered in the shopping cart data storage unit, and from the inside of the shopping cart to the outside When the product is gripped by the moved hand, the purchased product management unit for deleting the product registered in the shopping basket data storage unit;
A product management server.   Based on the optical flow of the hand image group data, a product area that extracts a vertically moving area that moves in the vertical direction and extracts an area other than the hand as a product area corresponding to the product in the vertically moving area The product management server according to claim 1, further comprising an extraction unit.   The product region extraction unit excludes a region having a shape similar to the shape of the hand and / or a region having color information similar to the color information of the hand from the up and down movement region. The product management server according to claim 2, wherein the product management server is extracted. A product identification unit for identifying the product based on the product region;
The product management server according to claim 2 or 3, wherein the product identification unit identifies a product using one or both of the following modes (a) and (b).
(A) an identification code mode for identifying the product based on an identification code included in the product area;
(B) A deep learning mode in which the product is identified based on an output value when numerical data related to the product region is input to each of the product discriminants for each product obtained by deep learning. The hand position detecting means is provided in two upper and lower stages near the opening of the shopping basket, and is composed of an upper detection surface and a lower detection surface that detect a light transmission state or a light shielding state,
The hand movement determination unit is
After the upper detection surface has changed from a light-transmitting state to a light-blocking state, when the lower detection surface has changed from a light-transmitting state to a light-blocking state, it is determined that the hand has moved from the outside to the inside of the shopping cart,
After the lower detection surface changes from a light shielding state to a light transmission state, when the upper detection surface changes from a light shielding state to a light transmission state, it is determined that the hand has moved from the inside of the shopping cart to the outside.
The merchandise management server according to any one of claims 1 to 4. The hand movement determination unit is
The distance between the upper detection surface and the lower detection surface and the time until the lower detection surface changes from the light passing state to the light shielding state after the upper detection surface changes from the light passing state to the light shielding state. Based on the above, calculate the downward movement speed when the hand moves from the outside to the inside of the shopping cart,
The distance between the upper detection surface and the lower detection surface and the time until the upper detection surface changes from the light shielding state to the light transmission state after the lower detection surface changes from the light shielding state to the light transmission state. Based on the above, the upward movement speed when the hand moves from the inside of the shopping cart to the outside is calculated.
The merchandise management server according to claim 5.   The product management server according to any one of claims 1 to 6, the shopping cart provided with the hand position detection means and the imaging means, and the product registered in the shopping cart data storage unit is settled. An automatic cash register system having a cash register apparatus. A product management program for managing products to be taken in and out of a shopping basket,
A hand position data acquisition unit for acquiring hand position data indicating the position of the hand from the hand position detection means provided near the opening of the shopping cart;
Based on the hand position data, a hand action determination unit that determines whether the hand has moved from the outside to the inside of the shopping cart or from the inside to the outside,
When the hand moves from the outside to the inside of the shopping cart or from the inside to the outside, the hand image group data including the image group of the hand is acquired from the imaging means provided near the opening of the shopping cart. A hand image group data acquisition unit;
Based on the hand image group data, when the product is held by a hand moved from the outside to the inside of the shopping cart, the product is registered in the shopping cart data storage unit, and from the inside of the shopping cart to the outside A product management program that causes a computer to function as a purchased product management unit that deletes the product registered in the shopping cart data storage unit when the product is held by a moved hand. A product management method for managing products to be taken in and out of a shopping basket,
A hand position data acquisition step for acquiring hand position data indicating the position of the hand from the hand position detection means provided in the vicinity of the opening of the shopping cart;
Based on the hand position data, a hand movement determination step for determining whether or not the hand has moved from the outside to the inside of the shopping cart or from the inside to the outside;
When the hand moves from the outside to the inside of the shopping cart or from the inside to the outside, the hand image group data including the image group of the hand is acquired from the imaging means provided near the opening of the shopping cart. A hand image group data acquisition step;
Based on the hand image group data, when the product is held by a hand moved from the outside to the inside of the shopping cart, the product is registered in the shopping cart data storage unit, and from the inside of the shopping cart to the outside A purchase product management step of deleting the product registered in the shopping basket data storage unit when the product is held by a moved hand;
A merchandise management method.

Images