JP2020017253A - Commodity management system and commodity management method - Google Patents

Abstract

To provide a commodity management system and a commodity management method that can easily perform number management of commodities mounted on a commodity shelf.SOLUTION: A commodity management system comprises: an imaging unit 11 that acquires a front face image D1 of commodities mounted on a front face head position of each row of a commodity shelf; a commodity type identification unit 10a that generates commodity type information D3 identifying commodity types and commodity mounting positions of commodities in each row on the basis of the front face image D1; a panel sensor S that acquires a bottom face image D4 of commodities placed on the commodity shelf and mounted on the commodity shelf; a number detection processing unit 20a that detects a number of commodities for each commodity type of each row of the commodity shelf on the basis of the commodity type information D3 and the bottom face image D4; and a commodity management controller that manages the commodity type and the number of commodities of each row of the commodity shelf to output.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a product management system and a product management method that can easily manage the number of products placed on a product shelf.

  In stores such as convenience stores and supermarkets, products are often displayed in showcases in which a plurality of product shelves are arranged in multiple stages. Stores need to manage the replenishable number of commodities placed on the merchandise shelf and replenish commodities appropriately and in a timely manner.

  Patent Literature 1 includes a RFID tag reading unit having a showcase attached to a product, transmits detection information to a store computer, and the store computer stores inventory information received from the showcase in a database. In addition, there is described a device provided with a theft determining means for determining occurrence of theft according to a predetermined rule based on product information taken out of a showcase with reference to stock information stored in a database.

JP 2008-290848 A

  By the way, inventory management of commodities displayed in a store can be performed using POS data. However, the POS data does not manage the actual items displayed on the product shelves in the store. That is, inventory management of the entire store can be performed using POS data, but inventory management of each product shelf cannot be performed using POS data. Therefore, the clerk visually checks the stock status of the products on each product shelf, and the work load on the clerk is large.

  The POS data indicates the number of products in the POS data when the products displayed on the product shelf are discarded without passing through the POS cashier terminal or when the products are displayed on the product shelf without passing through the POS cashier terminal. The number of products on the product shelf may be different.

  In addition, there is a device that captures an image of a product displayed on a product shelf and detects a missing product. However, in this missing product detection, shortage cannot be detected until the displayed product is missing. .

  The present invention has been made in view of the above, and an object of the present invention is to provide a product management system and a product management method capable of easily managing the number of products placed on a product shelf.

  In order to solve the above-described problems and achieve the object, a product management system according to the present invention includes a product on a product shelf in which products of the same product type are placed in a row extending in a depth direction from a front leading position of the product shelf. An article management system that performs quantity management, comprising: an imaging unit that acquires a front image of a product placed at the front front position of each row of the product shelf; and a product of a product in each row based on the front image. A product type identification unit that generates product type information that identifies a type and a product placement position; a panel sensor that is arranged on the product shelf and obtains a bottom surface image of the product placed on the product shelf; A number detection processing unit that detects the number of products for each product type in each row of the product shelf based on the type information and the bottom image, and manages and outputs the product type and the number of products in each row of the product shelf. And a product management unit. .

  Further, in the product management system according to the present invention, in the above invention, the number detection processing unit detects a replenishable number of products that can be replenished for each row based on the product type information and the bottom surface image. The product management unit manages and outputs the refillable number of products.

  Also, in the product management system according to the present invention, in the above invention, the product management unit outputs the refillable number of products to a terminal device including an electronic price tag and a tablet terminal.

  Further, in the product management system according to the present invention, in the above invention, the product type identification unit is configured to store the product of each product in each row based on product characteristic information which is characteristic information of the front image for each product type. The method is characterized by identifying a seed and a product placement position.

  Further, in the product management system according to the present invention, in the above invention, the number detection processing unit performs a template matching process on the bottom surface image using template matching information corresponding to the bottom image for each product type, and It is characterized by detecting the number of products in a row.

  Further, in the product management system according to the present invention, in the above invention, the number detection processing unit performs one or more image processes preset for each product type on the bottom surface image, and the image processing is performed. The number of products is detected by applying a product detection algorithm preset for each product type to the applied bottom image.

  Further, in the product management system according to the present invention, in the above invention, the product detection algorithm is an algorithm using template matching or labeling, and parameters of each algorithm are set in advance for each of the product types. It is characterized by.

  Further, the commodity management system according to the present invention is characterized in that, in the above invention, the commodity detection algorithm using the labeling counts the number of commodities based on the obtained number of pixels of each label.

  In addition, in the product management system according to the present invention, in the above invention, the number detection processing unit performs an image preset for each of the product types on the bottom surface image before performing the one or more image processes. The conversion processing is performed.

  In addition, in the product management system according to the present invention, in the above invention, the number detection processing unit performs a number of times corresponding to a preset number of averaging times for each of the product types before performing the one or more image processes. And obtaining a bottom image averaged by the averaging count as a bottom image to be subjected to image conversion processing.

  In addition, in the product management system according to the present invention, in the above invention, the number detection processing unit is configured to perform, based on a stock quantity of each product type in each row of the product shelf and dimensional information of the product type in a depth direction. An area in which the number of stocks is arranged from the front front side of the product shelf is set as a number detection area, and a process of detecting the number of articles is performed only on the number detection area.

  Further, in the product management system according to the present invention, in the above invention, the number detection processing unit performs no process or a product presence / absence determination process by acquiring only a bottom image for the product placement area outside the number detection area. Is set, and an error notification is performed when it is determined that there is a product by the product presence / absence determination process, or a process of detecting the number of products is performed using an area determined as having a product as the number detection region. It is characterized in that the processing is set and the set processing is executed.

  Further, in the product management system according to the present invention, in the above invention, in the case of replenishing the product, the number detection processing unit performs the process of detecting the number of products using the entire region of each row of the product shelf as the number detection region. Features.

  Further, the product management method according to the present invention is a product management method of a product management system for managing the number of products on a product shelf on which products of the same product type are placed in a row extending in the depth direction from the front front position of the product shelf. A front image obtaining step of obtaining a front image of a product placed at the front front position of each row of the product shelf; and a product type and a product placement position of the product in each row based on the front image. A product type identification step of generating product type information identifying the product, a bottom surface image obtaining step of obtaining a bottom image of a product placed on the product shelf and placed on the product shelf, and the product type information and the product type A number detection processing step of detecting the number of products for each product type in each row of the product shelf based on the bottom surface image; and a product management step of managing and outputting the product type and the number of products in each row of the product shelf. Including And it features.

  In addition, in the product management method according to the present invention, in the above invention, the number detection processing step performs one or more image processing preset for each product type on the bottom surface image, and the image processing is performed. The number of products is detected by applying a product detection algorithm preset for each product type to the applied bottom image.

  According to the present invention, it is possible to easily manage the number of products placed on a product shelf.

FIG. 1 is a block diagram showing a configuration of a product management system according to an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating a configuration of a showcase arranged in a store. FIG. 3 is a diagram illustrating an example of a state where the same product type is placed on each row of the product shelf. FIG. 4 is a diagram illustrating an imaging region of a front image captured by the imaging unit. FIG. 5 is a perspective view showing a configuration of a product shelf on which panel sensors are arranged on the upper surface. FIG. 6 is a diagram illustrating an example of the bottom surface image. FIG. 7 is a diagram illustrating an example of an electronic price tag displaying the number of refillable products according to a display switching instruction. FIG. 8 is a flowchart illustrating a product management process performed by the product management system. FIG. 9 is a block diagram showing a configuration of a panel sensor controller according to the first modification and a storage unit connected to the panel sensor controller. FIG. 10 is a schematic diagram illustrating an example of a showcase in which a product is placed on a product shelf. FIG. 11 is a diagram illustrating an example of product type information for seven product types. FIG. 12 is a diagram illustrating an example of the image conversion rule. FIG. 13 is a diagram illustrating an example of the image processing parameter setting information. FIG. 14 is a diagram illustrating an example of parameter setting information of the number-of-products detection process. FIG. 15 is a diagram conceptually showing the effect of the morphological processing. FIG. 16 is a diagram conceptually showing the relationship between the bottom surface image and the counting result. FIG. 17 is a diagram illustrating an example of the product quantity information. FIG. 18 is a flowchart illustrating the number detection processing procedure according to the first modification. FIG. 19 is a diagram illustrating an image conversion rule according to the second modification. FIG. 20 is an explanatory diagram for specifically explaining the generation of the bottom surface image averaged by the number of times of averaging. FIG. 21 is a diagram showing a product arrangement state on a product shelf to which the third modification is applied. FIG. 22 is a diagram illustrating an example of the parameter setting information on the number-of-products detection process according to the third modification. FIG. 23 is a diagram illustrating an example of product number information according to the third modification. FIG. 24 is a flowchart of a number detection process performed by the number detection processing unit according to the third modification.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

<System configuration>
FIG. 1 is a block diagram illustrating a configuration of a product management system 1 according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing a configuration of the showcase 2 arranged in the store. As shown in FIG. 1, in the product management system 1, a product type identification controller 10, a panel sensor controller 20, a product management controller 30, and an electronic price tag system 40 are connected to a network N1. The PC terminal 50 and the tablet terminal 60 are connected to the network N1 via the network N2.

  The imaging unit 11 and the storage unit 12 are connected to the product type identification controller 10. The imaging unit 11 acquires a front image D1 of the showcase 2. The storage unit 12 stores a front image D1, product characteristic information D2, and product type information D3. The product type identification controller 10 has a product type identification unit 10a.

  The product type identification unit 10a generates product type information D3 that identifies the product type and the product placement position of the product in each row of the product shelf based on the front image D1. At this time, the product type identification unit 10a identifies the product type of the front image D1 with reference to the product characteristic information D2 that is the characteristic information of the front image for each product type. The product type identification controller 10 sends the generated product type information D3 to the panel sensor controller 20. The product characteristic information D2 is information indicating a product packaging state including, for example, the color, pattern, design, and shape of the product.

  As shown in FIG. 2, in the showcase 2, products P are placed on a plurality of product shelves 2 a to 2 f via panel sensors S <b> 1 to S <b> 6, respectively. As shown in FIG. 3, on each of the product shelves 2a to 2f, products P of the same product type are placed in each row R extending from the front front position to the depth direction.

  As shown in FIG. 2, the imaging unit 11 is attached to, for example, a ceiling 101 and acquires front images D1 of the merchandise shelves 2a to 2f. As shown in FIG. 3, when a product P exists in each row R, all the products P appear on the front surface. As illustrated in FIG. 4, the imaging unit 11 acquires an image for a region E on the front surface of the showcase 2 as a front image D1. The area E is an area where the products P on all the product shelves 2a to 2f can be imaged. The front image D1 may be a set of front images divided for each of the product shelves 2a to 2f.

  In FIG. 1, a panel sensor S (S <b> 1 to S <b> 6) and a storage unit 22 arranged on a product shelf are connected to a panel sensor controller 20 via a hub 21. As shown in FIG. 2, the panel sensors S1 to S6 are arranged on the upper surfaces of the product shelves 2a to 2f. For example, as shown in FIG. 5, a panel sensor S1 is arranged on the upper surface of the product shelf 2a.

  The panel sensor S is a capacitance type panel sensor, detects a change in capacitance due to a product P arranged on the upper surface of the panel sensor S in a matrix, and displays a two-dimensional image of the change in capacitance. 6 as a bottom image D4 of the product as shown in FIG. FIG. 6 shows a bottom image D4 when a plastic bottle is arranged, and a circular image corresponding to the shape of the bottom of the plastic bottle is formed in an area where the plastic bottle is placed.

  In FIG. 1, the storage unit 22 stores product type information D3, a bottom surface image D4, template matching information D5, product number information D6, set number information D7, and product refillable number information D8. Further, the panel sensor controller 20 includes a number detection processing unit 20a.

  The number detection processing unit 20a detects the number of products for each product type in each row R of the product shelves 2a to 2f based on the product type information D3 and the bottom surface image D4, and can replenish the products in each row R. Detect the number of items that can be refilled.

  The number detection processing unit 20a performs template matching processing of the bottom surface image D4 using the template matching information D5 for each product type indicated by the product type information D3, and the number of locations determined to match in the template matching process is used as the number of products. Is detected. Further, the number detection processing unit 20a calculates a value obtained by subtracting the detected number of commodities from the maximum number of commodities that can be placed in each row R indicated by the set number information D7 as the number of refillable commodities of each example R. The number of products for each product type in each row R is stored as product number information D6. The refillable number for each product type in each row R is stored as refillable number information D8.

  When the number-of-products information D6 is not set, as shown in FIG. 6, the data size of the region ER where the product P does not exist is represented by the data of one product P indicated by the template matching information D5 used in each column R. The value divided by the size may be calculated as the refillable number. In FIG. 6, the region ER in which the product P does not exist is formed on the depth direction side, but is also formed in the arrangement region of the leading product in each row R and the intermediate region of each row. The number detection processing unit 20a calculates the refillable number of commodities based on the data size of the various regions ER and the data size of one commodity P.

  The panel sensor controller 20 sends the detected product number information D6 and the refillable number information D8 of each row to the product management controller 30.

  The storage unit 31 is connected to the product management controller 30, and stores the product number information D6 and the product refillable number information D8 sent from the panel sensor controller 20. The product management controller 30 manages a product type, a product placement position, a product number, and a product refillable number in each row. For example, the output processing unit 30a sends out the refillable number information D8 to the electronic price tag 40a of the electronic price tag system 40 and holds the information. The product refillable number information D8 includes the product placement position and the product type, and the product management controller 30 sends the product refillable number information D8 to the electronic price tag 40a arranged on the front of the row R.

  As shown in FIG. 7, the electronic price tag 40a normally displays a product type, a product price, and the like, but displays a product type and a refillable number in response to a display switching instruction from the remote controller 70. Thereby, the store clerk can operate the remote controller from the front of the showcase 2 to perform display switching to display the product type and the number of refillable products on the EPL 40a of each row, so that the number of products in each row is reduced. It is not necessary to perform a counting operation, and the product replenishment operation can be performed quickly and easily.

  The product management controller 30 may notify the PC terminal 50 or the tablet terminal 60 in the store of the product number information D6 and the product refillable number information D8 in addition to the electronic price tag 40a. In particular, in the case of a store in which the electronic price tag system 40 is not installed, the clerk does not need to carry out the work of counting the number of products in each row by checking the number of refillable products by carrying the tablet terminal 60. The refilling operation can be performed quickly and easily. It is preferable that an image similar to the front image D1 is displayed on the display screen of the tablet terminal 60, and the product type and the number of refillable products are displayed in association with each row position of each product shelf.

  When the panel sensor controller 20 does not detect a product in the area where the leading product of each row R is arranged, the product management controller 30 informs the product management controller 30 that there is no leading product in the row R for which no product was detected. Upon receiving the notification, the product management controller 30 notifies the tablet terminal 60, for example, of a product advance instruction in the row R in which no product is detected, to the tablet terminal 60. The product management controller 30 notifies the tablet terminal 60 of a product advance instruction when one or more products are detected in the row R in which no product is detected.

<Product management processing>
Next, the product management processing procedure by the product management system 1 will be described with reference to the flowchart shown in FIG. As shown in FIG. 8, first, the product type identification controller 10 acquires the front images D1 of the product shelves 2a to 2f by the imaging unit 11 (step S101). Thereafter, the product type identification unit 10a generates product type information D3 that identifies the product type and the product placement position of the product in each row R based on the front image D1 (Step S102). This product type information D3 is sent to the panel sensor controller 20 side.

  Thereafter, the panel sensor controller 20 acquires the bottom surface image D4 by the panel sensor S (Step S103). The number detection processing unit 20a detects the number of products for each product type in each row R based on the product type information D3 and the bottom surface image D4 (step S104). In the detection of the number of products, a portion where the images match in the template matching process using the template matching information D5 is detected as one product. Further, the number detection processing unit 20a detects the number of refillable products for each product type in each row R (step S105). The panel sensor controller 20 sends the detected number of products and the number of refillable products for each product type in each row R to the product management controller 30.

  Thereafter, the product management controller 30 manages the number of products and the number of refillable products for each product type in each row R, outputs the number of refillable products to the electronic price tag 40a (step S106), and ends the process. Note that the above series of processing is repeatedly performed at predetermined time intervals.

<Modification 1>
The number-of-products detection according to the above-described embodiment is performed by a template matching process. The template matching process is based on the premise that the bottom surface image D4 is stable. Depending on the product, the dielectric constant of the product may be largely different or the shape may be unstable, and the bottom image data having a fixed detection sensitivity may not be obtained, resulting in unstable bottom image data. Further, when the size of the product is small, the bottom surface image data of the obtained product is small, and the resolution is reduced. Therefore, if a uniform product detection algorithm is applied, the number of products may not be detected with high accuracy. Therefore, in the first modification, the number detection processing unit 20a performs one or more image processings set in advance for each product type on the bottom surface image D4, and applies the product processing to the bottom image subjected to the image processing. The number of commodities is detected by applying a commodity detection algorithm preset for each species.

  FIG. 9 is a block diagram illustrating a configuration of the panel sensor controller 20 according to the first modification and the storage unit 22 connected thereto. As shown in FIG. 9, the number detection processing unit 20a of the panel sensor controller 20 includes a preliminary image conversion unit 20b, an image processing unit 20c, a product number detection processing unit 20d, and a product replenishable number calculation unit 20e. The storage unit 22 is connected to the panel sensor controller 20. The storage unit 22 newly has an image conversion rule D11, image processing parameter setting information D12, product number detection processing parameter setting information D13, and product number detection algorithm information D14. The panel sensor controller 20 generates a difference image between the input bottom image D4 and the bottom image when no product is placed, and processes the difference image as the bottom image D4.

  The preliminary image conversion unit 20b converts the bottom image D4 into a bottom image corresponding to the product type using the image conversion rule D11. In the image conversion rule D11, parameters of a conversion coefficient and a use range are set for each product type. Some conversion coefficients have negative values detected depending on the product type, and the detected values are treated as positive values. In addition, the use range is one in which an upper limit and a lower limit of the detection value are set in order to remove noise.

  The image processing unit 20c performs one or more image processes corresponding to the product type on the converted bottom image D4 with reference to the image processing parameter setting information D12. The one or more image processes are one or more of a smoothing process, a binarization process, and a morphology process, and parameters of each image process are set in advance for each product type.

  The product number detection processing unit 20d detects the number of products by applying a product detection algorithm preset for each product type to the image-processed bottom image D4 with reference to the product number detection processing parameter setting information D13. I do. In the product number detection processing parameter setting information D13, a product detection algorithm set for each product type and various parameters for this process are set. The product detection algorithm is stored in the product number detection algorithm information D14, and is an algorithm using template matching or labeling. The template matching information D5 is template image information for each product type used when executing template matching as a product detection algorithm.

<Specific example information content in Modification Example 1>
Next, examples of various types of information such as the image conversion rule D11, the image processing parameter setting information D12, and the product number detection processing parameter setting information D13 will be described.

  First, FIG. 10 is a schematic diagram illustrating an example of a showcase in which a product is placed on a product shelf. FIG. 10 shows a state in which seven product types PA to PG having different product types are placed on the lowest product shelf. FIG. 11 shows an example of the product type information D3 for the seven product types PA to PG. As shown in FIG. 11, the product type information D3 indicates the product placement position using a shelf position, a panel sensor ID, and a display area defined by the left end and the right end of the front surface. 4 shows the relationship with the product types PA to PG placed at each product placement position.

  FIG. 12 is a diagram illustrating an example of the image conversion rule D11. As shown in FIG. 12, in the image conversion rule D11, a parameter of a conversion coefficient and a parameter of a use range are set for each of the product types PA to PG. When “+” of the conversion coefficient is set to “0”, the positive detection value is replaced with 0. When “−” of the conversion coefficient is set to “0”, the negative detection value is set to 0. Replace with When “plus” and “minus” of the conversion coefficient are “1”, the plus and minus detection values are replaced with “1”, respectively. Note that “plus” and “minus” are not limited to “1” and may be set to “2” or “3”. Note that the sign of the value of “minus” is inverted at the time of image conversion according to the image conversion rule D11.

  The use range indicates the lower limit and the upper limit of each product type when template matching is used as the number detection algorithm, and the detected value is replaced with the lower limit if less than the lower limit and replaced with the upper limit if it exceeds the upper limit. You. The use range of the product types PA and PB is set because template matching is applied as a number detection algorithm. On the other hand, for the product types PC to PG, since the labeling is applied as the number detection algorithm, the applicable range is not set.

  FIG. 13 is a diagram illustrating an example of the image processing parameter setting information D12. As shown in FIG. 13, image processing parameter setting information D12 includes image processing of smoothing processing (smoothing), binarization processing (detection threshold), and morphology processing (morphology) for each of the product types PA to PG. Parameters are set.

In the smoothing, the size of the smoothing filter used for the smoothing process is set by the number of pixels.
The smoothing process is performed to reduce noise when the SN ratio is low. When the number of pixels is “1”, the smoothing process is not performed.

  The detection threshold is used when creating a binarized image in the labeling process. If the pixel value is equal to or larger than the detection threshold, it is determined that there is a detection target. If the pixel value is not equal to or larger than the detection threshold, there is no detection target. Is determined.

  The morphology is setting information having items of type, minimum target, and number of times. Note that the morphology processing is performed to remove noise and to compensate for missing parts.

  As the type, any one of the four types of morphological processing “d”, “e”, “c”, and “o” is set. The type “d” is dilation, and if there is a pixel having a detection target in the reference target of the target pixel, the target pixel is set to the detection target. The type “e” is contraction, and if there is a pixel having no detection target in the reference target of the target pixel, the target pixel is set to no detection target. Type “c” is closing and expands and contracts. Type “o” is opening and contracts and expands.

  The reference target sets the number of pixels of the reference target to either “4” or “8”. In the case of "4", the four pixels at the top, bottom, left and right of the target pixel are set as reference targets. In the case of “8”, eight pixels around the target pixel are set as reference targets.

  The number of times specifies the number of repetitions of the processing by an integer. When “0” is set as the number of times, the morphology processing is not performed.

  Note that since the product types PA and PB perform template matching, the detection threshold and morphology parameters are not set.

  FIG. 14 is a diagram illustrating an example of the product number detection processing parameter setting information D13. As shown in FIG. 14, in the product number detection processing parameter setting information D13, parameters of an identification method, a dictionary, a connection target, a score threshold, and a conversion value (α, β) are set for each product type.

  “T” of the identification method indicates template matching, and “L” indicates labeling. The template matching is set to a product type having a stable bottom surface image D4 such as a product in a solid container having a stable shape. On the other hand, the labeling is set to a product type whose bottom image D4 such as a product in an unstable bag is unstable.

  In the dictionary, a dictionary file name to be used in the case of template matching is set. That is, the file name in the template matching information D5 is set.

  The connection target is a parameter of the raster scan performed only in the case of labeling. “4” sets four pixels in the upper, lower, left, and right directions with respect to the target pixel, and “8” sets eight pixels around the target pixel. Set.

  In the case of template matching, the score threshold value is a value obtained by normalizing the allowable range of the difference between the dictionary and the identification target so that it becomes 1 when they completely match. In the case of labeling, the score threshold indicates the lower limit of the number of pixels of the label, and it is determined that the number of pixels is equal to or larger than the threshold and that there is one product.

In the case of labeling, the conversion values (α, β) are obtained by calculating the number of products from the number of pixels of each label obtained by identification using the following equation (1), and the total value of the number of products is calculated as the count of each product type. Results.
Number of goods = ROUND (β + number of pixels / α, 0) (1)
Note that ROUND (X, 0) is a function for rounding X to the nearest integer.

  Here, FIG. 15 is a diagram conceptually showing the effect of the morphological processing. Note that in FIG. 15, black portions are pixels having a detection target. In FIG. 15A, the missing portion at the center of the image is complemented by closing, and the number of pixels with the detection target is the same as the number of pixels with the detection target when there is no missing at the center of the image. Due to this closing, the number of products can be counted with high accuracy even if the bottom image D4 is missing.

  FIG. 15B shows the opening, whereby the noise on the upper side of the image is removed. With this opening, the number of products can be counted accurately even if there is noise in the bottom image D4.

  FIG. 16 is a diagram conceptually showing a relationship between the bottom surface image D4 and the counting result. Note that, in FIG. 16, the black portion is a pixel having a detection target. FIG. 16A is a bottom view image of a product having a large product upper portion. The number of pixels having a detection target is two, and the number of products is counted as “2”. On the other hand, FIG. 16B is a bottom surface image of a prismatic product, in which the number of pixels to be detected is one lump. By using Expression (1), the number of products is set to “2”. Counted.

  FIG. 17 is a diagram illustrating an example of the product quantity information D6. As shown in FIG. 17, the product number information D6 describes a shelf position, which is a product placement position, and a detection result (count value) of the product number for each product type.

<Number detection processing according to Modification Example 1>
Next, a procedure of a number detection process according to the first modification will be described with reference to the flowchart shown in FIG. As shown in FIG. 18, first, the preliminary image conversion unit 20b converts the bottom image with reference to the image conversion rule D11 corresponding to the product type (step S201). Thereafter, the image processing unit 20c performs image processing on the bottom image D4 converted by the preliminary image conversion unit 20b using the image processing parameter setting information D12 according to the product type (Step S202).

  Thereafter, the product number detection processing unit 20d refers to the product number detection processing parameter setting information D13 and executes a product number detection algorithm according to the product type on the bottom surface image D4 image-processed by the image processing unit 20c. To detect the number of products (step S203). Thereafter, the product refillable number calculation unit 20e calculates the product refillable number (step S204), and ends this processing. Note that the above process is repeatedly executed at predetermined time intervals.

<Modification 2>
Next, Modification 2 will be described. In the second modification, before performing one or more image processes, the number detection processing unit 20a acquires the number of bottom images corresponding to the number of averagings set in advance for each product type and averages the number of bottom images by the number of averaging. The converted bottom image is converted into a bottom image to be subjected to image conversion processing.

  FIG. 19 is a diagram illustrating an image conversion rule D11 ′ according to the second modification. As shown in FIG. 19, in the image conversion rule D11 ′, the number of times of averaging is added to the image conversion rule D11.

  The prior image conversion unit 20b converts the bottom image D4 into a bottom image corresponding to the product type using the image conversion rule D11. In the second modification, the image conversion rule D11 ′ preliminarily converts each product type. Set the number of times of averaging. Then, the preliminary image conversion unit 20b acquires the number of bottom images D4 corresponding to the number of times of averaging, and converts the bottom image D4 averaged by the number of times of averaging.

<Specific Example of Modification 2>
FIG. 20 is an explanatory diagram specifically illustrating generation of the bottom surface image D averaged by the number of times of averaging. FIG. 20 shows a case where the number of times of averaging is two. As shown in FIG. 20, at the first time, a bottom image D31 in which the noise image DN1 is superimposed on the bottom image D20 when there is no noise is obtained. Further, in the second time, a bottom image D32 in which the noise image DN2 is superimposed on the bottom image D20 in the absence of noise is obtained. The preliminary image conversion unit 20b adds the first bottom image D31 and the second bottom image D32, and generates an averaged bottom image D33 obtained by dividing by two. The bottom image D33 is the bottom image D from which noise has been removed.

  When the value of the bottom surface image D is extremely small, the number of commodities cannot be counted because it is buried in noise. However, in this modified example 2, even in such a case, the bottom surface images for the number of times of averaging are averaged. Thus, the bottom image D from which noise has been removed is obtained, and the number of products can be counted.

<Modification 3>
Next, a third modification will be described. In the third modification, the number detection processing unit 20b allocates the stock quantity from the front side of the product shelf based on the stock quantity for each product type in each row of the product shelf and the depth dimension information of the product type. The set area is set as the number detection area, and the process of detecting the number of products is performed only on the number detection area.

  FIG. 21 is a diagram illustrating a product arrangement state on the product shelf 2a to which the third modification is applied. As shown in FIG. 21, the front of the product shelf 2 a is inclined vertically downward, and the product P is packed in the front direction, that is, the regulation plate 3 side by the weight of the product P. Instead of the weight of the product P, the product shelf 2a may be horizontal and a pressing mechanism for pressing the product P in the front direction may be provided.

  In such a product shelf 2a, since the products P are packed in the front direction, the product detection region ES1 where the products are actually arranged is smaller than the entire region ES of the row where the products are arranged. Therefore, by performing the process of detecting the number of products only on the product detection region ES1, the process of detecting the number of products can be performed quickly.

  FIG. 22 is a diagram illustrating an example of the product number detection processing parameter setting information D13 ′ according to the third modification. FIG. 24 is a diagram illustrating an example of the product quantity information D16 ′ according to the third modification. As shown in FIG. 23, the item number detection processing parameter setting information D13 'is obtained by adding dimension information and processing category items to the item number detection processing parameter setting information D13. The dimension information sets the length DT in the depth direction for each product type arranged in the row of the product shelf 2a. In the processing category, for the product placement area ES2 outside the number detection area ES1, either no processing or a product presence / absence determination process by acquiring only the bottom image is set, and further, the product presence / absence determination process determines that there is a product. It is set whether an error notification is made when it is determined, or whether a process of detecting the number of products is performed using an area determined as having a product as the number detection region.

  The setting category “0” is no processing, and is a setting in which the process of detecting the number of products in the product placement area ES2 is not performed. The setting category “1” is a setting of a process of performing a product presence / absence determination process by acquiring only a bottom surface image that has not been subjected to image processing, and performing an error notification when there is a product. In the setting section “2”, a product presence / absence determination process is performed by acquiring only the bottom surface image that has not been subjected to image processing, and when a product is present, an area is reset to include a product presence area in the product detection area ES1. This is a setting for performing detection processing of the number of products.

  The product detection area ES1 is determined by the dimension DT in the depth direction for each product type from the front side of the regulating plate 3 and the stock quantity of the product. The product detection area ES1 shown in FIG. 21 shows a case where the stock quantity is 2 and the size is DT. The stock quantity is a value before counting of the product quantity information D6 'shown in FIG.

<Number detection processing according to Modification 3>
FIG. 24 is a flowchart illustrating a procedure of a number detection process performed by the number detection processing unit 20b according to the third modification. As shown in FIG. 24, first, the number detection processing unit 20b determines whether or not the product is being refilled (step S300). If the product is to be replenished (step S300, Yes), the process of detecting the number of products in the entire area ES is performed (step S301), and the process ends. On the other hand, when the product is not being replenished (step S300, No), it is further determined whether the processing type of the product type is “0”, “1”, or “2” (step S302).

  If the processing category is "0" (step S302, "0"), the number detection area ES1 is calculated based on the dimension information and the number of commodities before counting (step S310). Thereafter, a process of detecting the number of products in the number detection area ES1 is performed (step S311), and the process ends.

  On the other hand, when the processing category is “1” (steps S302 and “1”), the same number of products is detected as in steps S310 and S311 (steps S320 and S321). Thereafter, it is determined whether or not there is a product in the product placement area ES2 based on the bottom surface image of the product placement area ES2 outside the number detection area ES1 (step S322). The bottom image is an image on which image processing or the like has not been performed. Further, whether or not there is a product is determined based on whether or not an image which does not exist in the image where no product is placed exists on the image.

  If there is a product in the product placement area ES2 (step S322, Yes), an error notification is performed (step S323), and the process ends. On the other hand, if there is no product in the product placement area ES2 (step S322, No), the process ends.

  If the processing category is "2" (steps S302, "2"), the same number of products is detected as in steps S310, S311 (steps S330, S331). After that, similarly to the processing section “1”, it is determined whether or not there is a product in the product placement area ES2 based on the bottom surface image of the product placement area ES2 outside the number detection area ES1 (step S332).

  If there is a product in the product placement area ES2 (step S332, Yes), the number detection area ES1 is reset by adding the area where the product is present to the product detection area ES1 (step S333). Note that the resetting of the number detection area ES1 may be performed on the entire remaining product placement area ES2. Thereafter, a process of detecting the number of products in the reset number detection region ES1 is performed, and the process ends. On the other hand, when there is no product in the product placement area ES2 (step S332, No), the present process ends. The above process is repeated every predetermined time.

  In the third modification, the detection processing load of the number of commodities is a necessary minimum, and inventory management for taking out commodities from the commodities shelf 2a can be performed quickly. Further, even if the goods on the goods shelf 2a are not moved smoothly, the goods detection area ES1 is reset, so that the work load for the occurrence of an error can be reduced.

  In addition, each configuration illustrated in the above-described embodiments and modified examples is a schematic function, and does not necessarily need to be physically configured as illustrated. In other words, the form of distribution / integration of each device and component is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed / integrated in arbitrary units according to various usage situations. Can be configured.

REFERENCE SIGNS LIST 1 product management system 2 showcase 2 a product shelf 3 regulation plate 10 product type identification controller 10 a product type identification unit 11 imaging unit 12, 22, 31 storage unit 20 panel sensor controller 20 a number detection processing unit 20 b pre-image conversion unit 20 c image processing Unit 20d commodity number detection processing unit 20e commodity refillable number calculation unit 21 hub 30 commodity management controller 30a output processing unit 40 electronic price tag system 40a electronic price tag 50 PC terminal 60 tablet terminal 70 remote controller 101 ceiling D1 front image D2 product characteristic information D3 product Seed information D4 Bottom image D5 Template matching information D6, D6 'Commodity quantity information D7 Set quantity information D8 Commodity refillable quantity information D11, D11' Image conversion rule D12 Image processing parameter setting information D13, FIG. Detection processing parameter setting information D14 Product number detection algorithm information E, ER area N1, N2 Network P Product PA to PG Product type R column S, S1 to S6 Panel sensor

Claims (15)

A product management system for managing the number of products on a product shelf in which products of the same product type are placed in a row extending in a depth direction from a front leading position of the product shelf,
An imaging unit that acquires a front image of a product placed at the front front position of each row of the product shelf;
A product type identification unit that generates product type information that identifies a product type and a product placement position of a product in each row based on the front image;
A panel sensor that is arranged on the product shelf and acquires a bottom surface image of the product placed on the product shelf,
A number detection processing unit that detects the number of products for each product type in each row of the product shelf based on the product type information and the bottom image;
A product management unit that manages and outputs the product type and the number of products in each row of the product shelf;
A product management system comprising: The number detection processing unit detects a replenishable number of commodities capable of replenishing commodities for each row based on the commodity type information and the bottom image,
The product management system according to claim 1, wherein the product management unit manages and outputs the refillable number of products.   The product management system according to claim 2, wherein the product management unit outputs the refillable number of the products to a terminal device including an electronic price tag and a tablet terminal.   The product type identification unit identifies a product type and a product placement position of a product in each row based on product feature information that is feature information of the front image for each product type. The merchandise management system according to any one of 1 to 3.   The said number detection processing part detects the number of articles | goods of each row | line by performing template matching processing with respect to the said bottom face image using the template matching information corresponding to the bottom face image for every goods type. 4. The product management system according to any one of items 4.   The number detection processing unit performs one or more image processing preset for each product type on the bottom image, and sets the image processing on the bottom image preset for each product type on the bottom image. The product management system according to claim 1, wherein the number of products is detected by applying a product detection algorithm.   7. The product management system according to claim 6, wherein the product detection algorithm is an algorithm using template matching or labeling, and parameters of each algorithm are set in advance for each product type.   The product management system according to claim 7, wherein the product detection algorithm using labeling counts the number of products based on the number of pixels of each obtained label.   9. The apparatus according to claim 5, wherein the number detection processing unit performs an image conversion process preset for each of the product types on the bottom surface image before performing the one or more image processes. Product management system according to any one of the above.   Before performing the one or more image processing, the number detection processing unit acquires the number of the bottom images corresponding to the number of times of averaging set in advance for each product type, and averages the number of the bottom images. The product management system according to claim 9, wherein the bottom image is converted as a bottom image to be subjected to image conversion processing.   The number detection processing unit is arranged from the front side of the product shelf for the number of stocks, based on the number of stocks for each product type in each row of the product shelf and the depth dimension information of the product type. The product management system according to any one of claims 5 to 10, wherein the region is set as a number detection region, and the process of detecting the number of products is performed only on the number detection region.   The number detection processing unit sets either no processing or a commodity presence / absence determination process by acquiring only the bottom surface image for the commodity placement area outside the number detection area, and further, there is a commodity presence / absence by the commodity presence / absence determination processing. If it is determined that an error notification is performed, or whether the area determined as having a product is detected is used as the number detection area to perform a process for detecting the number of products, the respective set processes are executed. The product management system according to claim 11, wherein   13. The product management system according to claim 11, wherein, in the case of product replenishment, the number detection processing unit performs the process of detecting the number of products using all regions of each row of the product shelf as a number detection region. A product management method of a product management system for managing the number of products on a product shelf in which products of the same product type are placed in a row extending in a depth direction from a front leading position of the product shelf,
A front image obtaining step of obtaining a front image of a product placed at a front front position of each row of the product shelf;
A product type identification step of generating product type information identifying a product type and a product placement position of a product in each row based on the front image;
A bottom surface image obtaining step of being arranged on the product shelf and obtaining a bottom image of the product placed on the product shelf,
A number detection processing step of detecting the number of products for each product type in each row of the product shelf based on the product type information and the bottom image;
A product management step of managing and outputting the product type and the number of products in each row of the product shelf;
A product management method comprising:   The number detection processing step performs one or more image processing preset for each product type on the bottom image, and sets the image processing on the bottom image preset for each product type on the bottom image. The product management method according to claim 14, wherein the number of products is detected by applying a product detection algorithm.

Images