WO2022024364A1 - 商品検知装置、商品検知システム、商品検知方法および記録媒体 - Google Patents

商品検知装置、商品検知システム、商品検知方法および記録媒体 Download PDF

Info

Publication number
WO2022024364A1
WO2022024364A1 PCT/JP2020/029490 JP2020029490W WO2022024364A1 WO 2022024364 A1 WO2022024364 A1 WO 2022024364A1 JP 2020029490 W JP2020029490 W JP 2020029490W WO 2022024364 A1 WO2022024364 A1 WO 2022024364A1
Authority
WO
WIPO (PCT)
Prior art keywords
product
image
model
shelf
shape
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/029490
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
莉奈 富田
裕司 田原
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to JP2022539960A priority Critical patent/JP7464129B2/ja
Priority to PCT/JP2020/029490 priority patent/WO2022024364A1/ja
Priority to US18/018,802 priority patent/US20230306741A1/en
Publication of WO2022024364A1 publication Critical patent/WO2022024364A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/87—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
    • G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders

Definitions

  • This disclosure relates to a product detection device, a product detection system, a product detection method, and a product detection program.
  • a method in which a learned model (hereinafter, also referred to as a model) in which an image of a displayed product is trained is used to detect a shortage of a product displayed on a product shelf or the like and a display disorder. ..
  • Patent Document 1 discloses a technique of capturing an image of the state of a product shelf and superimposing and displaying a color-coded image according to the display state so that a display shortage state can be understood.
  • Patent Document 2 describes a technique for notifying a product shelf to be replenished when a product is low and reordering for inventory storage.
  • Patent Document 1 and Patent Document 2 do not disclose a technique for improving the detection accuracy of product shortages and display disturbances in each store.
  • the shelves used may differ from store to store, or even if the shelves are the same, the display position, the orientation of the product display, and the mode of product display may differ. Therefore, if a model learned in one place is used, erroneous recognition is likely to occur in the detection of products in each store, and the detection accuracy is lowered.
  • One of the purposes of the present disclosure is to solve the above-mentioned problems and to provide a technique for improving the detection accuracy by using a model suitable for the display state of the store.
  • the product detection device in one aspect of the present disclosure is An image acquisition unit that acquires images of shelves displaying products, A determination unit for determining product display information including at least one of the shape of the shelf, the shape of the product, and the body of the display from the image. A selection unit that selects a model to be used for detecting the image based on the determined product display information. Using the selected model, it is provided with a detection unit that detects the display state of the product displayed on the shelf from the image.
  • the product detection system in one aspect of the present disclosure is With the product detection device described above, A camera that captures the image and sends it to the product detection device, A terminal that receives a notification regarding the detection from the product detection device, and To prepare for.
  • the product detection method in one aspect of the present disclosure is Get an image of a shelf displaying products, From the image, the product display information including at least one of the shape of the shelf, the shape of the product, or the body of the display is determined. Based on the determined product display information, select the model to be used for detecting the image, and select the model. Using the selected model, the display state of the product displayed on the shelf is detected from the image.
  • the recording medium for storing the product detection program in one aspect of the present disclosure is Get an image of a shelf displaying products, From the image, the product display information including at least one of the shape of the shelf, the shape of the product, or the body of the display is determined. Based on the determined product display information, select the model to be used for detecting the image, and select the model. Using the selected model, the computer is made to detect the display state of the product displayed on the shelf from the image.
  • the program may be stored on a non-temporary computer-readable recording medium.
  • a plurality of components are formed as one member, one component is formed of a plurality of members, one component is a part of another component, and one of the components. The part may overlap with a part of other components, and so on.
  • the order of description does not limit the order in which the plurality of procedures are executed. Therefore, when implementing the method and computer program of the present disclosure, the order of the plurality of procedures can be changed within a range that does not hinder the contents.
  • the methods of the present disclosure and the plurality of procedures of the computer program are not limited to being executed at different timings. Therefore, another procedure may occur during the execution of one procedure. Part or all of the execution timing of one procedure and the execution timing of another procedure may overlap.
  • the effect of this disclosure is to provide a technology to improve the detection accuracy by using a model suitable for the display state of the store.
  • FIG. 1 It is a figure which conceptually shows the structural example of the product detection system which concerns on 1st Embodiment of this disclosure. It is a figure which shows the internal structure example of the product detection apparatus which concerns on 1st Embodiment of this disclosure. It is a figure which shows the data structure example of image information. It is a figure which shows the data structure example of a shelf information. It is a figure which shows the data structure example of the product information. It is a figure which shows an example of the shelf image in the product shelf. It is a figure which shows an example of the shelf image in the product shelf. It is a figure which shows the internal structure example of a store terminal.
  • acquisition means that the own device obtains data or information stored in another device or recording medium (active acquisition), and is output to the own device from the other device. Includes at least one of the input of data or information (passive acquisition).
  • active acquisition include making a request or inquiry to another device and receiving the reply, and accessing and reading another device or recording medium.
  • passive acquisition may be receiving information to be delivered (or transmitted, push notification, etc.).
  • acquisition may be to select and acquire the received data or information, or to select and receive the delivered data or information.
  • FIG. 1 is a block diagram conceptually showing a configuration example of the product detection system 100 according to the first embodiment of the present disclosure.
  • the product detection system 100 includes a product detection device 1, a store terminal 2, and a camera 3.
  • the camera 3 and the store terminal 2 are connected to the product detection device 1 via a communication network 4 such as the Internet or an intranet.
  • the product detection device 1 may be provided in the store and connected to the camera 3 with a wired cable or the like.
  • Camera 3 is a camera provided in each store for taking pictures of product shelves.
  • the camera 3 may be a camera equipped with a fisheye lens for photographing a wide area.
  • the camera 3 may be a camera provided with a mechanism for moving in the store (for example, a mechanism for moving on a rail installed on the ceiling).
  • a plurality of cameras 3 may exist, and each camera 3 captures a shelf image which is a section of a product shelf.
  • the image of the product shelf taken by the camera is transmitted to the product detection device 1, and the product detection device 1 detects a product shortage or display disorder.
  • the product detection device 1 detects a product shortage or display disorder
  • the product detection device 1 notifies the store terminal 2 of the detection result.
  • the store terminal 2 presents the store clerk with information for correcting the product shortage and the display disorder.
  • the product detection device 1 includes an image acquisition unit 11, an image storage unit 12, a shelf information storage unit 13, a product information storage unit 14, a model storage unit 15, a determination unit 16, a selection unit 17, a detection unit 18, and a notification unit 19. ing.
  • the image acquisition unit 11 acquires a shelf image, which is a section of a product shelf for displaying products, taken by the camera 3.
  • the image shows the product and the background (shelf, etc.).
  • the image acquisition unit 11 stores the acquired image in the image storage unit 12 together with information related to the image (hereinafter, also referred to as image information).
  • the image storage unit 12 stores an image and image information acquired from the image acquisition unit 11.
  • the image information includes, for example, an image ID (Identifier), a shooting date and time, a store ID, a shelf position ID, and a product ID.
  • image ID Identifier
  • shooting date and time a shooting date and time
  • store ID a store ID
  • shelf position ID a product ID
  • the image ID is an identifier for uniquely identifying an image. For example, it may be a serial number in the shooting order.
  • a camera ID for uniquely identifying the camera may be assigned to the image ID. For example, in the case of the 100th image taken by the camera A, "image ID: A-100" is used.
  • the shooting date and time is the date and time when the camera 3 shot the shelf image. This may use the time stamp function provided in the camera 3. By determining the shooting date and time of the image, it is possible to select the shelf image of the latest shooting date and time, or to extract the shelf image shot at a specific date and time or period.
  • the store ID is an identifier that can uniquely identify the store where the image was taken.
  • the shelf position ID is an identifier for specifying the position of the image in the store. For example, suppose that a store A has 10 shelves (shelf numbers 1-10), and the shelves are classified into sections 1-5. In such a case, the store ID and the shelf position ID whose image indicates the image of the section 3 of the shelf number 5 are, for example, "A (store) -5 (shelf) -3 (section)".
  • the product ID is an identifier for identifying the product shown in the image.
  • information may be given in advance as to what product is displayed at the corresponding shelf position, or the product given to the front of the shelf in the image.
  • the tag information (for example, a product code) may be read by the image acquisition unit 11 and automatically input.
  • the image recognition engine may be mounted on the camera 3 or the product detection device 1 to identify the product and the product ID by the image recognition process.
  • a plurality of products may be shown in one image. For example, when canned juice A (product ID: KA) and canned juice B (product ID: KB) appear in a certain image, two product IDs, KA and KB, are given.
  • the shelf information storage unit 13 stores shelf information.
  • the shelf information is a combination of an image of a product shelf acquired in advance from the camera 3 and information about the product shelf. As shown in FIG. 4, for example, the shelf information includes a store ID, a shelf ID, a shelf type, a position ID, a partition presence / absence, a shooting date / time, and a shelf image.
  • the store ID is an identifier for uniquely identifying the store. It may be a store name.
  • the shelf ID is an identifier for uniquely identifying the shelf.
  • the position ID is an identifier for specifying the position of the shelf image in the store. For example, suppose a store has 10 shelves (shelf numbers 1-10), and each shelf is divided into 5 compartments (sections 1-5). In the case of the shelf image of the section 3 of the shelf number 1, the position ID is 1 (shelf number) -3 (position number).
  • the shelf type is information indicating the type of shelf. For example, hot showcases, normal temperature display shelves, refrigerated shelves, etc.
  • the presence or absence of a partition is information indicating whether or not there is a partition mechanism (for example, a partition, a rail, etc.) for partitioning a product, or whether or not there is a partition mechanism (only on a flat surface).
  • a partition mechanism for example, a partition, a rail, etc.
  • "1" is input for the presence or absence of a partition if there is a partition
  • "0" is input if there is no partition.
  • the shooting date and time is the date and time when the camera 3 shot the shelf image.
  • the shooting date and time may be acquired by using the time stamp function of the camera 3.
  • the shelf image is an image of a display shelf.
  • the product information storage unit 14 stores product information in which a certain product image and product image information are associated with each other.
  • the product image information includes, for example, a product name, a product ID, an orientation, and a product image, as shown in FIG.
  • the product name is the name of the product (for example, hashed potatoes).
  • the product ID is an identifier for uniquely identifying the product.
  • the orientation is the arrangement shape of the product (for example, flat placement, vertical placement, diagonal placement) taken from a plurality of angles. There may be many types of arrangement shapes.
  • the model storage unit 15 stores the model learned for each product shelf shape, product shape, and display body shape.
  • the model estimates the number of products from the image and estimates the display disorder from the image.
  • the model includes a first model and a second model.
  • the first model is included in the displayable area where the products on the shelf stand included in the first image taken at the first time can be displayed, and the second image taken at the second time after the first time.
  • This is a model in which the difference (first difference) from the displayable area of the shelf stand has been learned.
  • FIGS. 6 and 7 show shelf images of merchandise PET bottles on merchandise shelves.
  • the shelf image taken at the first time shown in FIG. 6 does not have a displayable area, but the shelf image taken at the second time after a predetermined period (see FIG.
  • the first model detects the area (area, position, etc. in the displayable area of FIG. 7) which is the first difference.
  • the second model calculates the difference (second difference) between the number of stocks of the product PET bottle at the shooting date and time of the shelf image of FIG. 6 and the stock number at the shooting date and time of the shelf image of FIG. 7 (second difference) based on these image information. do. For example, when the inventory quantity in FIG. 6 is 50 and the inventory quantity in FIG. 7 is 45, the second difference (quantity) corresponding to the first difference (region) in the product PET bottle is detected as 5.
  • the model can judge the disorder of the display by the shape of the displayable area of FIG. 7.
  • the displayable area is an area in which the upper side on the back side of the shelf and the lower side on the front side of the shelf are parallel to each other, and the display is not disturbed.
  • the shape of the displayable area is irregularly a circle or an ellipse, or if the border of the displayable area is an irregular curve, the model determines that the display disorder has occurred.
  • the determination unit 16 determines the product display information including at least one of the shape of the shelf, the shape of the product, and the body of the display from the image taken by the camera 3 for detection.
  • an image recognition engine by machine learning pattern recognition model such as a support vector machine is used for judgment.
  • the shape of the shelf is, for example, the type of product shelf and the shape of the product shelf (number of display stages, shape of display stage, etc.).
  • the determination unit 16 determines the shape of the shelf by comparing the image of the shelf in the image with the shelf information (see FIG. 4) stored in the shelf information storage unit 13.
  • the shape of the product is, for example, a shape according to the orientation of the product (flat shape, vertical shape, diagonal shape).
  • the determination unit 16 determines the shape of the product by comparing the image of the product in the image with the product information (see FIG. 5) stored in the product information storage unit 14.
  • the display body is, for example, a body in which products are arranged in a row along a partition on a display stand, or a body in which products are randomly arranged.
  • the body of the display may be judged by the presence or absence of a partition for displaying products.
  • the determination unit 16 may determine the body shape based on whether or not there is a partition in the shelf information (see FIG. 4) of the shelf determined to match the shape of the shelf.
  • the determination unit 16 may determine the appearance of the display using an image recognition engine.
  • the selection unit 17 selects a model to be used for image detection based on the product display information (information including at least one of the shape of the shelf, the shape of the product, or the body of the display) determined by the judgment unit 16. .. For example, assuming that the shape of the product shelf is 3 patterns, the shape of the product is 3 patterns, and the body shape of the display is 2 patterns, 18 models are stored in the model storage unit 15. The selection unit 17 selects a model that matches the determination result of the determination unit 16 from the model storage unit 15. The selection unit 17 notifies the detection unit 18 of the selected model.
  • the detection unit 18 detects the display state (for example, normal state, out-of-stock state, display disorder state) of the products displayed on the shelf from the image using the model selected by the selection unit 17.
  • the first model is in a displayable area in which the goods can be displayed in the first image of the shelf on which the goods are displayed in a certain product and in the second image acquired after the acquisition of the first image.
  • the first difference from the displayable area is detected.
  • the second model calculates the first difference in the product, the number of products shown in the first image, and the second difference, which is the difference between the number of products shown in the second image, and the calculation result is used. , Detects product shortages and product display disruptions.
  • the detection unit 18 uses the model to detect an abnormality in the display state of the product (for example, a product shortage or a display disorder).
  • the detection unit 18 is set with a value (for example, “5” for a product PET bottle) that determines that the product is out of stock as abnormal (requires replenishment of the product).
  • the detection unit 18 detects an abnormality in the display state of the product, the detection unit 18 notifies the notification unit 19 of the detection result. For example, when six product PET bottles are lost (purchased) from the product shelf, the detection result is notified to the notification unit 19.
  • the notification unit 19 When the notification unit 19 receives a notification from the detection unit 18 that an abnormality in the display state of the product (for example, a product shortage or a display disorder) has been detected, the notification unit 19 notifies the store terminal 2 of the result of the detection.
  • an abnormality in the display state of the product for example, a product shortage or a display disorder
  • the store terminal 2 is a terminal used by a store clerk for product management and the like.
  • the store terminal 2 includes, for example, a reading unit 21, a communication unit 22, an output unit 23, an input unit 24, and a control unit 25.
  • the reading unit 21 reads product information (bar code, etc.).
  • the communication unit 22 communicates between the store terminal 2 and an external device (for example, a product detection device 1 and a POS terminal (not shown)).
  • the output unit 23 displays the information read by the reading unit 21 and the information received from the external device (notification unit 19 of the product detection device 1) (for example, the detection result) on the display (not shown).
  • the input unit 24 is a keyboard, touch panel, or the like for a store clerk to input information to the store terminal 2.
  • the control unit 25 is connected to the reading unit 21, the communication unit 22, the output unit 23, and the input unit 24, and controls the operation of these units.
  • step S101 the image acquisition unit 11 acquires a shelf image, which is a section of the product shelf taken by the camera 3, and stores it in the image storage unit 12. Specifically, the image acquisition unit 11 generates image information related to the shelf image, and stores the shelf image and the generated image information in the image storage unit 12.
  • step S102 the determination unit 16 determines the product display information including at least one of the shape of the shelf, the shape of the product, and the body of the display from the shelf image. Specifically, the determination unit 16 acquires a shelf image from the image storage unit 12 and determines the shape of the shelf included in the shelf image, the shape of the product included in the shelf image, and the appearance of the product display. The determination unit 16 transmits the determined information to the selection unit 17.
  • step S103 the selection unit 17 selects a model to be used for detecting the shelf image based on the product display information determined by the determination unit 16. Specifically, the selection unit 17 selects a model to be used for detecting the shelf image from a plurality of models included in the model storage unit 15 based on the product display information determined by the determination unit 16. The selection unit 17 notifies the detection unit 18 of the selected model.
  • step S104 the detection unit 18 detects the display state of the products on the shelf from the shelf image using the model selected by the selection unit 17. Specifically, the detection unit 18 uses the model to detect abnormalities in the display of products included in the shelf image (for example, product shortages and display disturbances).
  • the detection unit 18 transmits a detection result (for example, a product shortage occurrence, a display disorder occurrence) to the notification unit 19, and the process proceeds to step S106. .. If no abnormality is detected (NO in step S105), this process ends.
  • step S106 the notification unit 19 transmits the detection result to the store terminal 2.
  • step S107 the notification unit 19 flags the corresponding shelf image in the image storage unit 12. This is because the shelf image in which the abnormality is detected can be extracted later.
  • the notification unit 19 may add an index to the corresponding shelf image in the image storage unit 12.
  • the flagged shelf image is used as teacher data to retrain (feedback) the model.
  • the detection accuracy can be improved by using a model suitable for the display state of products in a store.
  • the image acquisition unit 11 acquires an image of the shelf on which the product is displayed
  • the determination unit 16 determines the product display information including at least one of the shape of the shelf, the shape of the product, and the body of the display.
  • the selection unit 17 selects a model to be used for image detection based on the determined product display information
  • the detection unit 18 uses the selected model to display the products displayed on the shelves from the image. This is because it detects the state.
  • the shape of the product includes the shape of the product placed in one stage and the shape of the product placed in a plurality of stages, and the stacking state is taken into consideration to detect the abnormality of the product. The method will be explained.
  • FIG. 10 is a block diagram conceptually showing a configuration example of the product detection system 200 according to the second embodiment of the present disclosure.
  • the product detection system 200 includes a product detection device 1a, a store terminal 2, and a camera 3.
  • the product detection device 1a includes an image acquisition unit 11, an image storage unit 12, a shelf information storage unit 13, a product information storage unit 34, a model storage unit 35, a determination unit 36, a selection unit 37, a detection unit 18, and a notification unit 19. ing.
  • the product information storage unit 34 stores the product information.
  • the product information of the second embodiment will be described with reference to FIG.
  • the product information of the second embodiment includes, for example, a product name, a product ID, an orientation, a stacking presence / absence, and a product image.
  • the product name is the name of the product (for example, Frankfurt).
  • the product ID is an identifier for uniquely identifying the product.
  • the orientation is the arrangement shape (for example, diagonal placement) of the product taken from a plurality of angles.
  • the presence or absence of stacking is information for determining the stacking state (the shape of the product placed in one stage in the shape of the product and the shape of the product placed in multiple stages).
  • the presence / absence of stacking is information indicating whether or not the information is stacked and displayed in a plurality of stages, and is represented by, for example, "0" without stacking and "1" with stacking.
  • the product image is an image of the product as shown in FIG.
  • the model storage unit 35 stores the model learned for each product shelf shape, product shape, product stacking state, and display body.
  • the model storage unit 35 includes a first model storage unit 35a and a second model storage unit 35b.
  • the first model storage unit 35a has a displayable area in which the products on the shelf stand included in the first image taken at the first time of a certain product can be displayed, and the first model storage unit 35a takes a picture at the second time after the first time.
  • a model (first model) in which the difference (first difference) from the displayable area of the shelf stand included in the second image is stored is stored.
  • the second model storage unit 35b stores the second model and the conversion table.
  • the second model is a model in which the association between the first difference and the second difference between the number of products shown in the first image and the number of products shown in the second image in a certain product has been learned. Specifically, the second model is based on the above-mentioned first difference in a certain product and the second difference, which is the difference between the number of products shown in the first image and the number of products shown in the second image. It is a model that estimates the displayable area and the number of products. The second model outputs a conversion table as a result of estimating the displayable area and the number of products.
  • the conversion table is a table in which a change in the area of a certain product and a change in the number of products are associated with each other.
  • the conversion table may be updated as the detection accuracy of the second model is improved. By creating and updating the conversion table in this way, the calculation speed of the second model can be increased.
  • the conversion table will be described with reference to FIGS. 12 and 13.
  • the left column shows the area ratio and the right column shows the number.
  • the area ratio is the ratio of the area occupied by the product image in the shelf image.
  • the number is a number indicating the number of products included in the shelf image. For example, in the case of "area ratio 15%, number 1 to 3" in the first row from the top of conversion table 1 (see FIG. 12), the area ratio occupied by the product image in the shelf image is 15%, which is reflected in the shelf image. It is detected (estimated) that the number of products is 1 to 3.
  • the conversion table is updated as the detection accuracy of the second model is improved.
  • the left figure of FIG. 12 shows a shelf image 1 in which product croquettes are vertically placed and a shelf image 2 in which product croquettes are placed horizontally in a product shelf (hot showcase).
  • the right figure of FIG. 12 shows the result of detecting the goods and the number of goods from the shelf image 1 by the first and second models.
  • the conversion table 2 which is the result is shown.
  • the left figure of FIG. 13 shows a shelf image 3 in which the product Frankfurt is laid flat without stacking and a shelf image 4 in which the product Frankfurt is stacked in the product shelf (hot showcase).
  • the right figure of FIG. 13 shows the result of detecting the goods and the number of goods from the shelf image 3 by the first and second models.
  • the conversion table 4 which is the result is shown.
  • the determination unit 36 determines the product display information including at least one of the shape of the shelf, the shape of the product (including the stacked state of the products), or the body of the display from the image taken by the camera 3 for detection.
  • an image recognition engine by machine learning pattern recognition model such as a support vector machine is used for judgment.
  • the determination unit 36 compares the image of the product in the image with the product information (see FIG. 11) stored in the product information storage unit 34, and determines the shape of the product.
  • the selection unit 37 is an image based on the product display information (information including at least one of the shape of the shelf, the shape of the product (including the stacked state of the products), or the body shape of the display) determined by the judgment unit 36. Select the model to be used for detection. For example, assuming that the shape of the product shelf is 3 patterns, the shape of the product is 3 patterns, the stacked state of the products is 2 patterns, and the body of the display is 2 patterns, 36 types of area estimation models and conversion tables are in the model storage unit. It is stored in 35.
  • the selection unit 37 includes a first model selection unit 37a and a second model selection unit 37b.
  • the first model selection unit 37a selects a first model that matches the determination result of the determination unit 36 from the first model storage unit 35a.
  • the second model selection unit 37b selects a second model that matches the determination result of the determination unit 36 from the second model storage unit 35b.
  • the first model selection unit 37a and the second model selection unit 37b notify the detection unit 18 of the selected first model and second model.
  • step S201 the image acquisition unit 11 acquires a shelf image, which is a section of the product shelf taken by the camera 3, and stores it in the image storage unit 12. Specifically, the image acquisition unit 11 generates image information related to the shelf image, and stores the shelf image and the generated image information in the image storage unit 12.
  • step S202 the determination unit 36 determines from the shelf image the product display information including at least one of the shape of the shelf, the shape of the product (including the stacked state of the products), and the body of the display. Specifically, the determination unit 36 acquires a shelf image from the image storage unit 12, the shape of the shelf included in the shelf image, the shape of the product included in the shelf image, the stacked state of the product, and the body of the product display. Judge. The determination unit 36 transmits the determined information to the selection unit 37.
  • the selection unit 37 selects a model to be used for detecting the shelf image based on the product display information determined by the determination unit 36. Specifically, the first model selection unit 37a selects a first model that matches the determination result of the determination unit 36 from the first model storage unit 35a. In step S204, the second model selection unit 37b selects a second model that matches the determination result of the determination unit 36 from the second model storage unit 35b. The first model selection unit 37a and the second model selection unit 37b notify the detection unit 18 of the selected first model and second model.
  • step S205 the detection unit 18 detects the display state of the products on the shelf from the shelf image using the models (first model, second model) selected by the selection unit 37. Specifically, the detection unit 18 uses the model to detect abnormalities in the display of products included in the shelf image (for example, product shortages and display disturbances).
  • the detection unit 18 transmits a detection result (for example, a product shortage occurrence, a display disorder occurrence) to the notification unit 19, and the process proceeds to step S207. .. If no abnormality is detected (NO in step S206), this process ends.
  • step S207 the notification unit 19 transmits the detection result to the store terminal 2.
  • step S208 the notification unit 19 flags the corresponding shelf image in the image storage unit 12. This is because the shelf image in which the abnormality is detected can be extracted later.
  • the notification unit 19 may add an index to the corresponding shelf image in the image storage unit 12.
  • the flagged shelf image is used as teacher data to retrain (feedback) the model.
  • the detection accuracy can be further improved as compared with the first embodiment by using a model suitable for the display state of the store.
  • the image acquisition unit 11 acquires an image of the shelf on which the products are displayed
  • the determination unit 36 acquires at least one of the shape of the shelf, the shape of the products (including the stacked state of the products), or the body of the display.
  • the product display information including the above is determined
  • the selection unit 17 selects a model to be used for image detection based on the determined product display information
  • the detection unit 18 uses the selected model from the image. This is because it detects the display state of the products displayed on the shelves.
  • the determination unit 36 determines the shape of the product based on the information including the stacked state of the products (whether the shape of the products placed in one stage or the shape of the products placed in a plurality of stages). Because.
  • the product detection device 40 is the minimum configuration mode of the first embodiment and the second embodiment.
  • the product detection device 40 includes an image acquisition unit 41, a determination unit 42, a selection unit 43, and a detection unit 44.
  • the image acquisition unit 41 acquires an image of a shelf on which products are displayed.
  • the determination unit 42 determines from the image the product display information including at least one of the shape of the shelf, the shape of the product, and the body of the display.
  • the selection unit 43 selects a model to be used for image detection based on the determined product display information.
  • the detection unit 44 detects the display state of the products displayed on the shelves from the image using the selected model.
  • the detection accuracy can be improved by using a model suitable for the display state of the store.
  • the image acquisition unit 41 acquires an image of the shelf on which the product is displayed, and the determination unit 42 determines the product display information including at least one of the shape of the shelf, the shape of the product, and the body of the display.
  • the selection unit 43 selects a model to be used for image detection based on the determined product display information, and the detection unit 44 uses the selected model to display the products displayed on the shelves from the image. This is because it detects the state of.
  • each component of each device (product detection device 1, 1a, 40, etc.) included in the product detection systems 100, 200 indicates a block of functional units.
  • a part or all of each component of each device is realized by an arbitrary combination of an information processing device (computer) 500 and a program as shown in FIG. 16, for example.
  • the information processing apparatus 500 includes the following configurations.
  • -CPU Central Processing Unit
  • 501 -ROM Read Only Memory
  • RAM Random Access Memory
  • Drive device 507 that reads and writes the recording medium 506.
  • -Communication interface 508 to connect to the communication network 509 -I / O interface 510 for input / output of data -Bus 511 connecting each component
  • the program 504 that realizes the functions of each component of each device is stored in, for example, a storage device 505 or a RAM 503 in advance, and is read by the CPU 501 as needed.
  • the program 504 may be supplied to the CPU 501 via the communication network 509, or may be stored in the recording medium 506 in advance, and the drive device 507 may read the program and supply the program to the CPU 501.
  • each device may be realized by any combination of the information processing device 500 and the program, which are separate for each component.
  • a plurality of components included in each device may be realized by any combination of one information processing device 500 and a program.
  • each component of each device is realized by other general-purpose or dedicated circuits, processors, etc. or a combination thereof. These may be composed of a single chip or may be composed of a plurality of chips connected via a bus.
  • each component of each device may be realized by a combination of the above-mentioned circuit or the like and a program.
  • each component of each device When a part or all of each component of each device is realized by a plurality of information processing devices, circuits, etc., the plurality of information processing devices, circuits, etc. may be centrally arranged or distributed. May be good.
  • the information processing device, the circuit, and the like may be realized as a form in which each is connected via a communication network, such as a client-and-server system and a cloud computing system.
  • the product detection device according to Appendix 1, wherein the selection unit selects the model that matches the product display information from the model storage unit.
  • Appendix 3 The product detection device according to Appendix 1 or Appendix 2, wherein the shape of the product includes the shape of the product taken from a plurality of angles.
  • Appendix 4 The product detection device according to any one of Supplementary note 1 to Supplementary note 3, wherein the shape of the product includes the shape of the product placed in one stage and the shape of the product placed in a plurality of stages.
  • the model is In a certain product, the displayable area in the first image of the shelf on which the product is displayed and the displayable area in the second image acquired after the acquisition of the first image are the first.
  • the product detection device according to Appendix 1 or Appendix 2, which includes the first model in which the difference is learned.
  • the model is A second model in which the association between the first difference in a certain product and the second difference between the number of the products shown in the first image and the number of the products shown in the second image is learned.
  • the product detection device according to Appendix 5, including the above.
  • Appendix 7 The product detection device according to Appendix 1, further comprising a notification unit that notifies an external terminal of the detection result when an abnormality in the display state of the product is detected from the detection unit.
  • Appendix 8 The product detection device according to any one of Supplementary note 1 to Supplementary note 7, and the product detection device.
  • Appendix 9 Get an image of a shelf displaying products, From the image, the product display information including at least one of the shape of the shelf, the shape of the product, or the body of the display is determined. Based on the determined product display information, select the model to be used for detecting the image, and select the model. A product detection method comprising detecting the display state of the product displayed on the shelf from the image using the selected model.
  • the model matching the product display information is selected from the model storage means for storing one or more of the models learned to detect the product from the image corresponding to the product display information.
  • the model is A second model in which the association between the first difference in a certain product and the second difference between the number of the products shown in the first image and the number of the products shown in the second image is learned.
  • the product detection method according to Appendix 13 which comprises.
  • [Appendix 15] The product detection method according to Appendix 9, further comprising notifying an external terminal of the result of the detection when an abnormality in the display state of the product is detected in the detection.
  • [Appendix 16] Get an image of a shelf displaying products, From the image, the product display information including at least one of the shape of the shelf, the shape of the product, or the body of the display is determined. Based on the determined product display information, select the model to be used for detecting the image, and select the model.
  • the model matching the product display information is selected from the model storage means for storing one or more of the models learned to detect the product from the image corresponding to the product display information.
  • Appendix 19 The recording medium according to any one of Supplementary note 16 to Supplementary note 18, wherein the shape of the product includes the shape of the product placed in one stage and the shape of the product placed in a plurality of stages.
  • the model is In a certain product, the displayable area in the first image of the shelf on which the product is displayed and the displayable area in the second image acquired after the acquisition of the first image are the first. 1
  • the model is A second model in which the association between the first difference in a certain product and the second difference between the number of the products shown in the first image and the number of the products shown in the second image is learned.
  • the recording medium according to Appendix 20 including the above.
  • Appendix 22 The recording medium according to Appendix 16, further comprising notifying an external terminal of the result of the detection when an abnormality in the display state of the product is detected in the detection.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Business, Economics & Management (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Strategic Management (AREA)
  • Development Economics (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • General Business, Economics & Management (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
PCT/JP2020/029490 2020-07-31 2020-07-31 商品検知装置、商品検知システム、商品検知方法および記録媒体 Ceased WO2022024364A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2022539960A JP7464129B2 (ja) 2020-07-31 2020-07-31 商品検知装置、商品検知システム、商品検知方法および商品検知プログラム
PCT/JP2020/029490 WO2022024364A1 (ja) 2020-07-31 2020-07-31 商品検知装置、商品検知システム、商品検知方法および記録媒体
US18/018,802 US20230306741A1 (en) 2020-07-31 2020-07-31 Product detection device, product detection method, and recording medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/029490 WO2022024364A1 (ja) 2020-07-31 2020-07-31 商品検知装置、商品検知システム、商品検知方法および記録媒体

Publications (1)

Publication Number Publication Date
WO2022024364A1 true WO2022024364A1 (ja) 2022-02-03

Family

ID=80035284

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/029490 Ceased WO2022024364A1 (ja) 2020-07-31 2020-07-31 商品検知装置、商品検知システム、商品検知方法および記録媒体

Country Status (3)

Country Link
US (1) US20230306741A1 (https=)
JP (1) JP7464129B2 (https=)
WO (1) WO2022024364A1 (https=)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024130857A1 (zh) * 2022-12-20 2024-06-27 苏州万店掌网络科技有限公司 一种物品陈列检测方法、装置、设备及可读存储介质
WO2024201706A1 (ja) * 2023-03-28 2024-10-03 日本電気株式会社 検知装置、検知方法、及び非一時的なコンピュータ可読媒体
WO2024201799A1 (ja) * 2023-03-29 2024-10-03 日本電気株式会社 情報処理装置、情報処理システム、情報処理方法及びプログラムが格納された非一時的なコンピュータ可読媒体
JPWO2024201704A1 (https=) * 2023-03-28 2024-10-03

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019065212A1 (ja) * 2017-09-29 2019-04-04 日本電気株式会社 情報処理装置、情報処理システム、制御方法、及びプログラム
WO2019087519A1 (ja) * 2017-10-30 2019-05-09 パナソニックＩｐマネジメント株式会社 棚監視装置、棚監視方法、及び、棚監視プログラム

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019107157A1 (ja) * 2017-11-29 2019-06-06 株式会社Ｎｔｔドコモ 棚割情報生成装置及び棚割情報生成プログラム
US10949799B2 (en) * 2018-06-29 2021-03-16 Focal Systems, Inc. On-shelf image based out-of-stock detection
WO2020210825A1 (en) * 2019-04-11 2020-10-15 Carnegie Mellon University System and method for detecting products and product labels

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019065212A1 (ja) * 2017-09-29 2019-04-04 日本電気株式会社 情報処理装置、情報処理システム、制御方法、及びプログラム
WO2019087519A1 (ja) * 2017-10-30 2019-05-09 パナソニックＩｐマネジメント株式会社 棚監視装置、棚監視方法、及び、棚監視プログラム

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024130857A1 (zh) * 2022-12-20 2024-06-27 苏州万店掌网络科技有限公司 一种物品陈列检测方法、装置、设备及可读存储介质
WO2024201706A1 (ja) * 2023-03-28 2024-10-03 日本電気株式会社 検知装置、検知方法、及び非一時的なコンピュータ可読媒体
JPWO2024201706A1 (https=) * 2023-03-28 2024-10-03
JPWO2024201704A1 (https=) * 2023-03-28 2024-10-03
WO2024201799A1 (ja) * 2023-03-29 2024-10-03 日本電気株式会社 情報処理装置、情報処理システム、情報処理方法及びプログラムが格納された非一時的なコンピュータ可読媒体

Also Published As

Publication number Publication date
JP7464129B2 (ja) 2024-04-09
US20230306741A1 (en) 2023-09-28
JPWO2022024364A1 (https=) 2022-02-03

Similar Documents

Publication Publication Date Title
US11640576B2 (en) Shelf monitoring device, shelf monitoring method, and shelf monitoring program
US11823126B2 (en) Shelf label detection device, shelf label detection method, and shelf label detection program
JP7416292B2 (ja) 情報処理装置、情報処理システム、制御方法、及びプログラム
CN112074842B (zh) 价签信息显示方法、装置及货架系统
WO2020048492A1 (zh) 商品状态识别
JP7024351B2 (ja) 棚割生成プログラム、棚割生成方法及び棚割生成装置
WO2022024364A1 (ja) 商品検知装置、商品検知システム、商品検知方法および記録媒体
JP2012174154A (ja) 在庫状況管理方法および在庫状況管理装置
US11315074B2 (en) Smart shelf system
KR102319893B1 (ko) 재고 파악 방법 및 시스템
JP7396499B2 (ja) 学習モデル生成装置、学習モデル生成システム、学習モデル生成方法および学習モデル生成プログラム
JP7130945B2 (ja) 在庫検出プログラム、在庫検出方法及び在庫検出装置
JP7459947B2 (ja) 商品検知装置、商品検知システム、商品検知方法および商品検知プログラム
US20230281992A1 (en) Training data generation device, training data generation system, training data generation method, and recording medium
JP7167926B2 (ja) 情報処理装置、情報処理方法、およびプログラム
US20250045659A1 (en) Information processing device and information processing method
JP2021080087A (ja) 情報処理システムと情報処理装置と情報処理プログラムと情報処理方法
JP7779518B2 (ja) 画像処理装置、画像処理方法、及びプログラム
JP7613752B2 (ja) 価格情報決定装置、価格情報決定方法、及びプログラム
US20230063863A1 (en) Item management apparatus, item management method, and recording medium
JP7287015B2 (ja) 商品管理システム及び商品管理方法
US20210150502A1 (en) Cloud based self checkout system
JP7563625B2 (ja) 画像解析システム、画像解析方法およびプログラム
WO2024201715A1 (ja) 棚割りデータ更新装置、棚割りデータ更新方法、及びコンピュータ可読媒体

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20946935

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022539960

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20946935

Country of ref document: EP

Kind code of ref document: A1