JP2018206159A - System, method and program for managing commodities - Google Patents

Abstract

To provide proper management of commodities including analysis of photographed images.SOLUTION: A commodity management system 1 according to one embodiment of the present invention is configured as follows. If a weight detected by a weight sensor 38 has decreased, a management terminal 10 determines a commodity taken out of a commodity area on the basis of an image of the commodity area photographed by a commodity area photographing camera 36 when a human sensor 40 has detected the presence of a hand of a customer. If the weight detected by the weight sensor 38 has increased, the management terminal determines a commodity placed in the commodity area on the basis of an image of the commodity area photographed by the commodity area photographing camera 36 when the human sensor 40 does not detect the presence of the hand of the customer any more.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a system, method, and program for managing products in a product area.

  Conventionally, various methods for managing products placed in a product area such as a product shelf of a store have been proposed. For example, by taking a photographed image by a camera, taking out and arranging products in the product area A method for managing the system has been proposed. For example, the following Patent Document 1 is based on a comparison of a captured image of a user's hand before entering the inventory location where the product is placed, and a captured image of the user's hand after being released from the inventory location, A system for tracking the removal and placement of goods is disclosed. Specifically, in this system, when it is determined that the product is held in the user's hand of one of the images as a result of the comparison of the two images, the removal or arrangement of the product from the stock location is determined. The

JP-T-2006-532932 Publication

  However, the above-described conventional system determines whether or not a product is held in the user's hand by comparing two images. For example, a product having a small size is stored in the user's hand holding the product. As a result, it may be difficult to determine whether or not the product is held. As described above, in the conventional method for determining the take-out and arrangement of the product based on the analysis of the image of the user's hand, the accuracy of the determination may not be sufficient.

  An embodiment of the present invention has an object of appropriately managing a product with analysis of a captured image. Other objects of the embodiments of the present invention will become apparent by referring to the entire specification.

  A system according to an embodiment of the present invention is a system for managing products in a product area, and is placed in the product area, which is configured to capture an image of the product area. A detection unit configured to detect a predetermined physical quantity that increases / decreases in accordance with an increase / decrease of a product, and when the predetermined physical quantity detected by the detection unit decreases, before the predetermined physical quantity decreases The predetermined physical quantity is determined when the predetermined physical quantity is increased while the product taken out from the commercial area is determined based on the first image of the product area photographed by the photographing unit at the first timing. Arranged in the product area based on the second image of the product area photographed by the photographing unit at the second timing after the increase And and a determination unit configured to determine a product.

  A method according to an embodiment of the present invention is a method for managing products in a product area, which is executed by one or more computers, and the one or more computers capture an image of the product area. And a communicable connection with a detection unit configured to detect a predetermined physical quantity that increases or decreases according to the increase or decrease of the product placed in the product area. In the method, when the predetermined physical quantity detected by the detection unit is reduced, the product photographed by the photographing unit at a first timing before the predetermined physical quantity is decreased. Determining a product to be taken out from the product area based on a first image of the area, and the predetermined physical quantity detected by the detection unit is When added, the product arranged in the product area is determined based on a second image of the product area captured by the imaging unit at a second timing after the predetermined physical quantity increases. Steps.

  A program according to an embodiment of the present invention is a program for managing products in a product area, and is communicably connected to a photographing unit configured to photograph an image of the product area. In response to execution on a computer communicatively connected to a detection unit configured to detect a predetermined physical quantity that increases or decreases according to an increase or decrease of the product placed in the product area, the detection is performed on the computer. When the predetermined physical quantity detected by the unit decreases, based on a first image of the product area captured by the imaging unit at a first timing before the predetermined physical quantity decreases The process of determining the product to be taken out from the product area, and the predetermined physical quantity when the predetermined physical quantity detected by the detection unit increases A process of determining commodity arranged in the commodity area based on the second image of the product area to be captured by the imaging unit in the second timing later than increased, thereby to execute.

  Various embodiments of the present invention appropriately manage merchandise with analysis of captured images.

The lineblock diagram showing roughly the composition of merchandise management system 1 concerning one embodiment of the present invention. The figure which shows typically the mode of the store where management of the goods by the goods management system 1 is applied. The figure which illustrates the information managed in customer information database 151. The figure which illustrates the information managed in the merchandise information database. The figure which illustrates the information managed in the merchandise area information database 153. The schematic diagram which looked at four goods shelf SH from the front. The schematic diagram which looked at four goods shelf SH from the right side surface. 4 is a flowchart illustrating processing executed by the management terminal 10. FIG. The figure for demonstrating a mode that a customer takes out the goods currently put on goods shelf SH. The figure for demonstrating the method of determining the goods taken out by the customer. The figure for demonstrating a mode that a customer arranges goods in goods shelf SH. The figure for demonstrating the method to determine the goods arrange | positioned by the customer. The figure which illustrates the picked-up image of the product area at the time of a weight change.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram schematically showing a configuration of a product management system 1 according to an embodiment of the present invention, and FIG. 2 is a schematic view of a store to which product management by the product management system 1 is applied. FIG. As shown in FIG. 1, the product management system 1 includes a management terminal 10, a gate terminal 32, a plurality of in-store shooting cameras 34, a plurality of product area shooting cameras 36, a plurality of weight sensors 38, A plurality of human sensors 40 are provided, and these devices are connected to each other via the communication network 20 so as to communicate with each other. As shown in FIG. 2, the merchandise management system 1 manages taking out and arranging (returning) merchandise placed on each of a plurality of merchandise shelves SH in a store such as a convenience store and taking out the merchandise by the customer. Supports checkout (settlement) of cashier products. Thus, the merchandise management system 1 of this embodiment is a system for realizing a store without a cash register.

  The management terminal 10 is configured as a general computer, and, as shown in FIG. 1, a CPU (computer processor) 11, a main memory 12, a user I / F 13, a communication I / F 14, and a storage (memory). These components are electrically connected via a bus or the like (not shown).

  The CPU 11 reads various programs stored in the storage 15 or the like into the main memory 12 and executes various instructions included in the programs. The main memory 12 is configured by, for example, a DRAM.

  The user I / F 13 is various input / output devices for exchanging information with the user. The communication I / F 14 is implemented as hardware such as a network adapter, various types of communication software, and combinations thereof, and is configured to realize wired or wireless communication via the communication network 20 or the like.

  The storage 15 is configured by, for example, a magnetic disk or a flash memory. The storage 15 stores various programs including an operating system, various data, and the like.

  In the present embodiment, as shown in FIG. 1, the storage 15 includes a customer information database 151 that manages information related to customers, a product information database 152 that manages information related to products sold in the store, and products on which products are placed. And a product area information database 153 for managing information related to the area.

  FIG. 3 illustrates information managed in the customer information database 151. As shown in the figure, the customer information database 151 is associated with a “user ID” that identifies an individual customer, and includes “basic information” including name and payment information (including credit card information, etc.), customer, and the like. Information such as “customer image” that is the image of the customer, “in-store position” that indicates the position of the customer in the store, and “product list information” that is a list of products that the customer has taken out and held from the product area .

  In this embodiment, the customer who uses the store needs to be pre-registered, and the basic information in the customer information database 151 is set with information provided by the customer at the time of pre-registration. In addition, as the customer image, three images of the customer's hand, face, and whole body that are photographed by the in-store photographing camera 34 when the customer enters the store are set. As the in-store position, the position of the customer tracked by the management terminal 10 is set for the customer who has entered the store. Customer tracking by the management terminal 10 is performed on the basis of a photographed image by the in-store photographing camera 34. The in-store position is configured as coordinate information on a plane space corresponding to the floor of the store, for example. As will be described in detail later, the merchandise list information is updated according to the take-out and arrangement of merchandise by the customer determined by the management terminal 10.

  FIG. 4 illustrates information managed in the product information database 152. As shown in the figure, the product information database 152 manages information such as “product name”, “price”, “weight”, and “image feature amount” in association with “product ID” that identifies individual products. . These pieces of information are set in advance by a store manager or the like. The image feature amount is a feature amount in a product image, and is used for recognizing and identifying the product from a photographed image taken by the product area photographing camera 36.

  FIG. 5 illustrates information managed in the product area information database 153. As shown in the figure, the product area information database 153 manages information such as “in-store position” indicating the position of the product area in the store in association with the “product area ID” that identifies an individual product area. The in-store position is configured, for example, as coordinate information on a plane space corresponding to the floor of the store, similarly to the position in the customer's store, and is set in advance by a store manager or the like. In the present embodiment, one product area corresponds to one product shelf on which products are placed.

  The gate terminal 32 has a configuration as a general computer and is provided in a gate GT installed at a store entrance (see FIG. 2). The gate terminal 32 has a reading unit (such as a camera) for reading a QR code (registered trademark) presented via a user terminal (such as a smartphone) of the customer. A user ID for identifying individual customers is embedded in the QR code.

  As shown in FIG. 2, the plurality of in-store shooting cameras 34 are provided on the wall surface and ceiling of the store, and each camera 34 is configured to capture a predetermined position in the store.

  Each of the plurality of product area photographing cameras 36 is configured to photograph one corresponding product area. FIG. 6 is a schematic view of the four product shelves SH arranged in the vertical direction as viewed from the front, and FIG. 7 is a schematic diagram of the four product shelves SH as viewed from the right side. In the present embodiment, each of the three product shelves SH excluding the uppermost product shelf SH among the four product shelves SH forms one product area PA on which products are placed.

  As shown in FIG. 7, the product area photographing camera 36 is provided at the substantially lower center of the three product shelves SH except for the lowest product shelf SH among the four product shelves SH. The product placed on the product shelf SH positioned is photographed from above. That is, in this embodiment, one product area photographing camera 36 is provided for one product area PA (product shelf SH on which products are placed).

  As shown in FIGS. 6 and 7, the plurality of weight sensors 38 includes three product shelves SH (that is, a product area PA forming a product area PA, except for the top product shelf SH among the four product shelves SH). The product shelf SH) is provided on the upper surface of the product shelf SH, and is configured to detect the total weight of the products placed on one corresponding product shelf SH.

  As shown in FIGS. 6 and 7, the plurality of human sensors 40 are lower surfaces of three product shelves SH except for the lowest product shelf SH among the four product shelves SH, and are opposite to the wall W side. It is provided near the end on the side. Each human sensor 40 is configured to detect the presence of a human body (a customer's hand) in a product area PA located below the human sensor 40. Although the human sensor 40 is not limited to this, For example, it is comprised as an infrared sensor.

  Next, the operation of the product management system 1 configured as described above will be described. First, the operation when a customer enters the store will be described. In this embodiment, a customer who wants to enter a store presents his / her QR code to the gate terminal 32 at the gate GT. The gate terminal 32 reads the QR code presented by the customer via the user terminal or the like, extracts the user ID embedded in the QR code, and the extracted user ID is pre-registered in the customer information database 151. Make sure. Note that the QR code is an example, and the user ID is stored in a non-contact IC chip or the like provided in the customer's user terminal and the management terminal 10 reads the user ID through short-range wireless communication. May be.

  Further, in response to the extraction of the customer's user ID (reading the QR code) by the gate terminal 32, the in-store photographing camera 34 photographs the customer's hand, face, and whole body, and these photographed images are stored in the customer's hand. Are registered in the customer information database 151 as customer images. The in-store shooting camera 34 that captures the customer's hand is set up above the gate GT and is set to capture the vicinity of the reading portion of the gate terminal 32, and presents a QR code to the reading portion. The customer's hand can be taken from above.

  When it is confirmed by the gate terminal 32 that the customer is pre-registered in the customer information database 151, the customer becomes a target of tracking processing (object tracking processing) by the management terminal 10. Specifically, the management terminal 10 tracks the position of the customer in the store based on the captured image in the store by the in-store shooting camera 34 and updates the position in the store of the customer information database 151 as needed. In the tracking process, the above-described customer face and whole body images are used.

  The operation when the customer enters the store has been described above. Next, an operation for determining the take-out and arrangement of the product by the customer will be described. FIG. 8 is a flowchart illustrating processing executed by the management terminal 10. The process of FIG. 8 is executed for each product shelf (for each product area) every time the total weight of the products placed on the product shelf changes. Specifically, each time the weight detected by the weight sensor 38 increases or decreases by a predetermined threshold value or more, the management terminal 10 executes the process of FIG. 8 for the corresponding product area.

  When the weight detected by the weight sensor 38 changes, the management terminal 10 first identifies a customer close to the corresponding product area as shown in the figure, and acquires the customer's in-store position (step S100). ). Specifically, for example, a customer whose distance to the product area is within a predetermined range is determined as the in-store position of the product area (managed in the product area information database 153) and each customer's location. Based on the in-store position (managed in the customer information database 151), the in-store positions of the identified one or more customers are acquired. The customer specified here is a customer who is close to the product area when the total weight of the product placed in the product area changes.

  Subsequently, the management terminal 10 identifies the position where the weight has changed based on the photographed image of the product area when the weight has changed (step S110). The case where the weight detected by the weight sensor 38 changes includes the case where the customer lifts the product placed on the product shelf to take out the product and the case where the product is placed (returned) on the product shelf. . First, a case where a customer takes out a product placed on a product shelf will be described.

  FIG. 9 is a diagram for explaining a state in which a customer takes out a product placed on the product shelf SH, and shows changes in a photographed image by the product area photographing camera 36 in order from the top to the bottom. In this example, two first-type products PR1 are placed side by side on the left side of the product shelf SH, and two second-type products PR2 are placed side by side on the right side of the product shelf SH. It is. In addition, in this embodiment, the arrangement | positioning aspect of the goods in goods shelf SH is not limited to this, For example, only one kind of goods is placed in one goods shelf SH in this embodiment. Or three or more kinds of products may be placed, or a plurality of products may be stacked in the vertical direction.

  In FIG. 9, the time t1 corresponding to the first photographed image from the top is the timing when the human sensor 40 detects the presence of the customer's hand in the product shelf SH, and the customer's hand UH enters the product shelf SH. I'm starting. In FIG. 9, time t2 corresponding to the second photographed image from the top is the timing when the weight detected by the weight sensor 38 changes (decreases), and the product placed in front of the left side of the product shelf SH. PR1 is lifted by the customer's hand UH. In FIG. 9, the time t3 corresponding to the third photographed image from the top is the timing when the human sensor 40 no longer detects the presence of the customer's hand in the product shelf SH, and the customer holding the product PR1. The hand UH has left the product shelf SH.

  Returning to the flowchart of FIG. 8, in step S110, the management terminal 10 identifies the position where the weight has changed based on the photographed image of the product area at time t2, which is the timing at which the weight has changed (decreased). Specifically, as shown in FIG. 10, the photographed image at time t2 is analyzed to extract a customer hand region in the image, and the hand region is specified as a change region CA whose weight has changed. In this example, the change area CA is configured as a rectangular area including a part of the customer's hand. Such extraction of the customer's hand region in the image can be realized by applying a known region extraction technique in the technical field of image analysis.

  When the position where the weight has changed is specified, the management terminal 10 next determines that the position where the weight has changed and the presence sensor 40 is present in the customer's hand if the weight has decreased (YES in step S120). The product taken out by the customer is determined based on the photographed image of the product area at the time of detecting (when the sensor is ON) (step S130). Specifically, as shown in FIG. 10, the photographed image of the product area at time t1 is analyzed, and the product located in the specified change area CA is determined. Such product determination is performed using the image feature amount of each product managed in the product information database 152. For example, the management terminal 10 can determine a product located in the change area CA based on a comparison between the feature amount of the portion of the change area CA of the captured image and the image feature amount of each product.

  Here, since the change area CA is an area of the customer's hand when the weight changes, depending on the shape of the product and the manner in which the product is held, the position where the product is placed may be the change area. The case where it deviates from CA is assumed. In this case, for example, a product that is close to the change area CA and is located in a direction based on the arrangement of the customer's hand (for example, fingertip orientation, pose, etc.) when the weight changes is taken out by the customer. It can also be determined as a product.

  Thus, when the customer takes out the product placed on the product shelf SH, the product when the customer's hand at the timing before the weight changes starts to enter the product area (time t1). A product to be taken out (lifted) is determined using a photographed image of the area. Since the photographed image normally includes a product before being taken out clearly, it is relatively easy to determine the product.

  The case where the customer takes out the product placed on the product shelf has been described above. Next, a case where a customer places (returns) a product on the product shelf will be described. FIG. 11 is a diagram for explaining a state in which a customer places a product on the product shelf SH. Similarly to FIG. 9, a change in a photographed image by the product area photographing camera 36 is shown in order from top to bottom. ing.

  In FIG. 11, the time t1 corresponding to the first photographed image from the top is the timing when the human sensor 40 detects the presence of the customer's hand in the product shelf SH, and the customer's hand UH holding the product PR1 is the product. It has begun to invade the shelf SH. In FIG. 11, a time t2 corresponding to the second captured image from the top is a timing at which the weight detected by the weight sensor 38 changes (increases), and the customer's hand UH is placed on the left side of the product shelf SH. The product PR1 is arranged. In FIG. 11, the time t3 corresponding to the third photographed image from the top is the timing when the human sensor 40 no longer detects the presence of the customer's hand in the product shelf SH, and the customer's hand from the product shelf SH. UH has left.

  Returning to the flowchart of FIG. 8, even in the case where the product is arranged, in step S110, the management terminal 10 shoots the product area at time t2, which is the timing when the weight changes (increases), as in the case of taking out the product. Based on the image, the position where the weight has changed is specified. Specifically, as shown in FIG. 12, the photographed image at time t2 is analyzed to extract the customer's hand area in the image, and the hand area is specified as a change area CA whose weight has changed.

  When the position where the weight has changed is specified, the management terminal 10 next waits for the human sensor 40 to no longer detect the presence of the customer's hand if the weight has increased (NO in step S120). (Step S140), and arranged by the customer based on the position where the weight has changed and the captured image of the product area when the human sensor 40 no longer detects the presence of the customer's hand (when the sensor is OFF). The sold product is determined (step S150). Specifically, as shown in FIG. 12, the photographed image of the product area at time t3 is analyzed to determine the product located in the specified change area CA. Such product determination is performed using the image feature amount of each product managed in the product information database 152, as in the case of taking out a product.

  In this way, when the customer places (returns) the product on the product shelf SH, the product area when the customer's hand leaves the product area (time t3) at a later timing than the weight changes. The arranged product is determined using the photographed image. Since the photographed image normally includes the product after placement, it is relatively easy to determine the product.

  When the product taken out from the product shelf or the product placed on the product shelf is determined in this way, the management terminal 10 next takes out or places the product based on the change in weight and the weight of the product. The quantity of the product that has been subjected to is determined (step S160). Specifically, the quantity of the product that has been taken out or placed is calculated by dividing the change in weight by the weight of the determined product (managed in the product information database 152).

  Subsequently, the management terminal 10 includes a customer's hand included in the photographed image of the product area at the time of weight change and a hand photographed at the time of entry of each customer who was close to the product area identified in step S100. The similarity is calculated (step S170). Such calculation of similarity is realized by applying a known region extraction technique in the technical field of image analysis, a similarity determination technique based on comparison of feature amounts, and the like. Note that the processing for calculating the similarity may be a heavy load on the management terminal 10, but in this embodiment, only customers who are close to the product area at the time of weight change are targeted, so the store has entered the store. Compared with the case where the similarity is calculated for all customers, the load on the management terminal 10 can be reduced.

  Then, the management terminal 10 determines a customer who has taken out or placed a product based on the similarity of the customer's hand (step S180). Specifically, the management terminal 10 determines the customer having the highest hand similarity among the customers who are close to the product area at the time of weight change as the customer who has taken out or placed the product.

  Here, instead of or in addition to the similarity of the hand, the product was taken out or arranged based on the arrangement of the customer's hand (arm) included in the photographed image of the product area at the time of weight change. You may make it determine a customer. For example, as shown in FIG. 13, based on the direction DR in which the customer's hand (arm) included in the image at the time of weight change extends and the position of the customer at the time of weight change, The customer who is located within the range of the angle is determined as the customer who has taken out or placed the product.

  When the customer who has taken out or placed the product in this way is determined, the management terminal 10 updates the customer's product list (step S190). Specifically, when the customer picks up the product, the determined quantity of the product is added to the product list, and when the customer places the product, the determined quantity of the product is removed from the product list. The Information on the product list is managed in the customer information database 151 as described above.

  In this way, each time a customer takes out or arranges a product, the product list associated with the customer is updated. Then, when the customer leaves the store, settlement processing for the products included in the product list is performed. The exit from the customer's store is determined by, for example, tracking processing by the management terminal 10 or by the customer executing an input operation corresponding to the exit at the gate terminal 32. The settlement processing for the products included in the product list is executed after the confirmation operation by the customer by presenting information related to the product list to the user terminal of the customer in response to the customer leaving, for example.

  The merchandise management system 1 according to the present embodiment described above has the human sensor 40 when the management terminal 10 (determination unit) decreases the weight (predetermined physical quantity) detected by the weight sensor 38 (detection unit). When the (detecting unit) detects the presence of the customer's hand (first timing before the predetermined physical quantity decreases), the product area shooting camera 36 (shooting unit) captures the product area (first image). When the weight sensor 38 detects the presence of the customer's hand when the weight detected by the weight sensor 38 is increased (predetermined) Are arranged in the product area based on the photographed image (second image) of the product area by the product area photographing camera 36 at the second timing after the physical quantity increases). It was and is configured to determine a product. As described above, the product management system 1 according to the present embodiment is a captured image of the product area before the product is lifted and the weight is reduced, or the product area after the product is arranged and the weight is increased. Since the product taken out or arranged is determined based on the photographed image, for example, even when a small product is hidden in the customer's hand, photographing the product area where the product is exposed The product can be determined based on the image. As a result, a decrease in accuracy for determining a product can be suppressed.

  In the above-described embodiment, the weight sensor 38 is configured to detect the total weight of the products placed on one corresponding product shelf, but in another embodiment of the present invention, the weight sensor 38 Instead of 38, a sensor that outputs a detection result capable of specifying a position where the weight in the product shelf has decreased or increased is applied. For example, a sheet-type sensor that detects a pressure distribution in the product shelf may be applied. . In this case, the management terminal 10 changes the weight based on the detection result by the sensor as described above instead of specifying the position where the weight has changed based on the photographed image of the product area when the weight has changed. The specified position can be specified.

  When a sensor capable of detecting the weight distribution in the product shelf, such as a sheet-type sensor that detects the pressure distribution in the product shelf, is applied, the management terminal 10 determines the weight distribution, the weight of each product (product It is also possible to appropriately recognize the status of the products placed on each product shelf based on the captured image of the product area. In this case, the product area information database 153 manages the status of the product for each product area (for example, the type of product (product ID), the quantity, the position on the product shelf (coordinate information), etc.). The management terminal 10 may update the status of the product as appropriate (for example, every time the weight changes). In this way, even if the accuracy of the product determination based on the captured image of the product area is insufficient, for example, the status of the product placed on the corresponding product shelf and the past purchase record of the customer (It can be managed in the customer information database 151) and the like, it is possible to determine the product that has been taken out or placed. Further, when the management terminal 10 appropriately recognizes the status of the products placed on each product shelf, the product information database 152 or the like manages the number of products in the store, and the inventory number and the management terminal Consistency with the status of the product recognized by 10 may be confirmed as appropriate.

  In the above-described embodiment, the configuration is such that the product taken out or arranged is determined according to the change in the weight of the product placed on the product shelf, but the increase or decrease of the product placed in the product area is determined. Other physical quantities other than the weight that increases or decreases accordingly can also be used. For example, in another embodiment of the present invention, instead of the weight of a product, it is detected by a device that detects the volume of a product placed on a product shelf (for example, a device that measures the volume based on a three-dimensional image). Volume to be applied.

  In the embodiment described above, RFIDs storing product identification information (product ID, etc.) for identifying individual products are attached to each product, and the determination of the product that has been taken out or placed (steps S130 and S150), The RFID may be used. For example, in accordance with the specification of the position where the weight has changed in step S110, an RFID reader provided on the product shelf so as to correspond to the specified position is stored in the RFID of the product existing in the vicinity of the specified position. The product identification information being read may be read, and the product may be specified using the product identification information.

  In the embodiment described above, the merchandise management system 1 is configured as a system for realizing a store without a cash register, but the embodiment of the present invention can be applied to an example other than a store without a cash register, for example, The present invention can also be applied to the management of merchandise at a warehouse inventory location (managing of merchandise and management of merchandise by workers).

  The processes and procedures described in the present specification are implemented by software, hardware, or any combination thereof other than those explicitly described. For example, the processes and procedures described in this specification are realized by mounting logic corresponding to the processes and procedures on a medium such as an integrated circuit, a volatile memory, a nonvolatile memory, and a magnetic disk. The processing and procedure described in this specification can be implemented as a computer program corresponding to the processing / procedure and executed by various computers.

  Even if the processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software, multiple Component and / or multiple modules. Also, the software and hardware elements described herein can be implemented by integrating them into fewer components or by disassembling them into more components.

  In the present specification, when the constituent elements of the invention are described as one or a plurality, or when they are described without being limited to one or a plurality of cases, they should be understood separately in context. The component may be either singular or plural.

1 Product management system 10 Management terminal (determination unit)
15 Storage (storage unit)
20 Communication Network 32 Gate Terminal 34 Camera for Shooting in Store 36 Camera for Product Area Shooting (Shooting Unit)
38 Weight sensor (detector)
40 Human sensor (detector)
SH product shelf PA product area

Claims (13)

A system for managing products in a product area,
An imaging unit configured to capture an image of the product area;
A detection unit configured to detect a predetermined physical quantity that increases or decreases according to an increase or decrease of the product placed in the product area;
When the predetermined physical quantity detected by the detection unit decreases, based on a first image of the product area captured by the imaging unit at a first timing before the predetermined physical quantity decreases. Determining the product to be taken out from the product area, and when the predetermined physical quantity increases, the product area photographed by the photographing unit at the second timing after the predetermined physical quantity increases. A determination unit configured to determine a product arranged in the product area based on a second image,
system. The predetermined physical quantity includes the weight or volume of a product placed in the product area.
The system of claim 1. The determination unit specifies a position where the predetermined physical quantity is decreased or increased in the product area, and is extracted from the product area based on the specified position and the first image or the second image. Configured to determine a product or a product placed in the product area,
The system according to claim 1 or 2. The determination unit determines a position where the predetermined physical quantity is decreased or increased in the product area based on a third image of the product area captured by the imaging unit when the predetermined physical quantity is decreased or increased. Configured to identify,
The system of claim 3. The determination unit is configured to identify a position where the predetermined physical quantity has decreased or increased in the product area based on a position of a user's hand included in the third image.
The system of claim 4. The detection unit is configured to output a detection result capable of specifying a position where the predetermined physical quantity is decreased or increased in the product area,
The determination unit is configured to identify a position where the predetermined physical quantity is reduced or increased in the product area based on the detection result by the detection unit.
The system of claim 3. The determination unit is configured to identify a user who has taken out or arranged a product based on a third image of the product area captured by the imaging unit when the predetermined physical quantity is decreased or increased. ing,
The system according to claim 1. A storage unit that stores at least an image including each hand of the plurality of users;
The determination unit identifies the user who has taken out or placed the product based on a comparison between the user's hand included in the third image and the user's hand included in the image stored in the storage unit. Is configured to
The system of claim 7. The determination unit recognizes the position of each of a plurality of users in a facility including the product area, and selects a product based on the position of each of the plurality of users when the predetermined physical quantity decreases or increases. Configured to identify the user that has been removed or placed;
The system according to claim 7 or 8. The determination unit is configured to identify a user who has taken out or arranged a product based on an arrangement of a user's hand included in the third image.
The system of claim 9. A detector configured to detect entry into the product area of the user's hand and withdrawal from the product area;
The first timing is a timing at which the detection unit detects an entry of a user's hand into the product area,
The second timing is a timing at which the detection unit detects the user's hand leaving the product area.
The system according to claim 1. A method for managing products in a product area, executed by one or more computers,
The one or more computers are communicatively connected to an imaging unit configured to capture an image of the product area, and increase or decrease in accordance with an increase or decrease of the product placed in the product area. It is communicably connected to a detection unit configured to detect a physical quantity,
The method
When the predetermined physical quantity detected by the detection unit decreases, based on a first image of the product area captured by the imaging unit at a first timing before the predetermined physical quantity decreases. Determining a product to be taken out from the product area;
When the predetermined physical quantity detected by the detection unit increases, based on a second image of the product area captured by the imaging unit at a second timing after the predetermined physical quantity increases And determining a product arranged in the product area.
Method. A program for managing products in a product area,
The communication unit is configured to be communicably connected to an imaging unit configured to capture an image of the product area, and configured to detect a predetermined physical quantity that increases or decreases according to an increase or decrease of the product placed in the product area. In response to execution on a computer communicably connected to the detection unit,
When the predetermined physical quantity detected by the detection unit decreases, based on a first image of the product area captured by the imaging unit at a first timing before the predetermined physical quantity decreases. A process for determining a product to be taken out from the product area;
When the predetermined physical quantity detected by the detection unit increases, based on a second image of the product area captured by the imaging unit at a second timing after the predetermined physical quantity increases And determining the product arranged in the product area.
program.

Images