CN112446280A

CN112446280A - Sales management system and sales management method

Info

Publication number: CN112446280A
Application number: CN202010656447.XA
Authority: CN
Inventors: 岡村敦
Original assignee: Toshiba TEC Corp
Current assignee: Toshiba TEC Corp
Priority date: 2019-09-05
Filing date: 2020-07-09
Publication date: 2021-03-05
Also published as: US20210073554A1; JP7368982B2; US11887373B2; US11455803B2; US20220398849A1; JP2021039662A; EP3789977A1; JP2023181237A

Abstract

The invention discloses a sales management system and a sales management method, which can correctly specify a customer who takes away a commodity, and the sales management system comprises: an action monitoring device, a commodity monitoring device, a combination recognition device and a sales management device. The action monitoring device detects a purchaser who faces a displayed commodity, and identifies an action of the purchaser to take out the facing commodity and a taking-out direction thereof. The merchandise surveillance device recognizes movement of the merchandise being displayed and a movement direction of the merchandise. The merge recognition device, when recognizing the action of the purchaser to take out the product and the direction of taking out the product, and recognizing the movement of the product and the direction of movement of the product, designates the product whose movement is recognized as the purchased product of the purchaser whose action is recognized when the direction of taking out the product coincides with the direction of movement. The sales management apparatus registers a purchased product of a specified purchaser.

Description

Sales management system and sales management method

The present application claims priority to japanese application having application number JP2019-162003, application number 09/05 in 2019, and the contents of the above application are incorporated herein by reference in their entirety.

Technical Field

The embodiment of the invention relates to a sales management system and a sales management method.

Background

Currently, there are various proposals for sales processing systems in which no store clerk participates and which can sell products displayed in a store to customers.

For example, in a sales management system, it is detected that a customer has taken a commodity displayed in a store from an image or the like captured in the store by a camera. The sales management system registers the commodity that has been detected to have been taken by the customer as a purchased commodity of the customer.

In such a sales management system, for example, when two customers standing in front of the displayed product are holding their hands at substantially the same time toward the product and one of the customers takes the product, it is difficult to determine which customer (which side) has taken the product.

Disclosure of Invention

In view of the above problems, an object of the present invention is to provide a sales management system and a sales management method that can accurately specify a customer who has taken a product.

To solve the above problem, an embodiment of the present invention provides a sales management system, including: the device includes a detection unit, a first recognition unit, a second recognition unit, a designation unit, and a registration unit. The detection unit detects a purchaser who faces the displayed commodity. The first recognition unit recognizes the action of the purchaser to take out the opposing commodity detected by the detection unit and the direction of taking out the opposing commodity. The second recognition unit recognizes the movement of the displayed commodity and the movement direction of the commodity. When the movement of the commodity and the movement direction thereof are recognized by the second recognition unit while the movement of the commodity is recognized by the first recognition unit, the specification unit specifies the commodity, the movement of which is recognized by the second recognition unit, as a commodity to be purchased by the purchaser, the movement of which is recognized by the first recognition unit. The registration unit registers the purchased product of the purchaser specified by the specification unit.

According to the sales management system described above, the customer who has taken the product can be accurately specified based on the taking-out direction and the moving direction.

In the sales management system, the method further includes: a first time acquisition unit that acquires a first time at which the action of taking out the product is recognized by the first recognition unit; and a second time acquisition unit that acquires a second time at which the movement of the commodity is recognized by the second recognition unit, wherein the specification unit specifies the commodity, which is recognized by the second recognition unit as moving, as a purchased commodity that is a purchaser of the action in which the commodity is recognized by the first recognition unit when the moving direction of the commodity recognized by the second time coincides with the takeout direction of the takeout action recognized by the first time within a range that coincides with the second time.

According to the sales management system, the customer who takes the commodity can be accurately specified in time based on the taking-out direction, the moving direction, and the like.

In the sales management system, the method further includes: an identification unit that identifies a purchaser; a tracking unit that tracks the purchaser; and a purchaser specifying unit that specifies the purchaser detected by the detecting unit based on a result of tracking the purchaser identified by the identifying unit by the tracking unit, wherein the registering unit registers the purchased product for each purchaser specified by the purchaser specifying unit.

According to the sales management system described above, the purchased product can be registered based on the result of tracking the purchaser.

In the sales management system, the method further includes: and a measuring unit that measures the weight of the displayed product, wherein the second recognition unit recognizes the integer as the number of moved products when the weight measured by the measuring unit is reduced by an integer multiple of the unit weight of the displayed product.

According to the sales management system described above, the number of products taken out from the product shelf can also be specified.

In the sales management system, the method further includes: and a unit for transmitting information on the registered product to an information terminal owned by the purchaser.

According to the sales management system described above, the purchaser can sequentially confirm the purchased products using the information terminal.

In the sales management system, the detection unit controls a purchaser detection unit that detects the position of the purchaser by combining a beacon terminal and an information terminal owned by the purchaser.

According to the sales management system, the purchaser can be detected by detecting the information terminal.

In the sales management system, the method further includes: and a unit for transmitting the completion of payment of the registered commodity to the information terminal owned by the purchaser.

According to the sales management system, the purchaser can easily grasp the completion of the settlement by using the information terminal.

In the sales management system, when the first recognition unit recognizes the action of the purchaser to take out the product and the direction of taking out the product, and the second recognition unit recognizes the movement of the product and the direction of movement of the product, the purchaser detection unit may notify an abnormality when the purchaser cannot specify the position of the purchaser.

According to the sales management system, it is possible to easily check an abnormality.

In the sales management system, the detection unit controls the purchaser detection unit, the tracking unit controls the tracking detection unit that detects the position of the purchaser, and the purchaser detection unit and the tracking unit are used in common.

According to the sales management system, the system configuration can be simplified, and the cost can be reduced.

A sales management method according to another aspect of the present invention manages sales of displayed commodities, the sales management method including the steps of: a detection unit for detecting a purchaser who faces the displayed commodity; a first recognition unit recognizing the detected action of the purchaser to take out the opposing commodity and the direction of taking out the commodity; a second recognition unit recognizing the movement of the displayed commodity and the movement direction of the commodity; a specifying unit that, when the movement of the commodity and the movement direction thereof are recognized and the movement direction thereof is recognized, specifies the commodity recognized as the movement of the commodity recognized by the purchaser as a purchased commodity when the movement direction coincides with the movement direction; and a registration unit registering the purchased product of the specified purchaser.

According to the sales management method described above, it is possible to provide a method for accurately specifying a customer who has taken a product based on the pickup direction and the movement direction.

Drawings

Next, a sales management system and a sales management method according to the embodiments will be described with reference to the drawings. A more complete appreciation of the invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein the accompanying drawings are included to provide a further understanding of the invention and form a part of this application, and wherein the illustrated embodiments of the invention and the description thereof are intended to illustrate and not limit the invention, wherein:

FIG. 1 is a schematic diagram showing a schematic configuration of an embodiment of a sales management system;

fig. 2 is a block diagram showing a hardware configuration of an entrance gate device as a component of the sales management system;

fig. 3 is a block diagram showing a hardware configuration of a person tracking apparatus as a component of the sales management system;

FIG. 4 is a block diagram showing a hardware configuration of a behavior monitoring device as a component of the sales management system;

fig. 5 is a block diagram showing a hardware configuration of a product monitoring apparatus as a component of the sales management system;

fig. 6 is a schematic view of a product shelf to which a behavior monitoring device and a product monitoring device are applied, viewed from the front side;

fig. 7 is a schematic view of a product shelf to which a behavior monitoring device and a product monitoring device are applied, viewed from one side;

fig. 8 is a graph for explaining characteristics of the weight sensor provided in the commodity shelf;

fig. 9 is a block diagram showing a hardware configuration of a merge recognition device as a component of the sales management system;

fig. 10 is a block diagram showing a hardware configuration of a sales management apparatus as a component of the sales management system;

FIG. 11 is a diagram showing a data structure of a purchase list created in the sales management apparatus;

fig. 12 is a flowchart showing a main portion control procedure of a control portion in the entrance door apparatus;

fig. 13 is a flowchart showing a main portion control procedure of the control portion in the human tracking device;

fig. 14 is a flowchart showing a main portion control procedure of a control portion in the behavior monitoring device;

fig. 15 is a flowchart showing a main unit control procedure of the control unit in the merchandise surveillance device;

fig. 16 is a flowchart showing a main portion control procedure of a control portion in the merge recognition apparatus;

fig. 17 is a flowchart showing a main portion control procedure of the control portion in the sales management apparatus;

fig. 18 is a flowchart showing a main portion control procedure of the control portion in the sales management apparatus;

fig. 19 is a flowchart showing a main portion control procedure of the control portion in the sales management apparatus;

fig. 20 is a flowchart showing a main portion control procedure of the control portion in the sales management apparatus;

fig. 21 is a flowchart showing a main portion control procedure of the control portion in the sales management apparatus; and

fig. 22 is a flowchart showing a main portion control procedure of the control portion in the sales management apparatus.

Description of the reference numerals

10 entrance door device 20 people tracking device

30 action monitoring device 40 commodity monitoring device

50 merge recognition device 60 sales management device

70 communication network 100 sales management system

Detailed Description

Next, an embodiment of a sales management system and a sales management method that can accurately specify a customer who has taken a product will be described with reference to the drawings.

In the present embodiment, a shop in which shopping is possible without a store clerk.

Fig. 1 is a schematic configuration diagram of a sales management system 100 constructed by the store. The sales management system 100 includes an entrance door apparatus 10, a person tracking apparatus 20, a behavior monitoring apparatus 30, a product monitoring apparatus 40, a merge recognition apparatus 50, and a sales management apparatus 60. The sales management system 100 is connected to the entrance door apparatus 10, the person tracking apparatus 20, the action monitoring apparatus 30, the product monitoring apparatus 40, the merge recognition apparatus 50, and the sales management apparatus 60 via the communication network 70 so as to be capable of bidirectional communication. The communication Network 70 is, for example, a LAN (Local Area Network). The LAN may be a wired LAN or a wireless LAN.

The entrance door apparatus 10 has a login function for accepting a procedure for a purchaser who is a customer to enter a store, and a door function for selectively forming an open state in which the purchaser can enter the store and a closed state in which the customer is prevented from entering the store. The sales management system 100 may have only one entrance door apparatus 10 or may have a plurality of entrance door apparatuses 10.

Fig. 2 is a block diagram showing a hardware configuration of the entrance door apparatus 10. The entrance door device 10 includes a receiving unit 11, an opening/closing unit 12, a clock unit 13, a communication unit 14, a storage unit 15, and a control unit 16. Needless to say, the entrance door apparatus 10 may have other hardware configurations.

The receiving unit 11 is a device provided at an entrance of a store and receives registration of a purchaser. The member registration is performed in advance when the purchaser uses the store for the first time. By performing the member registration, a unique member ID is issued to the purchaser. The member ID is set in an information terminal such as a smartphone owned by a purchaser, for example. When an application program dedicated to the information terminal is activated, a barcode or a two-dimensional code indicating the member ID is displayed on a display (display unit) of the information terminal. In this case, a scanner for scanning the barcode or the two-dimensional code displayed on the display can be the receiving unit 11. The signal indicating the member ID may be transmitted from the information terminal by short-range wireless communication using a technique called RFID (Radio Frequency IDentification). In this case, the antenna corresponding to the short-range wireless communication can serve as the receiving unit 11. The member ID may be recorded in an IC card, a magnetic card, or the like and issued to the purchaser. In this case, a card reader such as an IC card or a magnetic card can be used as the receiving unit 11.

After the purchaser enters the store, the purchaser keeps the application program in operation. By doing so, the purchased state can be grasped through the display of the smartphone.

The opening/closing unit 12 is a device for switching opening and closing of doors provided at entrances of stores. When the door is in the closed state, entry into the store is prevented. When the door is in the open state, entry into the store is permitted. The opening/closing section 12 normally closes the door. When the registration in the reception unit 11 is approved, the opening/closing unit 12 temporarily opens the door. The opening/closing unit 12 has a sensor for detecting a person who has passed through the door.

The clock section 13 counts date and time. In the entrance door apparatus 10, the date and time measured by the clock section 13 is managed as the store entry date and time of the purchaser.

The communication unit 14 is connected to a communication network 70. The communication unit 14 performs data communication with the person tracking device 20, the action monitoring device 30, the product monitoring device 40, the merge recognition device 50, and the sales management device 60 via the communication network 70 according to a predetermined communication protocol.

The storage unit 15 includes a nonvolatile storage area and a volatile storage area. The storage unit 15 stores a control program for controlling the operation of the input gate device 10 in a nonvolatile storage area. The storage unit 15 stores data necessary for operating the access gate device 10 in a volatile storage area. The non-volatile Memory area is, for example, a ROM (Read Only Memory). Volatile Memory areas are, for example, RAM (Random Access Memory).

The control unit 16 is a control unit that realizes the functions of the entrance door apparatus 10 by controlling the reception unit 11, the opening/closing unit 12, the clock unit 13, and the communication unit 14 according to the control program. The control unit 16 is mainly configured by a processor. The processor is, for example, a CPU (Central Processing Unit). In the entrance door apparatus 10, the control unit 16 and the storage unit 15 constitute a computer.

The person tracking apparatus 20 is an apparatus for tracking a line of action in a store from a purchaser who enters the store through an exit in real time.

Fig. 3 is a block diagram showing a hardware configuration of the human tracking device 20. The human tracking device 20 includes a detection unit 21, a tracking unit 22, a clock unit 23, a communication unit 24, a storage unit 25, and a control unit 26. Needless to say, the person tracking apparatus 20 may have another hardware configuration.

The detection unit 21 (tracking detection unit) is a unit that detects the positions of a plurality of shoppers who have entered the store by a plurality of sensors arranged in the store. The sensors are mounted, for example, on the ceiling of the store, respectively. The sensors may be located, for example, along the customer aisles within the store. As the sensor, a known camera sensor such as an optical camera, an infrared camera, a TOF (Time of Flight) camera, or a stereo camera can be used. In this way, the person can be directly identified to be specified as the purchaser. Further, a beacon terminal that transmits an ID by an electric wave may be arranged at each place in a store, and the beacon terminal may be combined with a smartphone held by a shopper to detect the position of the purchaser. By doing so, the person holding the smartphone can be specified as the purchaser.

The tracking unit 22 is a unit that tracks the movement line of each purchaser in the store based on the positional information of each purchaser detected by the detection unit 21. The tracking unit 22 is configured to track the activity line of each purchaser by accumulating, in time series, tracking data including the positional information and the detection date and time of the purchaser, for example, for each purchaser who has been approved to log in by the entrance door apparatus 10.

The clock section 23 counts date and time. In the person tracking apparatus 20, the date and time counted by the clock unit 23 is managed as the detected date and time of the purchaser.

The communication unit 24 is connected to a communication network 70. The communication unit 24 performs data communication with the entrance gate device 10, the action monitoring device 30, the commodity monitoring device 40, the merge recognition device 50, and the sales management device 60 via the communication network 70 according to a predetermined communication protocol.

The storage unit 25 includes a nonvolatile storage area and a volatile storage area. The storage unit 25 stores a control program for controlling the operation of the human tracking apparatus 20 in a nonvolatile storage area. The storage unit 25 stores data necessary for the operation of the person tracking apparatus 20 in a volatile storage area. The non-volatile memory area is for example a ROM. A volatile memory area is for example RAM.

The control unit 26 is a unit that realizes the functions of the human tracking device 20 by controlling the detection unit 21, the tracking unit 22, the clock unit 23, and the communication unit 24 according to the control program. The control unit 26 is mainly configured by a processor. The processor is, for example, a CPU. In the human tracking device 20, the control unit 26 and the storage unit 25 constitute a computer.

The behavior monitoring device 30 is a device for monitoring the behavior of the purchaser who faces the displayed product. The behavior monitoring device 30 monitors whether or not a purchaser who stands in a place where a product is displayed, for example, in front of a product shelf and becomes a monitoring target takes out a behavior of gripping the displayed product. When such an action is performed, the action monitoring device 30 specifies the commodity removal direction. The sales management system 100 is typically provided with a behavior monitoring device for each product shelf. The sales management system 100 may be configured with one motion monitoring device 30 for a plurality of product shelves. The number of the action monitoring devices is not limited if the presence or absence of an action of taking out a product from the commodity shelf of a purchaser is monitored for all the commodity shelves in a store, and if the action is given, the direction of taking out the product can be specified.

Fig. 4 is a block diagram showing a hardware configuration of the behavior monitoring device 30. The action monitoring device 30 includes a purchaser detecting section 31, an action determining section 32, an extraction direction designating section 33, a clock section 34, a communication section 35, a storage section 36, and a control section 37. Needless to say, the behavior monitoring device 30 may have another hardware configuration.

The purchaser detecting unit 31 is a unit that detects a purchaser (person) who faces a displayed product by a sensor disposed in a store and mainly located near a product shelf. The purchaser detecting unit 31 regards a section of a predetermined width assumed to be on the ground in front of the commodity shelf as a monitoring target area. The purchaser detecting unit 31 detects a purchaser in the monitoring target area. The purchaser detecting unit 31 can simultaneously detect a plurality of purchasers in the monitoring target area. As the sensor, a known camera sensor such as an optical camera, an infrared camera, a TOF camera, or a stereo camera can be used. The sensor of the purchaser detecting unit 31 may be shared with the sensor necessary for the detecting unit 21 of the person tracking apparatus 20. For example, the location of the purchaser is detected by combining a beacon terminal and a smartphone held by the purchaser. By doing so, the person holding the smartphone can be detected as a purchaser. If the person holding the smartphone is rotated at the time of multi-person access, the person holding the smartphone can be detected as a purchaser. When multiple persons visit and settle accounts uniformly by one person, the smart phones can be held alternately to correspond to each other, so that the method is convenient.

Further, if the purchaser detecting unit 31 and the detecting unit 21 of the person tracking device 20 are shared, the system configuration can be simplified and the cost can be reduced.

The action determining unit 32 is a unit that determines the action of the purchaser detected by the purchaser detecting unit 31. The behavior determination unit 32 determines whether or not the purchaser has moved from the monitoring target area. The action determining unit 32 determines whether or not the purchaser takes action to hold the displayed product and take out the product. The action determination unit 32 determines the action of each purchaser when the purchaser detection unit 31 detects a plurality of purchasers. The action determination unit 32 can determine the presence or absence of an action by using a sensor used in the purchaser detection unit 31. The action determining unit 32 may determine the presence or absence of an action using another sensor.

The takeout direction designating unit 33 is a unit that designates the takeout direction of the commodity when the action of gripping and taking out the commodity displayed on the display is detected by the action determining unit 32. The extraction direction specification unit 33 can specify the extraction direction by using a sensor used in the purchaser detection unit 31, for example. The extraction direction specifying unit 33 may specify the extraction direction using another sensor.

The clock section 34 clocks date and time. In the motion monitoring device 30, the date and time measured by the clock unit 34 is managed as the date and time of detection of the motion of grasping and taking out the product.

The communication unit 35 is connected to a communication network 70. The communication unit 35 performs data communication with the entrance door apparatus 10, the person tracking apparatus 20, the product monitoring apparatus 40, the merge recognition apparatus 50, and the sales management apparatus 60 via the communication network 70 according to a predetermined communication protocol.

The storage unit 36 includes a nonvolatile storage area and a volatile storage area. The storage unit 36 stores a control program for controlling the operation of the behavior monitoring device 30 in a nonvolatile storage area. The storage unit 36 stores data necessary for the operation of the behavior monitoring device 30 in a volatile storage area. The non-volatile memory area is for example a ROM. A volatile memory area is for example RAM.

The control unit 37 is a unit that realizes the functions of the motion monitoring device 30 by controlling the purchaser detecting unit 31, the motion determining unit 32, the extraction direction specifying unit 33, the clock unit 34, and the communication unit 35 according to the control program. The control unit 37 is mainly configured by a processor. The processor is, for example, a CPU. In the behavior monitoring device 30, the control unit 37 and the storage unit 36 constitute a computer.

The product monitoring device 40 is a device for monitoring the movement of the displayed product. The article monitoring device 40 specifies an article taken out from the article shelf, and further specifies a moving direction of the article taken out from the article shelf. The sales management system 100 is typically provided with a product monitoring apparatus 40 for each product shelf. The sales management system 100 may be configured with one product monitoring apparatus 40 for a plurality of product shelves. The number of the product monitoring devices 40 is not limited if the product taken out from the product shelf and the moving direction of the product can be specified for all the product shelves in the store.

Fig. 5 is a block diagram showing a hardware configuration of the product monitoring device 40. The product monitoring device 40 includes a movement monitoring unit 41, a product specifying unit 42, a movement direction specifying unit 43, a clock unit 44, a communication unit 45, a storage unit 46, and a control unit 47. Needless to say, the product monitoring device 40 may have another hardware configuration.

The movement monitoring unit 41 monitors the displayed commodity with a sensor disposed on the commodity shelf or in the vicinity thereof, and detects the movement of the commodity. As the sensor, a known camera sensor such as an optical camera, an infrared camera, a TOF camera, or a stereo camera can be used. Alternatively, a camera sensor and a weight sensor or a distance sensor may be combined to monitor a product and detect the movement of the product.

The product specifying unit 42 is a unit that specifies the product and the number of products whose movement is detected by the movement monitoring unit 41. The product specification unit 42 can specify a product that has moved, for example, by using a sensor used in the movement monitoring unit 41. The product specification unit 42 may specify a product to be moved using another sensor.

The movement direction specifying unit 43 is a unit that specifies the movement direction of the commodity detected to move by the movement monitoring unit 41. The movement direction specifying unit 43 can specify the movement direction by using a sensor used in the movement monitoring unit 41, for example. The moving direction specifying unit 43 may specify the moving direction using another sensor.

The clock section 44 counts date and time. In the product monitoring device 40, the date and time counted by the clock unit 44 is managed as the date and time at which the movement of the product is detected.

The communication unit 45 is connected to a communication network 70. The communication unit 45 performs data communication with the entrance door apparatus 10, the person tracking apparatus 20, the action monitoring apparatus 30, the merge recognition apparatus 50, and the sales management apparatus 60 via the communication network 70 according to a predetermined communication protocol.

The storage unit 46 includes a nonvolatile storage area and a volatile storage area. The storage unit 46 stores a control program for controlling the operation of the product monitoring apparatus 40 in a nonvolatile storage area. The storage unit 46 stores data necessary for the operation of the product monitoring device 40 in a volatile storage area. The non-volatile memory area is for example a ROM. A volatile memory area is for example RAM.

The control unit 47 is a unit that realizes the functions of the product monitoring device 40 by controlling the movement monitoring unit 41, the product specifying unit 42, the movement direction specifying unit 43, the clock unit 44, and the communication unit 45 according to the control program. The control unit 47 is mainly configured by a processor. The processor is, for example, a CPU. In the product monitoring device 40, the control unit 47 and the storage unit 46 constitute a computer.

Here, an example of the product shelf 80 to which the behavior monitoring device 30 and the product monitoring device 40 are applied will be described with reference to fig. 6 to 8.

Fig. 6 is a schematic view of the product shelf 80 viewed from the front side, and fig. 7 is a schematic view of the product shelf 80 viewed from the side. In fig. 6 and 7, a commodity shelf 80 having a three-layer structure is illustrated, which is partitioned between a bottom plate 81 and a top plate 82 by two shelf partitions 83. Each floor is divided into three sections, i.e., a left section, a right section, and a center section, and one type of merchandise is displayed in one section.

The weight sensors 84 are disposed on the surfaces of the compartments on which the commodities are displayed. The weight sensor 84 is a sensor for measuring the total weight of the products displayed in the section. Further, a camera sensor 85 is mounted in each layer. The camera sensor 85 is, for example, an optical camera or an infrared camera, and monitors the commodities displayed on each layer. The mounting position of the camera sensor is not particularly limited as long as the product can be monitored. Fig. 6 shows an example in which one camera sensor 85 is disposed for each layer, but the number of camera sensors 85 is not particularly limited. For example, the camera sensor 85 may be disposed for each zone of each layer.

The side of the product shelf 80 on which the product is placed is referred to as the near side. In the present embodiment, as shown in fig. 6 and 7, on the ground surface on the near side, a section of the edge width H of the commodity shelf 80 is set as the monitoring target area AR 1. A camera sensor 91 for detecting the purchaser CA1 located in the monitoring target area AR1 is mounted above the commodity shelf 80, for example, on the ceiling CE 1. Preferably, the camera sensor 91 is installed at a position where the purchaser CA1 can be photographed from obliquely above. The camera sensor 91 detects that the purchaser CA1 is in the monitoring target area AR1 by, for example, reflecting the head of the purchaser CA1 in the imaging area AR 2. For example, when two heads are reflected in the imaging area AR2, the camera sensor 91 detects that there are two purchasers CA1 in the monitoring target area AR 1. The camera sensor 91 is a TOF camera, a stereo camera, or a 2D camera. By using a TOF camera, a stereo camera, or a 2D camera as the camera sensor 91, it is possible to estimate the body of the purchaser AC1 located in the monitoring target area AR1 and detect the action of the purchaser CA1 from the movement of the body. Specifically, the action of the purchaser CA1 to extend the forearm portion toward the commodity shelf 80 or the action to retract the forearm portion from the commodity shelf 80 can be detected. Further, it is possible to detect the operation in which the front arm portion extends from the near side of the commodity shelf toward the commodity shelf in which direction and, conversely, returns to which direction. The near side is a side on which the product is placed on the product shelf 80.

In the present embodiment, as shown in fig. 7, the near side of the commodity shelf 80 is defined as a screen-like behavior monitoring area AR 3. As shown in fig. 6, the behavior monitoring area AR3 is formed so as to cover the front surface of the product shelf 80. Further, a camera sensor 92 for detecting the action of the purchaser with respect to the action monitoring area AR3 is mounted on, for example, the ceiling CE 1. The camera sensor 92 is a TOF camera or a stereo camera. The camera sensor 92 detects the entrance or exit of the forearm of the action monitoring area AR 3.

Here, the camera sensor 91 is a sensor for constituting the purchaser detecting unit 31 of the action monitoring device 30. The camera sensor 91 and the camera sensor 92 are sensors for constituting the behavior determination unit 32 and the acquisition direction designation unit 33 of the behavior monitoring device 30. That is, by combining the information acquired from the camera sensor 91 and the camera sensor 92, the direction in which the product has been taken out can be detected more accurately. The camera sensor 85 is a sensor for constituting the movement monitoring unit 41 and the movement direction specifying unit 43 of the product monitoring device 40. The weight sensor 84 is a sensor for constituting the movement monitoring unit 41 and the product specifying unit 42 of the product monitoring device 40.

The weight sensor 84 measures the total weight of the commodity displayed in the section in which the weight sensor 84 is disposed as described above, and the commodity displayed in the section is of one type. Therefore, for example, when the measured weight is reduced from W2 to W1 in the weight sensor 84 of the section where the commodity AAA, whose weight per commodity is so-called the unit weight W0, is displayed, the quotient obtained by dividing the reduced weight Δ W (W2-W1) by the unit weight W0 is the number of the commodity AAA to be taken out from the section. In this way, the movement monitoring unit 41 can recognize the movement of the displayed commodity because the measured weight of the weight sensor 84 is reduced, and the commodity specifying unit 42 can specify the type and number of the moved commodities because the measured weight is reduced by an integral multiple of the weight of each commodity.

However, the timing at which the movement monitoring unit 41 detects the movement of the product and the timing at which the product specifying unit 42 specifies the product are offset. Further, depending on the characteristics of the weight sensor 84, a delay occurs from when the article actually starts to move to when the movement of the article is detected.

Fig. 8 is a graph for explaining characteristics of the weight sensor 84. In fig. 8, the vertical axis shows the weight measured by the weight sensor 84, and the horizontal axis shows the elapsed time. At time t0, weight sensor 84 measures a weight W2. At a subsequent time point t1, it is assumed that the commodity in the section in which the weight sensor 84 is disposed has been picked up, that is, the movement of the commodity has been started. At this time, the measured weight of the weight sensor 84 starts to decrease at a time point t2 when a certain delay time Δ t elapses. Therefore, the movement monitoring unit 41 can detect the movement of the commodity at the time point t 2. Then, at a time point t3 when the decrease in the measured weight to W1 is detected, the article specifying unit 42 can specify the type and the number of articles that have been moved. In this way, a deviation occurs between the time t2 at which the movement of the commodity is detected in the movement monitoring unit 41 and the time t3 at which the commodities and the number thereof that have moved in the commodity specifying unit 42 are specified. In addition, a deviation also occurs between the time point t1 at which the movement of the article has actually been started and the time point t2 at which the movement of the article is detected in the movement monitoring unit 41.

The integrated recognition device 50 is a device for associating the purchaser specified by the behavior monitoring device 30 with the product specified by the product monitoring device 40.

Fig. 9 is a block diagram showing a hardware configuration of the merge recognition device 50. The integrated recognition device 50 includes a product monitoring acquisition unit 51, an action monitoring acquisition unit 52, a matching unit 53, a purchase designation unit 54, a communication unit 55, a storage unit 56, and a control unit 57. Needless to say, the merge recognition means 50 may have another hardware configuration.

The product monitoring acquisition unit 51 is a unit that acquires product information from the product monitoring device 40. The article information to be taken out will be described later.

The behavior monitoring acquisition unit 52 is a unit that acquires the purchaser retrieval information from the behavior monitoring device 30 that monitors the behavior of the purchaser in front of the commodity shelf monitored by the commodity monitoring device 40 when the commodity monitoring acquisition unit 51 acquires the retrieval information from the commodity monitoring device 40. The purchaser retrieval information will be described later.

The collating unit 53 is a unit that collates the taken-out article information acquired by the article monitoring acquiring unit 51 and the purchaser taken-out information acquired by the action monitoring acquiring unit 52.

The purchase designating unit 54 is a unit for designating a purchaser who purchased a product and a product purchased by the purchaser, based on the result of the collation by the collating unit 53.

The communication unit 55 is connected to a communication network 70. The communication unit 55 performs data communication with the entrance door apparatus 10, the person tracking apparatus 20, the action monitoring apparatus 30, the commodity monitoring apparatus 40, and the sales management apparatus 60 via the communication network 70 according to a predetermined communication protocol.

The storage unit 56 includes a nonvolatile storage area and a volatile storage area. The storage unit 56 stores a control program for controlling the operation of the merge recognition device 50 in a nonvolatile storage area. The storage unit 56 stores data necessary for the operation of the merge recognition device 50 in a volatile storage area. The non-volatile memory area is for example a ROM. A volatile memory area is for example RAM.

The control unit 57 controls the action monitoring acquisition unit 52, the product monitoring acquisition unit 51, the matching unit 53, the purchase designation unit 54, and the communication unit 55 according to the control program, thereby realizing the functions of the integrated identification device 50. The control unit 57 is mainly configured by a processor. The processor is, for example, a CPU. In the merge recognition device 50, the control unit 57 and the storage unit 56 constitute a computer.

The sales management apparatus 60 is a unit that creates a list of products purchased by a purchaser and processes payment with the purchaser based on the list. Hereinafter, such a list is referred to as a purchase list.

Fig. 10 is a block diagram showing a hardware configuration of the sales management apparatus 60. The sales management device 60 includes a product information storage unit 61, a member information storage unit 62, a purchase list creation unit 63, a purchase list update unit 64, a payment processing unit 65, a communication unit 66, a storage unit 67, and a control unit 68. Needless to say, the sales processing apparatus 60 may have another hardware configuration.

The product information storage unit 61 is a unit that stores product information such as product ID, product name, and price of each product sold in a store. The product ID is a unique identification code that is set for each product to individually identify each product.

The member information storage unit 62 is a unit that stores member information such as a member ID, expiration date, reject flag, and payment account of a purchaser who has performed member registration. The reject flag is one bit of data set for a member who cannot enter the store due to an expiration date or improper use. The payment account is information related to a deposit account of a financial institution designated by a purchaser.

The product information storage unit 61 and the member information storage unit 62 are formed in a storage device such as an EEPROM (electrically Erasable Programmable Read-Only Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive).

The purchase list creation unit 63 is a unit that creates a purchase list 600 (see fig. 11) for accumulating the product ID and the number of purchases of the product purchased by the purchaser. The purchase list 600 is formed in a volatile storage area of the storage section 67, for example.

Fig. 11 is a diagram showing a data structure of the purchase list 600. As shown in fig. 11, the purchase list 600 has an area for storing paired data of the store entry time T0, the first ID, the plurality of product IDs, and the number of purchases in association with the member ID. The store entry time T0 and the first ID will be clear from the operation description to be described later.

The purchase list updating unit 64 is a unit that updates the purchase list 600 created by the purchase list creating unit 63.

The payment processing unit 65 is a unit that executes payment processing based on the data stored in the purchase list 600.

The communication unit 66 is connected to a communication network 70. The communication unit 66 performs data communication with the entrance door apparatus 10, the person tracking apparatus 20, the action monitoring apparatus 30, the commodity monitoring apparatus 40, and the merge recognition apparatus 50 via the communication network 70 according to a predetermined communication protocol.

The storage unit 67 includes a nonvolatile storage area and a volatile storage area. The storage unit 67 stores a control program for controlling the operation of the sales management apparatus 60 in a nonvolatile storage area. The storage unit 67 stores data necessary for operating the sales management apparatus 60 in a volatile storage area. The non-volatile memory area is for example a ROM. A volatile memory area is for example RAM.

The control unit 68 is a unit that realizes the functions of the purchase management device 60 by controlling the product information storage unit 61, the member information storage unit 62, the purchase list creation unit 63, the purchase list update unit 64, the settlement processing unit 65, and the communication unit 66, respectively, according to the control program. The control unit 68 is mainly configured by a processor. The processor is, for example, a CPU. In the sales management apparatus 60, the control unit 68 and the storage unit 67 constitute a computer.

The above completes the description of the hardware configuration of the sales management system 100. As described above, the entrance door apparatus 10, the person tracking apparatus 20, the action monitoring apparatus 30, and the product monitoring apparatus 40 are provided with the

clock units

13, 23, 34, and 44. These

clock sections

13, 23, 34 and 44 are synchronized. Alternatively, these

clock units

13, 23, 34, and 44 may be used as a common external clock, and the external clocks may be referred to by the

control units

16, 26, 37, and 47 of the entrance door apparatus 10, the person tracking apparatus 20, the action monitoring apparatus 30, and the product monitoring apparatus 40, respectively.

Fig. 12 is a flowchart showing a main portion control procedure of the control portion 16 in the entrance door apparatus 10. Fig. 13 is a flowchart showing a main part control procedure of the control unit 26 in the human tracking device 20. Fig. 14 is a flowchart showing a main portion control procedure of the control portion 37 in the behavior monitoring device 30. Fig. 15 is a flowchart showing a main unit control procedure of the control unit 47 in the product monitoring device 40. Fig. 16 is a flowchart showing the procedure of main portion control by the control portion 57 in the merge recognition device 50. Fig. 17 to 22 are flowcharts showing the essential part control procedure of the control part 68 in the sales management apparatus 60. Next, the operation of the sales management system 100 will be described with reference to these drawings. By this action explanation, the sales management method of the present embodiment becomes clear. The following description of the sequence of operations and the contents thereof is an example. The order and content are not limited as long as the same result can be obtained.

First, the control unit 16 of the entrance door apparatus 10 waits for receiving an input of a member ID as the ACT101 of fig. 12. When the purchaser performs a registration operation with respect to the receiving unit 11 at the entrance of the store, the member ID of the purchaser is input, and therefore, the control unit 16 determines YES in the ACT101 and proceeds to the ACT 102. The control unit 16 controls the communication unit 14 to notify the sales management apparatus 60 of the member. By this control, the communication unit 14 transmits a staff notification command to the sales management apparatus 60 via the communication network 70. The member notification command includes a member ID input by the login operation.

The control section 68 of the sales management apparatus 60 waits for an acceptance notification command as an ACT601 of fig. 17. When the notification command is received through the communication section 66, the control section 68 determines YES in ACT601 and proceeds to ACT 602. The control section 68 confirms as an ACT602 whether the received command is a member notification command. At this time, since the member communication command is received, the control unit 68 determines YES in the ACT602, and proceeds to the ACT611 of fig. 18.

The control unit 68 acquires the member ID from the member notification command as an ACT 611. If the member ID can be acquired, the control unit 68 searches the member information storage unit 62 with the member ID thereof as the ACT612, and acquires the member information associated with the member. The control unit 68 determines whether or not to permit entry into a store as ACT 613. For example, when the member information cannot be acquired from the member information storage unit 62, the control unit 68 determines that the entry into the store is denied. Alternatively, when the member information is acquired but the reject flag is set, the control unit 68 determines that the entry into the store is rejected. When the member information with the reject flag not set can be acquired, the control unit 68 determines that the store is permitted.

If the control unit 68 is permitted to enter the store, it is determined YES in ACT613 and the process proceeds to ACT 614. The control section 68 controls the communication section 66 to reply the permission response as an ACT 614. In contrast, if the entry is rejected, the control unit 68 determines NO in ACT613 and proceeds to ACT 615. The control section 68 controls the communication section 66 to return a response rejection response as an ACT 615. By these controls, the communication section 66 transmits the permission response command or the rejection response command to the entrance door apparatus 10 through the communication network 70. As described above, the control unit 68 ends the reception process of the member notification command.

The explanation returns to fig. 12.

The control section 16 of the entrance door apparatus 10 that has controlled the transmission of the member notification command waits for the reception of the response command from the sales management apparatus 60 as the ACT 103. Here, when the reject response command is received, the control section 16 determines NO in the ACT103 and proceeds to the ACT 104. The control unit 16 notifies the acceptance unit 11 of the prohibition of store entry as the ACT104 control. By this control, the acceptance unit 11 notifies the registered purchaser of the shop prohibition by display or voice. As described above, the control unit 16 ends the information processing in the sequence shown in the flowchart of fig. 12.

On the other hand, when the permission response command is received in the ACT103, the control section 16 determines YES and proceeds to the ACT 105. The control unit 16 controls the opening/closing unit 12 to open the door as an ACT 105. The opening/closing unit 12 opens the door by the control unit. The purchaser who has registered the purchase can enter the store by opening the door. When the purchaser passes through the door, the sensor of the opening/closing unit 12 detects entry into the store, and a store entry detection signal is output to the control unit 16.

The control unit 16 that controls the opening of the door waits for the reception of the store entrance detection signal as the ACT 106. When the store entry detection signal is input, the control unit 16 determines YES in ACT106 and proceeds to ACT 107. The control unit 16 controls the opening/closing unit 12 to close the door as an ACT 107. By this control, the door is closed by the opening/closing portion 12. The pass gate is closed so that the next purchaser is blocked from entering the store.

The control unit 16 that controls the door closing acquires the time counted by the clock unit 13 as the store entry time T0 as the ACT 108. The control unit 16 controls the communication unit 14 to notify the sales management apparatus 60 of the store entry as an ACT 109. Further, the control unit 16 controls the communication unit 14 to instruct the person tracking device 20 to start tracking as the ACT 110. By these controls, the communication unit 14 transmits the store entry notification instruction to the sales management apparatus 60 via the communication network 70. The store entry notification command includes the member ID and the store entry time T0 input by the registration operation in the reception unit 11. Further, the communication unit 14 transmits a tracking start command to the personal tracking device 20 via the communication network 70. The tracking monitor command includes a store entry time T0.

The control section 68 of the sales management apparatus 60 detects that the notification command is received in ACT601 of fig. 17, determines that the notification command is not a member notification command in ACT602, and proceeds to ACT 603. The control section 68 confirms whether or not the notification command is an entrance notification command as an ACT 603. At this time, since the store entry notification command is received, the control unit 68 determines YES in ACT603, and proceeds to ACT621 of fig. 19.

The control unit 68 obtains the member ID and the store entry time T0 from the store entry notification command as an ACT 621. The control unit 68 controls the purchase list creation unit 63 as an ACT622, and creates the purchase list 600 in the storage unit 67. The control unit 68 controls the purchase list update unit 64 as an ACT623 to register the member ID and the arrival time T0 obtained from the arrival notification command in the purchase list 600. As described above, the control unit 68 ends the reception process of the store entry notification command.

On the other hand, as shown in fig. 13, the control unit 26 of the human tracking device 20 waits for the reception of the tracking start command as an ACT 201. When the notification unit 24 receives the tracking start command, the control unit 26 determines YES in ACT201 and proceeds to ACT 202. The control unit 26 supplements a person who has passed through the entrance as a purchaser who newly approves the login, based on the positional information of each purchaser detected by the detection unit 21 as the ACT 202. The control unit 26 obtains the time counted by the clock unit 23 as the supplemental time T1 of the purchaser as the ACT 203. The control unit 26 gives a unique first ID to the purchaser as the ACT 204.

In this way, if the control unit 26 captures a purchaser who has been authorized to log in to the store and gives a unique first ID to the purchaser, it controls the communication unit 14 to perform a tracking notification to the sales management apparatus 60 as the ACT 205. By this control, the communication unit 14 transmits a tracking notification command to the sales management apparatus 60 via the communication network 70. The tracking notification command includes the replenishment time T1 for the purchaser who has approved login and the first ID.

The control unit 68 of the sales management apparatus 60 detects that the notification command is received in ACT601 of fig. 17, determines that the notification command is not the store entry notification command in ACT603, and proceeds to ACT 604. The control section 68 confirms as an ACT604 whether or not its notification command is a trace notification command. At this time, since the trace notification command is received, the control unit 68 determines YES in ACT604 and proceeds to ACT631 of fig. 20.

The control unit 68 obtains the first ID and the replenishment time T1 from the trace notification command as ACT 631. The control unit 68 refers to the store entry time T0 of each purchase list formed by the storage unit 67 as the ACT 632. Then, the control unit 68 selects, as ACT633, the purchase list 600 in which the store entry time T0 closest to the replenishment time T1 is registered. The control unit 68 controls the purchase list update unit 64 as an ACT634, and registers the first ID acquired from the tracking notification command in the selected purchase list 600. As described above, the control unit 68 ends the reception process of the tracking notification command.

The explanation returns to fig. 13.

The control unit 26 of the person tracking apparatus 20 that has performed the tracking notification starts the tracking process for the purchaser as the ACT206 instruction tracking unit 22. By this command, the tracking unit 22 accumulates tracking data including the position information and the detection date and time in association with the first ID of the purchaser. Since such tracking processing is well-known, detailed description thereof will be omitted.

The control section 26 waits for the purchaser who is being tracked to leave the store as an ACT 207. For example, when an exit is provided separately from the entrance, the control unit 26 determines that the purchaser has left the store when the tracking unit 22 confirms that the purchaser has passed the exit. For example, when the tracking unit 22 confirms that the purchaser has passed the door of the doorway when the entrance and the exit are the shared doorway, the control unit 26 determines that the purchaser has left the store.

When it is detected that the purchaser has left the store, the control section 26 determines YES in ACT207 and proceeds to ACT 208. The control unit 26 controls the communication unit 24 to notify the sales management apparatus 60 of settlement as an ACT 208. By this control, the communication unit 24 transmits a checkout notification command to the sales management apparatus 60 via the communication network 70. The checkout notification command includes a first ID assigned to the purchaser who has been detected to leave the store. The operation of the control unit 68 of the sales management device 60 that has received the checkout notification command will be described later. The following operation of the control unit 26 that performs the settlement notification is also described below.

The control unit 37 of the behavior monitoring device 30 waits for acceptance of the purchaser CA1 by the purchaser detecting unit 31 as the ACT301 of fig. 14. When the purchaser CA1 is detected, the control section 37 determines YES in ACT301 and proceeds to ACT 302. The control section 37 gives a unique second ID to the purchaser CA1 as the ACT 302. Then, the control unit 37 controls the action determination unit 32 and the extraction direction specification unit 33 as the ACT303 to start monitoring the action of the purchaser CA 1. By this control, the behavior of the purchaser CA1 is monitored by the behavior determination unit 32 and the takeout direction specification unit 33.

The control section 37 confirms as an ACT304 whether or not it is determined by the purchaser detecting section 31 that the purchaser CA1 has moved outside the monitoring target area AR 1. When the purchaser CA1 is within the monitoring target area AR1, the control unit 37 determines NO in ACT304 and proceeds to ACT 305. The control unit 37 checks as ACT305 whether the action determination unit 32 determines that the purchaser CA1 takes such an action as to grasp the displayed product and take it out. When the purchaser CA1 has not taken such an action, the control section 37 determines NO in ACT305 and returns to ACT 304. Here, in ACT304 or ACT305, the control unit 37 waits for the purchaser CA1 to move outside the monitoring target area AR1 or for the purchaser CA1 to take out the displayed product.

When the purchaser CA1 takes action to grasp a displayed product and take it out in the wait state of ACT304 or ACT305, the control unit 37 determines YES in ACT305 and proceeds to ACT 306. The control unit 37 obtains the time when the clock unit 34 is counting time as the execution time Tx as the ACT 306. The control unit 37 acquires, as the ACT307, the takeout direction Dx of the product specified by the takeout direction specifying unit 33. The control unit 37 then acquires, as the ACT308, the positional information Q1 of the purchaser CA1 detected by the purchaser detecting unit 31. In this way, the control unit 37 creates purchaser retrieval information including the second ID, the execution time Tx, the retrieval direction Dx, and the positional information Q1 as an ACT309, and writes the purchaser retrieval information in the storage unit 36. After that, the control section 37 returns to the wait-to-accept state of ACT304 or ACT 305.

When the purchaser CA1 moves out of the monitoring target area AR1 in the wait-accepted state of ACT304 or ACT305, the control section 37 determines YES in ACT304 and proceeds to ACT 310. The control unit 37 controls the action determination unit 32 and the extraction direction specification unit 33 as the ACT310, and ends monitoring the action of the purchaser CA1 to which the second ID is given. As described above, the control unit 37 ends the information processing in the sequence shown in the flowchart of fig. 14.

The control unit 47 of the product monitoring device 40 waits for reception of detection of picking up of a product by the movement monitoring unit 41 in ACT401 of fig. 15. When it is confirmed that the pickup of the product is detected due to, for example, a decrease in the measured weight of the weight sensor 84 (time t2 in fig. 8), the control unit 47 waits for the reception of the specification of the product by the product specification unit 42 as the ACT 402. When the measured weight is reduced by an integral multiple of the unit weight of the product and the product is designated (time t3 in fig. 8), the control unit 47 acquires the product ID and the number of purchases of the product as the ACT 403. For example, the storage unit 46 stores a data table in which the weight sensors 84 and the product IDs of the products displayed in the sections in which the weight sensors 84 are arranged are associated with each other, and the control unit 47 acquires the product ID corresponding to the matching weight sensor from the data table. Further, the controller 47 divides the amount of decrease Δ W in weight measured by the weight sensor 84 by the unit weight W0 of the product identified by the product ID (W2-W1), and obtains the quotient as the purchase number.

When the product ID and the number of purchases are acquired, the control unit 47 starts a process of monitoring the movement of the product by the movement monitoring unit 41 and the movement direction specifying unit 43 as an ACT 404. Then, the control unit 47 checks as the ACT405 whether or not the product to be monitored has been returned to its original place. For example, when the measured weight of the weight sensor 84 returns to the state before the change, it is determined that the product to be monitored has been returned to its original place. If the product to be monitored is not returned to its original place, the control unit 47 determines NO in ACT405 and proceeds to ACT 406. The control unit 47 confirms as an ACT406 whether or not the product to be monitored has been taken out by the purchaser. When the product is not taken out, the control unit 47 determines NO in ACT406 and returns to ACT 407. Here, the control unit 47 waits for the ACT405 or 406 to receive that the picked-up product to be monitored is returned to the original place or taken out by the purchaser.

When the product to be monitored has been replaced in the wait-accepted state of the ACT405 or the ACT406, the control unit 47 determines YES in the ACT405 and proceeds to the ACT 407. The control unit 47 ends the monitoring process of the movement of the commodity by the movement monitoring unit 41 and the movement direction specifying unit 43 as an ACT 407. As described above, the control unit 47 ends the information processing in the sequence shown in the flowchart of fig. 15.

When the product to be monitored has been taken out by the purchaser in the wait-to-accept state of the ACT405 or the ACT406, the control unit 47 determines YES in the ACT406 and proceeds to the ACT 408. The control unit 47 obtains the time counted by the clock unit 44 as the commodity takeout time Ty as an ACT 408. The control unit 47 also acquires the movement direction Dy designated by the movement direction designating unit 43 as ACT 409. In this way, the control unit 47 creates, as ACT410, the taken-out article information including the article ID, the number of purchases, the taking-out time Ty, and the moving direction Dy, and writes the taken-out article information into the storage unit 46.

After that, the control unit 47 checks as ACT411 whether or not the article taken out has been put back. If the operation is not returned to the original position, the control unit 47 determines NO in ACT411 and proceeds to ACT 412. The control unit 47 checks as an ACT412 whether or not a certain time, for example, 10 seconds, has elapsed since the production of the taken-out product information. If the predetermined time has not elapsed, the control unit 47 determines NO in ACT412 and returns to ACT 411. Here, the control section 47 waits for acceptance of that the product taken out by the purchaser is put back to the place or a certain time elapses as the ACT411 or 412.

When the product taken out by the purchaser has been put back in place in the wait-accept state of ACT411 or ACT412, the control section 47 determines YES in ACT411 and proceeds to ACT 413. The control unit 47 discards the taken-out article information written in the storage unit 46 as the ACT 413. As described above, the control unit 47 ends the information processing in the sequence shown in the flowchart of fig. 15.

When a predetermined time has elapsed without the product being returned, the control unit 47 determines YES in ACT412 and proceeds to ACT 414. The control unit 47 controls the communication unit 45 to issue a fetch notification to the merge recognition device 50 as an ACT 414. By this control, the communication unit 45 transmits a take-out notification command to the merge recognition device 50 via the communication network 70. The article removal notification command includes the article removal information stored in the storage unit 46.

The control unit 57 of the merge recognition device 50 waits for the reception of the take-out notification as the ACT501 of fig. 16. When the communication unit 55 receives the takeout notification command transmitted from the product monitoring device 40, the control unit 57 determines YES in ACT501 and proceeds to ACT 502. The control unit 57 obtains the article information, that is, the article ID, the number of purchases, the removal time Ty, and the moving direction Dy from the removal notification command as the ACT 502. Then, the control unit 57 stores the article information in the storage unit 56.

The control unit 57 acquires, as the ACT503, the purchaser retrieval information, that is, the second ID, the execution time Tx, the retrieval direction Dx, and the position information Q1 in the order in which the execution time Tx is new, for example, from the action monitoring device 30 for monitoring the action of the purchaser in front of the commodity shelf on which the commodity monitoring device 40 of the retrieval notification command transmission source is displayed. Then, the control section 57 selects purchaser retrieval information in which the time condition and the direction condition match each other as the ACT 504. Specifically, the control unit 57 calculates a time difference between the retrieval time Ty at which the product information is retrieved and the execution time Tx at which the purchaser retrieves the information. If the time difference is within the allowable range, the control unit 57 determines that the extraction time Ty matches the execution time Tx. Similarly, the control unit 57 calculates an angular deviation between the moving direction Dy of the extracted commodity information and the extraction direction Dx of the purchaser extraction information. If the angular deviation is within the allowable range, the control unit 57 determines that the moving direction Dy matches the extracting direction Dx.

In this way, if the purchaser retrieval information having the time condition and the direction condition matching each other is selected, the control unit 57 acquires the position information Q1 from the purchaser retrieval information as the ACT 505. Then, the control unit 57 searches the tracking data of each purchaser stored in the person tracking apparatus 20 as the ACT 506. The control section 57 specifies, as an ACT507, a purchaser who is at the place of the positional information Q1 at the execution time Ty of the purchaser retrieving information.

When the purchaser is specified, the control section 57 acquires the first ID of the purchaser from the person tracking apparatus 20 as an ACT 508. Further, the control unit 57 acquires the article ID and the number of purchases from the article information as the ACT 509. Then, the control section 57 controls the communication section 14 to perform a purchase notification to the sales management apparatus 60 as an ACT 510. By this control, the communication section 55 transmits a purchase notification command to the sales management apparatus 60 via the communication network 70. The purchase notification command includes the first ID, the product ID, and the purchase number. As described above, the control unit 57 ends the information processing in the sequence shown in the flowchart of fig. 16.

In addition, when the purchaser detecting unit 31 detects the position of the purchaser by combining the beacon terminal and the smartphone held by the purchaser, the purchaser cannot specify the position of the purchaser without holding the smartphone. At this time, the control unit 57 notifies the store management system, not shown, of information on the product shelf monitored by the product monitoring device 40 of the transmission source, and notifies an abnormality. By doing so, it is possible to easily confirm the abnormality, and the store clerk can support the purchaser of the commodity shelf.

The control unit 68 of the sales management apparatus 60 detects the reception of the notification command in ACT601 of fig. 17, determines that the notification command is not the trace notification command in ACT604, and proceeds to ACT605 when determining that the notification command is NO. The control section 68 confirms as an ACT605 whether or not its notification command is a purchase notification command. At this time, since the purchase notification command is received, the control unit 68 determines YES in ACT605 and proceeds to ACT641 of fig. 21.

The control unit 68 obtains the first ID, the product ID, and the purchase count from the purchase notification command as an ACT 641. Further, the control section 68 selects the purchase list 600 in which the first ID is registered as the ACT 642. The control unit 68 registers the product ID and the number of purchases in the purchase list 600 as ACT 643. As described above, the control unit 68 ends the reception process of the purchase notification command. In ACT643, the registered product ID and the purchase amount are transmitted to the action monitoring device 30 via the communication unit 66. When the purchaser detecting unit 31 is configured to detect the position of the purchaser by combining the beacon terminal and the smartphone held by the purchaser, the registered product ID and the purchase quantity (information on the registered product) are transmitted to the smartphone detected by the purchaser detecting unit 31 via the beacon terminal. That is, the transmission is performed to the smartphone of the purchaser who faces the displayed product.

The purchaser holding the smartphone can in turn confirm the purchase of the goods in the display of the smartphone. The smartphone is in a state in which the application program is activated.

On the other hand, the control section 68 of the sales management apparatus 60 detects that the notification command is received in ACT601 of fig. 17, determines that the notification command is not a purchase notification command in ACT605, and proceeds to ACT606 if it determines that the notification command is NO. The control section 68 confirms whether or not the notification command is a checkout notification command as ACT 606. Upon receiving the checkout notification command, the control unit 68 determines YES in ACT606 and proceeds to ACT651 in fig. 22.

As described above, the first ID is included in the settlement notification command output from the person tracking apparatus 20 to the sales management apparatus 60. The control unit 68 obtains the first ID from the checkout notification command as an ACT 651. Further, the control section 68 selects the purchase list 600, the first ID of which is registered, as the ACT 652. The control section 68 registers a member ID from its purchase list 600 as an ACT 653.

The control unit 68 searches the member information storage unit 62 for the member ID as an ACT 654. Then, the control unit 68 acquires information on the payment account stored in association with the member ID. The control unit 68 searches the product information stored in the product information storage unit 61 for the product ID registered in the purchase list 600 as an ACT655, and then obtains a price for each product ID. Then, the control unit 68 calculates the amount of payment from the price and the purchase amount.

The control unit 68 executes, as an ACT656, a settlement process for deducting a settlement amount from the payment account. In the checkout process, the control unit 68 may create an electronic ticket and download the electronic ticket to a smartphone of the purchaser.

When the payment processing is finished, the control unit 68 controls the communication unit 66 to notify the person tracking device 20 of the completion of payment as ACT 657. By this control, a checkout completion command is transmitted from the communication unit 66 to the person tracking apparatus 20 via the communication network 70. The checkout end command includes the first ID registered in the purchase list 600 on which the checkout process has been performed. As described above, the control unit 68 ends the reception process of the checkout notification command.

The explanation returns to fig. 13.

The control unit 26 of the person tracking device 20 that controls transmission of the settlement notification command in ACT208 waits for reception of the settlement end command as ACT 209. Upon detecting that the checkout end command is received via the communication unit 24, the control unit 26 determines YES in ACT209 and proceeds to ACT 210. The control unit 26 obtains the first ID from the checkout end command as the ACT 210. The control unit 26 ends the accumulation of the tracking data for the purchaser who has given the first ID and tracked as the ACT 211. As described above, the control unit 26 ends the information processing in the sequence shown in the flowchart of fig. 13. When the person tracking apparatus 20 detects the position of the purchaser by combining the beacon terminal and the smartphone held by the purchaser, the beacon terminal notifies the smartphone held by the purchaser of completion of payment. In this way, the completion of the checkout can be displayed on the display of the smartphone. The purchaser can easily grasp the checkout state.

As is clear from the above description, in the sales management system 100, the control unit 37 of the behavior monitoring device 30 constitutes a detection unit. That is, the control unit 37 controls the purchaser detecting unit 31 to execute the processes of the ACT301 and the ACT302 in fig. 14, thereby detecting a purchaser who faces a product being displayed using the purchaser detecting unit 31. The control unit 37 constitutes a first recognition unit. That is, the control unit 37 controls the action determination unit 32 and the takeout direction specification unit 33 to execute the processes of ACT303 to ACT310 in fig. 14, thereby recognizing the action of the commodity and the takeout direction of the commodity that the purchaser has taken out the commodity.

The control unit 47 of the product monitoring device 40 constitutes a second recognition unit. That is, the control unit 47 controls the movement monitoring unit 41, the product specifying unit 42, and the movement direction specifying unit 43 to execute the processes of ACT401 to ACT414 in fig. 15, thereby recognizing the movement of the displayed product and the movement direction of the product.

The control unit 57 of the merge recognition device 50 constitutes a specifying unit. That is, the control unit 57 controls the product monitoring acquisition unit 51, the action monitoring acquisition unit 52, the matching unit 53, and the purchase specifying unit 54 to execute the processes of ACT501 to ACT510 in fig. 16, thereby matching the direction in which the purchaser takes out the product with the direction in which the product moves. When the withdrawal direction matches the movement direction, the control unit 57 designates the commodity recognized as moving as the purchased commodity of the purchaser.

The control unit 68 of the sales management apparatus 60 constitutes a registration unit. That is, the control section 68 registers the purchased article of the specified purchaser in the purchase list 600 by controlling the purchase list updating section 64 to execute the processes of ACT641 to ACT643 of fig. 21.

Specifically, the control unit 37 of the behavior monitoring device 30 constitutes a first time point acquisition unit. That is, the control unit 37 acquires the execution time Tx as the first time at which the action of taking out the article is recognized by the first recognition unit by executing the process of ACT306 in fig. 14. The control unit 47 of the product monitoring device 40 constitutes a second time acquisition unit. That is, the control unit 37 acquires the pickup time Ty, which is the second time when the second recognition unit recognizes the movement of the product, by executing the process of ACT408 of fig. 15. Then, the control unit 57 of the integrated recognition device 50 functions as a specifying unit that specifies the purchased product of the purchaser when the moving direction of the product recognized to move at the pickup time Ty matches the pickup direction of the pickup action recognized at the execution time Tx within the range matching the pickup time Ty.

More specifically, the control unit 16 of the entrance door apparatus 10 constitutes an identification unit. That is, the control unit 16 controls the reception unit 11 and the opening/closing unit 12 to execute the processes of the ACTs 101 to 110 in fig. 12, and identifies the purchaser. The control unit 26 of the human tracking device 20 constitutes a tracking unit. That is, the control section 26 controls the detection section 21 and the tracking section 22, which are tracking detection sections, to execute the processes of the ACTs 201 to 210 of fig. 13, and tracks the purchaser. The control section 57 of the merge recognition device 50 constitutes a purchaser specification section by controlling the purchase specification section 54 to execute the processes of ACT505 to ACT 507. That is, the control unit 57 specifies the purchaser detected by the detection unit based on the result that the purchaser identified by the identification unit is tracked by the tracking unit. Then, the control unit 68 of the sales management apparatus 60 serves as a registration unit for registering the purchased product in the purchase list created for each of the purchasers specified by the purchaser specifying unit.

With the above configuration, for example, it is assumed that two purchasers CA1 and CA2 stand apart from each other on the left and right in the monitoring target area AR1 of the product 80 shown in fig. 6, and that the two purchasers CA1 and CA2 respectively extend their handles to the product AAA displayed in the center section of the second floor, but the purchaser CA1 standing on the left side toward the product shelf 80 has not taken the product AAA and has retracted its handle, and the purchaser CA2 standing on the right side has taken out the product AAA and retracted its handle. At this time, the movement monitoring device 30 recognizes the movement of taking out the product and the direction of taking out the product with respect to the purchaser CA1 and the purchaser CA 2. On the other hand, in the article monitoring device 40, the movement of the article AAA and the movement direction thereof are recognized. At this time, since the moving direction of the commodity AAA matches the withdrawal direction to the purchaser CA2, the purchaser CA2 is designated as the purchaser of the commodity AAA in the merge recognition device 50.

Further, suppose that another purchaser CA3 behind the purchaser CA2 takes out the commodity BBB from another section substantially simultaneously with the time when the purchaser CA2 takes out the commodity AAA. At this time, the movement monitoring device 30 recognizes the movement of taking out the product and the direction of taking out the product with respect to the purchaser CA2 and the purchaser CA 3. On the other hand, in the article monitoring device 40, the movement and the movement direction of the article AAA and the article BBB are recognized. At this time, since the moving direction of the commodity AAA matches the withdrawal direction to the purchaser CA2, the purchaser CA2 is designated as the purchaser of the commodity AAA in the merge recognition device 50. Further, since the moving direction of the commercial product BBB matches the direction of extraction from the purchaser CA3, the purchaser CA3 is designated as the purchaser of the commercial product BBB in the integrated identification device 50.

On the other hand, assume that after purchaser CA2 has taken out commodity AAA, purchaser CA3 has taken out commodity CCC from the same section as the commodity AAA. At this time, the moving direction of the commodity AAA and the commodity CCC substantially coincides with the picking direction to the purchaser CA2 and the purchaser CA 3. However, there is a time difference Δ Tx between the execution time Tx1 at which the purchaser CA2 recognizes the action of taking out the commodity AAA and the execution time Tx2 at which the purchaser CA3 recognizes the action of taking out the commodity CCC. Similarly, there is a time difference Δ Ty between the takeout time Ty1 at which the removal of the product AAA is recognized and the takeout time Ty2 at which the removal of the product CCC is recognized. Also, the execution timing Tx1 and the take-out timing Ty1 are considered to coincide, and the execution timing Tx2 and the take-out timing Ty2 are considered to coincide. Therefore, in the integrated identifying apparatus 50, the purchaser CA2 is specified as the purchaser of the commodity AAA. Further, in the incorporated recognition device 50, the purchaser CA3 is specified as the purchaser of the commercial product CCC.

In this way, the sales management system 100 can accurately specify a customer who has taken a product.

In addition, when the weight sensor 84 is used as one of the sensors of the movement monitoring unit 41 and the product specifying unit 42 in the product monitoring device 40, not only the type of the product taken out from the product shelf 80 but also the number of products can be specified. Therefore, in the sales management apparatus 60, the article ID as the article code of the article designated as the purchase article can be registered, and the purchase amount can also be registered.

In addition, a weight sensor may not necessarily be used. For example, the number of products taken out from the product shelf 80 may be determined by analyzing an image captured by a camera sensor. Further, by limiting the number of products taken out of the product shelf 80 by the purchaser to one, it is not necessary to find the number of products.

In the ACT504 of fig. 16, the control unit 57 of the merge recognition device 50 specifies the purchased product of the purchaser when the moving direction of the product recognized to move at the pickup time Ty matches the pickup direction of the pickup action recognized at the execution time Tx within the range matching the pickup time Ty. Here, the action determining unit 32 and the extraction direction specifying unit 33 of the action monitoring device 30 determine the action of the purchaser, and delay occurs compared to the actual action. Similarly, the movement monitoring unit 41, the article specifying unit 42, and the movement direction specifying unit 43 of the article monitoring device 40 determine the movement of the article, and delay occurs compared to the actual movement. The control unit 57 of the merge recognition device 50 determines whether or not the two match in consideration of these delays, and thus a more reliable result can be obtained.

Although the embodiments of the sales management system and the sales management method that can accurately specify the customer who has taken the product have been described above, the embodiments are not limited to these.

For example, in the above-described embodiment, the case where the settlement processing is performed by deducting the settlement amount from the payment account is exemplified. The checkout process is not limited to these. For example, the settlement processing may be performed by known credit card settlement, code settlement using a barcode, a two-dimensional code, or the like.

In the above embodiment, the case where the purchaser CA2 takes out the article from the article shelf 80 is exemplified. In this regard, the movement monitoring unit 41, the product specifying unit 42, and the movement direction specifying unit 43 of the product monitoring device 40 may detect an action in which the purchaser returns the product temporarily taken out to the product shelf 80. Therefore, when the product is returned to the product shelf 80, the product monitoring acquisition unit 51, the action monitoring acquisition unit 52, the matching unit 53, and the purchase designation unit 54 of the integrated recognition device 50 designate the product and the purchaser, and notify them to the sales management device 60. Upon receiving the notification, the control unit 68 of the sales management apparatus 60 deletes the data of the product from the purchase list 600 of the purchaser. By doing so, it is also possible to cope with return of goods before the payment by the purchaser.

Furthermore, the various embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments may be embodied in other various forms, and various omissions, substitutions, and changes may be made without departing from the spirit of the invention. These embodiments and modifications are included in the scope of the invention, and are included in the invention described in the scope of claims and the equivalent scope thereof.

Claims

1. A sales management system, comprising:

a detection unit for detecting a purchaser who faces a displayed commodity;

a first recognition unit that recognizes the action of the purchaser to take out the opposing commodity and the direction of taking out the opposing commodity detected by the detection unit;

a second recognition unit that recognizes the movement of the displayed commodity and the movement direction of the commodity;

a specifying unit configured to specify a commodity purchased by the purchaser who is the action of identifying the commodity taken out by the first identifying unit by the purchaser and the direction of taking out the commodity, and to specify the commodity purchased by the purchaser who is the action of identifying the commodity taken out by the first identifying unit when the direction of taking out coincides with the direction of movement when the action of identifying the commodity taken out by the purchaser and the direction of movement of the commodity are identified by the second identifying unit; and

and a registration unit that registers the purchased product of the purchaser specified by the specification unit.

2. The sales management system according to claim 1, further comprising:

a first time acquisition unit that acquires a first time at which the action of taking out the product is recognized by the first recognition unit; and

a second time acquisition unit that acquires a second time at which the movement of the commodity is recognized by the second recognition unit,

wherein the specifying unit specifies the commodity purchased by the purchaser who is identified as the movement of the commodity by the first identifying unit when the movement direction of the commodity identified as moving at the second time coincides with the takeout direction of the takeout action identified at the first time within a range coinciding with the second time.

3. The sales management system according to claim 1 or 2, further comprising:

an identification unit that identifies a purchaser;

a tracking unit that tracks the purchaser; and

a purchaser specifying unit that specifies the purchaser detected by the detecting unit based on a result of tracking the purchaser identified by the identifying unit by the tracking unit,

wherein the registration section registers the purchased product for each purchaser specified by the purchaser specifying section.

4. The sales management system according to claim 1, further comprising:

a measuring unit for measuring the weight of the displayed commodity,

wherein the second recognition unit recognizes the integer as the number of moved commodities when the weight measured by the measuring unit is reduced by an integer multiple of the unit weight of the commodities displayed.

5. The sales management system according to claim 1, further comprising:

and a unit for transmitting information on the registered product to an information terminal held by the purchaser.

6. The sales management system according to claim 1,

the detection unit controls a purchaser detection unit that detects a position of a purchaser by combining a beacon terminal and an information terminal owned by the purchaser.

7. The sales management system according to claim 1, further comprising:

and a unit for transmitting the completion of payment of the registered commodity to the information terminal owned by the purchaser.

8. The sales management system according to claim 1,

when the action of the purchaser to take out the product and the direction of taking out the product are recognized by the first recognition unit and the movement of the product and the direction of movement thereof are recognized by the second recognition unit, an abnormality is notified when the purchaser detection unit cannot specify the position of the purchaser.

9. The sales management system according to claim 3,

the detection unit controls a purchaser detection unit, the tracking unit controls a tracking detection unit that detects a position of a purchaser, and the purchaser detection unit and the tracking unit are shared.

10. A sales management method for managing sales of displayed commodities, comprising the steps of:

a detection unit for detecting a purchaser who faces the displayed commodity;

a first recognition unit recognizing the detected action of the purchaser to take out the opposing product and a direction of taking out the opposing product;

a second recognition unit recognizing the movement of the displayed commodity and the movement direction of the commodity;

a specifying unit that specifies a commodity, the commodity having been recognized as having moved when the direction of removal matches the direction of movement, as a purchased commodity of a purchaser who recognized the action of removing the commodity; and

the registration unit registers the purchased product of the specified purchaser.