TWI675337B - Unmanned goods management system and unmanned goods management method - Google Patents

Abstract

The invention provides an unmanned goods management system and an unmanned goods management method, which can automatically manage the goods in the tracking area. The method is to capture the tracking image of the tracking area, obtain the identity data of the person entering the tracking area, generate the person's posture model based on the tracking image, identify the position of each product in the product image, and judge any position based on the posture model and the position of each product. When a product is picked up by a person, the product information of the product is obtained, and the identity information and the product information are linked. The invention can effectively realize unmanned goods management, and can effectively save manpower, thereby reducing management costs.

Description

Unmanned goods management system and unmanned goods management method

The present invention relates to the management of goods, and particularly relates to an unmanned goods management system and an unmanned goods management method.

In the current unmanned store, the management system cannot actively determine which products the consumer has taken, and cannot actively checkout. This makes it necessary for the consumer to go to the checkout counter for self-checkout after taking the goods, resulting in users Poor experience.

In view of this, a management system that can proactively determine which products a consumer has taken is currently being proposed.

The invention provides an unmanned goods management system and an unmanned goods management method, which can proactively determine whether a person has taken any one of the goods and automatically establish a connection between the taken goods and the person who takes the goods when the personnel take the goods.

In one embodiment, an unmanned goods management method is used to manage multiple goods in the tracking area, including the following steps: shooting the tracking area through a tracking camera to obtain a tracking image; obtaining identity information of a person entering the tracking area ; Generate a person's pose model based on the tracking image; identify the goods position of each item in the goods image; judge any one based on the pose model and each item's position Obtain the goods information of the retrieved goods when the goods are taken by the personnel; and, link the identity information and the goods information.

In one embodiment, an unmanned goods management system is used to manage multiple goods in a tracking area, including a tracking camera, a database, and a control device. The tracking camera is used to capture a tracking area to obtain a tracking image. The database is used to store multiple goods data corresponding to multiple goods and multiple identity data corresponding to multiple persons. The control device is connected to a tracking camera and a database, and includes an identity recognition module, an attitude tracking module, a goods identification module, a retrieval analysis module and a processing module. The identification module is used to obtain the identity information of the person entering the tracking area. The attitude tracking module is used to generate a person's attitude model based on the tracking image. The goods identification module is used to identify the goods position of each goods in the goods image, and is used to obtain the goods data of the retrieved goods. The taking analysis module is used to judge whether any item is taken by the personnel according to the attitude model and the position of each item. The processing module is used to link the obtained identity data and the goods data of the retrieved goods.

The invention can effectively realize unmanned goods management, and can effectively save manpower, thereby reducing management costs.

1‧‧‧Unmanned Goods Management System

10‧‧‧Control device

100‧‧‧Processing Module

101‧‧‧Identification Module

102‧‧‧Attitude tracking module

103‧‧‧Face Tracking Module

104‧‧‧Hand Tracking Module

105‧‧‧ Take the analysis module

106‧‧‧ Goods Identification Module

107‧‧‧ Identity Confirmation Module

108‧‧‧Settlement module

11, 41-44‧‧‧ Goods camera

12, 40‧‧‧ tracking camera

13‧‧‧Identification device

14‧‧‧Storage device

140‧‧‧Computer Program

141, 210‧‧‧Database

15‧‧‧Communication device

20‧‧‧Internet

21‧‧‧Host

30-32‧‧‧Shelf

34‧‧‧Tracking Area

50-58‧‧‧ Goods

530-580 ‧‧‧ pattern label

6‧‧‧ personnel

60‧‧‧ Attitude Model

S10-S16‧‧‧The first management step

S20-S24‧‧‧Replacement steps

S30-S32‧‧‧Settlement steps

S400-S412‧‧‧Second Management Step

FIG. 1 is a structural diagram of an unmanned goods management system according to an embodiment of the present invention.

FIG. 2 is a structural diagram of a control device according to an embodiment of the present invention.

FIG. 3 is a flowchart of a method for managing unmanned goods according to the first embodiment of the present invention.

FIG. 4 is a partial flowchart of a method for managing unmanned goods according to a second embodiment of the present invention.

FIG. 5 is a partial flowchart of a method for managing unmanned goods according to a third embodiment of the present invention.

FIG. 6 is a flowchart of a method for managing unmanned goods according to a fourth embodiment of the present invention.

FIG. 7 is a first schematic diagram of unmanned goods management of the present invention.

FIG. 8 is a second schematic diagram of the unmanned goods management of the present invention.

FIG. 9 is a third schematic diagram of the unmanned goods management of the present invention.

FIG. 10 is a fourth schematic diagram of the unmanned goods management of the present invention.

FIG. 11 is a fifth schematic diagram of the unmanned goods management of the present invention.

FIG. 12 is a sixth schematic diagram of the unmanned goods management of the present invention.

A preferred embodiment of the present invention is described in detail below with reference to the drawings.

The invention mainly provides an unmanned goods management technology, which can manage multiple goods in the tracking area. The unmanned goods management technology of the present invention is mainly to automatically obtain the identity data of the personnel entering the tracking area, automatically detect whether the personnel have taken the goods, and detect the goods and the taken goods when the personnel are taken to the goods. Associates.

Furthermore, the unmanned goods management technology of the present invention can be applied to applications such as unmanned stores, self-rental stores, or unmanned storage.

Please refer to FIG. 1, which is a structural diagram of an unmanned goods management system according to an embodiment of the present invention. The unmanned goods management system (hereinafter referred to as the management system) 1 of the present invention mainly includes one or more tracking cameras 12, a storage device 14, and a control device 10 connected to the above devices.

The tracking camera 12 is used to capture a tracking area to obtain a tracking image. In an embodiment, the tracking camera 12 may include a color tracking camera (such as an RGB camera) and / or a depth tracking camera (such as a combination of an infrared transmitter and an infrared camera or a combination of an ultrasonic transmitter and an ultrasonic receiver). Hop). The color tracking camera is used to obtain a color tracking image of a tracking area, and the depth tracking camera is used to obtain a depth tracking image of a tracking area.

Furthermore, when the tracking camera 12 includes both a color tracking camera and a depth tracking camera, the function of the 3D camera can be realized. Specifically, the color tracking image and depth tracking image captured by the tracking camera 12 can be used to analyze the 3D spatial information of the tracking area (such as the number and size of objects in the area), and can accurately perform various detection processes (such as Detect if a person has entered or left the tracking area, or identify the current posture of the person).

The storage device 14 (such as a random access memory, a flash memory, a disk drive, a cache memory, or any combination of the above storage devices) is used to store data. The control device 10 (such as a processor or a microcontroller) is used to control the devices of the management system 1.

In an embodiment, the storage device 14 includes a local database 141, which is used to store data (such as identity data and product data described later) needed to implement unmanned goods management.

In one embodiment, the data needed to achieve unmanned goods management is stored in a network database. Specifically, the management system 1 includes a communication device 15 (such as a network card) connected to the control device 10. The communication device 15 can be connected to a network 20 (such as the Internet), and connected to a remote host 21 via the network 20. Database 210. The following will take the data required to realize the management of unmanned goods is stored in the local database 141 as an example for illustration.

In one embodiment, the management system 1 may include one or more product cameras 11 connected to the control device 10. The goods camera 11 is used for shooting the area where the goods are located in the tracking area to obtain goods images. The structure of the goods camera 11 is the same as or similar to that of the tracking camera 12 and will not be repeated here.

In one embodiment, the administrator of the tracking area can set a plurality of goods areas in which the goods are placed, and set up one or more goods cameras 11 in each goods area to obtain a clear picture of the goods in each area. For more accurate detection processing (more details later).

In one embodiment, the management system 1 may include an identification device 13 connected to the control device 10. The identification device 13 is used to obtain characteristic data of a person entering the tracking area.

For example, the identification device 13 may be an RFID card reader and disposed at the entrance of the tracking area. When a person wants to enter the tracking area, he must first sense the RFID tag (holding tag) with the identification device 13 to enter the identification code of the person stored in the RFID tag (that is, the aforementioned characteristic data).

In another example, the identification device 13 may be a biometric identification device (such as a fingerprint scanner, an iris scanner, or a vein scanner), and is disposed at the entrance of the tracking area. When a person wants to enter the tracking area, he must first input the biological data (that is, the aforementioned characteristic data, such as fingerprint characteristic data, iris characteristic data, or vein distribution characteristic data) to the identification device 13.

In an embodiment, the storage device 14 includes a non-transitory computer-readable medium, and stores a computer program 140, and the computer program 140 records computer-readable code. The control device 10 can execute a computer program 140 to implement the unmanned goods management method according to the embodiments of the present invention.

Please refer to FIG. 2 together, which is a structural diagram of a control device according to an embodiment of the present invention. Specifically, the control device 10 mainly interacts with other devices by executing a computer program 140 to perform various functions of the unmanned product management method of the present invention. In addition, the computer program 140 includes multiple sets of codes corresponding to the following functional modules, and the control device 10 can implement the following functional modules after executing the corresponding sets of codes: processing module 100: used for the goods to be taken by personnel When retrieved, the identity information of the personnel and the goods information of the retrieved goods are linked, and when the goods are returned by the personnel, the link between the identity information of the personnel and the goods information of the returned goods is released.

In one embodiment, the processing module 100 can determine whether a person is approaching the product, and control the product camera 11 to start shooting the area where the product is located when the person approaches the product.

Identity identification module 101: used to obtain identity data of persons entering the tracking area. In an embodiment, the identity recognition module 101 obtains the characteristic data from the recognition device 13 and checks it from the database 141 Ask for identity information that corresponds to this characteristic. In an embodiment, the identity recognition module 101 performs face recognition on the color tracking image captured by the tracking camera 12 to obtain facial features (ie, feature data) of a person entering the tracking area, and based on the feature data in a database 141 Query the corresponding identity information.

Attitude tracking module 102: used to generate a person's attitude model based on the tracking image captured by the tracking camera 12. In an embodiment, the attitude tracking module 102 determines all or part of the joint position of the person according to the color tracking image and the depth tracking image, and generates the aforementioned attitude model according to the determined joint position.

The face tracking module 103 is configured to track a person's face based on a color tracking image captured by the tracking camera 12.

The hand tracking module 104 is used to generate a person's hand posture model based on the color goods image and the depth goods image captured by the goods camera 11. In one embodiment, the hand tracking module 104 determines a plurality of joint positions of a person's hand according to the color goods image and a depth goods image, and generates a hand posture model according to the plurality of joint positions.

Handling analysis module 105: It is used to judge whether any goods are taken by personnel according to the attitude model (and / or the hand attitude model) and the positions of the goods.

Commodity identification module 106: used to identify the position of each product in the image of the product, and used to obtain the product data of the retrieved (or returned) product.

In one embodiment, the goods identification module 106 identifies the aforementioned goods image in the tracking image, or obtains the aforementioned goods image through the goods camera 11.

In one embodiment, the product identification module 106 identifies the pattern labels of each item in the color image of the product, and determines the corresponding position of the item according to the position of each pattern label.

In one embodiment, the goods identification module 106 queries the database 141 for corresponding goods data according to the appearance (such as the shape or color of the pattern) of the pattern label of the retrieved goods.

Identity verification module 107: Perform face recognition on clearer color goods images to confirm the identity information of the person who took the goods.

The settlement module 108 is configured to perform settlement processing on all the goods data linked to the identity data when the preset settlement conditions are met.

Through the above-mentioned function module, the present invention can effectively identify the person who enters the tracking area, accurately identify the goods taken / replaced by this person, and record it.

Please refer to FIG. 3 together for a flowchart of a method for managing unmanned goods according to the first embodiment of the present invention. The unmanned goods management method (hereinafter referred to as the management method) of the embodiments of the present invention can be implemented by the management system 1 shown in FIGS. 1 to 2. The management method of this embodiment includes the following steps.

Step S10: The control device 10 controls the tracking camera 12 to capture a tracking area (such as the tracking area 34 in FIGS. 7 to 12) to obtain a tracking image.

In an embodiment, the tracking camera 12 is a 3D camera (ie, includes a tracking color camera and a tracking depth camera), and can capture color tracking images and depth tracking images in a tracking area.

In an embodiment, the control device 10 controls the tracking camera 12 to start shooting and tracking when the self-identification device 13 receives the characteristic data of the person or detects that the person enters the tracking area through a person sensor (such as a PIR sensor) Area.

In one embodiment, the tracking camera 12 continuously captures the tracking area, performs human detection on the tracking image, and starts recording the tracking image when it is detected that the person enters the tracking area.

Step S11: The control device 10 obtains the identity data of the person entering the tracking area via the identity recognition module 101. Specifically, the administrator can profile all persons allowed to enter the tracking area in advance, and store the identity information of each person in the database 141, so that when identifying any person entering the tracking area, the corresponding identity data can be obtained.

In an embodiment, the database 141 may further store characteristic data corresponding to each identity data. The control device 10 obtains the characteristic data of the person entering (or intending to enter) the tracking area through the identification device 13 and queries the corresponding identity data in the database 141 according to the characteristic data.

Taking the identification device 13 as an RFID card reader as an example, before entering the tracking area, a person may use an RFID tag stored with its identification code (ie, characteristic data) to sense the identification device 13 to enter the identification code. Then, the control device 10 can query the identity information of the person in the database 141 according to the identification code, and can complete the identification of the person.

Taking the identification device 13 as a biometric identification device (such as a fingerprint scanner) as an example, a person may use the identification device 13 to enter fingerprint data before entering the tracking area. Then, the control device 10 can analyze the fingerprint data to obtain fingerprint characteristics (ie, characteristic data), and query the identity information of the person in the database 141 according to the fingerprint characteristics, so as to complete the identification of the person.

In an embodiment, the control device 10 analyzes the tracking images captured by the tracking camera 12 to obtain the image characteristics (ie, characteristic data) of the person entering the tracking area, and queries the corresponding identity in the database 141 according to the image characteristics. data.

For example, tracking images may include tracking color images. The control device 10 performs face recognition on the color tracking image to obtain the facial features (ie, feature data) of the person entering the tracking area, and queries the corresponding identity data in the database 141 according to the feature data.

In one embodiment, the management system 1 further includes a human-machine interface (such as a touch screen, a display, or a speaker) connected to the control device 10 and an access control device (not shown in the figure). The control device 10 can unlock (or open) the access control device to allow the person to enter the tracking area when obtaining the identity information of the person, and lock (or close) the access control device to prevent the person from entering the tracking area when the identity information of the person cannot be obtained, and Further notifications of unregistered personnel can be issued via the man-machine interface.

In addition, the control device 10 may further interact with the personnel through the human-machine interface to guide the personnel to register the identity data. With this, unregistered personnel can successfully enter the tracking area after completing registration.

Step S12: The control device 10 generates a pose model of the person according to the tracking image via the pose tracking module 102 (as shown in the pose model 60 of FIG. 8 to FIG. 12).

In an embodiment, the tracking image includes a color tracking image and a depth tracking image. The control device 10 can identify a person's image in the color tracking image and the depth tracking image, identify the current posture of the person, and construct a corresponding posture model. The captured color tracking image and / or depth tracking image continuously update the constructed attitude model. Thereby, the control device 10 can know the current position of the person (such as whether to approach or move away from the product) and the action (such as whether to pick up / down the product) through the attitude model.

In one embodiment, the control device 10 determines a plurality of joint positions of the person according to the color tracking image and the depth tracking image, and constructs or updates the posture model according to the plurality of joint positions.

In one embodiment, the control device 10 does not construct a full-body posture model of a person, but only a partial posture model (for example, only a hand posture model). Therefore, in this embodiment, according to the partial attitude model, it is still possible to judge whether a person picks up / downs the goods, and can effectively reduce the calculation amount.

Step S13: The control device 10 obtains the goods image, and identifies the goods position of each goods placed in the tracking area in the goods image through the goods identification module 106.

In an embodiment, the control device 10 analyzes and cuts the tracking image to obtain a product image including the product.

In an embodiment, the control device 10 controls the goods camera 11 to capture images to obtain goods images.

In an embodiment, the database 141 may further store product data corresponding to each product. The control device 10 can further identify each item in the item image, and obtain item data of the identified item.

In one embodiment, each product includes a pattern label (the pattern of the pattern label may be a bar code, a specific shape and / or a color) different from each other. The database 141 may further store label data (such as the barcode number of the product or the shape and / or color of the graphic label) indicated by the graphic label of each product data.

For example, the aforementioned graphic label can be pasted or printed on each product. The control device 10 can identify the image of each pattern label in the product image, and analyze the image of each pattern label to obtain label data (that is, the barcode number of the product or the shape and / or color of the pattern label), and then obtain it according to the label data. Corresponding product information. In addition, the control device 10 may determine the position of the goods according to the position of each pattern label.

Accordingly, the present invention can obtain the product position and product data of each product in the tracking area through image processing.

Step S14: The control device 10 determines whether any one of the goods has been taken by the personnel according to the posture model and the goods position of each goods via the taking analysis module 105.

In one embodiment, the control device 10 is based on the image of the product and the posture model's hand touching or continuously touching any product (for example, the position of the hand of the posture model continuously overlaps with the position of any product). , Judges take this product.

In one embodiment, the control device 10 determines whether the position of any item of the product moves with the posture model and / or the distance between the position of the product and the posture model (hand) is less than a preset picking distance (such as 30 (Cm), the judge will take this product.

If the control device 10 determines that any of the goods has been taken by a person, step S15 is performed. Otherwise, the control device 10 executes step S14 again.

Step S15: The control device 10 first obtains the goods data of the retrieved goods through the goods identification module 106, and then links the identity information of the person who retrieves the goods and the goods data of the retrieved goods through the processing module 100.

Taking an unmanned store as an example, the control device 10 may create a shopping cart list corresponding to the identity data in the storage device 14 and add the product data to the shopping cart list to establish a link between the identity data and the goods data.

Step S16: The control device 10 determines whether the person has finished taking the goods.

In one embodiment, the control device 10 determines that the person has finished taking the goods when the person leaves the tracking area based on the tracking image, and ends the management method.

In an embodiment, the control device 10 determines that the person has finished taking the product when the person judges that both hands take the product according to the attitude model and the position of the product, and ends the management method.

In an embodiment, the control device 10 determines that the person has finished taking the goods when the person has not taken the new goods for a preset time (e.g., 5 minutes) based on the attitude model and the goods position. And end the management method.

In one embodiment, the control device 10 determines that the person has finished taking the goods when the characteristic data of the person is received from the identification device 13 (for example, the person uses RFID tags for sensing again, or enters the biological data again), and ends the management. method.

In an embodiment, the control device 10 determines that the person has finished taking the goods when receiving the settlement operation of the person via the human-machine interface, and ends the management method.

By constructing and updating the posture model of each person, the present invention can accurately determine whether the person takes the goods. In addition, compared with using RFID technology to identify the goods being retrieved (such as attaching expensive RFID tags to each goods, and using an RFID card reader to sense the RFID tags of the goods being retrieved), the present invention uses the Cheap pattern tags and identification of goods based on image recognition technology can greatly reduce system construction costs (ie, no expensive RFID tags and RFID card readers are needed), and effectively implement unmanned goods management in the tracking area, which can effectively Save manpower and reduce management costs.

Please refer to FIG. 3 and FIG. 4 together. FIG. 4 is a partial flowchart of the unmanned goods management method according to the second embodiment of the present invention. In this embodiment, a product return detection function is further provided, which can be automatically Detect if personnel have returned the goods. Compared with the management method shown in FIG. 3, the management method of this embodiment includes the following steps after step S15.

Step S20: The control device 10 obtains the image of the product, and identifies the position of each product obtained by a person in the image of the product through the product identification module 106. The identification method used in step S20 is the same as or similar to step S13, and details are not described herein again.

Step S21: The control device 10 determines whether the person puts back any of the retrieved goods according to the posture model and the goods position of the goods that have been taken by the personnel via the taking analysis module 105.

In an embodiment, the control device 10 recognizes that a person's hand is separated from any product based on the image of the product and the posture model (such as the position of the hand of the posture model and the position of any product from an overlapping state to a non-overlapping state). ), The judge returns the item.

In an embodiment, the control device 10 determines whether the position of any item of the product is no longer moving with the posture model and / or the distance between the position of the product and the posture model (hand) is not less than a preset picking distance At that time, the judge returned the goods.

If the control device 10 judges that any product is returned by a person, step S22 is performed. Otherwise, the control device 10 executes step S13 again.

Step S22: The control device 10 first obtains the goods data of the returned goods through the goods identification module 106, and then releases the identity data of the person who placed the goods back through the processing module 100 and the goods data of the returned goods. link.

Taking an unmanned store as an example, the control device 10 may remove the goods data from the shopping cart list to release the link between the identity data and the goods data.

Step S23: The control device 10 determines whether the preset settlement conditions are met via the settlement module 108.

In one embodiment, the aforementioned settlement conditions are detected when a person leaves the tracking area, a preset time elapses, the characteristic data of the person is received from the identification device 13 again, and / or a settlement operation of the person is received via a human-machine interface.

If the control device 10 determines that the preset settlement conditions are met, step S24 is executed. Otherwise, the control device 10 executes step S13 again.

Step S24: The control device 10 executes settlement processing on all the goods data linked to the identity data.

Taking an unmanned store as an example, the control device 10 may perform a settlement process to check out all the goods data in the shopping cart list.

Taking unmanned storage as an example, the control device 10 can perform settlement processing to bind all the linked product data to this identity data (such as setting the ownership or holder of the goods data as this identity data).

With this, the present invention can effectively implement the goods return detection function and automatically perform settlement processing in a timely manner.

Continue to refer to FIG. 3, FIG. 4 and FIG. 5 together. FIG. 5 is a partial flowchart of the unmanned goods management method according to the third embodiment of the present invention. The management method of this embodiment is used in an unmanned store, and the checkout can be performed automatically when the settlement conditions are met. Specifically, compared to the management method shown in FIG. 3 and FIG. 4, step S24 of the management method of this embodiment further includes the following steps for performing settlement processing.

Step S30: The control device 10 obtains payment data corresponding to the identity data through the settlement module 108.

In an embodiment, the database 141 further stores a plurality of payment data (such as credit card data, stored value card data, or account data) corresponding to a plurality of identity data. The control device 10 obtains corresponding payment data according to the identity data of a person who meets the settlement conditions.

Step S31: The control device 10 calculates the total amount of the goods via the settlement module 108.

In an embodiment, the database 141 further stores price data of each product data. The control device 10 obtains price data of each product data linked to the identity data, and calculates the total amount of the product based on the obtained product price.

Step S32: The control device 10 performs the settlement according to the payment information and the total amount of the goods via the settlement module 108.

For example, the control device 10 may request payment from the credit card company corresponding to the credit card information according to the calculated total amount of goods, and deduct the total amount of goods from the balance of the stored value card information or the total amount of goods from the account information without limitation.

Thus, the present invention can be effectively used in an unmanned store and can automatically perform checkout.

Please refer to FIG. 6 together for a flowchart of a method for managing unmanned goods according to a fourth embodiment of the present invention. In this embodiment, the pattern labels of each product have the appearance of different shapes and / or colors, respectively (such as the pattern labels 530-550 shown in FIG. 7).

Moreover, in this embodiment, the management system 1 further includes one or more commodity cameras 11 (such as the commodity cameras 41 to 44 shown in FIG. 7). Each of the commodity cameras 11 is respectively located adjacent to the commodity and can shoot more. Clear image of goods.

Moreover, in this embodiment, the database 141 further records the correspondence between the product data of each product and the appearance of the pattern label. For example, the product data of the first product may correspond to a red triangle, the product data of the second product may correspond to a green triangle, the product data of the third product may correspond to a red rectangle, and so on. The management method of this embodiment includes the following steps.

Step S400: The control device 10 controls the tracking camera 12 to capture a tracking area to obtain a tracking image.

Step S401: The control device 10 obtains the identity data of the person entering the tracking area via the identity recognition module 101.

In this embodiment, the database 141 stores characteristic data corresponding to each identity data, and the tracking camera 12 is a 3D camera and can capture color tracking images and depth tracking images. The control device 10 performs face recognition on the color tracking image to obtain facial features (ie, feature data) of a person entering the tracking area, and queries corresponding identity data in the database 141 according to the feature data. Then, the control device 10 can execute steps S402 and S403 simultaneously (or alternately).

Step S402: The control device 10 generates a person's attitude model based on the continuously tracked and measured multiple color tracking images and multiple depth tracking images via the attitude tracking module 102, and continuously updates the attitude model.

Step S403: The control device 10 continuously recognizes a person's face image based on the continuously captured multiple color tracking images via the face tracking module 103 to continuously track the person's face position.

Step S404: The control device 10 determines whether the person approaches the goods in the tracking area according to the position of the pose model or the position of the face via the processing module 100.

Specifically, the goods position 4 in the tracking area of the goods (or the shelves on which the goods are placed) may be stored in advance in the storage device 14, and the control device 10 determines when the position of the posture model or the position of the face is close to the aforementioned goods position. Personnel approach the goods.

If the control device 10 determines that the person is approaching the goods, step S405 is performed. Otherwise, the control device 10 executes steps S402 and S403 again.

Step S405: The control device 10 controls the goods camera 11 to start shooting the area where the goods are located to continuously obtain the goods images.

In an embodiment, the product camera 11 is a 3D camera (that is, including a color product camera and a depth product camera), and can capture color product images and depth product images, but it is not limited thereto.

In one embodiment, the commodity camera 11 may include only one of a color commodity camera and a depth commodity camera.

Then, the control device 10 can execute steps S406, S407, and S408 simultaneously (or alternately).

Step S406: The control device 10 generates a person's hand posture model based on the color product image and the depth product image via the hand tracking module 104, and continuously updates the hand posture model based on the color product image and the depth product image.

The aforementioned generation method of the hand pose model is the same as or similar to the aforementioned generation of the pose model, and details thereof will not be repeated here.

Step S407: The control device 10 performs face recognition on the color goods image through the identity confirmation module 107 to obtain facial features (ie, feature data) of the person captured by the goods camera 11, and queries the database 141 based on the feature data Corresponding identity information.

In one embodiment, the control device 10 compares the facial features of the person currently or previously captured by the tracking camera 12 with the facial features of the person currently captured by the commodity camera 11 and determines whether the two facial features belong to the same When the persons (such as the facial features are the same or very similar), the identity information of the person photographed by the tracking camera 12 is directly used as the identity information of the person photographed by the goods camera 11. In this way, the present invention does not need to query the identity database 141 again to confirm the identity information of the personnel, and can reduce the load on the database 141.

Step S408: The control device 10 continuously recognizes the appearance of the pattern label of each item in the continuously captured multiple color product images through the product identification module 106, and continuously determines each item according to the position of each pattern label in the color product image Product location.

It is worth mentioning that when using a bar code as the pattern of the pattern label, because the bar code has a high degree of precision and the recognition result of the bar code is easily affected by light and shadow, the product image must have excellent image quality (such as extremely high resolution and minimal noise). (Information) can successfully identify the tag data embedded in the barcode and identify the goods, which will greatly increase the cost of the goods camera 11.

To solve the above problems, the present invention proposes to use different shapes, colors, or a combination thereof as the pattern of the pattern label. Compared with bar codes, pattern labels with shapes and colors as patterns have lower precision and are less susceptible to light and shadow, can have a higher recognition success rate, and can be applied to lower-level commodity cameras 11.

In an embodiment, the control device 10 continuously recognizes the shape and / or color (ie, appearance) of each pattern label captured, and further queries the corresponding product data in the database 141 according to the appearance of the pattern label, but does not Defined by this.

In an embodiment, the control device 10 only queries the corresponding product data when any of the photographed goods is picked up or returned by the personnel. Accordingly, the present invention can reduce the load of the database 141. Then step S409 is performed.

Step S409: The control device 10 determines whether any of the products has been taken by the personnel according to the hand posture model and the position of each product through the fetch analysis module 105. The foregoing determination of whether to take the goods is as described above, and is not repeated here.

If the control device 10 determines that any of the goods has been taken by a person, step S410 is performed. Otherwise, the control device 10 executes steps S402 and S403 again.

Step S410: The control device 10 queries the database 141 via the product identification module 106 according to the appearance of the pattern label of the retrieved product (that is, the shape, color, or combination of the pattern label, the aforementioned appearance can be identified in step S408). Corresponding product information.

In an embodiment, the control device 10 also confirms the identity information of the person who took the goods through the identity confirmation module 107 (the aforementioned identity information can be identified in step S407), but it is not limited thereto.

In an embodiment, when there is only a single person in the tracking area, the control device 10 may not perform step S407, but directly set the identity data obtained in step S401 as the identity data of the person who took the goods.

Step S411: The control device 10 links the confirmed identity data and the goods data of the retrieved goods (for example, the goods data is added to the shopping cart list corresponding to the identity data) via the processing module 100.

Step S412: The control device 10 determines whether the person leaves the tracking area according to the position of each person's posture model and / or the position of the facial image.

If the control device 10 determines whether a person has left the tracking area, the management method ends. Otherwise, the control device 10 executes steps S402 and S403 again.

In an embodiment, the control device 10 can automatically perform settlement processing on all the goods information of the connected identity data after the judges leave the tracking area (step S24 shown in FIG. 4 or steps S30-S32 shown in FIG. 5) .

In this way, the present invention can further improve the success rate of judging the taking of goods, and can more effectively realize the management of unmanned goods.

Please refer to FIGS. 7 to 12 together. FIG. 7 is a first schematic diagram of unmanned goods management of the present invention, FIG. 8 is a second schematic diagram of unmanned goods management of the present invention, and FIG. 9 is unmanned goods of the present invention. A third schematic diagram of management, FIG. 10 is a fourth schematic diagram of unmanned goods management of the present invention, FIG. 11 is a fifth schematic diagram of unmanned goods management of the present invention, and FIG. 12 is a sixth schematic diagram of unmanned goods management of the present invention. . 7 to 12 are used to exemplify one of the embodiments of the unmanned goods management of the present invention.

As shown in FIG. 7, in this example, the tracking camera 40 (a 3D camera in this example) captures the entire tracking area. There are three shelves in the tracking area 34. The goods 50-52 are placed on the first-level shelf 30, and a goods camera 41 photographs the first-level shelf 30 from above. The goods 53-55 are placed on the second shelf 31, and a goods camera 42 photographs the second shelf 31 from the side. The goods 56-58 are placed on the third shelf 32, and two goods cameras 43 take pictures of the third shelf 32 from the side. Photo. Moreover, in this example, the product cameras 41-44 are 3D cameras, and can capture color product images and depth product images at the same time.

In addition, in this example, the pattern of the pattern label is a combination of different shapes and colors. For example, the patterns of the pattern tags of goods 50, 53, 56 are red triangles, the patterns of the pattern tags of goods 51, 54, 57 are green triangles, and the patterns of the pattern tags of goods 52, 55, 58 are red squares.

Furthermore, the pattern label of the present invention can be combined with goods in any of the following ways. In one example, the pattern labels of goods 50-52 are designed as outer boxes, and can hold goods 50-52. In one example, the pattern labels 530-550 are adhered to the surfaces of the goods 53-55. In one example, graphic labels 560-580 are printed directly on goods 53-55.

As shown in FIG. 8, in this example, the management system 1 can continuously detect the color tracking image and / or the depth tracking image captured by the tracking camera 40, and detect the person 6 entering the tracking area 34. The person 6 performs face recognition to obtain the identity information of the person 6 and constructs a pose model 60 of the person 6 (in this example, the pose model 60 is presented in the form of a skeleton of the person 6).

Next, as shown in FIG. 9, during the process of the person 50 picking up the goods 50, the management system 1 may confirm the identity information of the person 6 picking up the goods 50 according to the image captured by the product camera 41 and identify the picked up goods. The pattern label of 50 is a red triangle to obtain the corresponding product information. In addition, the management system 1 links the product data of the goods 50 and the identity data of the personnel 6 (such as adding the product data of the goods 50 to the shopping cart list corresponding to the identity data of the personnel 6).

Next, as shown in FIG. 10, during the process of the person 6 taking another product 51, the management system 1 can confirm the identity information of the person 6 taking the product 51 according to the image captured by the product camera 41 and identify the person being taken The pattern label of the product 51 is a green triangle, and the corresponding product data is obtained. In addition, the management system 1 links the product data of the goods 51 and the identity data of the personnel 6 (such as adding the product data of the goods 51 to the shopping cart list corresponding to the identity data of the personnel 6).

Next, as shown in FIG. 11, in the process of returning the goods 51 by the personnel 6, the management system 1 can confirm the identity information of the personnel 6 returning the goods 51 based on the image captured by the goods camera 41, and identify the returned goods The pattern label of 51 is a green triangle, so as to obtain the corresponding product data. In addition, the management system 1 releases the connection between the product data of the goods 51 and the identity data of the personnel 6 (for example, the product data of the goods 51 is removed from the shopping cart list corresponding to the identity data of the personnel 6).

Then, as shown in FIG. 12, when the management system 1 detects that the person 6 leaves the tracking area 34, it performs settlement processing on the cargo information of all the goods 51 currently connected to the identification information of the person 6 (for example, based on the identification information of the person 6). The corresponding payment information will check out all the product information in the shopping cart list).

In this way, the present invention can effectively hope whether personnel can pick up or return the goods, and realize unmanned goods management.

The above description is only a preferred specific example of the present invention, and therefore does not limit the scope of the patent of the present invention. Therefore, any equivalent changes made by using the content of the present invention are included in the scope of the present invention in the same way. Bright.

Claims (22)

An unmanned goods management method for managing multiple goods in a tracking area includes the following steps: a) shooting the tracking area through a tracking camera to obtain a tracking image; b) obtaining a tracking image entering the tracking area An identification of the person; c) generating an attitude model of the person based on the tracking image; d) shooting a region of the plurality of goods through a goods camera to obtain a color goods image; e) identifying each of the color goods images A pattern label of the item, and a position of each item is determined according to the position of each pattern label; f) when any of the item is judged by the person according to the posture model and the position of the item Take a piece of product information of the product; and g) link the identity information and the product information. The method for managing unmanned goods according to claim 1, further comprising a step h) performing a settlement process on all the goods data linked to the identity data when a settlement condition is satisfied. The unmanned goods management method according to claim 2, wherein step h) further includes the following steps: h1) obtaining a payment information corresponding to the identity information; h2) calculating based on a price data of each of the linked goods information A total amount of goods; and h3) settlement based on the payment information and the total amount of goods. The unmanned goods management method according to claim 2, wherein the settlement condition is that the person is detected to leave the tracking area. The unmanned goods management method according to claim 1, wherein step b) is to obtain a characteristic data of the person through an identification device, and query the corresponding identity information in a database based on the characteristic data, wherein the characteristic The information is the person's identification code or biological information. The unmanned goods management method according to claim 1, wherein the tracking image includes a color tracking image; the step b) is to perform face recognition on the color tracking image to obtain a characteristic data of the person's face, and Query the corresponding identity information in a database based on the characteristic data. The unmanned goods management method according to claim 1, wherein the tracking image includes a color tracking image and a depth tracking image; the step c) is to determine multiple joints of the person based on the color tracking image and the depth tracking image Position, and generate the pose model according to the multiple joint positions. The method for managing unmanned goods according to claim 7, wherein after the step a) and before the step e), the method further comprises the following steps: i1) tracking the position of a face of the person according to the color tracking image; and i2) When it is determined that the person is approaching the plurality of goods according to the position of the posture model or the position of the face, a goods camera is controlled to start shooting the area where the plurality of goods are located. The unmanned goods management method according to claim 1, wherein the step d) is obtaining the color goods image and a depth goods image through the goods camera; wherein the step f) includes: f1) according to the color goods image and The deep product image generates a person's hand posture model; and f2) when judging any of the goods by the person based on the hand posture model and each of the product positions, based on the pattern label of the product being taken For the shape or color of the product, query the corresponding product data in a database. The method for managing unmanned goods according to claim 9, wherein step f) further includes the following steps: f3) Perform face recognition on the color goods image to confirm the identity information of the person who took the goods; wherein, step g) is to link the identified identity information and the goods information. The method for managing unmanned goods according to claim 1, further comprising the following steps: j1) obtaining the release when the person judges that the person puts back any of the goods based on the posture model and the location of the goods being taken Return the goods information of the goods; and j2) release the link between the identity information and the goods information of the goods returned. An unmanned goods management system for managing multiple goods in a tracking area includes: a tracking camera for shooting the tracking area to obtain a tracking image; and a database for storing the corresponding multiples. Multiple product data of each item and multiple identification data corresponding to multiple persons; a color product camera for shooting the area where the multiple items are located to generate a color product image; and a control device connected to the tracking camera, The database and the color goods camera, the control device includes: an identity recognition module for obtaining the identity data of a person entering the tracking area; an attitude tracking module for generating the person based on the tracking image An attitude model of a product; a product identification module for identifying a pattern label of each product in the color product image, and determining a product position of each product according to the position of each pattern label, and used to obtain the captured product Take a piece of product information of the item; a take analysis module, which is used to determine whether to take any responsibility based on the attitude model and the position of each item. The goods were to pick up the staff; and A processing module is used to link the obtained identity information and the goods information of the goods obtained. The unmanned goods management system according to claim 12, wherein the control device further includes a settlement module for performing a settlement process on all the goods data linked to the identity data when a settlement condition is satisfied. The unmanned goods management system according to claim 13, wherein the database further stores a plurality of payment data corresponding to a plurality of identity data and a price data of each of the goods data, and the settlement module obtains the identity data. The payment information and the price information of all the product information linked to the identity information are calculated based on the obtained price information for a total of the goods, and the billing is performed based on the payment information and the total of the goods. The unmanned goods management system according to claim 13, wherein the settlement condition is that the person is detected to leave the tracking area. The unmanned goods management system according to claim 12, wherein the database further stores a plurality of characteristic data corresponding to a plurality of identity data; wherein the unmanned goods management system further includes an identification device connected to the control device. , The identification device is used to obtain the characteristic data of the person entering the tracking area; wherein the identification module queries the identification data corresponding to the characteristic data from the database; wherein the identification device is an RFID card reader Device or biometric device, the characteristic data is the identification code or biometric information of the person. The unmanned goods management system according to claim 12, wherein the database further stores a plurality of characteristic data corresponding to a plurality of identity data; wherein the tracking camera includes a color tracking camera for generating a color tracking image ; The identity recognition module performs face recognition on the color tracking image to obtain a feature data of the face of the person entering the tracking area, and inquires the corresponding identity data in the database according to the feature data. The unmanned goods management system according to claim 12, wherein the tracking camera includes a color tracking camera for generating a color tracking image and a depth tracking camera for generating a depth tracking image. The attitude tracking module is A plurality of joint positions of the person are determined according to the color tracking image and the depth tracking image, and the posture model is generated according to the plurality of joint positions. The unmanned goods management system according to claim 18, wherein the control device further includes a face tracking module for tracking a person's face position based on the color tracking image; wherein the processing module is based on When the position of the posture model or the position of the face determines that the person is approaching the plurality of goods, the color goods camera is controlled to start shooting the area where the plurality of goods are located. The unmanned goods management system according to claim 12, wherein the unmanned goods management system further comprises a depth goods camera connected to the control device, the depth goods camera is used to photograph the area where the plurality of goods are located to generate a depth goods Image; wherein, the database further stores the shape or color of the pattern label of each of the product data; wherein the control device further includes a hand tracking module for generating the person based on the color product image and the depth product image The hand analysis model is based on the hand posture model and the positions of the goods to determine whether any of the goods have been taken by the person. Among them, the goods identification module is taken based on For the shape or color of the pattern label of the product, query the database for the corresponding product data. The unmanned goods management system according to claim 12, wherein the control device further includes an identity verification module, which performs face recognition on the color goods image to confirm the identity of the person who took the goods. Identity data; among them, the processing module is a link between the identified identity data and the goods data. The unmanned goods management system according to claim 12, wherein the retrieval analysis module determines whether the person puts back any of the goods based on the attitude model and the location of the goods being retrieved; among which, the goods The identification module obtains the product data of the returned product; wherein the processing module releases the link between the identity data and the product data of the returned product.