KR20230042803A - System for tracking product, method for tracking product and computer program stored in a recording medium to execute the method - Google Patents

Abstract

According to one embodiment of the disclosed invention, a product tracking method for tracking the movement of products on a store shelf comprises the steps of: obtaining, by a camera, an image including the store shelf displaying the products and at least one customer; setting the products displayed for each load cell area of the store shelf shown in the image using a coordinate designator; extracting a pixel cluster for each of a plurality of frames included in the image by a feature extraction unit; specifying, by a buyer confirmation unit, the customer who moved the products displayed on the load cell based on the characteristics of the pixel cluster included in a first frame and the characteristics of the pixel cluster included in a second frame temporally adjacent to the first frame; and determining, by the buyer confirmation unit, the specified customer as a buyer of the moved products. According to one aspect of the disclosed invention, in a situation in which a large number of customers are present in the image obtained in an unmanned store environment, when a product purchasing behavior occurs, it is possible to accurately distinguish who is a subject of the product purchasing behavior.

Description

A computer program stored in a recording medium to execute a product tracking system, a product tracking method, and a product tracking method

The present invention relates to a product tracking system and product tracking method capable of determining an actual purchaser of a product based on motion in an image acquired in an unmanned store environment.

Recently, unmanned stores have been widely introduced. In an unmanned store where a load cell rack is installed, when a purchase occurs by customers adjacent to the load cell rack, it is necessary to distinguish who the actual product purchaser is among a plurality of customers.

Optical flow can be used as a representative motion tracking technique for analyzing the motion of an object captured in an image. Optical flow is valid only when the premise that the local feature in the image does not change even if it is moved is satisfied, so the direction of movement of the pixel feature ( There is a problem that the accuracy of orientation is poor.

Therefore, it is necessary to estimate the actual purchaser of the corresponding product by analyzing the movement of the product and the customer without using the orientation of the pixel feature.

The present invention is to provide a product tracking system and a product tracking method that can accurately distinguish who is the subject of product purchase behavior when a product purchase behavior occurs in a situation where a large number of customers exist in an image acquired in an unmanned store environment. .

In addition, the present invention is based on the characteristics of other pixel clusters in addition to the moving direction of the pixel cluster, even if the color and object shape characteristics in a certain area, such as the customer's skin or the sleeve of the clothes, are similar in the image and the reliability of the moving direction of the pixel cluster is greatly reduced. It is to provide a product tracking system and a product tracking method capable of specifying a buyer who has moved a product displayed as .

A product tracking method for tracking the movement of goods in a store shelf according to an aspect of the disclosed invention includes acquiring, by a camera, an image including the store shelf where products are displayed and at least one customer; Presetting, by a coordinate designation unit, products displayed for each load cell area of the store shelf appearing in the image; extracting a pixel cluster for each of a plurality of frames included in the image by a feature extraction unit; Based on the characteristic of the pixel cluster included in the first frame and the characteristic of the pixel cluster included in the second frame temporally adjacent to the first frame, the buyer decision unit moves the product displayed in the load cell. identifying the customer who took the action; and determining the specified customer as a purchaser of the moved product by a purchaser determination unit.

In addition, detecting a weight change of the load cell by a weight sensor; determining, by the feature extraction unit, a first time when a change in weight of the load cell is detected; And determining, by the feature extraction unit, a second time that is before a predetermined time from the first time and a third time that is after a predetermined time from the first time, In the step of extracting a pixel cluster for each of a plurality of frames, a plurality of frames included between a frame obtained by the camera at the second time and a frame obtained by the camera at the third time are selected by the feature extraction unit. and extracting the pixel cluster based on the pixel cluster.

In addition, the characteristic of the pixel cluster is selected from among the scale of each pixel cluster, the average intensity of each pixel cluster, the angle between the coordinates of the center of each pixel cluster and the coordinates of the product, and the distance between the coordinates of the center of each pixel cluster and the coordinates of the product. Including at least one, specifying a customer who moved a product displayed on the load cell includes, by the buyer determining unit, the characteristics of the pixel cluster of the first frame and the pixel cluster of the second frame. determining a product movement direction based on a result of comparing characteristics; and specifying a customer who performed an action of moving a product displayed in the load cell based on the product movement direction and the coordinates of the at least one customer, by the buyer determination unit.

In addition, generating a machine learning model through a machine learning method by a machine learning unit by setting the characteristics of the pixel cluster determined based on the learning image as an input variable and setting the buyer information of the learning image as an output variable. can include more.

In addition, the step of specifying the customer who has moved the product displayed in the load cell is based on the pixel cluster included in the first frame and the pixel cluster included in the second frame by the buyer determination unit. It may include specifying a customer who performed an action of moving a product displayed on the load cell through the machine learning model.

In addition, generating a unique ID for each customer in the store based on information acquired by at least one lidar sensor or camera provided in the store by the customer ID setting unit; further comprising the moved The step of determining the purchaser of the product may include matching the ID of the specified customer with the moved product by the purchaser confirmation unit.

In addition, adding the information of the moved product to the purchase list of the ID of the specified customer by the purchase list creation unit; and generating, by a payment information generation unit, payment information used for product payment of the specified customer based on the purchase list when the specified customer leaves the store.

In addition, the step of specifying the customer who has moved the product displayed in the load cell is based on the pixel cluster included in the first frame and the pixel cluster included in the second frame by the buyer determination unit. generating a skeleton based on the customer's joints included in the image and tracking the movement of each joint if the customer who moved the product displayed on the load cell cannot be identified; and specifying, by the purchaser determining unit, a customer who performed an action of moving a product displayed on the load cell based on the movement of the joint.

A computer program according to an aspect of the disclosed invention may be stored in a computer-readable recording medium to execute the product tracking method.

A product tracking system for tracking the movement of goods in a store shelf according to an aspect of the disclosed invention includes a camera configured to acquire an image of the store shelf where products are displayed and at least one or more customers; a coordinate designation unit configured to preset products displayed for each load cell area of the shop shelf appearing in the image; a feature extraction unit configured to extract a pixel cluster for each of a plurality of frames included in the image; and based on the characteristics of the pixel cluster included in the first frame and the feature of the pixel cluster included in the second frame temporally adjacent to the first frame, a customer who moves the product displayed in the load cell. and a purchaser determining unit configured to specify and determine the specified customer as a purchaser of the moved product.

The weight sensor detects a change in weight of the load cell; and the feature extraction unit: determines a first time when the change in weight of the load cell is sensed; determining a second time that is before a predetermined time from the first time and a third time that is after a predetermined time from the first time; The pixel cluster may be extracted based on a plurality of frames included between frames acquired by the camera at the second time and frames acquired by the camera at the third time.

In addition, the characteristic of the pixel cluster is selected from among the scale of each pixel cluster, the average intensity of each pixel cluster, the angle between the coordinates of the center of each pixel cluster and the coordinates of the product, and the distance between the coordinates of the center of each pixel cluster and the coordinates of the product. It includes at least one, and the buyer determination unit determines a product movement direction based on a result of comparing the pixel cluster characteristics of the first frame and the pixel cluster characteristics of the second frame; And it may be configured to specify a customer who performed an action of moving a product displayed in the load cell based on the product movement direction and the coordinates of the at least one customer.

In addition, a machine learning unit configured to generate a machine learning model through a machine learning method by setting the characteristics of the pixel cluster determined based on the learning image as an input variable and setting the buyer information of the learning image as an output variable. can do.

In addition, the buyer determination unit specifies a customer who performed an action of moving a product displayed in the load cell through the machine learning model based on the pixel cluster included in the first frame and the pixel cluster included in the second frame. can be configured to

In addition, a customer ID setting unit configured to generate a unique ID for each customer in the store based on information acquired by at least one lidar sensor or camera provided in the store; further comprising the buyer determination unit, It may be configured to match the ID of the specified customer with the moved product.

In addition, a purchase list creation unit configured to add the information of the moved product to the purchase list of the ID of the specified customer; and a payment information generation unit configured to generate payment information used for product payment of the specified customer based on the purchase list when the specified customer leaves the store.

In addition, the buyer determination unit: If it is impossible to specify a customer who has moved a product displayed in the load cell based on the pixel cluster included in the first frame and the pixel cluster included in the second frame, included in the image Creating a skeleton based on the customer's joints to track the movement of each joint; And it may be configured to specify a customer who performed an action of moving a product displayed in the load cell based on the movement of the joint.

According to one aspect of the disclosed invention, in a situation where a large number of customers exist in an image acquired in an unmanned store environment, when a product purchasing behavior occurs, it is possible to accurately distinguish who the subject of the product purchasing behavior is.

In addition, according to an embodiment of the present invention, even if the reliability of the moving direction of the pixel cluster is greatly reduced because the color and object shape characteristics in a certain area, such as the customer's skin or the sleeve of the clothes, are similar in the image, pixels other than the moving direction of the pixel cluster Based on the characteristics of the cluster, the buyer who moved the displayed product can be identified.

1 is a block diagram of a product tracking system according to an embodiment.
2 is a diagram illustrating an image obtained by a camera according to an exemplary embodiment.
3 is a flowchart of a method for predicting an entity state according to an exemplary embodiment.

Like reference numbers designate like elements throughout the specification. This specification does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the disclosed invention belongs is omitted. The term '~unit' used in the specification may be implemented in software or hardware, and according to embodiments, a plurality of '~units' may be implemented as one component, or one '~unit' may constitute a plurality of components. It is also possible to include elements.

In addition, when a certain component is said to "include", this means that it may further include other components without excluding other components unless otherwise stated.

Terms such as first and second are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

Expressions in the singular number include plural expressions unless the context clearly dictates otherwise.

In each step, the identification code is used for convenience of description, and the identification code does not explain the order of each step, and each step may be performed in a different order from the specified order unless a specific order is clearly described in context. there is.

Hereinafter, the working principle and embodiments of the disclosed invention will be described with reference to the accompanying drawings.

In an unmanned store where a load cell shelf is installed, a customer can lift and move a product by hand in order to purchase a product displayed on the load cell shelf. In this case, the unmanned store system may automatically determine that a certain customer has purchased a certain product by analyzing an image acquired by a camera in the store. On the other hand, if there are several customers adjacent to a certain load cell shelf, it is necessary to distinguish which customer among the plurality of customers has moved the product displayed on the load cell shelf with an intention to purchase.

1 is a configuration diagram of a product tracking system according to an embodiment, and FIG. 2 is a diagram illustrating an image acquired by a camera according to an embodiment.

Referring to FIG. 1 , a product tracking system 100 according to an embodiment of the present invention may be configured to track product movement in a store shelf 110 .

The product tracking system 100 includes a store shelf 110, a weight sensor 111, a camera 101, a lidar sensor 102, a coordinate designation unit 103, a machine learning unit 104, and a feature extraction unit 105. ), a customer ID setting unit 106, a memory 120, a buyer determination unit 107, a purchase list generation unit 108, and a payment information generation unit 109.

The camera 101 may acquire an image including the store shelf 110 where products are displayed and at least one customer.

In an environment where there are obstacles such as narrow aisles and high shelves, such as convenience stores, and a large number of customers can make purchases in a similar form on adjacent streets at the same time, the position of the camera 101 that can check the shape of each customer without being covered is extremely limited. do.

Therefore, the camera 101 of the present invention can mainly use a bird eye view method that looks down vertically between the shelf and the customer, but is not limited thereto. is installed in an unmanned store in any way.

The coordinate designator 103 may preset products displayed for each load cell area of the store shelf 110 appearing in the image.

For example, the manager of an unmanned store can set in advance what products are displayed in each load cell through a direct input method through an input unit provided in the unmanned store system or through a communication unit provided in the unmanned store system by inputting the input to an external terminal, The designation unit 103 receives the information input by the manager from the input unit or the communication unit, and based on this, can set products displayed for each load cell area.

On the other hand, there is an optical flow method as a method of analyzing the motion of an object in an image. Optical flow is a method of tracking the movement of an object through the premise that the shape of an object in an image does not change in adjacent frames captured in a short period of time.

Optical flow searches for pixel features within a mask window of a certain size in an adjacent frame and expresses the movement position in orientation (direction) and magnitude (intensity). Reliability of orientation when color and shape features are similar within a certain area such as skin or sleeves There is a problem that a large drop in occurs.

The product tracking system 100 of the present invention finds a key frame for purchasing a product by obtaining a change in magnitude of pixels for a pre-set load cell area in an acquired image, and the intensity of pixels in images before and after the key frame. can be clustered by spatial clustering using the DBSCAN method, a distance-based clustering technique, and the movement patterns of the clusters can be analyzed.

Before the key frame when the customer picks up or puts down the product, the customer is approaching the shelf, and after the key frame, the hand is withdrawn, so find the customer ID that is close to the cluster corresponding to the phenomenon. For each frame, several pixel cluster features can be used.

The product tracking system 100 performs distance-based clustering on the characteristics of pixel clusters on the time axis, binds clusters between frames, defines a cost function for distance, and sorts them based on the size of cost. there is.

The ID of the customer who purchased the product can be specified by taking the distance between the coordinates of the center of each frame of the sorted clusters and the coordinates of the customer projected on the image as a cost.

The feature extractor 105 may extract a pixel cluster for each of a plurality of frames included in the image.

The pixel cluster may be a set of pixels having similar properties, such as a location or a state, among pixels included in a certain frame.

Referring to FIG. 2 , for example, if a customer's clothing sleeve is photographed in one frame of an image, pixels representing the customer's clothing sleeve are distributed in similar locations and may have similar colors or brightness. The feature extractor 105 may determine, as a pixel cluster, pixels having similar properties among pixels included in any one frame.

The buyer determination unit 107 may specify a customer who has moved a product displayed on the load cell based on characteristics of pixel clusters included in each of a plurality of frames.

Specifically, the buyer determination unit 107 moves the product displayed in the load cell based on the characteristics of the pixel cluster included in the first frame and the pixel cluster included in the second frame temporally adjacent to the first frame. You can identify the customer who took the action.

The feature of the pixel cluster may include at least one of the size of each pixel cluster, the average intensity of each pixel cluster, the angle between the coordinates of the center of each pixel cluster and the coordinates of the product, and the distance between the coordinates of the center of each pixel cluster and the coordinates of the product. there is.

The buyer determination unit 107 may determine a product movement direction based on a result of comparing the pixel cluster characteristics of the first frame and the pixel cluster characteristics of the second frame.

The purchaser determining unit 107 may specify a customer who performed an action of moving a product displayed on the load cell based on the product movement direction and the coordinates of at least one customer.

In conclusion, the buyer determination unit 107 determines that if a certain pixel cluster moves toward the load cell shelf in a certain area of the image and moves from the load cell shelf toward the corresponding area again, the customer located in the corresponding area moves the product on the load cell shelf. It can be identified as a customer.

The buyer determining unit 107 may determine the specified customer as a buyer of the moved product. That is, through the above-described method, the product tracking system 100 may determine whether a customer who has moved with the intention of actually purchasing the product is a purchaser, even if there are several customers around the load cell where the moved product was displayed.

As a result, the product tracking system 100 according to the present invention has similar color and shape features within a certain area such as the customer's skin or sleeves included in the image, even if the reliability of orientation (direction) is greatly reduced. ), it is possible to accurately classify the subject of a product purchasing act based on the characteristics of a pixel cluster different from ).

Coordinate designation unit 103, machine learning unit 104, feature extraction unit 105, customer ID setting unit 106, memory 120, buyer determination unit 107, purchase risk generation unit, payment information generation unit (109) may include any one of a plurality of processors included in the product tracking system (100). In addition, the object recognition method according to the embodiments of the present invention described so far and the embodiments to be described in the future may be implemented in the form of a program that can be driven by a processor.

Here, the program may include program commands, data files, and data structures alone or in combination. The program may be designed and manufactured using machine language codes or high-level language codes. The program may be specially designed to implement the above-described code correction method, or may be implemented using various functions or definitions that are known and usable to those skilled in the art in the field of computer software. A program for implementing the above information display method may be recorded on a recording medium readable by a processor. At this time, the recording medium may be the memory 120 included in the product tracking system 100.

The memory 120 may store a program for performing the above-described operation and an operation to be described later, and the memory 120 may execute the stored program. When there are a plurality of processors and memories 120, they may be integrated into a single chip or may be provided in physically separate locations. The memory 120 may include volatile memory such as static random access memory (S-RAM) and dynamic random access memory (D-lab) for temporarily storing data. In addition, the memory 120 may include a ROM (Read Only Memory), an Erasable Programmable Read Only Memory (EPROM), and an Electrically Erasable Programmable Read Only Memory (EEPROM) for long-term storage of control programs and control data. It may contain non-volatile memory.

The processor may include various logic circuits and arithmetic circuits, process data according to programs provided from the memory 120, and generate control signals according to processing results.

The camera 101 of the present invention may continuously acquire image information, but the product tracking system 100 does not always have to determine who the customer who moved the product is. Even if the customer is not in the unmanned store, the customer is near the load cell, or even if the customer moves, if the customer does not actually pick up the product displayed on the load cell, there is no need for the system of the unmanned store to track the movement of the product. . Therefore, it may be desirable to analyze the image only when the customer actually picks up the product displayed on the load cell.

Referring to FIG. 1 , a shop shelf 110 may include a weight sensor 111 . Specifically, the store shelf 110 may include weight sensors 111 corresponding to each load cell provided on the store shelf 110 .

The weight sensor 111 may detect a weight change of the load cell.

That is, when a customer picks up a product displayed on the load cell, the weight of the total products displayed on the load cell decreases, and the weight sensor 111 connected to the load cell can detect this change in weight.

The feature extractor 105 may determine a first time when a weight change of the load cell is detected. That is, the first time is the time when the customer picks up the product displayed on the load cell.

The first time may be a reference time for determining frames actually used to track a product among a plurality of frames obtained by the camera 101 .

The feature extractor 105 may determine the second time and the third time.

The second time may be a time from the first time before a predetermined time, and the third time may be a time from the first time to a predetermined time or later.

The feature extractor 105 may extract a pixel cluster based on a plurality of frames included between a frame acquired by the camera 101 at the second time and a frame obtained by the camera 101 at the third time.

Images obtained before the second time and after the third time may be images unrelated to a customer's act of purchasing a product displayed on a load cell whose weight has changed.

As a result, the product tracking system 100 may determine who the actual purchaser is by analyzing only frames of the image obtained between the second time and the third time.

On the other hand, the predetermined time required to determine the second time and the predetermined time required to determine the third time do not necessarily have to be the same, and it does not matter if either one of the two is zero.

The machine learning unit 104 may generate the machine learning model 121 through a machine learning method by setting the characteristics of the pixel cluster determined based on the learning image as an input variable and the buyer information of the learning image as an output variable. .

The machine learning unit 104 may learn the machine learning model 121 using outputs that are finally output through the machine learning model 121 according to the input. Machine learning model 121 may be stored in memory 120 of product tracking system 100 .

Machine learning may mean using a model composed of multiple parameters and optimizing the parameters with given data. Machine learning may include supervised learning, unsupervised learning, and reinforcement learning depending on the form of a learning problem. Supervised learning is to learn the mapping between inputs and outputs, and can be applied when input and output pairs are given as data. Unsupervised learning is applied when there are only inputs and no outputs, and regularities between inputs can be found.

The machine learning unit 104 according to an embodiment of the present invention may generate the machine learning model 121 based on the characteristics of the pixel cluster extracted from each frame of the training image and the actual purchaser information of each training image. .

The product tracking system 100 may determine a buyer through a deep learning method as well as a machine learning method, and may determine a purchaser in various ways.

The buyer determining unit 107 may specify a customer who performed an action of moving a product displayed in the load cell through the machine learning model 121 based on the pixel cluster included in the first frame and the pixel cluster included in the second frame. can

The customer ID setting unit 106 may generate a unique ID for each customer in the store based on information acquired by at least one lidar sensor 102 or camera 101 provided in the store.

If the customer ID setting unit 106 has an ID assigned to any one customer, even if the customer moves around the inside of the store, the corresponding customer is based on the information acquired by the lidar sensor 102 or camera 101 provided in the store. You can keep track of your customers.

In addition, the customer ID setting unit 106 may transmit ID information of a customer shown in a camera image to the purchaser determination unit 107 .

The buyer determination unit 107 may match the ID of the specified customer with the moved product.

The purchase list creation unit 108 may add the information of the moved product to the purchase list of the ID of the specified customer. In other words, the act of a customer picking up a product from the load cell shelf several times.

The payment information generation unit 109 may generate payment information used for product payment of the specified customer based on the purchase list when the specified customer leaves the store.

Payment information may be transmitted directly or indirectly to the purchaser's card company server or VAN company server, but a method of paying for a customer's product based on the payment information is not limited thereto.

On the other hand, even if the decision is made based on the pixel cluster as described above, there may be a case where it is not possible to specify a customer who moved a product displayed on the load cell.

The skeleton detection method can track the movement of each joint by generating a skeleton based on human joints. The skeleton detection method can track the detected wrist position of the customer, so it is possible to intuitively analyze the purchase behavior. If buyers cannot be identified based on pixel clustering, it may be desirable to identify buyers by analyzing the skeleton.

If the purchaser determining unit 107 cannot identify the customer who moved the product displayed in the load cell based on the pixel cluster included in the first frame and the pixel cluster included in the second frame, the customer's joint included in the image By creating a skeleton as a reference, the movement of each joint can be traced.

The purchaser determination unit 107 may specify a customer who performed an action of moving a product displayed on the load cell based on the motion of the joint.

At least one component may be added or deleted corresponding to the performance of the components described above. In addition, it will be easily understood by those skilled in the art that the mutual positions of the components can be changed corresponding to the performance or structure of the system.

3 is a flowchart of a method for predicting an entity state according to an exemplary embodiment. This is only a preferred embodiment for achieving the object of the present invention, and it goes without saying that some components may be added or deleted as needed.

Referring to FIG. 3, by the coordinate designation unit 103, products displayed for each load cell area of the store shelf 110 appearing in the image are preset.

The coordinate designation unit 103 may preset products displayed for each load cell area of the store shelf 110 appearing in the image (1001).

The camera 101 may acquire an image including the store shelf 110 where products are displayed and at least one customer (1002).

The weight sensor 111 may detect a weight change of the load cell (1003).

The feature extraction unit 105 determines the first time when the change in weight of the load cell is detected, the second time which is before a predetermined time from the first time, and the third time which is a time after the predetermined time from the first time. can be determined (1004).

A pixel cluster may be extracted for each of a plurality of frames included in the image by the feature extractor 105 (1005). In this case, the feature extractor 105 may extract a pixel cluster based on a plurality of frames included between the frame acquired by the camera 101 at the second time and the frame obtained by the camera 101 at the third time. there is.

The buyer determining unit 107 moves the product displayed in the load cell based on the characteristics of the pixel cluster included in the first frame and the pixel cluster included in the second frame temporally adjacent to the first frame. A customer can be specified (1006).

At this time, the purchaser determination unit 107 determines the product movement direction based on the result of comparing the pixel cluster characteristics of the first frame and the pixel cluster characteristics of the second frame, and determines the product movement direction and the coordinates of at least one customer. Based on this, it is possible to specify a customer who has moved the product displayed in the load cell.

The buyer determination unit 107 may determine the specified customer as a buyer of the moved product (1007).

As above, the disclosed embodiments have been described with reference to the accompanying drawings. Those skilled in the art to which the present invention pertains will understand that the present invention can be implemented in a form different from the disclosed embodiments without changing the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

100: product tracking system
101: camera
102: lidar sensor
103: coordinate designation unit
104: machine learning unit
105: feature extraction unit
106: customer ID setting unit
107: buyer confirmation unit
108: purchase list generation unit
109: payment information generation unit
110: store shelves
111: weight sensor
120: memory
121: machine learning model

Claims (17)

In the product tracking method for tracking the movement of products on a store shelf,
Obtaining, by means of a camera, an image including the store shelves and at least one customer;
Presetting, by a coordinate designation unit, products displayed for each load cell area of the store shelf appearing in the image;
extracting a pixel cluster for each of a plurality of frames included in the image by a feature extraction unit;
Based on the characteristic of the pixel cluster included in the first frame and the characteristic of the pixel cluster included in the second frame temporally adjacent to the first frame, the buyer decision unit moves the product displayed in the load cell. identifying the customer who took the action; and
Determining the specified customer as a purchaser of the moved product by a purchaser confirmation unit; Product tracking method comprising a. According to claim 1,
detecting a weight change of the load cell by a weight sensor;
determining, by the feature extraction unit, a first time when a change in weight of the load cell is detected; and
Determining, by the feature extraction unit, a second time that is before a predetermined time from the first time and a third time that is after a predetermined time from the first time,
The step of extracting the pixel cluster for each of the plurality of frames,
extracting, by the feature extraction unit, the pixel cluster based on a plurality of frames included between frames acquired by the camera at the second time and frames acquired by the camera at the third time; including , how to track the goods. According to claim 1,
The characteristics of the pixel cluster are:
at least one of a scale for each pixel cluster, an average intensity for each pixel cluster, an angle between a center coordinate for each pixel cluster and a coordinate of the product, and a distance between a center coordinate for each pixel cluster and a coordinate for the product;
The step of specifying the customer who moved the product displayed on the load cell,
determining, by the buyer determination unit, a product movement direction based on a result of comparing the pixel cluster characteristics of the first frame and the pixel cluster characteristics of the second frame; and
Including, the step of specifying a customer who moved the product displayed in the load cell based on the product movement direction and the coordinates of the at least one customer by the buyer determination unit. According to claim 3,
Generating, by a machine learning unit, a machine learning model through a machine learning method by setting the characteristics of the pixel cluster determined based on the learning image as an input variable and setting the buyer information of the learning image as an output variable; further comprising How to track your product. According to claim 4,
The step of specifying the customer who moved the product displayed on the load cell,
The buyer determination unit identifies a customer who performed an action of moving a product displayed in the load cell through the machine learning model based on the pixel cluster included in the first frame and the pixel cluster included in the second frame. Step of doing; containing, product tracking method. According to claim 1,
Generating, by a customer ID setting unit, a unique ID for each customer in the store based on information acquired by at least one lidar sensor or camera provided in the store; further comprising,
The step of determining as a buyer of the moved product,
, Product tracking method comprising: matching the ID of the specified customer with the moved product by the purchaser determination unit. According to claim 6,
adding information of the moved product to a purchase list of the ID of the specified customer by a purchase list creation unit; and
Generating, by a payment information generation unit, payment information used for product payment of the specified customer based on the purchase list when the specified customer leaves the store; further comprising a product tracking method. According to claim 1,
The step of specifying the customer who moved the product displayed on the load cell,
If the buyer determining unit cannot identify a customer who has moved a product displayed in the load cell based on the pixel cluster included in the first frame and the pixel cluster included in the second frame, the customer included in the image Tracking the movement of each joint by creating a skeleton based on the customer's joints; and
Including, the product tracking method comprising the; step of specifying a customer who moved the product displayed in the load cell based on the motion of the joint by the purchaser determination unit. A computer program stored in a computer-readable recording medium to execute the product tracking method according to any one of claims 1 to 8. In the product tracking system for tracking the movement of products on the store shelf,
a camera configured to acquire an image including the store shelves and at least one customer;
a coordinate designation unit configured to preset products displayed for each load cell area of the shop shelf appearing in the image;
a feature extraction unit configured to extract a pixel cluster for each of a plurality of frames included in the image; and
Based on the feature of the pixel cluster included in the first frame and the feature of the pixel cluster included in the second frame temporally adjacent to the first frame, a customer who has moved the product displayed on the load cell is specified. and a purchaser determination unit configured to determine the specified customer as a purchaser of the moved product. According to claim 10,
Further comprising a weight sensor for detecting a change in weight of the load cell;
The feature extraction unit:
determining a first time when a weight change of the load cell is sensed;
determining a second time that is before a predetermined time from the first time and a third time that is after a predetermined time from the first time; and
and extracting the pixel cluster based on a plurality of frames included between frames acquired by the camera at the second time and frames acquired by the camera at the third time. According to claim 10,
The characteristics of the pixel cluster are:
at least one of a scale for each pixel cluster, an average intensity for each pixel cluster, an angle between a center coordinate for each pixel cluster and a coordinate of the product, and a distance between a center coordinate for each pixel cluster and a coordinate for the product;
The buyer determination unit,
determining a product movement direction based on a result of comparing features of the pixel clusters of the first frame and features of the pixel clusters of the second frame; and
Based on the product movement direction and the coordinates of the at least one customer, a product tracking system configured to specify a customer who has moved a product displayed in the load cell. According to claim 12,
A machine learning unit configured to generate a machine learning model through a machine learning method by setting the characteristics of the pixel cluster determined based on the learning image as an input variable and setting the buyer information of the learning image as an output variable. tracking system. According to claim 13,
The buyer determination unit,
A product tracking system configured to specify a customer who has moved a product displayed in the load cell through the machine learning model based on the pixel cluster included in the first frame and the pixel cluster included in the second frame. . According to claim 10,
A customer ID setting unit configured to generate a unique ID for each customer in the store based on information acquired by at least one lidar sensor or camera provided in the store; further comprising,
The buyer determination unit,
A product tracking system configured to match the ID of the specified customer with the moved product. According to claim 15,
a purchase list creation unit configured to add information of the moved product to a purchase list of the specified customer ID; and
A product tracking system further comprising: a payment information generation unit configured to generate payment information used for product payment of the specified customer based on the purchase list when the specified customer leaves the store. According to claim 10,
The buyer determination unit:
If the customer who moved the product displayed on the load cell cannot be identified based on the pixel cluster included in the first frame and the pixel cluster included in the second frame, a skeleton is formed based on the joint of the customer included in the image. to track the movement of each joint by generating; and
Based on the movement of the joint, the product tracking system configured to specify a customer who has moved the product displayed in the load cell.

Images