KR102585358B1 - Method and apparatus for analyzing behavior pattern of customer in unmanned shop using ai-based image processing model - Google Patents

Abstract

One aspect of the present invention for achieving the above object discloses a method for analyzing customer behavior patterns in an unmanned store using a computing device. The method includes receiving image data captured from a first camera disposed in the unmanned store based on first sensing information related to the customer's entry and exit from the first sensor (the first sensor detects the customer's entry and exit into the store). (a sensor that detects), receiving second image data captured from a second camera placed in the unmanned store based on second sensing information related to the customer's product selection from a second sensor (the second sensor is (It is a sensor that is arranged in or around a product warehouse that stores products, and detects the customer's product selection behavior for the product warehouse) Based on payment initiation information from the payment terminal in the unmanned store, it is placed in the unmanned store. It includes receiving third image data captured from a third camera and analyzing customer behavior patterns by combining the first image data, the second image data, and the third image data.

Description

Method and device for analyzing behavioral patterns of unmanned store customers using an artificial intelligence-based video analysis model {METHOD AND APPARATUS FOR ANALYZING BEHAVIOR PATTERN OF CUSTOMER IN UNMANNED SHOP USING AI-BASED IMAGE PROCESSING MODEL}

The present invention relates to a pattern analysis method, and more specifically, to a method of analyzing the behavioral patterns of customers entering an unmanned store.

Crime prevention and security can be said to be very important factors in operating an unmanned store. The current crime prevention and security system may include observation through CCTV during open hours, analysis of movements after closing hours, and reporting processes through this. However, it may be difficult to receive user-optimized orders, resulting in inefficiency, and it is difficult to respond quickly and accurately to theft and damage. In particular, in the event of theft or damage while unattended for a long period of time, the manager must replay the entire time section of multiple CCTVs and carefully examine the behavior of all customers who entered during the time, so there is no risk of theft-related incidents. It is very inefficient when it comes to detecting criminals, and in reality, if this method is used, it is almost impossible to find the cause of the theft or damage.

Moreover, in the case of unmanned stores, although disasters due to smoking and drinking occur very frequently, there is no separate manual to respond to them, and the means for storing related information are insufficient.

Therefore, we provide store managers with an optimized usage environment not only in terms of methods and security, but also store management, provide means to respond to theft and damage, and develop technology that can immediately respond to accident events caused by smoking and drinking. This is required.

The purpose of one aspect of the present invention to solve the above-described problem is to use the results of image analysis according to an artificial intelligence-based machine learning model based on sensors provided in the unmanned store and images from a plurality of cameras. The goal is to provide a method and device for analyzing customer behavior patterns in unmanned stores that manages security-related situations and further efficiently extracts information related to store management.

According to an aspect of the present invention for achieving the above object, a method of analyzing customer behavior patterns in an unmanned store on a computing device includes receiving first sensing information related to the customer's entry into the store from a first sensor ( The first sensor is a sensor that detects the entrance and exit of a customer), providing the first sensing information to the first camera (based on the first sensing information, first image data captured from the first camera Receiving, the first image data includes images related to the customer's entry and exit), receiving second sensing information related to the customer's product selection from a second sensor (the second sensor is a (a sensor arranged in or around the product storage and detecting the customer's product selection behavior for the product storage), providing the second sensing information to a second camera, based on the second sensing information, Receiving second image data captured from a second camera (the second image data includes an image related to the customer's product selection), receiving payment information from the unmanned in-store payment terminal, the payment information providing to a third camera, receiving third image data captured from the third camera based on the payment information (the third image data includes images related to the customer's product payment), and It may include analyzing a customer's behavior pattern by combining the first image data, the second image data, and the third image data.

The step of analyzing the customer's behavior pattern includes assigning a label to each individual customer by individually identifying and labeling the customer who entered the unmanned store from the first image data, and labeling the customer. labeling a customer in the second image data and the third image data based on and analyzing customer behavior based on the labeled first image data, the second image data, and the third image data. May include steps.

By comparing the shape of the customer labeled through the first image data with the shape of the customer in at least one of the second image data and the third image data to determine whether they are the same customer, the second image data and Labeling of a customer in at least one of the third image data may be performed.

Determination of whether they are the same customer can be made through at least one of an artificial intelligence-based face tracking algorithm and a video analysis model that tracks a person's facial expressions and clothing.

The training data set for the artificial intelligence-based face tracking algorithm and the video analysis model that tracks people's facial expressions and clothing can be collected through crowd sourcing.

The artificial intelligence-based face tracking algorithm and the image analysis model that tracks a person's facial expression and clothing can be self-trained by generating the actual first to third image data analysis results as a learning data set.

When extracting video data related to the customer's label and analyzing the customer's behavior pattern, (i) the first video data is used to check whether the individual customer enters or exits, and (ii) the second video data is used to check whether the individual customer enters or exits. You can check whether an individual customer has selected a product using , and (iii) you can check whether an individual customer has paid for a product using the third video data.

In response to the fact that only the first image data exists for the first customer, and the second and third image data do not exist, the first customer enters the store but does not select the product or pay for the product (type 1) type) of customer.

In response to the fact that only first image data and second image data exist for the first customer and no third image data, the first customer only enters the store and selects a product, but does not pay for the product. It can be judged as a second type of customer.

In response to the existence of all first image data, second image data, and third image data for the first customer, the first customer is a third type of customer who has entered the store, selected a product, and paid for the product. It can be judged as follows.

The method includes acquiring fourth image data captured based on third sensing information related to exiting the store and labeling individual customers in the fourth image data based on the first image data. More may be included.

The method may further include analyzing the fourth image data to determine whether the labeled customer leaves with the product.

By analyzing the fourth image data and the third image data together, for the second customer labeled in the fourth image data, (i) there is no third image data paying for the product, and (ii) the third customer is labeled in the fourth image data. 4 Customers can be classified into a fourth type according to the fact that they carry out the product in the video data.

The method further includes the step of analyzing the second video data to detect the customer's act of picking up the product, the customer's act of putting the product into the shopping cart, and the customer's act of removing the product from the shopping cart, and generating a segmented image. It can be included.

By analyzing the second video data, information on the type and number of products selected by the customer is obtained, and the obtained information on the type and number of products is compared with the type and number of products in the payment history obtained from the payment terminal, and when payment is made, Missing products can be calculated.

Using an artificial intelligence-based machine learning model, the customer's behavior pattern analysis result and the store sales index can be used as a learning data set to derive the correlation between the customer's behavior pattern analysis result and store sales.

While changing the product display format, the product display format to maximize store sales can be inferred by considering the correlation with store sales derived based on the machine learning model.

According to another aspect of the present invention for achieving the above object, a device for analyzing customer behavior patterns in an unmanned store detects the customer's entry and exit into the store from a first sensor - the first sensor is a sensor that detects the customer's entry and exit. Receiving related first sensing information and providing it to a first camera, thereby receiving first image data captured from the first camera - the first image data includes images related to the customer's entrance and exit - 2 Sensor - The second sensor is arranged in or around a product warehouse storing products, and is a sensor that detects the customer's product selection behavior for the product storage - second sensing information related to the customer's product selection from the sensor. Receives second image data captured from the second camera - the second image data includes images related to the customer's product selection - by receiving and providing it to the second camera, and the unmanned in-store payment terminal A communication unit that receives payment information from and provides it to a third camera, thereby receiving third image data captured by the third camera - the third image data includes images related to the customer's payment for the product - and It may include a processor that analyzes the customer's behavior pattern by combining the first image data, the second image data, and the third image data.

According to another aspect of the present invention for achieving the above object, a method of analyzing customer behavior patterns in an unmanned store on a computing device is based on first sensing information related to the customer's entry into the store from a first sensor, Receiving image data captured from a first camera placed in the unmanned store (the first sensor is a sensor that detects the customer's entry and exit), based on second sensing information related to the customer's product selection from the second sensor Receiving second image data captured from a second camera disposed in the unmanned store (the second sensor is arranged in or around a product warehouse storing products, and the customer selects a product for the product warehouse) (a sensor that detects an action) receiving third image data captured from a third camera disposed in the unmanned store based on payment initiation information from the payment terminal in the unmanned store, and the first image data, the first It may include the step of analyzing the customer's behavior pattern by combining the second video data and the third video data.

According to the method and device for analyzing customer behavior patterns in unmanned stores of the present invention, the behavioral patterns of customers entering and exiting unmanned stores are analyzed using sensing and image data obtained from in-bound equipment in unmanned stores. It has the effect of strengthening the security of unmanned stores and further has the effect of deriving ways to increase sales of unmanned stores.

1 is a conceptual diagram showing an unmanned store system that performs customer behavior pattern analysis according to an embodiment of the present invention;
Figure 2 is a diagram showing an unmanned store system that performs customer behavior pattern analysis according to another embodiment of the present invention;
Figure 3 is a flow chart showing the information exchange process between equipment in the unmanned store of Figure 1 or Figure 2;
Figure 4 is a conceptual diagram illustrating a method of collectively managing customers by labeling video data acquired through multiple camera equipment in an unmanned store;
Figure 5 is a table expressing the results of analyzing the labeled image data of Figure 4;
Figure 6 is a flowchart showing a method of visualizing the video data analysis results so that users can intuitively understand them when the inventory quantity in an unmanned store does not match;
FIG. 7 is a conceptual diagram illustrating a method of efficiently reproducing video data corresponding to a user input during visualization according to the method of FIG. 6;
Figure 8 is a conceptual diagram for explaining the operation of displaying in-store customer behavior patterns in real time through the in-store display equipment and speakers of Figure 1;
Figure 9 is a flowchart illustrating a method of warning smokers through display equipment and speakers in the store when smoke is detected in the store;
Figure 10 is a conceptual diagram illustrating a method of learning an artificial intelligence model that infers the optimal product display method by utilizing data acquired through the system of Figure 1 for unmanned store management;
Figure 11 is a detailed block diagram specifically showing the configuration of a device that constitutes an unmanned store system that performs customer behavior pattern analysis according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention.

Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, the second component may be referred to as a first component without departing from the scope of the present invention. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted in an ideal or excessively formal sense unless explicitly defined in the present application. No.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. In order to facilitate overall understanding when describing the present invention, the same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

1 is a conceptual diagram showing an unmanned store system that performs customer behavior pattern analysis according to an embodiment of the present invention. As shown in Figure 1, the unmanned store system according to an embodiment of the present invention includes a door sensor 112, a camera 120, a product storage sensor 132, a camera 134, a payment terminal 140, It may include a camera 142, a display device 150, a speaker 160, an access point (AP) 170, a proxy server 180, and an integrated server 190.

Referring to Figure 1, the door sensor 112 is a sensor that detects customers entering an unmanned store. This may include at least one of a door opening/closing sensor, an access sensor, a motion recognition sensor, and an infrared sensor. When the door sensor 112 detects the customer's entrance, the camera 120 operates to capture the image of the customer entering the door. At this time, it is desirable to photograph the entire shape of the customer, and in particular, in order to use technology such as face tracking in the future, it is desirable to photograph in the direction in which the shape of the customer's face appears. The door sensor 112 can sense not only the customer's entrance but also the customer's exit, and can send a sensed signal to the camera 120 so that the customer's exit from the store can also be captured.

There may be multiple product storage units 130 within an unmanned store. Although the term product storage 130 is used, it may not be implemented as a box-shaped place for storing products inside, but may simply be implemented as a display stand for displaying products. In other words, it means the unit space where the product is placed. In one example, product storage 130 may include a refrigerator, freezer, display rack, etc. In addition, a sensor 132 is disposed within or around the product storage unit 120 to detect the customer's action of selecting a product for purchase (an action of picking up the product). The sensor 132 includes at least one of a door open/close sensor, an approach sensor, a motion recognition sensor, and an infrared sensor. In particular, in situations where products are stored with the door closed, such as in a refrigerator or freezer, it is desirable to detect the customer's action of selecting a specific product for purchase through a door opening/closing sensor. In an environment that does not require special door opening and closing operations, such as a display shelf, the customer's action of selecting a product can be detected through proximity sensors, motion recognition sensors, and infrared sensors.

Meanwhile, when the customer's action of selecting a product related to the specific product storage 130 is detected through the sensor 132, the corresponding camera 134 photographs the customer selecting the product. One or more sensors 132 and cameras 134 may be provided in each individual product storage 130. Therefore, when the customer's product selection behavior is detected by the sensor 132 of the first product storage 130, the corresponding camera 134 takes pictures, and the customer's product selection behavior is detected by the sensor of the other product storage. When this happens, shooting is done with another corresponding camera. Here too, photography can be performed in a direction looking at the front, back, or top of the customer.

After the customer selects a product, payment is required. Accordingly, payment is made by approaching the payment terminal 140 and entering payment-related user input into the payment terminal 140. At this time, the customer's product payment behavior can be detected through the execution of the payment terminal 140 and/or an access sensor (not shown) separately provided around the payment terminal 140. In response to the customer's payment (initiation) behavior detected in this way, the camera 142 starts filming. The payment terminal 140 includes a kiosk and a POS terminal. Alternatively, it may be a kiosk device that includes POS terminal functions.

According to an embodiment of the present invention, in-bound equipment within a store can communicate with each other using short-distance communication. At this time, short-distance communication may include Wi-Fi, Zigbee, Bluetooth, etc. When performing Wi-Fi communication, information is transmitted and received from each other using an internally distributed IP via the access point 170. That is, the access point 170 allocates individual IPs to the sensors 112 and 132, the payment terminal 140, and the cameras 120, 134, and 142, and remembers each other's IPs among the inbound devices to provide information. can be sent and received. For example, in an embodiment in which inbound devices communicate directly without going through the proxy server 180, the first sensor 112 transmits sensing information to the camera 120 through the IP of the camera 120, and the second sensor 112 transmits sensing information to the camera 120 through the IP of the camera 120. The sensor 132 transmits sensing information to the camera 134 through the IP of the camera 134, and the payment terminal 140 transmits payment initiation information to the camera 142 through the IP of the camera 142. Thus, each camera 120, 134, and 142 can be controlled to activate shooting. The administrator uses the administrator terminal 105 to transmit the sensed information to the individual sensors 112, 132 and/or the payment terminal 140 through the integrated server 190 or proxy server 180. IP information can be determined. And, you can change it whenever you want. In other words, the mapping relationship between the sensor/terminal and the camera can be arbitrarily set. In another example, inbound devices within a store do not necessarily have to use only short-range communications, but also use NarrowBand IoT (NB-IoT), LoRa, oneM2M, Open Connectivity Foundation (OCF), and Lightweight M2M (LwM2M). Information can be transmitted and received between each other or to an external device according to at least one of the standard communication protocols established by global IoT standards such as.

Additionally, sensing information from at least two sensors 112 and 132, image data captured from at least three cameras 120, 134 and 142 (this includes video), and payment terminal 140. Payment initiation and progress information and payment history information may be provided to the proxy server 180. And, the proxy server 180 can provide the corresponding data to the integrated server 190 using a broadband communication network.

The proxy server 180 is a server placed at an unmanned store site. This may exist in the form of an emulated daemon tool on a computing device deployed in an unmanned store. The server is linked with the integrated server 190. This may be a server or program installed in a member store that uses the unmanned store security and business management service provided by the integrated server 190.

However, according to an embodiment of the present invention, the proxy server 180 may not necessarily be needed. Instead of going through the proxy server 180, the sensors 112 and 132, the cameras 120, 134 and 142, and the payment terminal 140 may send related information directly to the integrated server 190.

The integrated server 190 analyzes customer behavior patterns by labeling customers in the video data based on the received sensing information, video data, and payment information, and sends the analysis results to the manager terminal 105 and the store. Controls output through the display equipment 150 and speaker 160. Output to the in-store display equipment 150 and speaker 160 may be performed through the proxy server 180. The display equipment 150 and the speaker 160 may be called an output device, which includes an information acquisition unit (not shown) that obtains the customer behavior pattern analysis results (and a method of expressing the analysis results) and an output unit ( (not shown) may be included. Additionally, the integration server 190 (or proxy server 180) may be called an output control device, a visualization device, a visual expression device, an auditory expression device, an expression control device, etc.

Meanwhile, the customer's behavior pattern is analyzed based on the video captured by the camera 120, the store entry behavior is analyzed, the customer's product selection behavior is analyzed based on the video captured by the camera 134, and the customer's product selection behavior is analyzed based on the video captured by the camera 142. It is derived by comprehensively analyzing the results of analyzing the customer's product payment behavior based on the captured video. Accordingly, some customers only entered and left the store, some customers only entered and selected products but left the store without making payment, and some customers left the store after completing everything from entry to product selection and payment. By checking whether customers have gone to the store, customers can be classified by type, and the classified results can be provided to the manager terminal 105 or the in-store output devices 150 and 160.

Additionally, the integrated server 190 stores the sensing information, image data, and payment information in the database 192 by dividing them by customer, product, and/or time. Then, using the stored information, measures to increase sales can be derived through an artificial intelligence-based machine learning model. One example of the methods derived could be a product display method that suits the customer's product selection tendency.

Meanwhile, functions in the integrated server 190 may be performed in the proxy server 180 placed within the store. That is, the analysis function provided by the integrated server 190 is installed in the proxy server 180 in the form of an agent, so that the proxy server 180 can operate itself to perform analysis and output analysis results. Since the integrated server 190 or proxy server 180 analyzes sensing information and/or image information, it may be called an analysis device, an image analysis device, etc.

The sensors 112 and 132 may be implemented as various types of IoT sensors other than those exemplified above. This may include a temperature sensor, humidity sensor, motion detection sensor, etc. Additionally, in addition to the sensors 112 and 132 and the sensors at the payment terminal 140, one or more other sensors may be present at other locations within the store or around the store. Additionally, at least one of the cameras 120, 134, and 142 can capture images in conjunction with sensors placed elsewhere as described above and transmit them to the integrated server 190.

The manager terminal 105 is a user terminal owned by a manager who performs store management, and may be implemented as a smart phone, desktop, laptop, or PDA (Personal Digital Assistants). The administrator terminal 105 can be linked with the integrated server 190 by installing an application (Application: hereinafter referred to as “app”) provided by the integrated server 190, and the information provided by the integrated server 190 is stored in the app. It can be displayed through the provided user interface (UX/UI). In special cases, the manager terminal 105 may be implemented as a terminal within a store, and the proxy server 180 and the manager terminal 105 may be implemented as a single device.

Figure 2 is a diagram showing an unmanned store system that performs customer behavior pattern analysis according to another embodiment of the present invention.

Referring to Figure 2, the relationship between a sensor and a camera attached to a product storage unit to detect a customer's product selection operation does not necessarily have to be a 1:1 relationship. Relationships can be 1:N, N:1, or N:M.

As in the embodiment of Figure 2, six sensors may correspond to four cameras. At this time, the matching relationship between the individual sensor and the corresponding camera may be predetermined. For example, the first sensor and the second sensor are connected to the first camera, the third sensor and the fourth sensor are connected to the second camera and the third camera, the fifth sensor is connected to the third camera, the fourth camera, and the sixth sensor. may be matched with the fourth camera. This scenario can be set in the integrated server, and can also be set through the integrated server and proxy server using an administrator terminal.

According to an embodiment of the present invention, at least some of the cameras are installed to be rotatable, and when sensing information from the sensor is received, they rotate in a direction toward the sensed product storage and look at the product storage in the rotated state. It can be controlled to start shooting in the viewing direction. This camera rotation includes PTZ (Pan Tilt Zoom) control. In addition, this rotation and PTZ control mechanism can be applied to cameras that film the customer's product selection motion, cameras that film the customer's entering and exiting the store, and cameras that film the product payment motion, etc.

Figure 3 is a flowchart showing the information exchange process between equipment in the unmanned store of Figure 1 or Figure 2.

Referring to FIG. 3, when the entrance sensor senses the customer's position (S310), related information is provided to the first camera (120 in FIG. 1) (S312). The first camera starts shooting based on the sensing information (S314). Filming continues for a preset time. For example, you can shoot for a period of 1 minute. However, it does not necessarily have to be set to 1 minute, and may be set to 2 minutes, 3 minutes, 4 minutes, 5 minutes, 10 minutes, 15 minutes, or 20 minutes. When the first camera captures an image, the captured image data is transmitted to the proxy server (S316). The proxy server transmits the received video data to the integrated server in real time (S318).

When the product storage door opening/closing sensor senses the opening/closing of the door (S320), this is regarded as the customer's product selection behavior, and the sensing information is provided to the second camera (134 in FIG. 1) (S322). The second camera starts taking images using the opening/closing sensing information as a trigger (S324) and provides the video to the proxy server (S326). The proxy server transmits this to the integrated server (S328).

Next, in relation to product payment, when the payment process proceeds at the payment terminal (S330), the payment terminal provides information related to the payment progress to the third camera (142 in FIG. 1) (S332). The third camera uses this as a trigger to start recording video (S334) and provides it to the proxy server (S336). The proxy server transmits this to the integrated server (S338).

In the embodiment of FIG. 3, it is explained that individual sensors directly provide sensing information to the corresponding camera, but instead of directly transmitting and receiving sensing information, information may be transmitted and received through a proxy server. For example, when an access sensor senses a customer's entrance, the sensing information is transmitted to the proxy server, and the proxy server identifies which sensor the sensing information is from and then takes a picture with the corresponding camera (i.e., the first camera). It can operate by sending a request message. In other words, the sensor transmits all sensing information to the proxy server, and the proxy server identifies the sensing information from the sensor, extracts the corresponding camera, and then delivers a shooting request message to the extracted camera. In this embodiment, the proxy server may maintain a mapping table representing the mapping relationship between sensors and cameras. Additionally, the administrator can change or set new settings at will.

Additionally, according to another embodiment of the present invention, when sensing is achieved, shooting may be started, but other operations may be performed while the camera continues shooting. The camera can be controlled to set a reference point for identifying customer behavior patterns or action steps based on sensing information. For example, the first camera may continuously capture images in the direction of the door, and in response to receiving sensing information from the entrance sensor, may recognize the point in time at which the sensing information is received as the point in time at which the customer enters the store. Accordingly, the customer's store entry point can be set based on the sensing information reception point and used for video analysis. For example, videos of a preset period of time before and after the reference point can be divided into store entrance videos related to the customer, and when store entrance videos are needed, the videos of this certain time can be called and used for analysis. In other words, the camera operates by assigning a viewpoint-related standard index to the image based on sensing information.

Figure 4 is a conceptual diagram to explain a method of labeling and collectively managing image data acquired through multiple camera equipment in an unmanned store.

Referring to FIG. 4, the integrated server collects first image data obtained from an entry detection camera, second image data obtained from a camera related to product selection, and third image data obtained from a camera related to product payment, and individual Labeling is performed for each customer present in the video. When more than one customer is present in one video, different labels are attached to each customer. There are two customers in the first video data, one is labeled #1 and the other is labeled #2. Detection of customers in the video is performed through an artificial intelligence (AI)-based machine learning model (hereinafter referred to as an ‘artificial intelligence model’). In other words, it is possible to recognize people in a video and distinguish a plurality of recognized people using face tracking or other image analysis/object detection models that track people's facial expressions and clothing.

After labeling is done in the first image data through the above artificial intelligence model, the labeled customer's image is detected from the second image data (product selection motion shooting data). Then, the same labeling is performed on the corresponding image. That is, a customer labeled as #1 is detected from a plurality of second image data, and the customer in the detected image is labeled as #1. In the same way, a customer labeled as #1 is detected in a plurality of third image data (product payment operation capture data), and the customer in the detected image is labeled as #1.

Then, the video containing the customer labeled #1 is saved to a specific physical location in the database (folder labeled #1). In the same way, videos related to customers labeled #2 are stored in a different physical location (folder labeled #2). When there are multiple customers in a video, one video data can be stored in multiple folders simultaneously. Such labeling may be performed for specific products, and image data for each product may be stored in a predetermined physical location.

Meanwhile, the integrated server can distinguish customer movements using an artificial intelligence-based motion recognition model. For example, a customer's entry and exit may not deviate significantly from the basic entry and exit motions, but the product selection motion may have various motions. For example, there may be actions such as picking up a product, picking it up and putting it back, opening and closing the refrigerator door, or testing various products.

These various movements can be divided into more detailed movements by generating various learning data through methods such as crowd sourcing. In other words, a massive amount of learning data set can be secured by collecting related videos from around the world through crowdsourcing. In addition, self-learning can be performed by generating image data captured while running an actual artificial intelligence model and its analysis results as a learning data set.

At this time, in relation to the segmentation of actions, the second video data should not be a video related to product selection, but rather a video of picking up a product in video 2-1 and an act of putting a product into a shopping cart in video 2-2. Videos can be divided into videos that instruct more detailed actions, such as videos related to removing products from the shopping cart, and can be applied to customer behavior patterns or action steps.

Regarding payment behavior, actions such as removing a product from the shopping cart, placing it at the checkout counter, reading the barcode in the product with a barcode reader, and inserting a payment card can be distinguished. Furthermore, the number of products placed at the checkout can be inferred based on the act of placing items at the checkout, and the number of products paid for can be inferred based on the act of reading the product barcode and payment history, and the difference between the two. By comparing, you can infer the number of products missing during payment.

In addition, in the second image data acquired through the camera 134, information on the type and number of products selected by the labeled customer is compared with the type and number of products paid in the payment history obtained through the payment terminal. Thus, it is possible to calculate whether a missing product exists and the number of missing products.

In particular, through artificial intelligence-based object detection and motion recognition technology, the more granular steps and/or patterns there are, the more useful it becomes as data for theft and security-related analysis and as management data to increase store sales.

Basically, the integrated server analyzes the existence of labeled customers in the first to third image data above and analyzes the customer's behavior to determine whether a specific customer is a customer who only entered the store or a customer who selected the product. You can distinguish whether a customer has made a payment or not. In other words, based on the fact that customer #1 exists in the first video data, the second video data, and the third video data, it can be determined as a customer who entered the store, selected the product, and paid for the product. . Based on the fact that customer #2 exists in the first image data but does not exist in the second and third image data, it can be judged as a customer who entered the store and left without selecting a product or paying for the product. Additionally, a customer who exists in the first and second image data but does not exist in the third image data may be determined as a customer who entered the store and selected a product, but did not pay for the product. At this time, it is desirable to segment the second image data to check whether there is data related to the act of placing the product in the road product storage. Moreover, by using the customer's exit video (fourth video data (not shown)), the customer's behavior pattern can be analyzed more effectively. This is because, through exit video analysis, it is possible to clearly check whether the customer left with the product or not.

In other words, the integrated server uses the first image data to check whether the individual customer enters or exits, uses the second image data to check whether the customer has selected a product, and uses the third image data and/or payment. You can use the information to check whether the customer has paid for the product. In addition, using exit video data, it is possible to finally check whether the customer left with the product or left in the same or similar manner as when he or she entered. At this time, if the bag that the customer carries the product in is a bag with the inside visible, such as a plastic bag, the type and number of products taken can be more clearly distinguished.

In particular, since all video data captured by the camera includes time information, customer behavior over time can be inferred more clearly. In other words, after entering the store, he went to the first refrigerator and selected a product, then went to the third refrigerator, then went to the second refrigerator and selected a product, and after paying for the product, he left at 14:05:12. You can almost completely understand the in-store behavior flow of a series of customers. Since there is sensing information and payment information for each important customer action as above, and there is a camera that operates based on this, the customer's action flow can be combined based on the index related to specific actions in the video.

Figure 5 is a table expressing the results of analyzing the labeled image data of Figure 4.

Referring to FIG. 5, the integrated server can classify behavior patterns into a plurality of preset types through the video analysis method of FIG. 4. In the embodiment of FIG. 4, it is divided into three types from A to C, but it will be obvious to those skilled in the art that it can be subdivided into more types by setting different behavior pattern scenarios.

First, the integrated server can classify customers like #1, who entered the store, selected products, and made payments, into type A. Additionally, customers such as #2 who entered the store and selected a product but did not make a payment can be classified as type B. Additionally, customers such as #3, who only entered the store but did not select products or pay, can be classified as Type C. In other words, using this classification method, the behavior patterns of all customers from the first time point to the second time point set by the user can be analyzed and classified into one of three types.

According to an embodiment of the present invention, the integrated server can control the above three types to be displayed on the display means with different visual expressions. Alternatively, the auditory expression may be different. For example, when displaying the results of customer behavior pattern analysis divided by type as above on the administrator terminal, the server displays Type A customers in blue, Type C customers in yellow, and Type B customers in red. You can control it. Relatively, in relation to theft, type A customers can be seen as those with the most normal purchasing behavior, followed by type C customers, and type B customers can be seen as customers with the most unusual behavior patterns. am. In other words, the reason they entered the store and selected the product but did not pay is because we cannot rule out the possibility that they just took the product out without paying for it.

Meanwhile, as previously mentioned in FIG. 4, the artificial intelligence model is capable of recognizing more diverse motions, and accordingly, it is also possible to set more detailed action types than three based on the detailed actions. For example, by analyzing store exit videos, it is possible to classify types depending on whether a product is taken with it upon exit. In particular, among type B customers, based on the video when leaving, customers who did not take the product with them can be further subdivided into type B-1, and customers who left with the product can be further subdivided into type B-2. At this time, B-2 type customers may be highlighted.

Figure 6 is a flowchart showing a method of visualizing the video data analysis results so that users can intuitively understand them when the inventory quantity in an unmanned store does not match.

Referring to Figure 6, the unmanned store manager can regularly check the inventory quantity (S610). The manager can set the period information from the first time when the inventory quantity has already been confirmed to the second time to confirm the next inventory quantity, and utilize the results of customer behavior pattern analysis. In other words, the inventory quantity within the period set by the manager is compared with the paid product quantity to check whether there are any abnormalities (S620). The inventory at the first point in time must match the inventory at the second point in time plus the paid products to ensure that all products have been paid for and taken out during that period without theft. This can be considered normal. However, if the above two values do not match, it is desirable to look at the results of customers' video analysis in relation to theft.

At this time, when displaying the video analysis results, the types of customers most likely to be involved in theft are displayed first. That is, type B customers are displayed first (S630), then type C customers (S640), and finally type A customers (S650). When displaying, customers of the corresponding type are displayed in a table format, as shown in Figure 5. You can display customer labels in rows and customer action steps in columns. It is okay to do the opposite.

The manager can sort through type B customers first, and then look at type C customers. Then, by looking at Type A customers, it identifies customers who have committed acts related to theft. This process will be described in more detail below with reference to FIG. 7.

FIG. 7 is a conceptual diagram illustrating a method of efficiently reproducing image data corresponding to a user input during visualization according to the method of FIG. 6 .

Referring to FIG. 7, the integrated server controls the customer's behavior pattern to be visualized and displayed, whether it is an administrator terminal, a display means connected to a proxy server, or an in-store display device. At this time, it is desirable to display it in a format that organizes customers by individual type, as in the method of FIG. 6. For example, when only type B customers are listed and displayed, as in the embodiment of FIG. 7, only customers #3, #7, and #8 can be displayed within the set period. These are all customers who entered the store and selected a product, but did not pay for the product.

At this time, the integrated server can further display the customer's stay time from entry to exit by considering the customer's exit video. In another example, as described above, it is possible to analyze and display whether the customer left with the product upon exiting.

Typically, when such a theft incident occurs, the manager plays backwards the CCTV footage installed in the store to find the scene at the time of the incident. In other words, even if the video for a set period is quickly forwarded, it is not efficient because the video in all time periods must be checked one by one. It is also difficult to find a screen of the actual theft incident.

However, according to an embodiment of the present invention, the unmanned store system selects a customer of a specific label on a user interface that displays customers by type in a table, and quickly loads and plays the customer's video data to prevent theft accidents and You can quickly check related videos.

In the embodiment of Figure 7, the manager who recognizes that customer #8's stay time is abnormally long can click customer #8 in the table, load all video data related to that customer, and then play it. At this time, video data may be prepared from all images related to customer #8 (for example, from the first image to the nth image). In addition, since individual video data is extracted and played back for a period of ±1 minute before and after the time when a specific action occurred, it does not take a long time to check it. By using this method, the administrator does not need to manually check the video for the entire set period. Here, the time ±n minutes before and after does not necessarily have to be 1 minute, and may be set to other periods such as 2 minutes, 3 minutes, 4 minutes, 5 minutes, 10 minutes, and 20 minutes. For example, when playing a video of customer #8 entering the store, the time of customer #8 entering the store is used as a reference point, and the video (captured through the camera 120) from 1 minute before entering the store to 1 minute after entering the store video) is played. In relation to product selection, when second video data related to customer #8 exists, videos from 1 minute before product selection to 1 minute after product selection are prepared and played based on the time of product selection. Regarding product selection, since selection can be made by looking through multiple refrigerators (refrigerator 1, refrigerator 3, refrigerator 4, ...), in the video data of multiple product selections, ±1 minute is recorded based on the time of product selection. Multiple videos can be played. Here, in the case of a refrigerator, the time of product selection may be the time of opening or closing the refrigerator door. In addition, if customer #8 is displayed on video data that is projected somewhere other than the door, refrigerator, payment terminal, etc., we prepare data from 1 minute before display to 1 minute after display based on the customer's display time. It can be regenerated. At this time, if the user wants to view the video for a longer time period before or after the reference point, the content for the section 1 minute before or 1 minute after the video can be played by inputting a command from the user.

As previously mentioned in the embodiment of FIG. 5 that it can be divided into more detailed operations, a product selection cancellation operation can be recognized in addition to a product selection on the table of FIG. 7, and the presence or absence of a product selection cancellation operation is displayed on the table. I can do it. In other words, while customers #3 and #8 have both a product selection action and a product selection cancellation action, the analysis results for #7 customer only have a product selection action and no product selection cancellation action can be displayed to the manager. Then, the manager will want to look at the detailed video of customer #7, and by looking at the video related to customer #7, the manager can check where the theft occurred.

According to an embodiment of the present invention, by specifically clicking on the customer label part 710 and the action parts 720-1, 720-2, ... in the table, only the screen corresponding to the corresponding action of the customer is displayed. You can control it. For example, if you click on the customer's label part 710, all videos related to that customer can be prepared. However, if a specific customer clicks on the store entry portion (720-1), only the first video data related to the store entry can be played. If you want to view only the product selection portion, click on the row portion (720-2) that indicates the product selection action within the corresponding customer's column, and only the second video data related to the corresponding action can be played.

FIG. 8 is a conceptual diagram illustrating an operation of displaying in-store customer behavior patterns in real time through the in-store display equipment and speakers of FIG. 1.

Referring to FIG. 8, the integrated server (or proxy server) can analyze customer behavior patterns within the store in real time and output them to a display device and/or speaker within the store. For example, the screen of the display device is divided to display multiple customers, and for currently entered customers, a video of the customer is output along with a customer label. At this time, act ahead of the customer's action stage. It can be identified and output in the same or similar form as pattern analysis. For example, if customer #1 enters, completes product selection, and is currently making a payment at the payment terminal, when the customer's entry is detected, the image from the first camera (120 in Figure 1) is displayed (and also labeled at the same time). When there is product selection sensing information, the second camera (134 in FIG. 1) displays an image of the second camera (134 in FIG. 1) and displays an action step called “product selection”, and payment is made at the payment terminal. When progress is detected, it can be controlled to display the operation stage “paying” while displaying the image of the third camera (142 in FIG. 1). This can have the effect of preventing theft and vandalism by making customers feel that their actions are continuously monitored when they are in the store. The expression of the action step is not only output as a video, but the customer's label and action step information can be controlled to be output by voice through the speaker, such as "Customer number 2 has entered" or "Customer number 3 is selecting a product." there is.

Meanwhile, the video of a specific customer within the display device can be deleted 5 minutes after the customer leaves. At this time, it does not necessarily have to be 5 minutes, and may be set to 1 minute, 2 minutes, etc.

Additionally, as the number of customers in the store increases, it is desirable to further segment the screen and secure a number of smaller screens. This can be implemented adaptively according to the number of customers entering.

Figure 9 is a flow chart to explain a method of warning smokers through display equipment and speakers within the store when smoke is detected within the store.

Referring to FIG. 9, the unmanned store system according to an embodiment of the present invention includes a system that detects and warns of smoking, etc. As described above, the unmanned store system according to an embodiment of the present invention can be linked with various IoT sensors. For example, smoke detection sensors can be arranged in certain areas to detect smoking smoke throughout the entire store. In fact, although smoking in unmanned stores is a big problem, it is not possible to arrest it because there are no people in the stores.

Therefore, the unmanned store system according to an embodiment of the present invention is equipped with a smoking detection sensor, and when smoke is detected, displays a warning message through an in-store display device, as shown in the embodiment of FIG. 8. At this time, the video from the camera corresponding to the labeled current location of the customer is also displayed. In the case of Zone B, there may be a mapping relationship with Camera 1, and in response to smoke detection in Zone B, Camera 1 captures the customer's image, and the captured image is displayed on the display device. At this time, a warning message is displayed by changing the background screen to red along with the phrase “No smoking.” At this time, it is desirable to also output a voice warning through a speaker. For example, a comment such as “Customer number 9, this is a no-smoking area. Please refrain from smoking” can be printed. Despite this primary warning, if smoking continues to be detected (for more than a threshold time) through the smoke detection sensor, the server may take action such as dispatching a security guard in conjunction with a preset security system. Alternatively, a smoking detection message is sent to the administrator's terminal to help the administrator take appropriate action.

Like smoking, drinking alcohol in stores is becoming a major social problem. The unmanned store system according to an embodiment of the present invention can detect that a mainstream product has been selected by a customer by placing sensors (door open/close sensor, access sensor, infrared sensor, etc.) in a location where mainstream products are arranged. When the choice of alcohol is detected, a video of the customer is captured through a camera corresponding to the sensor. At this time, if the act of consuming alcohol (drinking) is detected through an artificial intelligence-based motion detection model, the system Similarly, videos of alcohol consumption and warning messages are displayed together through in-store display devices. Additionally, a warning can be issued through the speaker. Despite the above primary warning, if alcohol consumption continues, the server takes action to dispatch security personnel in conjunction with the pre-set security system or sends an alcohol consumption detection message to the administrator terminal to take appropriate action. You can get drunk.

FIG. 10 is a conceptual diagram illustrating a method of learning an artificial intelligence model that infers the optimal product display method by using data acquired through the system of FIG. 1 for unmanned store management.

Referring to FIG. 10, the integrated server of the unmanned store system according to an embodiment of the present invention analyzes the number of store entries, number of product selections, number of paid products, store sales, etc. along with customer behavior patterns to generate data related to store management. can be derived.

As an example, at least some of the number of store admissions by time, number of product selections, number of products paid, and store sales indicators are used as a learning data set, and the product display is changed (A-B-C-D → B-C-A-D → ...), and the learning data set When input into an artificial intelligence-based machine learning model, the optimal product display method can be inferred to maximize store sales. At this time, parameters related to display can also be entered as input, and when changing the display parameters and inferring the relationship with the store sales index and/or the number of product selections, the store sales index and/or the number of product selections are increased. An artificial intelligence model is learned to infer the optimal product arrangement method. The server infers the optimal product arrangement method using the model learned in this manner.

In particular, optimal product display can be inferred from the relationship between the number of product selections and the number of purchased products. The integrated server can analyze the number of selections and payments for each product, detect products that have been selected a lot but have not yet been paid for, and display this to the manager. For example, product A, ice cream, is often selected, but there may be many cases where it is placed back in the refrigerator after selection. This can be confirmed by recognizing customer movement patterns for each product through the motion recognition artificial intelligence model explained earlier. The server determines that the current display format for these products is not appropriate, can change this, and input the changed product display format as learning data. In addition, result values such as changes in product display format and subsequent increase in sales or whether payment is made without returning after product selection according to change in product display format are generated as learning data to derive the optimal product display format. You can create, learn, and run artificial intelligence models.

In another example, the server extracts the corresponding time zone based on sales indicators by time zone and customer behavior pattern analysis results in response to the fact that there are many store entries at a certain time, but not many product selections and product payments are made. , It can provide help to the manager in the form of recommending events such as product discounts to customers who entered the extracted time zone.

Figure 11 is a detailed block diagram specifically showing the configuration of a device that constitutes an unmanned store system that performs customer behavior pattern analysis according to an embodiment of the present invention. As shown in FIG. 11, the device of the unmanned store system according to an embodiment of the present invention may include a communication unit 1110, a processor 1120, a memory 1130, and an input/output module 1140. Here, the device may be a proxy server or an integrated server.

Referring to FIG. 11, the communication unit 1110 is a component for communicating with other terminals using a wired or wireless network. The communication unit 1210 may include an antenna. The communication unit 1110 receives sensing information from sensors, payment information from a payment terminal, as well as image data from cameras.

The processor 1120 is a component that processes information received through the communication unit 1110. This can be implemented in hardware with a microprocessor. The processor 1120 identifies sensing information from the sensor and transmits a shooting command to the corresponding camera. Then, the collected video data is run through artificial intelligence models such as face recognition, object detection, and motion recognition to label each customer and analyze customer behavior patterns and stages.

The processor 1120 is connected to the memory 1130 through a signal line and executes the program stored in the memory 1130. As described above, the processor 1120 not only analyzes customer behavior related to theft and security, but also links the analyzed customer behavior pattern with sales indicators and processes it into data helpful for store management. At this time, an artificial intelligence-based learning model can also be executed.

The memory 1130 is a storage device that stores instructions related to programs to be performed by the processor 1120 and various data required by the processor 1120. The memory 1130 stores various sensing data obtained from sensors in the store and image data obtained from cameras. In addition, not only the mapping relationship between the sensor and the camera, but also a scenario such as what action to perform as the next action when sensing information is obtained may be stored in the form of a program language. The memory 1130 may be implemented as a local memory inside the device or as an external large-capacity database.

The input/output module 1140 includes information input means such as a keyboard and mouse, and information output means such as a monitor, TV, and touch screen. The input/output module 1140 can be used to change various settings, such as setting a time section subject to customer behavior pattern analysis, and to display analysis results.

The system or device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the systems, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), etc. ), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

Methods according to embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (19)

In a method of analyzing customer behavior patterns in an unmanned store on a computing device,
Receiving first sensing information related to a customer's entry into a store from a first sensor, wherein the first sensor is a sensor that detects the customer's entry and exit;
Providing the first sensing information to a first camera;
Receiving first image data captured from the first camera based on the first sensing information, the first image data including an image related to the customer's entrance and exit;
Receiving second sensing information related to the customer's product selection from a second sensor, wherein the second sensor is arranged in or around a product warehouse storing products, and detects the customer's product selection behavior for the product storage. It is a sensor that does;
Providing the second sensing information to a second camera;
Receiving second image data captured from the second camera based on the second sensing information, the second image data including an image related to a customer's product selection;
Receiving payment information from the unmanned in-store payment terminal;
providing the payment information to a third camera;
Receiving third image data captured from the third camera based on the payment information, the third image data including an image related to the customer's payment for the product; and
Comprising the step of analyzing a customer's behavior pattern by combining the first image data, the second image data, and the third image data,
Detect the occurrence of theft based on inventory at a first point in time, inventory at a second point in time, and the quantity of products paid between the first point in time and the second point in time, and detect the occurrence of the first image data, the second image data, and the Among the third video data, videos of customers of the type related to theft are displayed first, but based on analysis of the detailed behavior of the type of customer related to theft, the video of the customer who is estimated to have walked out with unpaid products is highlighted. A method of analyzing customer behavior patterns in unmanned stores. The method of claim 1, wherein analyzing the customer's behavior pattern comprises:
assigning a label to each individual customer by individually identifying and labeling customers who entered the unmanned store from the first image data;
labeling customers in the second image data and the third image data based on the assigned customer label; and
A method of analyzing customer behavior patterns in an unmanned store, comprising the step of analyzing customer behavior based on the labeled first image data, the second image data, and the third image data. According to claim 2,
By comparing the shape of the customer labeled through the first image data with the shape of the customer in at least one of the second image data and the third image data to determine whether they are the same customer, the second image data and A method of analyzing customer behavior patterns in an unmanned store, which performs labeling of a customer in at least one of the third image data. According to claim 3,
A method of analyzing customer behavior patterns in an unmanned store where the determination of whether the customer is the same is made through at least one of an artificial intelligence-based face tracking algorithm and a video analysis model that tracks a person's impression and clothing. According to claim 4,
A method of analyzing customer behavior patterns in an unmanned store in which the learning data set of the artificial intelligence-based face tracking algorithm and the video analysis model that tracks people's impressions and clothes is collected through crowd sourcing. According to claim 4,
The artificial intelligence-based face tracking algorithm and the image analysis model that tracks a person's impression and clothing are self-learned by generating the actual first to third image data analysis results as a learning data set. A method of analyzing customer behavior patterns in an unmanned store. . According to claim 3,
When analyzing customer behavior patterns by extracting video data related to the customer's label,
(i) Using the first image data, check whether an individual customer enters or exits,
(ii) Using the second video data, check whether an individual customer has selected a product,
(iii) A method of analyzing customer behavior patterns in unmanned stores that checks whether individual customers have paid for products using the third video data. According to claim 7,
In response to the fact that only the first image data exists for the first customer, and the second and third image data do not exist, the first customer enters the store but does not select the product or pay for the product (type 1) A method of analyzing customer behavior patterns in an unmanned store to determine the type of customer. According to claim 8,
In response to the fact that only first image data and second image data exist for the first customer and no third image data, the first customer only enters the store and selects a product, but does not pay for the product. A method of analyzing customer behavior patterns in unmanned stores to determine the second type of customer. According to clause 9,
In response to the existence of all first image data, second image data, and third image data for the first customer, the first customer is a third type of customer who has entered the store, selected a product, and paid for the product. A method of analyzing customer behavior patterns in unmanned stores. According to claim 10,
Obtaining fourth image data captured based on third sensing information related to leaving the store; and
A method of analyzing customer behavior patterns in an unmanned store, further comprising labeling individual customers in the fourth video data based on the first video data. According to claim 11,
A method for analyzing customer behavior patterns in an unmanned store, further comprising the step of analyzing the fourth video data to determine whether the labeled customer leaves with the product. According to claim 12,
By analyzing the fourth image data and the third image data together, for the second customer labeled in the fourth image data, (i) there is no third image data paying for the product, and (ii) the third customer is labeled in the fourth image data. 4 A method of analyzing customer behavior patterns in unmanned stores, classifying customers into 4 types in response to carrying out products in the video data. According to claim 1,
Analyzing the second video data, the unmanned device further includes the step of detecting the customer's actions of picking up the product, the customer's action of putting the product into the shopping cart, and the customer's action of removing the product from the shopping cart, and generating a segmented image. How to analyze customer behavior patterns in stores. According to claim 1,
Analyze the second video data to obtain information on the type and number of products selected by the customer,
A method of analyzing customer behavior patterns in an unmanned store, which compares the type and quantity information of the obtained product with the product type and quantity information of the payment history obtained from the payment terminal to calculate the missing product at the time of payment. According to claim 1,
Using an artificial intelligence-based machine learning model, the customer's behavior pattern analysis results and store sales indicators are used as learning data sets, and the correlation between the customer's behavior pattern analysis results and store sales is derived in an unmanned store. How to analyze customer behavior patterns. According to claim 16,
A method of analyzing customer behavior patterns in an unmanned store that changes the product display format and infers the product display format to maximize store sales by considering the correlation with store sales derived based on the machine learning model. In the device for analyzing customer behavior patterns in an unmanned store,
First sensing information related to the customer's entrance into the store is received from the first sensor - the first sensor is a sensor that detects the customer's entrance and exit - and is provided to the first camera, so that the first sensing information captured by the first camera is received. 1 image data - the first image data includes images related to the entry and exit of a customer - to receive, and a second sensor - the second sensor is arranged in or around a goods storage room storing goods, the goods storage area It is a sensor that detects the customer's product selection behavior - receives second sensing information related to the customer's product selection from and provides it to the second camera, so that second image data captured from the second camera - the first 2. The video data includes a video related to the customer's product selection. Payment information is received from the payment terminal in the unmanned store and provided to a third camera, so that the third video is captured from the third camera. a communication unit that receives data - the third image data includes images related to a customer's payment for a product; and
A processor that analyzes customer behavior patterns by combining the first image data, the second image data, and the third image data,
The processor detects the occurrence of theft based on inventory at a first time point, inventory at a second time point, and the quantity of products paid between the first time point and the second time point, and uses the first image data and the second time point. Among the video data and the third video data, the video of the customer of the type related to theft is displayed first, but based on the analysis of the detailed behavior of the customer of the type related to theft, the customer is estimated to have walked out with the product that has not been paid for. A device for analyzing customer behavior patterns in unmanned stores that highlights and displays videos. In a method of analyzing customer behavior patterns in an unmanned store on a computing device,
Receiving image data captured from a first camera placed in the unmanned store based on first sensing information related to the customer's entry into the store from a first sensor, wherein the first sensor is a sensor that detects the customer's entry and exit lim;
Receiving second image data captured from a second camera placed in the unmanned store based on second sensing information related to the customer's product selection from a second sensor, the second sensor being a product storage unit storing products It is a sensor arranged in or around the product storage and detects the customer's product selection behavior for the product storage; and
Receiving third image data captured from a third camera disposed in the unmanned store based on payment initiation information from the payment terminal in the unmanned store; and
Comprising the step of analyzing a customer's behavior pattern by combining the first image data, the second image data, and the third image data,
Detect the occurrence of theft based on inventory at a first point in time, inventory at a second point in time, and the quantity of products paid between the first point in time and the second point in time, and detect the occurrence of the first image data, the second image data, and the Among the third video data, videos of customers of the type related to theft are displayed first, but based on analysis of the detailed behavior of the type of customer related to theft, the video of the customer who is estimated to have walked out with unpaid products is highlighted. A method of analyzing customer behavior patterns in an unmanned store that further includes the step of displaying.

Images