KR102250504B1 - SYSTEM FOR PROVIDING CUSTOMIZED WELCOME DATA BY RECOGNIZING CUSTOMER USING IoT DEVICES - Google Patents

Abstract

The present invention is installed in the store at least one or more cameras for photographing a facial image of a visiting customer; At least one signal collection terminal installed in a store and collecting an identification code of a smart terminal possessed by a visiting customer; Create and store customer identification information by analyzing the facial image captured by the camera and identification code collected from the signal collection terminal, and match the payment history information for the product purchased by the visiting customer corresponding to the customer identification information with the customer identification information. A service server that stores and stores it; And a store employee terminal that displays at least one of customer identification information and payment history information to the store employee as a terminal installed inside the store or possessed by the store employee at work. It is about an offline store customer service system that identifies visiting customers in an offline store and provides customized customer service information.

Description

Offline store customer service system that identifies visiting customers in offline stores using various IoT devices and provides customized customer service information {SYSTEM FOR PROVIDING CUSTOMIZED WELCOME DATA BY RECOGNIZING CUSTOMER USING IoT DEVICES}

The present invention relates to an offline store customer service system.

In offline store operation, the more important issue than creating new customers is how to maintain and manage existing customers. To solve such a problem, the prior art has proposed various methods of operating a customer loyalty program (eg, earning points, etc.).

However, since the customer loyalty program according to such a conventional technology can check the customer's past payment information only when the customer pays the payment, it is extremely difficult to provide a service that recognizes the customer and responds kindly from the moment the customer visits the store. Was lacking.

Accordingly, the inventors of the present invention came to complete the present invention after long research and trial and error development in order to solve such a problem.

The present invention has been devised to solve the above problems, and an object of the present invention is to provide a high quality customer service before the customer's payment time by checking the customer's royalties from the moment the customer visits the store.

In addition, an object of the present invention is to provide a customer service with uniform quality at all times regardless of who is serving customers by sharing customer identification information and payment history information of visiting customers to store employees.

It is also an object of the present invention to provide a store customer service system that more accurately identifies customers using various IoT devices.

In addition, an object of the present invention is to provide a store customer service system that performs facial recognition based on machine learning to provide a uniform recognition rate despite various clothes, accessories, shooting angles, and lighting.

It is also an object of the present invention to provide a store customer service system that provides recommended products necessary for visiting customers by using customer identification information and payment history information of visiting customers.

A first aspect of the present invention is at least one camera installed in a store to photograph a facial image of a visiting customer;

At least one signal collection terminal installed in a store and collecting an identification code of a smart terminal possessed by a visiting customer;

Create and store customer identification information by analyzing the facial image captured by the camera and identification code collected from the signal collection terminal, and match the payment history information for the product purchased by the visiting customer corresponding to the customer identification information with the customer identification information. A service server that stores and stores it; And

It characterized in that it comprises a store employee terminal that displays at least one of customer identification information and payment history information to the store employee as a terminal installed inside the store or possessed by the store employee at work,

It provides an offline store customer service system that identifies customers visiting offline stores using various IoT devices and provides customized customer service information.

In a preferred embodiment, the payment history information of the visiting customer corresponding to the customer identification information is received from the service server, and a recommended product list is generated by analyzing the product design or product color preferred by the visiting customer based on the payment history information. , It is preferable to further include a fintech interlocking server for transmitting the recommended product list to the store staff terminal.

In a preferred embodiment, the camera includes a first camera disposed at the entrance of the store, and a plurality of second cameras disposed on the customer movement line in the store,

It is preferable that the service server improves the facial recognition rate of the visiting customer by performing machine learning on facial images of the same visiting customer collected from the first camera and the second camera and photographed in various lighting situations or angles.

In a preferred embodiment, the service server analyzes the customer identification information, extracts the number of visits by the visiting customer, the number of visits to the store, and determines the rating of the visited customer according to the number of visits and the visit cycle to determine whether the visiting customer is a regular It is preferable to transmit whether or not to the store staff terminal, and to additionally transmit at least one of the payment history information and recommended product list of the corresponding visiting customer.

In a preferred embodiment, the store staff terminal displays customer identification information of visiting customers who have visited the store on the display screen, and additionally adds at least one of payment history information matching the customer identification information, regularity status, and recommended product list. It is good to mark it.

In a preferred embodiment, the camera is installed on the product shelf inside the store to track the gaze viewed by the visiting customer,

The signal collection terminal is installed on the product display line inside the store to measure the movement line and residence time of visiting customers.

The service server analyzes the line of sight collected from the camera, the movement line collected from the signal collection terminal, and the time of stay, and generates customer interest information on the location of the product in which the visiting customer is interested,

The service server matches the product display map indicating which products are displayed at which locations in the store and customer interest information, creates a list of products of interest or a list of products of interest that the actual visiting customers watched with interest, and sends them to the store staff terminal. It is good.

By the above problem solving means of the present invention, the present invention extracts customer identification information and payment history information from the moment the customer visits the store and checks the customer's royalties, thereby providing high-quality customer service before the customer's payment time. can do.

In addition, the present invention can share customer identification information and payment history information of visiting customers to store staff to provide customer service of uniform quality at all times regardless of who is serving customers.

In addition, the present invention can provide a store customer service system that more accurately identifies customers by using various IoT devices.

In addition, the present invention can provide a store customer service system that provides a uniform recognition rate despite various clothes, accessories, shooting angles, and lighting by performing facial recognition based on machine learning.

In addition, the present invention can provide a store customer service system that provides recommended products necessary for visiting customers by using customer identification information and payment history information of visiting customers.

On the other hand, even if it is an effect not explicitly mentioned herein, it is added that the effect described in the following specification expected by the technical features of the present invention and the provisional effect thereof are treated as described in the specification of the present invention.

1 is a view showing a preferred embodiment of the store customer service system of the present invention.
2 is a diagram showing a preferred embodiment of the service server of the present invention.
3 is an example for explaining the effect of customer service using the present invention.
※ The accompanying drawings reveal that they are exemplified by reference for an understanding of the technical idea of the present invention, and the scope of the present invention is not limited thereto.

In describing the present invention, when it is determined that the subject matter of the present invention may be unnecessarily obscured as matters that are apparent to those skilled in the art for related known functions, detailed descriptions will be omitted.

1 is a view showing a preferred embodiment of the store customer service system of the present invention.

As can be seen in Figure 1, the store customer service system 100 of the present invention includes a camera, a signal collection terminal, a store staff terminal, a service server, and a fintech interlocking server.

At least one camera is installed in the store to capture facial images of visiting customers. The location where the camera is installed may be an entrance, and additionally, it may be installed on shelves and counters. Cameras installed at the entrances acquire facial images of visiting customers entering or leaving the store. The camera installed on the display rack acquires a facial image of a visiting customer staring at a product in front of the product display rack installed inside the store. In addition, the camera installed on the shelf can track the gaze of visiting customers. The camera installed at the counter acquires a facial image of a visiting customer who pays in front of the counter.

At least one signal collection terminal is installed in the store to collect the identification code of the smart terminal possessed by the visiting customer. In a preferred embodiment, the signal collection terminal is an IoT device that collects an identification code of a smart terminal using a Bluetooth search or Wi-Fi search function. In a preferred embodiment, the identification code collected by the signal collection terminal includes the MAC address of the smartphone, and may include various unique codes capable of identifying the device of the smartphone. Meanwhile, in the present specification, the term smart phone may be understood as a concept including a terminal device carried by a user such as a smart terminal, for example, a tablet PC, a wearable device, etc. in addition to a smart phone. A plurality of signal collection terminals are installed on the shelves inside the store to measure the movement and residence time of visiting customers. To this end, the signal collection terminal measures the strength of a Wi-Fi or Bluetooth signal generated from the smart terminal of the visiting customer. The signal collection terminal transmits the collected identification code of the smart terminal, the movement line of the visiting customer, and the time of stay to the service server.

The store employee terminal is a terminal installed inside the store or possessed by store employees who are working, and is an IoT terminal that includes a display screen and a communication module. In a preferred embodiment, the store employee terminal may be a POS terminal installed inside the store, or may be a smartphone held by the store employee. The store employee terminal displays at least one of customer identification information and payment history information to the store employee. In a preferred embodiment, the store staff terminal displays customer identification information of visiting customers who visited the store on the display screen, and additionally displays at least one of payment history information matching the customer identification information, whether a regular user, or a recommended product list. I can.

The service server generates and stores customer identification information by analyzing the facial image captured by the camera and the identification code collected from the signal collection terminal, and identifies payment history information for the product purchased by the visiting customer corresponding to the customer identification information. It matches the information and stores it. The service server transmits at least one of customer identification information and payment history information to the store employee terminal so that the store employee using the store employee terminal can use the information to provide a uniform store customer service. A more detailed description of the service server will be described with reference to FIG. 2.

The fintech interlocking server receives payment history information of visiting customers corresponding to customer identification information from the service server, analyzes the product design or product color preferred by visiting customers based on the payment history information, and creates a recommended product list, The recommended product list is sent to the store staff terminal.

2 is a diagram showing a preferred embodiment of the service server of the present invention.

As can be seen from FIG. 2, the service server of the present invention includes a facial image receiving unit, a machine learning unit, an identification code receiving unit, a customer identification information generation unit, a payment history information storage unit, a customer rating determination unit, and a customer service information transmission unit.

The facial image receiving unit receives the facial image of the visiting customer collected by the camera. In a preferred embodiment, the facial image receiving unit receives the unique number of the camera together with the facial image so that it is possible to distinguish an image obtained from any of the plurality of cameras. In addition, the facial image receiving unit may receive the facial image recording time together with the facial image.

The machine learning unit analyzes the facial image received by the facial image receiving unit and recognizes the face of the visiting customer. For this, the machine learning unit first extracts facial features from the facial image. In a preferred embodiment, the machine learning unit analyzes the coordinates of the eyes, nose, and mouth to extract facial features and stores numerical data. In an embodiment, the machine learning unit may use OpenCV's Dlib, CLM-Framework, etc. to analyze facial characteristics, but is not limited thereto.

The machine learning unit is a method of performing machine learning and deep learning using a plurality of facial images and facial feature data acquired from a plurality of cameras photographing a visiting customer in various lighting situations or angles. Improves the accuracy of facial recognition.

In a preferred embodiment, the machine learning unit performs machine learning using Generative Adversarial Networks (GAN) and improves the accuracy of facial recognition. The'Generative Adversarial Networks (GAN)' theory is a machine learning method in which different artificial intelligences (AI) compete with each other to improve mutual performance.

The process of improving the accuracy of facial recognition through a generative hostile neural network (GAN) is as follows.

In a generative hostile neural network (GAN), there is a generator that creates an image by itself and a discriminator that discriminates the image.

The creator creates an image close to reality, and the delimiter discriminates whether or not the creator's image is true using the facial image, which is an actual image. The constructor is programmed to deceive the delimiter, and the delimiter is programmed to better discriminate the images created by the constructor. In the process of competing between the two artificial intelligences, more accurate images are created.

In other words, in the generative hostile neural network (GAN), there is a “generator” that creates an image by itself and a discriminator that discriminates whether the image is real or fake, and the two compete. The creator creates content that is as close as possible to the real thing like a currency counterfeiter, and the separator discriminates whether it is close to the real thing like a policeman. If the creator and the classifier go through a competitive process, the machine can create content that is close to the correct answer without requiring supervised learning.

According to the above generative adversarial neural network theory, the machine learning unit of the present invention allows the creator to generate various virtual facial images of the visiting customer, and allows the separator to discriminate whether the virtual facial image is true based on the actual facial image. do. By doing this, the machine learning unit can more accurately recognize which visitor's face is the face of a specific visitor even if a new facial image (for example, if there is a change such as thickening makeup or wearing a new accessory) is input. have.

To this end, the machine learning unit may include a generation network unit that generates a virtual facial image using a generative hostile neural network and a discrimination network unit that verifies the generated virtual facial image. The generation network unit performs a function as a generator to generate a virtual facial image, and the discrimination network unit functions as a discriminator to verify the virtual facial image generated by the generation network unit, and the generation network The unit and the discrimination network unit perform functions competitively with each other, so that facial recognition becomes more accurate.

The present invention does not disclose a specific algorithm for improving the accuracy of facial recognition by generating a virtual facial image in the generation network unit for technical know-how, verifying the virtual facial image in the identification network unit, and performing it repeatedly by competing. I decided not to. In the present invention, the number of times the generating network unit and the discriminating network unit compete with each other can be limited to within a predetermined number in order to quickly derive the facial recognition result.

The identification code receiver receives the identification code of the smart terminal collected by the signal collection terminal. In a preferred embodiment, the identification code receiving unit receives the unique number of the signal collection terminal together with the identification code so that it can distinguish which of the plurality of signal collection terminals the identification code obtained from the signal collection terminal. In addition, the identification code receiving unit may receive the identification code confirmation time together with the identification code.

The customer identification information generation unit acquires the camera's unique number, facial image, and facial image shooting time from the facial image receiver, and acquires the signal collection terminal's unique number, the smart terminal's identification code, and the identification code verification time from the identification code receiver. The customer identification information generation unit stores information on which location of which store the camera with a specific unique number is installed in advance and information on which location of which store the signal collection terminal with a specific unique number is installed. . The customer identification information generation unit may use this information to distinguish whether the facial image acquired from the unique number of a specific camera is the facial image acquired from which location in a certain store. In addition, the customer identification information generation unit may use such information to distinguish whether the identification code of the smart terminal obtained from the unique number of the specific signal collection terminal is the identification code obtained from which location of a certain store. The customer identification information generation unit obtains a face recognition result from the machine learning unit that determines whose face the facial image is.

The customer identification information generation unit may synthesize the information listed above, extract information on who the visiting customer is, where they entered and moved, and how long they stayed in front of which display stand and store it in the customer identification information.

The payment history information storage unit collects payment information of a specific visiting customer and stores it as payment history information. In a preferred embodiment, the payment history information may include information on the type, color, quantity, etc. of payment products, payment time, payment method, and the like.

The customer rating determination unit acquires customer identification information from the customer identification information generation unit and obtains payment history information from the payment history information storage unit. The customer rating determination unit analyzes customer identification information to extract the number of visits to the store and the number of visits by the corresponding visiting customer, and analyzes the payment history information to extract payment products, payment cycles, and the number of payments. The customer rating determination unit analyzes such information, determines the rating of the visiting customer, sends whether or not the visiting customer is a regular, to the store staff terminal, and adds at least one of the visiting customer's payment history information and the recommended product list. I can send it.

The customer service information transmission unit transmits the information received from the customer grade determination unit to the store staff terminal.

In another embodiment, the service server may generate customer interest information on which location of the visiting customer is interested in viewing the product by analyzing the line of sight collected from the camera, the movement line collected from the signal collection terminal, and the residence time. The service server matches the product display map indicating which products are displayed at which location in the store and customer interest information to generate a list of products of interest or a list of products of interest that the actual visiting customers watched with interest, and transmit them to the store staff terminal. I can. Such a list of interested products is information different from a list of recommended products. Because the recommended product list analyzes the conventional payment history information and recommends products based on the past payment history of the visiting customer, the interest product list is based on which product the customer who visited the store on the same day watched the product in front of and for how long. Or, it is because it recommends related products.

3 is an example for explaining the effect of customer service using the present invention.

As can be seen in FIG. 3, if there is no system for identifying who the visiting customer is, the store employee cannot know who the visiting customer is, what tastes, and what products have been paid in the past. Therefore, there is no choice but to provide a formal store customer service. For example, when a customer visits, “I'll help you order”, “Add steak, chicken teriyaki, lunch sauce to chiyabata, and olive, pepper, and jalapeno. Oh, please put plenty of vegetables”, “Do you have a point card”, “Yes, I will enter it by number”. This is because it is impossible to know what kind of food taste the customer has or what point card they have. It is also similar in the case of a shoe store. For example, when a customer visits, we have no choice but to provide customer service such as "Hey, show me this", "Yes, what's the size?", "It's 235", "Yes, please wait." This is because it is impossible to know who the customer is and what the shoe size is.

On the other hand, when the present invention is used, the store customer service is greatly different. For example, when a visiting customer enters the store, the visiting customer visits the store employee terminal (possess terminal or, preferably, an IoT terminal carried by the store employee, a smartphone, etc.) 5 times, and the total purchase amount is 86,500 won. , Information such as the type of food as the purchase history and plenty of lettuce as a customer characteristic is displayed. Therefore, the store staff can check this and provide store customer service such as “Hello, can I prepare you with plenty of lettuce as you always eat?”, “Yes, thank you”, and “I have accumulated points”. This is because you can know the tastes of visiting customers and information on point cards in advance. Likewise, when a visiting customer enters a shoe store, the total number of visits is 3 times, the purchase amount is 430,000 won, the product type as the premises, and the foot size and foot ball as customer characteristics are displayed on the store staff terminal. Therefore, the store staff said in advance, “You have a size of 235, but this shoe has a narrow foot, so it would be nice to wear 240. We can provide an improved level of in-store customer service, such as "I'll show you both".

[Other Examples]

The service server can transmit to the store employee terminal whether the visiting customer who entered the store is a regular customer, and the store employee terminal sounds a specific pattern of vibration when the visiting customer is a regular customer, even if the store employee does not visually check the store employee terminal. You can check whether the visiting customer is a regular customer.

In addition, a speaker may be installed in the store to receive customer information from a service server and output a greeting when a visiting customer enters or leaves. Using the speaker, it is possible to provide a new user experience in which visiting customers hear their name at the store when they enter.

The scope of protection of the present invention is not limited to the description and expression of the embodiments explicitly described above. In addition, it is added once again that the scope of protection of the present invention may not be limited due to obvious changes or substitutions in the technical field to which the present invention pertains.

Claims (1)

At least one camera installed in a store to photograph a facial image of a visiting customer;
At least one signal collection terminal installed in a store to collect an identification code of a smart terminal possessed by a visiting customer;
By analyzing the facial image captured by the camera and the identification code collected from the signal collection terminal, customer identification information is generated and stored, and the payment history information for the product purchased by the visiting customer corresponding to the customer identification information is matched with the customer identification information. A service server that stores and stores; And
A terminal installed inside a store or possessed by a store employee who is working, and includes a store employee terminal that displays at least one of customer identification information and payment history information to the store employee,
Receives the payment history information of the visiting customer corresponding to the customer identification information from the service server, analyzes the product design or product color preferred by the visiting customer based on the payment history information, creates a recommended product list, and stores the recommended product list Further comprising a fintech interlocking server for transmitting to the employee terminal,
The camera is installed on the product shelf inside the store to track the gaze of visiting customers,
Cameras and signal collection terminals are arranged for each of a plurality of division areas formed by dividing the interior of the store,
The service server includes a customer identification information generation unit,
The customer identification information generation unit,
In advance, information on the location of which store the camera with a specific ID number is installed, and information about the location of the store where the signal collection terminal with a specific ID number is installed, are stored,
Using this information, it is possible to distinguish whether the facial image acquired from the specific number of a specific camera is the facial image acquired from a certain location in a certain store, and the identification code of the smart terminal acquired from the specific number of the specific signal collection terminal is It is possible to distinguish the identification code obtained from which location,
The service server,
By using the facial images and identification codes collected by the camera and signal collection terminal together, a list of products of interest that can accurately identify the products of interest viewed by customers is created, and
The signal collection terminal is installed on the product shelf inside the store and measures the strength of the identification code of the smart terminal possessed by visiting customers to measure the movement line and residence time of visiting customers.
The service server analyzes the line of sight collected from the camera, the movement line collected from the signal collection terminal, and the time of stay, and generates customer interest information about the location of the product in which the visiting customer is interested,
The service server matches the product display map indicating which products are displayed at which locations in the store and customer interest information, creates a list of products of interest or a list of products of interest that the actual visiting customers watched with interest, and sends them to the store staff terminal. ,
The camera includes a first camera disposed at the entrance of the store, and a plurality of second cameras disposed on the customer movement line in the store,
The service server improves the facial recognition rate of the visiting customer by performing machine learning on the facial images of the same visiting customer collected from the first camera and the second camera and photographed in various lighting situations or angles,
The service server includes a machine learning unit that analyzes the facial image received from the facial image receiving unit and recognizes the face of the visiting customer,
The machine learning unit performs machine learning using Generative Adversarial Networks (GAN) including a generator that creates an image and a discriminator that discriminates the image to improve the accuracy of facial recognition. ,
The creator generates various virtual facial images of visiting customers,
The separator discriminates whether or not the virtual facial image is true,
The service server analyzes the customer identification information, extracts the number of visits by the visiting customer, the number of visits to the store, and determines the level of the visited customer according to the number of visits and the visit cycle, and determines whether the visiting customer is a regular customer to the store staff terminal. And additionally transmit at least one of the payment history information and recommended product list of the visiting customer,
The store staff terminal displays customer identification information of visiting customers who visited the store on the display screen, and additionally displays at least one of payment history information matching the customer identification information, regularity, and recommended product list,
The service server transmits to the store employee terminal whether the visiting customer who entered the store is a regular customer, and the store employee terminal sounds a specific pattern of vibration when the visiting customer is a regular customer, even if the store employee does not visually check the store employee terminal. Offline store customer service system that identifies customers visiting offline stores using various IoT devices and provides customized customer service information, characterized in that it allows customers to check whether they are regulars.

Images