KR102060694B1 - Customer recognition system for providing personalized service - Google Patents

Abstract

The present invention relates to a customer recognition system for providing a personalized service, and more specifically, based on cameras installed in various stores, collects information about the face and movement of the customer to generate customer information for a specific customer identifier, Based on the customer information, we collect the order history of customers who have visited various stores to analyze customer tastes and provide personalized services that can enhance customer convenience and satisfaction by providing services related to the store when customers visit specific stores. A customer recognition system for providing. The present invention manages customers by recognizing their faces through deep learning-based artificial intelligence technology, and analyzes behaviors such as order history and movement route of customers based on them, and based on this, a menu suitable for customers visiting specific stores By recommending to the customer has the effect of increasing the convenience and satisfaction of customers using the store.

Description

Customer recognition system for providing personalized service

The present invention relates to a customer recognition system for providing a personalized service, and more specifically, based on cameras installed in various stores, collects information about the face and movement of the customer to generate customer information for a specific customer identifier, Based on the customer information, we collect the order history of customers who have visited various stores to analyze customer tastes and provide personalized services that can enhance customer convenience and satisfaction by providing services related to the store when customers visit specific stores. A customer recognition system for providing.

Currently, a customer management system that integrates and manages various stores serving food, collects order details of users who visited the stores and ordered food from a server, and provides various services based on the collected contents.

The customer management system analyzes the customer's taste based on the customer's order history, and based on this, recommends a menu suitable for the customer to support the customer's satisfaction with using the store.

However, the existing customer management system manages customers who visit a plurality of stores serving the same menu based on the member information of a user who is registered in advance. In addition to the difficulty of managing the stores that are in addition to the stores that serve these other products, to integrate and manage the member information of a specific customer and to share and manage the member information between stores, but the store already has its own In order to provide customer service, customer-related information is being organized, which makes it difficult to integrate customer-related information between different stores.

As a result, the existing customer management system is limited to a limited service menu provided by stores to be managed, thereby grasping the customer's taste, thereby greatly reducing scalability, and thus, it is difficult to cope with the constantly changing customer taste.

Korean Patent Publication No. 10-2012-0105589

The present invention recognizes the customer's face through deep learning-based artificial intelligence technology and manages the customer based on the analysis, and analyzes the behavior such as the order history or the movement route of the customer, and is suitable for the customer who visited the specific store based on this. Its purpose is to increase the convenience and satisfaction of customers using the shop by recommending the menu to customers.

In addition, the present invention has an object to increase the convenience of payment by automatically recognizing the customer visiting the store to support automatic payment when the customer uses the store after payment.

The customer recognition system for providing a personalized service according to the present invention includes a plurality of camera units installed at a place where a plurality of stores are located, a plurality of store terminals respectively corresponding to a plurality of stores, and a communication network with the store terminals. And a service server configured to receive an image of at least one of the plurality of camera units from the store terminal or directly receive an image from at least one of the plurality of camera units, wherein the service server includes a preset KLT in the image. Based on the tracker (Kanade-Lucas-Tomashi tracker), feature point information for each face feature point is extracted and stored, and feature point information extracted from a plurality of different images is clustered according to a preset deep learning algorithm. Do a unique identifier that corresponds to one or more feature point information for And the customer information and the order information including a unique identifier of a specific customer identified based on the feature point information in the image information based on the image information received from the store terminal and the order information on the order product of the customer. Accumulate and store the matching, and generate the taste information of the customer based on the cumulatively stored order information corresponding to the customer.

As an example related to the present invention, the service server receives image information and payment means information for the payment means of the customer from the store terminal, and the feature point information extracted from the image information is matched with a unique identifier for each customer. After identifying the unique identifier of the customer corresponding to the feature point information compared to the stored existing feature point information, and registering the payment means information in the customer information including the unique identifier of the customer identified corresponding to the feature point information from the store terminal Whenever the image information and the order information are received, the payment means information is extracted from the customer information corresponding to the unique identifier identified based on the feature point information extracted from the image information, and based on the extracted payment means information and the order information. Automatic payment for purchases of your order based on your order information It can be characterized by processing.

As an example related to the present invention, the service server tracks a specific customer through the facial feature point extraction based on the image information received from the plurality of camera units and the store terminal, and generates copper line information for a moving line. Matching and storing the customer information of a specific customer, and suitable for the taste of the specific customer targeting a plurality of stores located in the place based on pre-stored store information for each of a plurality of stores and the taste information of the specific customer. Select one or more recommended stores, and calculate the optimal copper line based on the location of each store according to the store information of one or more recommended stores selected in response to the taste of the specific customer and the copper line information accumulated in response to the specific customer. To generate optimal copper line information.

As an example related to the present invention, the customer information may include identification information of a customer terminal corresponding to a customer, and the service server may transmit the optimal copper wire information to the customer terminal.

As an example related to the present invention, the service server may provide a recommendation menu suitable for the specific customer based on the store information for each of the recommended stores selected corresponding to the specific customer based on the taste information of the customer and the taste information of the specific customer. And selecting recommendation menu information on the recommendation menu and transmitting the recommendation menu information to the customer terminal.

As an example related to the present invention, the store terminal receives image information from at least one of the plurality of camera units to identify a customer by recognizing or directly recognizing a face of a customer included in the image information with the service server. And registering the identified store use history of the customer in the shared ledger shared between the plurality of store terminals, so that sharing and mutual verification of the shared ledger including the store use history is performed through the communication between the plurality of store terminals. And, the service server may be characterized in that to provide a customized diet, diet information or a discount service to the specific customer based on the shared ledger.

The present invention integrates and manages a plurality of stores by unifying the same customers previously managed individually in each of a plurality of stores selling different types of products with unique identifiers based on facial feature points of the customers detected in the image. As a result, it is possible to easily analyze the customer's taste by product type, and based on this, it is possible to provide a more detailed and precise personalized service.

In addition, the present invention manages customers by recognizing their faces through deep learning-based artificial intelligence technology, and analyzes behaviors such as order history or movement route of customers based on them, and based on this, to customers visiting specific stores By recommending a suitable menu to customers, it is effective to increase the convenience and satisfaction of customers using the store.

In addition, the present invention has an effect of automatically recognizing the customer visiting the store to support the automatic payment when the customer uses the store after payment to increase the convenience of payment.

In addition, the present invention can unify and manage the customer management of a plurality of stores that sell different products based on a unique identifier given based on a facial feature point of a customer extracted from an image. At the same time, it is effective to increase the convenience of the user.

In addition, the present invention can automatically identify one or more stores that sell products preferred by the customer to customers who visited a specific place where a plurality of stores are located, and generated by identifying and tracking the customers through a plurality of camera-specific images. Considering customers' existing moving routes, we can provide customers with the optimal route to visit the one or more stores, which can greatly increase the customer's satisfaction by recommending menus for each store along with their shopping efficiency and convenience. have.

1 is a configuration diagram of a customer recognition system for providing a personalized service according to an embodiment of the present invention.
2 is a block diagram of a facial feature point extraction process of a service server according to an embodiment of the present invention.
3 and 4 are examples of operations for a customer identification process and an automatic payment processing process based on facial feature points of the customer recognition system for providing a personalized service according to an embodiment of the present invention.
5 to 7 are views illustrating an operation for identifying a moving line of a customer recognition system for providing a personalized service according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a configuration of a customer recognition system for providing a personalized service according to an embodiment of the present invention, a plurality of camera unit is installed in a specific place where a plurality of stores selling goods or services as shown 10, a plurality of store terminals 20 (or merchant terminals) connected to at least one of the plurality of camera units 10 and corresponding to a plurality of stores, a customer terminal 30 carried by a customer, and It may include a service server 100 for communicating with the store terminal 20 and the customer terminal 30 through a communication network through a communication network.

In this case, some of the plurality of camera units 10 may be located in a store, and may generate image information including a face of a customer visiting a specific store and transmit the image information to the store terminal 20.

In addition, the service server 100 may communicate with at least one of the plurality of camera units 10 through a communication network, and may receive image information from the camera unit 10. In this case, the camera unit 10 may be configured as an IP (Internet Protocol) camera.

In addition, the store terminal 20 may be configured as a point of sale (POS) device.

In addition, the store terminal 20 of a specific store is connected to the camera unit 10 installed in the specific store to receive the image information received from the camera unit 10, the communication server to the service server 100 The image information can be transmitted through the.

In addition, the service server 100 may recognize the face of the customer based on the image information received from the camera unit 10, and generate the customer information based on the recognized customer face.

In this case, the service server 100 may generate a unique identifier for the first recognized customer face and include the unique identifier in the customer information.

In this way, the service server 100 may classify customers using face recognition.

In this case, the face recognition technology is the most natural biometric technology because the face is most frequently used when a human recognizes another person. Facial recognition is recognized by analyzing about 50 parts of the face skeleton, such as the nose, mouth, eyebrows, and chin, rather than the entire face.

In addition, face area extraction process Face recognition is performed through two processes. It consists of the process of classifying faces in the whole image and the process of discriminating whose face is the face found. In the process of classifying the face in the whole image, the feature-based method using the infrared camera and the image-based method using the 3D face image is used.

In the process of discriminating which face is a face, the inputter is an unknown face to identify and identify a face and whether the face is his or her face by comparing with a stored database. The service server 100 Finds the pictures matched with the input pictures from the face-related feature point information already stored, and confirms or rejects the inputter's identity by matching the input face with the stored face data.

Provide personalized information (marketing) through face recognition, lie detection, crime suspect detection (detection of criminal suspects using mobile terminals), security authentication, entertainment (finding similar people), portrait photo management support, drowsiness driving prevention, etc. do.

The advantage of face recognition technology is the low reluctance to use the camera, and the user's follow-up can be traced since the user's photo is naturally submitted for reading.

Meanwhile, the service server 100 may communicate with the plurality of camera units 10 through a communication network, recognize a face of a specific customer based on the image information received from the camera unit 10, The face information may be compared with previously stored customer information to identify a specific customer based on the customer information that matches the face information.

Based on the above configuration, a detailed configuration of a customer recognition system for providing a personalized service according to an embodiment of the present invention will be described.

First, FIG. 2 is a block diagram of a customer information generation process of the service server 100 constituting a customer recognition system for providing a personalized service according to an embodiment of the present invention.

As shown, the service server 100 may receive a series of image information from the store terminal 20 and the camera unit 10 connected to the camera unit 10, and the corresponding image information to the service server It may be stored in an image DB (not shown) included in the 100.

In this case, the video DB may be included in the service server 100.

On the other hand, the service server 100 obtains a foreground removed from the image information stored in the image DB through image analysis, and in the foreground to identify the face region based on the KLT tracker (Kanade-Lucas-Tomashi tracker) Then, the KLT feature point may be detected from the face area.

In addition, the service server 100 in the surveillance space (monitoring target space) of the specific camera unit 10 via the KLT tracker preset in the image of the specific camera unit 10 of the plurality of camera unit 10. The KLT feature points detected for each location of the same object located may be collected to perform calibration on the specific camera unit 10, thereby performing a three-dimensional position on the surveillance space and the specific camera unit 10. Mapping information may be generated by mutually mapping a location in an image.

That is, the service server 100 may generate the mapping information corresponding to each of the plurality of camera units 10 through the above-described configuration, and through this, the object displayed in the image received from the specific camera unit 10. The KLT feature point detected in the face region may be converted into a 3D feature point based on the mapping information to extract and generate 3D feature point information on the face detected in the image of the specific camera unit 10.

Hereinafter, the feature point information described in the present invention may mean three-dimensional feature point information.

In this case, the service server 100 may store mapping information generated through calibration for each camera unit 10 in the image DB, and the mapping information may include identification information of the camera unit 10.

Accordingly, the service server 100 is based on the mapping information matching the identification information of the camera unit 10 included in the image information received from the camera unit 10 or the store terminal 20 in the image information 3 After the dimensional feature point is extracted, the feature point information may be generated accordingly.

On the other hand, as shown in Figure 3, the service server 100 is a customer DB to the feature point information consisting of the three-dimensional feature points extracted from the face of a specific customer identified through the KLT tracker in the image information stored in the image DB Can be stored at 101.

In addition, the service server 100 is generated through the above-described configuration based on the plurality of video information received from the plurality of camera units 10 and the store terminal 20 and stored in the customer DB 101 to each other. Objects having high similarity may be clustered and classified and classified through a deep learning algorithm (or artificial intelligence algorithm) preset for other feature point information.

In this case, the service server 100 may assign (allocate) a unique identifier corresponding to a specific cluster consisting of a plurality of feature point information clustered through the clustering, and the unique identifier is a member identifier (member ID) of a specific customer. It can be given as.

In addition, the specific cluster may mean facial feature points for a specific customer corresponding to a unique identifier matched with the specific cluster.

In addition, the service server 100 may generate and store customer information including the unique identifier in the customer DB 101, and store the customer information in the customer DB 101 by matching the customer information with the specific cluster.

Through the above-described configuration, the service server 100 may learn a facial feature point of a customer based on a deep learning algorithm, set the face feature point as a unique identifier of the customer, and set a specific customer to another customer based on the facial feature point. Can be distinguished from.

On the other hand, as shown in Figure 4, the service server 100 in the image information including the object for a particular customer of the image information received from the plurality of different camera unit 10 to the face of the specific customer Extract the feature point information, and compare the feature point information with the feature point information of the customer DB 101 to identify the feature point information in the customer DB 101 that matches the feature point information extracted from the image. The specific customer may be identified based on the unique identifier of the customer corresponding to (matched) the feature point information identified at 101.

In addition, the store terminal 20 of the specific store selling the product is the order customer in the order received from the camera unit 10 connected to the store terminal 20 with the order information on the order details of the order customer currently ordering The close-up image information may be transmitted to the service server 100.

In this case, the service server 100 extracts feature point information on the face of the order customer who is currently ordering based on the image information received from the store terminal 20 to obtain customer information of the order customer through the above-described configuration. DB 101 can be identified, and if the payment means information corresponding to the identified order customer does not exist in the customer DB 101 to the store terminal 20 the payment means related information of the identified order customer Transmits request information for requesting the request information, and receives payment means information related to a payment means for use by the order customer for the purchase of an order history (or an ordered product) from the store terminal 20 to the customer information of the order customer. It can be included to register the payment method of the order customer.

By doing so, the service server 100 receives the order information and the image information from the store terminal 20 of the other store when the order customer registered with the payment means visits another store and orders as described above, After identifying customer information corresponding to the ordering customer in the customer DB 101 based on the feature point information on the face of the ordering customer based on the image information, the pre-registered payment means information may be extracted from the corresponding customer information. The payment information may be generated based on the order detail information and the payment means information, and may perform automatic payment processing for the purchase of the order product according to the order information through communication with a predetermined payment server.

That is, the service server 100 visits multiple stores with only one registration for a payment method for use in purchasing a product at a specific store by a customer who is assigned a unique identifier based on the feature point information. May be automatically identified, and automatic payment may be performed based on the identified payment method of the customer.

Through this, the service server 100 can automatically identify (recognize) the customer based on the customer's face shown in the image and support automatic payment for the ordered product of the identified customer, thereby purchasing the product using the store. When a customer purchases a product, the customer can support the payment to be made simply by ordering the product without having to present a separate payment method to the store, thereby greatly improving the convenience of the customer.

In addition, the service server 100 manages a unique identifier assigned to the customer based on the facial feature point of the image as a member ID to generate a reward when purchasing a product of the customer corresponding to the payment information. It may be accumulated and matched with the customer information in order to pay to, and may provide the reward information for the entire rewards held by the customer who is ordering the product to the store terminal (20).

In this way, the service server 100 may support to use the rewards for the discount of the payment amount when the customer pays for the goods, for example, for the reward amount specified by the customer to use for payment from the store terminal 20 After receiving the usage information, the reward amount according to the usage information may be subtracted from the payment price according to the payment information, and then payment processing may be performed.

Through the above-described configuration, the service server 100 may unify and manage a plurality of store-specific customer management that sells different products based on a unique identifier given based on a facial feature point of a customer extracted from an image. In addition, it is possible to increase the convenience of the customer management of the store and at the same time the user convenience of payment.

Meanwhile, the service server 100 may store the order information of the customer identified by extracting the feature points of the face from the image by matching the customer information of the customer DB 101 to the customer DB 101 for a specific customer. Based on the accumulated and stored order information, a taste of a specific customer may be analyzed to generate taste information for this.

At this time, the taste information may include information about the customer's preferred product, preferred price, etc. for each product type.

In addition, the service server 100 may generate the taste information for the preferred product through analysis of the accumulated and stored order information based on a deep learning algorithm.

In addition, as shown in FIG. 5 and FIG. 6, the service server 100 may track a specific customer based on image information received from the plurality of camera units 10 and the store terminal 20. As shown in FIG. 7, copper wire information on the moving copper wire may be generated and matched with the customer information of the specific customer and stored in the customer DB 101.

In addition, the service server 100 may include a store DB (not shown) in which store information including product information about a sale product and a location of a store is stored for each of a plurality of different stores.

Through this, the service server 100 recommends one or more suitable for the taste of the specific customer for a plurality of stores located in a specific place based on the store information for each of the plurality of stores and the taste information of the specific customer. Selecting a store, and calculating the optimal copper line based on the location of the store according to the store information of one or more recommended stores selected according to the taste of the specific customer and the copper line information accumulated in response to the specific customer Information can be generated.

In addition, the service server 100 selects a recommended menu (or a recommended product) suitable for the specific customer based on the store information for each recommended store selected in correspondence with the specific customer and the taste information of the specific customer, and the recommended menu The recommendation menu information (or recommendation product information) for may be generated.

In this case, the customer information may include identification information of the customer terminal 30, the service server 100 may transmit the optimum copper wire information and menu information to the customer terminal 30 of the specific customer.

In this way, the service server 100 automatically identifies one or more recommended stores that sell products preferred by the customer to customers who visited a specific place where a plurality of stores are located, and provides them from the plurality of camera units 10. Based on the customer's existing moving route generated by identifying and tracking the customer through the video, the customer can provide a personalized service including an optimal moving route and recommendation menus for visiting the one or more stores. Through this, it is possible to greatly increase the customer's satisfaction through shop recommendation or product recommendation along with shopping efficiency and convenience.

In addition, the service server 100 may provide a system for identifying the customers and identifying the lines of the customers through facial three-dimensional modeling using a plurality of camera units 10 (security cameras). Can be used for payment systems.

According to the above-described configuration, the present invention can support the integrated customer management for a plurality of stores that sell different products based on the unique identifier given to the customer based on the facial feature points of the customer detected in the image. .

In addition, the present invention integrates a plurality of stores by unifying the same customer previously managed individually in each of a plurality of stores selling different kinds of products with a unique identifier based on the facial feature points of the customers detected in the image. As it can be managed, it can support the analysis of customer's taste by product type easily, and based on this, it can support to provide more detailed and precise personalized service.

On the other hand, the store terminal 20 may directly collect information about a specific customer's face and moving line, etc. through the communication with at least one of the plurality of camera unit 10 to generate customer information for a specific customer.

In this case, the store terminal 20 transmits the image information received from the camera unit 10 to the service server 100, for the specific customer recognized based on the image information from the service server 100 Customer information may be received and the customer information may be generated or collected through interworking with the service server 100.

In addition, the store terminal 20 may register and store store-related usage history (or order history) for a specific customer in a pre-stored shared ledger.

In addition, the shop terminal 20 of each of a plurality of different stores based on a shopping mall, a local small business association, and an affiliated store may be connected to each other through a peer-to-peer (P2P) identification client generated by each store terminal 20. Shared ledgers for the underlying usage history (or order history) can be shared between stores.

At this time, the usage history of the identified customer described in the shared shared ledger can be verified through a shared ledger stored in another adjacent store terminal 20 (when a history of simultaneous use occurs or is incorrectly recognized as the same customer). have.

In this way, the service identification (identification of the customized diet or collection of meal information, discount, etc.) for the customer identification and identification customer locally based on the trusted shared ledger can be distributed through a plurality of store terminal 20.

As a result, it is possible to perform distributed processing on a P2P basis without centrally processing personal identification and service processing for identification individuals that generate a large amount of traffic, thereby resulting in high speed, no traffic burden, and ultimately direct management of central processing. This eliminates the need for management costs.

10: camera unit 20: store terminal
30: customer terminal 100: service server

Claims (6)

A plurality of camera units installed at a place where a plurality of stores are located;
A plurality of store terminals respectively corresponding to the plurality of stores; And
And a service server which communicates with the store terminal through a communication network, and receives at least one image of the plurality of camera units from the store terminal or directly receives at least one image from the plurality of camera units.
The service server extracts and stores feature point information on facial feature points for each customer based on a preset KLT tracker (Kanade-Lucas-Tomashi tracker) from the image, and preset feature points extracted from a plurality of different images. A unique identifier is assigned in correspondence with one or more feature point information of a clustered specific customer clustered according to a running algorithm, and the image information is based on the image information received from the store terminal and the order information of the order product of the customer. Accumulate and store customer information including the unique identifier of a specific customer identified based on the feature point information and the order information, and generate preference information of the customer based on the accumulated order information corresponding to the customer;
The service server receives image information and payment means information for the payment means of the customer from the store terminal, and compares the feature point information extracted from the image information with the existing feature point information previously matched with a unique identifier for each customer. After identifying the unique identifier of the customer corresponding to the feature point information, and registering the payment means information in the customer information including the unique identifier of the customer identified corresponding to the feature point information, each time the image information and order information received from the store terminal Extracting the payment means information from the customer information corresponding to the unique identifier identified based on the feature point information extracted from the image information, and for the order goods according to the order information based on the extracted payment means information and order information We will automatically process your purchase.
The store terminal receives the image information from at least one of the plurality of camera unit to recognize or directly recognize the customer's face included in the image information in conjunction with the service server to identify the customer, including identification during the order Registering the store use history of the identified customer in the shared ledger shared between the plurality of store terminals, so that sharing and mutual verification of the shared ledger including the store use history is performed through communication between the plurality of store terminals. ,
The service server is a customer recognition system for providing a personalized service, characterized in that to provide a customized diet, diet information or a discount service to the specific customer based on the shared ledger.
delete The method according to claim 1,
The service server tracks a specific customer by extracting the facial feature points based on the image information received from the plurality of camera units and the store terminal, generates copper line information for a moving copper line, and matches the customer information of the specific customer. Select one or more recommended stores suitable for the taste of the specific customer based on pre-stored store information of each of the plurality of stores and the taste information of the specific customer; Generating optimal copper line information by calculating an optimal copper line based on the position of each store according to the store information of at least one recommended store selected in correspondence with the specific customer and the copper line information accumulated in correspondence with the specific customer. Customer recognition system for providing a personalized service characterized by.
The method according to claim 3,
The customer information includes identification information of a customer terminal corresponding to a customer, and the service server is a customer recognition system for providing a personalized service, characterized in that for transmitting the optimum copper wire information to the customer terminal.
The method according to claim 4,
The service server selects a recommendation menu suitable for the specific customer based on the store information for each of the recommended stores selected in correspondence with the specific customer and the taste information of the specific customer based on the taste information of the customer, The customer recognition system for providing a personalized service, characterized in that for generating the recommended menu information and transmitting to the customer terminal.
delete

Images