KR20230053269A - A payment system that tracks and predicts customer movement and behavior - Google Patents

Abstract

The present invention relates to a payment system that tracks and predicts the movement and behavior of customers and, more specifically, to a payment system that tracks and predicts the movement and behavior of customers, which analyzes the gaze and behavior of users for products in a store through image analysis and provides store managers with statistical information. The payment system can assign an ID to each user in a store to prevent the confusion between paying customers and the other customers, can detect the facial features of the user to prevent customer confusion, and can assign an ID to the user when the user enters the store and can control all cameras installed in the store, such as CCTVs, to track and manage the movement and behavior of the customers. The payment system comprises an image analysis unit recognizing the facial area of a user from collected captured images and analyzing the facial area.

Description

A payment system that tracks and predicts customer movement and behavior}

The present invention relates to a payment system that tracks and predicts the movements and behaviors of customers, and more particularly, provides statistical information to store managers by analyzing users' gazes and behaviors on products in a store through video analysis. It is about a payment system that tracks and predicts the movement and behavior of people.

In general, a store is equipped with a POS (Point Of Sales) terminal, and a store management system is established for card payment, sales settlement, and inventory management necessary for store operation and management. Stores introduce promotional strategies such as mileage accumulation through member registration or free gifts, and record and store customer card issuance, mileage accumulation according to purchases, and product purchase list together with customer personal information.

In this regard, Korean Patent Publication No. 10-2009-0000724 (published on January 8, 2009; hereinafter abbreviated as 'Patent Document 1') discloses a mobile communication terminal equipped with an NFC reader and a customer service in a store using PAN communication. A technology related to a payment system and method for tracking and predicting a customer's movement and behavior, which identifies the customer's movement or assigns and manages purchase points according to the customer's product purchase, is known.

According to Patent Document 1, by using a pre-established mobile communication network, it is possible to provide customized services to all subscribers of mobile communication terminals, whether store members or non-members, and purchase points are given through the mobile communication network according to customer purchases. and management to provide compatibility with a separate point providing system.

In addition, Korean Patent Publication No. 10-2009-0111622 (published on October 27, 2009; hereinafter referred to as 'Patent Document 2') discloses a store management system capable of providing various types of mileage and information to customers visiting the store, and A technology related to the driving method is known.

According to Patent Document 2, it is possible to secure potential customers and increase sales by providing various types of mileage such as visit mileage, purchase mileage, and district mileage to customers visiting the store, and customers' preferences for specific products and services was analyzed to enable specialized customer management and prompt and efficient information provision.

In addition, Korean Patent Publication No. 10-2011-0113922 (published on October 19, 2011; hereinafter referred to as 'Patent Document 3') discloses a technology related to a product marketing management method and management system using an offline store.

According to Patent Document 3, consumers' purchase intention is created using advertising means installed in offline stores that are easily accessible to consumers, and when sales results for actual advertising products occur, it acts as an intermediary between consumers and offline stores and stores. Thus, it is possible to increase the purchase performance of consumers, the sales performance of stores, and the sales performance of offline stores.

In addition, Korean Patent Publication No. 10-2012-0070330 (published on June 29, 2012; hereinafter abbreviated as 'Patent Document 4') provides shopping information according to consumer preferences and locations while being easy to manage and low cost. BACKGROUND OF THE INVENTION [0002] Techniques for shopping systems that can do this are known.

According to Patent Document 4, RFID tag information is received from a portable terminal to confirm a customer's entry or exit to the store, and when a shopping item list is received from the portable terminal, information on a recommended shopping route for the shopping item list and a store map are included. shopping information is provided to a mobile terminal, enabling efficient store management through analysis of consumer's product purchase propensity and consumption pattern.

On the other hand, in some stores, CCTVs that record in-store video in real time are installed, and the store is managed while determining situations such as theft, presence of seats, store cleanliness, or table movement that may occur in the store using the CCTV footage. making it possible

However, in such a conventional store management system, CCTV is managed and operated separately from the POS terminal, and most of the storage of sales data of the POS terminal is accumulation of data for simple sales analysis and evaluation, and image information captured by CCTV is also Regardless of the sales data of the POS terminal, only separate image information is stored and used only as reference data for later analysis. In addition, customer-related information is also stored, including customer personal information and accumulation of mileage, customer product purchasing propensity and pattern. Since it is stored separately considering only analysis, a method that can be used in conjunction with aggregated data of POS terminals has not been implemented.

Republic of Korea Patent Publication No. 10-2014-0010598, 'A system for analyzing the status of visiting customers in a store, a device for analyzing the status of visiting customers in a store, a device for analyzing the status of visiting customers by store, and a method for analyzing the status of visiting customers in a store', (2014.01.27. open)

One object of the present invention is to analyze the gaze and behavior of users visiting offline stores through video analysis to increase the visit rate and purchase rate of users' offline stores, tracking and predicting the movement and behavior of customers for reference. to provide a payment system.

Another object of the present invention is to operate an intelligent video analysis system that collects and analyzes images through a CCTV camera installed in the store, a webcam of a tablet PC, etc. in conjunction with a POS terminal in the store such as a professional franchise, thereby providing visitor analysis information in the store. Comparison of used POS terminal sales information, display of customized advertisements for visitors, calculation of waiting time for orders, and payment to track and predict customer movements and behaviors for efficient integrated operation and management of stores through regular customer visit notifications to provide the system.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, the present invention includes a plurality of cameras for photographing a user entering an offline store; an image collecting unit that collects captured images captured by the camera; an image analyzer for recognizing a facial region of the user from the collected images and then analyzing the facial region; and a product analyzer for determining which user's product is a product located in a product payment machine based on the user's data analyzed by the image analyzer, wherein the image analyzer recognizes the product picked by the user. A product recognition module to do; may include.

In addition, the image collection unit includes a face detection module for recognizing the user's face region, and after recognizing the user's face region through the face detection module, the user's existing face region is detected based on the recognized face region. It is determined whether the customer is a new customer or not, and if the customer is a new customer, an ID may be assigned and stored in a storage device along with facial feature points extracted through face analysis.

The image collection unit further includes an age recognition module for recognizing the user's age by analyzing the recognized facial region of the new customer through a pre-learned neural network, and when the user is an existing customer, Information on the ID, gender, and age of the existing customer corresponding to the extracted facial features may be called.

On the other hand, the image collection unit includes a behavior recognition module that analyzes the user's behavior, and recognizes the position of the user's hand in the collected images through the behavior recognition module so that the user can select a product in the ship. An action such as taking it into a cart or returning it may be analyzed, and action data related thereto may be mapped to the ID and stored in a storage device.

In addition, the image collection unit includes a gaze estimation module for analyzing the gaze of the user, and determines the direction of the user's head and the position of the pupils based on the collected images, and estimates the gaze of the user through this. can analyze the area of the ledge that is being watched.

Details of other embodiments are included in the detailed description and drawings.

According to embodiments of the present invention, the number of gazes defined in the present invention may correspond to the number of clicks on a product in a conventional online store, and the number of pickings in the present invention may correspond to the number of items placed in a shopping cart in a conventional online store. The conversion rate of the present invention may correspond to a ratio of purchasing products in a shopping cart in a conventional online store.

As a result, according to embodiments of the present invention, statistical information similar to that of an online store may be provided to a store manager even for an offline store. Therefore, the store manager can be provided with an opportunity to devise a plan to increase the purchase rate, such as reconstructing the product display of the online store, through various statistical information. That is, the store manager can identify products that are not popular and exchange them for alternative products to increase sales, and when a new product is released, it can arrange it to be the most prominent to check the popularity.

In particular, the present invention connects an intelligent video analysis (face recognition/motion tracking/object counting) system capable of collecting and analyzing images through webcams or CCTV cameras of multiple tablet PCs in the store with POS terminals in the store, It counts the number of visitors and regular customers in real time and provides analysis information by time (hourly, daily, monthly) that can be compared with the sales information of the POS terminal in the store to the POS terminal or the smartphone terminal of the person concerned, so the operation of the store It offers benefits that can improve efficiency.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in this specification should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in this specification, and are overly inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when the technical terms used in this specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to context, and should not be interpreted in an excessively reduced sense.

Also, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. In this application, terms such as "consisting of" or "having" should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps are included. It should be construed that it may not be, or may further include additional components or steps.

Also, terms including ordinal numbers such as first and second used in this specification may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. On the other hand, when a component is referred to as “directly connected” or “directly connected” to another component, it should be understood that no other component exists in the middle.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents or substitutes other than the accompanying drawings.

A payment system for tracking and predicting a customer's movement and behavior according to an embodiment of the present invention includes a plurality of cameras installed on each shelf of an offline store, a payment terminal for processing a payment for a user's product purchase in an offline store, and an offline It includes a store server installed for each store, a management server communicating with the store server of each store, and a manager terminal provided by the store manager. These components communicate with each other through a network. Here, the network includes a mobile communication network such as LTE (Long Term Evolution) and WCDMA (Wideband Code Division Multiple Access), a wired Internet network, and a local area wireless communication network such as Wi-Fi, and corresponds to a well-known conventional technology in the present invention Therefore, a detailed explanation is omitted. In this embodiment, it is described that the management server is separated from the store server and communicates with store servers of a plurality of stores, but it is not limited thereto, and the management server and store server may be integrated and built for each store.

A plurality of shelves for displaying products are installed in offline stores. A camera is installed on each shelf to take a picture of a customer purchasing or intending to purchase a product, that is, a user. The camera is installed to have an angle of view capable of photographing a user's appearance on the shelf and products displayed on the shelf. It is preferable to install one camera for each shelf, but is not limited thereto, and the camera may be installed at an appropriate location to photograph both the user and the shelf. For example, cameras may be installed above the shelf and on the left and right sides of the shelf. The camera may store the photographed image in real time or in advance in memory and transmit it to the store server at a designated time. A digital scale is installed on the shelf and a product may be placed on the digital scale, and the digital scale communicates with the store server to transmit weight information of the product to the store server. Alternatively, an ultrasonic sensor may be installed on the shelf to sense when a user's hand comes in and out, and transmit sensing information related thereto to the store server.

The store server is installed in an offline store, collects captured images from at least one camera in the store, stores the collected captured images in a storage device, or analyzes the captured images in real time. Also, the store server collects payment information from the payment terminal. To this end, the store server includes an image collection unit, an image analysis unit, and a payment information collection unit, and these components may be implemented as hardware or software, installed in a memory, and then executed by a processor. The store server includes a storage device such as a hard disk or Network Attached Storage (NAS).

The image collection unit collects captured images from cameras in the store. The image collection unit may receive a captured image in real time and store it in a storage device, or may receive a captured image captured and stored by a camera at regular intervals and store it in a storage device.

The image analysis unit analyzes the collected images, recognizes the user's face, analyzes the user's gender and age based on the recognized face, estimates the user's gaze, and recognizes the user's hand gesture. . The image analysis unit includes a face detection module, an age recognition module, a gender recognition module, a gaze estimation module, and an action recognition module.

After recognizing the user's face area, the face detection module determines whether the user is an existing customer or a new customer based on the recognized face area, and if the user is a new customer, an ID is assigned and stored together with facial feature points extracted through face analysis. stored in the device, and in the case of an existing customer, information on the ID, gender, and age of the existing customer corresponding to the extracted facial feature points is called from the storage device. The facial feature points may be, for example, data on areas that are difficult to change due to lighting, makeup, etc. (eg, the center of the eye, the tip of the nose, the tip of the mouth, the pupil, etc.) and distance data between these areas.

The age recognition module analyzes the recognized face region of the new customer through a pre-learned neural network to recognize the user's age, maps it to the ID, and stores it in a storage device. In addition, the gender recognition module analyzes the recognized face region of the new customer through a pre-learned neural network to recognize the user's gender, maps it to the ID, and stores it in a storage device.

The gaze estimation module determines the direction of the user's head and the position of the pupils based on the collected images, and estimates the gaze of the user through this, thereby analyzing the area of the shelf that the user gazes at (the area to be analyzed below). . The gaze estimation module analyzes the direction of the user's head and the position of the pupils, draws a normal line, and analyzes the area of the shelf that the normal line touches. The gaze estimation module maps information on the area of the shelf where the user's gaze stays for more than a predetermined time as gaze gaze data to the ID and stores the information in a storage device. The eye gaze data may include a user ID, shelf information, product information within an area to be analyzed, and occurrence time.

The action recognition module recognizes the position of the user's hand in the collected images, analyzes the user's action, such as whether the user took the product in the shelf and put it in the cart or returned it, and maps the action data related thereto to the ID. and save it to the storage device. In the case of a return act, when the hand holding the product is put into the shelf and the product is removed from the shelf, the situation in which the product is removed from the hand can be identified and analyzed. The action recognition module may precisely analyze action information such as whether the user has lifted or returned an object by further referring to weight information received from the digital scale of the shelf. Alternatively, an action may be analyzed more precisely by determining whether the user's hand has entered the shelf through an ultrasonic sensor installed on the shelf. Behavior data may include a user ID, shelf information, product information within an area to be analyzed, and occurrence time.

The payment information collection unit of the store server collects payment information related to a user's product purchase from a payment terminal installed in an offline store and stores it in a storage device. Here, the payment information includes product identification information, and the identification information may be, for example, barcode information. Here, the payment terminal may include a barcode reader capable of recognizing a barcode printed on a package of a product, and includes a card payment function such as a credit card.

The manager terminal is a portable communication terminal including a camera and a store analysis program. The manager terminal may communicate with the management server through a network. The manager terminal may access the management server through the store management program and perform member registration and login authentication. The manager registers the store he/she manages in the management server through the store management program installed in the manager terminal, sets at least one analysis target area for shelves and shelves in the store, and determines the products displayed in each analysis target area. information can be registered. In addition, the store management program of the manager terminal may receive various statistical information from the manager server and display it on the screen in graphics.

As described above, the management server communicates with the manager terminal to store store information in the storage device according to the manager's input, and also receives and stores information on each shelf in the store and display product information in the storage device. In addition, the management server communicates with the store server of each store to receive the video analysis result and payment information from the store server, calculates various statistical information based on the received video analysis result and payment information, and transmits it to the manager terminal.

The management server includes a shelf registration unit, a statistical calculation unit, and a statistical information providing unit, and these components may be implemented as hardware or software, installed in a memory, and then executed by a processor. The management server includes a storage device such as a hard disk or Network Attached Storage (NAS).

The shelf registration unit performs member registration and login authentication of store managers. The shelf registration unit may receive a name, ID, and password upon membership registration and store them in a storage device. In addition, the shelf registration unit receives store information managed by the store manager from the manager terminal of the store manager who has succeeded in login authentication, and also receives identification information of each shelf in the store, information on the analysis target area of each shelf, and each analysis target area. Receives the information of products displayed on the display and stores it in the storage device. Here, store information includes store name and address, shelf identification information may include location and serial number, analysis target area information may include coordinate information, and product information includes product name, barcode information, and the like. can do.

The statistical calculation unit communicates with the store server of each store to receive video analysis results and payment information from the store server and store them in a storage device, and based on the received video analysis results and payment information, various statistical information according to the store manager's request. Calculate The statistical calculation unit generates graphs having visual effects such as bar graphs or pie graphs for the total number of visitors to the store, the number of glances, the number of picking-returns, the number of purchases, and the conversion rate. Here, the number of glances, the number of picking-returns, the number of purchases, and the conversion rate may be calculated as overall statistics for the store, or may be calculated for each shelf or area to be analyzed. It is explained in detail below.

The statistical calculation unit calculates the total number of visitors, the number of gazes, the number of pickings, the number of returns, the number of purchases, and the conversion rate for each store by gender, age, time of day, or day of the week. The total number of visitors is the sum of the number of individual users recognized based on the video analysis results. The number of gazes is the sum of the number of gazes on a shelf in the store for a certain period of time, the number of picking is the sum of the number of times a product is picked up by hand on a shelf in the store, and the number of returns is the number of times it is returned to a shelf It is the sum of the number of times a product is placed on a shelf after being picked up by hand. The number of purchases is the sum of the total number of payments analyzed based on payment information. The conversion rate includes the gaze-to-picking conversion rate and the picking-to-purchase conversion rate, the gaze-to-picking conversion rate is the ratio of the number of pickings in the store divided by the number of gazes in the store, and the conversion rate of the picking to the purchase is the number of purchases in the store. It is the ratio divided by the number of picks in the store.

The statistical calculation unit may calculate the total number of visitors, the number of gazes, the number of pickings, the number of returns, the number of purchases, and the conversion rate for each shelf in the store by gender, by age group, by time of day, or by day of the week. The total number of visitors is the sum of the number of individual users who visited each shelf for each shelf recognized based on the video analysis result. Here, visiting a shelf may include not only giving eyes to the shelf, but also recognizing a user to a camera managing each individual shelf even without looking at the shelf. The number of gazes is the sum of the number of gazes staring at a specific shelf for a certain period of time, the number of picking is the sum of the number of times a product in a specific shelf is picked up by hand, and the number of returns is the sum of the number of times a product in a specific shelf is picked up by hand and then returned again. It is the sum of the number of times of placing on the shelf. The number of purchases is the sum of the number of payments for products displayed on a specific shelf that are analyzed based on payment information and video analysis results. The conversion rate includes the gaze-to-pick conversion rate and the picking-to-purchase conversion rate. The gaze-to-pick conversion rate is the ratio of the number of pickings on a particular shelf divided by the number of gazes on that shelf. It is the ratio divided by the number of picks on the shelf.

The statistical calculation unit may calculate the total number of visitors, the number of gazes, the number of pickings, the number of returns, the number of purchases, and the conversion rate for each analysis target area of each shelf in the store by gender, by age group, by time of day, or by day of the week. The total number of visitors is the sum of the number of individual users who visited each analysis target area recognized based on the video analysis result. Here, visiting the region to be analyzed may include not only giving a gaze to the region to be analyzed, but also recognizing a user within a predetermined distance from each region to be analyzed without giving the gaze. The number of gazes is the sum of the number of gazes gazing at a specific area to be analyzed for a certain period of time, the number of picking is the sum of the number of times holding a product within a specific area to be analyzed by hand, and the number of returns is the number of products within a specific area to be analyzed. This is the sum of the number of times the number of times you put it back on the shelf after lifting it with your hand. The number of purchases is the sum of the number of payments for products displayed in a specific analysis target area analyzed based on payment information and video analysis results. The conversion rate includes the gaze-to-picking conversion rate and the picking-to-purchase conversion rate, the gaze-to-picking conversion rate is the ratio of the number of product picks in a specific analysis target area divided by the number of gazes in the area to be analyzed, and the picking-to-purchase conversion rate is the specific analysis target It is the ratio of the number of purchases of products in an area divided by the number of picking products in the area to be analyzed.

The statistical information providing unit transmits the statistical information calculated by the statistical calculation unit to an administrator terminal through a network.

The manager terminal includes a memory, a memory controller, one or more processors (CPUs), a peripheral interface, an input/output (I/O) subsystem, a display device, an input device, a communication module, and a camera. These components communicate through one or more communication buses or signal lines. Various components described above may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and/or application specific integrated circuits. The manager terminal may be, for example, a smart phone, a tablet PC, or a laptop or personal computer, but is not limited thereto.

The memory may include high-speed random access memory, and may also include one or more magnetic disk storage devices, non-volatile memory such as flash memory devices, or other non-volatile semiconductor memory devices. Access to memory by other components of the manager terminal, such as the processor and peripheral interface, may be controlled by the memory controller. The memory may store various information and programs of the administrator's terminal.

The peripheral interface connects the input/output peripheral devices of the manager terminal with the processor and memory. One or more processors execute various software programs and/or instruction sets stored in memory to perform various functions for the manager terminal and process data. In some embodiments, the peripheral interface, processor and memory controller may be implemented on a single chip such as a chip.

The I/O subsystem provides an interface between peripheral interfaces and input/output peripherals of the manager terminal, such as a display device and an input device. The display device may use a liquid crystal display (LCD) technology or a light emitting polymer display (LPD) technology, and the display device may be a capacitive, resistive, or infrared touch display. The touch display provides an output interface and an input interface between the manager terminal and the user. The touch display presents visual output to the user. Visual output can include text, graphics, video, and combinations thereof. Some or all of the visual output may correspond to user interface objects. A touch display forms a touch sensitive surface that receives user input.

The processor is a processor configured to perform operations related to the manager terminal and to execute commands, and may control reception and manipulation of input and output data between components of the manager terminal using, for example, instructions retrieved from a memory. The software components are installed in the memory: an operating system, a graphics module and a store analytics program. The operating system may be, for example, a built-in operating system such as Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS or VxWorks, Android, etc., and general system tasks (eg, memory management, storage It includes various software components and/or devices that control and manage device control, power management, etc., and facilitates communication between the various hardware and software components The graphics module provides and displays graphics on display devices. The term “graphics” includes, without limitation, text, web pages, icons (e.g., user interface objects including soft keys), digital images, videos, animations, and the like; Includes all objects that can be displayed to the user.

The communication module performs communication through an external port or communication by an RF signal. The communication module converts electrical signals into RF signals and vice versa and can communicate with the communication network, other mobile gateway devices and communication devices through the RF signals. The communication module may include, for example, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, a memory, and the like. It may include well-known circuits for performing functions. The communication module uses the Internet, intranets and networks called the World Wide Web (WWW) and/or wireless networks such as cellular telephone networks, wireless LANs and/or MANs (metropolitan area networks), and short-range wireless communications. Can communicate with other devices.

The input device may receive a user input through a physical input interface such as a keyboard, a touch pad, a mouse, or a touch pen, or may receive a user input in conjunction with a touch display of a display device. The camera photographs the subject under the control of the camera program called by the store analysis program.

The store analysis program includes a shelf setting unit and a statistical information output unit.

The shelf setting unit performs membership registration processing and login processing for the management server and the user according to the store manager's input. The shelf setting unit receives the name, ID, and password from the store manager when registering as a member, transmits them to the management server, and transmits the ID and password to the management server when logging in.

The shelf setting unit receives store information from the store manager, transmits and registers store information to the management server, and also identifies information on each shelf in the store, information on the area to be analyzed for each shelf, and information on products displayed in each area to be analyzed. received from the store manager and transmitted to the management server for registration. Here, store information includes store name, address, etc., shelf identification information may include location and serial number, analysis target area information may include coordinate information, product information may include product name, barcode information, etc. can include

The shelf setting unit receives a store name managed by the store manager through the store name input window 701 . Preferably, the shelf setting unit may receive and display a store list from a management server that stores and manages the store list, and select one of them from a store manager, or may be searched for and selected by a store address or store name. In addition, the shelf setting unit receives input of a shelf name, an identification number, and a location from a store manager through a shelf name input window. Here, the location is information such as the third shelf from the top of the west wall, for example. In addition, the shelf setting unit displays the image of the shelf captured by the camera on the shelf image display window according to the operation of the store manager.

The shelf setting unit sets a predetermined number of analysis target regions for the shelf image displayed on the shelf image display window according to the store manager's input. The shelf setting unit receives input of the number of horizontal and vertical regions from the store manager through the division number input window, divides the shelf image equally according to the number of horizontal and vertical regions, and sets the layout. Alternatively, the shelf setting unit may automatically segment the shelf image by using an image segmentation technique among artificial intelligence deep learning technologies. That is, the shelf setting unit may recognize products displayed in the shelf image and recognize different products as different areas to be analyzed to set a layout. At this time, the shelf setting unit enables the store manager to edit each line of the layout and adjusts the top, bottom, left, and right positions of each line. For example, if the layout of products displayed on a specific shelf is 3 columns x 4 columns, and a total of 12 types of products are displayed, a line corresponding to 3 columns x 4 columns is automatically created and arranged according to the product area. After the line is set, as an example of a method of directly adjusting the detailed position of the line, the line displayed on the screen can be freely moved by touching and dragging.

After the analysis target area is set, the shelf setting unit receives information on products displayed in each analysis target area from the store manager. Product information may include identification information such as product name, price, and barcode. When receiving product information for each analysis target region, the shelf setting unit may set product information by reading a barcode printed on a wrapping paper of the product. Alternatively, the shelf setting unit may transmit a product name to a management server that stores and manages product information, receive and display a product list corresponding to the product name, and select one of them from the store manager.

The shelf setting unit transmits the store name, shelf name, shelf identification number, and analysis target area information input according to the method described above to the management server. The analysis target area information may include coordinate information of each analysis target area and product information. Meanwhile, the shelf setting unit may modify previously registered information on each shelf and analysis target area according to input from a store manager.

The statistical information output unit of the store analysis program transmits condition information input from the store manager to the management server, receives statistical information corresponding to the condition information from the management server, and displays the statistical information on the screen. The condition information includes the designation, period, gender, and age of the entire store or each shelf or area to be analyzed, and also includes statistical types such as the number of visitors, the number of gazes, the number of purchases, and the conversion rate. The statistical information output unit may receive a plurality of period information from a store manager and compare and display statistical information between different periods.

The store, shelf, and area to be analyzed can be set through the first input window, the period can be set through the second input window, the statistical type can be set through the third input window, and the fourth input window can be set. You can set gender and age. Statistical information is displayed in the search result display window according to the conditions set in this way. And when two different periods are set by the store manager, statistical information between the two periods is compared and displayed as a bar graph.

The store server collects captured images from cameras in the store. The store server may receive a captured image in real time and store it in a storage device, or may receive a captured image captured and stored by a camera at regular intervals and store it in a storage device.

The store server recognizes the user's face region from the collected images and then analyzes the facial feature points. Facial feature points may be, for example, data on areas that are difficult to change due to lighting or makeup (eg, the center of the eye, the tip of the nose, the tip of the mouth, the pupil, etc.) and distance data between these areas.

The store server determines whether a user recognized in the photographed image is an existing user or a new user based on the facial feature points analyzed in the analysis step. The store server compares the facial feature points for each user pre-stored in the storage device with the facial feature points analyzed in the analysis step, and determines that the user is an existing user if similar facial features are stored in the storage device, and otherwise determines the user as a new user.

In the case of a new user, the store server assigns a new ID to the new user and stores and registers the facial features analyzed in the analysis step in a storage device. In addition, the store server analyzes the new user's recognized face region through a pre-learned neural circuit, recognizes age and gender, maps it to the ID, and stores it in a storage device. On the other hand, in the case of an existing user, the store server calls the existing user's ID, age, and gender from the storage device.

After the user's ID, age, and gender are extracted in this way, the store server determines the direction of the user's head and the location of the pupils based on the captured images collected in the collection step, and estimates the user's gaze through this, so that the user can observe Analyze the area of the shelf to be used. Preferably, the store server maps information on an analysis target area of a shelf on which a user's gaze stays for a predetermined time or longer to the ID as eye gaze data and stores it in a storage device.

In addition, the store server recognizes the position of the user's hand in the collected images, analyzes the user's behavior, such as whether the user took the product in the shelf and put it in the cart or the like or returned it, and sends behavior data related to this to the ID. Map it and save it to the storage device. The store server may further refer to the weight information received from the digital scale of the shelf to precisely analyze action information such as whether the user has lifted or returned the product.

The store server repeats the process of area analysis to storage until the user leaves the store. The store server may recognize the user's exit from the store based on payment information (eg, member information) of the user, or may recognize the user's exit by analyzing an image captured by a camera installed at the checkout counter.

In the case of a conventional online store, various statistical information such as the number of product clicks, the number of products placed in a shopping cart, and the number of purchases of products placed in a shopping cart are provided to the seller. According to an embodiment of the present invention, the number of gazes defined in the present invention may correspond to the number of clicks on a product in a conventional online store, and the number of pickings in the present invention may correspond to the number of items placed in a shopping cart in a conventional online store. The conversion rate of the present invention may correspond to the number of purchases of products in a shopping cart in a conventional online store. Accordingly, statistical information similar to that of an online store may be provided to a store manager for an offline store. Therefore, the store manager can be provided with an opportunity to devise a plan to increase the purchase rate, such as reorganizing product display, through various statistical information. In addition, by allowing the store manager to directly set the analysis target area through the manager terminal, the store manager can quickly respond to the frequently changing store environment.

While this specification contains many features, such features should not be construed as limiting the scope of the invention or the claims. Also, features described in separate embodiments in this specification may be implemented in combination in a single embodiment. Conversely, various features that are described in this specification in a single embodiment may be implemented in various embodiments individually or in combination as appropriate.

It should be understood that the division of various system components in the above embodiments does not require such division in all embodiments. The program components and systems described above may generally be implemented as a package in a single software product or multiple software products.

The method of the present invention as described above may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form. Since this process can be easily performed by a person skilled in the art to which the present invention belongs, it will not be described in detail.

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention belongs that other modified examples based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein. It is self-evident to those skilled in the art. In addition, although specific terms have been used in the present specification and drawings, they are only used in a general sense to easily explain the technical content of the present invention and help understanding of the present invention, but are not intended to limit the scope of the present invention. Accordingly, the foregoing detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be selected by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (5)

A plurality of cameras that take pictures of users entering the offline store;
an image collecting unit that collects captured images captured by the camera;
an image analyzer for recognizing a facial region of the user from the collected images and then analyzing the facial region;
A product analysis unit for confirming which user's product is a product located in the product payment machine based on the user's data analyzed by the image analysis unit;
The video analysis unit,
A product recognition module for recognizing the product that the user picks; a payment system for tracking and predicting movement and behavior of customers, characterized in that it includes
According to claim 1,
The video collection unit,
Including; face detection module for recognizing the user's face region,
After recognizing the user's face region through the face detection module, based on the recognized face region, it is determined whether the user is an existing customer or a new customer, and if the user is a new customer, an ID is assigned and facial feature points extracted through face analysis A payment system that tracks and predicts customer movement and behavior, characterized in that it is stored in a storage device together with
According to claim 2,
The video collection unit,
An age recognition module for recognizing the user's age by analyzing the recognized facial region of the new customer through a pre-learned neural network;
If the user is an existing customer, a payment system for tracking and predicting the movement and behavior of a customer, characterized in that calling information on the ID, gender and age of the existing customer corresponding to the extracted facial feature from a storage device.
According to claim 1,
The video collection unit,
Including; action recognition module for analyzing the user's action;
The action recognition module recognizes the position of the user's hand in the collected images, analyzes the user's action, such as whether the user picked up a product in the ship, put it in a cart, etc., or returned it, and the action data related thereto A payment system that tracks and predicts the movement and behavior of customers, characterized in that they are mapped to IDs and stored in a storage device.
According to claim 1,
The video collection unit,
Including; a gaze estimation module for analyzing the gaze of the user;
Based on the collected images, the direction of the user's head and the position of the pupils are identified and the gaze of the user is estimated through this, thereby analyzing the area of the shelf that the user is watching, tracking the movement and behavior of the customer, predictive payment system