KR20170082299A - The object recognition and attention pursuit way in the integration store management system of the intelligent type image analysis technology-based - Google Patents

The object recognition and attention pursuit way in the integration store management system of the intelligent type image analysis technology-based Download PDF

Info

Publication number
KR20170082299A
KR20170082299A KR1020160001531A KR20160001531A KR20170082299A KR 20170082299 A KR20170082299 A KR 20170082299A KR 1020160001531 A KR1020160001531 A KR 1020160001531A KR 20160001531 A KR20160001531 A KR 20160001531A KR 20170082299 A KR20170082299 A KR 20170082299A
Authority
KR
South Korea
Prior art keywords
face
store
image
information
customer
Prior art date
Application number
KR1020160001531A
Other languages
Korean (ko)
Other versions
KR101779096B1 (en
Inventor
방승온
방수온
Original Assignee
방승온
방수온
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 방승온, 방수온 filed Critical 방승온
Priority to KR1020160001531A priority Critical patent/KR101779096B1/en
Publication of KR20170082299A publication Critical patent/KR20170082299A/en
Application granted granted Critical
Publication of KR101779096B1 publication Critical patent/KR101779096B1/en

Links

Images

Classifications

    • G06K9/00228—
    • G06K9/00281—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q20/00—Payment architectures, schemes or protocols
    • G06Q20/08—Payment architectures
    • G06Q20/20—Point-of-sale [POS] network systems
    • H04N5/2257—
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

The present invention relates to an object recognition and visual tracking method in an integrated store management system based on intelligent image analysis technology, and more particularly, to an object recognition method and an eye tracking method in an integrated store management system based on intelligent image analysis technology, And to grasp the gaze of the object.
To this end, the present invention relates to intelligent image analysis and real-time image collection and analysis of a visitor's image in a store, and an object recognition and a sight line tracking method by a store management server, A face detection step of detecting a face area of the detected object, a step of classifying the sex and age of the visitor in the face area of the detected object, determining whether the customer is a regular customer, calculating the number of visitors in the store and the number of regular customers, Face matching step to confirm customer information, co-location tracking to track the position of the nose in the face region by tracking and analyzing the body, face tracking, and head movement of the object in the real-time image frame acquired from the visitor image information in the store And the eye position of the object is determined by the position coordinates of the nose in the tracked face area He or including eye localization and time calculation step of calculating gaze fixed time position information of the matching object eye position and the position of the object of display products, allows you to efficiently integrate operate and manage the store.

Description

Technical Field [0001] The present invention relates to an object recognition and attention tracking method in an integrated store management system based on intelligent image analysis technology,

The present invention relates to object recognition and eye tracking in an integrated store management system based on intelligent image analysis technology. More particularly, the present invention relates to a method and system for collecting images of visitors through a webcam or a CCTV camera of a plurality of tablet PCs displaying in- The integrated store management system that operates intelligent image analysis system in cooperation with the POS terminal in the store, recognizes the object from the analysis information of the visitor image in the store and tracks the gaze so as to grasp whether the advertisement or the exhibition product is watched. To an object recognition and a gaze tracking method in an integrated store management system based on an intelligent image analysis technology capable of efficiently managing and managing advertisements or merchandise in a store.

In general, POS (Point Of Sales) terminals are installed in the store to establish a store management system for card settlement, sales settlement and inventory management necessary for the operation and management of stores. In order to carry out such store management more efficiently, The store introduces a promotion strategy such as accumulation of mileage through the membership registration, gift giving, etc., records the personal information of the customer together with the issuance of the customer card, the accumulation of mileage according to the purchase, and the purchase list of the product.

In this regard, Korean Patent Laid-Open No. 10-2009-0000724 (published on Jan. 1, 2009, hereinafter abbreviated as "Patent Document 1") discloses a method of transmitting a message to a customer using a PAN communication with a mobile communication terminal equipped with an NFC reader And a store management system and method for providing and managing purchase points according to customer's purchase of goods are known.

According to Patent Document 1, it is possible to provide customized services to all subscribers of the mobile communication terminal, whether a member of a store or a non-member, by using a pre-established mobile communication network, and to provide a purchasing point through a mobile communication network And provide compatibility with a separate point providing system.

Korean Patent Laid-Open No. 10-2009-0111622 (published on Oct. 27, 2009; hereinafter referred to as "Patent Document 2") includes a store management system capable of providing various kinds of mileage and information to customers visiting the store, A technique relating to the driving method thereof is known.

According to Patent Document 2, various kinds of mileage such as visit mileage, purchase mileage, and area mileage are provided to customers visiting a store, so that potential customer acquisition and sales improvement can be expected, and the customer's preference for specific goods and services To enable the management of specialized customers and the provision of information promptly and efficiently.

In addition, Korean Patent Laid-Open No. 10-2011-0113922 (published on October 19, 2011; hereinafter referred to as "Patent Document 3") discloses a technique related to a product marketing management method and a management system using an offline store.

According to Patent Document 3, a purchase intention of a consumer is created by using an advertising means installed in an offline store where a consumer can easily access, and a role of a relay between a consumer and an offline store and a retail store occurs when sales results of actual advertisement products occur So that consumers' purchasing performance, sales performance of the store, and sales performance of the offline store can be increased together.

In addition, Korean Patent Laid-Open No. 10-2012-0070330 (published on June 29, 2012; hereinafter referred to as "Patent Document 4") provides shopping information according to the tendency and position of the consumer while being easy to manage and requiring low cost A technique relating to a shopping system that can be used is known.

According to Patent Document 4, when the RFID tag information is received from the portable terminal to confirm the entrance or exit of the customer's store, and when the shopping article list is received from the portable terminal, information on recommended shopping lines for the shopping article list and store map are included To the portable terminal, enabling efficient store management by analyzing the purchase tendency and consumption pattern of the consumers.

On the other hand, in some stores, CCTVs are used to photograph real-time images in the store. CCTV images can be used to manage the stores while judging the thefts, presence of seats, cleanliness of stores, I can do it.

However, in the conventional store management system, the CCTV is managed and operated separately from the POS terminal. The storage of the sales data of the POS terminal is mostly accumulated for the sales analysis and evaluation, and the video information It is stored only as separate image information, regardless of the sales data of the POS terminal. Therefore, it is used only as reference data for future analysis. Also, the storage of customer related information is performed by accumulating the customer's personal information and mileage, Since it is separately stored considering only the analysis, a method that can be utilized in connection with the aggregated data of the POS terminal has not been implemented.

KR 10-2009-0000724 A 2009.01.08 Released KR 10-2009-0111622 A 2009.10.27 public KR 10-2011-0113922 A KR 10-2012-0070330 A 2012.06.29 public

SUMMARY OF THE INVENTION Accordingly, the present invention has been made keeping in mind the above problems occurring in the prior art, and an object of the present invention is to provide an intelligent image analysis system for collecting and analyzing a visitor's image through a webcam or a CCTV camera of a plurality of tablet PCs, In the integrated store management system operated by linking with the terminal, the object is recognized from the analysis information of the visitor image in the store, and the direction of the object and the position of the nose included in the face image are tracked to grasp the gaze of the object. The object recognition and gaze tracking method in the integrated store management system based on the intelligent image analysis technology that can efficiently manage and manage the advertisement or the product display in the store by quantifying quantitatively whether the advertisement or the exhibition product is watched .

According to an aspect of the present invention, there is provided an intelligent image analyzing system, which is connected to a first image capturing unit and a second image capturing unit installed in a store through a network, The integrated store management system integrates the store management server with the POS terminal for settlement of sales in the store and integrally manages the POS terminal by the intelligent image analysis and store management server. The intelligent image analysis and object recognition (A) a face detection step of detecting a face area of an object included in the image information by acquiring a real-time image frame from the visitor image information collected in the store collected through the image collection unit, (b) The face region of the detected object is distinguished from the sex and age of the visitor, A face matching step of determining the number of visitors in the chapter and the number of regular customers and determining the regular customer information to be displayed on the POS terminal; (c) a step of detecting the body and face of the object in the real- Tracking position of the nose within the face region by tracking and analyzing the tracking and head movement, and (d) tracking the nose position of the object with the position coordinates of the nose in the face region tracked in the nose position tracking step And an eye position and a time calculating step of calculating eye gaze position fixed time information of an object in which the position of the advertisement or the display product and the gaze position of the object coincide with each other, And eye tracking method.

According to the present invention, an intelligent image analysis system for collecting and analyzing images through a webcam or a CCTV camera of a plurality of tablet PCs in a store is interlocked with a POS terminal in a store such as a professional franchise, thereby realizing the number of real- (Hourly, daily, and monthly) analysis information that can be compared with the sales information of the POS terminal in the store, can be provided to the POS terminal or the smartphone terminal of the POS terminal in the store, so that the advantage of improving the operation efficiency of the store to provide.

According to the present invention, the object is recognized from the analysis information of the visitor image in the store, the gaze of the object is tracked by tracking the progress direction of the object and the position of the nose included in the face image, The degree of interest can be quantitatively quantified so as to provide an advantage that the employee or the person concerned can quantitatively judge the adequacy and efficiency of the advertisement or the product display in the store.

In addition, the present invention can calculate waiting time of a queue for settlement or order and provide it by reflecting it in the decision of moving line determination or recruitment of an employee, and also sending a visit notification SMS or SNS to an employee or an official when visiting a regular customer The new employee or part-time employee can quickly and efficiently perform routine customer-related tasks such as coupon issuance and mileage accrual during work, thereby improving the efficiency of the store operation.

FIG. 1 is a block diagram schematically illustrating an overall configuration of an integrated store management system based on an intelligent image analysis technology to which an object recognition and a gaze tracking method according to the present invention can be applied.
FIGS. 2A and 2B are exemplary views of a menu screen displayed on a POS terminal in an integrated store management system based on an intelligent image analysis technology to which an object recognition and a gaze tracking method according to the present invention is applied.
3A and 3B are flowcharts illustrating an object recognition and a gaze tracking method in an integrated store management system based on an intelligent image analysis technology according to the present invention.
4A to 4D are reference views for explaining an object recognition and a line-of-sight tracing process using the progress direction and nose position of an object according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an object recognition and a gaze tracking method in an integrated store management system based on an intelligent image analysis technology according to the present invention will be described in detail with reference to the accompanying drawings.

It is to be understood that the words or words used in the present specification and claims are not to be construed in a conventional or dictionary sense and that the inventor can properly define the concept of a term in order to describe its invention in the best possible way And should be construed in light of the meanings and concepts consistent with the technical idea of the present invention. Therefore, it should be understood that the embodiments described herein and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and that various equivalents and modifications may be substituted for them at the time of the present application shall.

FIG. 1 is a block diagram schematically illustrating an overall configuration of an integrated store management system based on an intelligent image analysis technology applicable to an object recognition and a gaze tracking method according to the present invention. FIGS. As an example of a menu screen displayed on a POS terminal in an integrated shop management system based on an intelligent image analysis technology to which a gaze tracking method is applied, as illustrated in FIG. 1, an intelligent image analysis technique Based integrated store management system may include a first image capturing unit 30 and a second image capturing unit 40 installed in a store, an intelligent image analysis and store management server 100, and a database 200 The integrated store management system based on the intelligent image analysis technology of this configuration is a store management system having a POS terminal 10 for sales settlement and inventory management, Is applied to the system can be implemented. At this time, an additional menu (refer to FIG. 2A) that can simply add the number of visitors and the number of regular customers in the store by time slot on the menu screen may be added to the POS terminal 10. In addition, The number of visitors in the store and the number of regular customers can be displayed on a detailed menu screen (detailed information: see FIG. 2B), which can be specifically displayed by distinguishing the number of shop visitors and the number of regular customers by hour, day, ) Can be set to be selectively displayed according to the button input on the additional menu.

The first image capturing unit 30 can be implemented as a tablet PC capable of displaying advertisement contents and transmitting / receiving data through a wired / wireless communication network, and can display an advertisement displayed on a screen easily by a shop visitor The location or the location of the store visitor can be easily photographed by the webcam, and one or a plurality of images may be provided in the store. The first image capturing unit 30 includes a webcam 31 for capturing image information, a local wireless communication unit 32 for noncontact local communication between the customer's smartphone terminal 20 and an intelligent Wireless communication module unit 33 for transmitting and receiving data to and from the image analysis and store management server 100 and is connected to the intelligent image analysis and store management server 100 through a communication network to transmit and receive data, And the store management server 100 and displays the advertisement content information on a screen, captures and collects the image information of the shop visitor in real time with the webcam 31, and transmits it to the intelligent image analysis and store management server 100 . The first image capturing unit 30 wirelessly receives customer information through a short distance wireless communication with a customer's smartphone terminal 20 and transmits the customer information to the intelligent image analysis and store management server 100, Information or mileage information is received from the intelligent image analysis and store management server 100 in real time to issue a coupon to the customer's smartphone terminal 20 or accumulate the mileage and accumulate coupon issuing information or mileage accumulation information according to the result, And returns it to the analysis and store management server 100.

The second image capturing unit 40 may be implemented as a CCTV camera capable of photographing image information of a shop visitor in real time and may be installed at a position where the image of a shop visitor can be easily photographed. May be provided. The second image capturing unit 40 includes an intelligent image analyzing unit through a wired or wireless communication network and a wired / wireless communication module for transmitting / receiving data to / from the store management server 100, And transmits the captured image information of the shop visitor to the intelligent image analysis and store management server 100 in real time.

The intelligent image analysis and store management server 100 collects and analyzes the visitor image information photographed by the image capturing units 30 and 40 in real time to classify the gender and the age group of visitors, It is possible to track the visitor's gaze while displaying the image on the image photographing unit 30, and provide the information such as the numerical information on how the sight line position of the visitor is staying in the advertisement being displayed in the store or the position of the product being displayed, And the real time waiting time information of the queue for ordering and settlement in the store are calculated and displayed on the POS terminal in the store together with the notification information at the time of confirming the regular customer visit or the registered smart phone terminal 50 ).

The intelligent image analysis and store management server 100 includes an image collection unit 110, an image analysis unit 120, an advertisement control unit 130, an advertisement watched amount quantification unit 140, a reward management unit 150, A normal customer visit notification unit 170, a force terminal menu management unit 180, and a manager management unit 190. [

The image collection unit 110 is connected to each of the image capturing units 30 and 40 through a communication network and collects visitor image information in the stores photographed by the image capturing units 30 and 40 in real time via a wired or wireless communication network.

The image analysis unit 120 performs angle test processing on the visitor image information collected by the image collection unit 110 to obtain a real-time image frame, and performs an adaboost face detection algorithm on the image frame whose difference between the obtained image frames is below the threshold A facial region of the object included in the image information is detected, a template matching algorithm is applied to the face region of the detected object to count the number of visitors and the number of visitors, and the face matching between the recognized face and the customer image information of the database To identify the visitor's gender and age group, count the number of regular customers, count the number of regular customers, and identify the number of visitors and regular customers in each store by time, day, month, quarter, And determines the regular customer information to be displayed on the force terminal.

Also, the image analysis unit 120 tracks the body movement of the object from the image analysis information of the visitor in the store, tracks the face, tracks the movement of the head and tracks the position of the nose within the face region, The coordinates of the traced nose are used to determine the gaze position of the object with respect to the position of the advertisement or the exhibition product in the store and to check whether the position of the advertisement or the exhibition product and the gaze position of the object are matched (fixed) The time information (the sight line position fixed time information of the object) during the time when the position of the displayed product matches the sight line position of the object is counted and calculated. Here, the location information of an advertisement or an exhibition product being displayed in a store may be a value previously registered and managed in advance as a position coordinate value.

The advertisement control unit 130 extracts the advertisement contents corresponding to the gender (male / female) and the age information of the object identified in the image analysis unit 120 from the advertisement content information of the database 200, Unit 30 to display the visitor-tailored advertisement in the store.

The advertisement audience volume quantification unit 140 confirms the gaze position fixed time information of the object with respect to the advertisement or the product position calculated by the image analysis unit 120 so that the advertisement being displayed or the product being displayed is displayed for a certain period of time Quantitatively quantifying the amount of advertisement or commodity viewing of the object according to the viewer's view.

The reward management unit 150 searches the customer information of the database 200 for coupon information and mileage information of the customer corresponding to the information of the smartphone terminal 20 of the customer transmitted from the first image capturing unit 30, The first image capturing unit 30 receives result information on coupon issuance or mileage accumulation processed in the smartphone terminal 20 of the customer and transmits the result information to the customer- Update information or mileage accumulated information.

The waiting time calculating unit 160 analyzes the visitor image information collected by the image collecting unit 110 to count the number of objects included in the queue for ordering or settlement in the store, Which may be, for example, the average time-to-value per object required for in-store ordering and billing).

The regular customer visit notification unit 170 checks the customer identification information of the object identified by the image analysis unit 120 and transmits the customer related information, for example, notification information for notifying the visiting of the regular customer, Related information and notification information for notifying the visiting of the customer to the POS terminal 10 in the store or transmits the information to the smartphone terminal 50 of the related party including the employee to the SMS or the SNS.

The POS terminal menu management unit 180 confirms the number of visitors and the number of regular customers in each store counted by the image analysis unit 120 and transmits them to the in-store POS terminal 10, 2A, the menu of the POS terminal is managed so as to display the number of visitors in the store and the number of regular customers by time, as the brief view information together with the real-time sales settlement information of the POS terminal 10 , And when the detailed view button (e.g., the 'GY system' button on the screen) is clicked by the POS terminal staff or an official on the sales settlement screen, the number of visitors and the number of regular customers The menu of the POS terminal is managed so as to express it concretely.

The manager managing unit 190 collects the real-time sales information of the POS terminal 10 and stores the number of visitors in the store and the number of customers in the store by the time analyzing unit 120 counted by the image analyzing unit 120 and the queue counted by the waiting time calculating unit 160 To the smartphone terminal 50 of the pre-registered party.

The database 200 stores customer information and advertisement content information registered through the POS terminal 10 and the intelligent image analysis and store management server 100 and customer-specific purchase information related to sales settlement processed at the POS terminal 10 Coupon information, mileage information, coupon issue information for each customer processed and returned by the first image capturing unit 30, and mileage accumulation information.

FIGS. 3A and 3B are flowcharts illustrating an object recognition and a gaze tracking process in an integrated store management system based on an intelligent image analysis technology according to the present invention. FIGS. 4A to 4D illustrate an operation flow of an object (A), a face matching step (b), a face matching step (b), a face matching step (b), and a face matching step (C), and a gaze position and time calculation step (d). The object recognition and gaze tracking method according to the present invention may include a first image capturing unit 30 and the second image capturing unit 40 through the network to collect and analyze the images of the visitors in the store in real time. The intelligent image analysis and store management server 100 is connected to the POS terminal 10 for settlement and the POS terminal 10 is operated by the intelligent image analysis and store management server 100. In the integrated store management system, Lt; / RTI >

The face detection step (a) is a step of detecting a face area of an object included in the image information by acquiring a real-time image frame from the visitor image information collected in the store collected through the image collection unit, (S121 and S122), the difference between the acquired image frames is compared with a preset threshold value, and an adaboost face detection algorithm is applied to an image frame whose difference between image frames is below a threshold value (S123-S125) of detecting the face region of the object included in the image information by applying the step S123-S125.

In the face matching step (b), the sex and age of the visitor are distinguished in the face region of the object detected in the face detecting step (a), and the number of the visitors in the store and the number of regular customers are checked, A step S126 of recognizing the face and counting the number of visitors by applying a template matching algorithm to the face region of the object detected in the face detecting step (a) (S126) In step S127, the face and face information of the face is recognized and the customer image information accessed in the database is used to identify the sex and age of the visitor through face matching, and whether the customer is a regular customer is counted The number of visitors in the counted stores and the number of regular customers are divided according to time periods such as hourly / daily / monthly / quarterly / yearly, respectively, in steps S128 and S129, And it may be formed by a step (S130) for deciding the regular customer information to express the POS terminal.

The step (c) of tracking the nose position traces and analyzes the body, face tracking and head movement of the object in the real-time image frame acquired from the visitor image information in the store in the face detection step (a) A step (S141, S142) of tracking a face of an object by tracing a body motion of the object in a real-time image frame obtained by angular-testing the image information of a visitor in a shop in a face detection step (a) (S143-S145) tracking and analyzing the head movement in the tracked face region image to track the position coordinates of the nose in the face region.

The eye gaze position and time calculation step (d) is a step of determining the gaze position of the object with the position coordinates of the nose in the face area tracked in the nose position tracking step (c) The method of claim 1, further comprising the step of: calculating the gaze position fixed time information of the object, wherein the gaze position of the object with respect to the position of the advertisement or the exhibit being displayed in the store is identified by the position coordinate of the nose in the face area tracked in the coarse position tracking step (c) (Step S146), confirming whether the position of the advertisement or the display product and the gaze position of the object are coincident (fixed) (S147), and determining whether the position of the advertisement or the display product and the gaze position of the object coincide And counting and calculating the sight line position fixing time information (S148).

The operation and effect of the object recognition and the gaze tracking method in the integrated store management system based on the intelligent image analysis technology according to the present invention will be described as follows.

First, in the face detection step (a), the image analysis unit 120 of the intelligent image analysis and store management server 100 performs initialization (S121) of variables for face recognition as illustrated in the operation flow chart of FIG. 3A (For example, rotating 5, 10, 15, 20, 25, and 30 degrees from the original image) using the enhanced face detection technique from the image information of the visitor collected in real time in the image collecting unit 110, (S122) through a plurality of image frames (refer to FIG. 4A) through S122, thereby making it possible to detect a face even when the angle of the acquired face photograph is large or there is an obstacle. (S123), an adaboost (Adaptive Boosting) face detection algorithm is applied to the image frames whose difference value between the image frames is not larger than the predetermined threshold (Y branch in S123) (S124) It is confirmed whether the face area is detected in the image frame (S125).

Next, in the face matching step (b), the image analysis unit 120 of the intelligent image analysis and store management server 100 performs a template matching algorithm on the image frame in which the face region is detected in the face region confirmation step (S125) (Step S126), and then the customer image information of the database 200 is accessed (step S127). Then, the face image is compared with the recognized face, (Male / female) of the visitors included in the image frame and the age group when the face matching is performed (Y branch in S128), and also checks whether the customer is a regular customer and counts the number of regular customers (step S128) S129). The number of visitors and the count of the number of regular customers are divided into time zones, and the number of visitors and the number of regular customers are calculated for each hour, day, month, quarter, The customer information to be displayed on the machine 10 is determined (S130). In particular, in the template matching algorithm, the image of the detected face region is compared with the face image of the customer stored in the database, and the method of matching the most similar face image is used. In the face matching algorithm, And compare them with the stored face images.

Next, in the coarse position tracking step (c), the image analysis unit 120 of the intelligent image analysis and store management server 100 performs an intra-store analysis in the face detection step (a) The body motion of the object is tracked from the real-time image frame (see FIG. 4A) acquired by angular-testing the image information of the visitor (S141), and the face of the object (see FIGS. 4C and 4D) (Step S142). The head movement in the tracked face area is tracked (S143) and analyzed (S144) to determine the position of the nose of the front face coordinate c1 (or the front face image f1 in FIG. 4D) The coordinates of the position of the nose with respect to the respective direction faces (see (c2) to (c10) in FIG. 4C or (f2) to (f10) in FIG. 4D) are traced (S145).

Finally, in the gaze position detection and time calculation step (d), the image analysis unit 120 of the intelligent image analysis and store management server 100 calculates the position coordinates of the nose in the face area tracked in the nose position tracking step (c) (S146), and it is confirmed whether or not the gaze position of the object with respect to the position of the advertisement or the exhibition product in the shop is matched (fixed) (S147) (Step S148), the time information (the sight line position fixed time information of the object) in which the gaze position of the object matched with the position of the advertisement or the displayed product is fixed is calculated. In this case, as shown in FIG. 4B, a method of tracking the position of an object in the image analysis unit 120 may be performed by applying a tablet algorithm to the frontal image of the object obtained by tracking the direction of the object, And sets a reference point and identifies the moving region of the nose of the object collected in the face region as illustrated in FIGS. 4C and 4D through the tracking of the head movement (S143) and the analysis (S144) The sight line position of the object is grasped through comparison with the reference point (S146).

Accordingly, in the integrated store management system of the present invention, the intelligent image analysis and store management server 100 collects and analyzes the image information captured by each of the image capturing units 30 and 40 in real time, thereby recognizing the face of the object, It is possible to grasp the sex and age of the object included in the image information and to display the customized advertisement in the first image capturing unit 30 in the store accordingly, The tracking of the object and the position of the nose within the face region are tracked to determine the position of the object's eye and the position of the object in the displayed advertisement or the displayed product is fixed and the eye- Or the quantity of items to be viewed can be quantified.

Also, in the integrated store management system of the present invention, the intelligent image analysis and store management server 100 transmits the coupon information of the customer corresponding to the information of the smartphone terminal 20 of the customer transmitted from the first image capture unit 30, The information is retrieved from the customer database 200 and returned, and the number of visitors and the number of regular customers in the store are counted by time (day / day / month / quarter / year) And the real-time waiting time information of the queue for settlement can be calculated and provided. As a result, stakeholders in the store can easily compare the sales settlement information provided by the POS terminal, the number of visitors in the store, and the number of regular customers, thereby effectively managing the store.

Also, in the integrated store management system of the present invention, the intelligent image analysis and store management server 100 determines whether the customer visits the regular customer based on the customer information previously stored in the database 200, and informs the visiting of the regular customer Information about the purchasing of the regular customer or the information related to the purchase of the regular customer and the notification information for notifying the visiting of the customer to the POS terminal 10 in the store or sending the SMS or SNS to the pre-registered smart phone 50 The number of visitors in the store and the number of regular customers in each counted time period are transmitted to and displayed in the POS terminal 10 in the store and the sales information of the POS terminal 10 is collected and the number of visitors in the store The regular customer number information and the real-time waiting time information of the queue to the smartphone terminal 50 of the pre-registered party.

As described above, the present invention provides an intelligent image analysis (face recognition / motion tracking / object counting) system capable of collecting and analyzing images through a webcam or CCTV camera of a plurality of tablet PCs in a store, (Hourly, daily, monthly, yearly) analysis information that can be compared with the sales information of the POS terminal in the store to the POS terminal or a smartphone terminal of an interested person by counting the number of real- It is advantageous that the employee or the store manager can compare and judge the advantages and disadvantages of the store operation or the cause and analysis of the cause of sluggish sales and the efficiency of the customer's response or management more efficiently .

Further, according to the present invention, it is possible to quantitatively determine whether a customer watches an advertisement in a store, and to provide a numerical value, thereby providing an advantage that an employee or an interested party can easily judge the suitability and quantitative effect of the advertisement content in the store have.

In addition, the present invention has an advantage in that the order waiting time of a queue for settlement or order is calculated and provided so that it can be reflected in the determination of a moving line or the decision of recruiting an employee. In addition, when visiting a regular customer, The new employee or part-time employee can quickly and efficiently perform the routine customer-related business affairs such as issuing the coupon and earning mileage, etc., so that the efficiency of the store operation can be improved.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, Modification is possible. Accordingly, it is intended that the scope of the invention be defined by the claims appended hereto, and that all equivalent or equivalent variations thereof fall within the scope of the present invention.

10: POS terminal 20: Customer smart phone terminal
30: First image capturing unit 31: Webcam
32: short range wireless communication unit 33: wired or wireless communication module unit
40: second image capturing unit 50: related smartphone terminal
100: Intelligent image analysis and store management server
110: image collecting unit 120: image analyzing unit
130: advertisement control unit 140:
150: Reward management unit 160: Waiting time calculation unit
170: regular customer visit notification part 180: POS terminal menu management part
190: manager manager 200: database

Claims (5)

An intelligent image analysis and store management server 100, which is connected to the first image capturing unit 30 and the second image capturing unit 40 installed in the store through a network to collect and analyze images of visitors in the store in real time, The intelligent image analysis and store management server 100 is connected to the POS terminal 10 for sales settlement and the POS terminal 10 is operated by the intelligent image analysis and store management server 100. In the integrated store management system, ), The object recognition method comprising:
(a) a face detection step of obtaining a real-time image frame from the visitor image information collected through the image collection unit and detecting a face region of the object included in the image information;
(b) classifying the sex and age of the visitor in the face region of the object detected in the face detection step (a), determining whether the customer is a regular customer, calculating the number of visitors in the store and the number of regular customers, A face matching step of determining customer information;
(c) tracking the position of the body of the object, the face tracking and the head movement in the real-time image frame acquired from the visitor image information in the store in the face detection step (a) Location tracking step; And
(d) capturing an eye position of the object with the position coordinates of the nose in the face area tracked in the coarse position tracing step (c), determining a gaze position fixation time of the object whose position of the advertisement or the display article matches the gaze position of the object And an eye position and a time calculating step of calculating the eye position and the time information of the eyes. 2. The method according to claim 1, wherein the face detection step (a)
Acquiring real-time image frames by performing angle test processing on visitor image information collected in real time in the image collecting unit;
Comparing the difference between the obtained image frames with a predetermined threshold value, and detecting the face region of the object included in the image information by applying the adaboost face detection algorithm to the image frame having the difference between the image frames less than or equal to the threshold value Object recognition and gaze tracking method in integrated store management system based on intelligent image analysis technology. 3. The method according to claim 2, wherein the face detection step (b)
Applying a template matching algorithm to the face region of the object detected in the face detection step (a) to recognize the face and counting the number of visitors;
Accessing customer image information of a database;
Counting the number of regular customers by identifying whether the customer is a regular customer by classifying the sex and age of the visitor through face matching between the face area in which the face is recognized and the customer image information accessed in the database; And
And counting the number of visitors in the counted stores and the number of regular customers by time / daily / monthly / quarterly / year, and determining the regular customer information to be displayed on the force terminal. Object Recognition and Eye Tracking in Integrated Store Management System Based on Image Analysis Technology. 2. The method according to claim 1, wherein the face detection step (c)
Tracking a face of an object by tracking a body motion of the object in a real-time image frame acquired from the visitor image information in the face detection step (a);
And tracing and analyzing head movement in the tracked face area image to track the position coordinates of the nose within the face area. The intelligent image analysis technology based integrated store management system according to claim 1, And eye tracking method. 5. The method according to claim 4, wherein the face detection step (d)
Capturing an eye gaze position of the object with the position coordinates of the nose in the face area tracked in the nose position tracking step (c);
Confirming whether or not the position of the advertisement or the display product being displayed in the store matches the gaze position of the object; And
And counting and calculating eye position fixed time information of an object whose position of the advertisement or the displayed product matches the gaze position of the object, and calculating the object position in the integrated shop management system based on the intelligent image analysis technology. And eye tracking method.
KR1020160001531A 2016-01-06 2016-01-06 The object pursuit way in the integration store management system of the intelligent type image analysis technology-based KR101779096B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020160001531A KR101779096B1 (en) 2016-01-06 2016-01-06 The object pursuit way in the integration store management system of the intelligent type image analysis technology-based

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020160001531A KR101779096B1 (en) 2016-01-06 2016-01-06 The object pursuit way in the integration store management system of the intelligent type image analysis technology-based

Publications (2)

Publication Number Publication Date
KR20170082299A true KR20170082299A (en) 2017-07-14
KR101779096B1 KR101779096B1 (en) 2017-09-18

Family

ID=59358594

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020160001531A KR101779096B1 (en) 2016-01-06 2016-01-06 The object pursuit way in the integration store management system of the intelligent type image analysis technology-based

Country Status (1)

Country Link
KR (1) KR101779096B1 (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102077805B1 (en) * 2018-11-26 2020-02-14 (주)에이텐시스템 apparatus for analyzing the purchasing behavior pattern of client and Driving method thereof
KR20200034975A (en) * 2020-03-19 2020-04-01 (주)엑스엠디 SYSTEM FOR PROVIDING CUSTOMIZED WELCOME DATA BY RECOGNIZING CUSTOMER USING IoT DEVICES
WO2020130339A1 (en) * 2018-12-17 2020-06-25 한화테크윈 주식회사 Camera control apparatus and method for processing image captured by at least one camera
WO2020184855A1 (en) * 2019-03-08 2020-09-17 삼성전자 주식회사 Electronic device for providing response method, and operating method thereof
KR20200128927A (en) * 2019-05-07 2020-11-17 주식회사 엘토브 O2O(On-line to Off-line) BASED SYSTEM AND METHOD FOR SUGGESTING CUSTOMIZED INFORMATION
WO2021168268A1 (en) * 2020-02-21 2021-08-26 Nec Laboratories America, Inc. Video analytics system for dwell-time determinations
KR102322310B1 (en) * 2020-12-04 2021-11-08 이경세 Advertisement Apparatus based on Visitor at Store and Method thereof
KR20220096436A (en) * 2020-12-31 2022-07-07 주식회사 메이아이 Method, device and non-transitory computer-readable recording medium for analyzing visitors based on images in edge computing environment
KR102632995B1 (en) * 2023-03-23 2024-02-02 네모시스템(주) In-store product sales guide system based on AI that recommends customized products through big data analysis
US11941604B2 (en) 2018-12-17 2024-03-26 Hanwha Vision Co., Ltd. Automatic payment system

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20190063516A (en) 2017-11-30 2019-06-10 곽근규 Server for store management having advanced online order system

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101216123B1 (en) * 2011-07-08 2012-12-27 김호 Method and device for generating tracking information of viewer's face, computer-readable recording medium for the same, three dimensional display apparatus
KR101224879B1 (en) * 2012-07-04 2013-02-05 김민정 Shop management system using face recognition and method thereof

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102077805B1 (en) * 2018-11-26 2020-02-14 (주)에이텐시스템 apparatus for analyzing the purchasing behavior pattern of client and Driving method thereof
WO2020130339A1 (en) * 2018-12-17 2020-06-25 한화테크윈 주식회사 Camera control apparatus and method for processing image captured by at least one camera
US11941604B2 (en) 2018-12-17 2024-03-26 Hanwha Vision Co., Ltd. Automatic payment system
WO2020184855A1 (en) * 2019-03-08 2020-09-17 삼성전자 주식회사 Electronic device for providing response method, and operating method thereof
KR20200128927A (en) * 2019-05-07 2020-11-17 주식회사 엘토브 O2O(On-line to Off-line) BASED SYSTEM AND METHOD FOR SUGGESTING CUSTOMIZED INFORMATION
WO2021168268A1 (en) * 2020-02-21 2021-08-26 Nec Laboratories America, Inc. Video analytics system for dwell-time determinations
KR20200034975A (en) * 2020-03-19 2020-04-01 (주)엑스엠디 SYSTEM FOR PROVIDING CUSTOMIZED WELCOME DATA BY RECOGNIZING CUSTOMER USING IoT DEVICES
KR102322310B1 (en) * 2020-12-04 2021-11-08 이경세 Advertisement Apparatus based on Visitor at Store and Method thereof
KR20220096436A (en) * 2020-12-31 2022-07-07 주식회사 메이아이 Method, device and non-transitory computer-readable recording medium for analyzing visitors based on images in edge computing environment
KR102632995B1 (en) * 2023-03-23 2024-02-02 네모시스템(주) In-store product sales guide system based on AI that recommends customized products through big data analysis

Also Published As

Publication number Publication date
KR101779096B1 (en) 2017-09-18

Similar Documents

Publication Publication Date Title
KR101779094B1 (en) A integration store management system using intelligent type image analysis technology
KR101779096B1 (en) The object pursuit way in the integration store management system of the intelligent type image analysis technology-based
US11501613B2 (en) Varied update rates of shopping data for frictionless shoppers
CN110033298B (en) Information processing apparatus, control method thereof, system thereof, and storage medium
CN109726759B (en) Unmanned vending method, device, system, electronic equipment and computer readable medium
US20110199486A1 (en) Customer behavior recording device, customer behavior recording method, and recording medium
CN107291732B (en) Information pushing method and device
JP3800257B2 (en) Attention information measurement method and apparatus, and various systems using the same
US8010402B1 (en) Method for augmenting transaction data with visually extracted demographics of people using computer vision
CN108876504B (en) Unmanned selling system and control method thereof
US20150208043A1 (en) Computer system and method for managing in-store aisle
CN105164619B (en) Detection watches user attentively to provide individualized content over the display
US20130030875A1 (en) System and method for site abnormality recording and notification
KR102060694B1 (en) Customer recognition system for providing personalized service
US20080304754A1 (en) Method and system for targeted marketing by leveraging video-based demographic insights
JP5002441B2 (en) Marketing data analysis method, marketing data analysis system, data analysis server device, and program
US20130328765A1 (en) Signage system and display method by the same
JP2017033401A (en) Customer information collection device, customer information collection system and customer information collection method
US11810067B2 (en) Digitally managed shelf space marketplace
CN114365175A (en) Store-use information distribution device, store-use information distribution system provided with same, and store-use information distribution method
US11488315B2 (en) Visual and geolocation analytic system and method
US20240112135A1 (en) Dynamic Product Marketing Through Computer Vision
AU2017231602A1 (en) Method and system for visitor tracking at a POS area
CN111784405A (en) Off-line store intelligent shopping guide method based on face intelligent recognition KNN algorithm
KR102077630B1 (en) System and method for analyzing commercial based on pos and video

Legal Events

Date Code Title Description
A201 Request for examination
E902 Notification of reason for refusal
E701 Decision to grant or registration of patent right
N231 Notification of change of applicant
GRNT Written decision to grant