KR102191044B1 - Advertising systems that are provided through contents analytics and recommendation based on artificial intelligence facial recognition technology - Google Patents

Abstract

The present invention relates to a system for providing advertisement through adjustment analysis and recommendation based on artificial intelligence facial recognition technology to perform active advertisement and customized advertisement. According to the present invention, the system comprises a facial recognition unit (2) and an advertisement management unit (3) collecting advertisement content and extracting customized advertisements in accordance with a recognized face. The facial recognition unit (2) comprises an image photographing unit (4) for photographing customers in a shop or pedestrians outside the shop to acquiring a video, and a data processing unit (5) analyzing the acquired video to extract one or more pieces of facial data at the same time, converting the facial data into a vector to process a function, and detecting facial feature points to recognize the faces. The data processing unit (5) comprises: an extraction unit (6) analyzing an image inputted in real-time through the image photographing unit (4) to extract one or more pieces of facial data at the same time and converting the facial data into a vector to process a function; a preprocessing unit (7) removing reflected light, the angle of view, dust, and noise from the facial data processed by the extraction unit (6) to perform post-processing; a feature detection unit (10) simultaneously analyzing and detecting one or more facial feature points by a defined feature point parameter defined by determining the similarity of the facial data processed by the preprocessor (7); a facial determination unit (15) registering one or more facial images detected by the feature detection unit (10) in a learned database (13) or comparing and analyzing previously registered facial feature points to recognize a face; and a control unit (9) controlling components to perform facial recognition.

Description

Advertising systems that are provided through contents analytics and recommendation based on artificial intelligence facial recognition technology

The present invention relates to a system for providing advertisements by face recognition, and more specifically, by recognizing faces from images input in real time, analyzing facial characteristics based on artificial intelligence technology, and matching collected advertisement contents. It relates to an advertisement providing technology capable of efficiently analyzing and recommending advertisements suitable for individuals.

In recent years, the importance of advertising is increasing. A large amount of advertisements are being conducted through broadcast media such as TV, newspapers, and leaflets.

In particular, a lot of large discount marts are entering, and competition for sales between marts and corners is also fierce. Accordingly, various exhibition and promotion methods are being mobilized.

In addition, in the past, in order to advertise discount event products or new products, the method of creating flyers was the main one.

Producing advertisement flyers in this way has a considerable cost burden because it is mainly printed on high-quality paper and color due to its characteristics, it is difficult to measure its effectiveness, and causes various problems such as waste discharge and environmental destruction.

Alternatively, a method of collectively sending spam information data such as e-mail or text message is widely used, but recently, due to too much spam information, there is a problem in that many customers delete or process spam before reading.

Moreover, since the above-described conventional public relations methods are not a voluntary approach from the point of view of a customer who needs information, but are unilateral information delivery, it can be said that the efficiency compared to the cost required is very low. Therefore, a new customized promotion method that can be voluntarily accessed by customers who need information is required.

Patent Publication No. 10-2009-0083643 (Method, system and apparatus for providing public relations service using private broadcasting)

Accordingly, the present invention was conceived to solve such a problem, and an object of the present invention is to recognize individual characteristics by recognizing the faces of customers in the store or pedestrians outside the store, and matching the collected advertisement contents to each individual. It is to provide a technology that enables active advertisement and targeted customized advertisement by providing appropriate advertisement contents.

Another object of the present invention is to provide a system capable of recognizing faces by mounting an artificial intelligence camera or an artificial intelligence set-top box on a smartphone App, a web service of a personal computer, a face recognition edge terminal, and an advertisement display device. Or, it provides a technology that can provide recommended advertisement contents according to individual preferences.

Another object of the present invention is to provide a system for providing an advertisement by face recognition capable of searching for a specific face by detecting only a partial area of the face by switching from the general mode to the quick mode when face recognition needs to be performed quickly. .

In order to achieve the above object, an embodiment of the present invention,

A face including a face recognition unit (2) that recognizes a face from a video shot of customers in a store or pedestrians outside the store, and an advertisement management unit (3) that collects advertisement contents and draws and displays customized advertisements according to the recognized faces. As a recognition-based advertisement providing system,

The face recognition unit 2 includes an image capturing unit 4 for obtaining a video by photographing customers in the store or pedestrians outside the store;

It includes a data processing unit 5 that analyzes the acquired video to extract one or more face data, converts it into a vector, performs function processing, and detects facial feature points to recognize a face,

The data processing unit 5,

An extracting unit 6 which analyzes an image input in real time through the image capturing unit 4, extracts one or more face data at the same time, and converts the extracted one or more face data into a vector for functional processing;

A preprocessor 7 for post-processing by removing reflected light, angle of view, dust and noise from the face data processed by the extraction unit 6;

A feature detection unit 10 that simultaneously analyzes and detects one or more facial feature points according to the defined feature point parameters by determining the similarity of the face data processed by the pre-processing unit 7;

One or more face images detected by the feature detection unit 10 are registered in an arbitrary database 13 by classification of age groups of men and women from the face images detected based on an algorithm learned by a learned deep learning method A face discriminating unit 15 for recognizing a face by comparing and analyzing the facial feature points;

An output unit 11 for outputting the recognized result; And

An advertisement providing system 1 that is matched by face recognition is provided, which includes a control unit 9 that controls the components to perform face recognition.

Another embodiment of the present invention,

Face recognition unit (2) recognizes faces from videos taken of customers in the store or pedestrians outside the store, searches for and retrieves the best advertisements according to gender and age of the recognized face, and collects and classifies advertisements to retrieve them. As a method of providing advertisements in a manner of matching advertisement contents based on face recognition and recommending advertisement contents in an order in which the advertisements are displayed through a display,

Obtaining a moving picture by the image capturing unit 4 (S10);

An extraction step (S20) of analyzing an image input in real time through the image capturing unit 4 by the extracting unit 6, extracting one or more face data, converting it into a vector, and performing function processing;

A pre-processing step (S25) of removing dust and noise such as reflected light, angle of view, etc. from the extracted one or more face data by the pre-processing unit 7;

A feature detection step (S30) of simultaneously analyzing and detecting one or more facial feature points according to a defined feature point parameter by determining the similarity of the extracted one or more face data when the controller 9 is set to the general mode;

One or more face images detected by the preprocessor 7 are compared and analyzed with face data registered in the database 13 by the face discrimination unit 15, or randomly annotated into the database with random face data to determine gender and age. Classifying and outputting according to the emotional state (S40) and;

A customized advertisement step (S50) of collecting advertisement contents by the advertisement management unit 3 and drawing and displaying suitable advertisements according to the clustered faces of customers; And

Including the step (S60) of outputting advertising content that is matched or recommended to the artificial intelligence algorithm,

When the control unit 9 is set to the quick mode, the face is divided into a plurality of areas, and features are sequentially detected and matched from a single area, but when not matched to a specific face, other areas except the symmetrically arranged areas And a quick recognition step (S70) capable of quickly performing face recognition by detecting features of and comparing matched face data.

As described above, the system for providing advertisements by face recognition according to an embodiment of the present invention has the following advantages.

First, by interlocking the face recognition unit and the advertisement management unit, the face is recognized from the image input in real time, and based on artificial intelligence technology, the individual's face characteristics are identified and then matched with the collected advertisement contents to efficiently provide advertisements suitable for each individual. There is an advantage it can provide.

Second, by recognizing the face of a person entering or passing through a specific location in an enclosed space such as a building or an open space such as outdoors, the person's gender, age, and emotional state such as emotions and sorrows are determined based on artificial intelligence technology. It has the advantage of being able to analyze and display a customized advertisement suitable for the situation as an image on a billboard.

Third, when providing advertisement content, there is an advantage in that it may include a recommendation type advertisement according to the preference of a specific product by gender or age in the surrounding commercial district.

Fourth, provision of advertisements in consideration of public interest also has the advantage of providing customized public notification advertisements by identifying the floating population in a specific area.

1 is a diagram schematically showing the structure of an advertisement providing system based on face recognition according to an embodiment of the present invention.
FIG. 2 is a block diagram schematically showing the structure of the data processing unit shown in FIG. 1.
3 is a block diagram schematically showing the structure of the extracting unit shown in FIG. 2.
FIG. 4 is a diagram illustrating a state in which face data is primarily divided into regions in order to recognize a face by the system for providing an advertisement based on face recognition shown in FIG. 1.
FIG. 5 is a diagram illustrating a state in which face data divided into regions shown in FIG. 4 is secondly calculated and analyzed by brightness for each pixel.
FIG. 6(a) is a diagram showing a state in which position, size, and ratio are recognized by distributing a plurality of reference points on an outline in face data, and FIG. 6(b) is a diagram showing a state in which a plurality of reference points are distributed inside a face.
7 is a diagram showing a CPU structure of the data processing unit shown in FIG. 1.
8 is a block diagram schematically showing the structure of the advertisement management unit shown in FIG. 1.
9 is a flow chart showing a method of providing face recognition-based advertisement according to another embodiment of the present invention.
10 is a flowchart showing the procedure of the method for providing advertisement based on face recognition of FIG. 9.
FIG. 11 is a flow chart showing a process of recognizing a face in a fast mode among the methods for providing advertisements based on face recognition shown in FIG. 9.

Hereinafter, a system for providing advertisements by face recognition according to the present invention will be described in detail with reference to the accompanying drawings.

As shown in Figs. 1 to 8, the system 1 for providing advertisements by face recognition proposed by the present invention includes a face recognition unit 2 for recognizing a face from a video shot of customers in a store or pedestrians outside the store. Wow; It is composed of an advertisement management unit 3 that collects advertisement contents and draws and displays appropriate advertisements according to the recognized customer's face.

In addition, such an advertisement providing system can be mounted on various types of media such as smartphones, PCs, and large billboards.

In more detail,

The face recognition unit 2 is an artificial intelligence-based face recognition terminal, and uses its own built-in camera or a general IP CCTV camera, and the network is a local area network (LAN) wide area network (WAN), virtual private network (VPN), mobile communication network, or All kinds of wired/wireless networks such as satellite communication networks are used.

The face recognition unit 2 includes an image photographing unit 4 for obtaining a moving picture by photographing customers in the store or pedestrians outside the store; And a data processing unit 5 that analyzes the acquired video to extract one or more face data, converts it into a vector, performs function processing, and detects facial feature points to recognize a face.

The image capture unit 4 includes a camera, for example, an IP camera, an IR camera, an HD camera, a depth camera, etc., and may also be a camera mounted on a smartphone.

The image data thus obtained may be analyzed by the data processing unit 5 to recognize a face.

The data processing unit 5 analyzes an image input in real time through the image capturing unit 4, extracts one or more face data at the same time, and converts the extracted one or more face data into a vector for functional processing ( 6) and; A preprocessor 7 for post-processing by removing reflected light, angle of view, dust and noise from the face data processed by the extraction unit 6; A feature detection unit 10 that simultaneously analyzes and detects one or more facial feature points according to the defined feature point parameters by determining the similarity of the face data processed by the pre-processing unit 7; A face determination unit 15 that registers one or more face images detected by the feature detection unit 10 in the learned database 13 or compares and analyzes the previously registered facial feature points to recognize the gender, age, and emotional state of the face Wow; In addition, it includes a control unit 9 that controls the components to perform facial recognition.

To describe this data processing unit 5 in more detail,

The extraction unit 6 extracts face data from the input video data.

The extraction unit 6 includes a face segmentation module 20 for dividing the face into a plurality of regions, as shown in FIG. 3; A local feature analysis module (22) for recognizing a shape of a pixel; A vector operation module (Vector feature analysis; 24) that calculates a vector for the brightness of a pixel; It includes a module (Surface texture analysis; 26) that analyzes the texture of the surface of the face.

In describing this extraction process, as shown in FIG. 4, a face area is extracted in a shape of a square or an oval from an image input in real time.

And, after dividing the extracted face area into a plurality of areas by the face segmentation module 20, a P point (Peak Point) is extracted from the face image, and eyes, nose, mouth, and ears are extracted around the P point. Become aware.

That is, when tracking or recognizing an object in an image, key-points are extracted from the image and matched. In this case, it is important to extract a corner point. There are various methods of extracting corner points, such as the FAST method.

Therefore, it is necessary to determine whether an arbitrary P point is a corner point, and it is determined by the following process.

As shown in FIG. 5, a random pixel is first selected by the local shape analysis module 22 from the extracted face data. At this time, the brightness of the selected pixel P is defined as Ip, and the threshold value is defined as t.

Then, a virtual circle is formed around the point P, and the radius of the circle is assumed to be about 3 pixels, and values of 16 pixels in contact with the circle are extracted. Of course, the size of the radius can be changed accordingly.

And, by the vector calculation module 24, n or more pixels that are brighter than Ip+t (>p+t) by a certain value or more (>p+t) are continuously in the circle made of 16 pixels, or are more than a certain value than Ip-t. If n or more dark (<pt) pixels are consecutive, p is determined as a corner point.

For example, whether pixels 1 and 9 are brighter than P is calculated first, and pixels 5 and 13 are calculated. If P is a corner, then at least three of the four pixels computed, i.e. pixels 1, 5, 9 and 13, are brighter than IP+t or darker than IP-t.

In this case, the number of consecutive pixels centered on P may vary and be applied in various ways, such as 4, 9, 12, as well as 3.

Further, in the above recognition process, when the number of n is small, for example, when it is less than 12, there is a problem in that the number of pixel candidates for selecting P is small, and thus a lot of pixel candidates cannot be removed. In addition, when detecting a feature of a face image, since it depends on the distribution of the corner shape, the pixel selection may not be optimal.

And, pixels judged not to be corners are excluded from the analysis. In addition, there are cases where it is difficult to determine a corner because multiple features may be detected around the corner.

Therefore, in order to compensate for these shortcomings, the following process is performed.

That is, corner points are detected by machine learning or deep learning.

In more detail, the process for detecting a corner point from a face image is performed in the same manner as described above.

And, for all detected pixels, each pixel (x) of 16 pixels around the pixel is stored as a vector.

At this time, when the brightness value of the pixel is expressed as a vector (P) as shown in the following equation by the vector calculation module 24, when it is much brighter than P (Pb), when it is much darker than P (Pd), when it is similar to P ( It is classified into three values of Ps). And, using this, the brightness distribution of the pixels on the circumference is expressed as a 16-dimensional ternary.

--식1

--Equation 1

A new Boolean variable is defined as Kp in Equation 1, where Kp is True if P is a corner, or False.

Therefore, the Kp variable is applied to the above equation and processed based on a decision tree.

Until the entropy of Kp becomes 0, the above equations Pd, Ps, and Pb are repeatedly applied.

Through this process, a decision tree capable of detecting each corner is created.

At this time, when a point P is recognized as a vertex, points adjacent to P are often detected as vertices.

Therefore, to solve this problem, a pre-processing step is performed using a non-maximal suppression method.

The non-maximum suppression method is a method in which when several adjacent points are detected as vertices, only the points with the maximum peak are left and the rest are removed.

In the method of determining P by the decision tree, since the vertex is determined in an On/Off method, it is difficult to quantify the degree of depth. A separate numerical score function as follows: (V) is applied (Equation 2)

------식(2)

------Equation (2)

{V: score function, x: pixel value, P: corner point}

When calculating the point P by the ratio-maximum suppression method, each of the corner points detected through the decision tree is input into the equation (2) to calculate V.

That is, V denotes a value obtained by summing all the absolute values (|x-p|) of the difference between p and the 16 pixel values (x) surrounding it. Then, V is calculated for two adjacent p-values, and points with the maximum cornerity are identified by discarding the smaller V value.

In this way, the shape and position of the eyes, nose tips, eyebrows, both ends of the mouth, and ears are determined by the reference points identified as corners. These points are useful in distinguishing each individual because their shape is greatly different for each individual and the surrounding shape is also complex.

Other reference points are placed on the cheek or in the middle of the forehead, etc. These points do not have many individual characteristics, but are necessary to grasp the overall shape of the face.

As described above, one or more face data extracted by the facial feature detection unit 10 are processed by the preprocessor 7, and the preprocessor 7 removes dust and noise such as reflected light, an angle of view, and the like from the face data.

After grasping the P points in this way, as shown in FIGS. 6A and 6B, the P points are used as detailed reference points to find the position , size , and ratio of the face.

That is, the facial feature detection unit 10 selects each reference point in a random order and converts it into a vector, and applies it to the similarity function, but changes the position little by little to increase the similarity of the face image image. In this process, the vector size is gradually increased from focus 1 to 5.

As shown in Fig. 6A, many reference points are distributed in the shape of the face, and the number of reference points is relatively small . As shown in FIG. 6B, a number of reference points are distributed at similar intervals inside the face .

In the case of focus 1 in the above process, if the previously set values are used as they are, 8 pixels will be the maximum size of the position difference vector. However, the direction and frequencies of the wavelet can be arbitrarily set by the user according to the training set or the degree of performance desired, so the pixel units in steps 1 and 2 are also adjusted accordingly, and the face is faced according to the pixel interval between the defined P points. Detect the feature point of

And, if necessary, the texture analysis module 26 analyzes the texture of the surface of the face.

In this way, features such as shape, brightness, and texture of multiple pixels of the face image are vectorized by the facial feature detection unit 10, processed and extracted by a function, and then combined with each other to obtain one or more facial feature points based on the feature point parameters. Simultaneous analysis and detection.

Meanwhile, in order to shorten the recognition time for the face image processed by the preprocessor 7, after detecting only a part of the face image, if a specific face does not match, the remaining undetected face image is additionally detected and the recognition process is performed. May be.

That is, in the case of changing and setting the normal mode to the quick mode by the control unit 9, the face is divided by predetermined regions by the face segmentation module 20, and then the facial features of the corresponding region are determined by the facial feature detection unit 10. After detection, the face determination unit 15 retrieves and compares matched face data from the database 13.

In more detail, horizontal and vertical lines are formed around the nose of the face and divided into a plurality of areas, for example, 4 areas, which are even numbers of symmetrical areas, and the first, second, and second areas are clockwise from the upper left area. It is set as the 3rd and 4th area.

Then, a feature of the left eye region, which is the first region, is first detected. Then, it is calculated whether there is a part in the database 13 that matches the corresponding area.

If there is no matched part, the feature of the fourth area, that is, the area immediately below the left eye area is detected, and a matching face image is searched.

That is, since features of the second area symmetrical with the first area can be processed using a mirror technique, it is preferable to detect the features of the fourth area that are not symmetrical.

And, when it is not detected in the fourth area, the entire face area is detected.

In this way, a face is divided into a plurality of areas, and features are sequentially detected and matched from a single area, but when a specific face is not matched, a matched face is detected by detecting features of other areas except the symmetrically arranged areas. By comparing the data, face recognition can be performed quickly.

In addition, prior to the detection of the feature points for the first to fourth regions, the recognition time may be quickly shortened by first performing the feature on the specific shape of the face. For example, if a certain part of the face has features such as large dots or scars, facial asymmetry, etc., and can be easily detected by the face discrimination unit 15, the recognition time can be shortened by processing these feature points with priority. .

Meanwhile, the face determination unit 15 compares one or more face images detected by the preprocessor 7 with face data registered in the database 13 to recognize the gender, age, and emotional state of the corresponding face.

In more detail,

The face discrimination unit 15 recognizes gender, age, and emotional state by extracting facial features by analyzing the shape of the face. The face determination unit 15 may be processed by a central processing unit (CPU) equipped with a face recognition analysis engine.

For reference, the central processing unit is a face analysis engine, GPU, ISP (Image Signal Processor), ALSA (Advanced Linux Sound Architecture), RTSP (Real Time Streaming Protocol), communication module, as shown in FIG. It transmits and receives signals.

The face determination unit 15 can use various algorithms to analyze the shape of the face. For example, it is a method that analyzes the shape by comparing the shape of the face using a deep learning method and determining the similarity. .

In other words, deep learning is a machine learning method built on the basis of an artificial neural network (ANN) to enable computers to learn on their own like humans using multiple data.

When an artificial neural network is used, classification and clustering of face data is possible, and similarity can be determined by placing various layers on the data desired for classification or clustering.

That is, the similarity can be determined by extracting the shape features of the face data with an artificial neural network and then classifying or clustering each shape by using the features as an input value of another machine learning algorithm.

Such an artificial neural network includes a deep neural network, and the deep neural network means a neural network composed of several layers among neural network algorithms.

In other words, the artificial neural network is composed of multiple layers, each layer is composed of several nodes, and each node actually classifies the shape, and this computational process simulates the process that occurs in the neurons constituting the human neural network. Designed to do.

When a node receives a stimulus larger than a certain size, it reacts, and the magnitude of the response is roughly proportional to the product of the input value and the node's coefficient (or weights).

In general, a node receives multiple inputs and has as many coefficients as the number of inputs. Therefore, it is possible to give different weights to different inputs by adjusting this coefficient.

Finally, all the multiplied values are added and the sum is fed into the input of the activation function. The result of the active function corresponds to the output of the node, and this output is ultimately used for classification or regression analysis.

Each layer consists of several nodes, and whether to activate/deactivate each node is determined according to the input value. At this time, the input data becomes the input of the first layer, and after that, the output of each layer becomes the input of the next layer again.

All coefficients change gradually during the shape learning process of the face data, and as a result, each node reflects which inputs are important. And training of neural networks is the process of updating these coefficients.

When learning the shape of face data, in this deep neural network, features of different layers are learned for each layer.

In other words, simple and specific features are learned for features at lower levels, {eg, shapes constituting the shape of face data}, and more complex and abstract features for features at higher levels. {Example: nose height, eyebrow shape, mouth tail shape, eye shape}

Through this abstract learning process, deep neural networks understand high-dimensional data, and hundreds of millions and billions of coefficients are involved in this process. (A nonlinear function is used in this process.)

In addition, deep neural networks can use the data to understand the potential structures of the data. In other words, potential structures such as contours of face data, nose height, eye shape, lip shape, eyebrow shape, and forehead shape can be identified. Through this, even if the data is not labeled, the similarity between data can be effectively grasped, and as a result, deep neural networks are effective in clustering face data.

For example, a large amount of face data can be input using a neural network, and similar face data can be collected and classified.

The difference between these deep neural networks from general machine learning is that feature extraction is performed automatically.

Previously, in order to extract effective features, experts in related fields had to devise and apply formulas or methods that directly extract features for a long time. This method has a problem that it takes a lot of time to develop, evaluate and supplement.

Deep neural networks are algorithms designed to allow computers to take over these processes, and they learn to perform much faster and more effectively than humans.

Even when learning unlabeled data, neural networks can automatically extract features of facial data. There are several methods of this automatic extraction. Usually, this process learns so that the output when passed through the neural network is the same as the input.

No matter what kind of label (using the input as it is/using a separate label), the neural network finds a correlation between the input and the output. In some cases, a neural network can be trained with labeled data to some extent, and then unlabeled data can be added to continue learning. This method can maximize the performance of the neural network.

The final layer of the deep neural network is the output layer. In most cases, the activation function of the output layer is logistic or softmax. In the output layer, the probability of a specific label can be finally obtained. For example, when the face data is input, whether the face contour is long vertically, horizontally, the height of the nose, the sagging of the corner of the mouth, the sagging of the tail of the eyes, the size and length of wrinkles, etc. can be obtained with each probability.

First, all coefficients of the neural net are initialized before learning starts. Then, the face data is repeatedly input to proceed with learning. If the learning proceeds smoothly, the coefficients will be updated to appropriate values, and various classifications and predictions are possible with this artificial neural network.

Within the learning process, the process of updating these coefficients occurs repeatedly.

The principle of coefficient update is to first estimate the coefficient, measure the error that occurs when using the coefficient, and then update the coefficient slightly based on the error.

In this case, the coefficients of the neural network are summed and called a model, and the model may be in an initialized state or a state in which training is completed.

The initialized model can't do any meaningful work, but as the learning progresses, the model outputs a result similar to the real thing, not an arbitrary value.

This is because the artificial neural network knows nothing before data is input, and the same is the reason for initializing the coefficients to arbitrary values. And as the data is read, the coefficients are gradually updated in the correct direction.

Through this update process, the artificial neural network can cluster similar face data by classifying the input face data.

At this time, the face data may be clustered according to types, for example, according to gender, age, and emotional state.

And, emotions can be classified into joy, sadness, anger, and depression.

The face data clustered in this way is registered in the database 13.

Meanwhile, as shown in FIG. 8, the advertisement management unit 3 collects advertisement contents and draws and displays appropriate advertisements according to the clustered faces of customers.

The advertisement management unit 3 includes: a collection module 30 for collecting and storing advertisement contents; A classification module 32 for classifying the collected advertisement content; A search module 34 for searching a path for selecting advertisement content corresponding to the detected facial feature; A recommendation module 36 for recommending the searched advertisement content; A display module 38 for displaying the recommended advertisement content; It includes a management module 40 for managing user information.

In more detail,

The collection module 30 collects and stores various advertisement contents online or offline. At this time, there are various methods of collecting advertisement contents online, and advertisement contents may be searched using, for example, a crawler or scraping.

The classification module 32 classifies the collected advertisement content according to characteristics. For example, it is classified into health-related advertisements, sports-related advertisements, health-related advertisements, and financial-related advertisements.

The search module 34 searches for a path for selecting advertisement content corresponding to the facial feature detected by the facial feature detection unit 10.

The recommendation module 36 recommends the advertisement content for which the path is searched by extracting the advertisement content from the database according to the classified face type. For example, advertisement content appropriate for age is retrieved. In the case of younger people, advertisement content for children is retrieved, and health-related content is retrieved for the elderly.

In addition, in the case of young people, advertising content related to health is withdrawn, and in the case of the elderly, financial related content is withdrawn.

In addition, advertisement content can be retrieved according to gender. For men, sports-related content is retrieved, and for females, cosmetic-related content can be retrieved.

In addition, advertising content can be withdrawn according to emotions. In the case of joy, travel content is withdrawn, in case of sadness, alcohol-related content is withdrawn, and in case of anger, female-related advertising content is withdrawn, and depression. In the case of status, bright content is fetched.

In this way, the recommendation module 36 selects and retrieves appropriate advertisement content according to the recognized face data.

In addition, the display module 38 may proceed with advertisement by displaying the recommended advertisement content on the advertisement display window.

At this time, the advertisement display window 32 includes the advertisement display window 32 of various types, for example, an LCD, an LED, and the like.

Hereinafter, an operation process of the advertisement providing system by face recognition according to a preferred embodiment of the present invention will be described in more detail with reference to the accompanying drawings.

As shown in FIGS. 9 and 10, the method of recognizing a face by the advertisement providing system 1 by face recognition is from a video of a customer in a store or a pedestrian outside the store by the face recognition unit 2 Faces are recognized, advertisement contents are collected, and appropriate advertisements are drawn out and displayed according to the recognized customer's face.

When explaining in detail the method of providing such a face recognition advertisement,

Obtaining a moving picture by the image capturing unit 4 (S10);

An extraction step (S20) of analyzing an image input in real time through the image capturing unit 4 by the extracting unit 6, extracting one or more face data, converting it into a vector, and performing function processing;

A pre-processing step (S25) of removing dust and noise such as reflected light, angle of view, etc. from the extracted one or more face data by the pre-processing unit 7;

A feature detection step (S30) of simultaneously analyzing and detecting one or more facial feature points according to a defined feature point parameter by determining the similarity of the extracted one or more face data when the controller 9 is set to the general mode;

Comparing and analyzing one or more face images detected by the preprocessor 7 with face data registered in the database 13 by the face determination unit 15, classifying and outputting them according to gender, age, and emotional state (S40) Wow;

A customized advertisement step (S50) of collecting advertisement contents by the advertisement management unit 3 and drawing and displaying suitable advertisements according to the clustered faces of customers; And

And outputting the advertisement content (S60).

In this advertisement providing method,

In the video acquisition step (S10), a face image is acquired in real time by photographing a video or the like by a camera.

In addition, in the face data extraction step S20, face data is extracted from an image input in real time.

That is, first, face data is extracted in a shape of a square or an oval, as shown in FIG. 4.

Then, P point (Peak Point) is extracted from the extracted face area, and eyes, nose, mouth, and ears are recognized around the P point.

That is, when tracking or recognizing an object in an image, keypoints are extracted from the image and matched. In this case, it is important to extract a corner point.

Therefore, it is necessary to determine whether an arbitrary point P is a corner point, and it is determined according to the following definition.

As shown in FIG. 5, a random pixel is first selected from the extracted face data. At this time, the brightness of the selected pixel P is defined as Ip, and the threshold value is defined as t.

Then, a virtual circle is formed around the point P, and the radius of the circle is assumed to be about 3 pixels, and values of 16 pixels in contact with the circle are extracted. Of course, the size of the radius can be changed accordingly.

In addition, n or more pixels (>p+t) brighter than Ip+t (>p+t) or more (<pt) pixels darker than Ip-t are continuous within a circle consisting of 16 pixels. If more than two are consecutive, p is judged as a corner point.

For example, whether pixels 1 and 9 are brighter than P is calculated first, and pixels 5 and 13 are calculated. If P is a corner, then at least three of the four pixels computed, i.e. pixels 1, 5, 9 and 13, are brighter than IP+t or darker than IP-t.

In this case, the number of consecutive pixels centered on P may vary and be applied in various ways, such as 4, 9, 12, as well as 3.

Further, in the above recognition process, when the number of n is small, for example, when it is less than 12, there is a problem in that the number of pixel candidates for selecting P is small, and thus a lot of pixel candidates cannot be removed. In addition, when detecting a feature of a face image, since it depends on the distribution of the corner shape, the pixel selection may not be optimal. And, pixels judged not to be corners are excluded from the analysis. In addition, there are cases where it is difficult to determine a corner because multiple features may be detected around the corner.

Therefore, in order to compensate for these shortcomings, the following process is performed.

That is, corner points are detected by machine learning or deep learning.

In more detail, the process for detecting a corner point from a face image is performed in the same manner as described above.

And, for all detected pixels, 16 pixels around the pixels are stored as vectors.

At this time, the brightness value of the pixel is classified into three values: when it is much brighter than P, when it is much darker than P, and when it is similar to P as shown in the equation below. And, using this, the brightness distribution of the pixels on the circumference is expressed as a 16-dimensional ternary.

In the above equation, a new Boolean variable is defined as Kp, where Kp is True if P is a corner, or False otherwise.

Therefore, the Kp variable is applied to the above equation and processed based on a decision tree.

Until the entropy of Kp becomes 0, the above equations Pd, Ps, and Pb are repeatedly applied.

Through this process, a decision tree capable of detecting each corner is created.

At this time, when a point P is recognized as a vertex, points adjacent to P are often detected as vertices.

Therefore, in order to solve this problem, post-processing is performed using a non-maximal suppression method.

The non-maximum suppression method is a method in which when several adjacent points are detected as vertices, only the points with the maximum peak are left and the rest are removed.

In the method of determining P by the decision tree, since the vertex is determined in an On/Off method, it is difficult to quantify the degree of depth, so a separate numerical function as follows is applied. do.

------식(2)

------Equation (2)

When calculating the point P by the ratio-maximum suppression method, each of the corner points detected through the decision tree is input into the equation (2) to calculate V.

That is, V denotes a value obtained by summing all the absolute values of the difference between p and 16 pixel values surrounding it. Then, V is calculated for two adjacent p-values, and points with the maximum cornerity are identified by discarding the smaller V value.

When defining the reference points in this way, the shape and position of the eyes, the tip of the nose, the eyebrows, the ends of the mouth, and the ears are basically determined. These points are useful in distinguishing each individual because their shape is greatly different for each individual and the surrounding shape is also complex.

Other reference points are placed on the cheek or in the middle of the forehead, etc. These points do not have many individual characteristics, but are necessary to grasp the overall shape of the face.

In the pre-processing step S25, the pre-processing unit 7 performs a pre-processing step of removing and correcting dust and noise such as reflected light, angle of view, and the like from the face data.

Then, in the face feature detection step (S25), after determining the P points, as shown in FIGS. 6A and 6B, the P points are used as detailed reference points to find the position , size , and ratio of the face.

In other words, each reference point is selected in a random order and the position is changed little by little to increase the similarity of the face image image. In this process, the focus is gradually raised from 1 to 5.

In the case of Fig. 6A, many reference points are distributed in the shape of the face, and the number of reference points is relatively small. In the graphs of FIG. 6B, a number of reference points are distributed at similar intervals inside the face .

In the case of focus 1 in the above process, if the previously set values are used as they are, 8 pixels will be the maximum size of the position difference vector. However, the direction and frequencies of the wavelet can be arbitrarily set by the user according to the training set or the degree of performance desired, so the pixel units in steps 1 and 2 are also adjusted accordingly, Detect feature points.

When the facial feature detection step (S25) is completed as described above, the face determination step (S40) proceeds.

In the face determination step (S40), one or more face images detected by the preprocessor 7 are compared with face data registered in the database 13 by the face determination unit 15 to identify the corresponding face, and They are classified according to their emotional state.

That is, the face determination unit 15 extracts features of the face by analyzing the shape of the face and distinguishes it from other faces. The face determination unit 15 may be processed by a central processing unit (CPU) equipped with a face recognition analysis engine.

The face determination unit 15 can use various algorithms to analyze the shape of the face. For example, it is a method that analyzes the shape by comparing the shape of the face using a deep learning method and determining the similarity. .

In other words, when learning the shape of face data, in the deep neural network, features of different layers are learned for each layer.

In other words, simple and specific features are learned for features at lower levels, {eg, shapes constituting the shape of face data}, and more complex and abstract features for features at higher levels. {Example: nose height, eyebrow shape, eye shape, mouth tail}

Through this abstract learning process, deep neural networks understand high-dimensional data, and hundreds of millions and billions of coefficients are involved in this process. (A nonlinear function is used in this process.)

In addition, deep neural networks can use the data to understand the potential structures of the data. In other words, potential structures such as contours of face data, nose height, eye shape, lip shape, eyebrow shape, and forehead shape can be identified. Through this, even if the data is not labeled, the similarity between data can be effectively grasped, and as a result, deep neural networks are effective in clustering face data.

For example, a large amount of face data can be input using a neural network, and similar face data can be collected and classified.

The final layer of the deep neural network is the output layer. In most cases, the activation function of the output layer is logistic or softmax. In the output layer, the probability of a specific label can be finally obtained. For example, when face data is input, whether the contour of the face is long vertically, horizontally, the height of the nose, and the sagging of the tail of the eye can be obtained with each probability.

In this way and by repeatedly inputting face data, learning is performed. If the learning proceeds smoothly, the coefficients will be updated to appropriate values, and various classifications and predictions are possible with this artificial neural network.

Within the learning process, the process of updating these coefficients occurs repeatedly.

Through such an update process, the artificial neural network classifies the input face data, thereby clustering similar face data according to gender, age, and emotional state. Then, the clustered face data is registered in the database 13.

On the other hand, as another embodiment of the present invention, a quick recognition step (S70) may be performed. That is, in the quick recognition step (S70), in order to shorten the recognition time for the face image extracted by the extraction unit 6, only a part of the face image is detected, and if a specific face is not matched, the remaining undetected face images are added. It is detected by and quickly progresses the recognition process.

That is, when the control unit 9 is set to the fast mode, the face is divided into a plurality of areas, and features are sequentially detected and matched from a single area, but when a specific face is not matched, the symmetrically arranged area is excluded. Face recognition can be quickly performed by detecting features of different regions and comparing matching face data.

The rapid recognition step (S70) includes a first step (S100) of changing and setting the normal mode to a fast mode by the control unit 9, as shown in FIG. 11; A second step (S110) of dividing the face into a plurality of even areas by the face division module 20; A third step (S120) of primarily detecting facial features sequentially from a singular number of areas by the facial feature detection unit 10; A fourth step (S130) of secondaryly detecting features of regions that are not symmetrically arranged if they do not match; If not matched in the fourth step (S130), a fifth step (S140) of thirdly detecting and matching features of the entire face is included.

In more detail, in the first step (S100), if the user needs to quickly perform face recognition, the mode of the control unit 9 can be quickly performed by changing the mode of the controller 9 from the normal mode to the fast mode.

In the second step (S110), the face is divided into a plurality of regions by the face segmentation module 20. That is, horizontal and vertical lines are formed around the nose of the face and divided into a plurality of areas, for example, 4 areas, which are even numbers of symmetrical areas, and the first, second, third, and fourth areas are clockwise from the upper left area. Set to the area.

In the third step (S120), the facial feature detection unit 10 primarily detects facial features sequentially from a singular number of areas. That is, the feature of the left eye region, which is the first region, is first detected. Then, it is calculated whether there is a part in the database 13 that matches the corresponding area.

If there is no matched part, as a fourth step (S130), a feature of a fourth region, that is, a region immediately below the left eye region is detected to search for a matched face image.

That is, since features of the second area symmetrical with the first area can be processed using a mirror technique, it is preferable to detect the features of the fourth area that are not symmetrical.

In addition, if it is not detected even in the fourth region, the entire face region is detected as a fifth step (S140).

In this way, a face is divided into a plurality of areas, and features are sequentially detected and matched from a single area, but when a specific face is not matched, a matched face is detected by detecting features of other areas except the symmetrically arranged areas. By comparing the data, face recognition can be performed quickly.

In addition, in the third step (S120), prior to the detection of the feature points for the first to fourth regions, the recognition time may be quickly shortened by adding a step of first performing a feature on the peculiar shape of the face. . For example, if a certain part of the face has features such as large dots or scars, facial asymmetry, etc., and can be easily detected by the face discrimination unit 15, the recognition time can be shortened by processing these feature points with priority. .

When the step of recognizing faces is completed as described above, a customized advertisement step (S50) of collecting advertisement contents by the advertisement management unit 3 and drawing and displaying suitable advertisements according to the clustered customer's faces is performed.

In more detail,

First, the collection module 30 collects and stores various advertisement contents online or offline. At this time, there are various methods of collecting advertisement contents online, and advertisement contents may be searched using, for example, a crawler or scraping.

When advertisement contents are collected, the classification module 32 classifies the collected advertisement contents according to characteristics. For example, it is classified into health-related advertisements, sports-related advertisements, health-related advertisements, and financial-related advertisements.

Further, the search module 34 searches for a path for selecting an advertisement content corresponding to the facial feature detected by the facial feature detection unit 10.

When the path is searched, the recommendation module 36 recommends the advertisement content for which the path is searched by extracting the advertisement content from the database according to the classified face type. For example, advertisement content appropriate for age is retrieved. In the case of younger people, advertisement content for children is retrieved, and health-related content is retrieved for the elderly.

In addition, in the case of young people, advertising content related to health is withdrawn, and in the case of the elderly, financial related content is withdrawn.

In addition, advertisement content can be retrieved according to gender. For men, sports-related content is retrieved, and for females, cosmetic-related content can be retrieved.

In addition, advertising content can be withdrawn according to emotions. In the case of joy, travel content is withdrawn, in case of sadness, alcohol-related content is withdrawn, and in case of anger, female-related advertising content is withdrawn, In the case of status, bright content is fetched.

In this way, the recommendation module 36 selects and retrieves appropriate advertisement content according to the recognized face data.

In addition, the display module 38 may perform advertisement by displaying the recommended advertisement content on the advertisement display window.

At this time, the advertisement display window 32 includes the advertisement display window 32 of various types, for example, an LCD, an LED, and the like.

Claims (10)

A face including a face recognition unit (2) that recognizes a face from a video shot of customers in a store or pedestrians outside the store, and an advertisement management unit (3) that collects advertisement contents and draws and displays customized advertisements according to the recognized faces. As a recognition-based advertisement providing system,
The face recognition unit 2 includes an image capturing unit 4 for obtaining a video by photographing customers in the store or pedestrians outside the store;
It includes a data processing unit 5 that analyzes the acquired video to extract one or more face data, converts it into a vector, performs function processing, and detects facial feature points to recognize a face,
The data processing unit 5,
An extracting unit 6 which analyzes an image input in real time through the image capturing unit 4, extracts one or more face data at the same time, and converts the extracted one or more face data into a vector for functional processing;
A preprocessor 7 for post-processing by removing reflected light, angle of view, dust and noise from the face data processed by the extraction unit 6;
A feature detection unit 10 that simultaneously analyzes and detects one or more facial feature points according to the defined feature point parameters by determining the similarity of the face data processed by the pre-processing unit 7;
A face determination unit 15 that registers one or more face images detected by the feature detection unit 10 in the learned database 13 or compares and analyzes previously registered facial feature points to recognize a face;
An output unit 11 for outputting the recognized result; And
An extraction unit 6, a pre-processing unit 7, a feature detection unit 10, a face discrimination unit 15, and a control unit 9 that controls the output unit 11 to perform facial recognition,
The extraction unit 6 divides the face area into a plurality of squares or ovals from the image input in real time, extracts a P point (Peak Point) from the extracted face area, and focuses on the P point. The nose, mouth, and ears are recognized, and when determining whether the point P is a corner point, a virtual circle with a predetermined pixel radius is formed around the point P, and the value is determined by the pixel value in contact with the circle.
If there are n or more consecutive (>p+t) pixels brighter than point P (>p+t) or n or more dark (<pt) pixels darker than a certain value, p is determined as a corner point,
To determine whether P is a vertex, it is determined by a decision tree method,
The brightness value of a pixel is classified into three values: a case much brighter than P, a case much darker than P, and a case similar to P. After expressing the brightness distribution of pixels on the circumference as a 16-dimensional ternary, this Enter the vector into the decision tree to determine whether the point P is,
The vertex is recognized by processing using a non-maximal suppression method,
In the method of determining P by the decision tree, since the vertex is determined in an On/Off method, a function is applied according to the following equation,
When calculating the point P by the non-maximum suppression method, input each corner point detected through the decision tree into Equation (1) to calculate V, and then a V value higher than the corresponding point P among adjacent corner points. An advertisement providing system (1) by face recognition that grasps points with the greatest cornerity in a manner that sequentially removes corner points having a.

------Equation 1
{V: score function, x: pixel value, P: corner point} The method of claim 1,
The extraction unit 6 includes a face segmentation module 20 that divides a face into a plurality of regions; A local feature analysis module (22) for recognizing a shape of a pixel; A vector operation module (Vector feature analysis; 24) that calculates a vector for the brightness of a pixel; An advertisement providing system (1) by face recognition including a module (Surface texture analysis; 26) for analyzing the texture of the face of the face. delete delete delete The method of claim 1,
The advertisement management unit 3 includes a collection module 30 for collecting and storing advertisement contents; A classification module 32 for classifying the collected advertisement content; A search module 34 for searching a path for selecting advertisement content corresponding to the detected facial feature; A recommendation module 36 for recommending the searched advertisement content by artificial intelligence; A display module 38 for displaying the recommended advertisement content; Advertisement providing system (1) by face recognition including a management module (40) for managing user information. The method of claim 1,
The control unit 9 can perform face recognition by changing the normal mode to the quick mode, and the face division module 20 divides the face into a plurality of even areas, and the face feature detection unit 10 First, facial features are detected sequentially from four areas, and if not matched, features of areas that are not symmetrically arranged are secondarily detected, and if not matched, features of the entire face are thirdly detected and matched. Advertisement provision system by face recognition (1). The method of claim 7,
The control unit 9 is a system (1) for providing advertisements based on face recognition capable of shortening the recognition time by first performing a feature on a peculiar shape of the face before detecting the feature points for the first to fourth regions. delete delete

Images