KR101170676B1 - Face searching system and method based on face recognition - Google Patents

Abstract

The face search system of the present invention includes a face information storage unit storing face information collected from an image of a detection target, a query input unit receiving a query for a search target, and the search among the face information based on the received query. A face search unit for searching for face information corresponding to the target person, wherein the query includes at least one of a keyword, a still image, and a video.

Description

Face recognition system based on face recognition and its method {FACE SEARCHING SYSTEM AND METHOD BASED ON FACE RECOGNITION}

The present invention relates to a face recognition system based on face recognition and a method thereof.

Recently, the demand for video surveillance using CCTV in airports, harbors, research institutes or special facilities has increased rapidly. As such, one of the most important elements of a video surveillance system for security and surveillance in a specific area is a video search component that enables a manager to quickly and accurately find a scene of interest. In particular, since it is limited to directly check the scene where a person appeared from a large collection of databases, it was necessary to process the scene where a person appeared in order to support a quick and accurate search for a suspect or a specific person. In other words, there is a demand for a technology for selectively processing and databasening scenes of human appearance in surveillance images.

In addition, a query method using a still image may be used to search for a specific person in the stored surveillance image. However, in the related art, the similarity between the detected face and the registered faces is compared to determine that the face of the same person as the registered face is larger than the threshold, and the comparison is repeated with other registered faces when smaller than the threshold. Was done. However, this method guarantees a high recognition rate, but has a problem that it is not suitable for a retrieval system that needs to perform recognition in real time on a large database because of its low processing speed. Accordingly, there is a need for a new type of system having a high retrieval rate while enabling face recognition based search in a large database.

An object of the present invention is to provide a system and method for retrieving face information using keywords, still images or moving images.

As a technical means for achieving the above-described technical problem, the face retrieval system according to the first aspect of the present invention, the face information storage unit that stores the face information collected from the image photographing the detection target, the query for the search target And a face search unit that searches for face information corresponding to the search target among the face information based on the query input unit and the received query, wherein the query includes at least one of a keyword, a still image, and a video.

The face searching method according to the second aspect of the present invention may further include collecting face information from an image of an object to be detected, receiving a still image of a person to be searched, and collecting the face image based on the received still image. And searching for face information corresponding to the search target among the extracted face information, wherein the face information includes face image data generated by dividing an area where a face is detected among captured images captured from the captured image, And feature vector data extracted from face image data, divided image data generated by temporally dividing a face-detected image among the captured images, and metadata generated by using face detection information of the captured image.

In addition, the method for storing face information according to the third aspect of the present invention comprises: generating a captured image by capturing an image of a detection target; generating face image data by dividing an area where a face is detected among the captured images; Extracting feature vector data from the face image, time-dividing a face-detected image of the captured image to generate split image data, and using face detection information on the captured image. Generating metadata, wherein the metadata includes: detection date data indicating a date when a face was detected from the captured image, detection time data indicating a time when a face was detected from the captured image, and the detection target; At least one of image capturing unit information data indicating information about a device It includes.

According to any one of the above-described problem solving means of the present invention, the user can perform a face search using a still image or a video as a query as well as a keyword, there is an advantage that it is easy to process a large face information database.

In addition, according to any one of the problem solving means of the present invention described above, when using the SR (sparse representation) algorithm, various lighting changes, when the other image overlaps or disguises the original image, when the original image is altered, low resolution, etc. Even in a variety of environments that impair recognition performance, there is an advantage of ensuring a very high face recognition rate without using the front face image reduction method.

In addition, according to any one of the above-described problem solving means of the present invention, even if new face information is added, learning about the new face information in the parasitic matrix, without having to re-learn all the face information during face search Since only the result needs to be added as column data, it is excellent in scalability and advantageous for processing a large amount of images.

In addition, according to any one of the above-described problem solving means of the present invention, when performing a face search using the image as a query, it is possible to perform a search for a specific person in real time in combination with a monitoring device such as CCTV.

In addition, according to any one of the above-described problem solving means of the present invention, when a user searches for an image or video for a specific search target, it is possible to search not only the image or video for the search target, but also metadata thereof. There is an advantage.

1 is a block diagram illustrating a face searching system according to an exemplary embodiment of the present invention.
2 is a block diagram illustrating a face information generator according to an exemplary embodiment of the present invention.
3 is a structural diagram showing face information data according to an embodiment of the present invention.
4 is a flowchart illustrating a face search algorithm according to an embodiment of the present invention.
5 is a diagram illustrating an SR algorithm according to an embodiment of the present invention.
6 is a view showing a display unit according to an embodiment of the present invention.
7 illustrates face search cumulative accuracy according to an embodiment of the present invention.
8 is a flowchart illustrating a face searching method according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . In addition, when a part is said to "include" a certain component, which means that it may further include other components, except to exclude other components unless otherwise stated.

1 is a block diagram illustrating a face searching system according to an exemplary embodiment of the present invention. 2 is a block diagram illustrating a face information generation unit according to an embodiment of the present invention, and FIG. 3 is a structural diagram showing face information data according to an embodiment of the present invention.

The face retrieval system 10 according to an embodiment of the present invention receives a query for a search target in the form of a keyword, a video, or a still image, and performs face retrieval using a sparse representation algorithm based on the received query. Perform.

In particular, the face retrieval system 10 according to an embodiment of the present invention performs a face search for a search target from a database in which the face information 260 is stored, and the face information 260 includes an image, an image, and metadata thereof. It includes.

As shown in FIG. 1, the face searching system 10 includes an image capturing unit 100, a face information generating unit 200, a face information generating unit 200, a face information storing unit 300, and a query input unit 400. ), A face searching unit 500 and a display unit 600.

The image capturing unit 100 captures an image of a detection target. The image capturing unit 100 may be, for example, an image generating apparatus such as CCTV, and generates an image by photographing a detection target. The detection target may be an unspecified target, such as an arbitrary person or an object, and the generated image may be transmitted to the face information generator 200.

The face information generator 200 generates the face information 260 using the image received from the image capturing unit 100. The face information 260 may include metadata related to an image, an image, a feature vector, and a face detection environment where a face is detected. The face information generator 200 and the face information 260 will be described in detail with reference to FIGS. 2 and 3.

As illustrated in FIG. 2, the face information generator 200 may include an image capture unit 210, a face detector 220, a face feature extractor 230, an image divider 240, and a detection information generator ( 250). In addition, the face information 260 generated by the face information generator 200 may include face image data 261, feature vector data 262, split image data 263, and detection date data, as illustrated in FIG. 3. 266, detection time data 267, and image capturing unit information data 268.

The image capture unit 210 captures an image received from the image capturing unit 100 and converts the image into an image. In addition, the image capture unit 210 may further include a process of preprocessing the captured image. For example, the image capture unit 210 may perform gray color conversion, interpolation, histogram smoothing, smoothing, resizing, and noise reduction on the captured image. cancelling).

The image captured by the image capture unit 210 may be configured in plurality. Also, since the image is a captured image of a reproduced image, a plurality of images containing various angles and expressions of the same object may be captured.

The face detector 220 detects a face image by using the captured image. Since the image provided by the image capturing unit 210 is captured by capturing an unspecified detection target, the image may not include the detection target. It may include.

Meanwhile, since the face search system 10 according to the present invention performs a search using a face of a subject as a biometric feature, the face detector 220 detects a face image of the captured image to generate a database to be used for face search. Detect. The face detector 220 may use a haar-like feature method. However, the present invention is not limited thereto, and face detection may be performed using any method. The face image detected by the face detector 220 may configure face image data 261 of the face information 260 as shown in FIG. 3.

The facial feature extractor 230 extracts a feature vector from the face image detected by the face detector 220. The facial feature extractor 230 may use, for example, a non-negative matrix factorization (NMF) algorithm. NMF algorithms are used to extract independent features from a data set. The basis vectors of feature spaces derived by the NMF show partial image shapes similar to eyes, noses, mouths, and so on. Can be represented. The facial feature extractor 230 according to the present embodiment is illustrated as using an NMF algorithm, but is not limited thereto. The facial feature extractor 230 may be used for facial feature extraction such as a SVD (singular value decomposition) method or a PCA method (principal component analysis). Any method can be used. The feature vector extracted by the face feature extractor 230 may configure feature vector data 262 of the face information 260 as shown in FIG. 3.

The image dividing unit 240 divides the image portion in which the face is detected from the image received by the image capturing unit 100. When the face is detected in the image received by the image capturing unit 100, the image divider 240 may divide the image in time. However, the image splitter 240 may further perform spatial segmentation on the face part according to an exemplary embodiment. As shown in FIG. 3, the image divided by the image splitter 240 may configure the divided image data 263 among the face information 260.

The detection information generator 250 generates metadata 265 about the temporal and spatial environment where the face is detected. The temporal environment may include detection date information, detection time information, and the like, and the spatial environment may include image capturing information. If a plurality of image capturing units 100 are fixedly installed, spatial information of face detection may be obtained through image capturing unit information, for example, reference number information of the image capturing unit 100.

The metadata 265 generated by the detection information generator 250 may function as data for the face image data 261 or the divided image data 263. If the user searches for an image or an image of a specific search target, the face search system 10 may not only face image data 261 or the divided image data 263 of the search target but also metadata 265 thereof. You can also search together. In addition, when the user requests a search based on the information on the metadata 265, the face search system 10 may perform the search based on this.

As shown in FIG. 3, the metadata generated by the detection information generator 250 may include the detection date data 266, the detection time data 267, and the image capturing information 268 of the face information 260. It may include at least one. However, the present invention is not limited thereto, and the metadata 265 may include any information related to an environment in which a face is detected.

Referring back to FIG. 1, the face information storage unit 300 stores the face information 260 generated by the face information generator 200. Since the face information 260 is generated when a face is detected in the image photographed by the image capturing unit 100 or the image captured by the image capturing unit 100, the face information storage unit 300 receives a plurality of face information 260 from a plurality of detection targets. ) Can be stored. As the face information 260 accumulates, the face information storage unit 300 may build a large face information database. The constructed face information database may perform a search by a query input by a user, which will be described in detail below.

The query input unit 400 receives a query for a search target from a user. A query is a search condition for searching for an image or an image in which a face is detected. Since a user directly checks the large-scale face information database stored in the face information storage unit 300, there is a limit, and thus, a quick and accurate search can be performed using a query.

The query may include a query-by-keyword and a query-by-example. The query by keyword is a search performed by using a keyword, and the keyword may be related to detection date data, detection time data, and image capturing unit information data in the face information 260. For example, the keywords may be 'November 1, 2010', '10 pm to 11 pm ', and' 3 CCTV '.

In addition, the query according to the example is to perform a search using a still image or a video, and a user may perform a search using a still image or a video of a target to be searched.

The face search unit 500 searches for the face information 260 stored in the face information storage unit 300 based on the query received by the query input unit 400. Since the query may include a query by keyword and a query by example, the face search unit 500 may perform a search by using a different method according to the query. This will be described in detail with reference to FIG. 4.

4 is a flowchart illustrating a face search algorithm according to an embodiment of the present invention.

The face search unit 500 first determines whether the query is an example (S100). When the user inputs an image or a video, it is determined as a query by an example, and when a keyword is input, it is determined as a query by a keyword.

If it is determined that the query is a query by example, it is determined whether a still image is input (S110). The query according to the example may include input of a still image and input of a video, and thus, it is determined whether the input target is a still image or a video.

If a video other than a still image is input, face detection is performed after capturing the video (S120 and S130). In addition, when a still image is input, face detection is performed on the still image (S130). Since the face retrieval system 10 according to the present invention performs a search using a face of a subject as a biometric feature, the face retrieval unit 500 detects a face image among still images input by a user. Face detection can be performed using a Haar-like feature method.

When face detection is completed, a feature vector is extracted (S140). Feature vector extraction may be performed using a non-negative matrix factorization (NMF) algorithm. In this case, the feature vector extraction may use the same method as the extraction method used by the face information generator 200.

Next, a search using the SR algorithm is performed (S150). This will be described with reference to FIG. 5.

5 is a diagram illustrating an SR algorithm according to an embodiment of the present invention.

의 열로 정렬 될 수 있다. 사용자로부터 입력된 이미지로부터 추출된 특징 벡터가 y라면 y는 A에 대한 하기 수학식 1로 표현될 수 있다.Referring to FIG. 5, first, a matrix A is generated by using face information 260 stored in the face information storage unit 300. In detail, a high dimensional data matrix A having n dimensions is generated using the feature vector data among the face information 260 stored in the face information storage unit 300. Each column data of the matrix A may be a feature vector for each face information. For example, when generating the matrix A from the i class, n face information 260, the n th training image feature vector belonging to the i th class is matrix

Can be sorted in columns. If the feature vector extracted from the image input from the user is y, y may be represented by Equation 1 for A.

[Equation 1]

In Equation 1, x is a coefficient vector. If the feature vector belongs to class i, the coefficient vector x has a value equal to 0 or 0 except for the training data value associated with i, and is represented by Equation 2 below.

[Equation 2]

Equation 2 can be solved by solving Equation 1 to obtain an x value, and may be changed to a problem of obtaining an optimization of x.

&Quot; (3) "

-노름(norm) 최소화 문제(수학식 3)는 다음 컨벡스 관련 최적화 문제를 이용하여, 근사해인

-노름 최소화 문제로 풀 수 있다.Finding a solution using Equation 3 is an NP-hard problem and difficult to find. But if the solution of x is sparse enough,

The norm minimization problem (Equation 3) is approximated using the following convex-related optimization problem.

Can be solved by the problem of minimizing gambling.

&Quot; (4) "

The face search unit 500 may extract a search target person corresponding to a class having a relatively high value.

When using the SR (Sparse Representation) algorithm, the front face image is reduced in various environments such as various lighting changes, overlapping or disguising other images on the original image, altering the original image, and low resolution. It has the advantage of ensuring a very high face recognition rate without using it.

Next, the search result is transmitted to the display unit 600 (S160). The search result may include all face information about the search subject. For example, the search result may include face image data, feature vector data, split image data, detection date data, detection time data, image capturing unit information data, and the like of the search target. The transmitted search result is displayed by the display unit 600.

On the other hand, as shown in Figure 4, when the query by the keyword is received from the user instead of the query by the example, the search using the metadata is performed (S170).

The keyword may be related to detection date data, detection time data, and image capturing unit information data in the face information. For example, when the keywords are 'November 1, 2010', '10 pm to 11 pm ', and' 3 CCTV ', the face search unit 500 searches for the face information storage unit 300. The face information corresponding to the keyword condition may be searched for. The retrieved face information is transmitted to the display unit 600 in the same manner as the query by the example.

The user can perform a face search using not only a keyword but also a still image or a video as a query, which is advantageous in processing a large face information database.

In addition, when performing a face search using a video as a query, it is possible to perform a search for a specific person in real time by combining with a surveillance device such as CCTV. For example, after storing face information about a suspect in a database, the detected person can be automatically searched for the same person as the suspect in the database.

In addition, even if new face information is added, the face search does not need to relearn all the face information, and only the learning results of the new face information need to be added as column data to the parametric matrix. There is an advantage that it is advantageous for processing a large amount of images.

Referring back to FIG. 1, the display unit 600 displays face information retrieved by the face searching unit 500. This will be described in detail with reference to FIG. 6.

6 is a view showing a display unit according to an embodiment of the present invention.

6 illustrates the display unit 600 when a video is input during the query. The display unit 600 includes an input image display unit 610, an input image display unit 620, an input face image display unit 630, a search image display unit 640, a search image display unit 650, and a search metadata display unit 660. Include.

The input image display unit 610, the input image display unit 620, and the input face image display unit 630 are input images received by the query input unit 400 and images captured by the face search unit 500 and face detection is performed. Is displayed. When a user inputs an image as shown in the input image display unit 610, it is captured as a still image shown in the input image display unit 620, and the detected face image is displayed on the input face image display unit 630. This has been described in detail above.

The search image display unit 640, the search image display unit 650, and the search metadata display unit 660 show face information on which the search is completed by the face search unit 500. The search image display unit 640 displays the face image data 261 of the face information 260 searched by the face search unit 500, and the search image display unit 650 displays the divided image data 263 of the face information 260. ) Is displayed. In addition, the metadata display unit 660 displays the metadata 265 of the found face information 260. For example, the metadata display unit 660 may display the image capturing unit information data 268, the detection date data 266, and the detection time data 267 among the face information 260 (661, 662, 663). .

7 illustrates face search cumulative accuracy according to an embodiment of the present invention.

7 illustrates a search performance of the face search system according to an embodiment of the present invention. In the experiment, the face information database was collected using a total of 28 face images and 15 face images per person. After preparing 15 images of each face and other images used in the face information database, we used this as a query to test the search performance. As a result of the search using the example query, the accuracy of the first recommendation image recorded a high performance of 99.5% as shown in FIG. 7.

8 is a flowchart illustrating a face searching method according to an embodiment of the present invention.

First, face information is collected (S200).

Face information is collected from the image of the detection target. The detection target may be an unspecified object such as an arbitrary person or an object, and may be photographed using an image generating device such as CCTV. The face information includes face image data, feature vector data, segmented image data, and metadata.

The captured image is captured as a captured image, and then a face region is detected. When turned on, a face image in which the face area is divided based on the image is generated. In addition, a feature vector is extracted from the face image. The feature vector may be extracted using, for example, a non-negative matrix factorization (NMF) algorithm. Meanwhile, the divided image is generated from the image of the face of the captured image. The divided image may be generated by dividing the captured image in time.

In addition, metadata is generated using face detection information of an image. The metadata is data regarding temporal and spatial environments for face detection, and the temporal environment may include detection date information, detection time information, and the like, and the spatial environment may include image capturing information. The metadata may include at least one of detection date data, detection time data, and image capturing unit information data.

Next, a query for a search target is received (S210).

A query is a search condition for searching for an image or an image in which a face is detected, and may include a query by keyword and a query by example. The query by keyword is to search by keyword, and the example query is to search by video or still image.

Next, the face information corresponding to the search target is searched among the collected face information based on the query (S220).

The face information may be searched based on the received query. The query can include a query by keyword and a query by example, so that the search can be performed using different methods depending on the query. This has been described in detail with reference to FIG. 4.

In particular, when the still image is received as a query, the feature vector is extracted from the still image, and the feature vector data corresponding to the extracted feature vector is searched among the collected feature vector data. In addition, the search for the feature vector data may be performed using a sparse representation (SR) algorithm. This has been described in detail with reference to FIG. 5.

Next, the detected face information is displayed (S230).

All data of the collected face information may be displayed, and this is described in detail with reference to FIG. 6.

For reference, components shown in FIGS. 1 and 2 according to an embodiment of the present invention mean software components or hardware components such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), and have a predetermined role. Perform them.

However, 'components' are not meant to be limited to software or hardware, and each component may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors.

Thus, as an example, a component may include components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, procedures, and subs. Routines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

Components and the functionality provided within those components may be combined into a smaller number of components or further separated into additional components.

One embodiment of the present invention may also be embodied in the form of a recording medium including instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes any information delivery media, including computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism.

The foregoing description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

10: face detection system 100: video recording unit
200: face information generation unit 210: image capture unit
220: face detection unit 230: facial feature extraction unit
240: Image segmentation unit 250: Detection information generation unit
260: face information 300: face information storage unit
400: query input unit 500: face search unit
600: display unit

Claims (14)

In the face search system,
A face information storage unit storing face information collected from an image of a detection target;
A query input that receives a query for the search target, and
A face searching unit searching for face information corresponding to the search target among the face information based on the received query,
The query includes at least one of a keyword, a still image, and a video.
And if the received query is a moving picture or a still image, the face search unit performs a search according to a sparse representation algorithm.
The method of claim 1,
The face information is,
Face image data generated by dividing an area where a face is detected among captured images captured from the captured image;
Feature vector data extracted from the face image data;
Divided image data generated by temporally dividing an image of a face detected among the photographed images;
Metadata generated using face detection information of the captured image
Face search system comprising a.
The method of claim 2,
The metadata,
Detection date data indicating a date when a face was detected from the captured image;
Detection time data indicating a time when a face was detected from the captured image;
And at least one of image capturing unit information data representing information about a device photographing the detection target.
The method of claim 2,
If the received query is a still image,
And a face search unit extracting a feature vector from the still image and searching for feature vector data corresponding to the extracted feature vector among the feature vector data stored in the face information storage unit.
The method of claim 2,
If the received query is a video,
The face search unit captures the video, extracts a feature vector from the captured image of the video, and searches a feature vector data corresponding to the extracted feature vector among the feature vector data stored in the face information storage unit. system.
The method of claim 2,
If the received query is a keyword,
And the face searching unit searches for metadata corresponding to the keyword among metadata stored in the face information storage unit.
delete The method of claim 1,
And a display unit which displays the retrieved face information.
In the face search method,
Collecting face information from an image of the detection target;
Receiving a still image for the search target; and
Searching for face information corresponding to the search target among the collected face information based on the received still image;
The face information is,
Face image data generated by dividing an area where a face is detected among captured images captured from the captured image;
Feature vector data extracted from the face image data;
Divided image data generated by temporally dividing an image of a face detected among the photographed images;
Include metadata generated by using the face detection information for the captured image,
Searching for the face information,
Extracting a feature vector from the received still image; and
Retrieving feature vector data corresponding to the extracted feature vector from the feature vector data;
And retrieving feature vector data corresponding to the extracted feature vector from the feature vector data is performing a search according to a sparse representation algorithm.
The method of claim 9,
Collecting the face information,
Generating a captured image by capturing an image of a detection target;
Generating face image data by dividing an area where a face is detected among the captured images;
Extracting feature vector data from the face image;
Generating segmented image data by temporally dividing an image of a face detected from the photographed images;
Generating metadata using face detection information of the captured image;
Face detection method comprising a.
The method of claim 9,
The metadata,
Detection date data indicating a date when a face was detected from the captured image;
Detection time data indicating a time when a face was detected from the captured image;
And at least one of image capturing unit information data representing information about a device photographing the detection target.
delete delete delete

Images