KR101827538B1 - Device and method for gender reconition - Google Patents

Abstract

The present invention relates to a gender recognizing technology and, more specifically, to a device and a method for recognizing a gender by using a convolutional neural network (CNN). According to an embodiment of the present invention, a gender can be accurately recognized by using a model which learns genders of various people based on deep learning.

Description

[0001] DEVICE AND METHOD FOR GENDER RECONITION [0002]

The present invention relates to gender recognition technology, and more particularly, to a device and method for gender recognition based on deep learning using a CNN (Convolutional Neural Network).

The existing gender recognition system is mainly used in the near environment. In addition, information on body and clothes, accessories, hairstyle, etc. are recognized by gender. However, due to the limitation of the near - field environment, there are many problems in gender recognition in the outdoor environment other than the indoor environment.

The background art of the present invention is disclosed in Korean Patent Laid-Open Publication No. 10-2016-0122452 (October 26, 2014).

The present invention is intended to provide a deep learning-based gender recognition apparatus and method for performing gender recognition with improved accuracy not only in an indoor environment but also in an outdoor environment.

The present invention is intended to provide a deep learning-based gender recognition apparatus and method that performs highly reliable gender recognition through learning using body region information of a person who is a target, not only in a near place but also in a remote place.

According to an aspect of the present invention, a gender recognition apparatus is provided.

The gender recognition apparatus according to an embodiment of the present invention includes a body region information generation unit that detects a person who is a target in an input image and extracts an image of a body region of a person to generate body region information, An image feature extraction unit for extracting image feature information using a running-based Convolutional Neural Network (CNN) model, and a gender recognition unit for recognizing a sex of a person using the extracted image feature.

According to another aspect of the present invention, a gender recognition method and a computer program for executing the method are provided.

A gender recognition method and a computer program for executing the same according to an embodiment of the present invention include a step of detecting a person from an inputted visible ray image and generating an image of a visible ray body region of the detected person, Applying a deep-run-based neural network model to the visible light body region image to extract a visible light image feature, extracting a visible light body region image, Applying a run-based circuit neural network model to extract thermal image features and combining the extracted visible light image features and thermal image features and using the combined image features to recognize the sex of the detected person .

According to the embodiment of the present invention, the gender of the photographed person can be accurately recognized by using the line neural network model that learns gender classification using various body region information based on Deep Learning.

In addition, according to the embodiment of the present invention, it is possible to perform highly reliable gender recognition on a person who is photographed using body region information of a person who is a target, not only in an indoor environment but also in a remote area as well as in an outdoor environment.

1 is a view for explaining a gender recognition system according to an embodiment of the present invention;
2 to 6 are views for explaining a gender recognition apparatus according to an embodiment of the present invention.
7 to 9 are views for explaining a gender recognition method according to an embodiment of the present invention.
10 to 14 are graphs showing gender classification results of a gender recognition method based on a line neural network based learning according to an embodiment of the present invention.

While the present invention has been described in connection with certain exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and similarities. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In addition, the singular phrases used in the present specification and claims should be interpreted generally to mean "one or more " unless otherwise stated.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or corresponding components throughout. .

1 is a view for explaining a gender recognition system according to an embodiment of the present invention.

Referring to FIG. 1, the gender recognition system includes a photographing device 10 and a gender recognizing device 20. As shown in FIG.

The image photographing apparatus 10 photographs a person who is a target and inputs image information. The image capturing apparatus 10 may be, for example, a closed-circuit television (CCTV) located in an indoor environment or an outdoor environment, but is not limited thereto. The imaging device may include at least one of a visible light camera and a thermal imaging camera. Here, the thermal imaging camera may be a camera that senses heat and imaged using infrared rays. That is, the image capturing apparatus 10 can input at least one of visible light image information and thermal image information.

The gender recognition apparatus 20 recognizes the gender of the person who is the target in the input image. The gender recognition apparatus 20 detects a person who is a target in the input image, and extracts a body region image of a person to generate body region information. The gender recognition apparatus 20 extracts the image feature information using the deep-learning-based circuit neural network model with the generated body region information. Here, the gender recognition apparatus 20 repeatedly performs learning and verification using a database including a human image for deep learning. The gender recognition apparatus 20 performs gender recognition for classifying gender using the extracted image feature information. The gender recognition device 20 will be described in more detail below with reference to FIGS. 2 to 6. FIG.

FIGS. 2 to 6 are views for explaining a gender recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 2, the gender recognizing apparatus 20 includes a body region information generating unit 100, an image feature extracting unit 200, a gender recognizing unit 300, and a gender aware learning unit 400.

The body region information generation unit 100 detects a person who is a target in the input image, and extracts a body region image of a person to generate body region information.

Referring to FIG. 3, the body region information generation unit 100 includes a body detection unit 120 and a body region image generation unit 140.

The object detecting unit 120 detects a person (object) from the input image information. The object detection unit 120 can detect a person using a known object detection algorithm. When the visible light ray image information and the thermal image information are simultaneously used as the input image, the object detecting unit 120 can detect the target chain person with less influence from the surrounding environment and the weather and with high accuracy.

The body region image generating unit 140 generates a body region image from the detected human image. Body region image generating section 140 may generate at least one of a visible light body region image and a thermal image body region image. The body region image generating unit 140 may generate a body region image with a predetermined size, for example, 183 × 119 pixels through a size normalization step. Here, the size normalization process is to compensate for the difference between the case where the image of the person to be detected is near and the case where the image is far away. In addition, the body-area image generator 140 may normalize the illumination of the body-area image by performing a zero-center method.

The image feature extraction unit 200 extracts image feature information using the deep-learning-based Convolutional Neural Network (CNN) model. The image feature extraction unit 200 may extract image feature information for gender recognition by inputting at least one image of the visible light body region image and the thermal image body region image. The image feature extraction unit 200 may apply a data augmentation method and a dropout method in order to alleviate an over-fitting problem. Here, the data extension method is to add a few different extension data slightly different from the original images. In this case, the preset pixels are removed from the upper and lower sides of the original image, and a virtual extended data is generated and added using the pixels. The dropout method can be performed by separating the connections of the previous layer and the next layer in the fully connected layer of the circuit neural network model and setting the output of hidden neurons with a certain probability to zero. Here, the specific probability is dependent on the input data and can be determined through experimentation.

4 and 5, the image feature extraction unit 200 includes an image input layer 220, a feature extraction layer 240, and a classification layer 260.

The image input layer 220 inputs a body region image of a predetermined scale. The image input layer 220 may, for example, input a visible light body region image of a size of 183 x 119 pixels or an image body region image of the same size.

The feature extraction layer 240 extracts image features by applying a line neural network model to the input body region image. Feature extraction layer 284 may include (1) five convolution layers, (2) each convolution layer may comprise a rectified linear unit; ('ReLU') layer, and (3) a pooling layer.

Here, the first convolution layer is an image of 183 x 119 pixels (visible ray body region image or thermal image of the body region image) input, and in the horizontal and vertical directions, 96 x 11 pixels of size 11 x 11 pixels (Conv: 96 @ 11 x 11; Stride 2 x 2). The first convolution layer outputs 96 feature maps of size 87 x 55 pixels. The CNN structure is processed and downsampled by a first Max-Pulling layer that applies the 96 feature maps to a maximum of 3 x 3 spatial neighboring pixels in a two-pixel stride to make the image transformation more robust. As a result, the output of the first convolution layer is 96 feature maps of size 43 x 27 pixels in size.

The second convolution layer includes 128 filters of size 5x5x96 with 2 pixel padding, stride 1 pixel to fine tune the first convolution layer (Conv: 128 @ 5x5x96 ; Stride: 1x1; Padding: 2x2). The second convolution layer outputs 128 feature maps of size 21 x 13 pixels in size. The first convolution layer and the second convolution layer may be used to extract low-level image features, such as, for example, texture features of edge or color blobs.

The third convolution layer outputs 256 feature maps of size 21 × 13 pixels by applying 256 1 × 1 × 1 pixels with padding of 1 × 1 pixels and a size of 3 × 3 × 128 (Conv: 256 @ 3 x 3 x 128; Stride 1 x 1; Padding: 1 x 1).

The fourth convolution layer outputs 256 feature maps of size 21 × 13 pixels (Conv: 256) by applying 256 pixels of size 3 × 3 × 256 with 1 × 1 pixel padding and stride 1 × 1 pixels @ 3 x 3 x 256; Stride 1 x 1; Padding: 1 x 1).

The fifth convolution layer convolutionizes 1 × 1 pixel padding, stride 1 × 1 pixel, 128 filters of size 3 × 3 × 256, and outputs 128 feature maps of size 10 × 6 pixels (Conv: 128 @ 3 x 3 x 256; Stride 1 x 1; Padding: 1 x 1).

Then, the feature extraction layer 240 extracts 128 feature maps of size 10 x 6 pixels output using the first through fifth convolution layers to three fully connected layers (1, 2, 3, 4, Fully Connected Layers).

The classification layer 260 may include only two neurons in the last fully connected layer and may extract image feature vectors in a 1024-dimensional space for each visible light body image or a thermal image of the body image. The classification layer 260 is classified into two genders according to features extracted by applying a line neural network model.

Referring again to FIG. 2, the gender recognition unit 300 recognizes the sex of a person using the extracted image characteristics.

The gender recognition unit 300 may combine the image features extracted from the visible light body region image and the thermal image body region image and recognize the gender using a support vector machine (SVM).

Referring to FIG. 6, the gender recognition unit 300 includes a feature level fusion unit 320 and a score level fusion unit 340.

The feature level fusion unit 320 selects an optimal feature among the visible light image feature and the thermal image feature extracted by the Principal Component Analysis (PCA) and performs the gender recognition using the support vector machine method. Here, the principal component analysis is to remove the noise and reduce the feature dimension before recognizing the sex. This is because the extracted image feature may include redundant information because the body region image of the object may have a very wide variation due to differences in background, clothing, ornament, body pose, hairstyle, and other factors to be. The support vector machine method learns the machine by supervised learning method to obtain various support vectors, xi with classification parameters with learning data. The classification level of the verification image by the support vector machine method of the gender recognition apparatus 20 can be evaluated by equation (1).

(1)

(One)

Here, yi is a classification level corresponding to the support vector xi, and K (x) is a kernel function used for mapping the input data to the high dimensional feature space at low dimension.

In the present invention, two different support vector machine kernels may be used, such as the linear based function of equation (2) and the radial basis function (RBF) of equation (3).

(2)Linear kernel:

(2)

(3)RBF kernel:

(3)

Further, in the present invention, the LibSVM library which is an open source library can be used for applying the support vector machine method.

The feature level fusion unit 320 can associate the visible light image feature and the thermal image feature extracted using the circuit neural network model using the following equation (4).

(4)

(4)

Where fc concatenates image features of two types of images with fusing characteristics, and fv and ft are image features extracted from visible light and infrared images.

The score level fusion unit 340 performs gender recognition for each of the visible light image feature and the thermal image feature using the support vector machine method. The score level fusion unit 340 selects the visible light optimum feature through principal component analysis on the visible light image feature and selects the thermal image optimal feature through principal component analysis on the thermal image feature. The score level fusion unit 340 may be configured to generate a visual ray support vector machine score (a sex probability (a male probability or a female probability) for visible light ray image features calculated by applying a support vector machine method for each of the visible light ray feature ) (Score of visible light image)) and the thermal image support machine score (the score for the sex probability (thermal image score) for the thermal image feature). The score level fusion unit 340 generates a score fusion feature (support vector machine score combination) that is a new feature by combining the calculated visible light support vector machine score and the thermal image support vector machine score, and performs gender recognition using the score fusion feature . The score level fusion unit 340 may use another classification layer based on the support vector machine for generating the score fusion characteristic.

Referring again to FIG. 2, the gender aware learning unit 400 learns and verifies a person's gender recognition using a database of human images. Here, the database may include, for example, images of 412 persons, including 254 males and 158 spiritualities. Each person may include 10 visible light images and 10 thermal images. As a result, the database may contain 8240 images, including 4120 visible light images and 4120 thermal images. All included images can be captured using dual visible light and thermal cameras located approximately 6 meters high when compared to the observation scene to simulate the actual operation of the surveillance system in the laboratory. A database can contain various configurations of texture and body pose.

The gender awareness learning unit 400 may perform a 5x cross validation method to measure the accuracy of the recognition system. The gender awareness learning unit 400 can perform the learning and verification of the sub database five times by dividing the entire database. Each segmentation method can use approximately 80% of the images of male and female for sub-database learning, and the remaining 20% of images can be assigned to the verification. As a result, the gender aware learning unit 400 can use five learning subdatabases and a verification subdatabase. Here, each learning sub-database includes 204 male images and 127 female images, and each of the verification sub-databases may include 50 male images and 31 female images.

Used self-configuration database Database Males    Females Total Number of Persons 254 158 412 (persons) Number of Images 5080 3160 8240 (images)

The gender awareness learning unit 400 may generate an extended database to enlarge the database size in the original database in order to reduce the over-fitting problem described above. The gender awareness learning unit 400 may expand the number of expanded images of the male image to be smaller than the number of expanded images of the female image to make the number of male images and the number of images of the female similar to each other when the number of male images is larger than the number of female images . The gender awareness learning unit 400 may further generate 18 images from each male image by removing 2 or 4 pixels from the left and right and upper and lower sides of the original image to generate 30 images from each female image. The gender awareness learning unit 400 can construct the extended database shown in Table 2 as a result.

Extended database configuration Database Males Females Total Augmented Database Learning Database Number of Persons 204 (persons) 127 (persons) 331 (persons) Number of Images 73,440 (204 x 20 x 18 images) 76,200 (127 x 20 x 30 images) 149,640 images Testing Database Number of Persons 50 (persons) 31 (persons) 81 (persons) Number of Images 18,000 (50 × 20 × 18 images) 18,600 (31 x 20 x 30 images) 36,600 images

7 to 9 are diagrams for explaining a gender recognition method according to an embodiment of the present invention.

Referring to FIG. 7, a gender recognition method according to an embodiment of the present invention will be described. In step S710, the gender recognition apparatus 20 detects a person from the input visible light ray image, Create an image.

In step S720, the gender recognition device 20 detects a person from the input thermal image and generates an image of the detected person's thermal image.

In step S730, the gender recognition apparatus 20 applies a deep learning-based circuit neural network model to the generated visible light body region image to extract a visible light image feature.

In step S740, the gender recognition apparatus 20 applies the deep learning-based circuit neural network model to the generated thermal image area image to extract thermal image features.

In step S750, the gender recognition apparatus 20 combines the extracted visible light image characteristic and the thermal image characteristic, and recognizes the sex of the detected person using the combined image characteristic.

In step S760, the gender recognition device 20 outputs the recognized gender result.

Referring to FIG. 8, in step S810, the gender recognition apparatus 20 detects a person from the input visible light ray image, Lt; / RTI > body-area image of the patient.

In step S820, the gender recognition device 20 detects a person from the input thermal image and generates an image of the detected person's thermal image.

In step S830, the gender recognition apparatus 20 applies a deep learning-based circuit neural network model to the generated visible light body region image to extract a visible light image feature.

In step S840, the gender recognition apparatus 20 applies a deep learning-based circuit neural network model to the generated thermal image area image to extract thermal image features.

In step S850, the gender recognition apparatus 20 selects an optimal feature among the visible light image feature and the thermal image feature extracted by the Principal Component Analysis (PCA), and performs gender recognition using the support vector machine method.

In step S860, the gender recognition device 20 outputs the recognized gender result.

Referring to FIG. 9, the gender recognition method using the score level convergence according to an embodiment of the present invention will be described. In step S910, the gender recognition apparatus 20 detects a person from the inputted visible ray image, Lt; / RTI > body-area image of the patient.

In step S920, the gender recognition apparatus 20 detects a person from the input thermal image and generates a thermal image of the body region of the detected person.

In step S930, the gender recognition apparatus 20 extracts visible light image features by applying a deep learning-based line neural network model to the generated visible light body region image, and obtains a visible light optimal feature by Principal Component Analysis (PCA) .

In step S940, the gender recognition apparatus 20 extracts the thermal image feature by applying a deep learning-based circuit neural network model to the generated thermal image of the body region of the thermal image, and calculates the thermographic image optimum by Principal Component Analysis (PCA) Select a feature.

In step S950, the gender recognizing device 20 performs gender recognition using the support vector machine method for the selected visible ray optimum feature to calculate the visible ray support vector machine score.

In step S960, the gender recognition apparatus 20 calculates the thermal image support vector machine score using the support vector machine method for the selected thermal image best feature.

In step S970, the gender recognition apparatus 20 generates a support vector machine score combination using the calculated visual beam support vector machine score and the thermal image support vector machine score, and performs gender recognition using the generated support vector machine score combination do.

In step S980, the gender recognition device 20 outputs the recognized gender result.

FIGS. 10 to 14 illustrate gender classification results of the gender recognition method based on the circuit neural network based learning according to the embodiment of the present invention.

The present invention performs learning and verification procedures to learn an image feature extraction model based on a line neural network for a visible light body region image and a thermal image body region image, respectively. The learning and validation database used were 74,820 images (36,720 male images (204 x 10 x 18) and 38,100 female images (127 x 10 x 30)) as learning data, including visible light and thermal images, 18,300 images (9000 male images (50 x 10 x 18) and 9300 female images (31 x 10 x 30)) were used as verification data. In addition, MATLAB execution was used to study the neural network model. In addition, the number of epochs in the circuit neural network model was set to 60, and the initial learning rate was set to 0.01 with a learning rate drop factor of 0.1 after each 20 epochs.

Referring to FIG. 10, the average EER (Equal Error Rate) based on the circuit neural network using only the visible ray image in the ROC (Receiver-Operating Curve) curve is about 17.216% as a result of the gender recognition according to the present invention, The average EER based on the circuit neural network was approximately 16.610%. Here, the Equal Error Rate (EER) means a rate at which the subject is recognized as another person or can not be recognized at the same error rate.

11, the average EER based on the circuit neural network based on the principle component analysis and the support vector machine method in the visible light image in the ROC (Receiver-Operating Curve) curve is about 17.064% The average EER based on the circuit neural network using principal component analysis and support vector machine method in the image was approximately 16.114%.

Referring to FIG. 12, the average EER based on the line-by-line neural network based on the receiver-operating curve (ROC) curves of the visible light image and the thermal image according to the present invention is about 11.439% The average EER based on the line neural network using the level fusion method was approximately 11.713%. As shown in Fig. 12, when the visible light image and the thermal image are combined, the recognition accuracy is higher than when the visible light image or the thermal image is used alone.

Comparison of recognition accuracy between gender recognition method and other methods according to the present invention Method Using Single Visible Light Images Using Single Thermal Images Feature Level Fusion Score Level Fusion HOG + SVM 17.817 20.463 16.632 16.227 EWHOG + SVM 15.113 19.198 14.767 14.135 wHOG + SVM 15.219 18.257 14.819 13.060 Invention 17.064 16.114 11.439 11.713

Referring to Table 3, it can be seen that the gender recognition method according to the present invention improves recognition accuracy significantly compared to other methods.

13 and 14 show a sample showing the result of sex recognition by the gender recognition method according to the present invention.

Referring to FIG. 13, the gender recognition method according to the present invention correctly recognizes the sex of people even in the case of the front view or the rear view image of people.

14 (a) through 14 (c) are examples in which the male image is mistakenly recognized as a woman, and FIGS. 14 (d) through 14 (f) Are examples of female images mistakenly recognized as males. However, the image of the body region shown in Fig. 14 is very vague and sometimes wearing a thick winter clothing in the rear view, so that it may not be easy to recognize sex even if it is based on the recognition of the general public. Moreover, since the general surveillance system shoots images at a distance of about 10 meters from the camera, the quality of the captured images is generally poor, which can have a negative impact on the recognition system.

The gender recognition method according to an embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions to be recorded on a computer-readable medium may be those specially designed and constructed for the present invention or may be available to those skilled in the computer software arts. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; - Hardware devices specifically configured to store and execute program instructions, such as magneto-optical media and ROM, RAM (Random Access Memory), flash memory, etc. Examples of program instructions include those made by a compiler As well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The embodiments of the present invention have been described above. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

10: Video shooting device
20: Gender recognition device

Claims (12)

In the gender recognition apparatus,
A body region information generating unit for detecting a person who is a target in the input image and extracting a body region image of a person to generate body region information;
An image feature extracting unit for extracting image feature information by using a deep-learning-based Convolutional Neural Network (CNN) model; And
And a gender recognition unit for recognizing a gender of a person using the extracted image feature,
The image feature extraction unit
An image input layer for inputting a body region image of a preset scale;
A feature extraction layer for extracting image features by applying a line neural network model to the input body image; And
And a classification layer for classifying the two sexes by using the extracted image feature,
The feature extraction layer
Five convolution layer sets and three fully connected layer sets,
The five convolution layer sets
A first convolution layer for inputting an image of 183 × 119 pixels and converting the 96 feature maps of size 43 × 27 pixels into convolutions by applying 96 filters of size 11 × 11,
A second convolution layer for transforming the 96 feature maps of 43 × 27 pixels into convolution using 128 filters of size 5 × 5 × 96 and outputting 128 feature maps of size 21 × 13 pixels;
A third convolution layer for convolutionizing the 128 feature maps of 21x13 pixels by applying 256 filters of size 3x3x128 to output 256 feature maps of size 21x13 pixels;
A fourth convolution layer for convolutionizing 256 feature maps of 21x13 pixels by applying 256 filters of size 3x3x256 to output 256 feature maps of size 21x13 pixels; And
And a fifth convolution layer for convolutionizing 256 feature maps of 21x13 pixels by applying 128 filters of size 3x3x256 to output 128 feature maps of size 10x6 pixels, Recognition device.
The method according to claim 1,
The body-area-information generating unit
A target object detection unit for detecting a target person in the input image information; And
And a body region image generating unit for generating a body region image from the detected person image.
delete delete delete The method according to claim 1,
Wherein the body region information is visible light body image information and thermal image body image information,
The gender-
The visible ray image feature extracted from the visible light body image information through the circuit neural network model and the thermal image feature extracted from the thermal image body image information through the circuit neural network model are selected using the principal component analysis and the support vector machine method To perform gender recognition.
The method according to claim 1,
Wherein the body region information is visible light body image information and thermal image body image information,
The gender-
In the visible light body image information, the visible light ray feature extracted from the line neural network model is selected through principal component analysis,
The thermal image optimum feature is selected through the principal component analysis on the thermal image feature extracted through the circuit neural network model in the thermal image body image information,
A support vector machine score combination is generated using the visual beam support vector machine score and the thermal image support machine score calculated by applying the support vector machine method for each,
And a gender recognition device for performing gender recognition using the generated support vector machine score combination.
The method according to claim 1,
And a gender awareness learning unit that learns and verifies gender recognition of a person using a database of human images.
delete delete delete delete

Images