KR20170121664A - Method and apparatus for multiple image information generation and the processing for optimal recognition performance in a deep learning - Google Patents

Abstract

Disclosed are a method and apparatus for generating and processing multiple image information for optimal performance in a deep learning structure. A facial color feature learning apparatus according to an embodiment of the present invention includes: a multiplexing unit which generates color images for a plurality of predetermined colors with respect to a facial image and performs processing using a predetermined processing technique on each of the generated color images to generate image information for each of the color images; and a selecting unit which selects one or more image information among the image information on each of the generated color images; and a fusion unit for fusing the feature of each of the one or more selected image information to generate fusion characteristics. Accordingly, the present invention can achieve strong recognition performance for an input facial image which is variously changed.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a method and apparatus for generating and processing multiple image information for optimal performance in a deep running structure,

The present invention relates to an apparatus and method for generating and processing multiple image information for optimal performance in a deep running structure.

In face recognition and facial expression recognition, various input face image changes such as lighting change of input face, pose change, resolution change, and the like are known to make recognition difficult.

Recently, the deep learning based recognition method has attracted great attention in the environment having variously changing input face images. In this case, in order to obtain very good performance, the latent characteristic of the input data must be sufficiently learned by the deep learning structure, and according to the existing studies, it is necessary to perform the deep learning with a large number of layers Structure is known to be required.

In other words, the deep learning structure should be designed according to the environment of the input image, otherwise the recognition performance deteriorates.

However, a deep learning structure with a very large number of layers is known to be very difficult to learn and requires a lot of learning time, and such a deep learning structure design is highly dependent on the developer's a priori knowledge. In addition, as the size of the deep running structure increases, it has a large number of floating point parameters, which requires a lot of resources in terms of capacity and calculation.

Therefore, there is a demand for a design method capable of achieving optimum performance in a fixed deep-running structure, and studies on weight reduction of the deep-running structure are also under way.

The present invention proposes a technique for multiplexing and processing input data to achieve robust recognition performance for input face images that vary in a fixed depth running structure.

Embodiments of the present invention provide a method and apparatus that can perform multiplexing and processing of input data to achieve robust recognition performance for input face images that vary in a fixed depth running structure.

Further, the embodiments of the present invention can be applied to a method of improving learning performance of a deep learning structure by learning a fixed deep learning structure for an input image having various changes, a deep learning structure having a small number of layers, Device.

Embodiments of the present invention also provide a method and apparatus that enable effective and efficient design in terms of computational complexity and storage capacity since an optimal performance can be obtained even in a deep running structure having a small number of layers.

The facial color feature learning apparatus according to an embodiment of the present invention generates color images for a predetermined plurality of colors with respect to a face image and processes each of the generated color images using a predetermined processing technique A multiplexing unit for generating image information for each of the color images; A selecting unit selecting at least one image information among the image information for each of the generated color images; And a fusion unit for fusing features of each of the selected at least one image information to generate fusion characteristics.

Further, the facial color feature learning apparatus according to an embodiment of the present invention may further include a feature extracting unit that extracts features of each of the selected at least one image information, The convergence characteristic can be generated by fusing the features of each of the above-mentioned image information.

The feature extraction unit may extract characteristics of each of the at least one or more image information by learning features of deep convolutional neural networks (DCNN) or deep neural networks (DNN) of each of the at least one image information.

The multiplexing unit may generate image information for each of the color images through at least one of a preprocessing technique such as illumination correction, pause correction, image alignment, and super resolution.

The fusion unit may generate the convergence feature using a concatenation technique for fusing features of each of the at least one image information into one vector or a fusion technique using a weight based on depth-of-neural network learning.

The fusion unit may generate DNN (deep neural networks) for each feature of the at least one image information to generate the fusion feature.

The DNN may comprise a plurality of fully connected layers.

A face color feature learning method according to an embodiment of the present invention includes: generating color images for a predetermined plurality of colors for a face image; Generating image information for each of the color images through processing using a predetermined processing technique for each of the generated color images; Selecting at least one image information among the image information for each of the generated color images; And generating a convergence feature by fusing features of each of the at least one or more selected image information.

Further, the face color feature learning method according to an embodiment of the present invention may further include extracting a feature of each of the selected at least one image information, and the step of generating the fusion feature may include extracting The fusion features may be generated by fusing features of each of the at least one image information.

The extracting of the feature may include extracting features of each of the at least one or more image information by learning features of deep convolutional neural networks (DCNN) or deep neural networks (DNN) have.

The generating of the image information may generate image information for each of the color images through at least one of a preprocessing technique such as illumination correction, pose correction, image alignment, and super resolution.

The convergence feature generation step may generate the fusion feature using a concatenation technique of fusing the features of each of the at least one image information into one vector or a fusion technique using a weight by depth neural network learning have.

The generating the fusion feature may generate the fusion feature by learning deep neural networks (DNNs) for each feature of the at least one image information.

The DNN may comprise a plurality of fully connected layers.

According to embodiments of the present invention, multiplexing and processing of input data can be performed to achieve robust recognition performance on input face images that vary in a fixed deep-running structure.

Further, according to the embodiments of the present invention, it is possible to sufficiently learn in a deep learning structure having a fixed depth learning structure and a deep learning learning structure having a small number of layers for an input image having various changes, thereby improving the recognition performance of the deep learning structure .

In addition, according to the embodiments of the present invention, optimal performance can be obtained even in a deep-running structure having a small number of layers, thereby enabling effective and efficient design in terms of computational complexity and storage capacity.

1 is a conceptual diagram illustrating a framework of a facial color feature learning apparatus according to an embodiment of the present invention.
FIG. 2 shows an example of multiple image information.
FIG. 3 is a diagram illustrating an exemplary process for extracting multiple features for a multiple input image.
FIG. 4 shows an exemplary diagram for explaining a process of generating fused features for extracted multiple features.
FIG. 5 is a flowchart illustrating an operation of a face color feature learning method according to an exemplary embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. In addition, the same reference numerals shown in the drawings denote the same members.

Embodiments of the present invention are directed to a method and apparatus for performing deep processing of input data by performing multiplexing and processing of input data in order to achieve robust recognition performance for an input image having various changes in a fixed deep running structure and a deep running structure having a small number of layers, Thereby improving the recognition performance of the structure.

At this time, the embodiments of the present invention can achieve an optimum performance even in a deep-running structure having a small number of layers, thereby enabling effective and efficient design in terms of computational complexity and storage capacity.

FIG. 1 is a conceptual diagram illustrating a framework of a facial color feature learning apparatus according to an exemplary embodiment of the present invention, and shows a configuration of an apparatus for performing facial color feature learning using multiple image information generation and processing in a deep learning structure will be.

1, the facial color feature learning apparatus 100 according to an exemplary embodiment of the present invention includes a multiplex processing unit 110, a selection unit 120, a feature extraction unit 130, and a fusion unit 140 .

Here, the multiplexing processing unit 110 is configured to perform multiplexing and processing of input images, the selecting unit 120 is configured to select multiple information, and the feature extracting unit 1310 extracts features of a depth- And the fusion unit 140 may be configured to fuse the extracted multiple features.

The constituent elements of the facial color feature learning apparatus 100 will now be described.

1. Input image multiplexing and processing unit (multiplexing processing unit)

The multiplex processing unit 110 is a module for utilizing multiple information of an input image in order to obtain optimal performance in a limited depth running structure, and generates a plurality of multiple colors through various color models.

In this case, the multiplexing processing unit 110 may convert RGB, YCbCr, YIQ (x, y) into the input image, for example, the face image using the RGB, YCbCr, YIQ, XYZ, HSV, RIQ, RQCr, YQCr, , XYZ, HSV, RIQ, RQCr, YQCr, and La * b *, respectively.

Furthermore, the multiplexing processing unit 110 performs processing on the multi-color image generated using the environment-adaptive preprocessing technique such as illumination correction, pause correction, image alignment, and super-resolution .

For example, the multiplexing processing unit 110 performs preprocessing on each image channel based on an illumination correction method, for example, histogram equalization, differential operator-based filtering, Weber face, local binary pattern, Images can be generated.

That is, the multiplexing processing unit 110 transforms the input face image into various color spaces to generate a multicolor image, and then processes the multicolor image through the preprocessing method. As a result, as shown in FIG. 2, Multiple image information can be generated.

2. Multiple information selection unit (selection unit)

The selecting unit 120 is a module for selecting necessary information for the multiplexed image information processed by the multiplexing processing unit 110. [

At this time, the selection unit 120 can select necessary information based on a combination of information suited to the limited number of layers, for example, a color combination, the number of selected color channels, image information generated by various processes, and the like.

For example, the selection unit 120 may include all of the available information through selection of a large number of image information when there are relatively few training data in a highly varied environment, and a deep running structure having a relatively shallow depth of about 2-3 layers Can be utilized.

Here, the selection unit 120 may select various optimal combinations of information by introducing various objective functions.

3. Deep Learning Based Feature Extractor (Feature Extraction)

The feature extraction unit 130 is a module for extracting features from information selected by the selection unit 120. The feature extraction unit 130 may include a deep convolutional neural network (DCNN) composed of various convolutional layers or a fully connected deep neural networks (DNNs) composed of a layer of high-order neural network (DNN).

This deep learning structure can be learned by the same type of input, the same multiplexing and processing method, and the same multiple information selection method.

That is, as shown in FIG. 3, the feature extraction unit 130 designates an independent feature extractor (deep learning-based feature extractor) for each of the multiple pieces of image information (input image 1 to input image n) , And obtains complementary and discerning features (features 1 to n) from each feature extractor.

In other words, the feature extracting unit 130 acquires information including the feature in the color space selected through the DCNN learning or the DNN learning from the information of the selected color space.

4. Multi-feature fusion part (fusion part)

The fusion unit 140 is a module for fusing a feature that maximizes the discrimination power on the feature vector extracted by the feature extraction unit 130. The feature extraction unit 130 fuses the extracted feature vectors, Lt; RTI ID = 0.0 > fused < / RTI > feature or fused feature vector.

Here, the fusion unit 140 extracts features extracted from the features extracted by the feature extraction unit 130 using a concatenation technique that fuses the independent features into a single vector, fusion using weighted by neural network learning Features can be merged.

That is, as shown in FIG. 4, the fusion unit 140 fuses the learned features (feature 1 to feature n) from the selected various color spaces to generate fused features.

Here, the convergence feature in which the various color features are fused can be obtained by concatenating and combining all feature vectors from various color spaces in order to obtain a high face recognition rate. However, the discriminative capabilities of the feature vectors in the various color spaces are different, and therefore maximum performance is not guaranteed to simply chain all feature vectors. In addition, a dimensional problem due to an increased dimension may also occur.

The convergence feature in the present invention can be generated or obtained by learning DNNs (deep neural networks) composed of a plurality of fully connected layers.

Therefore, the facial color feature learning framework of the present invention learns and collects the learned feature vectors in the selected color space among various color spaces, and the finally output feature vectors contain only the essential contents, which can improve the discrimination ability of the learned features have.

The feature vectors concatenated by the DNN parameter can be mapped to a mixed feature space of low dimension and mixed color features from various color spaces can describe the input information in compressed form. Furthermore, in the case of discernible feature vectors, it can be emphasized by the learned weight of DNN.

As described above, the apparatus according to the embodiment of the present invention can be applied to a variety of input face images in a limited deep learning structure based on a framework for performing multiple information extraction and processing, multiple information selection, feature extraction and fusion by a deep learning structure A robust recognition performance can be achieved.

Here, the present invention uses an information selection method for generating multiple color information and extracting multiple information, and for processing multiple information and maximizing discrimination power for multiple information input. The information obtained as described above is used for optimal deep learning It can be used in structure learning.

Further, the apparatus according to the embodiment of the present invention is configured with a relatively small number of layers through multiple information inputs, thereby making it possible to construct a high-performance deep-running structure. At this time, the number of layers of the deep learning structure can be determined by the number, type, and processing method of multiple images.

Therefore, the present invention utilizes the multiple information for the input image, so that optimal learning can be performed for a deep learning structure having a limited layer, which enables high performance recognition.

According to the apparatus according to the embodiment of the present invention, an optimal performance can be obtained even in a deep running structure having a small number of layers, thereby enabling effective and efficient design in terms of computational complexity and storage capacity.

Particularly, the present invention enables effective and efficient design in terms of operation for storing and recognizing parameters of a deep learning structure by learning through a proposed method for a deep learning structure having a limited number of layers or a small number of layers . That is, it enables a high-speed, low-power design.

FIG. 5 is a flowchart illustrating an operation of a facial color feature learning method according to an exemplary embodiment of the present invention, and is a flowchart illustrating an operation performed by the apparatuses of FIGS. 1 to 4 described above.

Referring to FIG. 5, a face color feature learning method according to an embodiment of the present invention generates a multi-color image for an input image, for example, a face image (S510).

In step S510, RGB, YCbCr, YIQ, XYZ, HSV, RIQ, RQCr, and YQCr for the input image are calculated using RGB, YCbCr, YIQ, XYZ, HSV, RIQ, RQCr, YQCr, And La * b * color images, respectively.

After the multi-color image is generated in step S510, preprocessing is performed on each of the multiple color images generated using the preprocessing technique, for example, illumination correction, pose correction, image alignment, and super resolution (S520).

Here, in step S520, a multi-color image robust to illumination change can be generated by performing preprocessing on each image channel based on the illumination correction method, for example, histogram equalization, differential operator-based filtering, Weber face, local binary pattern, have.

At least one piece of image information is selected from the multicolor image processed in step S520 (S530).

Here, in step S530, necessary image information can be selected on the basis of an information combination suitable for a limited number of layers, for example, a color combination, the number of color channels to be selected, image information generated by various processes and the like.

Of course, in step S530, a large number of image information can be selected when there are relatively few training data in a highly varied environment and a relatively shallow deep learning structure, such as two or three layers, and various objective functions ) Can be introduced to select the optimum combination of information.

If the image information is selected in step S530, the feature of each image information is extracted from the selected image information (S540).

Here, the step S540 may extract the feature of each of the selected image information using a deep convolutional neural network (DCNN) composed of various convolution layers or a deep neural network (DNN) composed of various completely connected layers.

When the feature of each of the image information selected in step S540 is extracted, a fusion feature is maximized for the extracted feature vector to generate a fusion feature (S550).

Here, in step S550, fusion characteristics may be generated using a concatenation technique for merging the extracted independent features into a single vector, and a fusion based on weighted learning using depth neural network learning.

Step S550 may concatenate and combine all of the extracted feature vectors to generate a fusion feature. This convergence feature can be created or obtained by learning a DNN consisting of a plurality of fully connected layers.

The system or apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the systems, devices, and components described in the embodiments may be implemented in various forms such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array ), A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to embodiments may be implemented in the form of a program instruction that may be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (14)

A multiplexing processing unit for generating color images of a predetermined plurality of colors for a face image and generating image information for each of the color images through a process using a predetermined processing technique for each of the generated color images, ;
A selecting unit selecting at least one image information among the image information for each of the generated color images; And
A fusion unit for fusing features of each of the selected at least one image information to generate fusion features,
Wherein the face color feature learning device comprises:
The method according to claim 1,
A feature extracting unit for extracting a feature of each of the at least one selected image information,
Further comprising:
The fusion unit
Wherein the fusion features are generated by fusing features of each of the at least one image information extracted by the feature extraction unit.
3. The method of claim 2,
The feature extraction unit
Characterized by extracting characteristics of each of the at least one image information by learning characteristics of deep convolutional neural networks (DCNN) or deep neural networks (DNN) of each of the at least one image information, Device.
The method according to claim 1,
The multiplexing processor
Wherein the image information for each of the color images is generated through processing using at least one pre-processing method of illumination correction, pose correction, image alignment, and super resolution.
The method according to claim 1,
The fusion unit
Characterized in that the convergence feature is generated using a concatenation technique for fusing features of each of the at least one image information into one vector or a convergence technique for weighting by depth neural network learning .
The method according to claim 1,
The fusion unit
And the convergence feature is generated by learning DNN (deep neural networks) for each feature of the at least one image information.
The method according to claim 6,
The DNN
And a plurality of fully connected layers.
Generating, for the face image, color images for a predetermined plurality of colors;
Generating image information for each of the color images through processing using a predetermined processing technique for each of the generated color images;
Selecting at least one image information among the image information for each of the generated color images; And
Generating fusion features by fusing features of each of the at least one or more selected image information
A face color feature learning method.
9. The method of claim 8,
Extracting features of each of the selected at least one image information
Further comprising:
The step of generating the fusion feature
Wherein the merging feature is generated by merging features of each of the at least one image information extracted by the feature extracting unit.
10. The method of claim 9,
The step of extracting the feature
Characterized by extracting characteristics of each of the at least one image information by learning characteristics of deep convolutional neural networks (DCNN) or deep neural networks (DNN) of each of the at least one image information, Way.
9. The method of claim 8,
The step of generating the image information
Wherein the image information for each of the color images is generated through at least one of a preprocessing method of illumination correction, pose correction, image alignment, and super resolution.
9. The method of claim 8,
The step of generating the fusion feature
Wherein the convergence feature is generated by using a concatenation technique for fusing features of each of the at least one image information into one vector or a convergence technique for weighting by depth neural network learning .
9. The method of claim 8,
The step of generating the fusion feature
And the convergence feature is generated by learning DNN (deep neural networks) for each feature of the at least one image information.
14. The method of claim 13,
The DNN
And a plurality of fully connected layers.

Images