KR20180135616A - Structure of deep network and deep learning based visual image recognition system - Google Patents

Abstract

A deep learning-based image recognition system according to an embodiment of the present invention comprises: a first network including an input image receiving unit for receiving an input image including an object, a first extracting unit for extracting low-level feature information representing a low-level feature corresponding to the input image, a second extracting unit for extracting middle-level feature information representing a middle-level feature corresponding to the low-level feature information, and a third extracting unit for extracting high-level feature information representing a high-level feature corresponding to the middle-level feature information; a second network including a fourth extraction unit for extracting middle-level feature information representing a middle-level feature corresponding to the low-level feature information extracted by the first extraction unit and a fifth extracting unit for extracting high-level feature information representing a high-level feature corresponding to the middle-level feature information extracted by the fourth extraction unit; and a recognition unit for recognizing an element corresponding to the object using the feature information extracted by the first network and the feature information extracted by the second network.

Description

TECHNICAL FIELD [0001] The present invention relates to a deep network structure and a deep learning based image recognition system,

The present invention relates to a deep network structure and a deep learning based image recognition system, and more particularly, to a deep network structure and a deep learning based image recognition system capable of implementing a model ensemble technique using a branched network structure It is about.

Deep learning and deep learning can be accomplished through a combination of several nonlinear transformation techniques with a high level of abstractions (summarizing core content or functions in large amounts of data or complex data) , Which is a set of machine learning algorithms that can be said to be a field of machine learning that teaches computers to people's minds in a large framework.

When there is any data, it is represented by a form that can be understood by the computer (for example, pixel information is represented by a column vector in the case of an image), and many studies As well as how to create models to learn them. As a result of these efforts, we have developed diverse deep networks such as deep neural networks (DNN), convolutional deep neural networks (CNN), and deep belief networks Running techniques are applied to fields such as computer vision, speech recognition, natural language processing, and voice / signal processing, and show cutting-edge results.

Thus, a deep network used in various fields includes many layers. In 2012, the AlexNet architecture, which uses only 8 layers, greatly improved the existing image recognition rate, and more than 1000 networks were announced due to rapid technological development and deepening network structure.

While these deep networks provide the best performance in many areas, the architectural capabilities of over 1000 layers require a lot of memory and computation to learn and verify. In recent years, as the tendency to use increasingly deeper and deeper networks has become stronger, high-performance image recognizers have been developed that use expensive equipment to learn high-performance image recognizers and transcend human capabilities.

However, as the network becomes deeper, the network contains more learnable parameters, which can cause a significant amount of memory and the number of parameters to become excessive.

U.S. Patent No. 5,030,447

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems of the prior art, and it is an object of the present invention to provide a branching network (hereinafter, referred to as " branch " and a deep network based image recognition system capable of reducing memory and parameter requirements by implementing a model ensemble technique using a branched network structure.

In addition, the correlation between networks can be reduced by normalizing by applying a label smoothing technique.

The deep learning-based image recognition system according to an embodiment of the present invention includes an input image receiving unit for receiving an input image including an object, a first extracting unit for extracting low-level feature information corresponding to the input image, A second extracting unit for extracting intermediate level feature information indicating an intermediate level feature corresponding to the low level feature information and a third extracting unit for extracting high level feature information indicating a high level feature corresponding to the intermediate level feature information, A first network comprising; A fourth extraction unit for extracting intermediate level feature information representing an intermediate level feature corresponding to the low level feature information extracted by the first extraction unit and a second extraction unit for extracting intermediate level feature information corresponding to the intermediate level feature information extracted by the fourth extraction unit Level feature information indicating a high-level feature, and a fifth extracting unit extracting high-level feature information indicative of a high-level feature, and a feature extraction unit extracting features corresponding to the object using the feature information extracted by the first network and the feature information extracted by the second network And the like.

In one embodiment, the first network and the second network may extract the feature information using a label smoothing technique.

According to an embodiment of the present invention, a branched network structure in which a layer for extracting low-level characteristic information corresponding to a low-level characteristic is shared and a layer for extracting characteristic information at an intermediate level or higher is branched is used By implementing the model ensemble technique, it is possible to reduce the occupancy rate and learning speed of the spatial memory.

In addition, the correlation between networks can be reduced by normalizing by applying a label smoothing technique.

Figures 1 and 2 illustrate a structure for abstracting and recognizing and validating an input image as a low level feature, a mid level feature, and a high level feature according to an embodiment FIG.
FIG. 3 is a block diagram for explaining an embodiment of a deep learning-based image recognition system according to the present invention.
FIG. 4 is a block diagram illustrating a network structure of the deep learning-based image recognition system shown in FIG. 3. Referring to FIG.
FIG. 5 is a block diagram for explaining another embodiment of the network structure of the deep learning-based image recognition system shown in FIG.
6 is a graph for explaining a one-hot vector encoding technique and a label smoothing technique.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by one of ordinary skill in the art to which this invention belongs. Also, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

Further, the embodiments described herein will be described with reference to cross-sectional views and / or schematic drawings that are ideal illustrations of the present invention. Thus, the shape of the illustrations may be modified by manufacturing techniques and / or tolerances. In addition, in the drawings of the present invention, each component may be somewhat enlarged or reduced in view of convenience of explanation. Like reference numerals refer to like elements throughout the specification and "and / or" include each and every combination of one or more of the mentioned items.

Hereinafter, the configuration of a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

Hereinafter, Deep learning may represent a set of machine learning algorithms that attempt a high level of abstractions through a combination of various nonlinear transformation techniques. For example, deep learning can represent machine learning that teaches computers how people think. Here, the abstraction can represent an operation of extracting core data from a plurality of data.

Figures 1 and 2 illustrate a structure for abstracting and recognizing and validating an input image as a low level feature, a mid level feature, and a high level feature according to an embodiment FIG.

The deep learning structure 100 shown in FIG. 1 can abstract the input image 101 sequentially using features corresponding to a plurality of levels. For example, the deep-running structure 100 of FIG. 1 may abstract the input image 101 using a low-level feature 110, an intermediate-level feature 120, and a high-level feature 130.

Here, the feature may represent a core data in which a plurality of data (e.g., a training image) is abstracted and learned. For example, in the present specification, a feature may include a feature image in which any image is abstracted and learned. The feature image may be learned as an image generated by performing convolution filtering using a predetermined number of filters of a predetermined size with respect to a training image. The number of learned feature images may correspond to the number of the filters described above.

For example, in FIG. 1, the low-level feature 110 may be represented using one of the low-level feature images 111 and the intermediate-level feature 120 may be represented using one of the intermediate- And the high-level feature 130 may be represented using one of the high-level feature images 131. The low-level feature images 111 are subjected to convolution filtering on the training images, and the intermediate-level feature images 121 are subjected to different convolution filtering on the low-level feature images 111, Images, and high-level feature images 131 may be subjected to another convolution filtering on the intermediate-level feature images 121 to display the learned images.

The result of abstracting the input image 101 in the deep learning structure 100 shown in FIG. 1 may be expressed as feature information indicating a feature corresponding to the input image 101 from the features corresponding to the respective levels. For example, the feature information may include a feature value indicating an arbitrary feature image. The deep learning structure 100 can extract feature information corresponding to each level using a pre-learned layer corresponding to each level.

For example, in Figure 1, the deep-running structure 100 may be used to extract low-level feature information representing a low-level feature 110 corresponding to an input image 101 and to represent intermediate level features 120 corresponding to low- It is possible to extract the intermediate level feature information and extract the high level feature information indicating the high level feature 130 corresponding to the intermediate level feature information.

In the structure 100 shown in FIG. 1, the module for performing recognition / verification 140 may sequentially perform an abstracting process at each level and perform recognition and verification using feature information of the last level. For example, in FIG. 1, recognition and verification may be performed using only the high-level feature information indicating the high-level feature 130. FIG. In this case, the low-level feature information and the intermediate level feature information may be lost.

2, the module 240 for performing recognition / verification can perform recognition and verification using feature information corresponding to all levels. For example, the module 240 performing recognition / verification in FIG. 2 may perform recognition and verification of the input image 101 using low-level feature information, intermediate-level feature information, and high-level feature information. In FIG. 2, three levels (for example, low level, middle level, and high level) have been exemplarily described. In the following description, the characteristics of an image may be abstracted into at least two levels. As described above, by utilizing all feature information output from each layer, a recognition rate and a verification rate for an image can be secured.

The deep running structure 200 according to one embodiment can be applied to recognize and verify various input images 101. [ For example, the input image 101 may include an image associated with the object (e.g., an image representing the shape of the object). An object may represent a person (e.g., a person's face, body, etc.), an animal, or an object that is contained within a region of interest (ROI) of the image. For example, the deep-running structure 200 according to one embodiment can be used to recognize a person's face and recognize and authenticate the user. Also, the deep learning structure 200 can be used to automatically search for and manage vast amounts of content (e.g., multimedia such as photos, videos, etc.).

The deep learning structure 200 may be implemented in software or hardware such as a chip, and may be mounted on an electronic device. For example, the electronic device may include a mobile device (e.g., a mobile phone, a smart phone, etc.) and a home appliance (e.g., a TV, etc.).

The deep running structure 200 can be applied to an apparatus for recognizing an image and an apparatus for verifying an image. A device for recognizing an image and a device for verifying an image can be learned using training data. The training data may include training images, training elements, and training information associated with the training objects.

For example, training data for learning of an image recognizing device may include training images and training elements associated with the training images. An apparatus for recognizing an image can be learned such that a training element is output from a training image. Here, the training element may be a value indicating a training object included in the training image.

For another example, the training data for learning of the apparatus for verifying an image may include training image pairs and training information. An apparatus for verifying an image can be learned such that training information is output from a pair of training images. Here, the training information may be a value indicating whether the corresponding training image pair includes the same training object.

According to one embodiment, the deep-running structure may include an artificial neural network, for example, a Deep Convolutional Neural Network (DCNN).

A neural network may include an input layer, a hidden layer, and an output layer. Each layer includes a plurality of nodes, and nodes between adjacent layers can be connected to each other with connection weights. Each node may operate based on an activation model. The output value corresponding to the input value can be determined according to the activation model. The output value of an arbitrary node can be input to the node of the next layer connected to that node. A node of the next layer can receive values output from a plurality of nodes. In the process of inputting the output value of an arbitrary node to the node of the next layer, a connection weight can be applied. The node of the next layer can output the output value corresponding to the input value to the node of the next layer connected to the node based on the activation model.

The output layer may include nodes corresponding to a plurality of elements. The nodes of the output layer can output characteristic values corresponding to a plurality of elements. As will be described in detail below, the feature values output from the neural network can be transformed into elements through linear classifiers for a plurality of elements.

FIG. 3 is a block diagram for explaining an embodiment of a deep learning-based image recognition system according to the present invention. FIG. 4 is a block diagram for explaining an embodiment of a deep learning- FIG.

3, the deep learning-based image recognition system according to an embodiment of the present invention includes an input image receiving unit 310, a first network 320, a second network 330, and a recognition unit 340 .

The input image receiving unit 310 may receive an input image including an object. Here, the input image may be a preprocessed image. Here, the preprocessed image represents an image processed so that an arbitrary image has a predetermined size, a predetermined resolution, and a ratio between the object and the background in the image (where the background is a portion excluding an object in the image) .

The first network 320 and the second network 330 are for implementing a model ensemble technique to improve the performance of the DIP network. The first network 320 and the second network 330 may have different initial values . ≪ / RTI >

The first network 320 includes a first extracting unit 321 for extracting a low level feature, a second extracting unit 322 for extracting an intermediate level feature, and a third extracting unit 323 for extracting a high level feature can do.

The first extracting unit 321 can extract low-level feature information indicating a low-level feature from an input image input through the input image receiving unit 310. [ Here, the low-level feature may be a feature point such as an edge or a blob, which is the most basic element forming an image. 4, the first extracting unit 321 may include a plurality of convolution layers for extracting low-level feature information, and may include a convolution layer for extracting low-level feature information from a layer connected to the input image receiving unit 310, The feature information corresponding to the higher dimensional feature can be extracted as the layer moves.

The second extracting unit 322 can extract the intermediate level feature information indicating the intermediate level feature corresponding to the low level feature extracted by the first extracting unit 321 in the input image. Wherein the intermediate level feature information may represent a feature value representing at least one intermediate level feature corresponding to any of the plurality of intermediate level features. Here, the intermediate level feature may include a learned feature image in which the low level feature is abstracted. The second extracting unit 322 may include a plurality of convolution layers for extracting intermediate level feature information.

The third extracting unit 323 may extract high-level feature information indicating a high-level feature corresponding to the intermediate level feature extracted by the second extracting unit 322 in the input image. Here, the high-level feature information may represent a feature value representing at least one high-level feature corresponding to any of the plurality of high-level features. Here, the high level feature may include the learned feature image in which the intermediate level feature is abstracted. In addition, the third extracting unit 323 may include a plurality of convolution layers for extracting high-level feature information.

The second network 330 may extract the intermediate level feature information and the high level feature information using the low level feature information extracted by the first extracting unit 321 of the first network 320. [ That is, as shown in FIG. 4, the second network 330 may have a structure in which the first network 320 and the first extracting unit 321 are shared. In other words, the first network 320 and the second network 330 may share the first extracting unit 321 for extracting low-level feature information, and may be branched at a mid-level or higher. Accordingly, it is possible to reduce the number of memories and parameters required to obtain a plurality of outputs for one input image.

In one embodiment, the second network 330 may include a fourth extractor 332 and a fifth extractor 334.

The fourth extracting unit 332 can extract the intermediate level information indicating the intermediate level feature corresponding to the low level feature information extracted by the first extracting unit 321. [ Wherein the intermediate level feature information may represent a feature value representing at least one intermediate level feature corresponding to any of the plurality of intermediate level features. Here, the fourth extracting unit 332 may include a plurality of convolution layers for extracting intermediate level feature information.

The fifth extracting unit 334 may extract high level information indicating a high level feature corresponding to the intermediate level feature information extracted by the fourth extracting unit 3320. The high level feature information may represent a feature value representing at least one high level feature corresponding to any of the plurality of high level features. Here, the high level feature may include the learned feature image in which the intermediate level feature is abstracted. Here, the fifth extracting unit 334 may include a plurality of convolution layers for extracting high-level feature information.

In one embodiment, the first extracting unit 321 to the fifth extracting unit 334 may include a convolution layer and a pooling layer as shown in FIG.

The convolution layer can be used to perform convolutional filtering to filter information extracted from the previous layer or previous extractor using a filter of a predetermined size (e.g., 7 x 7 or 3 x 3). In Fig. 4, it can be shown as "conv." For example, the convolution layer of the first extracting unit 321 may filter a predetermined edge. Here, as a result of the convolution filtering, filtering images corresponding to the number of filters can be generated according to the number of filters included in the convolution layer. The convolution layer may consist of the nodes included in the filtered images. Each node included in the convolution layer can receive a filtered value from a region of a predetermined size of a feature image of the previous layer or the previous extractor (in the case of the first extractor 621, the input image). ReLU (Rectifier Linear Unit) can be used as an activation model of each node included in the convolution layer. ReLU is a model that outputs 0 for inputs below 0 and outputs a linearly proportional value for inputs beyond 0.

The pooling layer can be used to extract representative values from feature images of the previous layer through pooling. It can be shown as "pool" in FIG. Here, the flipping layer may be any of max pooling, average pooling, or stochastic pooling. In an embodiment, the pooling layer of the first extracting unit 321 may use max pooling, and the pooling layer of the third extracting unit 323 and the fifth extracting unit 334 may use average pooling. Pooling images can be generated for each pool of feature images. The pooling layer may be composed of the nodes included in the pooling images. Each node included in the pooling layer can receive the pooled value from the region of the size of the corresponding feature image. For example, the pulling layer included in the first extracting unit may extract representative values from information corresponding to the filtered input image.

In the convolution layer and the pulling layer described above, nodes between adjacent layers are partially connected, and connection weights can be shared.

The filters of the convolution layer included in the second extracting unit 322 and the fourth extracting unit 332 may have complex edges compared to the filters of the convolution layer included in the first extracting unit 321, Can be filtered. In the pooling layer included in the second extracting unit 322 and the fourth extracting unit 332, the filtered image filtered by the convolution layer of the second extracting unit 322 and the fourth extracting unit 332, (E.g., filtered first feature information) may be extracted. As described above, in the layers included in the second extracting unit 322 and the fourth extracting unit 332, feature information having higher complexity than the first extracting unit 321 can be extracted. For example, the feature information corresponding to the intermediate level may be extracted with respect to the input image. (E. G., The intermediate level may have intermediate complexity)

Convolution filtering may be performed using filters of a predetermined size in the convolution layer of the third extracting unit 323 and the fifth extracting unit 334. Each of the filters may filter a predetermined edge. The filters of the convolution layer of the third extracting unit 323 and the fifth extracting unit 334 have more complicated edges than the filters of the convolution layer of the second extracting unit 322 or the fourth extracting unit 332 Can be filtered. For example, feature information corresponding to a high level with respect to an input image may be extracted. (E.g., high level may have high complexity)

However, the number of extraction units and the structure of the layers included in the extraction unit are not limited to the above-described ones, but may be changed according to the design.

The recognition unit 340 receives the low level feature information, the intermediate level feature information, and the high level feature information extracted by the first network and the intermediate level feature information and the high level feature information extracted by the second network, Can recognize an element corresponding to an object included in the object. Here, the element corresponding to the input image may include, for example, a value indicating an object included in the reference images that have been learned from the training data and stored.

In one embodiment, the recognizer 630 may include at least one layer and may include, for example, a fully connected layer or the like (shown as "FULLY CON" in FIG. 4) ). In the fully connected layer, nodes between adjacent layers are fully connected, and connection weights can be set individually. For example, each of the fully connected layers may comprise 2048 nodes.

In an embodiment, the recognition unit 340 may calculate the average value of the result of the feature information extracted by the first network 320 and the result of the feature information extracted by the second network 330, The average value can be followed as the final result.

Although not shown, it may further include a learning unit that learns parameters of the recognition unit 340 to recognize an element from the input image.

FIG. 5 is a block diagram for explaining another embodiment of the network structure of the deep learning-based image recognition system shown in FIG. 3. FIG. 6 illustrates a one-hot vector encoding scheme and a label smoothing scheme FIG.

4, when the first network 320 and the second network 330 are implemented in a structure sharing the first extracting unit 321, even if the intermediate level and the high level are independently learned from each other A correlation may occur between the first network 320 and the second network 430 so that when one-hot vector encoding technique is used, compared with ensuring an independent network, May be reduced.

5, the basic structure of the first network 320 and the second network 330 is the same as that of the embodiment of FIG. 4, except that a correlation is established between the first network 320 and the second network 330 The first network 320 and the second network 330 extract feature information by using a label smoothing technique.

In one embodiment, the label smoothing technique may encode a ground truth label using Equation 1 below.

Equation 1

Here, k means the number of classses. This label smoothing technique can be interpreted as a method of pre-normalizing the distribution of predicted values from the model to a uniform distribution. Also, if the two classes are very similar, the one-hot vector always indicates that the two classes (the features extracted by the first network and the second network) are completely different, whereas the label smoothing technique reduces the difference between the two classes The network can learn the knowledge extracted from the learning data.

As described above, the correlation between the first network 320 and the second network 330 can be reduced by normalizing the model by defining a properly smoothly encoded vector using a label smoothing technique as a ground truth label .

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components.

For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute one or more software applications that are executed on an operating system (OS) and an operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions to be recorded on the medium may be those specially designed and constructed for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not intended to limit the invention to the particular forms disclosed. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

310: input image receiving unit
320: first network
321:
322:
323: Third extracting unit
330: Second network
332: fourth extracting unit
334: fifth extracting unit
340:

Claims (6)

An input image receiving unit for receiving an input image including an object;
A second extraction unit for extracting intermediate level feature information representing an intermediate level feature corresponding to the low level feature information and a second extraction unit for extracting intermediate level feature information corresponding to the low level feature information, And a third extracting unit for extracting high-level feature information indicating a high-level feature corresponding to the level feature information;
Level feature information corresponding to the low-level feature information extracted by the first extracting unit, and a fourth extracting unit extracting intermediate level feature information indicating an intermediate level feature corresponding to the low-level feature information extracted by the first extracting unit, A second network including a fifth extractor for extracting high-level feature information indicative of a high-level feature; And
A recognition unit for recognizing an element corresponding to the object using the feature information extracted by the first network and the feature information extracted by the second network; Based learning system.
The apparatus of claim 1,
Wherein the element corresponding to the object is recognized using the characteristic information extracted by the first network and the characteristic information extracted by the second network.
The apparatus according to claim 1, wherein the first extracting unit to the fifth extracting unit comprise:
Level feature information, and at least one convolution layer for extracting low-level feature information, intermediate-level feature information, or high-level feature information, respectively.
2. The method of claim 1, wherein the first network and the second network comprise:
A deep learning based image recognition system for extracting the feature information using a label smoothing technique.
A first low-level feature extracting unit for extracting low-level feature information representing a low-level feature corresponding to the input image, a first intermediate level feature extracting unit for extracting intermediate-level feature information representing an intermediate level feature corresponding to the low- And a first high-level feature extraction unit for extracting high-level feature information representing a high-level feature corresponding to the intermediate level feature information; And
A second intermediate level feature extraction unit that is branched from the low level feature extraction unit and extracts intermediate level feature information indicating an intermediate level feature corresponding to the low level feature information; A second network including a second high-level feature extraction unit for extracting high-level feature information representing a high-level feature corresponding to the intermediate level feature information;
≪ / RTI >
6. The method of claim 5, wherein the first network and the second network comprise:
A deep network structure that encodes the label of the ground truth using label smoothing.

Images