KR20180119013A - Method and apparatus for retrieving image using convolution neural network - Google Patents

Abstract

The present invention provides a method for operating an image retrieving apparatus applying a convolution neural network and the image retrieving apparatus. The method according to the present invention includes the steps of: obtaining an image frame; generating a plurality of color feature maps and a plurality of edge feature maps by convoluting a plurality of color sensitivity kernels and a plurality of edge sensitivity kernels in the image frame, respectively; generating the maximum color activity map for each of a color and an edge using an index of the color sensitivity kernel or an index of the edge sensitivity kernel matched to the maxim activity value at each pixel position based on the plurality of color feature maps and the plurality of edge feature maps; spatially pooling the maximum activity map for each of the color and the edge; and retrieving an object which exists in the image frame by comparing a previously store feature value with a result value concatenating the spatially pooled value. Accordingly, the present invention can more efficiently retrieve a specific object in image data.

Description

TECHNICAL FIELD [0001] The present invention relates to an image retrieval method using convolutional neural networks,

The present invention relates to an image retrieval method and apparatus using a convolution neural network.

A variety of industrial sites may be streamlined by using a machine or processor capable of identifying an entity in a visual representation (e.g., image, etc.). The field of computer vision is to provide an algorithm for identifying and detecting an object of an image wherein the object can be characterized by descriptors identifying one or more points (e.g., all pixel points, interest keypoints, etc.) have. In general, object recognition may involve identifying points of interest in an image for purposes of feature identification and / or object recognition. These points of interest need to be processed so that they are invariant to image scale changes and / or rotations, provide a significant range of distortion, perspective changes, and / or robust matching against noise and illumination changes. In particular, in the field of surveillance systems that collect a large number of images continuously, there is a need to ensure that certain entities can be accurately matched to a large database with high probability. In addition, the field of surveillance systems requires efficient and low-cost computational processing for large databases.

To this end, the Scale Invariant Feature Transform (SIFT) algorithm reduces the cost of indexing images by sorting the objects queried in the initial stage based on the features of the corresponding entity. However, it still requires a large amount of processing in the process of extracting features.

U.S. Published Patent Application No. 2017/0076195 (entitled DISTRIBUTED NEURAL NETWORKS FOR SCALABLE REAL-TIME ANALYTICS)

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to provide an image retrieval system capable of efficiently managing image data and searching / retrieving image data by efficiently describing image data using a convolutional neural network The purpose is to provide. Another object of the present invention is to provide a high efficiency / low cost image search system by reducing the processing load of the convolutional neural network.

It should be understood, however, that the technical scope of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

As a technical means for achieving the above-mentioned technical object, a first aspect of the present invention is a method for acquiring an image frame, comprising: obtaining an image frame; Generating a plurality of color feature maps and a plurality of edge feature maps by convoluting a plurality of color sensitivity kernels and a plurality of edge sensitivity kernels respectively in an image frame; Based on a plurality of color feature maps and a plurality of edge feature maps, an index of a color sensitivity kernel or an index of an edge sensitivity kernel that matches a maximum active value at each pixel position, Generating a maximum activity map for the maximum activity map; Spatial pooling a maximum activity map for each of the color and edge; And comparing the result obtained by concatenating the space-pooled values with pre-stored feature values to search for an entity existing in the image frame.

The second aspect of the present invention also includes a memory for storing a program for retrieving an image using a convolution neural network and a processor for executing the program, Acquires an image frame as it is executed, inputs the acquired image frame to the convolutional neural network, and compares the output value of the convolutional neural network with a previously stored feature value to search for an object existing in the image frame. The convolution neural network includes at least one convolution layer for generating a plurality of color feature maps and a plurality of edge feature maps by convoluting a plurality of color sensitivity kernels and a plurality of edge sensitivity kernels respectively in an image frame, Based on the color feature map and the plurality of edge feature maps, the index of the color sensitivity kernel or the index of the edge sensitivity kernel, which matches the maximum active value at each pixel position, At least one pooling layer for spatial pooling the generated maximum activity map, and at least one full-connected layer for concatenating the pooled values.

A third aspect of the present invention provides a computer-readable recording medium on which a program for implementing the method of the first aspect is recorded.

According to an aspect of the present invention, an embodiment of the present invention describes a feature of image data through a learnable convolution neural network, so that a specific entity in image data can be more efficiently searched / retrieved. In addition, an embodiment of the present invention can allow a pooling layer of the convolution neural network to space-pool using the index of the kernel, thereby enabling a specific object in image data to be searched / retrieved at low cost.

1 is a schematic diagram showing an image retrieval system according to an embodiment of the present invention.
FIG. 2 illustrates a convolution layer according to an embodiment of the present invention. Referring to FIG.
3 is a diagram illustrating an example in which a spatial maximum activity map is constructed from color sensitivity feature maps in a pooling layer according to an embodiment of the present invention.
4 is a diagram illustrating an example of a color histogram according to an embodiment of the present invention.
5 is a flowchart illustrating an operation method of the processor of FIG. 1 according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when a part is referred to as "including " an element, it does not exclude other elements unless specifically stated otherwise.

In this specification, the term " part " includes a unit realized by hardware, a unit realized by software, and a unit realized by using both. Further, one unit may be implemented using two or more hardware, or two or more units may be implemented by one hardware.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

 1 is a schematic diagram of an image retrieval system 10 according to an embodiment of the present invention. Referring to FIG. 1, an image retrieval system 10 includes a surveillance camera 11 (11a, 11b, ...) and an image retrieval apparatus 12. At this time, the surveillance camera 11 and the image search device 12 can exchange data through a wired / wireless network.

The surveillance camera 11 processes the image frame acquired through the image sensor (not shown) and provides the processed image frame to the image search device 12. At this time, the surveillance camera 11 may be for monitoring a specific entity (e.g., a person, a vehicle, etc.) at various places such as a CCTV.

The image retrieval apparatus 12 retrieves a specific object from the image frames collected by the surveillance camera 11. To this end, the image search apparatus 12 includes a memory 121, a feature histogram storage unit 122, and a processor 123. The image search apparatus 12 further includes a communication unit (not shown) for receiving an image frame from the surveillance camera 11, a display unit (not shown) for outputting information processed in the image search apparatus 12, and the like And may further include other components depending on the implementation method.

Hereinafter, the configuration and operation method of the image search apparatus 12 will be described in detail with reference to FIGS. 1 to 5. FIG.

The memory 121 may be implemented as a nonvolatile memory, a volatile memory, a hard disk drive (HDD), or a solid state drive (SSD). When the image search apparatus 12 searches for a specific object And stores the program. The program may include a convolution neural network that generates an image descriptor in accordance with an embodiment of the present invention. Here, the convolutional neural network is a neural network designed to encode the characteristics of image data under the assumption that the input data is an image. Therefore, the convolution neural network is composed of a convolution layer, a pooling layer and a full-connected layer implemented in three dimensions of width, length, and depth (RGB channel) According to an embodiment, each layer may be composed of a single layer or a plurality of layers. On the other hand, the convolutional neural network can be driven as the image frame is input to the convolution layer, under the control of the processor 123. Hereinafter, a convolutional neural network according to an embodiment of the present invention will be described in detail with reference to FIGS. 2 to 4. FIG.

FIG. 2 illustrates a convolution layer according to an embodiment of the present invention. Referring to FIG. The parameters of the convolution layer are composed of a set of learnable convolution kernels 201 and are convolved with input data (i.e., image frame) 211 to output a feature map 213. According to an embodiment of the present invention, the convolution kernel set can be a kernel in known AlexNet model, ConvNet model, LeNet-5 model, and the like. However, the present invention is not limited thereto, and kernels used in various convolution neural network models can be used. Meanwhile, the kernel can be named as a filter according to an embodiment.

In this case, according to an embodiment of the present invention, the convolution kernel set 201 is divided into a color sensitivity kernel 202a and an edge sensitivity kernel 202b according to the hue and edge sensitivity of each kernel. Since the inter-channel standard deviation is sensitive to the sensitivity of the kernel to the color and the inter-pixel standard deviation is sensitive to the sensitivity to the edge, the sensitivity to color and edge of each kernel is calculated by the following equations (1) and .

는 i번째 커널 K의 커널 계수에 대한 RGB 채널 사이에서 계산된 표준편차(

)의 합으로 계산된 색상 민감도를 나타낸다.In this equation, i (i = 1, 2, .. I) is the index of the kernel, and m is the width and height of the i-th kernel. Also,

Is the standard deviation calculated between the RGB channels for the kernel coefficients of the i < th > kernel K (

) ≪ / RTI >

는 i 번째 커널 K의 모든 색상 채널에서 연속적인 수평 및 수직 커널 계수간에 계산된 표준편차(

)의 합으로 계산된 에지 민감도를 나타낸다. In the above equation,

Is the calculated standard deviation between successive horizontal and vertical kernel coefficients in all color channels of the i < th > kernel K (

) ≪ / RTI >

)가 2.0 이상이면, 해당 커널은 색상 감도 커널로 분류될 수 있다. 또는, 특정 커널은 다른 커널들보다 상대적으로 높은 색상 민감도(

) 또는 에지 민감도(

)를 가짐에 따라 색상 감도 커널 또는 에지 감도 커널로 분류될 수 있다. For example, the color sensitivity of a particular kernel (

) Is 2.0 or higher, the kernel can be classified as a color sensitivity kernel. Alternatively, a particular kernel may have a relatively high color sensitivity (

) Or edge sensitivity (

), It can be classified into a color sensitivity kernel or an edge sensitivity kernel.

The input data 211 is then convolved with each of the convolution kernels 212a included in the color sensitivity kernel 202a to generate a plurality of color feature maps 213a, And is convolved with each convolution kernel 212b included in the edge sensitivity kernel 202b to generate a plurality of edge feature maps 213b. The above process may be performed in parallel, but may be performed sequentially according to an implementation.

The convolution layer outputs a plurality of generated color feature maps 213a and a plurality of edge feature maps 213b.

Next, instead of pooling the maximum active value from the color feature maps and the edge feature maps, the pooling layer constructs a maximum activity map using a kernel index corresponding to the maximum active value of each pixel. That is, the maximum activity map according to an exemplary embodiment of the present invention is generated for each color and edge, and each of the indexes is composed of color sensitivity kernel indexes or edge sensitivity kernel indexes. This is expressed by the following equations (3) and (4).

는 각 픽셀 위치(x, y)(x

X , y

Y, X는 가로값, Y는 세로값)는 자연수)에서의 최대 활성값을 나타내며,

는 색상 감도 커널의 개수를 나타내고,

은 색상 특징맵을 나타낸다. 또한,

는 색상 특징맵(

)으로부터 생성되는 최대 활성도 맵을 나타낸다. In the above equation,

(X, y) (x

X, y

Y, X are horizontal values, and Y is vertical values) is a natural number)

Represents the number of color sensitivity kernels,

Represents a color feature map. Also,

Is a color feature map (

). ≪ / RTI >

는 에지 감도 커널의 개수이며,

는 에지 특징맵을 나타낸다. 또한,

는 에지 특징맵(

)으로부터 생성되는 최대 활성도 맵을 나타낸다. In the above equation,

Is the number of edge sensitivity kernels,

Represents an edge feature map. Also,

Is an edge feature map

). ≪ / RTI >

FIG. 3 is a diagram illustrating an example in which a maximum activity map 320 is constructed from color feature maps 310 in a pooling layer according to an embodiment of the present invention. Referring to FIG. 3, when the maximum activity value at the (1,1) pixel position 311 of the color feature maps 310 corresponds to the third color sensitivity kernel, 1) 3 is stored in the pixel. Thus, the pooling layer generates a maximum activity map for color from the color feature maps and generates a maximum activity map for the edges from the edge sensitivity feature maps.

K, K =1,2,..

) 별 빈도값(frequency)으로 표현될 수 있다. 이와 같이, 본 발명의 일 실시예에 따른 풀링 레이어는 공간 정보를 제외하여 간단한 풀링을 수행함으로써 이미지를 저차원적 특징을 추출할 수 있다. The information of the maximum activity maps is then collected as a histogram through spatial pooling. Specifically, the color histogram is collected according to the color sensitivity kernel specific frequency from the maximum activity map for color, and the edge histogram is collected according to the edge sensitivity kernel-specific frequency from the maximum activity map for the edge. 4 is a diagram illustrating an example of a color histogram according to an embodiment of the present invention. As shown in FIG. 4, the color histogram includes a color sensitivity kernel k

K, K = 1,2, ...

) Can be expressed as a frequency value. As such, the pooling layer according to an embodiment of the present invention can extract low-order features of an image by performing simple pulling excluding spatial information.

Next, a full-connected layer connects the color histogram and the edge histogram. That is, the full connected layer outputs a feature histogram as a result of the color histogram and the edge histogram being fully connected. This feature histogram serves as a descriptor that describes the features (i.e., features) of the image.

Meanwhile, in the pooling layer, a space pooling method other than the above embodiment may be applied. For example, according to an embodiment, the pooling layer may space-pool color feature maps and edge feature maps by spatial pyramid matching, quadrant based spatial pooling, and the like. In this case, the output values of the pulling layer can be obtained by obtaining histograms for the respective regions of the image frame, and then connecting the output histograms to output the feature histogram.

Referring again to FIG. 1, the feature histogram storage unit 122 stores feature histograms (i.e., feature values) of image frames obtained through the convolutional neural network, under the control of the processor 123. 1, the feature histogram storage unit 122 and the memory 121 are shown as separate components, but the feature histogram storage unit 122 may be included in the memory 121. [ Or feature histogram storage unit 122 may be a separate database device located external to image retrieval device 12. [

The processor 123 controls the overall operation of the image search apparatus 12. [ The processor 123 may further include a graphics processor unit (GPU) for graphics processing and a digital signal processor (DSP) for signal processing, in addition to a central processing unit (CPU) And may be implemented as a system on chip (SoC) incorporating at least one of the above.

Specifically, the processor 123 can control the memory 121 and the feature histogram storage unit 122 to detect / retrieve objects existing in the image frame received from the surveillance camera 11. [

Referring to FIG. 5, the processor 123 receives an image frame from the surveillance camera 11 (S510). Thereafter, the processor 123 inputs the image frame to the above-described convolutional neural network (S520). Accordingly, the convolutional neural network is driven, and the convolution layer (CONV), the pulling layer (POOL) and the full connected layer (FC) described in Figs. 2 to 4 operate. Specifically, the convolution layer CONV convolutes the color sensitivity kernels and the edge sensitivity kernels respectively in the image frame to generate a plurality of color feature maps and an edge feature map T (S521). Next, the pooling layer (POOL) generates a maximum activity map for each color and edge from the color feature maps and the edge feature maps, and then pools each maximum activity map (S522). Accordingly, the information of each maximum activity map is collected as a color histogram and an edge histogram.

Next, the full connected layer FC links the color histogram and the edge histogram, and outputs the characteristic histogram (S523). That is, the full connected layer outputs a feature histogram as a result of the color histogram and the edge histogram being fully connected.

Thereafter, the processor 123 compares the feature histogram output from the full connected layer with the feature histogram of the pre-stored image frames in the feature histogram storage unit 122 to obtain a pre-stored image containing the same or similar entities as the corresponding image frame (S530). For example, the processor 123 may rank the pre-stored image frames using the similarity score or non-default score between the feature histogram extracted from the image frame and the pre-stored feature histogram (i.e., feature value).

On the other hand, an embodiment of the present invention may also be realized in the form of a recording medium including instructions executable by a computer such as a program module executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. The computer-readable medium may also include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

10: Image search system
11: Camera
12: Image search device
121: Memory
122: Feature histogram storage unit
123: Processor

Claims (8)

A method of operating an image retrieval apparatus using a convolution neural network, the method comprising:
Obtaining an image frame;
Generating a plurality of color feature maps and a plurality of edge feature maps by convoluting the plurality of color sensitivity kernels and the plurality of edge sensitivity kernels respectively in the image frame;
Based on the plurality of color feature maps and the plurality of edge feature maps, an index of a color sensitivity kernel or an index of an edge sensitivity kernel matching a maximum active value at each pixel position, Generating a maximum activity map for each;
Spatially pooling a maximum activity map for each of the hue and edge; And
And comparing the result of concatenating the space-pooled values with pre-stored feature values to search for an entity present in the image frame. The method according to claim 1,
The step of space pooling the maximum activity map
Calculating a frequency per color sensitivity kernel from the maximum activity map for the color, collecting the frequency per kernel in a color histogram, calculating a frequency per edge sensitivity kernel from the maximum activity map for the edge, and collecting it as an edge histogram. 3. The method of claim 2,
The concatenated result value
Wherein the histogram is a feature histogram fully connected to the color histogram and the edge histogram. The method of claim 1, wherein
The pre-stored feature value is a feature histogram of pre-stored image frames,
The step of searching for the entity
Wherein the ranking of the pre-stored image frames is determined according to the degree of similarity or non-preference between the linked resultant value and the characteristic histogram of the pre-stored image frames. The method according to claim 1,
The color sensitivity kernel or edge sensitivity kernels may include:
Wherein the learnable convolution kernels are classified as color sensitive kernels or edge sensitive kernels, depending on color and sensitivity scores for edges. 6. The method of claim 5,
The sensitivity score for the color is the sum of standard deviations between the RGB color channels for the kernel coefficients of the convolution kernels,
Wherein the sensitivity score for the edge is a sum of standard deviations between successive horizontal and vertical kernel coefficients in an RGB color channel. A program for retrieving an image using a convolution neural network is stored in a memory
And a processor for executing the program,
The processor, as the program is executed,
Searching an object existing in the image frame by comparing the output value of the convolutional neural network with a previously stored feature value to obtain an image frame, inputting the obtained image frame to the convolutional neural network,
The convolution neural network
At least one convolution layer for convoluting a plurality of color sensitivity kernels and a plurality of edge sensitivity kernels respectively in the image frame to generate a plurality of color feature maps and a plurality of edge feature maps, And a maximum value generated for each color and edge using the index of the color sensitivity kernel or the index of the edge sensitivity kernel that matches the maximum active value at each pixel position based on the plurality of edge feature maps. At least one pooling layer for spatial pooling an activity map, and at least one full-connected layer for concatenating said pooled values. A computer-readable recording medium on which a program for implementing the method of any one of claims 1 to 6 is recorded.

Images