KR101995523B1 - Apparatus and method for object detection with shadow removed - Google Patents

Abstract

The present invention relates to an object detecting apparatus and method, and more particularly, to an object detecting apparatus and method for detecting an object shadow in a low resolution image, and accurately detecting the object by removing the detected shadow. The present invention can accurately detect the object from which the shadow is removed by classifying the object and the shadow of the object based on the circuit neural network in the low resolution image. In addition, the present invention is robust to changes in the lighting environment since object detection is performed by converting a low resolution image into an HSV color model.

Description

Object detection apparatus and method {APPARATUS AND METHOD FOR OBJECT DETECTION WITH SHADOW REMOVED}

The present invention relates to an object detecting apparatus and method, and more particularly, to an object detecting apparatus and method for detecting an object shadow in a low resolution image, and accurately detecting the object by removing the detected shadow.

Recently, as the intelligent surveillance system is expanded, researches on technologies for detecting, tracking, and recognizing objects in images captured by the surveillance camera have been actively conducted. Most intelligent surveillance systems detect objects in visible light images, and it is important to accurately separate shadows and objects in order to detect objects accurately. For this purpose, existing intelligent surveillance systems use NIR and thermal cameras, which require additional lighting or are expensive.

Existing shadow detection algorithms mainly detect color features of objects and shadows. However, since there are many variables in the real environment, it is difficult to accurately detect an object.

Background art of the present invention has been published in the Republic of Korea Patent Publication No. 2015-01739.

The present invention provides an object detection apparatus and method for detecting a shadow of an object based on a convolutional neural network (CNN) of a low resolution image, and accurately detecting the object by removing the detected shadow.

According to one aspect of the invention, an object detecting apparatus is provided.

An object detecting apparatus according to an embodiment of the present disclosure receives an input image, performs a difference operation between the input image and the background image, and detects a foreground region, and refers to the detected foreground region in the input image. A window image generator for generating an approximate object area as a window image in pixel units, a circuit neural network learner for learning a convolutional neural network (CNN) using training data constructed from a plurality of window images; And an object detector configured to transform the window image into a predetermined size by interpolation, classify the modified window image into an object or a shadow using a learned circuit neural network, and detect an object region except for the region classified as a shadow. do.

According to another aspect of the present invention, an object detection method is provided.

An object detecting method according to an embodiment of the present invention is to detect an foreground area by receiving an input image and performing a difference operation between the input image and a background image, and approximate an object in the input image with reference to the detected foreground area. Generating an area as a window image in pixel units, learning a convolutional neural network (CNN) using training data constructed from a plurality of window images, and transforming the window image to a predetermined size by interpolation. And classifying the modified window image into an object or a shadow by using the learned circuit neural network, and displaying an object area except the area classified as a shadow on the input image.

The present invention can accurately detect the object from which the shadow is removed by classifying the object and the shadow of the object based on the circuit neural network in the low resolution image.

In addition, the present invention is robust to changes in the lighting environment since object detection is performed by converting a low resolution image into an HSV color model.

1 to 4 are diagrams for describing an object detecting apparatus according to an embodiment of the present invention.
5 and 6 are diagrams for explaining a method of detecting an object using an object detecting apparatus according to an embodiment of the present invention.
7 to 9 are diagrams illustrating a result of classifying an object and a shadow of the object detection method through the circuit neural network-based learning according to an embodiment of the present invention.
10 and 11 are diagrams showing the results of comparing the performance of the object detection method according to an embodiment of the present invention and the existing object detection method.
12 is a view comparing object detection results with open data in an object detection method according to an embodiment of the present invention.

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, when it is determined that the detailed description of the related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. In addition, when a part is said to "include" a certain component, it means that it may further include other components, except to exclude other components unless specifically stated otherwise.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, the same reference numerals will be used for the same means regardless of the reference numerals in order to facilitate the overall understanding.

1 to 4 are diagrams for describing an object detecting apparatus according to an exemplary embodiment.

Referring to FIG. 1, the object detecting apparatus 100 may include a foreground area detector 110, a window image generator 120, a circuit neural network learner 130, and an object detector 140.

The foreground area detector 110 receives an input image and detects a foreground area by performing a difference operation between the input image and the background image. The input image and the background image may be RGB color model images acquired through a visible light camera.

The window image generator 120 generates an approximate object region in the input image as a window image in units of pixels with reference to the detected foreground region. In this case, the window image generating unit 120 converts the window image into the HSV color model through Equation 1 and generates three window images.

는 k프레임에서 x, y좌표의

i번째 윈도우 이미지,

은 F의 각 채널,

는 k프레임에서의 입력 이미지의 S값, V값,

는 참조 영상의 V값이다. In Equation 1

Is the x, y coordinate

i-th window image,

Each channel of f,

Is the S, V, and

Is the V value of the reference image.

The circuit neural network learner 130 learns a convolutional neural network (CNN) based on VGG Net-16 based on training data constructed from a plurality of window images. At this time, the circuit neural network learner 130 classifies the window image into object data and shadow data based on a ground truth image. Here, referring to FIG. 2, the measured image is an image in which an object and a shadow are manually divided by referring to the input image 210, and the object is displayed in blue and the shadow in red. With reference to the measured image 220, the circuit neural network learner 130 classifies the window image into object data when the center pixel coordinate of the window image corresponds to the object in the measured image, and the center pixel coordinate of the window image is the measured image. In the case of shadow in the window image can be divided into shadow data.

The circuit neural network learning unit 130 transforms the window image divided into the object data and the shadow data into a predetermined size by bi-linear interpolation and uses the input as the input of the circuit neural network. For example, the circuit neural network learner 130 may transform a window image having a size of 21 × 21 × 3 pixels into a size of 224 × 224 × 3 pixels and use it as an input of the circuit neural network. Here, the structure of the circuit neural network will be described with reference to FIGS. 3 and 4.

Referring to FIG. 3, a convolutional neural network according to an embodiment of the present invention is designed to finally classify an input image into an object or a shadow through 13 convolution layers 310 and 3 hidden layers 320.

Referring to FIG. 4, the convolutional neural network includes (1) an image input layer 400, (2) five convolutional layer groups 410-450, and (3) three hidden layer groups 460-480. .

(1) The image input layer 400 inputs a window image having a preset pixel size. In this case, the window image may use a three-channel image of the HSV color model and may have a size of 224 × 224 × 3 pixels.

(2) Five convolutional layer groups 410 to 450 extract a feature by applying a convolutional neural network to the input window image. The five convolutional layer groups may include one or more convolutional layers, each convolutional layer comprising a Rectified Linear Unit (ReLU) layer and / or a Max Pooling layer. It may include.

The first feature extraction layer group 410 includes two convolutional layers. An image of 224 × 224 × 3 pixels is input to the first convolution layer, and may be convolved by one padding at intervals of one pixel by using 64 filters having a size of 3 × 3 × 3. In this case, the weight size per filter is 3 × 3 × 3 = 27, and the total number of parameters in the first convolutional layer is (27 + 1) × 64 = 1792, where 1 represents a bias. The size of the feature map output from the first convolutional layer is 224 × 224 × 64. The output height (or width) is 224 (= (224-3 + 2 × 1) / 1 + 1 using the formula Input height (or width)-filter height (or width) + 2 × padding) / stride +1 Can be calculated as The outputs are used as inputs of the second convolutional layer via the ReLU layer. The second convolutional layer convolves the input image by one padding at intervals of one pixel using 64 filters of size 3 × 3 × 64, passes through the calibration linear unit layer, and then moves the image to 2 pixel intervals. It is processed by a first max pulling layer applying filters of size x 2 and then down sampled to size 112 x 112 x 64 pixels.

The second feature extraction layer group 420 includes two convolutional layers. An image of 112 × 112 × 64 pixels is input to the third convolution layer, and may be convolved by one padding at intervals of one pixel by using 128 filters having a size of 3 × 3 × 64. The outputs are used as inputs to the fourth convolutional layer via the ReLU layer. The fourth convolutional layer convolves the input image by one padding at intervals of one pixel using 128 filters having a size of 3 × 3 × 128, passes through a calibration linear unit layer, and has a size of 2 × 2 at two pixel intervals. It is processed by a second max pooling layer that applies filters of and then downsampled to 56 × 56 × 128 pixels.

The third feature extraction layer group 430 includes three convolutional layers. An image of 56 × 56 × 128 pixels is input to the fifth convolution layer, and may be convolved by one padding at intervals of one pixel by using 256 filters having a size of 3 × 3 × 128. The outputs are used as inputs to the sixth convolutional layer via the ReLU layer. The sixth convolution layer convolves the input image by 1 padding at intervals of 1 pixel by applying 256 filters having a size of 3 × 3 × 256, and uses it as an input of the seventh convolution layer through a correction linear unit layer. do. The seventh convolution layer convolves the input image by 1 padding at 1 pixel intervals by applying 256 filters of 3 × 3 × 256 size, passes through the correction linear unit layer, and 2 × 2 size at 2 pixel intervals. It is processed by a third max pooling layer to apply filters of and then downsampled to 28 × 28 × 256 pixel size.

The fourth feature extraction layer group 440 includes three convolutional layers. An image of 28 × 28 × 256 pixels is input to the eighth convolution layer, and may be convolved by one padding at intervals of one pixel using 256 filters having a size of 3 × 3 × 256. The outputs are used as inputs to the ninth convolutional layer via the ReLU layer. The ninth convolution layer convolves the input image by one padding at intervals of one pixel by using 512 filters having a size of 3 × 3 × 512, and uses the correction linear unit layer as an input of the tenth convolution layer. do. The 10th convolution layer convolves the input image by 1 padding at 1 pixel intervals using 512 filters of 3 × 3 × 512 size, passes through the calibration linear unit layer, and 2 × 2 sizes at 2 pixel intervals. It is processed by a fourth max pooling layer applying filters of and then downsampled to a size of 14x14x512 pixels.

The fifth feature extraction layer group 450 includes three convolutional layers. An image of 14 × 14 × 512 pixels is input to the eleventh convolution layer and may be convolved by one padding at intervals of one pixel using 512 filters having a size of 3 × 3 × 512. The outputs are used as inputs to the twelfth convolutional layer via the ReLU layer. The twelfth convolution layer convolves the input image by one padding at intervals of one pixel by using 512 filters having a size of 3 × 3 × 512 and uses the correction linear unit layer as an input of the thirteenth convolution layer. do. The thirteenth convolution layer convolves the input image by one padding at one pixel interval by using one filter having a size of 3 × 3 × 512, passes through a calibration linear unit layer, and then has a size of 2 × 2 at 2 pixel intervals. It is processed by a fifth max pooling layer to apply filters of and then downsampled to a size of 7x7x512 pixels.

(3) The three hidden layer groups 460-480 are a fully connected layer, a calibration linear unit layer (ReLU layer), a dropout layer, a softmax layer and a classification layer. (Classification layer) may be included.

The first hidden layer group 460 outputs 4096 × 1 nodes via a first fully connected layer and a calibration linear unit layer.

The second hidden layer group 470 outputs 4096 × 1 nodes via a second fully connected layer, a calibration linear unit layer, and a dropout layer. In this case, the dropout layer is used to prevent overfitting, and the output of each hidden node may be set to 0 based on a preset probability. Here, the preset probability may be 0.5.

The third hidden layer group 480 outputs a 2 × 1 node through the third fully connected layer, the softmax layer, and the classification layer, so that the probability that the image input to the image input layer 400 is finally classified as an object or a shadow. To calculate.

The object detector 140 classifies the input window image into an object or a shadow by using the learned circuit neural network, and detects and displays an object area except the area classified as a shadow on the input image.

5 and 6 are diagrams for describing a method of detecting an object using an object detecting apparatus according to an embodiment of the present invention.

Referring to FIG. 5, in operation S510, the object detecting apparatus receives an input image and detects a foreground area by performing a difference operation between the input image and the background image. The input image and the background image may be RGB color model images acquired through a visible light camera.

In operation S520, the object detecting apparatus generates an approximate object region from the input image as a window image in units of pixels with reference to the detected foreground region. Here, the window image may be converted into an HSV color model through Equation 1 to generate a three-channel window image.

는 k프레임에서 x, y좌표의

i번째 윈도우 이미지,

은 F의 각 채널,

는 k프레임에서의 입력 이미지의 S값, V값, B는 참조 영상의 V값이다. In Equation 1

Is the x, y coordinate

i-th window image,

Each channel of f,

Is the S value, the V value, and B of the input image in k frames.

In operation S530, the apparatus for detecting an object learns a convolutional neural network (CNN) using learning data constructed from a plurality of window images.

In operation S540, the object detecting apparatus classifies the input window image into an object or a shadow by using the learned circuit neural network.

 In operation S540, the object detecting apparatus displays the object region except the region classified as the shadow on the input image.

6 illustrates an image of a result of performing the above-described steps in FIG. 5.

In FIG. 6, (a) is an image input to the object detecting apparatus 100, and the foreground image detected by performing a difference operation with the background image is the same as that of FIG. In FIG. 6, (c) is a result of generating an approximate object area in a pixel unit window image from the input image with reference to the detected foreground image. As shown in (c) of FIG. 6, the window image includes an object region including a shadow of the object. The present invention classifies the shadow and the object as shown in (d) of FIG. 6 by applying the learned circuit neural network to the window image to detect the object area from which the shadow is removed from the window image. The removed object area can be detected.

In order to evaluate the performance of the object detecting apparatus 100 according to an embodiment of the present invention, the Logitech webcam C600 is trained using data of 1,802,147 photographed in various environments having changes in day, night, dawn, weather, temperature, and lighting. And verification experiments were conducted. The data used for learning and verification are window images created by dividing objects and shadows based on the measured images as shown in Table 1, and classified into two groups to perform 2-fold cross validation.

Window image Object Shadow Number of data 900,443 901,704 Group 1 450,230 450,875 Group 2 450,213 450,829

Training and verification was done using a graphics card NVIDIA GeForce GTX TITAN X (3,072 CUDA cores) computing device with an Intel® Core ™ i7-6700 CPU @ 3.40 GHz (4 CPUs), 64 GB memory and 12 GB of memory.

The object detection apparatus 100 trained the circuit neural network model, and finally changed the 21 × 21 × 3 pixel size window image to 224 × 224 × 3 pixel size by bi-linear interpolation. It was used for. Here, the learning of the circuit neural network model is the same as the method described above with reference to FIGS. 3 and 4.

7 to 9 are diagrams illustrating a result of classifying an object and a shadow of an object detection method through circuit neural network-based learning according to an embodiment of the present invention.

Referring to FIG. 7, the graph y-axis represents classification learning accuracy and the x-axis represents a training epoch in FIG. 7. Here, the training epoch is assumed that the data to be learned is 10,000 pieces, and the 10 epochs means that 100,000 pieces are learned. As shown in FIG. 7, the accuracy of the object detection method converges near 100 and the loss converges near zero.

Referring to FIG. 8, the object detecting apparatus 100 classifies objects and shadows with accuracy of 98.13% and 99.15% as a result of performing two cross-validation on the window image according to the object detecting method described above.

Referring to FIG. 9, the object detecting apparatus 100 exhibited an average performance of 98% when the TPR, PPV, ACC, and F_score were measured using Equations 2 to 5 below. In Equation 2, #TP, #TN, #FP, and #FN represent the numbers of true positives (TPs), true negatives (TNs), false positives (FPs), and false negatives (FNs), respectively.

10 and 11 are diagrams showing the results of comparing the performance of the object detection method according to an embodiment of the present invention and the existing object detection method.

Referring to FIG. 10, it can be seen that the object detection method of the present invention has high accuracy with almost no error compared to existing methods.

Referring to FIG. 11, the object detection method of the present invention showed higher accuracy than the existing method by 97.67% when comparing the performance with the existing method using the CAVIAR dataset.

12 is a diagram illustrating object detection results of an object detection method according to an embodiment of the present invention compared with open data.

(A) and (b) of FIG. 12 are input images provided from open data, and measured images classified by objects and shadows. 12C illustrates a result of classifying an object and a shadow of an input image of open data using the object detection method of the present invention. As shown in FIG. 12, the measured image (b) of the open data and the resulting image (c) of the present invention are very similar.

The object detecting method according to various embodiments of the present disclosure may be implemented in the form of program instructions that may be executed through computer means such as various servers. In addition, a program and an application for executing the object detecting method according to the present invention may be installed in computer means and recorded in a computer readable medium. Computer-readable media may include, alone or in combination with the program instructions, data files, data structures, and the like. The program instructions recorded on the computer readable medium may be those specially designed and constructed for the present invention, or may be known and available to those skilled in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Hardware devices specially configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory and the like.

So far, the present invention has been described with reference to the embodiments. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

100: object detection device
110: foreground area detection unit
120: window image generating unit
130: circuit neural network learning unit
140: object detection unit

Claims (9)

In the object detecting apparatus,
A foreground region detector which receives an input image and detects a foreground region by performing a difference operation between the input image and the background image;
A window image generating unit generating an object region as a window image in pixel units from the input image with reference to the detected foreground region;
A circuit neural network learning unit for learning a convolutional neural network (CNN) using training data constructed from a plurality of window images; And
And transforming the window image into a predetermined size by interpolation, classifying the modified window image into an object or a shadow using a trained circuit neural network, and detecting an object area except an area classified as a shadow. ,
The window image generation unit,
Equation 1 below to convert the window image to an HSV color model to generate a three-channel window image,

In Equation 1

Is the x, y coordinate

i-th window image,

Each channel of f,

Is the S, V, and

Is an object detection device that means a V value of the reference image.
delete The method of claim 1,
The circuit neural network learning unit
The window image included in the training data is divided into object data and shadow data based on a ground truth image, and the object data and the shadow data are transformed into a predetermined size by interpolation to input the circuit neural network. Object detection device to use.
The method of claim 1,
The circuit neural network
An image input layer for inputting a window image having a preset pixel size;
A convolution layer group for extracting features by applying a convolutional neural network to the input window image; And
And a hidden layer group that classifies the window image into an object or a shadow based on the extracted feature.
The method of claim 4, wherein
The convolutional neural network includes five convolutional layer groups and three hidden layer groups,
The five convolutional layer groups include one or more convolutional layers, each convolutional layer comprising a Rectified Linear Unit (ReLU) layer and / or a Max Pooling layer. ,
The three hidden layer groups include a fully connected layer, a correction linear unit layer (ReLU layer), a dropout layer, a Softmax layer, and a classification layer. Object detection device.
In the object detection method,
Detecting a foreground area by receiving an input image and performing a difference operation between the input image and the background image;
Generating an object region as a window image in pixel units from the input image with reference to the detected foreground region;
Training a convolutional neural network (CNN) using training data constructed from a plurality of window images;
Transforming the window image into a predetermined size by interpolation, and classifying the modified window image into an object or a shadow using a learned circuit neural network; And
Displaying an object area except the area classified as a shadow on the input image,
Generating an object region as a window image in pixel units from an input image with reference to the detected foreground region,
Equation 1 below to convert the window image to an HSV color model to generate a three-channel window image,

In Equation 1

Is the x, y coordinate

i-th window image,

Each channel of f,

Is the S, V, and

Is an object detection method for meaning a V value of a reference image.
delete The method of claim 6,
Learning a convolutional neural network (CNN) using the training data constructed from the plurality of window images
And a window image included in the training data into object data and shadow data based on a ground truth image, transformed into a predetermined size by interpolation, and used as input of a circuit neural network.
A computer program stored in a computer-readable recording medium for executing the object detecting method of any one of claims 6 and 8.

Images