KR102478335B1 - Image Analysis Method and Server Apparatus for Per-channel Optimization of Object Detection - Google Patents

Abstract

An image analysis method and server device for optimizing object detection for each channel are disclosed.
a plurality of object detection units for each channel based on a neural network trained to detect and recognize an object from an input image for each channel; and a learning data collection unit that collects learning data for each channel by acquiring the input image for each channel and the detection result output from the object detection unit for each of the plurality of channels. Provided is an image analysis server device including a learning unit that individually learns the object detection unit for each of the plurality of channels using the collected learning data.

Description

Image Analysis Method and Server Apparatus for Per-channel Optimization of Object Detection}

The present invention relates to an image analysis method and server device for optimizing object detection for each channel.

The contents described in this part merely provide background information on the present embodiment and do not constitute prior art.

Surveillance systems using CCTV (Closed Circuit Television) and DVR (Digital Video Recorder) are expanding their scope from a technology in which an administrator simply monitors and stores images with the naked eye to an intelligent video surveillance system. An intelligent video surveillance system refers to a system capable of detecting, recognizing, classifying, and tracking an object by analyzing video information input from a camera in real time. The intelligent image detection system is a technology that can maximize the efficiency of post-prevention management and search after determining and analyzing whether an object has generated an event related to security, providing information to a manager or storing data and event contents.

A CCTV video surveillance system adopts a deep learning technology such as a convolutional neural network (CNN) for object detection, recognition, classification, and tracking. In the case of using a convolutional neural network (CNN)-based technology, an object may be detected for each image frame and information on the detected object may be utilized. Since this object detection process requires a large amount of computation, the object detection process after image capture is mainly performed in the server. Therefore, the number of video channels capable of providing a service is limited according to the hardware performance of the server device, and as the number of service video channels increases, a lot of cost is incurred.

In addition, when an object is detected using the learned result, even if the object does not actually exist in the image, the object is frequently incorrectly detected due to factors such as lighting environment, weather change, and user's camera position adjustment. can A false alarm due to false detection hinders accurate recognition of dangerous situations and rapid response. In particular, since similar background images continue to exist in CCTV images, once an erroneous detection occurs, there is a problem in that false alarms continue to occur for similar images.

Embodiments of the present invention are intended to provide an image analysis method and server device that can reduce the operating cost of a server device for object detection while minimizing false detection of an object in a fixed camera environment such as CCTV.

According to an embodiment of the present invention, a plurality of object detection units for each channel based on a neural network learned to detect and recognize an object from an input image for each channel; and a learning data collection unit that collects learning data for each channel by acquiring the input image for each channel and the detection result output from the object detection unit for each of the plurality of channels. Provided is an image analysis server device including a learning unit for individually re-learning the object detection unit for each of the plurality of channels using the collected learning data.

According to an embodiment of the present invention, the learning unit trains a separate neural network based on a 32-bit operation using the training data, performs calibration to optimize the separate neural network for a neural network based on an 8-bit operation, and It is possible to individually re-learn the object detection unit for each channel of .

According to an embodiment of the present invention, the learning data collection unit collects learning data for each channel at a predetermined time period for each channel, and the learning unit collects a predetermined number of learning data for each channel. In this case, the object detection unit for each of the plurality of channels may be individually re-learned.

According to an embodiment of the present invention, the process of obtaining an input image; generating a detection result by an object detector learned to detect and recognize an object from the input image; collecting learning data for each channel by acquiring the input image for each channel and the detection result output from the object detection unit for each of the plurality of channels; and individually re-learning the object detector for each of the plurality of channels using the collected learning data.

As described above, according to the embodiments of the present invention, by periodically acquiring an input image for each channel and performing optimization for object detection, even if the environment in the image changes, the occurrence of false positives can be prevented or minimized. It works.

In addition, by reducing the amount of computation and memory required for object detection, it is possible to provide a larger number of video channel services under limited hardware performance of the server device, thereby reducing costs required for service operation.

1 is a block diagram of a video surveillance system according to an embodiment of the present invention.
2 is a block diagram briefly showing the configuration of a video analysis server device according to an embodiment of the present invention.
3 is a block diagram illustrating a learning method of an object detection unit according to an embodiment of the present invention.
4 is a diagram for explaining an operation process of a learning unit according to an embodiment of the present invention.
5 is a flowchart of an image analysis method according to an embodiment of the present invention.
6 is a flowchart illustrating an example of an image analysis method according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the present invention, '... Terms such as 'unit' and 'module' refer to a unit that processes at least one function or operation, and may be implemented as hardware, software, or a combination of hardware and software.

The present invention intends to use the characteristic of a fixed camera environment such as CCTV, that is, the characteristic that a similar background image is continuously captured with little change in order to minimize false detection of an object. More specifically, we propose a method of minimizing the detection of an object from a background image in which no object exists by retraining (updating) a neural network for object detection and recognition with a similar background image.

1 is a block diagram of a video surveillance system according to an embodiment of the present invention.

The video surveillance system 100 includes an image capture device 110 , a video analysis server device 120 and a monitoring device 130 .

The image capture device 110 generates an image to be input to the image analysis server device 120 in real time. The image capture device 110 may be implemented as a fixed camera such as a CCTV camera, but is not necessarily limited thereto and may include any device capable of generating an input image of the image analysis server device 120. As shown, one or more image capture devices 110 are included in the video surveillance system 100 and are installed in different locations to generate images of various locations.

The image analysis server device 120 analyzes the input image received from the image capture device 110 in real time to detect and recognize an object, and detects one or more predefined events based on the analysis result. The video analysis server device 120 transmits information according to the detected event to the monitoring device 130. The video analysis server device 120 transmits images captured by one or more imaging devices 110 that are matched with each monitoring device 130, and prevents access to images captured by other imaging devices 110. Encrypt and manage

The image analysis server device 120 trains a neural network designed in advance to detect and recognize objects, and detects and recognizes objects in an input image based on the learned neural network. For example, a convolutional neural network (CNN) may be used as a neural network. A convolutional neural network (CNN) includes one or more convolution layers that perform convolution operations on input images and one or more pooling layers that sample outputs of the convolution layers. However, the convolutional neural network (CNN) is only an example and is not necessarily limited thereto, and various other neural networks such as a recurrent neural network (RNN) or a combination of CNN and RNN may be used.

The monitoring device 130 may receive an image captured by the imaging device 110 and display it in real time. In addition, the monitoring device 130 may generate an appropriate event such as an alarm according to information received from the video analysis server device 120 .

2 is a block diagram briefly showing the configuration of a video analysis server device according to an embodiment of the present invention.

The image analysis server device 120 includes an input image acquisition unit 210, an object detection unit 220, a learning data collection unit 230, and a learning unit 240. Each component shown in FIG. 2 may be implemented as a hardware chip or may be implemented as software, and a microprocessor may execute software functions corresponding to each component.

The input image acquisition unit 210 receives an input image from the image capture device 110 and captures it. The input image acquisition unit 210 resizes the captured input image to the size of an input layer of an object detection and recognition neural network. The adjusted input image is transmitted to the object detection unit 220 . Also, the adjusted input image is transmitted to the learning data collection unit 230 for re-learning (update) of the object detection unit 220 .

The object detection unit 220 detects and recognizes an object from an input image based on the learned neural network. The object detection unit 220 derives detection candidates through calculations in several layers (eg, convolution layer, max-pooling layer, etc.) included in the neural network. The object detection unit 220 generates and outputs a final detection result by applying a threshold to detection candidates. The output final detection result is transmitted to the learning data collection unit 230 for re-learning (update) of the object detection unit 220 .

The output detection result includes at least one of object location information, object class information, and object reliability. Here, the location information of the object represents information (eg, location coordinates, width and height of the area) on the area (or box) in which the object is detected. Class information of an object indicates which class (eg, person, vehicle, etc.) the object is classified into. The reliability of an object represents the probability that an object belongs to a finally determined class. For example, if the object does not exist, the ideal reliability value is 0. When a plurality of detection areas exist in one input image, the above detection result may be output for each detection area.

The object detection unit 220 detects and recognizes an object based on an 8-bit arithmetic-based neural network. The object detection unit 220 is a configuration that outputs an object detection result for a new input using a neural network that has been trained on learning data consisting of given input-output, and outputs a result through an inference process of the learned neural network. , At this time, the parameters constituting the learned neural network are expressed in 8 bits, and the calculations necessary for the inference process are also performed in units of 8 bits.

Although not shown in FIG. 2 , the video analysis server device 120 may further include a component for determining a situation based on a detection result. A component (not shown) for determining a situation may detect a situation occurring within the input image using the detected object. For example, this component may determine whether a person is detected when the situation to be detected is a situation in which a person appears, and determine whether a person and a vehicle are simultaneously detected when a situation to be detected is a situation where a person is getting out of a vehicle. there is. In addition, this component may make a situation judgment according to the reliability value of each detected object.

The learning data collection unit 230 obtains an input image and an object detection result from the image input acquisition unit 210 and the object detection unit 220 to collect learning data. The learning data collection unit 230 classifies and obtains input images received from the plurality of imaging devices 110 for each channel, and obtains a detection result output from the object detection unit 220 according to the input images for each channel. to collect learning data for each channel. The collected learning data is used to re-learn the object detection unit 220, and is composed of an input image and a pair of detection results corresponding thereto.

The learning data collection unit 230 may periodically collect learning data. A time period for collecting learning data in the learning data collection unit 230 may be set differently for each channel, and learning data may be collected for a set time or at regular time intervals. In addition, the learning data collection unit 230 may detect a change in the input image and collect learning data whenever a change is detected. Changes in the input image may include various cases, including cases in which only the illuminance changes while the object in the image remains the same as it goes from daytime to nighttime, or cases where background objects move due to rain or wind in the background of the image due to climate change. there is.

In order to optimize the performance of detecting each object for each input image for each channel, the learning data collection unit 230 classifies and collects learning data for each channel so that the object detection unit 220 can learn individually. In addition, the learning data collection unit 230 collects input images that can appear in various ways according to changes in the environment in the image in order to prevent false detection of objects, and learns the object detection unit 220 to learn about the changed environment. Enables proper object detection results to be output.

The learning unit 240 learns the object detection unit 220 using the collected learning data. The learning unit 240 may individually learn the object detection unit 220 for each of a plurality of channels, which operates individually for a plurality of channels, using the learning data collected for each channel. Learning the object detector 220 means updating pre-learned parameters (or weights) between nodes included in the neural network. The learning unit 240 may determine whether a preset number of learning data has been collected as appropriate for learning, or may collect learning data for a set period of time to train the object detection unit 220 .

In the present invention, the input image acquisition unit 210 and the object detection unit 220 are components that perform image analysis to detect an object in the video analysis server device 120, and in FIG. 2, one input image acquisition unit ( 210) and object detection unit 220 are shown, but the same number as the number of image capturing devices 110 is provided in the system to which the present invention is applied, and object detection results for different channels are provided from input images for each channel. work individually to be able to To this end, the object detection unit 220 for each channel detects and recognizes an object based on a neural network having different parameters. In the system to which the present invention is applied, since the imaging device 110 continuously captures the same place in many cases, the object detection unit 220 for each channel learns images of the same place captured at different times, The environment can be learned, and as a result, object detection performance optimized for each channel can be shown.

Hereinafter, a method for learning an object detection unit in a learning unit according to an embodiment of the present invention will be described in detail with reference to FIGS. 3 and 4 . 3 is a block diagram illustrating a learning method of an object detection unit according to an embodiment of the present invention. 4 is a diagram for explaining an operation process of a learning unit according to an embodiment of the present invention.

The learning data collection unit 230 collects learning data by periodically obtaining an input image for each channel and a detection result according to the input image. The learning data collected by the learning data collector 230 includes different data according to each channel, and the object detection unit 220 for each channel assigned to each of the plurality of image capturing devices 110 uses the learning data for each channel. are individually learned, and operate independently of each other using the learned results.

The learning unit 240 is configured to train the object detection unit 220 using learning data, and includes a 32-bit operation unit 310 and an 8-bit operation optimization unit 320. The training of the learning unit 240 is first performed in a neural network based on 32-bit operation, and the object detection unit 220 is trained by performing calibration to optimize it for 8-bit operation. That is, the learning data collected in the present invention is learned based on 32-bit (FP32) operation, and in real-time object detection, inference is made based on 8-bit (INT8) operation.

The 32-bit operation unit 310 uses the collected learning data to learn an input image and a detection result accordingly. The 32-bit arithmetic unit 310 learns training data using a 32-bit based neural network used in conventional neural network calculation.

The 8-bit operation optimizer 320 performs an operation to optimize the 32-bit-based neural network learned in the 32-bit operation unit 310 to the 8-bit-based neural network. If a parameter that was previously expressed in 32 bits is quantized as it is in 8 bits, information loss inevitably occurs, so the 8-bit operation optimizer 320, as shown in FIG. 4, the data range of 32-bit operation By approximating the distribution to the data range of 8-bit operation, calibration is performed to optimize it so that it can operate properly even in an 8-bit based neural network.

The object detection unit 220 outputs a detection result for an input image based on a neural network optimized for 8-bit operation by the 8-bit operation optimization unit 320 . A technique for detecting an object in an image using a neural network detects motion in a specific region of an image and does not require high-precision calculation. Therefore, in the present invention, in order to process input images generated from a plurality of image capturing devices 110 in real time in the image analysis server device 120, an 8-bit arithmetic-based neural network is used, and the server requests for real-time image analysis. It reduces the amount of computation and memory required for computation, and also reduces the time required for computation.

Hereinafter, an image analysis method according to an embodiment of the present invention will be described with reference to FIGS. 5 and 6 . 5 is a flowchart of an image analysis method according to an embodiment of the present invention.

Referring to FIG. 5 , first, an input image is acquired in step S510. Specifically, an input image delivered from an imaging device is captured, and the captured input image is resized to the size of an input layer of an object detection and recognition neural network.

When an input image is obtained, a detection result is generated by an object detector learned to detect and recognize an object from the input image (S520). The object detector derives detection candidates through calculations in several layers (eg, convolution layer, max-pooling layer, etc.) included in the neural network. The object detector generates and outputs a final detection result by applying a threshold to detection candidates.

At this time, the object detector detects and recognizes the object based on a neural network based on 8-bit operation. That is, the object detector outputs the result through the inference process of the learned neural network. At this time, the parameters constituting the learned neural network are expressed in 8 bits, and the calculation required in the inference process is also performed in units of 8 bits.

The output detection result includes at least one of object location information, object class information, and object reliability. Here, the location information of the object represents information (eg, location coordinates, width and height of the area) on the area (or box) in which the object is detected. Class information of an object indicates which class (eg, person, vehicle, etc.) the object is classified into. The reliability of an object represents the probability that an object belongs to a finally determined class. For example, if the object does not exist, the ideal reliability value is 0. When a plurality of detection areas exist in one input image, the above detection result may be output for each detection area.

In step S530, learning data is collected by periodically acquiring the input image for each channel in step S510 and the object detection result in step S520. Specifically, input images received from a plurality of imaging devices are obtained by dividing them into respective channels, and learning data for each channel is collected by obtaining a detection result according to the input images of each channel. The training data is used to train the neural network based on the object detector, and is composed of an input image and a pair of detection results corresponding thereto.

A time period for collecting learning data at this time may be set differently for each channel, and learning data for a set time may be collected or learning data may be collected at regular time intervals. In addition, a change in the input image may be detected, and learning data may be collected whenever a change is detected.

Finally, the object detector is trained using the collected learning data (S540). The learner may individually train a plurality of object detectors that operate individually for a plurality of channels using learning data collected for each channel. Learning the object detector means updating pre-learned parameters (or weights) between nodes included in the neural network. The learner may learn the object detector when a preset number of learning data is collected as it is determined to be appropriate for learning, or the object detector may be trained using the learning data collected for a certain period of time.

The learner learns training data using a separate neural network based on 32-bit operation, and then performs calibration optimized for 8-bit operation to learn the object detector. That is, the learning data collected in this embodiment is learned based on 32-bit operation, and object detection performed in real time is inferred based on 8-bit operation.

A separate neural network for 32-bit arithmetic-based learning learns the input image and its detection result using the collected learning data. Thereafter, an operation to optimize the learned 32-bit-based separate neural network to an 8-bit-based neural network is performed. If a parameter that was previously expressed in 32 bits is quantized into 8 bits, information loss inevitably occurs, so the data range distribution of 32-bit operations is approximated to the data range of 8-bit operations, The object detector is trained by performing calibration that is optimized to operate properly in the neural network.

6 is a flowchart illustrating an example of an image analysis method according to an embodiment of the present invention.

After obtaining an input image in step S610, an object in the input image is detected and recognized to generate a detection result (S620). Then, a situation may be judged using the generated detection result (S630). That is, a situation occurring in the input image can be detected using the detected object. For example, if a situation to be detected is a situation in which a person appears, it may be determined whether a person is detected, and if a situation in which a person is getting out of a vehicle is to be detected, it may be determined whether a person and a vehicle are simultaneously detected. In addition, a situation may be determined according to the reliability value of each detected object.

Meanwhile, in order to determine whether to re-learn the object detector, first, input images and object detection results for each channel are obtained, and learning data according to a set time period is collected (S640). In step S640, learning data is periodically collected. When the number of collected learning data reaches a set value (S650), since the learning data is collected to a sufficient extent for learning, the object detector for each channel is trained (S660).

5 and 6 describe that each process is sequentially executed, but is not necessarily limited thereto. In other words, since the processes described in FIGS. 5 and 6 may be changed and executed or one or more processes may be applied in parallel, FIGS. 5 and 6 are not limited to a time-series sequence.

The image analysis method according to the present embodiment described in FIGS. 5 and 6 may be implemented as a program and recorded on a computer-readable recording medium. A computer-readable recording medium on which a program for implementing the image analysis method according to the present embodiment is recorded includes all kinds of recording devices storing data that can be read by a computing system.

The above description is merely an example of the technical idea of the present embodiment, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights of this embodiment.

100: video surveillance system

Claims (10)

a plurality of object detection units for each channel based on a neural network trained to detect and recognize an object from an input image for each channel; and
A learning data collection unit that collects learning data for each channel by acquiring the input image for each channel and the detection result output from the object detection unit for each of the plurality of channels. consists of pairs of detection results for it;
A learning unit that individually relearns the object detection unit for each of the plurality of channels using the collected learning data.
including,
The learning unit,
Using the learning data, a separate neural network is trained based on 32-bit operation, and calibration is performed to optimize the separate neural network for a neural network based on 8-bit operation, thereby individually re-learning the object detection unit for each of the plurality of channels. Image analysis server device characterized by. delete According to claim 1,
The learning unit,
Approximating the data distribution range of the separate neural network to an 8-bit data range and optimizing the 8-bit operation-based neural network. According to claim 1,
The learning data collection unit,
Collecting learning data for each channel at a predetermined time period for each channel;
The learning unit,
When a predetermined number of learning data for each channel is collected, the object detection unit for each of the plurality of channels is individually re-learned. According to claim 1,
The learning data collection unit,
The video analysis server device, characterized in that for detecting a change in the input image for each channel, and collecting learning data for each channel by obtaining the input image for each channel and the detection result whenever a change is detected. Acquiring an input image for each channel;
a process of generating a detection result by a plurality of object detectors for each channel learned to detect and recognize an object from an input image for each channel;
Acquiring the input image for each channel and the detection results output from the object detectors for each of the plurality of channels to collect learning data for each channel, the learning data for a given channel is the input image of the corresponding channel and the detection result for it Consists of pairs of; and
Process of individually re-learning the object detector for each of the plurality of channels using the collected learning data
including,
The process of individually learning the object detector,
A process of individually retraining the object detector for each of the plurality of channels by training a separate neural network based on a 32-bit operation using the learning data and performing calibration to optimize the separate neural network for a neural network based on an 8-bit operation Image analysis method comprising a. delete According to claim 6,
The process of individually relearning the object detector,
The image analysis method characterized in that the data distribution range of the separate neural network is approximated to an 8-bit data range and optimized for the 8-bit operation-based neural network. According to claim 6,
The process of collecting the learning data,
Collecting learning data for each channel at a predetermined time period for each channel;
The process of individually learning the object detector,
When a predetermined number of learning data for each channel is collected, individually re-learning the object detector for each of the plurality of channels. According to claim 6,
The process of collecting the learning data,
and detecting a change in the input image for each channel, acquiring the input image for each channel and the detection result whenever a change is detected, and collecting learning data for each channel.

Images