KR102514301B1 - Device for identifying the situaton of object's conduct using sensor fusion - Google Patents

Abstract

The present invention relates to a behavioral analysis technology using heterogeneous sensor fusion, and more particularly, to improve the accuracy of object detection by fusing object information in an RGB image with object information in a thermal image, and a portion of a detected object rather than the entire image. The present invention relates to a behavioral analysis device capable of time-sequentially analyzing an image to determine what a situation an object's behavior means with a relatively lightweight system. An apparatus for analyzing behavior using heterogeneous sensor fusion according to an exemplary embodiment fuses first feature information extracted from an RGB image of a first sensor and second feature information extracted from a thermal image of a second sensor to obtain an object. It includes an object detection unit that detects and a situation determination unit that analyzes partial images of the detected object in time series to determine the situation of the object.

Description

Behavioral analysis device using heterogeneous sensor fusion {DEVICE FOR IDENTIFYING THE SITUATON OF OBJECT'S CONDUCT USING SENSOR FUSION}

The present invention relates to a behavioral analysis technology using heterogeneous sensor fusion, and more particularly, to improve the accuracy of object detection by fusing object information in an RGB image with object information in a thermal image, and a portion of a detected object rather than the entire image. The present invention relates to a behavioral analysis device capable of time-sequentially analyzing an image to determine what a situation an object's behavior means with a relatively lightweight system.

BACKGROUND ART As the range of human activity is widened, lifesaving activities in accidents caused by fire, plane crash, fishing boat sinking, etc. are required in various fields. In particular, in the case of an accident at sea, the accident vessel and the survivors quickly leave the search range due to the influence of sea and current, so unlike on land, information on a wide range of accident areas must be quickly and accurately collected.

However, with the current rescue guidelines and technologies, the observer directly observes the video of the accident site and passively collects situational information necessary for lifesaving. In addition, maritime accidents that occur at night lead to the greatest loss of life, and in situations where the characteristics of the night environment where it is difficult to secure visibility are taken into consideration, it takes a lot of manpower and time because it relies on simply illuminating lights or visual search using flares. It is becoming.

Demand for various monitoring services that recognize abnormal situations by analyzing images, such as detection of intruders in the field of security and monitoring of abnormal behavior of specific animals in zoos, as well as detection of victims at sea are increasing.

Object detection based on image processing techniques in the traditional computer vision field requires very heavy hardware specifications for image processing because it is a method of identifying objects by processing the entire image. However, there is a problem in that it is difficult to apply traditional image processing techniques to mobile navigation devices such as drones due to limited power and hardware specifications.

In order to overcome this problem, a technique for detecting an object in a relatively lightweight method such as CNN has been proposed, but CNN only detects the object itself for a single frame of still image, and does not know what the object is doing or what the object is doing. You can't judge what situation you're in.

In addition, in the existing object detection technique, a person directly defines features for object detection, and usually defines color, size, and shape as representative features of an object, and detects the object by extracting the features from a captured image. way.

However, existing object detection techniques have limitations in that new features must be newly extracted every time the object features change, and it is difficult to generalize specific features of the object. This inevitably lowers the reliability of object detection performance.

Therefore, it is required to develop a technology that can increase the reliability of object detection and analyze the behavior of the detected object to determine the object's situation.

One problem to be solved by the present invention is to provide an active and intelligent object detection technology that overcomes the passivity and low reliability of the conventional image processing-based object detection technology.

Another problem to be solved by the present invention is to provide a technique for identifying an object's behavior pattern with relatively light system resources by analyzing a partial image including the detected object instead of the entire image.

Another problem to be solved by the present invention is to provide a technology for supplementing functional limitations of each sensor by fusing an RGB sensor and a thermal image sensor.

In order to solve the above problem, the present invention detects an object by fusion of first feature information extracted from an RGB image of a first sensor and second feature information extracted from a thermal image of a second sensor. an object detecting unit; and a situation determining unit configured to analyze a partial image of the detected object in a time-sequential manner and determine a situation of the object.

The behavior analysis device may further include a pre-processing unit that performs viewing angle correction on the RGB image and the thermal image before inputting the RGB image and the thermal image to the object detection unit, and performs edge correction on the thermal image. there is.

The object detection unit may include: a first detection module extracting first feature information including a bounding box (bb _c ) and a reliability score (S _C ) of the object from the RGB image; a second detection module extracting second feature information including a bounding box (bb _t ) and a reliability score (S _T ) of the object from the thermal image; a sensor fusion module that resets reliability scores (S _C , S _T ) for each image so that the outputs of the first detection module and the second detection module have a more meaningful correlation with an actual correct answer (ground-truth); and a final detection module for deriving final feature information based on the reset reliability scores (S _C ', S _T ').

Here, the sensor fusion module uses the first feature information and the second feature information to determine reliability scores for each image (S _C , S _T ), bounding boxes for each image (bb _c , bb _t ), and bounding boxes for each image At least among the average values of the center coordinates (μ _x , μ _y ), the variance values (σ _x , σ _y ) of the center coordinates of the bounding boxes for each image, and the minimum bounding box (bb _m ) including all the bounding boxes for each image An extraction module for deriving fusion feature information (F) including one; And a multi-layer perceptron whose target value is set to the relative width (Intersection Of Union; IOU) of the intersection of the boundary boxes (bb _c , bb _t ) for each image and the bounding box (bb _g ) of the actual correct answer (ground-truth) ( It may include a learning module that inputs the fusion feature information (F) to a multi-layer perceptron.

In addition, the situation determination unit, a calculation module for deriving an output value at time t by reflecting the feature information of the object detected at time t -1 to the feature information of the object detected at time t (t is an integer greater than or equal to 1) as a weight; and an analysis module configured to determine a situation of the object by analyzing a sequence of output values of the calculation module from a time point at which the object is detected to a time point set in advance.

The calculation module includes an input layer (x), a hidden layer (h), and an output layer (y), and a circular neural network to which a circular edge is added to the hidden layer to transfer detection information of an object at time t-1 to time t ( recurrent neural networks) can be used.

In addition, the calculation module may use a long short term memory (LSTM) network in which a memory block for storing parameters is added to the recurrent neural network.

The behavior analysis apparatus of this embodiment includes a communication module for transmitting the determined situation to a remote control terminal through a wireless network; A movement module for moving the behavior analysis device to a destination; And it may further include at least one of a GPS module for determining the location of the behavior analysis device.

According to the embodiments of the present invention, since intelligent and active object detection by deep learning is possible, whenever object features are updated in the conventional image processing-based object detection technology, it is cumbersome to newly input feature information every time and object It is possible to overcome the limitation that it is difficult to generalize the specific characteristics of

According to the embodiments of the present invention, since it is not necessary to process the entire image to analyze the behavior of an object, it is possible to grasp the behavior pattern of an object with sufficient quality even in a mobile device having limited resources, such as a drone.

According to embodiments of the present invention, reliable object detection and behavioral analysis are possible by supplementing the detection performance deterioration when the amount of external light is insufficient or excessive, which is a limitation of the RGB sensor, with a thermal image sensor.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a configuration diagram of a behavior analysis system according to a first embodiment and a behavior analysis device included in the system.
2 is a conceptual diagram illustrating the operating principle of a CNN.
3 is a block diagram illustrating an operation process of an object detection unit.
4 is a conceptual diagram illustrating in detail the operation principle of the sensor fusion module.
5 is a conceptual diagram explaining the operation algorithm of RNN.
6 is a conceptual diagram illustrating an operating algorithm of a situation determination unit.
FIG. 7 is a conceptual diagram intuitively illustrating the principle of the algorithm of FIG. 6 .
8 is a configuration diagram of a behavior analysis system and a behavior analysis device included in the system according to the second embodiment.

Since the present invention can apply various transformations and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, it should be understood that this is not intended to limit the present invention to specific embodiments, and includes all transformations, equivalents, and substitutes included in the spirit and scope of the present invention.

In describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings in the context of the related art, and unless explicitly defined in this application, they should not be interpreted in ideal or excessively formal meanings. don't

The term "MODULE" described in this specification refers to a unit that processes a specific function or operation, and may mean hardware or software or a combination of hardware and software.

In addition, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

In the present invention, an "object" refers to an analysis target included in an RGB image or a thermal image, and may be defined in various ways according to setting conditions such as a person, a vehicle, and a pet. In this specification, the object is assumed to be a person and described, but is not necessarily limited thereto.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. description is omitted.

<Example 1>

Embodiment 1 relates to a behavior analysis device that autonomously performs image capturing, object detection, and situation determination.

1 is a configuration diagram of a behavior analysis system according to a first embodiment and a behavior analysis device included in the system.

As shown in FIG. 1 , the behavior analysis system of the first embodiment includes a behavior analysis device 1 and a remote control terminal 2 .

The behavior analysis device 1 includes a first sensor 10 that captures an RGB image, a second sensor 20 that captures a thermal image, first feature information extracted from the RGB image of the first sensor 10, and first sensor 10 that captures a thermal image. The object detection unit 200 detects an object by fusion of the second feature information extracted from the thermal image of the second sensor 20 and time-sequentially analyzes the partial image of the detected object to determine the location of the object. It includes a situation determination unit 300 that determines the situation, and may further include a pre-processing unit 100 that corrects the RGB image and the thermal image so that object detection is easy.

For better understanding, it is assumed that in the system of Example 1, the behavior analysis device 1 is implemented as a drone for searching for victims of a marine accident, and the control terminal 2 is implemented as a mobile communication terminal connected to the drone through a wireless communication network. do.

In this case, the behavior analysis device 1, the GPS module 30 for determining its location, the situation determined by the situation determination unit 300 and / or the location information determined by the GPS module 30 through a wireless network It may further include a communication module 40 for transmitting to a control terminal in a remote place, and a movement module 50 for moving the behavior analysis device to a destination.

The moving module 50 is implemented as various means of movement such as wheels, caterpillars, propellers, jet propellers, quadcopter fans, or the like. When implemented as a flying drone as in the above assumption, a quadcopter fan may be applied.

The pre-processing unit 100 performs viewing angle correction on the RGB and thermal images before inputting the RGB and thermal images to the object detection unit 200, and performs edge correction on the thermal images. Either one of the viewing angle correction and the edge correction may be selectively performed.

The first sensor 10 and the second sensor 20 generate an image by projecting a 3D object into a 2D image. Since the positions where each sensor is installed are different, even if each sensor looks at the same object, it is photographed according to the viewing angle. A difference occurs in the image. Since sensor fusion, which will be described later, is easy when the positional information of the RGB image and the thermal image of the same scene are identical, the positional information of the two images needs to be matched. Accordingly, the pre-processing unit 100 matches the viewing angles of the plurality of thermal static images collected from the thermal imaging camera and the plurality of color static images collected from the color camera.

In addition, since the first sensor 10 acquires an RGB image displayed with a myriad of pixels and colors, the second sensor 20 obtains a gray scale thermal image. The edge or outline of an object cannot be accurately expressed. Accordingly, the preprocessor 100 corrects the edge of the object by applying an edge filter to increase the visual contrast of the thermal image. Examples of the edge filter include a Sobel edge detection filter, a Canny edge detection filter, and the like, and other than these, any filter having the same function can be applied.

Hereinafter, the object detection unit 200 and the situation determination unit 300 will be described in detail.

The object detection unit 200 may use a Convolution Neural Network (CNN) to detect an object from an RGB image and a thermal image.

2 is a conceptual diagram illustrating the operating principle of a CNN.

As shown in FIG. 2, CNN is a network that extracts features from input image data through a convolution operation and detects objects included in the image data through the extracted features. A feature extraction layer that extracts feature information of the object ( It consists of feature extraction layers and fully connected layers that classify objects using extracted feature information. In other words, CNN can be defined as one of object detection technologies that derives (localizes) geometric information such as the size or position of a specific object included in image data and classifies the specific object.

The feature extraction layer is composed of a plurality of convolution layers, and the convolution layer at one stage is a new feature map (feature map) map) is generated, and the generated feature map is input to the convolution layer in the next step. Each convolution layer is composed of weights and biases, which are parameters whose values change according to learning, and the value of the synthetic kernel is automatically adjusted during the learning process to extract useful information for object detection, such as color, shape, and contour.

The entire connection layer consists of a flatten layer that converts the multi-dimensional feature map extracted from the feature extraction layer into a one-dimensional one and a multi-layer perceptron (MLP). Parameters of image data are adjusted by inputting the one-dimensionally converted feature map to the learning algorithm, MLP.

The object detection unit 200 performs a CNN feature extraction layer for each of the RGB image and the thermal image, and performs a full connection layer by inputting the sensor fusion-applied flattening layer to the multi-layer perceptron to perform the feature of each image. After re-evaluating the reliability of the feature, the object is finally detected as a highly reliable feature.

3 is a block diagram illustrating an operation process of the object detection unit 200.

As shown in FIG. 3 , the object detection unit 200 includes a first detection module 210 , a second detection module 220 , a sensor fusion module 230 and a final detection module 240 .

The first detection module 210 detects an object from the RGB image and outputs feature information of the object. The second detection module 220 detects an object from the thermal image and outputs feature information of the object. That is, it can be understood that the first detection module 210 and the second detection module 220 perform the feature extraction layer of the CNN described above for the RGB image and the thermal image, respectively.

CNN's feature extraction layer is divided into a single-stage detector and a two-stage detector according to the number of stages for object detection. The ‘Two-stage detector’ consists of two steps: proposing a region of interest (ROI) where an object is likely to exist inside the image data before detecting an object, and detecting the object within that region. On the other hand, in the ‘Single-stage detector’, the process of finding the position of an object included in the image data and the process of detecting the object are performed simultaneously.

Examples of single-step detectors include YOLO v1, v2, v3, SSD, DSSD, DSOD, RetinaNet, RefineDet, and M2Det, and examples of two-step detectors include R-CNN, Fast R-CNN, Faster R-CNN, R- FCN, Mask R-CNN, etc. may be mentioned.

Hereinafter, an example in which the first detection module 210 and the second detection module 220 use the real-time object detection algorithm YOLO (You Only Look Once) among single-stage detectors will be described. Other algorithms can easily be applied as substitutes. The first detection module 210 can be used interchangeably with the name YOLO-C in the sense that it targets RGB images, and the second detection module 220 is used interchangeably with the name YOLO-T in the sense that it targets thermal images It can be.

YOLO divides each image into S × S grids and calculates the reliability of the grids. Confidence reflects accuracy in recognizing objects within the grid. Initially, a bounding box that is far from object recognition is set, but by calculating the reliability and adjusting the position of the bounding box, a bounding box with the highest object recognition accuracy can be obtained. To calculate whether an object is included in the grid, an object class score is calculated. As a result, a total of S x S x N objects is predicted. Most of the grids have low reliability, and in order to increase the reliability, unnecessary parts are removed by setting a threshold after merging neighboring grids.

The sensor fusion module 230 fuses the detection results of the first detection module 210 and the second detection module 220 to increase the accuracy of object detection or enables object detection even if an object is not detected by either sensor. let it do

Since the first sensor 10 that captures an RGB image expresses visible light reflected from an object as high-resolution image data and represents image data similar to human vision, the shape and color of an object to be detected among various recognition sensors are best appear

However, because it is vulnerable to the external environment, the object to be detected does not appear well in the image data when the amount of light is insufficient or when the amount of light is suddenly input excessively. For example, the detection performance of the RGB image-based object detection system is rapidly degraded when the light intensity is insufficient at night or when the direct sunlight is greatly affected by external light.

In contrast, the second sensor 20 that captures a thermal image converts radiant energy emitted by an object according to infrared rays into temperature and displays it as image data. Since the second sensor 20 is less affected by the external environment, it expresses the object to be detected well even in a situation where the amount of light suddenly becomes excessive. However, since the resolution of image data is low, object detection tends to be less accurate.

The sensor fusion module 230 fuses the detection results based on the bounding boxes and reliability scores output from YOLO-C and YOLO-T to mutually complement the strengths and weaknesses of each sensor, thereby enabling high reliability and detection performance.

4 is a conceptual diagram illustrating in detail the operating principle of the sensor fusion module 230 .

As shown in FIGS. 3 and 4 , the sensor fusion module 230 includes an extraction module 231 partially corresponding to the CNN flattening layer and a learning module 232 corresponding to the MLP.

The extraction module 231 selects reliability scores (S _C , S _T ) and bounding boxes (bb _c , bb _t ) from among the feature information output from the first detection module 210 and the second detection module 220, Calculate the average value (μ _x , μ _y ) of the central coordinates of the two bounding boxes and the variance values (σ _x , σ _y ) of the central coordinates of the two bounding boxes, and include both bounding boxes (bb _c , bb _t ). Calculate the minimum bounding box (bb _m ). where the bounding box (bb _c , bb _t , As described above, bb _m ) refers to information including center coordinates, width, and height of a rectangular area including an object.

Reliability scores (S _C , S _T ), bounding boxes (bb _c , bb _t ), mean values of central coordinates (μ _x , μ _y ), and variance values of central coordinates (σ _x , σ _y ) , When the minimum bounding box (bb _m ) is defined as input data, the input data may be expressed as fusion feature information (F) input to the MLP by the following equation.

[Equation 1]

F = (S _C , S _T , bb _c , bb _t , μ _x , μ _y , σ _x , σ _y , bb _m )

The learning module 232 _is a multi-dimensional non-linear mapping ₍ for example, _multi -layer perceptron, MLP). Through MLP learning, the confidence score is reset so that the outputs of YOLO-C and YOLO-T have a more meaningful correlation with the actual correct answer.

Specifically, the learning module 232 calculates the target value of the MLP as the relative area of the intersection of the bounding boxes (bb _c , bb _t ) output from YOLO-C and YOLO-T and the bounding box of the actual correct answer (bb _g ) (Intersection Of Union; IOU), it learns to have a higher correlation with the actual correct answer than the existing reliability score.

The final detection module 240 uses the two bounding boxes (bb _c , bb _t ) output from the sensor fusion module 130 and the confidence level ( _SC ', S _T ') updated by the MLP to determine the result according to a predetermined rule. The final economy box and reliability score (bb _f , S _F ) are derived according to

In the above description, the object detection unit 200 performed optimal object detection using an algorithm modified from CNN. CNN can extract objects with high accuracy, but there is a difficulty in running a large number of images with CNN. Accordingly, in another embodiment, the object detection unit 200 may reduce the number of bounding boxes required for object classification through the R-CNN algorithm considering a region.

Next, the operation of the situation determination unit 300 will be described in detail.

The situation determination unit 300 determines the situation of the object by analyzing the image of the object detected by the object detection unit 200, that is, the image of a specific area including the object.

When a large number of objects, for example, a large number of people are included in an image captured by a drone, a large amount of information processing resources are required by traditional image processing, but the limited power and information processing resources of a drone cannot handle it. Therefore, the present invention proposes a two-step analysis technique that first finds a person with an object detection network such as CNN and then analyzes only the images of a specific area that includes the person in time series to determine what the person's behavior means. do.

In particular, it is difficult to judge a person's behavior with a single frame still image. For example, in a drone system that detects a person in distress, even if a person floating on the sea is detected using an object detection network such as CNN, It is difficult to determine whether the person is swimming normally in a safe situation or requesting rescue in a distress situation with only a single still image of the person waving. Therefore, a person is first detected through an object detection network, and the sequence of a person's action is analyzed for several consecutive image frames in time series rather than a single image frame in the detection area. judge the type of movement.

When several image frames that are successive in time series are defined as time series data, the time series data is dynamic data having a variable length and changing characteristics according to time. In particular, in order to analyze time series data for a long period of time, it is necessary to identify the sequence of data from the point at which the data was acquired to the last point in time. There are limitations in understanding .

Therefore, in the present invention, a Recurrent Neural Network (RNN) is used as an example algorithm for effectively analyzing several frames of a video of an object detected by a CNN. RNN determines the situation of an object, that is, the situation of an object by analyzing sequentially input time series data.

To this end, the situation determination unit 300 reflects the feature information of the object detected at time t -1 to the feature information of the object detected at time t (t is an integer greater than or equal to 1) as a weight to derive an output value at time t It includes a calculation module (not shown) and an analysis module (not shown) that analyzes a sequence of output values of the calculation module (not shown) from the time the object is detected to a preset time to determine the situation of the object. .

5 is a conceptual diagram explaining the operation algorithm of RNN.

As shown in FIG. 5, RNN is a combination of MLP and cyclic edge ( W ), and receives past output values through cyclic edges when time-series data is input. Therefore, given that the output value of the RNN at time t is affected by the output value at the previous time point t-1, it can be defined as a network of feedback structures including 'memory' information that remembers past values.

6 is a conceptual diagram illustrating an operating algorithm of the situation determination unit 300, and FIG. 7 is a conceptual diagram intuitively illustrating the principle of the algorithm of FIG.

As shown in FIG. 6, unlike the typical RNN of FIG. 5 that applies a circular edge to one piece of data (for example, a single still frame), the RNN of the situation decision unit 300 uses time-sequentially arranged data (for example, It is a so-called "unfolded RNN" in which the feedback structure of the RNN is recurrent for a plurality of consecutive video frames).

The RNN of the calculation module (not shown) plays a role of finding parameters that predict an output value close to the target value from given input data such as MLP, receives new data according to the flow of time, and outputs the output value through the weight at the previous time. Calculate

As shown in Figure 6, the RNN consists of an input layer (x), a hidden layer (h), and an output layer (y), and circular edges are added to the hidden layer. The mild edge (W) serves to transfer the information of time t-1 to time t.

According to time, X=(x ^t=1 , x ^t=2 , ..., x ^t=T ) are sequentially input, and the hidden layer value is calculated by reflecting the weight U to the input data X. The hidden layer value h ^(t) at time t is calculated according to the input value x ^(t) and the hidden layer value h ^{(t-1) at} the immediately preceding time point t-1, and Y=(y ^t=1 , depending on time y ^t=2 , ..., y ^t=T ) are output. If such a process is expressed as a mathematical expression, it is as shown in Equation 2.

[Equation 2]

h ^(t) = f(h ^(t-1) * W+x ^(t) * U) (if t>=2)

h ^(t) = f(x ^t * U) (for t=1)

Here, f(x) is an activation function.

The analysis module analyzes a sequence of output values of a calculation module (not shown) from a time point when the object is detected to a time point set in advance to determine the state of the object.

On the other hand, when time-series data is processed by RNN, if the time point at which the input data is acquired is long, the 'gradient vanishing' problem in which parameters disappear may occur. Accordingly, in another embodiment, the calculation module (not shown) of the situation determination unit 300 may use Long Short Term Memory (LSTM) processing long time series data by adding a memory block to the RNN.

Since the value of the hidden layer at time t is affected by the value of the hidden layer at the previous time point t-1, LSTM calculates the final output value y ^(T) by figuring out the context of the input data until the end of the dynamic image time point. , the context of the object is judged through this process of understanding the context.

<Example 2>

Embodiment 2 relates to a case in which a photographing terminal is in charge of capturing and transmitting images, and a behavior analysis server located in a remote location is in charge of detecting an object and determining a situation.

8 is a configuration diagram of a behavior analysis system and a behavior analysis device included in the system according to the second embodiment.

The behavior analysis system of the second embodiment includes a photographing terminal 3 and a behavior analysis server 4 .

For better understanding, in the system of Example 2, the behavior analysis device 3 is implemented as a CCTV that captures customer behavior in a department store, and the behavior analysis server 4 is a security server connected to the CCTV and a wireless or wired communication network. Assume it is implemented.

The photographing terminal 3 includes a first sensor 11 that captures an RGB image of a store situation, a second sensor 21 that captures a thermal image of the same store situation, and behavioral analysis of the two captured images through a wired or wireless network. It includes a communication module 41 for transmission to the server 4. The configurations of the first sensor 11 and the second sensor 21 are the same as those of the first sensor 10 and the second sensor 20 of the first embodiment described above.

The behavior analysis server 4 fuses the first feature information extracted from the RGB image of the first sensor 11 and the second feature information extracted from the thermal image of the second sensor 21 to detect the object ( It includes an object detection unit 201 to detect and a situation determination unit 301 to determine the situation of the object by analyzing the time-sequential images of the detected object, and RGB images and thermal images are provided to facilitate object detection. A pre-processing unit 101 for correcting and a communication module 42 for communicating with the photographing terminal 3 may be further included. Here, the configuration of the pre-processing unit 101, the object detection unit 201, and the situation determination unit 301 is the same as the pre-processing unit 100, the object detection unit 200, and the situation determination unit 300 of the first embodiment described above. do.

In the above-described embodiment, the case where the behavior analysis device is mainly used for lifesaving and search has been described, but it is not necessarily limited thereto.

For example, after classifying specific customers by age and gender at an offline clothing store, it is possible to identify which products a woman in her 30s wears the most and purchases the most through behavioral analysis. In addition, through CCTV installed in daycare centers, it is possible to analyze which classes a particular child is active in and what behavioral characteristics appear by age. The behavioral classification data analyzed in this way can be used in various fields, such as identifying buyers' tastes in the process of establishing a product launch plan.

All or partial functions of the behavior analysis device or behavior analysis system described above are implemented as software that translates a series of program commands, data files, data structures, etc. into machine language alone or in combination, or such software can be read through a computer. Those skilled in the art will be able to easily understand that it can be included in a recording medium that can be provided. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and floptical disks. Included are hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, USB memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like, in addition to machine language codes such as those produced by a compiler. The hardware device may be configured to operate as one or more software modules to perform the operations of the present invention.

In addition, although the above has been described with reference to several embodiments related to the present invention, those skilled in the art can within the scope not departing from the spirit and scope of the present invention described in the claims below. It will be appreciated that it can be modified and changed in various ways.

1: behavior analysis device 2: control terminal
3: shooting terminal 4: behavior analysis server
10, 11: first sensor 20, 21: second sensor
30: GPS module 40, 41: communication module
50: movement module
100: pre-processing unit 200: object detection unit
210: first detection module 220: second detection module
230: sensor fusion module 240: final detection module
300: situation judgment unit

Claims (10)

an object detection unit that detects a human object by fusing first feature information extracted from an RGB image of a first sensor with second feature information extracted from a thermal image of a second sensor; and
A situation determination unit configured to time-sequentially analyze partial images of the detected object to determine whether the action of the human being is an action in a distress situation;
The situation judgment department,
An operation module for deriving an output value at time t by reflecting the feature information of the object detected at time t -1 as a weight to the feature information of the object detected at time t (t is an integer greater than or equal to 1);
Characterized in that it comprises an analysis module for determining whether the action of the human being is an action in a distress situation by analyzing a sequence of output values of the calculation module from the time when the object is detected to the time set in advance.
Behavioral analysis device using heterogeneous sensor fusion. According to claim 1,
A pre-processing unit that performs viewing angle correction on the RGB and thermal images before inputting the RGB and thermal images to the object detection unit, and performs edge correction on the thermal images.
Behavior analysis device using heterogeneous sensor fusion further comprising a. According to claim 1,
The object detection unit,
A first detection module extracting first feature information including a bounding box (bb _c ) and a reliability score (S _C ) of the object from the RGB image;
a second detection module extracting second feature information including a bounding box (bb _t ) and a reliability score (S _T ) of the object from the thermal image;
a sensor fusion module that resets reliability scores (S _C , S _T ) for each image so that the outputs of the first detection module and the second detection module have a correlation with an actual correct answer (ground-truth); and
A final detection module for deriving final feature information based on the reset reliability scores (S _C ', S _T ')
Behavior analysis device using heterogeneous sensor fusion, characterized in that it comprises a. According to claim 3,
The sensor fusion module,
Using the first feature information and the second feature information, the reliability score (S _C , S _T ) for each image, the bounding box (bb _c , bb _t ) for each image, and the average value of the center coordinates of the bounding box for each image (μ _x , μ _y ), variance values (σ _x , σ _y ) of the central coordinates of the bounding boxes for each image, and fusion feature information including at least one of a minimum bounding box (bb _m ) including all bounding boxes for each image (F) an extraction module that derives; and
A multi-layer perceptron (multi-layer perceptron) whose target value is set to the relative width (Intersection Of Union; IOU) of the intersection area of the bounding box (bb _c , bb _t ) for each image and the bounding box (bb _g ) of the actual correct answer (ground-truth). Learning module for inputting the convergence feature information (F) to the -layer perceptron)
Behavior analysis device using heterogeneous sensor fusion, characterized in that it comprises a. According to claim 3 or 4,
The first detection module and the second detection module use a You Only Look Once (YOLO) algorithm for object detection. delete According to claim 1,
The calculation module,
It includes an input layer (x), a hidden layer (h), and an output layer (y), and in the hidden layer, a recurrent neural network to which a circular edge is added that transmits the detection information of the object at time t-1 to time t Behavior analysis device using heterogeneous sensor fusion, characterized in that for use. According to claim 7,
The calculation module,
Behavior analysis device using heterogeneous sensor fusion, characterized in that using a long short term memory (LSTM) network in which a memory block for storing parameters is added to the recurrent neural network. According to claim 1,
Behavior analysis device using heterogeneous sensor fusion further comprising a communication module for transmitting the determined situation to a remote control terminal through a wireless network. According to claim 9,
A movement module for moving the behavior analysis device to a destination; and
GPS module for determining the location of the behavior analysis device
Behavior analysis device using heterogeneous sensor fusion further comprising a.

Images