KR101077967B1 - Apparatus and method for surveillance and tracking - Google Patents

Abstract

Mobile monitoring and tracking devices and methods are disclosed. The position estimator estimates in real time the current position of the mobile robot that changes in accordance with the movement in the surveillance region by wireless communication with a plurality of beacons located in the surveillance region. The imaging unit photographs a preset photographing area at the current location to generate a first image frame. The image input unit sequentially receives a plurality of second image frames photographed by a fixed camera installed in the surveillance area through a network. The object position determiner detects the moving objects included in the first image frame and the second image frame and tracks the movement of the moving object by the object tracking filter. When the moving object is detected from the second image frame, the moving object is the last. The estimated position of the moving object is determined based on the position information of the moving object in the detected second image frame. The path generation unit determines a tracking path from the current position to the expected position of the moving object. The driving unit moves the mobile robot to the expected position along the determined tracking path, and outputs a control command for moving the mobile robot according to the movement of the moving object detected from the first image frame. According to the present invention, even when the surveillance area is large and there are many obstacles, intrusion detection can be efficiently performed without using blind spots with fewer cameras than when performing unmanned surveillance using a plurality of stationary cameras.

Description

Mobile Apparatus and Method for Surveillance and Tracking

The present invention relates to a mobile monitoring and tracking device and method, and more particularly, to a device and method for allowing a mobile robot to recognize its location and to map the surrounding area to track the objects appearing in the surveillance area It is about.

As the number of crimes and accidents increases, the importance of security is increasing in public facilities, businesses and homes. As a result, an effective security system needs to be established. Currently, most security companies detect an intrusion using a human body sensor (PIR) or a magnetic sensor, and transmit data to a server through a public switch telephone network (PSTN) network. have. Such a system is not only vulnerable to user's negligence or malfunction of the sensor, but also has a vulnerability as a security system because it does not have information about an intruder. In addition, security systems such as unmanned security systems and CCTV surveillance cameras are exposed to problems such as intrusion detection failure due to negligence in management because they rely on human concentration and judgment.

Compared to conventional CCTV, the multimedia unmanned surveillance system is inferior to manual monitoring by humans, and it is possible to efficiently use and manage the recording media in that it can record high quality images only at the same time as tracking. It has the advantage of being possible, and it has the advantage of enabling the prevention of errors and negligence of monitoring personnel by manual labor and further reducing labor.

In particular, an intelligent surveillance system using a network camera and a mobile robot can obtain a valid image not only in the place where the camera is not installed but also in the blind spot of the camera by using the mobility of the autonomous mobile robot. In addition, since a large number of cameras can be used to monitor a large area, it is an effective system that combines the reliability and economy of the system.

In order to effectively implement such an intelligent monitoring system, a mobile robot must recognize a location, detect an obstacle, and move to a destination. In order for the mobile robot to recognize its location and form information about the environment without any information about the surrounding environment, localization and mapping processes must be performed at the same time. This is called Simulaneous Localization And Mapping (SLAM).

Currently, most SLAM algorithms have been implemented based on distance sensors such as laser sensors or ultrasonic sensors. In the case of using a laser sensor, relatively accurate positional information about the surrounding environment can be obtained, which shows good performance, but it is difficult to be practical due to the expensive equipment. However, in the case of using an ultrasonic sensor, although there is an advantage in terms of price, a post-processing process for processing inaccurate distance information is required.

For this reason, recently, studies on implementing a SLAM algorithm of a mobile robot using a vision sensor have been conducted. When the repeatability and reliability of visual feature points extracted from images are low, the robustness of the SLAM algorithm is reduced, and the computational efficiency is large. There is a problem of lowering.

To solve this problem, an algorithm is needed to accurately recognize the location of the mobile robot and to map the area around it, effectively tracking the objects detected in the surveillance area.

The technical problem to be achieved by the present invention is to provide the information necessary for the movement by detecting the object displayed in the surveillance area to quickly move to the location of the object by recognizing its location and obtaining distance information of the surrounding area to create a map. To provide a mobile monitoring and tracking device and method.

Another technical problem to be achieved by the present invention is to recognize the location of the object so that it can quickly move to the location of the object detected in the surveillance area, obtain the distance information of the surrounding area to create a map to obtain the information necessary for the movement The present invention provides a computer-readable recording medium having recorded thereon a program for executing a portable monitoring and tracking apparatus and method provided by a computer.

In order to achieve the above technical problem, the mobile monitoring and tracking device according to the present invention is provided in a mobile robot moving in a surveillance area to perform unmanned surveillance and wireless communication with a plurality of beacons located in the surveillance area. A position estimating unit for estimating a current position of the mobile robot that changes in accordance with movement in the surveillance region by a real time; An imaging unit for photographing a preset photographing area at the current position to generate a first image frame; An image input unit which sequentially receives a plurality of second image frames photographed by a fixed camera installed in the surveillance area through a network; The moving object included in the first image frame and the second image frame is detected to track the movement of the moving object by an object tracking filter. When the moving object is detected from the second image frame, the moving object is detected. An object position determiner which determines an expected position of the moving object based on position information of the moving object in the second detected image frame; A path generation unit determining a tracking path from the current location to an expected location of the moving object; And a driving unit which moves the mobile robot to the expected position along the determined tracking path and outputs a control command for moving the mobile robot according to the movement of the mobile object detected from the first image frame.

In order to achieve the above technical problem, the mobile monitoring and tracking method according to the present invention allows the mobile robot moving in the surveillance region to perform unattended surveillance, and (a) with a plurality of beacons located in the surveillance region. Estimating in real time the current position of the mobile robot that changes with movement in the surveillance area by wireless communication; (b) generating a first image frame by photographing a preset shooting area at the current position; (c) sequentially receiving a plurality of second image frames photographed by a fixed camera installed in the surveillance area through a network; (d) detecting the moving objects included in the first image frame and the second image frame to track the movement of the moving object by an object tracking filter, and when the moving object is detected from the second image frame, Determining an expected position of the moving object based on the position information of the moving object in the second image frame last detected by the moving object; (e) determining a tracking path from the current position to an expected position of the moving object; And (f) outputting a control command for moving the mobile robot to the expected position along the determined tracking path and moving the mobile robot according to the movement of the mobile object detected from the first image frame. Have

According to the mobile monitoring and tracking device according to the present invention, by detecting and tracking the moving object from the image frame received in real time from the fixed camera and the image frame photographed by the self-capturing unit, a large surveillance area and a lot of obstacles Even in case of using unmanned surveillance using a large number of fixed cameras, intrusion detection can be performed efficiently with fewer cameras without blind spots. In addition, by estimating the position of the mobile robot by using the RF signal transmitted from the beacon, it can accurately estimate the location even in the indoor environment can be used for mapping the surveillance area and tracking of the moving object.

1 is a block diagram showing the configuration of a preferred embodiment of a mobile monitoring and tracking device according to the present invention;
2 is a view showing an example of a surveillance area to which the mobile monitoring and tracking device according to the present invention is applied;
3 is a view showing an example of a Zigbee module used for communication with a beacon in the position estimator;
4 is a diagram illustrating an example of mapping an estimated current position on a previously stored map;
5 is a view showing a stereo camera used as the imaging unit,
FIG. 6 is a diagram illustrating an example of estimating distance information from two first image frames obtained by an image capturing unit; FIG.
FIG. 7 is a diagram illustrating an example of distance information obtained by two first image frames photographed by a stereo camera and registration; FIG.
8 is a diagram illustrating an example in which distance information is expressed in a two-dimensional coordinate system;
9 is a diagram illustrating an example in which a partial region for extracting distance information is set from among photographing regions of an image capturing unit;
10 is a diagram illustrating an example of detecting coordinates at which a moving object moves;
11 illustrates an example of searching for an optimal path using an A-star algorithm;
12 is a flowchart illustrating a process of performing a preferred embodiment of the mobile monitoring and tracking method according to the present invention;
13A and 13B are graphs showing the data measured for the X coordinate and the Y coordinate when the mobile robot moves and the data obtained by the correction by the Kalman filter compared with the actual data, and
14A and 14B are diagrams illustrating a system in which a mobile monitoring and tracking device according to the present invention is implemented.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the mobile monitoring and tracking device and method according to the present invention.

1 is a block diagram showing the configuration of a preferred embodiment of a mobile monitoring and tracking device according to the present invention.

Referring to FIG. 1, the mobile monitoring and tracking device according to the present invention is provided in a mobile robot that autonomously moves in a surveillance area, and performs unmanned monitoring. 130, an image input unit 140, an object position determiner 150, a path generator 160, and a driver 170.

As described above, an unmanned surveillance system using a mobile robot is an economical and effective system. The present invention is provided in a mobile robot used in such an unmanned surveillance system to perform an unmanned surveillance in the surveillance area.

2 is a diagram illustrating an example of a surveillance area to which a mobile monitoring and tracking device according to the present invention is applied. Referring to FIG. 2, the surveillance region is divided into a plurality of spaces, and each separated space may be moved through a door. The mobile robot can move autonomously or move by remote control between the separated spaces. The image capturing unit 130 of the mobile monitoring and tracking device according to the present invention detects a shooting area set in the surveillance area while the mobile robot moves. Shoot in real time. The photographing area depends on the specific performance of the imaging unit 130.

A plurality of RF beacons arranged in each separated space are used for wireless communication with the position estimator 110, and the position estimator 110 is based on the strength of the signal received from the RF beacon. You can estimate your current location. This will be described in detail later.

Meanwhile, the fixed camera installed at the center of the surveillance region continuously photographs the fixed region within the surveillance region. The stationary camera transmits an image in real time by performing communication through a network with a mobile monitoring and tracking device or a mobile robot according to the present invention. The object position determiner 150 continuously analyzes the image photographed by the image capturing unit 130 and the image transmitted from the fixed camera to track the movement path of the detected moving object when the moving object is detected from the image. The driving unit 170 may continuously track the moving object by moving the moving robot along the moving path of the moving object.

Each separated space of the surveillance area, the position of the RF beacon and the position of the stationary camera may be mapped on a map and stored in advance. In this case, the position estimator 110 estimates the current position of the mobile robot in real time and maps it on a map. The position estimator 110 detects the moving object detected from the image captured by the imaging unit 130 and the image transmitted from the fixed camera. Locations can also be mapped on the map. However, if there is no information about the surveillance area, the mobile robot randomly moves in the surveillance area, and the map generator 120 may create a new map for the surveillance area.

Hereinafter, the operation of each component of the mobile monitoring and tracking device according to the present invention will be described in detail.

The position estimator 110 estimates in real time the current position of the mobile robot that changes according to movement in the surveillance region by wireless communication with a plurality of beacons located in the surveillance region.

Wheel encoders and gyroscopes are used to estimate the position of a mobile robot moving randomly in a certain area without a predetermined path in real time. The wheel encoder is a sensor that measures the traveling distance of the mobile robot, and the gyroscope measures the moving direction of the robot. However, the encoder has low accuracy due to various reasons such as slippage between the floor and the wheels of the mobile robot and alignment of the wheels. As the robot moves, odometric errors accumulate as the robot moves, It is difficult to know the exact location of the.

Therefore, an auxiliary method is used to correct the error of the position estimation of the mobile robot by the encoder. Location estimation methods that can be used as auxiliary methods include client-based design and reference node-based methods. In the client-based method, a signal transmitted from reference nodes is received at a radio wave receiving client such as a tag to calculate a client's own location, and GPS is a typical method. The fingerprinting method through a WLAN is a method of determining a location by transmitting radio waves from an access point (AP) to a client, which also corresponds to a client-based location recognition system.

The reference node-based location recognition method is a method of calculating a location of a tag by receiving a signal transmitted from a tag at a reference node. This method determines the position of the client using the strength or direction of the signal received from the client, the propagation time difference of the signal, and the like. The distance-based position estimation method estimates the position by triangulation by measuring the distance. In order to apply the location estimation method in indoor environment, various communication methods such as RFID, laser, sonar, radar and infrared have been used, but recently Wi-Fi Ways to use infrastructure, UWB and ZigBee are rapidly evolving.

The position estimator 110 of the present invention estimates the position of the mobile robot using a ZigBee of the RF communication method to correct an error caused by using the wheel encoder and the gyroscope. 3 is a diagram illustrating an example of a Zigbee module used for communication with a beacon in the position estimator 110. Beacons capable of ZigBee communication are arranged throughout the surveillance area in which the mobile robot moves, and the position estimator 110 measures the strength of radio waves received from each beacon and triangulates the location of the mobile robot. Estimate

In general, the change according to the distance of the radio signal strength is expressed by Equation 1 below.

Here, A is a proportionality constant, a is an index indicating strength attenuation, r is a distance, S _r is the strength of the signal received by the position estimator 110, and S ₀ is the strength of the signal transmitted from the beacon. Since A and s change according to the measurement environment and time, it is difficult to measure the exact value, and even if the initial value is measured initially, an error occurs over time.

A representative method of positioning according to the measurement component of a moving client is Received Signal Strength Indication (RSSI), which uses the strength of a received signal, which is a method of receiving radio waves from a transmitter attached to an object or a tag. Using the strength, the distance from the sensor to the transmitter is measured according to the weakening of the radio wave strength, and the measured distance can be used to estimate the current position through triangulation techniques.

The following Table 1 shows the error of the position estimation through the experiment, it can be seen that the average error distance is more than 1m in the case of the position estimation using only beacons need correction.

X (m) Y (m) Distance (m) Average 0.85 0.92 1.04 Dispersion 0.56 0.38 0.78 Standard Deviation 1.24 1.14 1.21

As described above, the RF communication method with the encoder and the beacon used to estimate the position of the mobile robot is less accurate and generates a lot of errors. Thus, the Kalman Filter can be used to compensate for this. The Kalman filter is a set of mathematical equations that provides an efficient means of calculation in a way that minimizes errors to track the state of the process. The Kalman filter also has excellent separation of signal and noise to prevent instantaneous position jumps, and is used to reliably detect accurate position, speed and time.

The position estimator 110 estimates the current position of the mobile robot by applying a Kalman filter to the position measured by the encoder and the position measured by the strength of the RF signal received from the beacon.

The Kalman filter's algorithm consists of a time update equation and a measurement update equation for prediction. The time update process is a step of predicting the current state in advance, and delivers the estimation result of the current state in the forward direction. The measurement update process adjusts the estimated state values delivered by the actual measurement at that time. The Kalman filter basically assumes the following equation (2).

Where x _k is a state vector at a specific time k, u _k is a user input at that time, A _k is a state transition matrix based on the previous state x _{k -1} at that time, and B _k is a state transition by user input. The matrix, and w _{k -1} are noise variables.

In addition, the state vector x _k and the vector z _k actually obtained when the vector is measured have the same relationship as in Equation 3 below.

Where H is the matrix involved in the measurement at that time and v _k is the noise variable.

As described above, the time update process corresponds to the prediction step of the Kalman filter, and is deduced as shown in Equation 4 below.

The first equation of Equation 4 relates to the deductive state prediction, and the second equation relates to the deductive covariance prediction.

Next, the measurement update process corresponds to a calibration step of the Kalman filter, and is inductively performed as in Equation 5 below.

That is, the position estimator 110 generates an optimal gain of the Kalman filter by the first equation, and corrects the value of the current state by the second equation. The third equation is also a formula for correcting the covariance matrix.

By using the RF communication with the beacon in this way it is possible to accurately estimate the current position of the mobile robot even in the indoor environment.

When the current position of the mobile robot is estimated by the above process and the map created for the surveillance area is stored in advance, the current position of the mobile robot estimated by the position estimator 110 is mapped onto the stored map. Can be However, if the information on the surveillance area is not known in advance, a new map of the surveillance area should be prepared by the SLAM algorithm described above before performing unattended monitoring. 4 is a diagram illustrating an example of mapping an estimated current position on a previously stored map.

The map generator 120 extracts a distance extracted from a plurality of first image frames obtained by capturing a photographing area at regular intervals by panning of the imaging unit 130 when there is no map previously stored in the surveillance area. Based on the information, as the mobile robot moves, nodes are arranged in the surveillance area at predetermined intervals to generate a map of the surveillance area.

As a method for creating a map of the surveillance area based on the current position estimated by the position estimator 110 in real time, there is a method using a distance measuring sensor such as a laser sensor and an ultrasonic sensor, and using a vision sensor. There is a way. The method of using the vision sensor consists of constructing a map of the surrounding environment using visual feature points extracted from the image.

In the mobile monitoring and tracking device according to the present invention, a map of a surveillance area is prepared using a stereo camera capable of estimating distance information. That is, the imaging unit 130 is implemented in the form of a stereo camera and is used not only to detect the moving object but also to map the surveillance area. 5 illustrates a stereo camera used as the imaging unit 130.

In the pre-processing means of the stereo camera, the performance of stereo matching may be improved by minimizing the distortion of the camera through independent brightness adjustment and quadrature of the left and right images captured by the stereo camera. In the stereo image geometry, one point of one image corresponds to a line in another image, which is called an epipolar line. By using such a conjugate constraint, the corresponding point corresponding to one point in one image may be limited to the conjugate line rather than finding the corresponding point in the whole other image.

If a feature of a specific region is extracted from two images, and the same feature appears in two images, disparity between the two images may be used for proximity calculation, and distance information may be obtained from the information. The map generation unit 120 calculates the inconsistency from the left and right image inputs processed by the preprocessing means and displays the brightness information.

및

로 나타내며, 점 E_L과 E_R을 공액으로 표현할 수 있다. 이와 같은 방법을 사용하여 두 개의 제1영상프레임에서 서로 대응되는 지점을 찾을 수 있다.FIG. 6 is a diagram illustrating an example of estimating distance information from two first image frames obtained by the imaging unit 130. In Fig. 6, the epipolar plane is based on points X, O _L and O _R. O _L and O _R representing the two cameras provided in the image pickup unit 130, and P _L and P _{R, which} are points at which the observed random point P is projected to each first image frame, are respectively on the first image frame.

And

The points E _L and E _R can be expressed as conjugates. Using this method, points corresponding to each other in two first image frames may be found.

Next, two first image frames may be matched based on a mismatch between corresponding points in the two first image frames. As a method of matching image frames, a method of dividing an image frame into a plurality of blocks having a predetermined size and obtaining disparity of each block in a reference image frame may be used.

7 is a diagram illustrating an example of two first image frames photographed by a stereo camera and distance information obtained by registration. The upper two images of FIG. 7 represent the left first image frame and the right first image frame, respectively, and the bottom image shows the distance information in brightness. First, the A point (A_left, A_right) and the B point (B_left, B_right) are displayed on the left and right first image frames, respectively, and a mismatch between the A point and the B point can be calculated by Equation 6 below. .

If the value of D (A) is larger than D (B) in the mismatch for each point thus obtained, it can be seen that the point A is located closer to the mobile robot than the point B.

In this way, if a mismatch is calculated for each point constituting the block, the mismatch for an image frame may be expressed by Equation 7 below.

Where d _min and d _max are the minimum and maximum values of mismatch obtained for each point, and m is the size of the mask.

Referring to the bottom image of FIG. 7, the distance from the mobile robot is represented by a bright color, and the distance information is represented by the darker the distance.

The map generator 120 processes the first image frame continuously obtained as the mobile robot moves as described above to extract distance information. The obtained distance information may be expressed in a two-dimensional coordinate system of (r, θ) in which the height z is omitted from the three-dimensional cylindrical coordinate system. In addition, in order to express distance information expressed by actual distance to each point in coordinate system, it is expressed as reduced distance using scale.

사이의 값으로 표현할 수 있다.8 is a diagram illustrating an example of representing distance information in a two-dimensional coordinate system. As shown in FIG. 8, distance information is represented by a cylindrical coordinate system of r-θ on a two-dimensional xy plane, where r is a distance reduced from an actual distance and θ is a value determined by the angle of view of a camera.

Can be expressed as a value in between.

The map generator 120 may create a map of the entire surveillance area based on the current position of the mobile robot estimated by the position estimator 110 and distance information extracted from the first image frame photographed by the imaging unit 130. have. When creating a map, nodes can be arranged at regular intervals according to the movement path of the mobile robot so that they can be used to move the mobile robot to a target point later.

In this case, as the mobile robot moves, all first image frames obtained in real time may be analyzed, and distance information may be extracted by analyzing the first image frames at regular intervals by referring to the current position of the mobile robot to reduce the amount of calculation. . In addition, to set the analysis interval of the first image frame, it is possible to improve the speed of mapping by moving the mobile robot to an area where analysis of distance information is not completed without relying on autonomous movement of the mobile robot.

That is, when distance information on the photographing area is obtained by analyzing the first image frame at the current position, the driving unit 170 moves to the point farthest from the current mobile robot among the areas where no node is disposed in the photographing area. Move, and the map generator 120 places a new node at the point where the driving unit 170 moves the mobile robot. In addition, the location estimator 110 estimates the current position at the point where the node is newly placed, and the map generator 120 analyzes the first image frame to extract distance information and create a map.

On the other hand, in order to reduce the amount of calculation and to improve the speed of mapping, the area from which the distance information is extracted from the photographing area may be limited in consideration of the size or height of the mobile robot. 9 is a diagram illustrating an example in which a partial region for extracting distance information is set from among photographing regions of the imaging unit 130. In the image of the lower part of FIG. 9, the area indicated by the thick line is an area for extracting the distance information, and the upper image indicates the distance information extracted from the set area. As such, extracting only the distance information of the part that actually affects the movement of the mobile robot in the surveillance area can improve the speed of mapping.

The image input unit 140 sequentially receives a plurality of second image frames photographed by a fixed camera installed in the surveillance area through a network.

As shown in FIG. 2, a fixed camera is installed in the surveillance area, and the number of the fixed cameras may be one or more. The stationary camera is designed to communicate over a network and can wirelessly transmit captured images. In addition, when the map of the surveillance area is stored in advance, the location information of the fixed camera is displayed on the map. When the map generator 120 creates a map of the surveillance area, the communication with the fixed camera is performed. The location information can be grasped.

The fixed camera continuously photographs the fixed area to generate a plurality of second image frames, and the plurality of second image frames are sequentially input to the image input unit 140 through a network. The intruder may be detected when an intruder occurs through comparison between adjacent second image frames.

The object position determiner 150 detects moving objects included in the first image frame and the second image frame, tracks the movement of the moving object by the object tracking filter, and when the moving object is detected from the second image frame. The estimated position of the moving object is determined based on the position information of the moving object in the second image frame in which the moving object was last detected.

The imaging unit 130 continuously photographs the photographing area while the mobile robot moves to sequentially output a plurality of first image frames, and a plurality of second image frames input from the fixed camera are also sequentially input. When detecting a moving object such as an intruder from the first image frame and the second image frame, a known object detection method may be used, and as an example, a motion estimation method using block matching may be used.

Block matching is a method of evaluating similarity using Mean Absolute Different (MAD) between blocks set in two temporally adjacent video frames. That is, the coordinate in the t-th image frame I _t (k, l) to M × N Assuming that the size of the block position, the block and the t-1 th frame image I _{t -1} in the coordinates (x + k, l The average absolute difference with the block located at + y) may be calculated as in Equation 8 below.

According to Equation 8, the block having the smallest MAD value is found in the search window set in the t-1 th image frame, and the position change between the two image frames is detected as a motion vector. The moving path of the moving object is tracked according to the detected motion vector, and the moving robot is moved according to the moving path of the moving object as needed.

10 is a diagram illustrating an example of detecting coordinates at which a moving object moves. First, as shown in the upper drawing of FIG. 10, a 320 × 240 image frame is divided into 3 × 5 blocks, and a value of 1 is allocated when the MAD value of the previous image frame exceeds a reference threshold, and a value of 0 is not exceeded. The block having the minimum value passing through the 3 × 3 mask window is determined as the moving coordinate of the moving object. To this end, the weight of the block determined as the movement coordinate is calculated by Equation 9 below.

Referring to the bottom view of FIG. 10, the block B [1] [3] having the maximum value is determined as the movement coordinate.

If the moving object is detected in the second image frame, since the fixed camera cannot move according to the movement of the moving object, the moving path of the moving object is continuously tracked in the plurality of second image frames, and the moving object is the second image. When moving away from the area corresponding to the frame, the expected position of the moving object is determined. The expected position of the moving object is determined to move the mobile robot having the imaging unit 130 to the place where the moving object is located so as to continuously track the moving object. However, even before the moving object is moved out of the area corresponding to the second image frame, if the same moving object is not detected from the first image frame but only from the second image frame, the moving robot may be immediately moved to the position of the moving object. have.

To keep track of moving objects in an image frame, we can use a particle filter based on a probability model. Particle filters are a popular method for tracking objects in the field of computer vision, and their performance depends on the accuracy of prior probabilities. Accordingly, when tracking the moving object by the object position determiner 150, the prior probability density function may be generated by the block matching method described above.

와 영상프레임에서 관측된 데이터 Z_{1 :t}에 의해 유도될 수 있다. 파티클 필터에서 사후 확률은 N개의 상태 집합과 샘플링을 통한 파티클

으로 나타낼 수 있다.A particle filter, an object tracking filter, can be used to track the movement of the moving object and determine the expected position. All data about the moving object to be tracked is the posterior probability density function for state X _t .

And data Z _{1 : t} observed in the image frame. In particle filters, the posterior probabilities are particles with N state sets and sampling.

It can be represented as

는 시간 t-1에서의 사후 확률로부터 다음의 수학식 10과 같이 유도될 수 있다.Particle filters can be divided into two phases, the prediction phase and the observation phase, and the prior probability at time t in the prediction phase.

Can be derived from Equation 10 below from the posterior probability at time t-1.

Next, the observation data for the state X in the observation step can be expressed by the following equation (11).

를 이용하여 유도할 수 있다.Here, V _t represents noise of observation data independent of the state X _t . And observed data Z _t

It can be derived using.

In this way, the object position determiner 150 tracks the movement of the moving object using the particle filter and determines the expected position, so that the intruder may not detect the moving object even from the first image frame obtained by the imaging unit 130. By detecting, the mobile robot can quickly move to the intruder's location to track the intruder.

The path generator 160 determines a tracking path from the current position of the mobile robot estimated by the position estimator 110 to the expected position of the moving object determined by the object position determiner 150. As a method of determining the tracking path, an existing path determination method may be used. When the nodes are arranged at regular intervals when the map generator 120 creates the map for the surveillance area, or when the nodes are arranged on the map of the previously stored surveillance area, the path generation unit 160 displays the current state of the mobile robot. The tracking path can be determined by the shortest distance connecting the node closest to the position to the node closest to the expected position of the moving object.

One way to determine the tracking path is to use the A-Star (A ^* ) algorithm, which is a graph search algorithm that finds the path from the initial node to the target node, and calculates the shortest path in the graph. Search with. 11 is a diagram illustrating an example of searching for an optimal path using an A-star algorithm. The node marked 'S' is the starting node, and the node marked 'b' is the destination node. The number on the straight line connecting the two nodes represents the cost of moving between the two nodes. The optimal path connecting S node and b node is {S, c, a, b}, and the cost at this time is 5 + 3 + 1 + 0 = 9, which is the minimum cost.

As such, the path generation unit 160 may determine a path having a minimum cost among a plurality of paths obtained by connecting nodes disposed between the current location and the expected location by using the A-star algorithm.

The driving unit 170 moves the mobile robot to the expected position along the determined tracking path, and outputs a control command for moving the mobile robot according to the movement of the moving object detected from the first image frame. As described above, when the moving object is detected from the second image frame photographed by the fixed camera and the moving object is not detected from the first image frame photographed by the imaging unit 130, the moving object is the second image. Once out of the area corresponding to the frame, there is a problem that no more tracking is possible. Therefore, there is a need to continuously track the movement of the moving object by moving the mobile robot to the point where the moving object is located.

When the object position determiner 150 determines the expected position of the moving object and the path generator 160 determines the tracking path from the current position to the expected position, the driving unit 170 moves the position along the determined tracking path. Move the robot to the point where When the moving object is detected from the first image frame photographed by the imaging unit 130, the object position determiner 150 tracks the movement of the moving object within the first image frame, and the driving unit 170 tracks the movement. In other words, a control command for moving the mobile robot according to a motion vector extracted from two adjacent temporally adjacent first image frames is output.

As such, when there is no control by the driving unit 170, the mobile robot moves autonomously within the monitoring area or moves by a remote monitoring means. When the moving object is detected, the mobile robot is controlled by the driving unit 170. It will track the moving object. If the driving unit 170 does not control the movement of the mobile robot because the moving object is not detected, the mobile robot stays in the area where the intruder is particularly likely to be generated or the intruder is likely to be in a long time even in the surveillance area. You can also quickly detect objects.

As described above, the mobile monitoring and tracking apparatus according to the present invention detects a moving object from the second image frame received in real time from the fixed camera and the first image frame photographed by the imaging unit 130 provided therein. By tracking, even if the surveillance area is large and there are many obstacles, intrusion detection can be performed efficiently with fewer cameras than with blind spots compared to the case of performing unmanned surveillance using a plurality of stationary cameras.

12 is a flowchart illustrating a preferred embodiment of the mobile monitoring and tracking method according to the present invention.

Referring to FIG. 12, the position estimator 110 detects a current position of a mobile robot that changes according to movement by autonomous movement or remote monitoring in the surveillance region by wireless communication with a plurality of beacons located in the surveillance region. It estimates (S1210). The imaging unit 130 generates a first image frame by photographing a preset shooting area at the current location (S1215).

When the map for the surveillance region is not stored in advance (S1220), the map generator 120 may distance the plurality of first image frames obtained by capturing the photographing regions at regular intervals by panning of the imaging unit 130. The information is extracted, and nodes are arranged at predetermined intervals in the surveillance region according to the movement of the mobile robot to generate a map of the surveillance region (S1225).

Meanwhile, the image input unit 140 sequentially receives a plurality of second image frames photographed by the fixed camera installed in the surveillance area (S1230), and the object position determiner 150 receives the first image frame and the second image frame. The moving object is detected from.

When the moving object is detected from the second image frame photographed by the fixed camera (S1235), the object position determiner 150 tracks the movement of the moving object by an object tracking filter, for example, a particle filter, and the moving object ends. The estimated position of the moving object is determined based on the position information of the moving object in the detected second image frame (S1240). The path generation unit 160 determines the tracking path from the current position of the mobile robot to the expected location (S1245), and the driving unit 170 moves the mobile robot to the expected location along the determined tracking path (S1250).

When the moving robot moves to the point where the moving object is located and the moving object is detected from the first image frame or the moving object enters the shooting area of the imaging unit 130 from the beginning and the moving object is detected from the first image frame (S1255). ), The object position determiner 150 tracks the movement of the moving object in the first image frame, and the driving unit 170 moves the mobile robot according to the tracked movement (S1260) so that the moving object is captured by the imaging unit 130. It allows you to keep track of the scene without getting out of it.

Experiments were conducted to evaluate the performance of the present invention. Hanul Robotics' Hanuri-RS robot was used for the experiment, which is a synchronous drive system with three driving wheels, designed to simultaneously drive and steer the wheels. The wheel encoder and main sensor were stereo cameras from Point Gray Research. The TRICLOPS library provided by Point Gray Research was used to measure the distance information for mapping. Four beacons and one reference coordinator among the Zigbee modules for position estimation were used, and Radiopulse's MG2455 chipset was used. In addition, the PC specifications used are shown in Table 2 below.

Device Specification CPU Intel® Core (TM) 2 Duo RAM 2 GB RAM VGA Nvdia Geforce 9600 GT CAMERA PointGrey BurnableBee2

13A and 13B are graphs showing the data measured for the X coordinate and the Y coordinate and the data obtained by the correction by the Kalman filter when the mobile robot moves with the actual data. 13A and 13B, it can be seen that position estimation can be performed with an accuracy of 80% or more by correcting the measured value by the Kalman filter.

14A and 14B are diagrams illustrating a system in which a mobile monitoring and tracking device according to the present invention is implemented. RSSI values are received from four Zigbee beacons in real time, and each beacon ID is identified and matched with the RSSI value. In addition, the direction of the mobile robot received from the gyroscope is expressed in real time, and the distance information obtained from the stereo camera is displayed. In FIGS. 14A and 14B, the stereo cameras face different directions, and the direction information of the stereo camera is also recorded and displayed on the map. As such, by connecting the mobile monitoring and tracking device according to the present invention through an external monitoring means and a wireless network, the unmanned monitoring state can be checked in real time.

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and may be implemented in the form of a carrier wave (for example, transmission via the Internet) . The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

Although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific preferred embodiments described above, and the present invention belongs to the present invention without departing from the gist of the present invention as claimed in the claims. Various modifications can be made by those skilled in the art, and such changes are within the scope of the claims.

110-position estimator
120-Map Generator
130-imaging unit
140-Video input unit
150-object positioning unit
160-path generator
170-drive section

Claims (17)

In the mobile monitoring and tracking device which is provided in the mobile robot moving in the surveillance area to perform unmanned surveillance,
A position estimator for estimating in real time the current position of the mobile robot that changes in accordance with movement in the surveillance region by wireless communication with a plurality of beacons located in the surveillance region;
An imaging unit for photographing a preset photographing area at the current position to generate a first image frame;
An image input unit which sequentially receives a plurality of second image frames photographed by a fixed camera installed in the surveillance area through a network;
The moving object included in the first image frame and the second image frame is detected to track the movement of the moving object by an object tracking filter. When the moving object is detected from the second image frame, the moving object is detected. An object position determiner which determines an expected position of the moving object based on position information of the moving object in the second detected image frame;
A path generation unit determining a tracking path from the current location to an expected location of the moving object; And
And a driving unit for moving the mobile robot to the expected position along the determined tracking path and outputting a control command for moving the mobile robot according to the movement of the mobile object detected from the first image frame. Mobile monitoring and tracking device. The method of claim 1,
The position estimator estimates the current position by triangulation using a distance to each beacon calculated based on the strength of RF signals received from a plurality of adjacent beacons, and uses the Kalman filter to estimate the current position. Mobile monitoring and tracking device, characterized in that the correction. 3. The method according to claim 1 or 2,
In the surveillance area according to the movement of the mobile robot based on the distance information on the shooting area extracted from the plurality of first image frames obtained by photographing the shooting area at regular intervals by a panning technique of the imaging unit. It further comprises a map generator for generating a map for the surveillance area by arranging nodes at predetermined intervals,
And the path generation unit determines a tracking path based on a shortest distance connecting a node closest to the current location and a node closest to the expected location on a map generated for the surveillance area. The method of claim 3,
The driving unit moves the mobile robot to a point farthest from the mobile robot based on the distance information obtained for the area where the node is not disposed in the photographing area.
The map generating unit is a mobile monitoring and tracking device, characterized in that for placing the node at the point where the drive moves the mobile robot. The method of claim 3,
The imaging unit is a stereo camera,
The map generator extracts the distance information by matching two first image frames simultaneously photographed by the stereo camera. 3. The method according to claim 1 or 2,
The object tracking filter is a particle filter,
And the object position determiner generates a prior probability density function of the particle filter by applying a block matching algorithm to a plurality of temporally contiguous second image frames including the moving object. 3. The method according to claim 1 or 2,
The path generation unit determines the tracking path by an A-star algorithm for searching for a path having a minimum cost among a plurality of paths obtained by connecting nodes arranged between the current location and the expected location. And tracking device. 3. The method according to claim 1 or 2,
And the mobile robot autonomously moves within the monitoring area when the driving unit does not output the control command. In the mobile monitoring and tracking method for the mobile robot moving in the surveillance area to perform unattended monitoring,
(a) estimating in real time the current position of the mobile robot that changes with movement in the surveillance region by wireless communication with a plurality of beacons located in the surveillance region;
(b) generating a first image frame by photographing a preset shooting area at the current position;
(c) sequentially receiving a plurality of second image frames photographed by a fixed camera installed in the surveillance area through a network;
(d) detecting the moving objects included in the first image frame and the second image frame to track the movement of the moving object by an object tracking filter, and when the moving object is detected from the second image frame, Determining an expected position of the moving object based on the position information of the moving object in the second image frame last detected by the moving object;
(e) determining a tracking path from the current position to an expected position of the moving object; And
(f) moving the mobile robot to the expected position along the determined tracking path, and outputting a control command for moving the mobile robot according to the movement of the mobile object detected from the first image frame; Mobile monitoring and tracking method characterized in that. The method of claim 9,
In the step (a), using the distance to each beacon calculated based on the strength of the RF signal received from a plurality of adjacent beacons to estimate the current position by triangulation method, using the Kalman filter Mobile monitoring and tracking method characterized by correcting the current position. 11. The method according to claim 9 or 10,
Between step (b) and step (c),
(g) within the surveillance region according to the movement of the mobile robot based on distance information on the photographing region extracted from a plurality of first image frames obtained by photographing the photographing regions at regular intervals by a panning technique; Generating a map for the surveillance area by arranging nodes at preset intervals,
In the step (e), the mobile monitoring characterized in that the tracking path is determined by the shortest distance connecting the node closest to the current position and the node closest to the expected position on the map generated for the surveillance area; Tracking method. 12. The method of claim 11,
In the step (g),
(g1) moving the mobile robot to a point farthest from the mobile robot based on distance information obtained for the area where the node is not disposed in the photographing area; And
(g2) disposing the node at a point where the mobile robot is moved; is repeatedly performed for the entire surveillance area. 12. The method of claim 11,
In the step (b), the first image frame is generated by a stereo camera,
In the step (g), the mobile surveillance and tracking method characterized in that to extract the distance information by matching the two first image frames taken at the same time by the stereo camera. 11. The method according to claim 9 or 10,
The object tracking filter is a particle filter,
In the step (d), the mobile monitoring and tracking method comprising generating a prior probability density function of the particle filter by applying a block matching algorithm to a plurality of temporally contiguous second image frames including the moving object. . 11. The method according to claim 9 or 10,
In the step (e), the tracking path is determined by an A-star algorithm searching for a path having a minimum cost among a plurality of paths obtained by connecting nodes arranged between the current location and the expected location. Removable surveillance and tracking method. 11. The method according to claim 9 or 10,
And the mobile robot autonomously moves within the surveillance area when the mobile robot does not move by the control command. A computer-readable recording medium having recorded thereon a program for executing the mobile monitoring and tracking method according to claim 9 or 10.

Images