KR102161343B1 - Stream reasoning surveilance system and missing child alarming method using the same - Google Patents

Abstract

The present invention relates to a stream reasoning monitoring system capable of automatically tracking the departure or movement of a specific person in a place where many people come and go, and a stream reasoning monitoring method using the same. To this end, the present invention provides a stream reasoning monitoring system comprising a common device interface layer for collecting data, an image analysis module for analyzing a collected image, and a trajectory inference module for inferring a trajectory of a moving object, the stream The reasoning monitoring system includes the steps of collecting images captured by a plurality of cameras by the common device interface layer; Merging the collected images by the image analysis module; Determining, by the trajectory inference module, the same group for the moving object in the merged image; It provides a stream reasoning monitoring method for performing the step of determining whether there is a moving object that satisfies the departure condition in the group.
Accordingly, according to the present invention, it is possible to provide a stream reasoning monitoring system capable of efficiently monitoring the situation with a small number of people by automatically recognizing a specific situation at a site where many people come and going, and giving an alarm to a monitorer.

Description

Stream reasoning surveilance system and missing child alarming method using the same}

The present invention relates to a stream reasoning monitoring system and a missing child alarm method using the same, and specifically, a stream reasoning monitoring system capable of automatically tracking the departure or movement of a specific person in a place where many people come and go, and a missing child alarm using the same. It's about how.

CCTVs are being installed everywhere to protect citizens from violent crimes and accidents. Such CCTVs are increasing by an average of 15 to 20% each year, but the number of personnel to operate and monitor CCTVs is insufficient, so monitoring is not being properly performed.

Therefore, the need for research on intelligent CCTV that can automatically grasp the situation is increasing because the monitoring system through manpower has its limitations in reality.

The current intelligent CCTV focuses on recognizing and tracking objects through image analysis.

According to Patent No. 10-1732115 shown in FIG. 1, the steps of displaying an image captured by a main surveillance camera on a main screen area, and displaying images taken by a plurality of sub surveillance cameras on a plurality of sub screen areas, respectively, Recognizing an object included in an image displayed in the main screen area and extracting object features. If one of the objects recognized in the main screen area and extracted object features are selected by the user, the main screen area and a plurality of sub-screen areas Calculating a similarity to an object selected by a user for each recognized object; And displaying the calculated similarity for each object.

However, in the above patent, only the function of calculating the similarity of one object selected by the user is not possible, and the relationship with various people is not automatically recognized. By grasping these relationships, we can more accurately infer the trajectories of various people.

In order to immediately recognize the situation occurring in the field where a large number of people move, it is essential to understand the relationship with people by collecting and analyzing various image information. For this, smart city middleware and stream reasoning technology are required.

On the other hand, according to Korean Patent Publication No. 2003-0065014, the device for preventing lost children using Bluetooth is composed of a main module worn by a parent and a sub module worn by a child, and the main module detects the transmission power drawn from the sub module. Measure the reception level.

However, such a device is inconvenient for both parents and children to wear a missing child prevention device, and there is a limitation that it cannot protect a child who does not wear a separate child loss prevention device.

The present invention has been made to achieve the above object, and to provide a stream reasoning monitoring system capable of automatically recognizing a situation at a site where many people come and go, and notifying a monitor.

In addition, it is to provide a stream reasoning monitoring system capable of inferring the trajectory of people and determining whether a specific person's movement path is abnormal.

And, it is to provide a method of automatically notifying the occurrence of missing children in places where there are many people without wearing a separate child loss prevention device.

In order to achieve the above object, the present invention provides a stream reasoning monitoring system comprising a common device interface layer for collecting data, an image analysis module for analyzing a collected image, and a trajectory inference module for inferring a trajectory of a moving object. A missing child generation alarm method using, comprising: collecting images captured by a plurality of cameras by the common device interface layer; Merging the collected images by the image analysis module; Determining, by the trajectory inference module, the same group for the moving object in the merged image; A missing child alarm method is provided in which determining whether there is a moving object that satisfies the departure condition in the group.

The image analysis module generates a grid and a reference line at regular intervals from the merged image, measures the distance between the moving object and the reference line and the height of the moving object using the grid and the reference line, and uses the distance and height to estimate the key. It is preferable to perform the step of searching the database and estimating the key of the moving object.

It is preferable that the trajectory inference module determines whether there is a moving object whose height is less than a certain key among the same group, and determines whether there is a moving object that satisfies the departure condition only when there is a moving object less than a certain key.

The image analysis module extracts the rgb value for each pixel of the moving object from the image collected by one camera, and also extracts the rgb value for each pixel of the moving object from the image collected by another camera, and the same object for each rgb. A step of calculating the similarity may be performed.

The image analysis module may determine the color of the moving object, and the trajectory inference module may determine the same object from a plurality of different images using the color.

The image analysis module extracts the trajectory data for the moving object, the trajectory inference module determines the same object based on the autocorrelation coefficient, and generates an entire trajectory for the same object using the trajectory data. It is desirable to do.

The trajectory inference module preferentially analyzes the image collected by a camera that photographs an image of an area adjacent to one moving object to infer the trajectory of the moving object, and if there is no image collected from a specific camera, the specific camera is photographed. It is preferable to infer the trajectory of the moving object by analyzing the image collected by the camera that photographs the area closest to the area.

The trajectory inference module may determine whether the group is the same based on a trajectory similarity with respect to the trajectory of each moving object and a distance between moving objects by time slot.

Preferably, the image analysis module divides and merges some of the images collected from a plurality of cameras into camera groups, and the trajectory tracking module infers a trajectory for each of the camera groups.

It is preferable that the trajectory inference module infers the trajectory by analyzing the image collected from the camera for a predetermined time, and updates the image every predetermined time to infer the trajectory.

According to the present invention, it is possible to provide a stream reasoning monitoring system capable of efficiently monitoring the situation with a small number of people by automatically recognizing a specific situation at a site where many people come and going, and giving an alarm to a monitorer.

In addition, by automatically inferring the trajectory of people, it is possible to respond immediately when a lost child occurs.

1 is an exemplary view for explaining an object tracking method using a surveillance camera according to the prior art;
2 is a block diagram showing the configuration of a smart city middleware system used in the stream reasoning monitoring system according to the present invention;
3 is a block diagram showing a procedure of a stream reasoning monitoring system according to the present invention;
4 is an exemplary diagram showing an example of a camera group used in FIG. 3;
5 is an explanatory diagram showing a data format of the stream reasoning monitoring system according to the present invention;
6 is an exemplary view for explaining a method for determining the same object according to the present invention;
7 is an exemplary view for explaining the same object processing method according to the present invention;
8 is an exemplary view for explaining the same group determination method according to the present invention;
9 is an exemplary diagram for explaining a processing method when data is not received;
10 is an exemplary diagram for explaining a method for determining the occurrence of an automatic missing child according to the present invention;
11 is a flow chart for explaining a method for determining the occurrence of an automatic missing child according to the present invention;
12 is an explanatory diagram for explaining a method for estimating the height of a moving object according to the present invention.

The configuration and operation of the embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 2, the stream reasoning monitoring system according to the present invention may be implemented using a smart city server 100, and the smart city server 100 includes a common device interface layer 110, an image analysis module ( 230), and a trajectory inference module 250.

The smart city middleware collects data from a plurality of cameras that photograph each area through the common device interface layer 110.

3 shows a general stream reasoning monitoring method according to the present invention. When data including images captured by each camera is collected, these data are adjusted to a format determined by the common device interface layer 110. In addition, these data are divided into a plurality of camera groups determined in advance by the image analysis module 230, and each camera group infers the trajectory of the moving object by analyzing images at predetermined time intervals.

Specifically, each camera photographs a certain area, and the image analysis module 230 divides the area to group the areas photographed by a plurality of cameras to form a camera group.

Referring to FIG. 4, a total of 9 cameras photograph each area. That is, camera 1 is the upper left, camera 2 is the upper center, and camera 5 is the center. In this case, the image analysis module 230 merges the images transmitted from cameras 1, 2, 4, and 5 to generate a camera group A. In addition, images transmitted from cameras 2, 3, 5, and 6 are merged to create camera group B, and images transmitted from cameras 4, 5, 7, and 8 are merged to create camera group C. The images transmitted from cameras 6, 8, and 9 are merged to create camera group D.

Meanwhile, the trajectory inference module 250 packetizes and divides the image data for one minute among the image data merged into the camera group, and infers the trajectory of the moving object by using the image data packet for one minute in the packet time. In addition, the image data packet is updated every 3 seconds to generate a new image data packet.

Referring to FIG. 5, when a moving object A is detected, the image analysis module generates a data format including an ID for the moving object. The data format may include the CCTV ID, the color of the clothes, the height, the color of the hair, the presence or absence of carrying a bag, the presence or absence of wearing glasses, the entry time, the trajectory data, and the like in which the image in which the moving object is found.

The trajectory inference module 250 first determines whether a moving object of an image captured by one camera and a moving object of an image captured by another camera are the same.

Specifically, the image analysis module extracts color values of r, g, and b for each pixel of the moving object 1 photographed by camera 1, and the color value for each pixel of the moving object 2 photographed by another camera, camera 2. Rgb histogram is generated by extracting the r, g, and b values. The x-axis (horizontal axis) of the rgb histogram indicates brightness, and the y-axis (vertical axis) indicates the number of pixels for the corresponding brightness.

Then, in the rgb histogram by moving object 1, depending on the number of pixels

X(1,1), X(1,2), X(1,3),… , A sequence of X(1,255) is created,

In the rgb histogram by moving object 2, depending on the number of pixels

X(2,1), X(2,2), X(2,3),… , A sequence of X(2,255) is generated.

Then, the similarity T of the same object of moving object 1 and moving object 2 is calculated as follows.

X_bar means an average of the sequence X, and the same object similarity can be calculated for each r, g, and b.

The trajectory inference module 250 determines an object with the same object similarity of 0.7 or more as the same object, and when there are multiple objects with the same object similarity of 0.7 or more, the object with the highest similarity of the same object is determined as the same object.

The data format generated by the image data includes trajectory data, and the trajectory data stores coordinates of the moving object for each time. When the trajectory inference module determines that the object is the same, the image analysis module adds and stores the ID information for the same object in the data format as shown in FIG. 7. In this way, it is possible to quickly determine the same object even if it is an image captured by another camera.

In addition, the trajectory inference module uses the trajectory data to merge the trajectory data as shown in FIGS. 5 and 6 to generate an entire trajectory for the same object in the camera group. At this time, the coordinates of each moving object at the same time are generated.

When the entire trajectory for the same object is generated, the trajectory inference module calculates the trajectory similarity between the trajectories. The trajectory similarity P is calculated as follows.

The locus of moving object 1 is (x11, y11), (x12, y12), (x13, y13),… , (x1n, y1n) and

The locus of moving object 2 is (x21, y21), (x22, y22), (x23, y23), ... , (x2n, y2n)

In addition, the trajectory inference module 250 measures the distance between moving bodies at predetermined time intervals to determine whether the relative distance is located within a predetermined distance. For example, the relative distances of moving objects 1 and 2 can be measured for 5 minutes at 5 second intervals, and if the relative distances of moving objects 1 and 2 are within 10m, they can be stored as proximity distances, and if they are outside 10m, they can be stored as departure distances .

The trajectory inference module 250 calculates the number of times that the relative distances of the moving object 1 and the moving object 2 are stored as the proximity distance for 5 minutes, and determines whether the number of proximity distances is 70% or more compared to the total number of measurements.

In conclusion, the trajectory inference module 250 determines that the trajectory similarity Px and Py of the moving object 1 and the moving object 2 are both 0.7 or more, and if the proximity distance is 70% or more, the moving object 1 and the moving object 2 are the same group. On the other hand, if the above conditions are not satisfied, it is not determined that the mobile 1 and the mobile 2 are the same group.

The trajectory inference module 250 infers the trajectory of the moving object by first analyzing the image collected by a camera that photographs an image of an adjacent area when one moving object disappears from one camera image. At this time, as shown in FIG. 9, if a specific camera fails or data is not received due to other causes, the trajectory inference module expands the analysis area and collects the area closest to the area photographed by the camera for which no data is received. Analyze the image and infer the trajectory of the moving object.

Next, with reference to FIGS. 10 and 11, a description will be given of an alarm method for missing children using the stream reasoning monitoring system according to the present invention.

The image analysis module 230 searches for a moving object whose height is less than a certain height among moving objects in the same group.

Referring to FIG. 12, the image analysis module 230 recognizes the boundary line between the floor and the wall and generates a grid on the floor using the reference line as a starting point. In this case, the spacing of the grids means an actual regular spacing, but the spacing of the grids generated on the screen becomes narrower as the distance from the reference line increases due to a visual effect.

When a moving object is detected, the image analysis module calculates how far away from the reference line is by using the grid and measures the distance between the moving object and the reference line. In addition, the height of the moving object is calculated by recognizing it from the feet to the head. At this time, the height of the moving object is the height shown on the screen photographed by the camera.

The image analysis module searches the estimated key database by using the measured height and distance data. Estimated keys for each height and distance are stored in the estimated key database.

The trajectory inference module 250 determines whether there is a moving object with a height of 120 cm or less of the moving object estimated by the image analysis module in the same group, and performs trajectory inference only in a group with a moving object whose height is 120 cm or less.

The trajectory inference module 250 determines that the moving object 1 or the moving object 2 has left the group when the departure condition is satisfied in the moving object belonging to the same group. The departure condition may be configured to be achieved when the trajectory similarities Px and Py of the moving body 1 and the moving body 2 are both less than 0.7 and the proximity distance is less than 70%.

When there are three or more moving objects in the same group, when the departure condition is satisfied between the moving object 1 and the moving object 2, the trajectory inference module analyzes whether the departure condition is satisfied between the remaining moving objects in the same group and the moving objects satisfying the departure condition. As a result of the analysis, it is determined that the side with fewer moving objects belonging to the same group has left the group.

For example, if the separation condition between the moving object 3 and the moving object 1 is satisfied and the separation condition between the moving object 3 and the moving object 2 is not satisfied, it is determined that the moving object 1 has left the group.

In the trajectory inference module, when the departure condition is satisfied, the ID of the moving object that has deviated and the camera number that photographs the area are stored in a separate storage. If necessary, it can also trigger an alarm. This stored data is displayed again so that it can be checked if necessary.

Although the above has been described with reference to embodiments of the present invention, those of ordinary skill in the art can variously modify and modify the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims. You will understand that you can change it.

100: smart city middleware 110: common device interface layer
130: common application interface layer 200: context reasoning computing layer
230: image analysis module 250: trajectory inference module
300: cloud computing layer

Claims (3)

In the missing child alarm method using a stream reasoning monitoring system comprising a common device interface layer that collects data, an image analysis module that analyzes the collected image, and a trajectory inference module that infers a trajectory of a moving object,
Collecting images captured by a plurality of cameras by the common device interface layer;
Merging the collected images by the image analysis module;
Determining, by the trajectory inference module, the same group for the moving object in the merged image;
A step of determining whether there is a moving object that satisfies the departure condition in the group is performed,
The trajectory inference module calculates the trajectory similarity with respect to the moving object, calculates the number of proximity distances in which the relative distance of another moving object is less than a certain distance for a certain period of time, so that the trajectory similarity is above a threshold value, If it is more than a certain number, it is judged as the same group, otherwise it is judged as a departure.
The image analysis module generates a grid and a reference line at regular intervals from the merged image, measures the distance between the moving object and the reference line and the height of the moving object using the grid and the reference line, and uses the distance and height to estimate the key. Performs the step of estimating the key of the moving object by searching the database, and it is determined whether there is a moving object whose height is less than a certain key by the image analysis module among the same group, and if there is a moving object less than a certain key, the departure condition is satisfied Lost child generation alarm method, characterized in that determining whether there is a moving object. delete delete

Images