KR100983057B1 - Behavior monitoring system and method thereof - Google Patents

Abstract

The present invention provides a video camera for photographing an area to be monitored and transmitting photographing data; A behavior recognition device attached to the body of the monitoring subject to recognize the behavior of the monitoring subject and to transmit behavior recognition data; And a monitoring server, wherein the monitoring server receives the shooting data captured by the video camera when the monitoring target is within the recordable area of the video camera, and the monitoring target is outside the recordable area of the video camera. In the case, the behavioral recognition data for the monitored object is received from the behavioral awareness device attached to the monitored object.

Video camera, monitoring, sensor node

Description

Behavior monitoring system and behavior monitoring method {BEHAVIOR MONITORING SYSTEM AND METHOD THEREOF}

TECHNICAL FIELD The present invention relates to a user surveillance system, and more particularly, to increase the scope and effect of surveillance in a video surveillance area by using an attached human behavior recognition device.

As technology advances and industrialization progresses rapidly, more and more security is required for certain areas.

In addition to the increased risk of exposure to crimes and risks, there is an increasing need for surveillance and the accuracy of specific areas or persons. For example, there may be a case where a certain amount of monitoring is required for prisoners in prison facilities.

Currently, in the case of a surveillance system or a monitoring system in a specific security zone or surveillance zone, it is built on the basis of a video camera. The monitoring system using such a video camera focuses on the most frequently traveled or mainly active areas within the surveillance area.

In other words, by installing a video camera along a frequent moving path, or by installing a video camera in a main activity area and receiving data as an image through the camera, the behavior of the monitored subject, the moving path of the monitored subject, and the monitored subject You can check what happens.

However, since the conventional monitoring system is built on the basis of the video camera, there is a surveillance zone that is difficult to shoot naturally.

To eliminate this blind spot, additional video cameras may be installed, but installing a video camera to cover all existing monitoring blind spots is not cost effective.

In addition, according to the monitoring system using a video camera, in addition to the blind spots due to the limitation of the shooting range that the video camera can shoot, the change of lighting or the amount of light in the day or night in the environment where the video camera is installed, etc. There exists a possibility of discriminating the same person as another person when there exists.

In addition, as the distance from the video camera increases, there may be a problem that the image of the monitoring target captured by the video camera becomes smaller than the minimum image size that can accurately recognize the target.

Currently, for a particular security zone or surveillance zone, a monitoring system for monitoring or observing people in the zone is built on a video camera as described above. As a result, blind spots are difficult to shoot video naturally.

The present invention provides a method that can effectively monitor the subject's behavior even when there is a subject to be monitored in such a blind spot.

In addition, the present invention provides a method for solving the problem of limitation of the monitoring target area, for example, because the target area is dark, making it difficult to monitor the subject through the captured video image.

The present invention also provides a method for accurately patterning a subject's behavior by continuously monitoring the subject and extracting detailed data.

In order to solve the above problems, the present invention provides a video camera for photographing the area to be monitored and transmitting photographing data; A behavior recognition device attached to the body of the monitoring subject to recognize the behavior of the monitoring subject and to transmit behavior recognition data; And a monitoring server, wherein the monitoring server receives the shooting data captured by the video camera when the monitoring target is within the video camera's shootable area, and the monitoring target receives the shooting data of the video camera. In case of deviation, behavioral awareness data about the monitored object is received from the behavioral awareness device attached to the monitored object.

Further, in the behavior monitoring system of the present invention, the video camera transmits a recognition start notification to the behavior recognition device to which the monitoring target is attached when the monitoring target person is outside the photographable area, and the behavior recognition device starts recognition from the video camera. In the case of receiving the notification, behavioral awareness data about the monitored subject may be collected and transmitted to the monitoring server.

In addition, in the behavior monitoring system of the present invention, the video camera transmits an out of zone notification to the monitoring server when the monitoring target is out of the imageable area, and the monitoring server receives out of zone notification from the video camera. In response to the recognition start notification received from the monitoring server, the behavior recognition device collects and transmits the behavior recognition data for the monitoring target to the monitoring server. You can also get started.

In addition, in the behavior monitoring system of the present invention, the behavior recognition apparatus transmits a recognition start notification to the video camera when the monitoring target enters the imageable area of the video camera, and the video camera sends the recognition start notification from the behavior recognition apparatus. In the case of the reception, the photographing data for the monitoring subject may be transmitted to the monitoring server.

In addition, in the behavior monitoring system of the present invention, the behavior recognition apparatus sends an area entry notification to the monitoring server when the monitoring dialog enters the imageable area of the video camera, and the monitoring server enters the region from the behavior recognition apparatus. Upon receipt of the notification, a recognition start notification is sent to the video camera, and when the video camera receives the recognition start notification from the monitoring server, the video camera may begin to transmit photographing data for the monitoring subject.

In addition, in the behavior monitoring system of the present invention, the behavior recognition apparatus may include at least three sensors for sensing acceleration in different directions.

The present invention comprises the steps of initializing the data of the video camera and the behavior recognition device; Determining whether a monitoring subject is present in a recordable area of the video camera; If the monitoring target is within the recordable area of the video camera, the video camera captures the monitoring target; and if the monitoring target is outside the recordable area of the video camera, the behavior recognizing apparatus detects whether the monitoring target's behavior is detected; And transmitting data about the monitored subject that has been photographed or perceived to a monitoring server.

In addition, in the behavior monitoring method of the present invention, if it is determined that the monitoring target is out of the imageable area of the video camera between the determination step and the shooting / detecting step, it is determined that the video camera is behavioral. The method may further include transmitting an acknowledgment initiation notification to the device.

In addition, in the behavior monitoring method of the present invention, if it is determined that the monitoring target is out of the captureable area of the video camera between the determination step and the shooting / detecting step, it is determined that the monitoring target exists. Sending an out of zone notification to the monitoring server; And the monitoring server receiving the out of zone notification sends an acknowledgment initiation notification to the behavior recognition apparatus.

In addition, in the behavior monitoring method of the present invention, if it is determined that the monitoring target is in the imageable area of the video camera between the determination step and the photographing / detecting step, it is determined that the behavior recognition device is a video. The method may further include transmitting a recognition start notification to the camera.

In addition, in the behavior monitoring method of the present invention, if it is determined that the monitoring target is in the imageable area of the video camera between the determination step and the photographing / detecting stage, the behavior recognizing apparatus is monitored. Sending an area entry notification to a server; And the monitoring server receiving the area entry notification sends a recognition start notification to the video camera.

According to the present invention, the subject's behavior can be monitored even in a blind spot of video shooting in a monitoring system built on the basis of a video camera.

According to the present invention, it is possible to utilize the existing video camera-based monitoring system, it is possible to reduce the cost of building a new monitoring system.

According to the present invention, even when the subject has left the video capture area, monitoring is automatically performed, whereby the behavior pattern of the subject can be continuously monitored.

According to the present invention, since the subject's behavior can be three-dimensionally patterned, the subject's behavior for a predetermined time can be analyzed in detail.

According to the present invention, by using the video camera and the attached sensor node together, it is possible to accurately grasp the behavior of the subject even when there is a sudden environmental change in the monitoring area.

According to the present invention, by attaching a sensor node that recognizes a behavior to a monitored subject, the monitoring subject's behavior can be grasped in a section other than the coverage of the video surveillance apparatus, and the video surveillance apparatus and the attached sensor node complement each other. As a result, a monitoring system can be used to monitor and analyze the behavior of the subject based on more accurate and richer data.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail.

1 schematically illustrates a monitoring target region and a shootable region of a video camera in a behavior monitoring system according to the present invention.

As shown in the drawing, when monitoring the monitoring target area 10 based on the video cameras 11a, 11b, 11c, and 11d, the area 13 that can be photographed by the video cameras 11a, 11b, 11c, and 11d. And the area 15 which cannot be photographed appears.

In the present invention, when the monitoring target is in the imageable area 13 of the video cameras 11a, 11b, 11c, and 11d in the monitoring target area 10, the video cameras 11a, 11b, 11c, and 11d are monitored. If the subject is monitored and the subject is in an area 15 that the video cameras 11a, 11b, 11c, and 11d cannot capture, the behavior recognition device (not shown) to which the subject is attached monitors the subject's behavior. Monitor.

2 is a flowchart illustrating a method for recognizing and monitoring the behavior of the monitoring target in the behavior monitoring system according to the present invention.

The behavior monitoring system includes a video camera for photographing a region to be monitored, a behavior recognition device attached to the body of the subject to be monitored, and a monitoring server for controlling the system.

When power is supplied to the system to start operation of the system (step S1), the video camera and the behavior recognition device are initialized by themselves or under the control of the monitoring server (step S2).

As will be described later, the behavior recognition apparatus may include a sensor unit for detecting a behavior of a monitoring target, an A / D (Analog to Digital) converter for converting the detected behavior data from an analog value to a digital value, a controller for calculating and processing data, and data It includes a storage unit for storing the wireless transmission and reception unit for transmitting data.

Initialization of the behavior recognition device, that is, when the internal devices as described above are initialized, each of the set values are reset to the initial value. The storage unit deletes existing storage data and secures a storage space. The sensor unit is initialized by the controller, and loads and uses a user's setting value set in the storage unit or the controller. In the case of a video camera, once the power is applied, the camera may start capturing the monitored person within the monitored area immediately.

 The video camera has a wireless transceiver capable of communicating with a monitoring server or behavior recognition device. As will be described later, according to the embodiment of the present invention, a communication module such as a Zigbee Phy module based on, for example, IEEEE 802.15.4 is mounted on a video camera, a behavior recognition device, and a sensor node of a system server. Construct a low power wireless communication network.

By initialization, the wireless transmission / reception unit of the video camera also starts to operate so that the shooting data can be transmitted to the monitoring server.

After operation, the behavior monitoring system determines whether there is a monitoring target in the imageable area of the video camera (step S4). If necessary, it may first be determined whether or not there is a monitoring target in the entire monitoring target region (S3 step).

In order to determine whether there is a monitored object in the monitored area, the monitoring server is based on whether there is a signal from the wireless transceiver of the behavior recognition device from the monitored area or monitors the image data sent from the video camera. The presence or absence of a subject to be monitored may be determined by considering whether there is a subject and whether there is a signal from a behavior recognition device sent from an area where the video camera cannot capture.

If it is determined that there is no monitoring target in the monitored area, the monitoring server terminates the behavior monitoring system (S5). This prevents unnecessary energy consumption by operating the monitoring system even when no monitoring target exists.

When the system is shut down, the user may directly restart the system (step S1), the system may be automatically restarted (step S1) after a predetermined time elapses, and the video camera is maintained at low power. The system may automatically start (step S1) when motion is detected.

If it is determined that there is a monitoring target in the zone to be monitored, it is determined whether there is a monitoring target in the imageable area of the video camera (step S4).

If there is a monitoring target in the imageable area of the video camera, the monitor monitors and recognizes the behavior of the monitoring target (S6). If there is no monitoring target in the imageable area of the video camera, the behavior recognition device attached to the monitoring target recognizes and grasps the target's behavior (step S7).

If there is a monitoring target in the video captured by the video camera, it is automatically recognized, and it is determined that the monitoring target is in the captureable area to recognize and grasp the behavior through the video camera.

If there is no monitoring target in the captured image, it is determined that the monitoring target is out of the captureable area, and the behavior recognition apparatus recognizes and grasps the behavior of the monitoring target.

In this case, the behavior recognition apparatus may initiate the recognition by sending the recognition start notification from the video camera to the behavior recognition apparatus, and the monitoring server that transmits the area departure notification from the video camera to the monitoring server and receives the area departure notification. The behavior recognition device may initiate recognition by sending a notification start notification to the behavior recognition device.

Alternatively, when the position sensor is mounted in the behavior recognition device attached to the body of the subject to be monitored, and the position sensor confirms that the subject is present in the imageable area, the position sensor recognizes the behavior of the subject through the video camera. If it is confirmed that the camera is out of the photographable area, the behavior of the monitoring subject may be recognized by the behavior recognition device.

In this case, by sending the recognition start notification from the behavior recognition device to the video camera, the video camera may be made aware of the behavior of the monitoring target, sending the area entry notification from the behavior recognition device to the monitoring server, and receiving the area entry notification. A monitoring server may send the acknowledgment initiation notification to the video camera to cause the video camera to initiate the acknowledgment.

 Data of the monitoring subject who has recognized the behavior by the video camera or the behavior recognition device is transmitted to the monitoring server through a wireless transceiver installed in each device (step S8).

The video camera may transmit only image data to the monitoring server, or may transmit data obtained by analyzing the captured image to the monitoring server. In the latter case, for example, by including and transmitting the same type of data as the behavior recognition device, the monitoring server may be able to change even if the subject monitoring the monitoring object changes from the video camera to the behavior recognition device or from the behavior recognition device to the video camera. You can also make your data more efficient.

The wireless transceiver of the video camera and the behavior recognition device includes a Zigbee PHY module based on IEEE 802.15.4 and forms a local area network with a monitoring server based on Zigbee Phy.

Therefore, the video camera and the behavior recognition device detect the sensor node of the monitoring server and transmit the data of the monitored target to the monitoring server.

If necessary, a Zigbee Phy-based local area network should be established only between the video camera and the behavioral recognition device, and a wired connection between the video camera and the monitoring server allows the video camera to mediate the connection between the behavioral awareness device and the monitoring server. It may be.

In addition, the behavior recognition device and the video camera may be used at the same time to monitor the behavior of the monitoring target. In this case, the monitoring can be performed more accurately by contrasting the user ID recognized by the behavior recognition device with the monitoring target recognized by the video camera image. For example, if the monitored object in the video and the user ID of the behavior recognition device appear to be inconsistent, the server may be notified to check for possible dangerous situations, and may be caused by changes in the system environment such as lighting. If it is found that an error has occurred, it may be possible to activate an algorithm to recalibrate the system or adjust the settings of the system, such as video cameras or lights, to match the user ID of the behavioral awareness device.

By repeating the above-described process, the behavior monitoring system can monitor the behavior pattern of the monitored subject in real time, and the monitoring server accumulates data transmitted from the video camera or the behavior recognition apparatus, and monitors the behavior taken by the monitored subject for a certain time. You can grasp it concretely.

3 is a block diagram schematically showing the internal configuration of the behavior recognition device 30 according to the present invention.

The acceleration sensor unit 31 of the behavior recognition device includes at least three sensors 31a, 31b, and 31c for detecting accelerations in different directions. The sensor detects acceleration in three X, Y, and Z directions by setting the Z axis perpendicular to the ground with respect to the ground, and two directions perpendicular to the Z axis and perpendicular to each other as the X and Y axes, respectively. The axis acceleration sensor 31a, the Y axis acceleration sensor 31b, and the Z axis acceleration sensor 31c may be used. In addition, the sensor may sense an acceleration in any direction capable of forming a three-dimensional space.

The analog-to-digital conversion unit 33 converts analog data measured by the acceleration sensor unit 31 into digital data.

The controller 35 transmits the digital data of the acceleration converted through the A / D converter 33 through the wireless transceiver 39 or stores it in the storage 37.

The controller 35 communicates with the monitoring server or the video camera through the wireless transceiver 39. The wireless transceiver 39 includes a Zigbee Pi (PHY) module based on IEEE 802.15.4, and a sensor node and a video camera of a monitoring server also include a Zigbee Pie (PHY) module based on IEEE 802.15.4. .

As described above, according to the present invention, communication is performed between the behavior recognition apparatus, the video camera, and the monitoring server of the behavior monitoring system using the IEEE 802.15.4 based Zigbee technology.

Due to the characteristics of Zigbee technology, wireless communication can be carried out with minimal power, which makes it possible to reduce the size, weight, and longevity of the behavior recognition device and the video camera.

The controller 39 automatically transmits the digitally converted acceleration data through the wireless transceiver 39. If the control unit 35 fails to retrieve the sensor node of the monitoring server, the control unit 35 may transmit data to the video camera so that the data is transferred to the monitoring server, or store the data in the storage unit 37. If a certain time elapses, the sensor node of the monitoring server may be rescanned to transmit data.

In addition, the controller 35 compares the reference value of the preset acceleration with the acceleration data digitally converted by the A / D converter 33, and transmits it to the monitoring server only when the acceleration data is greater than or equal to the reference value of the acceleration. It may be stored in the storage unit 37. In this case, the reference value of the acceleration may be set in advance in the controller 35 or the storage 37. Therefore, it is possible to monitor only meaningful behaviors among the monitored subject's behaviors.

In addition, although not shown, the behavior recognition device may further include a data input unit capable of receiving data from the outside, through which the desired acceleration condition may be newly set.

In addition, the control unit 35 stores the main data such as the average value, the maximum value, the period, and the generation time of each direction with respect to the digital values of the accelerations in the different directions transferred from the A / D converter 33. The calculation rule may be set in advance for the control unit 35 or the storage unit 37.

As the storage unit 37, various small memories, including flash memory, may be used. In addition to digitized acceleration data to be transmitted to an external server, the storage unit 37 may select a predetermined or newly set value, for example, an acceleration transmitted or stored as data. It can store various setting values such as acceleration reference value, physical condition such as height or stride length of the monitoring target, time period for searching the monitoring server, detection period, and sampling frequency per time.

The wireless transceiver 39 includes a Zigbee PHY module based on IEEE 802.15.4. The wireless transceiver 39 communicates with a monitoring server or video camera via this Zigbee Phy module.

Since the wireless transceiver 39 includes a Zigbee communication module effective for short-range low-power communication, it is possible to monitor the subject more effectively using this.

Specifically, a plurality of behavioral recognition devices may be attached to different body parts of the same monitoring subject.

The behavior recognition device attached to different body parts transmits the acceleration data measured at each part through each wireless transceiver 39.

At this time, the acceleration reference value that the control unit 35 compares with the measured acceleration data may be set differently for each behavior recognition device according to the attached body part.

The monitoring server may receive cognitive data detected from a plurality of body parts with respect to the same monitored subject, thereby analyzing the subject's behavior more broadly.

4 illustrates an example of a message format transmitted to the monitoring server in the behavior monitoring system according to the present invention.

At the very top of the message format is the sensor ID. The sensor ID is a field for distinguishing whether the data is input from an attached sensor node, that is, data acquired from a behavior recognition device or extracted from an image captured by a video camera. For example, the sensor ID may be displayed as a WSN (Wearable Sensor Node) when the data is input from the behavior recognition device, or as a VSN (Vedio Sensor Node) when the data is obtained from the video camera.

The time stamp indicates the time the data was received from an attached sensor node or video. In order to monitor accurate behavior patterns, it has a structure that records not only the year and month but also the hour and minute in the same way as year: month: day (YY: MM: DD) -hour: minute: second (HH: MM: SS). .

The subsequent null data is entered to sort the data.

The user ID is an ID assigned to each person to be monitored. In case of monitoring a plurality of subjects at the same time, the monitoring server can store and analyze the data transmitted from each behavior recognition device and the video camera for each subject through this user ID.

If a plurality of attached sensor nodes, that is, behavior recognition devices, are used to identify specific behavior patterns of the monitoring target, a lower ID assigned to each attachment site is stored. In other words, if the behavior recognition device is attached to the upper body and the lower body for the monitoring target A, user IDs assigned to the upper body and the lower body of A, respectively, are stored there.

Subsequent null data is allocated as a reserved space for further information.

The behavior type field may include a behavior recognition result of a monitored subject determined through an attached sensor node, that is, a behavior recognition apparatus or a video camera.

The result of the behavior recognition is determined based on a predetermined predetermined value. For example, the acceleration reference value or the acceleration section value of the body part during walking is determined in a statistical manner, and the acceleration value measured for each direction is measured. If it corresponds to a criterion or section, the behavior of the subject to be monitored is determined to be walking, and the type of behavior is displayed as Walk. Similarly, by setting acceleration reference values or interval values corresponding to various behavior types, and comparing the measured acceleration values with each other, it is possible to determine what type of behavior the monitoring subject is taking.

Finally, it may have a recognition probability field. The recognition probability field indicates the recognition rate of the user's behavior recognition result and indicates the reliability of the behavior type indicated by the behavior type field. For example, using a previously calculated distribution of statistics, it is a numerical indication of the degree of confidence that the actual monitored subject's behavior is the same as the indicated behavior type.

The data format having the configuration as shown in FIG. 4 is transmitted to the monitoring server by periodically repeating a predetermined time. Although only specific data as described above is illustrated in FIG. 4, this is only an embodiment of the present invention, and various data may be extracted and transmitted in addition to the illustrated data.

The monitoring server accumulates data transmitted periodically for a certain time, so that the monitoring server can grasp specific and continuous behavior patterns of the actions taken by the monitoring target during the corresponding time, thereby enabling various behavioral analysis.

 As described above, the technical spirit of the present invention has been described in detail through preferred embodiments, but the above-described embodiment is for explaining the technical spirit of the present invention, and the present invention is not limited thereto. In addition, those skilled in the art or similar to the present invention will appreciate that various embodiments are possible within the scope of the technical idea of the present invention.

1 schematically illustrates a monitoring target region and a shootable region of a video camera in a behavior monitoring system according to the present invention.

2 is a flowchart illustrating a method for recognizing and monitoring the behavior of the monitoring target in the behavior monitoring system according to the present invention.

3 is a block diagram schematically illustrating an internal configuration of a behavior recognition apparatus according to the present invention.

4 illustrates an example of a message format transmitted to the monitoring server in the behavior monitoring system according to the present invention.

Claims (11)

A video camera for photographing the area to be monitored and transmitting photographing data; A behavior recognition device attached to the body of the monitoring subject to recognize the behavior of the monitoring subject and to transmit behavior recognition data; And Receives shooting data captured by the video camera when the monitoring target is within the recordable area of the video camera, and recognizes that the monitoring target is attached when the monitoring target is outside the recordable area of the video camera. A monitoring server that receives behavioral awareness data about the monitored person from the device, The video camera sends a notification start notification to the behavior recognizing apparatus to which the monitoring target is attached when the monitoring target is out of the imageable area, And when the behavior recognition apparatus receives the notification of initiation from the video camera, collects behavior awareness data for the monitored subject and starts transmitting to the monitoring server. delete The method of claim 1, The video camera transmits an area departure notification to the monitoring server when the monitoring target is out of the photographable area. The monitoring server receives an out of zone notification from the video camera, and transmits a recognition start notification to the behavior recognition apparatus to which the monitoring target is attached, And the behavior recognition apparatus, when receiving the notification of initiation notification from the monitoring server, collects behavior awareness data for the monitored subject and starts transmitting to the monitoring server. The method of claim 1, The behavior recognition device transmits a recognition start notification to the video camera when the monitoring target is in the photographable area of the video camera, And when the video camera receives the notification of initiation notification from the behavior recognition apparatus, the video camera starts to transmit photographing data about the monitored person to the monitoring server. The method of claim 1, The behavior recognizing apparatus transmits an area entry notification to the monitoring server when the monitoring target is in the photographable area of the video camera, The monitoring server receives an area entry notification from the behavior recognition device, sends a recognition start notification to the video camera, And the video camera, upon receiving a recognition start notification from the monitoring server, starts to transmit photographing data for the monitoring target to the monitoring server. The method according to any one of claims 1 and 3 to 5, The behavior recognition device comprises at least three sensors for measuring acceleration in different directions. Initializing data of the video camera and the behavior recognition device; Determining whether a monitoring target exists in a captureable area of the video camera; When the monitoring target is within the recordable area of the video camera, the video camera captures the monitored target, and when the monitoring target is out of the recordable area of the video camera, the behavior recognition device performs the action of the monitored target. Detecting cognition; And And transmitting the data about the monitored subject detected or perceived to a monitoring server, Between the determination step and the photographing / detecting step, If it is determined that the monitoring target is out of the captureable area of the video camera, the video camera further comprises the step of transmitting a recognition start notification to the behavior recognition device,  Behavior monitoring method. delete The method of claim 7, wherein Between the determination step and the photographing / detecting step, If it is determined that the monitoring target is out of the captureable area of the video camera, the video camera transmitting an area departure notification to the monitoring server; And And sending, by the monitoring server receiving the deviation notification, the recognition initiation notification to the behavior recognition device. The method of claim 7, wherein Between the determination step and the photographing / detecting step, And if it is determined that the monitoring target is in a photographable area of the video camera, the behavior recognizing apparatus further comprises transmitting a recognition start notification to the video camera. The method of claim 7, wherein Between the determination step and the photographing / detecting step, As a result of the determination, if it is determined that the monitoring target is in the photographable area of the video camera, the behavior recognizing apparatus transmits an area entry notification to the monitoring server; And And the monitoring server receiving the area entry notification sends a recognition start notification to the video camera.

Images