KR20230115231A - Intruder location-based dynamic virtual fence configuration and multiple image sensor device operation method - Google Patents

Abstract

A method for constructing a dynamic virtual fence based on an intruder's location and operating a multi-image sensor device is disclosed. A virtual fence-based intruder monitoring method according to an embodiment includes an operation of detecting an intruder invading a virtual fence set in a protection area using a first image sensor device, and the first image sensor device based on information about the intruder. It may include an operation of tracking the intruder through cooperation of second image sensor devices adjacent to .

Description

Intruder location-based dynamic virtual fence configuration and multi-image sensor device operating method

The disclosure below relates to a method for constructing a dynamic virtual fence based on an intruder's location and operating a multi-image sensor device.

In order to protect the country's major facilities or industrial facilities from outside intruders, it is necessary to set up a virtual surveillance line that defends the protected facilities, detect intruders who cross this virtual line, perform tracking, and implement a response strategy. In order to monitor such an outside intruder, an image sensor composed of a camera equipped with a magnification lens and an image processing device is mainly used, but the surveillance area is limited depending on the characteristics of the magnification lens and the image sensor due to the optical characteristics of the camera. It is necessary to use this effectively to set up and operate the surveillance area.

The background art described above is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and cannot necessarily be said to be known art disclosed to the general public prior to the present application.

In order to protect major facilities from outside intruders, a system that installs a virtual fence outside the facility with an image sensor, detects an intruder that crosses the virtual fence with image deep learning technology, and continuously tracks the detected intruder. need to be built At this time, in order to detect an intruder, the magnification of the zoom lens of the camera must be adjusted so that the image of the intruding object has pixels of a certain size or more, and the observation direction of the camera must be controlled (eg, PT control (panning tilting)). Since the direction of the camera and the magnification of the lens must be adjusted according to the path the detected intruder moves, as the initially set observation range of the camera changes, a blind spot in observation occurs, so a technique to overcome this is required. .

In one embodiment, a virtual fence is installed based on a plurality of image sensors to protect human and material assets from outside intruders, and a camera is installed when an intrusive object (eg, intruder) is detected and the intrusive object is continuously tracked. To overcome surveillance blind spots (or observation blind spots) for intruding objects caused by changes in the observation range of the system, use redundant image sensor devices or dynamically configure and operate virtual fences based on cooperation between adjacent image sensor devices. can provide a way to do it.

An embodiment is a method for dynamically configuring and operating a virtual surveillance line (eg, a virtual fence) through cooperation of a plurality of image sensor devices by analyzing the movement characteristics of an intruding object (eg, an intruder) intruding into a surveillance area. can provide

According to an embodiment, a virtual fence is set in the image sensor device and the virtual fence is dynamically reset according to the moving path of the intruder, so that monitoring of the intruder may not be missed.

An embodiment can track multiple intruders by reducing deep learning inference time by classifying and performing intruder detection and tracking deep learning engine.

In one embodiment, in order to distinguish multiple intruders, location information of an intruder who first appears in a protected area is set as a unique identifier, and the intruder's location information is continuously managed as a data attribute, so that multiple intruders can be identified and tracked. .

However, the technical challenges are not limited to the above-described technical challenges, and other technical challenges may exist.

A virtual fence-based intruder monitoring method according to an embodiment includes an operation of detecting an intruder invading a virtual fence set in a protection area using a first image sensor device, and the first image sensor device based on information about the intruder. It may include an operation of tracking the intruder through cooperation of second image sensor devices adjacent to .

The method may further include dynamically setting the virtual fence through cooperation of the second image sensor devices according to controlling the PTZ of the first image sensor device.

The method may further include setting, by the first image sensor device, initial location information at which the intruder is detected as an identifier for distinguishing the intruder.

The tracking operation may include selecting an image sensor device to track the intruder from among the second image sensor devices based on information about the intruder when the intruder is out of the observation range of the first image sensor device. and assigning a task of tracking the intruder to the selected image sensor device.

The tracking operation may further include calculating a Neighborhood Takeover Region, which is a blind spot of the first image sensor device, generated as the first image sensor device tracks the intruder.

The Neighborhood Takeover Region may be a difference between an initial observation range of the first image sensor device and an updated observation range of the first image sensor device that is updated while tracking the intruder.

The tracking operation may be performed based on the time when the intruder leaves the observation range of the first image sensor device and the time required for the second image sensor devices to observe the Neighborhood Takeover Region by controlling the PTZ. An operation of setting a cooperative distance between the sensor device and the second image sensor devices may be further included.

The operation of selecting the second image by comparing the time required for the intruder to leave the observation range of the first image sensor device and the time required for the second image sensor device to control the PTZ to observe the Neighborhood Takeover Region. An operation of selecting an image sensor device that takes less time from among sensor devices may be included.

The method may further include an operation of allocating a task of tracking the intruder to a spare image sensor device when the intruder cannot be tracked through cooperation of the second image sensor devices.

A method for monitoring an intruder based on a virtual fence according to an embodiment includes an operation of detecting an object invading a virtual fence set in a protection area, an operation of tracking the intruder in response when the detected object is an intruder, and an operation of tracking the intruder transmitted from a control center. and controlling a PTZ of the image sensor device according to a command to control an observation range of the image sensor device, wherein the virtual fence controls the PTZ of the image sensor device so that the image sensor devices adjacent to the image sensor device It may be dynamically set through cooperation.

An apparatus for monitoring an intruder based on a virtual fence according to an embodiment includes a processor, and a memory electrically connected to the processor and storing instructions executable by the processor, when the instructions are executed by the processor. , The processor performs a plurality of operations, and the plurality of operations include an operation of detecting an intruder invading a virtual fence set in a protection area using a first image sensor device, and an operation of detecting an intruder based on information about the intruder An operation of tracking the intruder through cooperation of second image sensor devices adjacent to the first image sensor device may be included.

The plurality of operations may further include dynamically setting the virtual fence through cooperation of the second image sensor devices according to controlling the PTZ of the first image sensor device.

The plurality of operations may further include setting, by the first image sensor device, initial location information at which the intruder is detected as an identifier for distinguishing the intruder.

The tracking operation may include selecting an image sensor device to track the intruder from among the second image sensor devices based on information about the intruder when the intruder is out of the observation range of the first image sensor device. and assigning a task of tracking the intruder to the selected image sensor device.

The tracking operation may further include calculating a Neighborhood Takeover Region, which is a blind spot of the first image sensor device, generated as the first image sensor device tracks the intruder.

The Neighborhood Takeover Region may be a difference between an initial observation range of the first image sensor device and an updated observation range of the first image sensor device that is updated while tracking the intruder.

The tracking operation may be performed based on the time when the intruder leaves the observation range of the first image sensor device and the time required for the second image sensor devices to observe the Neighborhood Takeover Region by controlling the PTZ. An operation of setting a cooperative distance between the sensor device and the second image sensor devices may be further included.

The operation of selecting the second image by comparing the time required for the intruder to leave the observation range of the first image sensor device and the time required for the second image sensor device to control the PTZ to observe the Neighborhood Takeover Region. An operation of selecting an image sensor device that takes less time from among sensor devices may be included.

The plurality of operations may further include assigning a task of tracking the intruder to a spare image sensor device when the second image sensor devices cannot track the intruder through cooperation.

1 shows an intruder detection system based on a virtual fence according to an embodiment.
2 shows a schematic block diagram of an image sensor device according to an exemplary embodiment.
3 shows a schematic block diagram of a control center according to an embodiment.
4 is a diagram for explaining an operation of operating a multi-image sensor device according to an exemplary embodiment.
5 is a diagram for explaining an operation of dynamically configuring and operating a virtual fence when an intruder invades a protection area according to an embodiment.
6 is a diagram for explaining an operation of configuring an intruder protection area based on a dynamic virtual fence according to an exemplary embodiment.
7 is a diagram for explaining an operation of managing an intruder protection area in a control center according to an exemplary embodiment.
8 is a diagram for explaining a task of tracking an intruder based on cooperation between a plurality of image sensor devices according to an exemplary embodiment.
9 is a flowchart illustrating a mission procedure of an image sensor device for operating a dynamic virtual fence according to an exemplary embodiment.
10 is a flowchart illustrating a task procedure of a control center for operating a dynamic virtual fence according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed and implemented in various forms. Therefore, the form actually implemented is not limited only to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 shows an intruder detection system based on a virtual fence according to an embodiment.

Referring to FIG. 1 , the intruder detection system 10 may include one or more image sensor devices 2000-1, 2000-2, 2000-3, and 2000-4 and a control center 4000. Image sensor devices (2000-1, 2000-2, 2000-3, 2000-4) are installed in the protected area (or surveillance area) to detect intrusive objects (e.g., intruders 200 and 300) invading the protected area (or surveillance area). can be installed The image sensor devices 2000-1, 2000-2, 2000-3, and 2000-4 may include a camera and a processor (eg, a GPU) that performs an image deep learning algorithm. The control center 4000 may control and manage the image sensor devices 2000-1, 2000-2, 2000-3, and 2000-4. Each of the image sensor devices 2000-1, 2000-2, 2000-3, and 2000-4 and the control center 4000 may interwork with each other through a wired or wireless network. The intruder detection system 10 may further include an image learning server 5000 .

The image sensor devices (2000-1, 2000-2, 2000-3, 2000-4) analyze the video of the camera (eg, the video taken by the camera) by deep learning algorithm (eg, Yolo, ResNet, RE3, etc.) It can perform the function of detecting and tracking intruders. In the case of the cameras of the image sensor devices (2000-1, 2000-2, 2000-3, 2000-4), there are restrictions on the distance in detecting and tracking intruders due to optical limitations, so cooperation between cameras is required for surveillance within the protected area. this may be needed

In order to efficiently detect and track intruding objects 200 and 300 invading the protected area, a virtual fence (eg, a virtual surveillance line or fence line) 100 may be set in the protected area. The image sensor devices 2000-1, 2000-2, 2000-3, and 2000-4 may detect and track intruders 200 and 300 through image deep learning of an object crossing the virtual fence 100. Each virtual fence 100 is within the monitoring area boundaries 3110 and 3210, which is the size of the image observed (or acquired) through the cameras of the image sensor devices 2000-1, 2000-2, 2000-3, and 2000-4. (Example: virtual fence A_Image 3120, virtual fence B_Image 3220) is installed, and an object invading the virtual fence 100 is detected and tracked through an image deep learning algorithm. The virtual fence A_Image 3120 is a virtual fence 100 set within the surveillance area boundary 3110 observed through the camera of the image sensor device 2000-1, and the virtual fence B_Image 3220 is the image sensor device 2000-1. It may be the virtual fence 100 set within the surveillance area boundary 3210 observed through the camera of 2).

The image sensor devices (2000-1, 2000-2, 2000-3, 2000-4) have a function of detecting an intruder by downloading and loading a model (eg, image deep learning algorithm) learned from the image learning server 5000. can be done In the process of detecting and tracking an intruder, each image sensor device (2000-1, 2000-2, 2000-3, 2000-4) can control the angle of view (eg, angle of view (2010, 2110)) by adjusting the magnification of the lens. there is.

The control center 4000 manages each of the image sensor devices 2000-1, 2000-2, 2000-3, and 2000-4, and tracks the moving path of an intruder (eg, intruder 200, 300) to the corresponding image sensor. When the device (eg, image sensor device 2000-1, image sensor device 2000-2) is out of the monitoring area, an adjacent image sensor device (eg, image sensor device 2000-3) or a spare image sensor A function of assigning a task to the device to track a specific intruder (eg, intruders 200 and 300) may be performed.

2 shows a schematic block diagram of an image sensor device according to an exemplary embodiment.

Referring to FIG. 2 , according to an embodiment, an image sensor device 2000 (eg, the image sensor devices 2000-1, 2000-2, 2000-3, and 2000-4 of FIG. 1) is an image sensor 2010 ) (eg, a camera), a processor 2020, and a communication unit 2090. The image sensor device 2000 may further include a memory 2095.

Memory 2095 may store instructions (or programs) executable by processor 2020 . For example, the instructions may include instructions for executing an operation of the processor 2020 and/or an operation of each component of the processor 2020 .

Memory 2095 may include one or more computer-readable storage media. The memory 2095 may include non-volatile storage elements (eg, magnetic hard disk, optical disk, floppy disk, flash memory, electrically programmable memories (EPROM), electrically erasable and programmable (EEPROM).

Memory 2095 may be non-transitory media. The term “non-transitory” may indicate that the storage medium is not implemented as a carrier wave or propagated signal. However, the term "non-temporary" should not be interpreted as the memory 2095 being immovable.

The processor 2020 may process data stored in the memory 2095. Processor 2020 may execute computer readable code (eg, software) stored in memory 2095 and instructions invoked by processor 2020 .

The processor 2020 may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. For example, desired operations may include codes or instructions included in a program.

For example, a data processing unit implemented in hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , Application-Specific Integrated Circuit (ASIC), and Field Programmable Gate Array (FPGA).

The processor 2020 includes an object detection unit 2030, a detection object movement path prediction unit 2040, a PTZ control signal generator 2050, detection It may include a detection object location information extraction unit 2060, an object location tracking unit 2070, and an observation range management unit 2080. Object detection unit 2030, detection object movement path prediction unit 2040, PTZ control signal generation unit 2050, detection object location information extraction unit 2060, object location tracking unit 2070, and observation range management unit 2080 may be embedded in the processor 2020 or loaded into the processor 2020 from the memory 2095.

The image sensor 2010 (eg, a camera) may transmit a photographed image to the object detector 2030 . The object detector 2030 may detect an object in an image (or screen) using an image deep learning algorithm and then determine whether the detected object is an intruder. If it is determined that the detected object is an intruder, the object detector 2030 may transmit information about the intruder to the detected object moving path predictor 2040 and the detected object location information extractor 2060 .

The detected object location information extractor 2060 may extract the location information of the detected object by combining the object location on the image (or screen) with pan tilt zoom (PTZ) information mounted in the image sensor device 2000 . PTZ information may be acquired from the image sensor 2010 and/or the image sensor device 2000. The detection object location information extraction unit 2060 may transmit location information of the detection object to the detection object location tracking unit 2070 . That is, the location information of the detection object may be obtained by converting the location of the detection object into GPS location information by combining the size and coordinates of the bounding box of the detection object in the image and the PTZ information of the image sensor 2010 equipped with GPS.

The detection object position tracking unit 2070 may track the position of the detection object in units of frames of an image. The detection object movement path predictor 2040 may control the PTZ of the image sensor 2010 by predicting the movement path of the detection object based on the tracked information. Also, the detection object position tracking unit 2070 may transmit the movement path of the detected object to the control center 4000 through the communication unit 2090 so that the control center 4000 can display the movement path of the detected object. The detection object movement path predictor 2040 generates a movement path (or estimated movement path) of the detection object in real time based on information such as the movement direction and speed of the detection object, and generates a PTZ control signal for the movement path of the detection object. It can be sent to unit 2050.

The PTZ control signal generating unit 2050 may control the image sensor 2010 by generating a PTZ control signal (eg, a PTZ control signal for the image sensor 2010) based on the movement path of the detected object. The PTZ control signal generator 2050 generates a PTZ control signal in response to a control command transmitted from the control center 4000 (eg, a command for the control center 4000 to directly control each image sensor 2010). can

The PTZ control signal generation unit 2050 may also transmit the generated PTZ control signal to the observation range management unit 2080. The observation range management unit 2080 determines the currently observable range of the image sensor 2010 in real time based on the characteristics of the image sensor 2010 (eg, characteristics such as zoom magnification and angle of view) in response to the PTZ control signal. , and the updated information may be transmitted to the control center 4000 through the communication unit 2090.

3 shows a schematic block diagram of a control center according to an embodiment.

Referring to FIG. 3 , according to an embodiment, a control center 4000 (eg, a control server device) may include a communication unit 5010 and a processor 5080. The control center 4000 may further include a memory 5090.

The communication unit 5010 may perform a communication function with the image sensor device 2000 (eg, each image sensor device 2000 - 1 to 2000 - 4 in FIG. 1 ).

Memory 5090 may store instructions (or programs) executable by processor 5080 . For example, the instructions may include instructions for executing an operation of the processor 5080 and/or an operation of each component of the processor 5080 .

Memory 5090 may include one or more computer-readable storage media. The memory 5090 may include non-volatile storage elements (eg, magnetic hard disk, optical disk, floppy disk, flash memory, electrically programmable memories (EPROM), electrically erasable and programmable (EEPROM).

Memory 5090 may be non-transitory media. The term “non-transitory” may indicate that the storage medium is not implemented as a carrier wave or propagated signal. However, the term "non-temporary" should not be interpreted as meaning that the memory 5090 is immovable.

The processor 5080 may process data stored in the memory 5090. Processor 5080 may execute computer readable code (eg, software) stored in memory 5090 and instructions invoked by processor 5080 .

The processor 5080 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include codes or instructions included in a program.

For example, a data processing unit implemented in hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , Application-Specific Integrated Circuit (ASIC), and Field Programmable Gate Array (FPGA).

The processor 5080 includes a control message processing unit 5020, an intruder tracking unit 5030, a surveillance area management unit 5050, and an image sensor cooperation unit 5050. unit 5060, and an intruder management unit 5070. The control message processing unit 5020, intruder tracking unit 5030, surveillance area management unit 5050, image sensor cooperation unit 5060, and intruder management unit 5070 are embedded in the processor 5080 or from the memory 5090 to the processor 5080. ) can be loaded.

The control message processing unit 5020 may generate a message (eg, command) for controlling the image sensor 2010 of the image sensor device 2000 and transmit the message to the image sensor device 2000 through the communication unit 5010. . The intruder tracking unit 5030 may track the location of a detection object (eg, intruder). The monitoring area manager 5050 may manage the monitoring area of each image sensor device 2000 (eg, the image sensor devices 2000-1 to 2000-4 of FIG. 1) in real time. The image sensor cooperation unit 5060 continuously tracks the intruder when the intruder detected by the image sensor device (eg, the image sensor device A (2000-1) of FIG. 1) moves and moves out of the observation range of the corresponding image sensor device. For this purpose, another image sensor device (eg, image sensor device C 2000-3 of FIG. 1) may be given a corresponding task. The intruder management unit 5070 may assign a unique identifier to an intruder appearing in a monitoring area (eg, an intruder monitoring area) and manage the intruder. For example, a method of assigning a unique identifier to each intruder may include a method of selecting a GPS time when an intruder is detected for the first time as a unique identifier and continuously managing a movement path.

4 is a diagram for explaining an operation of operating a multi-image sensor device according to an exemplary embodiment.

FIG. 4 shows a method for operating an image sensor device within an observation range (or photographing range) 2500 of image sensor device A 2000-1 and an observation range 2600 of image sensor device B 2000-2 of FIG. 1 . It may be to describe the action.

An image sensor (e.g., the image sensor 2010 of FIG. 2), that is, a camera sets a narrow-angle screen 2530 that enlarges the size of an image of a subject by increasing the magnification of the lens from the wide-angle screen 2510 where the magnification of the lens is 1. It is possible to obtain an image for detecting an intruder. In the wide-angle screen 2510, the range of observing the intruder is wide, but the size of the intruder in the image is small, so it may be difficult to detect the intruder by image deep learning. While the image size of the subject is increased, the observation range may be reduced.

Referring to FIG. 4 , an image sensor device (eg, image sensor device A 2000-1 or image sensor device B 2000-2) adjusts the magnification of a lens of an image sensor 2010 to display an initial setting screen 2520. ) (e.g. initial surveillance area) can be set. After setup, the image sensor device (e.g. image sensor device A (2000-1), image sensor device B (2000-2)) starts the task of detecting an intruder (e.g. detection object) and produces a clear image of the intruder. or track the intruder as it moves. A screen of the image sensor 2010 when panning tilting zooming (PTZ) of the image sensor 2010 is controlled for tracking may be the screen 2530 . At this time, as the screen of the image sensor 2010 becomes smaller (eg, 2520 -> 2530), a difference from the original wide-angle screen 2510 occurs, and this may be referred to as a tolerance area 2550 (eg, an observation range). That is, the image sensor devices (eg, image sensor device A (2000-1) and image sensor device B (2000-2)) freely control the PTZ of the image sensor 2010 within the range of this tolerance area (2550). Intruders can be detected or tracked. Neighborhood takeover (handover) region 6710 of FIG. 8 or third party takeover (handover) region 6200 may be determined within the tolerance area 2550.

Considering the environment in which a plurality of image sensor devices (e.g., image sensor device A (2000-1) and image sensor device B (2000-2)) are installed and operated, the observation range of the image sensor device A (2000-1) If (2500) is reduced and an observation blind spot occurs, in this visual zone, the image sensor device B (2000-2) (eg, the image sensor device adjacent to the image sensor device A (2000-1)) replaces the intruder surveillance task. can be performed.

In an environment where a plurality of image sensor devices are operated, a cooperative distance 2560 between adjacent image sensor devices may be set. The cooperative distance 2560 is the time t_out when the intruder leaves the narrow angle screen 2530 based on the average moving speed of the intruder(s) and the image sensor device (eg, the image sensor device B (2000-2)) adjacent to the image sensor By comparing the stabilization time t_setup required to observe the tolerance area 2550 of the image sensor device A (2000-1) by controlling the PTZ of (2010), t_setup may be set within a range less than t_out. On the other hand, when the lens magnification of the image sensor 2010 is controlled, a clear image can be obtained only within the range of the depth of field 2540 of the image sensor 2010. A preemptive control method may be applied.

5 is a diagram for explaining an operation of dynamically configuring and operating a virtual fence when an intruder invades a protection area according to an embodiment.

At time t1, the image sensor device A (2000-1) having a variable view angle 2010 configures the virtual fence A_Image_t1 (3121) to monitor the intruder 200, and the image sensor device B (2000-2) constructs the virtual fence B_Image_t1 3221 can be configured to monitor the intruder 200.

The camera angle of view of the image sensor device 2000-1 that monitors the intruder 200 whose location information is "location information_t1 (Xt1, Yt1, Zt1) 201" at time t1 is FoV_t1 (2011), and the image sensor The apparatus 2000-1 may detect the intruder 200 through deep learning on a captured image. Thereafter, the intruder crosses the virtual fence (e.g., the virtual fence 100 of FIG. 1) and continues invading, and when the location information_t2 (Xt2, Yt2, Zt2) 202 is obtained at time t2, the image sensor device A (2000-1) may adjust the angle of view of the image sensor 2010 to FoV_t2 (2012) in order to enlarge the image of the intruder 200 for the purpose of continuously and precisely tracking the intruder 200. While the magnification of the lens of the image sensor 2010 is increased to obtain a large image, the angle of view of the image sensor device A (2000-1) is reduced to FoV_t2 (2012) and the monitoring area for the virtual fence is also reduced, so that the image sensor device A ( 2000-1) may configure a virtual fence A_Image_t2 (3122).

To compensate for the reduced surveillance area of the image sensor device A (2000-1), the image sensor device B (2000-2) controls the PTZ of the image sensor 2010 included in the image sensor device 2000-2. Thus, the virtual fence B_Image_t2 (3222) can be configured. Accordingly, in order to compensate for the reduction of the monitoring area, the image sensor device C (2000-3) also controls the PTZ of the image sensor 2010 included in the image sensor device C (2000-3) to configure a virtual fence C_Image_t2 (3322). can do. In this way, the virtual fence (eg, the virtual fence 100 of FIG. 1 ) may be changed from the original (eg, time t1 ) straight line shape to a combination of a straight line and an oblique line.

After that, when an intruder continuously invades the inside of the protected area, the image sensor device A (2000-1) adjusts the angle of view of the image sensor 2010 of the image sensor device A (2000-1) to FoV_t3 (2013), so that the virtual A fence A_Image_t3 (3123) can be configured. Accordingly, in order to compensate for the surveillance area of the image sensor device A (2000-1), the image sensor device B (2000-2) sets the PTZ of the image sensor 2010 included in the image sensor device B (2000-2). The virtual fence B_Image_t3 (3222) is formed by controlling the virtual fence, and the image sensor device C (2000-3) also controls the PTZ of the image sensor 2010 included in the image sensor device C (2000-3) to form the virtual fence C_Image_t3 (3223). can be configured. At this time, the location information of the intruder 200 may be location information_t3 (Xt3, Yt3, Zt3) 203.

6 is a diagram for explaining an operation of configuring an intruder protection area based on a dynamic virtual fence according to an exemplary embodiment.

FIG. 6 may show surveillance areas 3100, 3200, and 3300 in charge of each of the image sensor devices 2000, 2100, and 2200 when configuring an intruder protection area based on the operation of the dynamic virtual fence described in FIG. 5. . The monitoring area 3100 of the image sensor device A (2000-1) is defined by the boundary line 3110 of the monitoring area of the image sensor device A (2000-1), the area of the monitoring area A (3120), and the observation time "virtual fence_Image_t1". (3121), virtual fence_Image_t2 (3122), and virtual fence_Image_t3 (3123). Similarly, the area of the monitoring area 3200 of the image sensor device B 2000-2 is the same as the area B 3220 of the monitoring area, and the area of the monitoring area 3300 of the image sensor device C 2000-3 The area may be equal to the surveillance zone area C 3320 .

The image captured by the image sensor 2010 of each image sensor device 2000-1 to 2000-3 has a clear contour only for an object within the depth of field of the image sensor 2010, so the image is deep. An area in which an object can be detected by running may be determined. An observation range (eg, a camera observation range) of the image sensor devices 2000-1 to 2000-3 may be determined according to a PTZ control value of the image sensor 2010 included in the corresponding device.

At time t1, the camera PTZ control value of image sensor device A (2000-1) is Ca_A_PTZ_t1 = (PA1, TA1, ZA1) (2051), and the camera PTZ control value of image sensor device B (2000-2) is Ca_B_PTZ_t1 = (PB1, TB1, ZB1) (2151). At time t2, the PTZ control value of the camera of the image sensor device A (2000-1) is Ca_A_PTZ_t2 = (PA2, TA2, ZA2) (2052), and the PTZ control value of the camera of the image sensor device B (2000-2) is Ca_B_PTZ_t2 = (PB2, TB2, ZB2) 2152, and the camera PTZ control value of the image sensor device C 2000-3 may be Ca_C_PTZ_t2 = (PC2, TC2, ZC2) 2252. At time t3, the PTZ control value of the camera of the image sensor device A (2000-1) is Ca_A_PTZ_t3 = (PA3, TA3, ZA3) (2053), and the PTZ control value of the camera of the image sensor device B (2000-2) is Ca_B_PTZ_t3 = (PB3, TB3, ZB3) 2153, and the camera PTZ control value of the image sensor device C (2000-3) may be equal to Ca_C_PTZ_t3 = (PC3, TC3, ZC3) 2253. The range in which the image sensor devices 2000-1 to 2000-3 can detect intruders, track and classify, and extract abnormal behaviors may be the same as the range 6000.

As shown in FIG. 6 , at time t1, a virtual fence_Image_t1 3121 corresponding to the range 6100 capable of detecting and tracking intruding objects is set, and at time t2, the range capable of classifying and tracking intruding objects ( 6200), a virtual fence_Image_t2 (3122) is set, and at time t3, the virtual fence_Image_t3 ( 3123) may be set.

7 is a diagram for explaining an operation of managing an intruder protection area in a control center according to an exemplary embodiment.

The control center 4000 sets monitoring areas of image sensor devices 2000, 2100, 2200, and 2300 and virtual fences 3120, 3220, 3320, and 3420 that detect objects (eg, intruders) intruding into an intruder protection area. can control. In addition, the control center 4000 may perform a function for tracking the movement of the detected intruders 200, 210, and 220.

In FIG. 7, the area of the entire monitoring area 3000 in the intruder protection area is S, and the areas monitored by each of the image sensor devices 2000-1 to 2000-4 are A (3130), B (3230), and C ( 3330), and D (3430). The image sensors 2010 of the image sensor devices 2000-1 to 2000-4 may be set so that the sum of areas monitored by the image sensor devices 2000-1 to 2000-4 is S.

Assuming that the characteristics of the image sensor 2010 of each image sensor device 2000-1 to 2000-4 installed in the intruder protection area are the same, considering the depth of field and the magnification of the lens of the image sensor 2010, the image A limit at which the sensor devices 2000-1 to 2000-4 can monitor an intruder may be a monitoring depth (eg, distance D) 3500. Each of the image sensor devices 2000-1 to 2000-4 and the control center 4000 may perform a function of detecting and tracking an intruder through mutual cooperation as in the following example scenario.

① At time t1, intruder #1 (200) appears within the surveillance area boundary 3110 of the video sensor device A (2000-1), and intruder #2 (210) monitors the video sensor device D (2000-4) It is assumed that the intruder #3 (220) appears within the border line 3410 of the area and that the intruder #3 (220) appears within the border line 3310 of the monitoring area of the image sensor device C (2000-3). Each of the image sensor devices 2000-1, 2000-3, and 2000-4 detects an intruder through image deep learning since each intruder 200, 210, and 220 exists within the virtual fence 3120, 3420, and 3320, Location information of the detected intruder (e.g. location information (X1, Y1, Z1) of intruder #1 (200), location information (X2, Y2, Z2) of intruder #2 (210), location information (X2, Y2, Z2) of intruder #3 (220) After extracting the location information (X3, Y3, Z3), continuously tracking the movement path of the intruder (200, 210, 220) detected based on the frame of the image, information (e.g., tracked information) about this is sent to the control center ( 4000). The control center 4000 may classify the entire monitoring area into areas monitored by each of the image sensor devices 2000-1 to 2000-4 and manage them. In addition, the control center 4000 may use the location information of the intruder detected by each of the image sensor devices 2000-1, 2000-3, and 2000-4 as an intruder identification identifier and manage the intruder as follows.

- Identifier of intruder #1 (200): A_X1, Y1, Z1, attributes of intruder #1 (200): A, X1_t1, Y1_t1, Z1_t1

- Identifier of intruder #2 (210): D_X2, Y2, Z2, attributes of intruder #2 (210): D, X2_t1, Y2_t1, Z2_t1

- Identifier of intruder #3 (220): C_X3, Y3, Z3, attribute of intruder #3 (220): C, X3_t1, Y3_t1, Z3_t1

② At time t2, the intruder #1 (200) moves into the boundary line 3210 of the surveillance area of the video sensor device B (2000-2), and the intruder #2 (210) also monitors the video sensor device B (2000-2). It is assumed that the intruder #3 (220) moves within the area boundary 3210 and the intruder #3 (220) moves within the surveillance area boundary 3110 of the image sensor device A (2000-1). Each of the image sensor devices 2000-1 to 2000-4 may continuously track and transmit the moving direction and speed of the detected object (eg, intruder) to the control center 4000 after time t1 at which the intruder is detected. The control center 4000 determines which image sensor device (eg, image sensor device A (2000-1) or image sensor device B (2000-2)) the intruder is moving in the direction of the surveillance area, and determines which image sensor device (eg, image sensor device (2000-2)) : Image sensor device A (2000-1) and image sensor device B (2000-2) are notified of this (eg, intruder tracking handover start message), and the notified image sensor device (eg, image sensor device A ( 2000-1) and the image sensor device B (2000-2)) may transmit an approval message (eg, intruder tracking handover approval) to the control center 4000 when it is determined that tracking of the intruder is possible. Thereafter, the image sensor devices (eg, image sensor device A (2000-1) and image sensor device B (2000-2)) that have approved the intruder tracking handover refer to this when controlling the PTZ of the image sensor 2010. Thus, the handed over intruder can be continuously tracked and the new intruder can be monitored at the same time. In this case, attributes of the intruder managed by the control center 4000 may be as follows.

- Identifier of intruder #1 (200): A_X1, Y1, Z1, attributes of intruder #1 (200): A_B, X1_t2, Y1_t2, Z1_t2

- Identifier of intruder #2 (210): D_X2, Y2, Z2, attribute of intruder #2 (210): D_B, X2_t2, Y2_t2, Z2_t2

- Identifier of intruder #3 (220): C_X3, Y3, Z3, attribute of intruder #3 (220): C_A, X3_t2, Y3_t2, Z3_t2

8 is a diagram for explaining a task of tracking an intruder based on cooperation between a plurality of image sensor devices according to an exemplary embodiment.

As shown in FIG. 8 , the initial monitoring area 6300 of the image sensor device B (2000-2) is the monitoring area 6200 of the image sensor device A (2000-1) and the image sensor device C (2000-3). may be adjacent to the monitoring area 6400 of ) and the monitoring area 6500 of the image sensor device D (2000-4). When an intruder appears in the monitoring area 6300, the image sensor device B (2000-2) adjusts the magnification of the lens and the PTZ of the image sensor 2010 so that the screen of the image sensor 2010 is reduced in the process of detecting the intruder. As a result, a blind spot in the monitoring area 6300 may occur. The blind area can be supplemented through cooperation with the image sensor 2010 of the neighboring image sensor devices (2000-1, 2000-3, 2000-4), and this blind area is referred to as the Neighborhood takeover (or handover) region (6110, 6120). , 6130).

The control center 4000 monitors the status of neighboring image sensor devices 2000-1, 2000-3, and 2000-4 (eg, presence or absence of intruder detection and tracking, current Camera observation range) is analyzed and a PTZ control command is given to the adjacent image sensor devices (2000-1, 2000-3, 2000-4) in the blind spot to monitor including the Neighborhood takeover (handover) region (6110, 6120, 6130). Blind spots can be eliminated by transmission. Each image sensor device (2000-1, 2000-3, 2000-4) analyzes the video frame by frame in response to the PTZ control command transmitted from the control center (4000), estimates the moving direction of the intruder, and then detects the intruder Information about the intruder (eg, movement direction, movement speed, current coordinate information) of the intruder may be provided to the control center 4000 . The control center 4000 receiving the information about the intruder sets up a target image sensor device to continuously track the intruder when the intruder leaves the area of the current observation image sensor device (eg, image sensor device B (2000-2)). After selection, the intruder tracking mission can be assigned to the selected image sensor device.

In FIG. 8 , when an intruder moves in the direction of ①, the image sensor device D (2000-4) is assigned a task of tracking the intruder from the observation center 4000 and may perform the task of tracking the intruder. If the intruder moves in the direction of ② and there is no video sensor device nearby, the control center (4000) will issue a surveillance command to the spare video sensor device R to monitor the Third party Takeover (Handover) Region (6200). can As such, the virtual fence operation area 6700, the neighborhood takeover (handover) operation limit 6710, and the third party takeover (handover) operation limit 6720 may be as shown in FIG. 8 .

In addition, the observation mission of the image sensor device for monitoring the intruder as the intruder moves may be performed as follows.

① When the observation range (area) of the image sensor device is reduced within a certain threshold value (S _Th1 ),

- When the observation range, which is reduced according to the moving speed and direction of the intruder, is within the range (S _{Th_N} ) in which the intruder can be tracked without the cooperation of the adjacent image sensor device for a certain period of time, the image sensor device can continuously track the intruder. there is.

[Equation 1]

- When the observation range, which is reduced according to the moving speed and direction of the intruder, is reduced to more than the range (S _{Th_N} ) in which the intruder can be tracked only through the cooperation of the neighboring image sensor device, in the Neighborhood takeover (handover) region, the adjacent image sensor device It can perform missions to track intruders.

[Equation 2]

② When the observation range (area) of the image sensor device is reduced beyond a certain threshold value (S _Th1 ),

- The control center (4000) can perform the task of tracking intruders through a spare image sensor device to monitor the Third party Takeover (Handover) Region.

9 is a flowchart illustrating a mission procedure of an image sensor device for operating a dynamic virtual fence according to an exemplary embodiment.

In operation 8010, the image sensor device (eg, the image sensor devices 2000-1 to 2000-4 of FIG. 1) detects an object by performing deep learning on the image acquired by the image sensor 2010, and detects the object. It is possible to determine whether or not is an intruder.

In operation 8020, when the detection object is an intruder, the image sensor device 2000 may estimate information about the intruder (eg, the intruder's location and moving direction) based on the image frame.

In operation 8030, the image sensor device 2000 may track the intruder by controlling the PTZ of the image sensor 2010 according to information about the intruder (8030).

In operation 8040, the image sensor device 2000 may transmit PTZ information to the control center 4000. The control center 4000 may manage the observation range of each image sensor device 2000 (eg, image sensor devices 2000-1 to 2000-4) based on the PTZ information.

In operation 8050, it is possible to analyze (eg, determine) whether an intruder leaves the surveillance area of the image sensor device 2000 that is currently being detected. When not leaving the monitoring area, the image sensor device 2000 may continuously track the intruder and transmit information about the intruder to the control center (8020 to 8050).

In operation 8060, when the intruder leaves the monitoring area, the image sensor device 2000 transmits information about the intruder to the control center 4000 in order to allocate the intruder's tracking task to an adjacent image sensor device and/or a spare image sensor device. Information (eg, direction and speed of movement of an intruder) and PTZ information can be transmitted.

In operation 8070, the image sensor device 92000 may wait for a command from the control center 4000. The image sensor device 2000 may continuously detect and track an object while waiting.

10 is a flowchart illustrating a task procedure of a control center for operating a dynamic virtual fence according to an embodiment.

In operation 8510, the control center 4000 may initially allocate an observation range of each image sensor device (eg, image sensor devices 2000-1 to 2000-4 of FIG. 1).

In operation 8520, the control center 4000 may receive PTZ information transmitted by each of the image sensor devices 2000-1 to 2000-4.

In operation 8530, the control center 4000 may continuously update the observation range of each image sensor device 2000-1 to 2000-4 based on the PTZ information.

In operation 8540, the control center 4000 may calculate a Neighborhood Takeover (Handover) Region, which is a blind spot generated in the process of each image sensor device 2000-1 to 2000-4 detecting and tracking an intruder. A difference between the initial observation range of the image sensor devices 2000-1 to 2000-4 and the updated observation range of the image sensor devices 2000-1 to 2000-4 may be a neighborhood takeover (handover) region.

In operation 8550, the control center 4000 may analyze the observation ranges of the adjacent image sensor devices 2000-1 to 2000-4.

In operation 8560, the control center 4000 may analyze whether the neighboring image sensor devices 2000-1 to 2000-4 can take charge of the Neighborhood Takeover (Handover) Region based on the analysis result. A parameter for setting a takeover (handover) criterion of the adjacent image sensor device 4000 may be determined as follows.

(1) Moving direction of intruder: Select an image sensor device adjacent to the moving direction of the intruder as a handover target image sensor device

(2) Moving speed of intruder: The time taken to estimate the moving speed of the intruder and get out of the observation range of the current image sensor device is I_t, and it takes the neighboring image sensor device to control the PTZ to observe the Neighborhood Takeover (Handover) Region. When the set-up time is N_S_t,

-I_t In case of N_S_t: Perform Neighborhood Takeover (Handover) procedure

(3) When R_S_t is the stabilization time by controlling the PTZ for the spare image sensor device to observe the Neighborhood Takeover (Handover) Region,

- N_S_t In case of R_S_t: Perform third party takeover (handover) region procedure

In operation 8570, the control center 4000 transmits Neighborhood Takeover (Handover) Region information to a neighboring image sensor device when a Neighborhood Takeover (Handover) procedure is required to assign an intruder tracking task.

In operation 8580, if the camera setup time required to track the intruder with the adjacent image sensor device is long, or if there is no adjacent image sensor device, information about the intruder (eg, moving direction and moving speed of the intruder) is continuously obtained from the image sensor device and PTZ information can be received.

In operations 8590 and 8595, the control center 4000 calculates a third party takeover (handover) region and transmits information thereon to a preliminary image sensor device to assign a task of tracking an intruder.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. A computer readable medium may store program instructions, data files, data structures, etc. alone or in combination, and program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques may be performed in an order different from the method described, and/or the components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (19)

In the virtual fence-based intruder monitoring method,
detecting an intruder invading a virtual fence set in a protection area by using a first image sensor device; and
An operation of tracking the intruder through cooperation of second image sensor devices adjacent to the first image sensor device based on the information about the intruder.
Including, intruder monitoring method.
According to claim 1,
An operation of dynamically setting the virtual fence through cooperation of the second image sensor devices as the PTZ of the first image sensor device is controlled.
Further comprising, intruder monitoring method.
According to claim 1,
An operation of setting, by the first image sensor device, the initial location information at which the intruder is detected as an identifier for distinguishing the intruder.
Further comprising, intruder monitoring method.
According to claim 1,
The tracking operation is
selecting an image sensor device to track the intruder from among the second image sensor devices based on information about the intruder when the intruder is out of the observation range of the first image sensor device;
Allocating the task of tracking the intruder to the selected image sensor device
Including, intruder monitoring method.
According to claim 4,
The tracking operation is
An operation of calculating a Neighborhood Takeover Region, which is a blind spot of the first image sensor device, generated as the first image sensor device tracks the intruder.
Further comprising, intruder monitoring method.
According to claim 5,
The Neighborhood Takeover Region,
A difference between an initial observation range of the first image sensor device and an updated observation range of the first image sensor device updated as the intruder is tracked.
According to claim 5,
The tracking operation is
Based on the time required for the intruder to leave the observation range of the first image sensor device and the time required for the second image sensor devices to observe the Neighborhood Takeover Region by controlling the PTZ, the first image sensor device and the second image sensor device Operation of setting a cooperative distance between image sensor devices
Further comprising, intruder monitoring method.
According to claim 5,
The selection operation is
The time required for the intruder to leave the observation range of the first image sensor device and the time required for the second image sensor devices to control the PTZ to observe the Neighborhood Takeover Region are compared to obtain the Operation of selecting an image sensor device with less time
Including, intruder monitoring method.
According to claim 1,
Assigning a task of tracking the intruder to a spare image sensor device when the intruder cannot be tracked through cooperation of the second image sensor devices
Further comprising, intruder monitoring method.
In the virtual fence-based intruder monitoring method,
detecting an object invading a virtual fence set in the protected area;
tracking the intruder in response when the detection object is the intruder; and
An operation of controlling the observation range of the image sensor device by controlling the PTZ of the image sensor device according to a command transmitted from the control center.
including,
The virtual fence,
In accordance with controlling the PTZ of the image sensor device, the intruder monitoring method is dynamically set through cooperation of image sensor devices adjacent to the image sensor device.
Further comprising, intruder monitoring method.
An apparatus for monitoring an intruder based on a virtual fence,
processor; and
A memory electrically connected to the processor and storing instructions executable by the processor.
including,
When the instructions are executed by the processor, the processor performs a plurality of operations;
The plurality of operations,
detecting an intruder invading a virtual fence set in a protection area by using a first image sensor device; and
An operation of tracking the intruder through cooperation of second image sensor devices adjacent to the first image sensor device based on the information about the intruder.
Including, device.
According to claim 11,
The plurality of operations,
An operation of dynamically setting the virtual fence through cooperation of the second image sensor devices as the PTZ of the first image sensor device is controlled.
Further comprising a device.
According to claim 11,
The plurality of operations,
An operation of setting, by the first image sensor device, the initial location information at which the intruder is detected as an identifier for distinguishing the intruder.
Further comprising a device.
According to claim 11,
The tracking operation is
selecting an image sensor device to track the intruder from among the second image sensor devices based on information about the intruder when the intruder is out of the observation range of the first image sensor device;
Allocating the task of tracking the intruder to the selected image sensor device
Including, device.
According to claim 14,
The tracking operation is
An operation of calculating a Neighborhood Takeover Region, which is a blind spot of the first image sensor device, generated as the first image sensor device tracks the intruder.
Further comprising a device.
According to claim 15,
The Neighborhood Takeover Region,
A difference between an initial observation range of the first image sensor device and an updated observation range of the first image sensor device updated as the intruder is tracked.
According to claim 15,
The tracking operation is
Based on the time required for the intruder to leave the observation range of the first image sensor device and the time required for the second image sensor devices to observe the Neighborhood Takeover Region by controlling the PTZ, the first image sensor device and the second image sensor device Operation of setting a cooperative distance between image sensor devices
Further comprising a device.
According to claim 15,
The selection operation is
The time required for the intruder to leave the observation range of the first image sensor device and the time required for the second image sensor devices to control the PTZ to observe the Neighborhood Takeover Region are compared to obtain the Operation of selecting an image sensor device with less time
Including, device.
According to claim 11,
The plurality of operations,
Assigning a task of tracking the intruder to a spare image sensor device when the intruder cannot be tracked through cooperation of the second image sensor devices
Further comprising a device.

Images