JP6967868B2 - Surveillance systems, surveillance programs, and storage media - Google Patents

Description

The present invention relates to a monitoring system, a monitoring program, and a storage medium, and more particularly to a monitoring system, a monitoring program, and a storage medium for identifying a flying object.

Surveillance cameras have been used for a long time, but with the spread of Internet lines, they are connected to networks and are increasingly operated by remote servers for monitoring. In addition, the number of pixels of the camera has increased, and it has become possible to recognize accidents and crimes by analyzing the recorded images after the fact. Although the current situation of such surveillance cameras is the most common, an unmanned surveillance camera is installed in a building, and security personnel such as security companies monitor the images of the surveillance cameras in remote areas to help prevent crime. It is a thing. Security guards actually observe a large number of images from surveillance cameras on a large number of display devices, recognize suspicious objects (suspicious persons and suspicious objects), dispatch security guards if necessary, and send them to suspicious persons. I'm warning you.

However, security personnel will see images of surveillance cameras in different locations, different buildings, in which the public areas of roads, parks, passageways, or adjacent irrelevant There are also buildings and sites, so it is often difficult to immediately understand where it is okay and where it should be regarded as a suspicious person. In some cases, even if a suspicious person invades a private land without a partition or an area where intrusion is not allowed, the security officer may not notice it. In recent years, due to terrorism and crimes, the number of surveillance cameras installed has increased sharply, but such a situation has increased the burden on security personnel to monitor and induces surveillance errors.

Conventional techniques for automating surveillance cameras include image processing devices, image processing systems and programs (see Patent Document 1). This is to "set a gaze area in the panoramic image of the monitoring area, analyze the image of the set gaze area by the image analysis means to detect a moving object, and operate the zoom camera of the camera unit based on the detection. The moving body is photographed. At that time, the spatial relationship information with the panoramic image and the temporal relationship information showing the transition on the time axis in the panoramic image of the gaze partial image are created in the gaze portion image, and the gaze portion is photographed. The related information is added to the image and stored in the image storage unit. At the time of searching, the panoramic image and the gaze part image are searched using the above related information, and both images are superimposed and displayed on the display unit. " It is a technology such as.

Japanese Unexamined Patent Publication No. 2010-233185

In the above-mentioned prior art, after setting a monitoring area, a moving object in the monitoring area is zoomed and photographed as a suspicious object. This is a technology that automatically tracks and zooms suspicious persons, but it cannot "automatically set the monitoring area". Further, when the monitoring area is wide or when a plurality of moving objects enter the monitoring area, there is a demerit that the processing cannot be completed in time unless the high-speed arithmetic unit is used, and it becomes impossible to capture a suspicious object. As described above, in the prior art, in order to set the monitoring area, it is essential for a human to set it, and there is no automatic setting technique. Even when the surveillance area is set by the surveillance camera setter, whether or not the surveillance area to be set can be set properly depends on the skill of the surveillance camera setter and whether or not the situation of the building or site is known. Will be done. With such a human setting, it is not easy to eliminate the setting error of the monitoring area.

In addition, surveillance cameras may mistakenly recognize weather changes such as light and rain as suspicious persons or objects in both automatic recognition and human observation, but in this case, the number of pixels of the camera is increased. We have responded by letting them do it, but it has not been fundamentally resolved.

By the way, in the conventional technology of a monitoring device other than a camera, there is a set of a ray transmitter and a ray receiver, and a monitoring line is set between them. When the target blocks the monitoring line, it is a device that issues an alarm as a suspicious person. It is still widely used as a night security system. However, even in this conventional technology, once the monitoring line machine is installed, it is necessary to relocate the machine itself in order to change the monitoring line, so the monitoring line can be easily relocated or changed. It had the drawback of being difficult. In addition, there is a drawback that a set of a transmitting unit and a receiving unit is required for the number of monitoring lines.

Furthermore, recently, the technology of small flying objects called drones has been developed, and flying objects capable of extremely precise flight control are becoming widespread. Cameras and microphones are mounted on this projectile, increasing the risk of voyeurizing or tapping private homes and buildings. Therefore, in the past, it was sufficient to consider the intrusion of people in the security of buildings and facilities, but it is necessary to build a monitoring system in consideration of the intrusion of drones, voyeurism and wiretapping by drones, and dangerous acts. There is sex.

In addition, there are many types of flying objects, from those that are at the level of children's toys and have a relatively low risk to those that have high performance and high risk. The correspondence between toy-level flying objects and high-performance flying objects should be changed, but surveillance systems and crime prevention systems have not yet been developed to deal with such situations.

Therefore, an object of the present invention is to provide a monitoring system, a monitoring program, and a storage medium that solve the above problems, and in particular, provide a monitoring device, a monitoring system, a monitoring program, and a storage medium for identifying a flying object. Is.

In order to solve the above-mentioned problems, the monitoring system (device) according to the first invention is
An audio projectile database containing audio data and projectile information associated with the audio data, an image projectile database containing image-related data, and projectile information associated with the image-related data, and a projectile. A storage unit for storing information and a flying object risk database including risk information associated with the flying object information.
A voice acquisition unit that acquires voice, and
An image acquisition unit (or an image pickup unit that captures images) that acquires a stereo image,
A voice analysis unit that analyzes the acquired voice and generates voice analysis data,
An image analysis unit that analyzes the acquired stereo image and generates image analysis data,
The audio data included in the audio projectile database is compared with the generated audio analysis data, and the image data included in the image projectile database is compared with the generated image analysis data. Based on the comparison result, the flying object determination unit that obtains the flying object information associated with the voice data or the image-related data having the highest degree of matching, and the flying object determination unit.
Based on the requested flying object information, the risk information identification unit for which risk information is requested by referring to the flying object risk database, and
Have.

Further, the monitoring system according to the second invention is
The storage unit
Further stores the risk response database containing the projectile response information associated with the risk information,
The monitoring system is
Based on the requested risk information, it also has a projectile response decision unit that seeks projectile response information by referring to the risk response database.
It is characterized by that.

Further, the monitoring system according to the third invention is
An audio projectile database containing audio data and projectile information associated with the audio data, an image projectile database containing image-related data, and projectile information associated with the image-related data, and a projectile. A storage unit for storing information and a flying object monitoring area database including monitoring area information associated with the flying object information.
A voice acquisition unit that acquires voice, and
An image acquisition unit (or an image pickup unit that captures images) that acquires a stereo image,
A voice analysis unit that analyzes the acquired voice and generates voice analysis data,
An image analysis unit that analyzes the acquired stereo image and generates image analysis data,
The audio data included in the audio projectile database is compared with the generated audio analysis data, and the image data included in the image projectile database is compared with the generated image analysis data. Based on the comparison result, the projectile determination unit that obtains the projectile information associated with the one with the highest degree of matching, and the projectile determination unit,
Based on the obtained flying object information, the risk information identification unit that obtains the monitoring area information (monitoring line, monitoring surface, etc.) by referring to the flying object monitoring area database, and
Have.

Further, the monitoring system according to the fourth invention is
The storage unit
Further stores the risk response database containing the projectile response information associated with the risk information,
The monitoring system is
Based on the requested risk information, it also has a projectile response decision unit that seeks projectile response information by referring to the risk response database.
It is characterized by that.

Further, the monitoring system according to the fifth invention is
It also has a BIM / CIM information acquisition unit that acquires BIM / CIM information including structure information or topographical information.
The risk information identification department
Change or correct the monitoring area information based on the acquired BIM / CIM information.
It is characterized by that.

Further, the monitoring system according to the sixth invention is
An audio projectile database containing audio data and projectile information associated with the audio data, an image projectile database containing image-related data, and projectile information associated with the image-related data, and a projectile. A storage unit for storing information and a flying object monitoring area database including monitoring area information associated with the flying object information.
A voice acquisition unit that acquires voice, and
An image acquisition unit (or an image pickup unit that captures images) that acquires a stereo image,
A BIM / CIM information acquisition unit that acquires BIM / CIM information including structure information or topographical information,
A voice analysis unit that analyzes the acquired voice and generates voice analysis data,
An image analysis unit that analyzes the acquired stereo image and generates image analysis data,
The audio data included in the audio projectile database is compared with the generated audio analysis data, and the image data included in the image projectile database is compared with the generated image analysis data. Based on the comparison result, the projectile determination unit that obtains the projectile information associated with the one with the highest degree of matching, and the projectile determination unit,
Based on the obtained flying object information and the acquired BIM / CIM information, the risk information identification unit for obtaining monitoring area information (monitoring line, monitoring surface, etc.) by referring to the flying object monitoring area database, and
Have.

Further, the monitoring system according to the seventh invention is
An audio projectile database containing audio data and projectile information associated with the audio data, an image projectile database containing image-related data, and projectile information associated with the image-related data, and a projectile. Flying object monitoring area database including information and monitoring area information associated with the flying object information, and structure information (information on structures such as buildings, facilities, houses, windows, openings, etc., BIM / CIM information) A storage unit for storing the information of the above and a structure correction database including monitoring area correction information based on the structure information.
A voice acquisition unit that acquires voice, and
An image acquisition unit (or an image pickup unit that captures images) that acquires a stereo image,
The structure information acquisition unit that acquires structure information,
A voice analysis unit that analyzes the acquired voice and generates voice analysis data,
An image analysis unit that analyzes the acquired stereo image and generates image analysis data,
The audio data included in the audio projectile database is compared with the generated audio analysis data, and the image data included in the image projectile database is compared with the generated image analysis data. Based on the comparison result, the projectile determination unit that obtains the projectile information associated with the one with the highest degree of matching, and the projectile determination unit,
Based on the obtained flying object information, the risk information identification unit that obtains the monitoring area information (monitoring line, monitoring surface, etc.) by referring to the flying object monitoring area database, and
A monitoring area correction unit that obtains monitoring area correction information based on the acquired structure information with reference to the structure correction database and corrects the monitoring area information based on the obtained monitoring area correction information.
Have.

Further, the monitoring system according to the eighth invention is
It further has a GPS unit that acquires the position information of the image pickup unit or the monitoring system.
The structure information acquisition unit
With reference to the structure information database in which the position information and the structure information are associated with each other, the structure information in which the imaging unit or the monitoring system is located is acquired based on the acquired position information.
It is characterized by that.

Further, the monitoring program according to the ninth invention is
It is a monitoring program that makes one or more arithmetic processing units function as the monitoring system according to the first to eighth inventions.

Further, the computer-readable storage medium according to the tenth invention is
A computer-readable storage medium containing the monitoring program according to the ninth aspect of the invention.

Further, the monitoring system (device) according to the first invention according to another aspect is
A surveillance system that includes a stereo camera and a server,
The stereo camera
An image pickup unit that captures stereoscopic images, and
A position measuring unit (GPS, etc.) that measures the position of the own device,
A direction measuring unit (3-axis geomagnetic sensor, etc.) that measures the direction of the image pickup unit of the own device,
A communication unit that transmits the stereo image, the position, and the direction to the server.
Have,
The server
A communication unit that receives the stereo image, the position, and the direction transmitted from the stereo camera.
The acquisition department that acquires site information about the site at the above location,
Based on the position, the direction, and the site information, a boundary line (coordinates of the boundary) forming at least a part of the contour of the site in the stereo image is obtained, and the boundary line is monitored in the stereo image. The monitoring line setting unit to be set to
A monitoring unit that analyzes the stereo image and recognizes an object that comes into contact with the set monitoring line (or crosses the monitoring line and invades the site).
An alarm output unit that outputs a predetermined alarm (sending e-mail, outputting or transmitting a warning voice, reporting to a security company, police, etc.) when recognizing an object that comes into contact with the monitoring line.
Have,
It is characterized by that.

Further, the monitoring system according to the second invention according to another aspect is
The server
When the target in contact with the monitoring line is recognized instead of the alarm output unit, the distance from the stereo camera to the target is obtained, and the distance and the position of the target in contact with the monitoring line to the stereo camera are obtained. A second that outputs a predetermined alarm (e-mail, warning voice, warning sound, notification to a security company, police, etc.) only when the distance is almost the same or the difference is less than or equal to a predetermined value. Has an alarm output unit,
It is characterized by that.

Further, the monitoring system according to the third invention according to another aspect is
A surveillance system that includes a stereo camera and a server,
The stereo camera
An image pickup unit that captures stereoscopic images, and
A position measuring unit (GPS, etc.) that measures the position of the own device,
A direction measuring unit (3-axis geomagnetic sensor, etc.) that measures the direction of the image pickup unit of the own device,
A communication unit that transmits the stereo image, the position, and the direction to the server.
Have,
The server
A communication unit that receives the stereo image, the position, and the direction transmitted from the stereo camera.
The acquisition department that acquires site information about the site at the above location,
Based on the position, the direction, and the site information, a boundary line (coordinates of the boundary) forming at least a part of the contour of the site in the stereo image is obtained, and the boundary line is on or near the boundary line in the stereo image. If a fence (fence, etc.) is provided on the wall, the upper end of the wall (slightly below the upper end to prevent false alarms (that is, a few cm, a dozen centimeters from the end, and a position away from the end)). It is preferable to set to), and the monitoring line setting unit that sets the monitoring line to
A monitoring unit that analyzes the stereo image and recognizes an object that comes into contact with the monitoring line.
An alarm output unit that outputs a predetermined alarm (e-mail, warning voice, warning sound, notification to a security company, police, etc.) when it recognizes an object that comes into contact with the monitoring line.
Have,
It is characterized by that.

Further, the monitoring system according to the fourth invention according to another aspect is
A surveillance system that includes a stereo camera and a server,
The stereo camera
An image pickup unit that captures stereoscopic images, and
A position measuring unit (GPS, etc.) that measures the position of the own device,
A direction measuring unit (3-axis geomagnetic sensor, etc.) that measures the direction of the image pickup unit of the own device,
A communication unit that transmits the stereo image, the position, and the direction to the server.
Have,
The server
A communication unit that receives the stereo image, the position, and the direction transmitted from the stereo camera.
The acquisition department that acquires site information about the site at the above location,
Based on the position, the direction, and the site information, a boundary line (coordinates of the boundary) forming at least a part of the contour of the site in the stereo image is obtained.
A monitoring line setting unit that sets the boundary line as a monitoring line in the stereo image, and further sets at least one line forming a boundary in the air vertically moved upward from the boundary line as an additional monitoring line. ,
A monitoring unit that analyzes the stereo image and recognizes an object that comes into contact with the monitoring line or an additional monitoring line.
An alarm output unit that outputs a predetermined alarm (e-mail, warning voice, warning sound, notification to a security company, police, etc.) when it recognizes an object that comes into contact with the monitoring line or an additional monitoring line.
Have,
It is characterized by that.

Further, the monitoring system according to the fifth invention according to another aspect is
A surveillance system that includes a stereo camera and a server,
The stereo camera
An image pickup unit that captures stereoscopic images, and
A position measuring unit (GPS, etc.) that measures the position of the own device,
A direction measuring unit (3-axis geomagnetic sensor, etc.) that measures the direction of the image pickup unit of the own device,
A communication unit that transmits the stereo image, the position, and the direction to the server.
Have,
The server
A communication unit that receives the stereo image, the position, and the direction transmitted from the stereo camera.
The acquisition department that acquires site information about the site at the above location,
Based on the position, the direction, and the site information, a boundary line (coordinates of the boundary) forming at least a part of the contour of the site in the stereo image is obtained, and the boundary surface including the boundary line. A monitoring surface setting unit that sets a boundary surface that extends vertically upward from the boundary line (such as the spatial coordinates of the surface that separates the outside from the boundary line) as the monitoring surface in the stereo image.
A monitoring unit that analyzes the stereo image and recognizes an object that comes into contact with the monitoring surface.
An alarm output unit that outputs a predetermined alarm (e-mail, warning voice, warning sound, notification to a security company, police, etc.) when it recognizes an object that comes into contact with the monitoring surface.
Have,
It is characterized by that.

Further, the monitoring system according to the sixth invention according to another aspect is
The server
When the target in contact with the monitoring line is recognized instead of the alarm output unit, the distance from the stereo camera to the target is obtained, and the distance and the position of the target in contact with the monitoring surface to the stereo camera are obtained. A second that outputs a predetermined alarm (e-mail, warning voice, warning sound, notification to a security company, police, etc.) only when the distance is almost the same or the difference is less than or equal to a predetermined value. Has an alarm output unit,
It is characterized by that.

Further, the monitoring system according to the seventh invention according to another aspect is
A surveillance system that includes a stereo camera and a server,
The stereo camera
An image pickup unit that captures stereoscopic images, and
A position measuring unit (GPS, etc.) that measures the position of the own device,
A direction measuring unit (3-axis geomagnetic sensor, etc.) that measures the direction of the image pickup unit of the own device,
A communication unit that transmits the stereo image, the position, and the direction to the server.
Have,
The server
A communication unit that receives the stereo image, the position, and the direction transmitted from the stereo camera.
The acquisition department that acquires site information about the site at the above location,
Based on the position, the direction, and the site information, a boundary line (coordinates of the boundary) forming at least a part of the contour of the site in the stereo image is obtained, and the boundary surface including the boundary line. A monitoring surface setting unit that sets a boundary surface that extends vertically upward from the boundary line (such as the spatial coordinates of the surface that separates the outside from the boundary line) as the monitoring surface in the stereo image.
A monitoring unit that analyzes the stereo image and recognizes an object that comes into contact with the set monitoring surface (or an object that crosses the monitoring surface and invades the site or the space above the site).
An alarm output unit that outputs a predetermined alarm (e-mail, warning voice, warning sound, notification to a security company, police, etc.) when recognizing an object that comes into contact with the monitoring surface or an invading object. ,
Have,
It is characterized by that.

The monitoring system uses a server, but it is also possible to incorporate the server functions into the monitoring device as follows.

Further, the monitoring device according to the eighth invention according to another aspect is
An image pickup unit that captures stereoscopic images, and
A position measuring unit (GPS, etc.) that measures the position of the own device,
A direction measuring unit (3-axis geomagnetic sensor, etc.) that measures the direction of the image pickup unit of the own device,
The acquisition department that acquires site information about the site at the above location,
Based on the position, the direction, and the site information, a boundary line (coordinates of the boundary) forming at least a part of the contour of the site in the stereo image is obtained, and the boundary line is monitored in the stereo image. The monitoring line setting unit to set the line and
A monitoring unit that analyzes the stereo image and recognizes an object that comes into contact with the set monitoring line (or recognizes an object that crosses the monitoring line and invades the site).
An alarm output unit that outputs a predetermined alarm (sending an e-mail, warning voice, warning sound, notification to a security company, police, etc.) when recognizing an object that comes into contact with the monitoring line.
Have,
It is characterized by that.
Further, it is possible to provide a communication unit and transmit the output to a mobile terminal, a server, a PC or the like via a network.

Further, the monitoring device according to the ninth invention according to another aspect is
An image pickup unit that captures stereoscopic images, and
A position measuring unit (GPS, etc.) that measures the position of the own device,
A direction measuring unit (3-axis geomagnetic sensor, etc.) that measures the direction of the image pickup unit of the own device,
The acquisition department that acquires site information about the site at the above location,
Based on the position, the direction, and the site information, a boundary line (coordinates of the boundary) forming at least a part of the contour of the site in the stereo image is obtained, and the boundary surface including the boundary line. A monitoring surface setting unit that sets a boundary surface that extends vertically upward from the boundary line (such as the spatial coordinates of the surface that separates the outside from the boundary line) as the monitoring surface in the stereo image.
A monitoring unit that analyzes the stereo image and recognizes an object that comes into contact with the set monitoring surface.
An alarm output unit that outputs a predetermined alarm (sending an e-mail, warning voice, warning sound, notification to a security company, police, etc.) when recognizing an object that comes into contact with the monitoring surface.
Have,
It is characterized by that.

As described above, the means for solving the present invention have been described as a system or an apparatus, but the present invention can also be realized as a storage medium in which a method, a program, or a program substantially corresponding to these is recorded. It should be understood that these are also included in the scope of. In addition, each step of the following method and program uses an arithmetic processing device such as CPU and DSP as necessary in data processing, and the input data and processed / generated data are taped. , HDD, memory, etc. are stored in storage devices.

For example, the program according to the tenth invention according to another aspect, which realizes the present invention as a program, is the present invention.
It is a monitoring program that makes one or more arithmetic processing units function as the monitoring system according to the first to tenth inventions.

In addition, the program according to the eleventh invention according to another aspect is
It is a monitoring program that makes the arithmetic processing unit function as the monitoring device according to the sixth or seventh invention.

Further, for example, the computer-readable recording medium program according to the twelfth invention, which realizes the present invention as a computer-readable recording medium, is
A computer-readable storage medium containing the monitoring program according to the tenth or eleventh invention.

According to the present invention, it is possible to provide a monitoring device and a monitoring system for monitoring a flying object by further utilizing not only images but also sounds. According to another aspect of the present invention, it is possible to provide a monitoring device and a monitoring system capable of automatically setting a monitoring line and a monitoring surface.

FIG. 1 is a block diagram showing an outline of a monitoring system according to an embodiment of the present invention. FIG. 2 is a sequence diagram showing an example of the processing executed by the system shown in FIG. FIG. 3 is a sequence diagram showing another example of the processing executed by the system shown in FIG. FIG. 4 is a schematic diagram showing the automatic setting of the monitoring line when the stereo camera of this system is installed. FIG. 5 is a schematic diagram illustrating what the camera looks like on the screen when the camera is pointed in the other direction in FIG. FIG. 6 is a sequence diagram showing another example of the processing executed by the system shown in FIG. FIG. 7 is a schematic diagram illustrating what the monitoring line looks like on the screen when the monitoring line is installed at the upper end of the fence as shown in FIG. FIG. 8 is a sequence diagram showing another example of the processing executed by the system shown in FIG. FIG. 9 is a schematic diagram showing automatic setting of a plurality of monitoring lines when the stereo camera of this system is installed. FIG. 10 is a schematic diagram illustrating a scene in which an object approaching a site under the supervision of this system is detected as a suspicious object. FIG. 11 is a schematic diagram illustrating how each scene of FIG. 7 looks on the screen. FIG. 12 is a block diagram showing an outline of a monitoring system according to an embodiment of the present invention. FIG. 13 is a sequence diagram showing another example of the processing executed by the system shown in FIG. FIG. 14 is a schematic diagram showing automatic setting of a plurality of monitoring surfaces when a stereo camera of this system is installed. FIG. 15 is a schematic diagram showing when the automatic setting of a plurality of monitoring surfaces when the stereo camera of this system is installed is completed. FIG. 16 is a block diagram showing an outline of a monitoring system according to an embodiment of the present invention. FIG. 17 is a flowchart showing an example of processing executed by this system. FIG. 18 is a schematic diagram illustrating a scene in which an object approaching a site under the supervision of this system is detected as a suspicious object. FIG. 19 is a flowchart showing an example of processing executed by this system. FIG. 20 is a flowchart in which step D34-1 of FIG. 19 is divided into more detailed steps. FIG. 21 is a schematic diagram illustrating a setting status of a monitoring surface for a flying object approaching a building under the monitoring of this system. FIG. 22 is a flowchart showing an example of processing executed by this system. FIG. 23 is a schematic diagram illustrating the setting status of the monitoring surface for a flying object approaching a building under the monitoring of this system. FIG. 24 is a schematic diagram illustrating a setting status of a monitoring line when monitoring a water purification plant.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing an outline of a monitoring system according to an embodiment of the present invention. As shown in the figure, the monitoring system MS1 includes a stereo camera SC1 and a server SV1. The server SV1 has a control unit (CPU, arithmetic processing unit, processor) CON, an input unit (not shown), an output unit (not shown), a communication unit COM, a storage unit MEM, and a display unit DIS. The stereo camera SC1 has a control unit (CPU, arithmetic processing unit, processor, not shown) and a communication unit COM, but further includes an input unit (not shown), an output unit (not shown), and a storage unit (not shown). It may have a display unit (not shown) and a display unit (not shown). A stereo camera is a camera that captures a plurality of (usually two) images with a plurality of (usually two) cameras (imaging units). Normally, the optical axes of the two cameras are parallel to each other, and the object is photographed at the same time. Further, the two images taken by this are referred to as stereo images, and further, an image obtained by various processing from these two images, an information set after processing, and a combination thereof are also included in the stereo image. It may be a three-lens stereo image taken by a camera with three or more eyes. In addition, it is preferable to use a high-performance camera that can shoot with light in a slight environment even at night.

The stereo camera SC1 has an imaging unit CAM that captures a stereo image, a position measuring unit (GPS, etc.) that measures the position of its own device, and a direction measuring unit DIM (3-axis geomagnetic sensor) that measures the direction of the imaging unit of its own device. Etc.), and a communication unit COM that transmits the stereo image, the position, and the direction to the server.

The site related to the site where the server SV1 is located from the site information server RES or the storage unit MEM connected via the network NET to the communication unit COM that receives the stereo image, position, and direction transmitted from the stereo camera. Based on the acquisition unit ACQ that acquires information, the position, direction, and site information, the boundary line (boundary coordinates) that forms at least a part of the contour of the site in the stereo image is obtained, and the boundary is obtained in the stereo image. A monitoring line setting unit MLS that sets a line as a monitoring line, and a monitoring unit MON that analyzes a stereo image and recognizes an object that invades the site or touches the monitoring line beyond the set monitoring line. It has an alarm output unit WOT that outputs a predetermined alarm (e-mail, warning voice, warning sound, notification to a security company, police, etc.) when it recognizes an object that comes into contact with the monitoring line. The control unit CON is a distance calculation unit that calculates the distance between a specific area (a part where an object exists, etc.) of a stereo image (two images) and its own device using a known stereo image distance analysis technique or the like. Including DE. The distance calculation unit and the monitoring line setting unit may be realized by software, but it is possible to perform higher-speed processing by forming a circuit and realizing it as hardware. The display unit DIS can display the information stored in the apparatus and the generated information.

It is preferable that each functional unit included in the control unit of the server SV1 is realized by a program module read into the memory space of the control unit. This is because the stereo camera SC1 does not show the control unit, but each function unit is realized by a program module read into the memory space of the control unit, or each function unit is realized by a built-in GPS unit or camera unit. It is possible to realize it. Normally, the PC operates as the present device by downloading software that causes the processor to function as each part of the present device from a storage unit or a website, installing the software on the PC, and starting the software. It should be noted that each functional unit provided in the control unit is merely a collection of steps having a certain functional cohesiveness, and a plurality of functional units may be combined into one functional unit, or a part thereof may be used as another function. It can be incorporated into parts or divided into other independent functional parts.

In this way, the generated / extracted information, intermediate data, and acquired data can be transmitted to the outside and displayed on the display unit, and the generated / extracted information, intermediate data, and acquired data can be stored in the storage unit. It should be noted that this is also possible in other real-world embodiments described below. This device is a computer such as a general-purpose computer, a special-purpose computer, a server, a PC, or a mobile terminal such as a smartphone, or a program module that realizes (executes) the functions and processing procedures (methods) of this device on the computer. It is preferable to construct the present device on the computer by holding it in the CPU or storage unit of the computer or reading it from an external server or storage, and the same applies to each subsequent embodiment. In addition, each functional unit may be distributed to separate computers and devices connected by a network. Further, a plurality of functional units may be combined into one, or a part of the processing step may be made into another functional unit. Further, in the present embodiment, the server and the stereo camera are defined as separate ones, but a part or all of each function of the server may be included in the stereo camera.

FIG. 2 is a sequence diagram showing an example of the processing executed by the system shown in FIG. As shown in the figure, the three devices of the stereo camera SC1, the server SV1, and the site information server RES1 cooperate to perform processing. Once the server SV1 stores the site information in the site information server RES1 and other servers, it is possible to eliminate the need for cooperation with the site information server RES1.

In step S11 of the stereo camera SC1, the image pickup unit CAM captures a stereo image. Incidentally, although imaging may be performed continuously, it is preferable to thin out frames or perform still images at intervals of several seconds in order to reduce the amount of communication and processing. Next, in step S12, the position measuring unit POM measures the position of the own device. Next, in step S13, the direction measuring unit DIM measures the direction of the imaging unit of the own device. Next, in step S14, the communication unit COM transmits a stereo image, a position, and a direction to the server. At this time, it is preferable to also transmit the image pickup set values (angle of view, zoom, number of pixels, etc.) of the camera. The stereo image, position, and direction may be transmitted together or individually.

After that, the processing moves to the server SV1. In step V11 of the server SV1, the communication unit COM receives the stereo image, position, and direction transmitted from the stereo camera. Next, in step V12, the acquisition unit ACQ transmits a site information request regarding the location to the site information server RES1. Alternatively, the acquisition unit ACQ may acquire the site information regarding the site at the received position by reading it from the storage unit MEM in the own device.

After that, the processing is transferred to the site information server RES1. In step R11 of the site information server RES1, the site information request regarding the location is received. Then, in step R12, the site information (latitude, longitude, site shape, area, in the case of a rectangle, the coordinates of the four corners, etc.) is read out based on the position (absolute coordinates such as address, latitude / longitude, etc.). In step R13, the read site information is transmitted to the server SV1.

After this, the process returns to the server SV1. The server SV1 receives the site information, and then, in step V13, the boundary where the monitoring line setting unit MLS forms at least a part of the outline of the site in the stereo image based on the position, the direction, and the site information. A line (each coordinate of the boundary) is obtained, and the boundary line is set as a monitoring line in the stereo image. Then, in step V14, the monitoring unit MON analyzes the stereo image and continues monitoring until it recognizes an object that comes into contact with the set monitoring line (or crosses the monitoring line and invades the site). Finally, in step V15, when the alarm output unit WOT recognizes an object that comes into contact with the monitoring line, a predetermined alarm (sending an e-mail, outputting or transmitting a warning voice, reporting to a security company, police, etc., etc.) ) Is output. Alternatively, the captured image may be transmitted to the mobile terminal of the owner of the site (such as a contractor of a security company).

In addition, some further determination condition may be added between steps V14 and V15. For example, this server has a function of finding the height of an object in contact with a monitoring line, and by using this, only those with a height of 150 cm or more, 150 cm to 190 cm (that is, adults) can proceed to step V15. Other than that, it may be excluded. This makes it possible to effectively exclude children smaller than 150 cm or small animals such as dogs, cats, birds and insects.

In addition, a Doppler sensor is installed in or near the camera to measure the heart rate of the target in contact with the monitoring line, and only those within the human heart rate range are advanced to step V15, and the others are excluded. May be good. This makes it possible to effectively exclude small animals such as dogs, cats, birds and insects.

FIG. 3 is a sequence diagram showing another example of the processing executed by the system shown in FIG. The process of FIG. 3 is the same as the process of FIG. 2 unless otherwise mentioned. The different process is that in FIG. 3, steps V16-V19 are performed instead of step V15 in FIG. The processing of FIG. 2 can be realized with only one image of a stereo image (two images) except for the recognition accuracy and the speed of image recognition. FIG. 3 is an embodiment in which distance recognition to an object is utilized by using a distance recognition technique such as a parallax image, which is an advantage of a stereo image. As shown in FIG. 3, after recognizing an object (person, object, etc.) in contact with the monitoring line in step V14, the distance calculation unit DE obtains the distance from the stereo camera to the object in step V16. .. In step V17, whether or not the obtained distance and the distance from the position of the monitoring line in contact with the object to the stereo camera are approximately the same (plus or minus 30 cm, about 1 m (when the difference is less than or equal to a predetermined value)). Set to a practical range depending on the situation and camera accuracy). If the distances are not the same (when the determination conditions are not satisfied), the process proceeds to step V19, which is regarded as a false recognition of the target and no alarm is output. Typical misidentifications are insects, birds, rain, smoke, snow, etc. that cross nearby, or distant birds, humans, vehicles, small animals, rain, smoke, snow, etc. Can be excluded. In addition, although light and the like also cause misidentification, stereo images can be recognized with reduced influence of light. When the determination condition of step V17 is satisfied, that is, when the distances are substantially the same, the alarm output unit outputs an alarm in step V18 and ends the process.

FIG. 4 is a schematic diagram showing the automatic setting of the monitoring line when the stereo camera of this system is installed. As shown in the figure, the stereo camera SC1 is installed on the site SS of the map MAP. This system identifies the position of the site SS by the GPS unit of the camera and acquires the site information of the site SS. According to the site information, the site SS is a rectangle, and the coordinates of each of the four boundary points A, B, C, and D are obtained. The four sides connecting these four points are the boundary lines, and these four sides are set as the monitoring line ML (virtual fence). The camera is installed in the direction of photographing the monitoring line connecting the boundary point A and the boundary point D, which are the upper sides. That is, finally, in the stereo image SI of the camera (direction sensor, attitude sensor, GPS), the monitoring line (virtual fence) is automatically set at the position of the obtained coordinates. No matter which direction the camera is facing, if the boundary line is visible, the coordinates of the boundary line are mapped (projected) in the image, and the part occupied by the captured boundary line is automatically set as the monitoring line. can. If several cameras are installed so that the boundary line can be seen, the user can surround the site with a monitoring line and put it completely under surveillance.

FIG. 5 is a schematic diagram illustrating what the camera of FIG. 4 looks like on the screen when pointed in the other direction. As shown in the figure, when the stereo camera is pointed at the boundary point D from the lower right side of the site, the monitoring line ML, the monitoring line ML1, and the boundary point D are projected on the stereo image SI. This device treats these two monitoring lines ML and monitoring line ML1 as target areas to be monitored. Preferably, the image is divided into an appropriate number of blocks, such as the block-divided image SI-S below, and only the blocks covered by the monitoring line ML and the monitoring line ML1 are monitored. This can significantly reduce the amount of processing. In the figure, it is divided into 4 × 6 = 24 blocks, and since the number of monitored blocks is only 7 out of 24, it is possible to reduce the arithmetic processing to 7/24. If the number of divisions is increased to an appropriate number, the arithmetic processing can be further significantly reduced. It is preferable that the monitoring unit MON and the distance measuring unit DE monitor only the blocks of these monitoring targets to capture the suspicious target or calculate the distance to the supplemented suspicious target.

FIG. 6 is a sequence diagram showing another example of the processing executed by the system shown in FIG. The process of FIG. 6 is the same as the process of FIG. 3 unless otherwise mentioned, but a part of the process of the server SV1 is extracted. In FIG. 6, instead of step V13 in FIG. 3, steps V13-a and V13-b are performed. As shown in FIG. 5, in step V13-a, the boundary line of the site is obtained in the stereo image. In Tep V13-b, if there is a fence or fence on or near the boundary line, the monitoring line is set (that is, moved) at the upper end of the fence. If the monitoring line is on the ground (ground plane, that is, 0 m in height) and there is a fence or fence, a suspicious person who climbs on the fence or fence will be regarded as a suspicious person until it completely enters the site. Cannot be captured. In order to prevent this and reliably capture the suspicious person, monitoring lines ML4 and ML5 are set at the upper end (preferably slightly lower side) of the physical fence PF1. Next, in step V14, an object (person, object, etc.) that comes into contact with each monitoring line set at the upper end of the fence is recognized. For the recognition of fences and fences, it is possible to cut out edges by changing the feature amount of the edge portion and the fence end portion by image processing, and such a conventional image processing technique is used. If the heights of the fences are different, it is preferable to provide a monitoring line at the vertically vertical ends connecting the fences.

FIG. 7 is a schematic diagram illustrating what the monitoring line looks like on the screen when the monitoring line is installed at the upper end of the fence as shown in FIG. The stereo image SI11 sets the boundary line as the monitoring line as it is, and the stereo image SI12 sets the boundary line not as the boundary line but as the monitoring line at the upper end of the fence (to be exact, slightly below). As shown in the figure, in the stereo image SI11, since the monitoring lines ML2 and ML3 are set on the ground, even if a suspicious person UIP gets on the physical fence PF1, an alarm cannot be issued. On the other hand, in the stereo image SI12, since the monitoring lines ML4 and ML5 are set at the upper ends of the physical fence PF1, it is possible to detect that the suspicious person UIP has entered the physical fence PF1 and issue an alarm. In addition, it is possible to detect suspicious acts such as searching the inside of the site by touching the monitoring line. Further, since the heights of the monitoring lines ML4 and ML5 are different, it is necessary to set a monitoring line other than the upper end. That is the monitoring line ML6. As described above, in the case of a physical fence, it is preferable to set a monitoring line at the "end" other than the upper end. In addition, by setting a monitoring line slightly below the upper end or toward the inside of the end, it is possible to exclude unrelated persons who just pass outside and prevent false alarms.

FIG. 8 is a sequence diagram showing another example of the processing executed by the system shown in FIG. The process of FIG. 8 is the same as the process of FIG. 3 unless otherwise mentioned. The different process is that in FIG. 3, steps V13-1 and V14-1 are performed instead of steps V13 and V14 in FIG. As shown in FIG. 8, in step V13-1, the boundary line of the site is obtained in the stereo image, the boundary line is set as the monitoring line, and the boundary line is set above the boundary line at a predetermined height or a predetermined number at a predetermined interval. Set up multiple monitoring lines up to. Next, in step V14, an object (person, object, etc.) that comes into contact with each monitoring line is recognized. In this way, by setting several monitoring lines as if they were fences, it is possible to make them function as a substantial monitoring surface.

FIG. 9 is a schematic diagram showing automatic setting of a plurality of monitoring lines when the stereo camera of this system is installed. As shown in the figure, the left side has the monitoring line set for the stereo image, and the right side has the corresponding arrangement of the monitoring line on the site. First, one monitoring line ML1 is set as in the stereo image SI10 on the upper left. Corresponding to this is the monitoring line ML1 on the site SS10 on the upper right. The boundary line of the site SS10, which is a ground plane parallel to the X-axis and the Y-axis, is the monitoring line.

Based on this monitoring line ML1, a monitoring line ML2, which is a second line at a predetermined height from the monitoring line ML1, is set upward along the Z-axis perpendicular to the X-axis and the Y-axis (vertical direction). This is repeated, and the monitoring lines ML3 and ML4 are sequentially set. These plurality of monitoring lines make it possible to set a substantial monitoring surface MP on the boundary line of the site. The advantage of this is that since the monitoring is performed by a line, it is not necessary to cover the entire monitoring surface as a monitoring target, so that the amount of calculation for monitoring can be significantly reduced. By setting such a substantial monitoring surface at the boundary line of all four sides, it is possible to put the site under complete monitoring. In the figure, for convenience of drawing, the thick monitoring line can actually be thinned to about several cm.

FIG. 10 is a schematic diagram illustrating a scene in which an object approaching a site under the supervision of this system is detected as a suspicious object. As shown in the figure, the above scene SCN21 does not report that the drone DR1 is not a suspicious object because it is far from the monitoring surface. A stereo camera SC2 is installed in the building CONS2 of the site SS20, and a substantial monitoring surface MP23 is projected in the image. Then, on the screen, the drone DR1 recognizes that it is in contact with the substantially monitoring surface MP23. This is the stage of step V14-1 in FIG. However, as shown in the figure, since the drone DR21 is far away from the monitoring surface, the distance from the camera to the monitoring surface and the distance from the camera to the target drone DR1 are different due to the processing of steps V16 and V17 (the distance from the camera to the target drone DR1 is different (). Since it is not almost the same), it will not be reported as not a suspicious object.

As shown in the figure, in the scene SCN22 below, since the drone DR1 is in contact with the monitoring surface MP23, it is detected as a suspicious object and an alarm is issued (alarmed). At this time, the following figure will explain how the drone looks on the actual screen.

FIG. 11 is a schematic diagram illustrating how each scene of FIG. 10 looks on the screen. The upper stereo image SI21 is a scene SCN21 in which the drone DR1 is far from the monitoring surface, and the lower stereo image SI2221 is a scene SCN22 in which the drone DR1 is also in contact with the monitoring surface. When the device detects an object (drone, person, etc.) that is in contact with the actual monitoring surface on the screen, it temporarily determines that a suspicious object has been detected. That is, in both of the two scenes, it is temporarily determined that a suspicious object has been detected. However, in such a judgment, drones, people, animals, or nearby objects (insects, raindrops, petals, snow, hail, garbage, etc.) outside the site SS20 will be regarded as suspicious objects. It becomes a false alarm. Therefore, using the distance recognition function of the stereo camera, the distance to the object that is temporarily regarded as a suspicious object is calculated, and it is determined whether or not the object is touching the monitoring surface. Only touching objects (only those whose distance indicates a value near the monitoring surface) are regarded as suspicious objects, and others are excluded as non-suspicious objects, noise, or false detection. As shown in the figure, distance recognition can be used to eliminate distant objects (rain, drones, people, etc.), nearby raindrops, insects, birds, etc., dramatically reducing false alarms. It becomes.

FIG. 12 is a block diagram showing an outline of a monitoring system according to an embodiment of the present invention. As shown in the figure, the monitoring system MS2 includes a stereo camera SC2 and a server SV2. The server SV2 has a control unit (CPU, arithmetic processing unit, processor) CON, an input unit (not shown), an output unit (not shown), a communication unit COM, a storage unit MEM, and a display unit DIS. The stereo camera SC has a control unit (CPU, arithmetic processing unit, processor, not shown) and a communication unit COM, but further includes an input unit (not shown), an output unit (not shown), and a storage unit (not shown). It may have a display unit (not shown) and a display unit (not shown).

The monitoring system of FIG. 12 is the same as that of FIG. 1 unless otherwise noted. The major difference is that in FIG. 12, a monitoring surface is set and monitored instead of a monitoring line.

The site related to the site where the server SV2 is located from the site information server RES or the storage unit MEM connected via the network NET to the communication unit COM that receives the stereo image, position, and direction transmitted from the stereo camera. Based on the acquisition unit ACQ for acquiring information, the position, the direction, and the site information, a boundary line (coordinates of the boundary) forming at least a part of the contour of the site in the stereo image is obtained, and the boundary line is included. A monitoring surface setting that sets a boundary surface that extends vertically upward from the boundary line (such as the spatial coordinates of the surface that separates the outside from the outside by the boundary line) as the monitoring surface in the stereo image. When the MPS and the monitoring unit MON that analyzes the stereo image to recognize the object that comes into contact with the monitoring surface and the object that comes into contact with the monitoring surface are recognized, a predetermined alarm (e-mail, warning sound, warning sound, security) It has an alarm output unit WOT that outputs (reports to the company, police, etc.). The control unit CON is a distance calculation unit that calculates the distance between a specific area (a part where an object exists, etc.) of a stereo image (two images) and its own device using a known stereo image distance analysis technique or the like. Including DE. The distance calculation unit and the monitoring line setting unit may be realized by software, but it is possible to perform higher-speed processing by forming a circuit and realizing it as hardware. The display unit DIS can display the information stored in the apparatus and the generated information.

FIG. 13 is a sequence diagram showing another example of the processing executed by the system shown in FIG. The process of FIG. 13 is the same as the process of FIG. 2 unless otherwise mentioned. The different process is that in FIG. 13, steps V23-V25 are performed instead of steps V13-15 in FIG. That is, the process of FIG. 13 is to process the monitoring surface instead of the monitoring line. As shown in the figure, the server SV2 receives the site information, and then in step V23, the monitoring surface setting unit MPS determines the outline of the site in the stereo image based on the position, direction, and site information. A boundary line (coordinates of the boundary) that forms at least a part of the boundary line is obtained, and a boundary surface that includes the boundary line and extends vertically upward from the boundary line (the boundary line separates the outside from the outside). (Spatial coordinates of the surface, etc.) are set as the monitoring surface in the stereo image. Then, in step V24, the monitoring unit MON analyzes the stereo image and continues monitoring until it recognizes an object that comes into contact with the set monitoring surface (or enters the site beyond the monitoring surface). Finally, in step V25, when the alarm output unit WOT recognizes an object that comes into contact with the monitoring surface, a predetermined alarm (sending an e-mail, outputting or transmitting a warning voice, reporting to a security company, police, etc., etc.) ) Is output.

As in the process of FIG. 3, the process of finding the distance from the camera to the target and issuing an alarm only when the distance from the camera to the monitoring surface is almost the same, and not issuing an alarm otherwise, is The same can be done with the system of FIG.

FIG. 14 is a schematic diagram showing automatic setting of a plurality of monitoring surfaces when a stereo camera of this system is installed. As shown in the figure, it is assumed that there is a building CONS2 on the site SS20 as in the scene SCN21. First, it is assumed that there is a boundary line BDY of the site along the X axis as in the scene SCN22. This system sets a monitoring surface MP21 that extends this boundary line BDY vertically to a predetermined height (preferably up to several meters above the building) in the Z-axis direction. Subsequently, it is assumed that there is a boundary line of the site along the Y axis as in the scene SCN23. This system sets a monitoring surface MP22 (virtual fence) that extends this boundary line vertically from the ground plane (XY plane) to a predetermined height (preferably up to several meters above the building) in the Z-axis direction. do. After that, the monitoring surfaces MP23 and MP25 are sequentially set. Finally, the monitoring surface MP24 is set so as to close the upper surface. It surrounds the entire site and building like a bird cage, and if multiple cameras are installed to cover all of these monitoring surfaces, the site can be completely under surveillance. .. As a result, it becomes possible to appropriately recognize a suspicious person who approaches from the sky with a drone or a suspicious person who invades from above with a crane or the like as a monitoring target.

FIG. 15 is a schematic diagram showing when the automatic setting of a plurality of monitoring surfaces when the stereo camera of this system is installed is completed. As shown in the figure, in the scene SCN25, the building CONS2 and the site SS20 are completely surrounded by the five monitoring surfaces MP21, 22, 23, 24, and 25, and it is possible to take a perfect monitoring system. ..

FIG. 16 is a block diagram showing an outline of a monitoring system according to an embodiment of the present invention. As shown in the figure, the monitoring system MS3 includes a stereo camera SC2 and a server SV2. The server SV3 has a control unit (CPU, arithmetic processing unit, processor) CON, an input unit (not shown), an output unit (not shown), a communication unit COM, a storage unit MEM, and a display unit DIS. The stereo camera SC has a control unit (CPU, arithmetic processing unit, processor, not shown) and a communication unit COM, but further includes an input unit (not shown), an output unit (not shown), and a storage unit (not shown). It may have a display unit (not shown) and a display unit (not shown).

The monitoring system of FIG. 16 is the same as that of FIG. 12 unless otherwise noted. The major difference is that in FIG. 16, the voice information can be further utilized to form a configuration suitable for monitoring a flying object, that is, a drone. The configuration will be described in detail below.

The storage unit MEM of the server SV3 stores the audio data, the audio projectile database SFI including the projectile information associated with the audio data, the image-related data, and the projectile information associated with the image-related data. Image Flying object database IFI including, flying object information, and flying object risk database FRI including flying object information and risk information associated with the flying object information, and risk response database RCDB containing flying object correspondence information associated with the risk information, To store more.

Further, the acquisition unit ACQ of the control unit CON of the server SV3 includes an audio acquisition unit that acquires audio and an image acquisition unit (or an image pickup unit that captures an image) that acquires a stereo image. The control unit CON includes the analysis unit ANA. The analysis unit ANA includes a voice analysis unit that analyzes the acquired voice and generates voice analysis data, and an image analysis unit that analyzes the acquired stereo image and generates image analysis data. The control unit CON compares the voice data included in the voice flying object database with the generated voice analysis data, and further compares the image data included in the image flying object database with the generated image analysis data. However, based on these comparison results, the flying object determination unit FOD that obtains the flying object information associated with the voice data or the image-related data having the highest degree of matching, and the flying object risk database based on the obtained flying object information. Further includes a risk information identification unit RID for requesting risk information, and the like. The control unit CON further has a projectile response determination unit FCD that seeks projectile response information by referring to the risk response database based on the requested risk information.

FIG. 17 is a flowchart showing an example of processing executed by this system. For convenience of explanation, the description is given in the form of processing on the server, but some of these processing may be executed by a stereo camera or another terminal (not shown).

As shown in the figure, in step D11, the audio projectile database and the image projectile database are stored. In step D12, audio information and image information are acquired. Further, in step D13, the acquired voice information is acoustically analyzed to generate voice analysis data. In addition, the acquired stereo image is analyzed to generate image analysis data. Next, in step D14, a drone model that exceeds a predetermined threshold value in matching with each database (one having a predetermined degree of matching or higher, or one having the highest degree of matching) is specified as the corresponding drone model. Further, in step D15, the audio data included in the audio projectile database and the generated audio analysis data are compared, and further, the image data included in the image projectile database and the generated image analysis data are compared. The comparison is performed, and based on these comparison results, the flying object information associated with the audio data or the image-related data having the highest degree of matching is obtained.

In step D15, based on the obtained flying object information (specified drone model), the flying object risk database is referred to, the risk information is obtained, and the response is determined based on the risk information. In this example, the process branches to three types of correspondences, steps D16, D17, and D18, based on the risk information. If the risk is relatively low, the process proceeds to step D16 to warn the administrator or the like of the invasion of the flying object (invasion of the drone). If the risk is moderate, proceed to step D17, track the projectile (drone), track it with a tracking drone, identify where the projectile will return, or photograph the face of someone operating a suspicious drone. To do. If the risk is high, proceed to step D18 to capture (may fall) the drone (physical means such as water, nets, wind, drone, or radio means such as radio interference, radio control hijacking, etc.).

The projectile risk database referenced in step D15 can be implemented in various embodiments, but some will be exemplified.
<Model number and weight information>
Flying object information association risk information model number association weight MD001 association 150g (risk small)
KX999 association 1kg (medium risk)
YZZ707 association 5kg (high risk)
<Flight ability and attack ability, maximum speed, or navigation ability>
Flying object information Association risk information Flying ability association Attack ability, maximum speed, or navigation ability 3 rotor blades Association Very small (risk very small)
4 rotor blades associated small (small risk)
6 rotor blades associated medium (medium risk)
8 rotor blades associated high (high risk)
Wireless control association Medium (medium risk)
Autonomous navigation association High (high risk)
<Shooting ability and voyeur ability>
Flying object information Association Risk information Shooting ability Association Voyeur ability Camera 300,000 pixels Association Small (risk small)
Camera 3 million pixels Association Medium (medium risk)
Camera 12 million pixels High association (high risk)

The model number and name can be converted into flight ability, shooting ability, weight information and the like, that is, by specifying the model number and name, it is possible to obtain risk information by referring to the above table.

The risk response database referenced in step D15 can be implemented in various ways, but some will be exemplified.
<Risk information and projectile response information>
Risk Information Association Flyer Response Information Risk Small Association Warn Administrators of Drone Intrusion Risk Medium Association Track Drone Tracked with Drone, Rator, etc. Risk Large Association Capture, Drop, Takeover Control, Disable Radio Control, etc.

FIG. 18 is a schematic diagram illustrating a scene in which an object approaching a site under the supervision of this system is detected as a suspicious object. As shown in the figure, in the scene SCN31, the drone DR3 is in contact with the monitoring surface. A stereo camera SC3 is installed in the building on the site, and the actual monitoring surface MP3 is projected in the image. In this system, the drone is specified by voice and image, so the response is determined by the risk of the specified drone. For example, in the case of a model with low specifications such as model xxx01 and function: shooting with 1 million pixels, it is determined that the risk is low, and only a warning is given as in the countermeasure AP1. Of course, suspicious drone monitoring (shooting, audio acquisition) will continue. Also, in the case of model: zzz03, function: shooting with 16 million pixels, autonomous flight function with GPS, it is judged that the risk is medium, and like the countermeasure AP2, the tracking drone is skipped and the suspicious drone is tracked. Respond to tracking like that. In addition, in the case of model: yyy02, function: shooting with 2 million pixels, and luggage release function up to 1 kg, it is judged that the risk is high and capture is taken. The risk judgment and the response decision may be made based on the function by using AI or the like, but it is preferable to appropriately use each database provided by this system.

FIG. 19 is a flowchart showing an example of processing executed by this system. Steps D31-D38, which are the processes of FIG. 19, are almost the same as the processes of FIG. 17 unless otherwise indicated.

As shown in the figure, step D34-1 is a newly added process. In step D34-1, the monitoring surface (which may be a monitoring line) is changed (set) based on the obtained risk information of the flying object model. This makes it possible to set a monitoring area according to the risk, that is, the performance of the drone model. That is, it is possible to take a stronger monitoring system for dangerous drones and a flexible monitoring system such as loosening the monitoring system for toy-like drone models. The process of step D34-1 will be described in more detail in the following figure.

FIG. 20 is a flowchart in which step D34-1 of FIG. 19 is divided into more detailed steps. As shown in the figure, in step D34-11, based on the obtained flying object information, the monitoring area information (monitoring line, monitoring surface, etc.) is obtained by referring to the flying object monitoring area database. Then, in step D34-12, the monitoring area correction information is obtained with reference to the structure correction database based on the acquired structure information, and the monitoring area information (for example, monitoring) is obtained based on the obtained monitoring area correction information. Correct the line and monitoring surface). According to this process, it is possible to set the monitoring area according to the risk information of the flying object specified as the monitoring target. The greatest advantage of this configuration is that it is possible to suppress unnecessary warning issuance due to detection when a low-performance drone such as a child's toy passes far enough. Of course, on the contrary, for high-performance drones, it is possible to expand the monitoring area and take a sufficient monitoring system. In this system, it is possible to set a monitoring area individually for the monitoring target.

For the above processing, this system may be configured as follows.

The monitoring system
An audio projectile database containing audio data and projectile information associated with the audio data, an image projectile database containing image-related data, and projectile information associated with the image-related data, and a projectile. Flying object monitoring area database including information and monitoring area information associated with the flying object information, and structure information (information on structures such as buildings, facilities, houses, windows, openings, etc., BIM / CIM information) A storage unit for storing the information of the above and a structure correction database including monitoring area correction information based on the structure information.
A voice acquisition unit that acquires voice, and
An image acquisition unit (or an image pickup unit that captures images) that acquires a stereo image,
The structure information acquisition unit that acquires structure information,
A voice analysis unit that analyzes the acquired voice and generates voice analysis data,
An image analysis unit that analyzes the acquired stereo image and generates image analysis data,
The audio data included in the audio projectile database is compared with the generated audio analysis data, and the image data included in the image projectile database is compared with the generated image analysis data. Based on the comparison result, the projectile determination unit that obtains the projectile information associated with the one with the highest degree of matching, and the projectile determination unit,
Based on the obtained flying object information, the risk information identification unit that obtains the monitoring area information (monitoring line, monitoring surface, etc.) by referring to the flying object monitoring area database, and
A monitoring area correction unit that obtains monitoring area correction information based on the acquired structure information with reference to the structure correction database and corrects the monitoring area information based on the obtained monitoring area correction information.
Have.

Alternatively, the system may be configured as follows.
It ’s a monitoring system,
An audio projectile database containing audio data and projectile information associated with the audio data, an image projectile database containing image-related data, and projectile information associated with the image-related data, and a projectile. A storage unit for storing information and a flying object monitoring area database including monitoring area information associated with the flying object information.
A voice acquisition unit that acquires voice, and
An image acquisition unit (or an image pickup unit that captures images) that acquires a stereo image,
A BIM / CIM information acquisition unit that acquires BIM / CIM information including structure information or topographical information,
A voice analysis unit that analyzes the acquired voice and generates voice analysis data,
An image analysis unit that analyzes the acquired stereo image and generates image analysis data,
The audio data included in the audio projectile database is compared with the generated audio analysis data, and the image data included in the image projectile database is compared with the generated image analysis data. Based on the comparison result, the projectile determination unit that obtains the projectile information associated with the one with the highest degree of matching, and the projectile determination unit,
Based on the obtained flying object information and the acquired BIM / CIM information, the risk information identification unit for obtaining monitoring area information (monitoring line, monitoring surface, etc.) by referring to the flying object monitoring area database, and
Have.

FIG. 21 is a schematic diagram illustrating a setting status of a monitoring surface for a flying object approaching a building under the monitoring of this system. As shown in the figure, it is assumed that there is a building on a certain site.
First, it is assumed that the monitoring surface V3 (this is a standard monitoring surface) is set as in the scene SCN31. At this time, there is a drone D31 that approaches the monitoring surface. This system acquires the sound and image of an approaching drone and identifies the drone model.

This system changes and corrects the monitoring surface based on the risk information or model performance according to the specified drone (flying object) model. Alternatively, the information in which the monitoring surface (monitoring area) is set in advance for each model may be used. When a high-performance drone DR33 approaches, as in the scene SCN33, it is preferable to set a wide monitoring surface on the side to which the drone approaches, and to make it like the monitoring surface V33. Alternatively, when a drone DR32 with low performance and low risk approaches, as in the scene SCN32, if the drone approaches, the monitoring surface on that side may be set narrower, or the overall monitoring surface may be set narrower. Therefore, it is preferable to make it like the monitoring surface V32.

FIG. 22 is a flowchart showing an example of processing executed by this system. As shown in the figure, in step D41, the BIM / CIM information acquisition unit provided in the control unit of the server acquires the BIM / CIM information including the structure information or the topographical information.
Next, in step D42, the risk information specifying unit changes or corrects the monitoring area information based on the acquired BIM / CIM information. As the BIM / CIM information, the BIM / CIM information BCI stored in the BIM / CIM information server BIM3 may be acquired via the network, or the BIM / CIM information BCI stored in the storage unit may be acquired. .. BIM / CIM information includes CAD information (including opening information such as windows) of buildings that are structures, topographical information such as rivers and mountains, and information on civil engineering structures such as dams and bridges.

FIG. 23 is a schematic diagram illustrating the setting status of the monitoring surface for a flying object approaching a building under the monitoring of this system. As shown in the scene SCN41 in the figure, the monitoring surface on the windowless side of the building CONS41, like the monitoring surface V41, is only a predetermined distance from the building (may be the site) on the windowless side based on BIM / CIM information. Set the monitoring surface at a remote location. This is because the side without a window is not likely to be observed from the outside such as voyeurism, so there is little need for strict monitoring. On the other hand, as shown in the scene SCN41, the monitoring surface on the side of the building CONS42 with the window WIND is set far away so that the monitoring surface can be monitored widely and far, like the monitoring surface V42, to widen the monitoring area. In this way, it is possible to correct or set the monitoring area according to the BIM / CIM information, and by combining the BIM / CIM information and the drone model information (risk information), It is also possible to set the monitoring area more precisely.

FIG. 24 is a schematic diagram illustrating a setting status of a monitoring line when monitoring a water purification plant. As shown in the figure, in the scene SCN51, the monitoring line VF51 is set around the water purification plant CLW without considering the river RV. In this case, when the drone DR51 approaches the upstream of the river RV, it cannot be detected because it is not a monitoring target area. However, in the case of a water purification plant, water is taken from a river, so it is better to monitor upstream. Therefore, it is the scene SCN52 that uses the BIM / CIM information for upstream monitoring.

As shown in the scene SCN52, the monitoring line VF1 (which may be the monitoring surface) is widely set (may be changed or corrected) from the intake to the upstream of the river based on the BIM / CIM information. By doing so, it becomes possible to detect the drone DR52 approaching the sky upstream of the river VR.

Although the present invention has been described with reference to the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and modifications based on the present disclosure. Therefore, it should be noted that these modifications and modifications are included in the scope of the present invention. For example, the processes and functions included in each part and each step can be rearranged so as not to be logically inconsistent, and multiple means / parts and steps can be combined or divided into one. It is possible. Alternatively, it should be noted that some components, functions, processes, steps, etc. of the devices, methods, programs, etc. according to the present invention can be arranged on a remote server or the like. Further, although the present invention has been described as a real-time monitoring device, it can also be used as an editing device or an editing system that cuts out a time zone or a moment taken by a suspicious object by inputting a captured image.

A, B, C, D Boundary point ACQ acquisition unit BDY Boundary line CAM Imaging unit COM Communication unit CON Control unit CONS2 Building DE Distance calculation unit DE Distance measurement unit DIM Direction measurement unit DIS display unit DR1 Drone DR21 Drone MAP Map MEM storage unit ML monitoring line ML1 monitoring line ML2 monitoring line ML3 monitoring line ML3 monitoring line ML4, ML5 monitoring line MLS monitoring line setting unit MON monitoring unit MP Substantial monitoring surface MP21 monitoring surface MP22 monitoring surface MP23 monitoring surface MP24 monitoring surface MP25 monitoring surface MPS monitoring Surface setting unit MS1 Monitoring system MS2 Monitoring system NET network PF1 Physical fence POM Position measurement unit RES Site information server RES1 Site information server SC Stereo camera SC1 Stereo camera SC2 Stereo camera SCN21 Scene SCN22 Scene SCN23 Scene SI Block split image SI10 Stereo image SI11 Stereo Image SI12 Stereo image SI21 Stereo image SI2221 Stereo image SS Site SS10 Site SS20 Site SV1 Server SV2 Server UIP Suspicious person WOT Alarm output unit

Claims (10)

It ’s a monitoring system,
An audio projectile database containing audio data and projectile information associated with the audio data, an image projectile database containing image-related data, and projectile information associated with the image-related data, and a projectile. A storage unit for storing information and a flying object risk database including risk information associated with the flying object information.
A voice acquisition unit that acquires voice, and
An image acquisition unit that acquires stereoscopic images, and
A voice analysis unit that analyzes the acquired voice and generates voice analysis data,
An image analysis unit that analyzes the acquired stereo image and generates image analysis data,
The audio data included in the audio projectile database is compared with the generated audio analysis data, and the image- related data included in the image projectile database is compared with the generated image analysis data. Based on the comparison result of, the flying object determination unit that obtains the flying object information associated with the voice data or the image-related data having the highest degree of matching, and
Based on the requested flying object information, the risk information identification unit for which risk information is requested by referring to the flying object risk database, and
Monitoring system with. In the monitoring system according to claim 1,
The storage unit
Further stores the risk response database containing the projectile response information associated with the risk information,
The monitoring system is
Based on the requested risk information, it also has a projectile response decision unit that seeks projectile response information by referring to the risk response database.
A monitoring system characterized by that. It ’s a monitoring system,
An audio projectile database containing audio data and projectile information associated with the audio data, an image projectile database containing image-related data, and projectile information associated with the image-related data, and a projectile. A storage unit for storing information and a flying object monitoring area database including monitoring area information associated with the flying object information.
A voice acquisition unit that acquires voice, and
An image acquisition unit that acquires stereoscopic images, and
A voice analysis unit that analyzes the acquired voice and generates voice analysis data,
An image analysis unit that analyzes the acquired stereo image and generates image analysis data,
The audio data included in the audio projectile database is compared with the generated audio analysis data, and the image- related data included in the image projectile database is compared with the generated image analysis data. Based on the comparison result of, the flying object determination unit that obtains the flying object information associated with the one with the highest degree of matching, and the flying object determination unit,
Based on the obtained flying object information, the risk information identification unit for which the monitoring area information is obtained by referring to the flying object monitoring area database, and
Monitoring system with. In the monitoring system according to claim 3,
The storage unit
Further stores the risk response database containing the projectile response information associated with the risk information,
The monitoring system is
Based on the requested risk information, it also has a projectile response decision unit that seeks projectile response information by referring to the risk response database.
A monitoring system characterized by that. In the monitoring system according to claim 3 or 4.
It also has a BIM / CIM information acquisition unit that acquires BIM / CIM information including structure information or topographical information.
The risk information identification department
Change or correct the monitoring area information based on the acquired BIM / CIM information.
A monitoring system characterized by that. It ’s a monitoring system,
An audio projectile database containing audio data and projectile information associated with the audio data, an image projectile database containing image-related data, and projectile information associated with the image-related data, and a projectile. A storage unit for storing information and a flying object monitoring area database including monitoring area information associated with the flying object information.
A voice acquisition unit that acquires voice, and
An image acquisition unit that acquires stereoscopic images, and
A BIM / CIM information acquisition unit that acquires BIM / CIM information including structure information or topographical information,
A voice analysis unit that analyzes the acquired voice and generates voice analysis data,
An image analysis unit that analyzes the acquired stereo image and generates image analysis data,
The audio data included in the audio projectile database is compared with the generated audio analysis data, and the image- related data included in the image projectile database is compared with the generated image analysis data. Based on the comparison result of, the flying object determination unit that obtains the flying object information associated with the one with the highest degree of matching, and the flying object determination unit,
Based on the obtained flying object information and the acquired BIM / CIM information, the risk information identification unit for which the monitoring area information is obtained by referring to the flying object monitoring area database, and
Monitoring system with. It ’s a monitoring system,
An audio projectile database containing audio data and projectile information associated with the audio data, an image projectile database containing image-related data, and projectile information associated with the image-related data, and a projectile. Stores a flying object monitoring area database containing information and monitoring area information associated with the flying object information , and a structure correction database including structure information and monitoring area correction information based on the structure information. Memory and
A voice acquisition unit that acquires voice, and
An image acquisition unit that acquires stereoscopic images, and
The structure information acquisition unit that acquires structure information,
A voice analysis unit that analyzes the acquired voice and generates voice analysis data,
An image analysis unit that analyzes the acquired stereo image and generates image analysis data,
The audio data included in the audio projectile database is compared with the generated audio analysis data, and the image- related data included in the image projectile database is compared with the generated image analysis data. Based on the comparison result of, the flying object determination unit that obtains the flying object information associated with the one with the highest degree of matching, and the flying object determination unit,
Based on the obtained flying object information, the risk information identification unit for which the monitoring area information is obtained by referring to the flying object monitoring area database, and
A monitoring area correction unit that obtains monitoring area correction information based on the acquired structure information with reference to the structure correction database and corrects the monitoring area information based on the obtained monitoring area correction information.
Monitoring system with. In the monitoring system according to claim 7,
It further has a GPS unit that acquires the position information of the image pickup unit or the monitoring system.
The structure information acquisition unit
With reference to the structure information database in which the position information and the structure information are associated with each other, the structure information in which the imaging unit or the monitoring system is located is acquired based on the acquired position information.
A monitoring system characterized by that. A monitoring program that causes one or more arithmetic processing units to function as the monitoring system according to claim 1. A computer-readable storage medium containing the monitoring program according to claim 9.

Images