WO2019225147A1

WO2019225147A1 - Airborne object detection device, airborne object detection method, and airborne object detection system

Info

Publication number: WO2019225147A1
Application number: PCT/JP2019/013456
Authority: WO
Inventors: 信太郎吉國
Original assignee: パナソニックＩｐマネジメント株式会社
Priority date: 2018-05-24
Filing date: 2019-03-27
Publication date: 2019-11-28
Also published as: JP2019203821A

Abstract

This airborne object detection device is provided with: a communication unit capable of communicating with a first camera of which an optical axis can be adjusted and which is disposed so as to be able to capture an image of a monitoring area, or with a microphone array which is disposed so as to be able to pick up sounds in the monitoring area; and a processor which, in the case when the noise level of a sound picked up in the monitoring area is equal to or greater than a predetermined value, performs suppression processing to bring down, by a given value, the sound parameter representing sound magnitude, and which detects the presence or absence of an unmanned air vehicle in the monitoring on the basis of a sound parameter representing the magnitude of the picked-up sound or the sound parameter that has undergone suppression processing. In the case when the optical axis direction of the first camera is to be adjusted toward a direction in which an unmanned air vehicle has been detected, the processor skips the execution of suppression processing for the sound parameter.

Description

Flying object detection device, flying object detection method, and flying object detection system

The present disclosure relates to a flying object detection device, a flying object detection method, and a flying object detection system that detect a flying object in flight for monitoring.

Conventionally, a flying object monitoring apparatus capable of detecting the presence of an object and detecting the flying direction of the object using a plurality of sound detection units that detect sound generated in the monitoring area for each direction is known. (For example, refer to Patent Document 1). When the processing device of this flying flying object monitoring device detects the flying object and its flying direction by detecting sound with a non-directional microphone, it directs the monitoring camera in the flying direction of the flying object. Further, the processing device displays the video captured by the monitoring camera on the display device.

Japanese Unexamined Patent Publication No. 2006-168421

However, in the prior art including Patent Document 1, depending on the surrounding sound environment, it may be erroneously detected that there is a flying object that should not originally exist under the influence of the sound environment, or There is a problem that a flying object that should have already been detected may be lost, and the recognition accuracy of the flying object deteriorates.

The present disclosure has been devised in view of the above-described conventional situation, and a flying object detection device that accurately reduces false detection of an unmanned air vehicle according to the environment of a monitoring area and suppresses deterioration in recognition accuracy of the unmanned air vehicle. It is an object of the present invention to provide a flying object detection method and a flying object detection system.

The present disclosure provides a communication unit that communicates with a first camera that can adjust an optical axis direction and is arranged so as to be able to capture an image of a monitoring area, or a microphone array that is arranged so as to be able to collect sound in the monitoring area. And when the noise level included in the sound of the monitoring area collected by the microphone array is equal to or higher than a predetermined value, the sound parameter indicating the volume of the sound is suppressed by a predetermined value, and the microphone array A processor for detecting the presence or absence of an unmanned air vehicle in the monitoring area based on a sound parameter indicating the volume of the collected sound or the sound parameter subjected to the suppression process, and the processor includes the unmanned vehicle. When the first camera adjusts the optical axis direction in the detection direction in which the flying object is detected, execution of the sound parameter suppression process Omitted, it provides a flying object detecting device.

Further, the present disclosure is a flying object detection method in a flying object detection apparatus that is communicably connected to a first camera that can adjust an optical axis direction or a microphone array, and the microphone array collects sound. When the noise level included in the sound of the monitoring area is equal to or higher than a predetermined value, the step of suppressing the sound parameter indicating the sound volume by a predetermined value, and the volume of the sound collected by the microphone array Detecting the presence or absence of an unmanned aerial vehicle in the monitoring area based on the sound parameter indicating the suppression parameter or the sound parameter subjected to the suppression process, and the step of detecting the presence or absence of the unmanned aerial vehicle includes: When the first camera adjusts the optical axis direction in the detection direction in which the flying object is detected, the execution of the sound parameter suppression process is omitted. Based on the sound parameter indicating the magnitude of the sound picked up by the microphone array to detect the presence or absence of the unmanned air vehicle, to provide a flying object detection method.

In addition, the present disclosure provides a first camera that can adjust the optical axis direction and is arranged so as to be able to capture an image of the monitoring area, a microphone array that is arranged so as to be able to pick up sounds in the monitoring area, and the first camera. Or a flying object detection device that communicates with the microphone array, wherein the flying object detection device has a noise level included in the sound of the monitoring area collected by the microphone array equal to or higher than a predetermined value. The sound parameter indicating the volume of the sound is suppressed by a predetermined value, and the monitoring is performed based on the sound parameter indicating the volume of the sound collected by the microphone array or the sound parameter subjected to the suppression process. When the presence or absence of an unmanned air vehicle in the area is detected, and the first camera adjusts the optical axis direction in the detection direction in which the unmanned air vehicle is detected Omitted execution of suppression of the sound parameter, provides a flying object detection system.

According to the present disclosure, erroneous detection of an unmanned air vehicle can be accurately reduced according to the environment of the monitoring area, and deterioration in recognition accuracy of the unmanned air vehicle can be suppressed.

The figure which shows the system structural example of the flying object detection system which concerns on Embodiment 1. FIG. The figure which shows the external appearance example of a sound source detection unit Block diagram showing an example of the internal configuration of a microphone array Block diagram showing an internal configuration example of an omnidirectional camera Block diagram showing an internal configuration example of a PTZ camera Block diagram showing an example of the internal configuration of the monitoring device Timing chart showing examples of detection sound signal patterns for drones registered in memory Timing chart showing an example of frequency change of detected sound signal obtained as a result of frequency analysis processing Sequence diagram showing an example of the overall operation procedure of the flying object detection system according to the first embodiment The flowchart which shows the example of an operation | movement procedure of the sound control process of step T13 of FIG. Explanatory drawing which shows typically the example of suppression of the sound pressure level of step S5 of FIG. The figure which shows the example of a display screen with which the sound pressure heat map produced | generated based on the sound pressure level which is not suppressed is superimposed on the omnidirectional picked-up image data of a monitoring area The figure which shows the example of a display screen on which the sound pressure heat map produced | generated based on the sound pressure level suppressed by detecting the drone and operating the PTZ camera is superimposed on the omnidirectional captured image data of the monitoring area The figure which shows the example of the sequential scanning of the directional direction performed in the monitoring area at the time of drone detection The flowchart which shows the example of an operation | movement procedure of the detection determination process of the drone of step T16 of FIG. The figure which shows the example of a display screen which displayed the omnidirectional captured image and PTZ captured image side by side when a drone was detected

Hereinafter, an embodiment that specifically discloses a flying object detection device, a flying object detection method, and a flying object detection system according to the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of already well-known matters and repeated descriptions for substantially the same configuration may be omitted. This is to avoid the following description from becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

As an example of a flying object detected by the flying object detection system, an unmanned air vehicle (eg UAV (Unmanned Aerial Vehicle) such as a drone) flying in a monitoring area to be monitored (for example, an outdoor area such as a residential area of a city resident) To explain. The present disclosure can also be defined as a flying object detection device constituting a flying object detection system, a flying object detection method executed in the flying object detection device, and a flying object detection method executed in the flying object detection system. It is.

Hereinafter, a user of the flying object detection system (for example, a supervisor who looks around and monitors the monitoring area) is simply referred to as a “user”.

In the first embodiment, the flying object detection device (for example, the monitoring device 10) can at least collect the sound of the monitoring area, the PTZ camera CZ arranged so that the optical axis direction can be adjusted and the monitoring area can be imaged. Communicating with the microphone array MA arranged in the. When the noise level included in the sound of the monitoring area collected by the microphone array MA is equal to or higher than a predetermined value, the flying object detection device suppresses the sound parameter indicating the loudness by a predetermined value and also suppresses the microphone array. The presence / absence of an unmanned air vehicle (for example, a drone) in the monitoring area is detected based on a sound parameter indicating the volume of sound collected by the MA or a sound parameter subjected to suppression processing. When the PTZ camera CZ adjusts the optical axis direction in the detection direction in which the unmanned flying object is detected, the flying object detection device omits the execution of the sound parameter suppression process.

FIG. 1 is a diagram showing a system configuration example of a flying object detection system 5 according to the first embodiment. The flying object detection system 5 detects the presence or absence of an unmanned flying object (see, for example, the drone DN shown in FIG. 12) that is a detection target in the monitoring area (see above). The unmanned aerial vehicle uses, for example, a GPS (Global Positioning System) function to autonomously ascend, descend, turn left, move left, turn right, move right, or a combination thereof. A drone that flies by performing an action with a certain degree of freedom, or a radio controlled helicopter that flies by a third party's radio control. The unmanned air vehicle can be used for multiple purposes such as aerial photography of a destination or target object, monitoring, chemical spraying, transportation of goods, and the like.

Hereinafter, a multi-copter type drone DN equipped with a plurality of rotors (in other words, rotor blades) will be described as an example of an unmanned air vehicle. In the multi-copter type drone DN, when the number of rotor wings is generally two, a harmonic having a frequency twice as high as a specific frequency and a harmonic having a frequency multiplied by that frequency are generated. Similarly, when the number of rotor wings is three, harmonics having a frequency three times as high as a specific frequency and further harmonics having a frequency multiplied by the harmonics are generated. The same applies when the number of rotor wings is four or more.

The flying object detection system 5 includes a plurality of sound source detection units UD, a monitoring device 10, a monitor MN, and a recorder RC. In FIG. 1, only one sound source detection unit UD is shown to simplify the drawing. The plurality of sound source detection units UD have the same configuration and are connected to the monitoring apparatus 10 via the network NW. The sound source detection unit UD includes a microphone array MA, an omnidirectional camera CA (an example of a second camera), and a PTZ camera CZ (an example of a first camera).

In the sound source detection unit UD, the microphone array MA picks up sound in all directions (that is, 360 degrees) in a non-directional state in the monitoring area (see above) where the device is installed. The microphone array MA has a housing 15 (see FIG. 2) in which a circular opening having a predetermined width is formed at the center. The sound collected by the microphone array MA is, for example, a mechanical operation sound such as a drone DN, a sound generated by a person or the like, a mechanical sound generated by the PTZ camera CZ (for example, pan rotation or tilt rotation, zoom processing) Drive sound of the zoom lens) and other sounds, not only the sound in the audible frequency range (that is, 20 Hz to 20 kHz), but also the low frequency sound lower than the audible frequency and the ultrasonic sound exceeding the audible frequency are included. Good.

The microphone array MA includes a plurality of omnidirectional microphones M1 to Mq (see FIG. 3). q is a natural number of 2 or more. The microphones M1 to Mq are arranged around the circular opening provided in the housing 15 in a concentric predetermined interval (for example, a uniform interval) along the circumferential direction. As the microphones M1 to Mq, for example, electret condenser microphones (ECM) are used. The microphone array MA transmits sound data signals obtained by collecting the sounds of the microphones M1 to Mq to the monitoring device 10 via the network NW. The arrangement of the microphones M1 to Mq is an example, and other arrangements (for example, a square arrangement or a rectangular arrangement) may be used, but the microphones M1 to Mq are preferably arranged at equal intervals. .

The microphone array MA includes a plurality of microphones M1 to Mq (for example, q = 32) and a plurality of amplifiers PA1 to PAq (see FIG. 3) that amplify output signals of the plurality of microphones M1 to Mq, respectively. Analog signals output from each amplifier are converted into digital signals by A / D converters A1 to Aq (see FIG. 3). The number of microphones in the microphone array MA is not limited to 32, but may be other numbers (for example, 16, 64, 128).

An omnidirectional camera CA that substantially matches the volume of the circular opening is accommodated inside the circular opening formed in the center of the casing 15 (see FIG. 2) of the microphone array MA. That is, the microphone array MA and the omnidirectional camera CA are integrated with each other so that the center of each case is in the coaxial direction (see FIG. 2). The omnidirectional camera CA is a camera equipped with a fisheye lens 45a (see FIG. 4) capable of capturing images in all directions (that is, 360 degrees) in a monitoring area (see above) as an imaging area of the omnidirectional camera CA. In the first embodiment, the sound collection area of the microphone array MA and the imaging area of the omnidirectional camera CAk are both described as a common monitoring area, but the spatial size (for example, volume) of the sound collection area and the imaging area is It does not have to be the same. For example, the volume of the sound collection area may be larger or smaller than the volume of the imaging area. In short, the sound collection area and the imaging area only need to have a common space. The omnidirectional camera CA functions as a monitoring camera capable of imaging an imaging area where the sound source detection unit UD is installed, for example. That is, the omnidirectional camera CA has an angle of view of, for example, vertical direction: 180 ° and horizontal direction: 360 °, and images the monitoring area 8 (see FIG. 14), which is a hemisphere, for example, as an imaging area.

In the sound source detection unit UD, the omnidirectional camera CA and the microphone array MA are coaxially arranged by fitting the omnidirectional camera CA inside the circular opening of the housing 15. Thus, the optical axis of the omnidirectional camera CA and the central axis of the housing of the microphone array MA coincide with each other, so that the imaging area and the sound collection area in the axial circumferential direction (that is, the horizontal direction) are substantially the same. The position of the subject in the omnidirectional image captured by the omnidirectional camera CA (in other words, the direction indicating the position of the subject viewed from the omnidirectional camera CA) and the position of the sound source to be collected by the microphone array MA (in other words, The direction of the sound source as viewed from the microphone array MA can be expressed in the same coordinate system (for example, coordinates indicated by (horizontal angle, vertical angle)). The sound source detection unit UD is attached so that, for example, the upward direction in the vertical direction becomes the sound collection surface and the imaging surface in order to detect the drone DN flying in the sky (see FIG. 2).

The monitoring device 10 as an example of a flying object detection device is configured using a computer such as a PC (Personal Computer) or a server. The monitoring apparatus 10 forms directivity with the arbitrary beam direction as the main beam direction based on the user's operation (that is, beam forming) in the omnidirectional sound collected by the microphone array MA. Can emphasize the sound. The details of the process of forming the directivity in the sound data by beam forming the sound collected by the microphone array MA are known techniques as shown in, for example, Reference Patent Documents 1 and 2.

(Reference Patent Document 1) Japanese Laid-Open Patent Publication No. 2014-143678 (Reference Patent Document 2) Japanese Laid-Open Patent Publication No. 2015-029241

The monitoring apparatus 10 uses a captured image captured by the omnidirectional camera CA to cut out an omnidirectional captured image having a field angle of 360 degrees or a part of a specific range (direction) of the omnidirectional captured image to obtain a two-dimensional view. Panorama captured images (hereinafter, “captured images having an angle of view of 360 degrees” and “panoramic captured images” are collectively referred to as “omnidirectional captured images”). Note that the omnidirectional captured image may be generated not by the monitoring device 10 but by the omnidirectional camera CA.

The monitoring device 10 generates an omnidirectional image captured by the omnidirectional camera CA based on a calculated value of a sound parameter (for example, a sound pressure level) that specifies the volume of sound collected by the microphone array MA. An image of the sound pressure heat map (see FIG. 13) is superimposed and displayed on the monitor MN.

The monitoring apparatus 10 uses the first visual image (for example, an identification mark not shown) that is easy for the user to visually identify the detected drone DN, and the position (that is, coordinates) of the corresponding drone DN in the omnidirectional captured image IMG1. May be displayed. In the first visual image, for example, in the omnidirectional captured image IMG1, when the user views the omnidirectional captured image IMG1, the position of the drone DN is explicitly shown to such an extent that it can be clearly distinguished from other subjects. The same shall apply hereinafter.

The monitor MN displays the omnidirectional captured image IMG1 output from the monitoring device 10 or a composite image in which an identification mark (see above) is superimposed on the omnidirectional captured image IMG1. The monitor MN may be configured as a device integrated with the monitoring device 10.

The recorder RC is configured by using a semiconductor memory such as a hard disk (Hard Disk Drive), a solid state drive (Solid State Drive), or a flash memory, for example, and data of various captured images generated by the monitoring device 10 and respective sound sources. The omnidirectional captured image data and sound data sent from the detection unit UD are recorded. Note that the recorder RC may be configured as an apparatus integrated with the monitoring apparatus 10 or may be omitted from the configuration of the flying object detection system 5.

In FIG. 1, a plurality of sound source detection units UD and a monitoring device 10 each have a communication interface and are connected to each other via a network NW so that data communication is possible. The network NW may be a wired network (for example, an intranet, the Internet, a wired LAN (Local Area Network), or a wireless network (for example, a wireless LAN), and each of the sound source detection units UD or a part of the sound source detection units UD. The monitoring device 10 may be directly connected to the monitoring device 10 without going through the network NW, and the monitoring device 10, the monitor MN, and the recorder RC are all installed in the monitoring room RM where the user resides at the time of monitoring.

FIG. 2 is a view showing an example of the appearance of the sound source detection unit UD. The sound source detection unit UD includes a microphone array MA, an omnidirectional camera CA, a PTZ camera CZ, and a support base 70 that mechanically supports them. The support base 70 includes a tripod 71, two rails 72 fixed to the top plate 71 a of the tripod 71, and a first attachment plate 73 and a second attachment plate 74 attached to both ends of the two rails 72, respectively. And has a combined structure.

The first mounting plate 73 and the second mounting plate 74 are mounted so as to straddle the two rails 72 and have substantially the same plane. The first mounting plate 73 and the second mounting plate 74 are slidable on the two rails 72, respectively, and are adjusted and fixed at positions separated or approached from each other.

The first mounting plate 73 is a disk-shaped plate material. A circular opening 73 a is formed at the center of the first mounting plate 73. The casing 15 of the microphone array MA is accommodated and fixed in the circular opening 73a. On the other hand, the second mounting plate 74 is a substantially rectangular plate material. A circular opening 74 a is formed in a portion near the outside of the second mounting plate 74. The PTZ camera CZ is accommodated and fixed in the circular opening 74a.

As shown in FIG. 2, the optical axis L1 of the omnidirectional camera CA housed in the casing 15 of the microphone array MA and the optical axis L2 of the PTZ camera CZ attached to the second mounting plate 74 are in the initial installation state. Are set to be parallel to each other.

The tripod 71 is supported by the grounding surface by three legs 71b, and the position of the top plate 71a can be moved in the vertical direction with respect to the grounding surface by manual operation, and the top and bottom can be moved in the pan and tilt directions. The direction of the plate 71a can be adjusted. Thereby, the sound collection area of the microphone array MA (in other words, the imaging area of the omnidirectional camera CA or the monitoring area of the flying object detection system 5) can be set in an arbitrary direction.

FIG. 3 is a block diagram showing an example of the internal configuration of the microphone array MA. The microphone array MA shown in FIG. 3 includes a plurality of microphones M1 to Mq (for example, q = 32), a plurality of amplifiers PA1 to PAq for amplifying sound data signals output from the plurality of microphones M1 to Mq, and each amplifier PA1 to This configuration includes a plurality of A / D converters A1 to Aq that convert analog sound data signals output from PAq into digital sound data signals, a compression processing unit 25, and a transmission unit 26, respectively.

The compression processor 25 generates a sound data signal packet based on the digital sound data signal output from the A / D converters A1 to An. The transmission unit 26 transmits the packet of the sound data signal generated by the compression processing unit 25 (hereinafter simply referred to as “sound data”) to the monitoring device 10 via the network NW.

As described above, when the microphone array MA receives the sound distribution request (see FIG. 9) from the monitoring device 10, the microphone array MA converts the sound data signals output from the microphones M1 to Mq to the amplifiers PA1 to PA1 corresponding to the amplifiers (Amp). It is amplified by PAq and converted into a digital sound data signal by corresponding A / D converters A1 to Aq. Further, the microphone array MA generates a sound data signal packet in the compression processing unit 25, and continues to transmit the sound data signal packet to the monitoring device 10 via the network NW.

FIG. 4 is a block diagram showing an example of the internal configuration of the omnidirectional camera CA. The omnidirectional camera CA shown in FIG. 4 includes a CPU 41, a communication unit 42, a power management unit 44, an image sensor 45, a fisheye lens 45a, a memory 46, and a network connector 47.

The CPU 41 performs signal processing for overall control of operations of each unit of the omnidirectional camera CA, data input / output processing with other units, data calculation processing, and data storage processing. Instead of the CPU 41, a processor such as MPU (Micro Processing Unit) or DSP (Digital Signal Processor) may be provided.

For example, the CPU 41 is a two-dimensional panorama obtained by cutting out an image in a specific range (direction) from captured image data having a field angle of 360 degrees generated by the image sensor 45 according to the designation of the user who operates the monitoring device 10. Image data (that is, image data obtained by two-dimensional panorama conversion) is generated and stored in the memory 46.

The image sensor 45 is configured using, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor, and photoelectrically converts an optical image of incident light from the monitoring area collected by the fisheye lens 45a. Thus, the captured image data of the monitoring area having an angle of view of 360 degrees is generated and sent to the CPU 41.

The fisheye lens 45a receives and collects incident light from all directions of the imaging area (that is, the monitoring area), and forms an optical image of the incident light on the imaging surface of the image sensor 45.

The memory 46 includes a ROM 46z storing a program for defining the operation of the omnidirectional camera CA and set value data, and panoramic imaging in which captured image data having a field angle of 360 degrees or a part of the range is cut out. A RAM 46y that stores image data and work data, and a memory card 46x that is detachably connected to the omnidirectional camera CAk and stores various data.

The communication unit 42 is a communication interface that controls data communication with the network NW connected via the network connector 47.

The power management unit 44 supplies DC power to each unit of the omnidirectional camera CA. Further, the power management unit 44 may supply DC power to devices connected to the network NW via the network connector 47.

The network connector 47 transmits picked-up image data or panoramic picked-up image data (that is, the above-described omnidirectional picked-up image data) having an angle of view of 360 degrees to the monitoring apparatus 10 via the network NW, and a network cable. It is a connector that can be fed via.

FIG. 5 is a block diagram showing an example of the internal configuration of the PTZ camera CZ. About each part similar to omnidirectional camera CA, the code | symbol corresponding to each part of FIG. 4 is attached | subjected, and the description is abbreviate | omitted. The PTZ camera CZ is a camera that can adjust the optical axis direction (that is, the imaging direction of the PTZ camera CZ) and the zoom magnification according to the PTZ control request (see FIG. 9) from the monitoring device 10.

Similar to the omnidirectional camera CA, the PTZ camera CZ includes a CPU 51, a communication unit 52, a power management unit 54, an image sensor 55, an imaging lens 55a, a memory 56, and a network connector 57, an imaging direction control unit 58, and a lens driving motor. 59. When the CPU 51 receives a PTZ control request (see FIG. 9) from the monitoring device 10, the CPU 51 notifies the imaging direction control unit 58 of an angle of view change instruction.

The imaging direction control unit 58 controls at least one of the panning direction and the tilting direction of the PTZ camera CZ in accordance with the view angle change instruction notified from the CPU 51, and further changes the zoom magnification as necessary. Control signal is sent to the lens drive motor 59. The lens drive motor 59 drives the imaging lens 55a according to this control signal, changes its imaging direction (direction of the optical axis L2 shown in FIG. 2), and adjusts the focal length of the imaging lens 55a to increase the zoom magnification. change.

The imaging lens 55a is configured using one or two or more lenses. In the imaging lens 55a, the optical axis direction of pan rotation and tilt rotation or the zoom magnification is changed by driving the lens drive motor 59 according to the control signal from the imaging direction control unit 58.

FIG. 6 is a block diagram showing an example of the internal configuration of the monitoring apparatus 10. The monitoring apparatus 10 illustrated in FIG. 6 includes a communication unit 31, an operation unit 32, a signal processing unit 33, an SPK 37, a memory 38, and a setting management unit 39. SPK is an abbreviation for speaker.

The communication unit 31 receives the omnidirectional captured image data transmitted from the omnidirectional camera CA and the sound data transmitted from the microphone array MA, and sends them to the signal processing unit 33.

The operation unit 32 is a user interface (UI) for notifying the signal processing unit 33 of the contents of the user's input operation, and includes an input device such as a mouse or a keyboard. The operation unit 32 may be configured using, for example, a touch panel or a touch pad that is arranged correspondingly on the display screen of the monitor MN and can be directly input by a user's finger or stylus pen.

When the user designates the red region RD1 of the sound pressure heat map (see FIG. 13) superimposed on the omnidirectional captured image IMG1 of the omnidirectional camera CA on the monitor MN, the operation unit 32 displays the designated position. The indicated coordinates are acquired and sent to the signal processing unit 33. The signal processing unit 33 reads out the sound data collected by the microphone array MA from the memory 38, forms directivity from the microphone array MAk in the direction toward the actual sound source position corresponding to the designated position, and generates the directivity from the SPK 37. Output sound. Thus, the user can check not only the drone DN but also the sound at the position specified by the user on the omnidirectional captured image IMG1 in an emphasized state.

The signal processing unit 33 is configured using a processor such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array). The signal processing unit 33 (that is, the processor) is configured to control and control the operation of each unit of the monitoring apparatus 10, data input / output processing with other units, data calculation (calculation) processing, Data storage processing is performed. The signal processing unit 33 includes a sound source direction detection unit 34, an output control unit 35, a directivity processing unit 63, a frequency analysis unit 64, an object detection unit 65, a detection result determination unit 66, a scanning control unit 67, and a detection direction control unit 68. including. The monitoring device 10 is connected to the monitor MN.

The sound source direction detection unit 34 estimates the sound source position in the monitoring area using the sound collected by the microphone array MA, for example, in accordance with a known whitening cross-correlation method (CSP (Cross-power Spectrum Spectrum Analysis) method). To do. In the CSP method, the sound source direction detection unit 34 divides the monitoring area 8 shown in FIG. 14 into a plurality of blocks (see later), and when sound is picked up by the microphone array MA, the sound source direction detection unit 34 generates a sound for each block (see later). Of the sound parameters indicating the magnitude of the sound (for example, the normalized output value of the cross-correlation value between the sound data respectively collected by the two microphones of the plurality of microphones M1 to Mq constituting the microphone array MA) ) Exceeds a threshold (predetermined value), the sound source position in the monitoring area 8 can be roughly estimated.

The sound source direction detection unit 34 constitutes omnidirectional captured image data in the monitoring area 8 based on the omnidirectional captured image data captured by the omnidirectional camera CA and the sound data collected by the microphone array MA ( For example, a sound parameter indicating a loudness at a position corresponding to the block for each pixel group including a predetermined number of pixels such as “2 * 2” and “4 * 4”. (For example, the above-described normalized output value of the cross-correlation value or sound pressure level) is calculated. The sound source direction detection unit 34 sends the calculation result of the sound parameter for each block constituting the omnidirectional captured image data to the output control unit 35. The sound parameter calculation process described above is a known technique, and a detailed description of the process is omitted.

The sound source direction detection unit 34 constantly monitors the noise level included in the sound data collected by the microphone array MA, and when the noise level exceeds a predetermined value, a sound parameter indicating the size of the sound data ( For example, the cross-correlation value normalized output value or sound pressure level) is suppressed by a predetermined value QT (see FIG. 11).

The reason for this suppression processing is as follows.

Specifically, if the noise level is high enough to exceed a certain value (predetermined value) even though the drone DN does not exist in the monitoring area 8, the monitoring device 10 turns the drone into the monitoring area. This is because there is a high possibility of being erroneously detected as having appeared. Here, the ghost drone is a virtual drone in which the substance of the drone DN is not detected, but it is erroneously detected that the drone DN exists in the noise level higher than the above-mentioned fixed value. That's it.

FIG. 11 is an explanatory diagram schematically showing an example of suppression of the sound pressure level in step S5 of FIG. Note that FIG. 11 is not limited to the example of suppressing the sound pressure level, and can also be applied as an example of suppressing the normalized output value of the cross-correlation value described above. The detailed description of FIG. 10 will be described later, but here, the sound pressure level suppression processing will be briefly described with reference to FIG. In each of the two graphs shown in FIG. 11, the horizontal axis indicates the frequency, and the vertical axis indicates the sound pressure level. The characteristic Cv1 is an example of the characteristic of the sound pressure level before the sound pressure level suppression process is performed (that is, the frequency characteristic of the sound data collected by the microphone array MA). Similarly, the characteristic Cv2 is the characteristic of the sound pressure level after the sound pressure level suppression process is performed (that is, the frequency characteristic in which the frequency characteristic of the sound data collected by the microphone array MA is suppressed by the predetermined value QT). (Characteristic). The frequency fp indicates the frequency when the peak value of the sound pressure level in the characteristics CV1 and Cv2 is obtained. The first threshold value is a threshold value for determining whether or not the drone DN exists, and is specifically referred to in the object detection unit 65 described later.

When the noise level (not shown) included in the sound data collected by the microphone array MA exceeds the above-described constant value, the sound source direction detection unit 34 sets the frequency characteristic of the sound pressure level indicated by the characteristic Cv1 to a predetermined value QT. The suppression processing is performed as much as possible (see FIG. 11). As a result, the monitoring apparatus 10 suppresses (that is, lowers) the frequency characteristic of the sound pressure level by the predetermined value QT in a state where a noise level higher than the above-described constant value is generated. In addition, a frequency characteristic (see characteristic Cv2) in which the peak value of the sound pressure level is less than the first threshold value is obtained, and erroneous detection of the presence of the ghost drone described above can be effectively suppressed.

However, although details will be described with reference to FIG. 9, the PTZ camera CZ performs PTZ control when receiving a PTZ control request from the monitoring device 10 in response to detection of the drone DN. At this time, a loud sound (see above) is generated when the PTZ camera CZ is rotated by panning, tilting, or moving the imaging lens 55a. When the sound source direction detection unit 34 regards the large generated sound as a noise level higher than the above-described constant value and suppresses the frequency characteristic of the sound pressure level by a predetermined value QT, the peak value of the sound pressure level is less than the first threshold value. There is a problem that the drone DN that should originally exist does not exist (in other words, the drone DN is lost).

Therefore, the sound source direction detection unit 34 suppresses the predetermined value QT of the frequency characteristic of the sound pressure level described above during the operation of the PTZ camera CZ (that is, during pan rotation, tilt rotation, and movement of the imaging lens 55a). It is conceivable to perform control so as to omit the execution of.

However, if the execution of the suppression process of the predetermined value QT of the frequency characteristic of the sound pressure level described above is omitted when the drone DN does not exist, the ghost drone as described above appears in the monitoring area 8 and the drone DN There is a possibility of being falsely detected as being detected.

Therefore, in the first embodiment, the sound source direction detection unit 34 suppresses the predetermined value QT of the frequency characteristic of the sound pressure level (see FIG. 10) when either the following first condition or second condition is satisfied. The execution of the indicated step St5) is omitted (see FIG. 10).

The first condition is when the noise level included in the sound data collected by the microphone array MA is not higher than a certain value. When the first condition is satisfied, since the noise level is lower than a certain value, it is considered that the noise level does not cause the above-described ghost drone, and that the monitoring device 10 does not erroneously detect the presence of the ghost drone. It is.

The second condition is that when the presence of the drone DN is detected by the monitoring device 10 and the PTZ camera CZ operates, the noise level included in the sound data collected by the microphone array MA is recognized to be higher than a certain value. Is the case. When the second condition is satisfied, if the PTZ camera CZ operates (for example, pan rotation, tilt rotation, zoom processing) when the drone DN is detected, if the suppression processing is performed, the drone DN originally exists. This is because although the sound pressure level is lowered, the substance of the drone DN cannot be detected and is missed, and the detection accuracy of the drone DN of the monitoring device 10 is deteriorated. Details regarding whether or not to suppress the predetermined value QT of the frequency characteristic of the sound pressure level in the sound source direction detector 34 will be described in detail with reference to FIG.

The setting management unit 39 holds a coordinate conversion formula relating to the coordinate conversion of the position specified by the user on the display screen of the monitor MN on which the omnidirectional captured image data captured by the omnidirectional camera CA is displayed. This coordinate conversion formula is based on, for example, the physical distance difference between the installation position of the omnidirectional camera CA (see FIG. 2) and the installation position of the PTZ camera CZ (see FIG. 2). This is a mathematical formula for converting the coordinates of the designated position (that is, (horizontal angle, vertical angle)) into coordinates in the direction viewed from the PTZ camera CZ.

The signal processing unit 33 uses the coordinate conversion formula held by the setting management unit 39 to determine the position designated by the user from the installation position of the PTZ camera CZ with reference to the installation position of the PTZ camera CZ (see FIG. 2). The coordinates (θMAh, θMAv) indicating the directivity direction toward the actual sound source position corresponding to is calculated. θMAh is a horizontal angle in a direction toward the actual sound source position corresponding to the position designated by the user as viewed from the installation position of the PTZ camera CZ. θMAv is a vertical angle in a direction toward the actual sound source position corresponding to the position designated by the user as viewed from the installation position of the PTZ camera CZ. As shown in FIG. 2, the distance between the omnidirectional camera CA and the PTZ camera CZ is known, and the optical axes L1 and L2 are parallel. It can be realized by geometric calculation. The sound source position is an actual sound source position corresponding to the position designated from the operation unit 32 by the operation of the user's finger or stylus pen with respect to the omnidirectional captured image data displayed on the monitor MN.

As shown in FIG. 2, in the first embodiment, the omnidirectional camera CA and the microphone array MA are arranged so that the optical axis direction of the omnidirectional camera CA and the central axis of the casing of the microphone array MA are coaxial. Has been placed. For this reason, the coordinates of the user-specified position calculated by the omnidirectional camera CA according to the user's specification for the monitor MN on which the omnidirectional captured image data is displayed are the sound emphasis direction (also referred to as the directional direction) viewed from the microphone array MA. ). In other words, when the user designates the monitor MN on which the omnidirectional captured image data is displayed, the monitoring apparatus 10 transmits the coordinates of the designated position on the omnidirectional captured image data to the omnidirectional camera CA. As a result, the omnidirectional camera CA uses the coordinates of the designated position transmitted from the monitoring device 10, and the coordinates (horizontal angle, vertical) indicating the direction of the sound source position corresponding to the user designated position as seen from the omnidirectional camera CA. Corner). Since the calculation process in the omnidirectional camera CA is a known technique, a description thereof will be omitted. The omnidirectional camera CA transmits a calculation result of coordinates indicating the direction of the sound source position to the monitoring device 10. The monitoring device 10 can use the coordinates (horizontal angle, vertical angle) calculated by the omnidirectional camera CA as coordinates (horizontal angle, vertical angle) indicating the direction of the sound source position as viewed from the microphone array MA.

However, when the omnidirectional camera CA and the microphone array MA are not arranged coaxially, the setting management unit 39 performs, for example, an omnidirectional camera according to a method described in Japanese Patent Application Laid-Open No. 2015-029241. It is necessary to convert the coordinates calculated by the CA into coordinates in the direction viewed from the microphone array MA.

The setting management unit 39 compares the sound pressure level for each block constituting the omnidirectional captured image data calculated by the sound source direction detection unit 34 with a first threshold th1, a second threshold th2, and a third threshold th3 (for example, FIG. 9). Here, the sound pressure level is used as an example of a sound parameter indicating the volume of sound generated in the monitoring area 8 collected by the microphone array MA, and the volume indicating the volume of sound output from the SPK 37 It is a different concept. The first threshold th1, the second threshold th2, and the third threshold th3 are thresholds that are compared with the sound pressure level of the sound generated in the monitoring area 8, for example, for determining the presence or absence of the sound generated by the drone DN It may be set as a threshold value. In addition to the first threshold th1, the second threshold th2, and the third threshold th3 described above, a plurality of thresholds can be set, and only the first threshold th1 may be used. Here, for the sake of simple explanation, for example, the first threshold th1, the second threshold th2 that is smaller than this, and the third threshold th3 that is smaller than this are set (first threshold th1>). Second threshold th2> third threshold th3, see FIG.

As will be described later, in the sound pressure heat map generated by the output control unit 35, the omnidirectional captured image data is stored in the red region RD1 (see FIG. 16) of the pixel in which the sound pressure level larger than the first threshold th1 is obtained. On the displayed monitor MN, for example, it is drawn in red. In addition, the pink region PD1 of the pixel in which the sound pressure level greater than the second threshold th2 and less than or equal to the first threshold th1 is obtained is rendered, for example, in pink on the monitor MN on which the omnidirectional captured image data is displayed. The blue region BD1 of the pixel for which the sound pressure level greater than the third threshold th3 and less than or equal to the second threshold th2 is obtained is rendered in blue, for example, on the monitor MN on which the omnidirectional captured image data is displayed. Further, the sound pressure level region N1 (see FIG. 16) of the pixel equal to or smaller than the third threshold th3 is drawn, for example, in a colorless manner on the monitor MN on which the omnidirectional captured image data is displayed. No change in display color.

The SPK 37 is a speaker, and the sound data of the monitoring area 8 collected by the microphone array MA, the sound data after the sound pressure level of the sound data is suppressed by the predetermined value QT, or the sound data collected by the microphone array MA The sound data having the directivity formed by the signal processing unit 33 is output as sound. The SPK 37 may be configured as a separate device from the monitoring device 10.

The memory 38 is configured using, for example, a ROM or a RAM, and holds various data including sound data of a certain section, setting information, a program, and the like. The memory 38 also has a pattern memory (see FIG. 7) in which a sound pattern unique to each drone DN is registered. Further, the memory 38 stores sound pressure heat map data generated by the output control unit 35. In addition, data of an identification mark (not shown) that schematically represents the position of the drone DN is registered in the memory 38. The identification mark used here is, for example, a star symbol. The identification mark is not limited to a star shape, but may be a circle, a triangle, a square, or a symbol or character such as a “卍” shape that reminds a drone DN. Further, the display mode of the identification mark may be changed between daytime and nighttime. For example, it may be a star shape during the daytime and a square that is not mistaken for a star at nighttime. Further, the identification mark may be dynamically changed. For example, a star symbol may be blinked or rotated, and the user can be further alerted.

FIG. 7 is a timing chart showing an example of the detection sound pattern of the drone DN registered in the memory 38. The detected sound pattern shown in FIG. 7 is a combination of frequency patterns, and includes sounds of four frequencies f1, f2, f3, and f4 generated by the rotation of the four rotors mounted on the multicopter type drone DN. The signals of the respective frequencies are signals having different sound frequencies that are generated, for example, with the rotation of a plurality of wings pivotally supported by each rotor.

In FIG. 7, the frequency region indicated by the diagonal lines is a region where the sound pressure level is high. The detected sound pattern may include not only the number of sounds having a plurality of frequencies and the sound pressure level but also other sound information. For example, a sound pressure level ratio representing a ratio of sound pressure levels at each frequency can be used. Here, as an example, the detection of the drone DN is determined by whether or not the sound pressure level of each frequency included in the detection sound pattern exceeds a threshold value.

The directivity processing unit 63 uses the sound data signals collected by the non-directional microphones M1 to Mq, performs the above-described directivity formation processing (beam forming), and makes the sound having the direction of the monitoring area 8 as the directivity direction. Data signal extraction processing is performed. The directivity processing unit 63 can also perform a sound data signal extraction process in which the range in the direction of the monitoring area 8 is the directional range. Here, the directivity range is a range including a plurality of adjacent directivity directions, and is intended to include a certain extent of directivity direction compared to the directivity direction.

The frequency analysis unit 64 performs frequency analysis processing on the sound data signal extracted in the directivity direction by the directivity processing unit 63. In this frequency analysis process, the frequency and the sound pressure level included in the sound data signal in the pointing direction are detected.

FIG. 8 is a timing chart showing an example of the frequency change of the detected sound signal obtained as a result of the frequency analysis process. In FIG. 8, four frequencies f11, f12, f13, f14 and sound pressure levels of the respective frequencies are obtained as detected sound signals (that is, detected sound data signals). In the figure, the fluctuation of each frequency that changes irregularly occurs due to, for example, a rotational fluctuation of the rotor (rotary blade) that slightly changes when the drone DN controls the attitude of the drone DN itself.

The object detection unit 65 performs the detection process of the drone DN using the frequency analysis processing result of the frequency analysis unit 64. Specifically, in the detection process of the drone DN, the object detection unit 65 detects the pattern of the detection sound obtained as a result of the frequency analysis process (see the frequencies f11 to f14 shown in FIG. 8) in the monitoring area 8. The detected sound patterns (see frequencies f1 to f4 shown in FIG. 7) registered in advance in the pattern memory of the memory 38 are compared. The object detection unit 65 determines whether or not the patterns of both detection sounds are approximate.

Whether or not both patterns are approximated is determined, for example, as follows. When the sound pressure of at least two frequencies included in the detected sound data among the four frequencies f1, f2, f3, and f4 exceeds a threshold value (for example, the first threshold th1 shown in FIG. 9 or FIG. 11), the sound pattern is As an approximation, the object detection unit 65 detects the drone DN. Note that the drone DN may be detected when other conditions are satisfied.

When it is determined that the drone DN does not exist, the detection result determination unit 66 instructs the detection direction control unit 68 to shift to detection of the drone DN in the next directivity direction. The detection result determination unit 66 notifies the output control unit 35 of the detection result of the drone DN when it is determined that the drone DN exists as a result of the scanning in the directivity direction. The detection result includes information on the detected drone DN. The information of the drone DN includes, for example, identification information of the drone DN and position information (for example, direction information) of the drone DN in the monitoring area 8.

The detection direction control unit 68 controls the direction for detecting the unmanned air vehicle dn in the sound collection space based on an instruction from the detection result determination unit 66. For example, the detection direction control unit 68 detects an arbitrary direction of the directivity range BF1 (see FIG. 11) including the sound source position estimated by the sound source direction detection unit 34 in the entire sound collection space (that is, the monitoring area 8). Set as direction.

The scanning control unit 67 instructs the directivity processing unit 63 to perform beam forming using the detection direction set by the detection direction control unit 68 as the directivity direction.

The directivity processing unit 63 performs beam forming in the directivity direction instructed from the scanning control unit 67. In the initial setting, the directivity processing unit 63 sets the initial position in the directivity range BF1 (see FIG. 14) including the sound source position estimated by the sound source direction detection unit 34 as the directivity direction BF2. The directivity direction BF2 is sequentially set by the detection direction control unit 68 within the directivity range BF1.

The output control unit 35 controls each operation of the monitor MN and the SPK 37, displays the omnidirectional captured image data sent from the omnidirectional camera CA on the monitor MN, and further outputs the sound data sent from the microphone array MA. Audio is output to the SPK 37. When the drone DN is detected, the output control unit 35 superimposes an identification mark (not shown) indicating the drone DN on a corresponding position (that is, a coordinate indicating the position of the drone DN) of the omnidirectional captured image data. And display it on the monitor MN.

The output control unit 35 uses the sound data collected by the microphone array MA and the coordinates indicating the direction of the sound source position derived by the omnidirectional camera CA, and the directivity of the sound data collected by the microphone array MA. By performing the forming process, the sound data in the directivity direction is emphasized. The sound data directivity forming process is a known technique described in, for example, Japanese Patent Application Laid-Open No. 2015-029241.

The output control unit 35 uses the sound pressure level for each block constituting the omnidirectional captured image data calculated by the sound source direction detection unit 34, and positions the corresponding block for each block constituting the omnidirectional captured image data. A sound pressure map to which the calculated value of the sound pressure level is assigned is generated. Further, the output control unit 35 performs a color conversion process on the sound pressure level for each block of the generated sound pressure map to a visual image (for example, a colored image) so as to be visually and easily discriminable for the user. Thus, the sound pressure heat map shown in FIG. 16 is generated.

The output control unit 35 has been described as generating a sound pressure map or a sound pressure heat map in which the sound pressure level calculated in units of blocks is assigned to the position of the corresponding block. Alternatively, the sound pressure level may be calculated and a sound pressure map or a sound pressure heat map in which the sound pressure level for each pixel is assigned to the position of the corresponding pixel may be generated.

Next, the operation of the flying object detection system 5 according to the first embodiment will be described in detail with reference to FIGS.

FIG. 9 is a sequence diagram showing an example of the overall operation procedure of the flying object detection system 5 according to the first embodiment. When power is supplied to each device (for example, the monitor MN, the monitoring device 10, the PTZ camera CZ, the omnidirectional camera CA, and the microphone array MA) of the flying object detection system 5, the flying object detection system 5 starts its operation. .

First, the signal processing unit 33 of the monitoring apparatus 10 sends an image distribution request to the PTZ camera CZ via the communication unit 31 (T1). The PTZ camera CZ starts imaging processing according to power-on in accordance with the image distribution request sent from the monitoring device 10 (T2). Similarly, the signal processing unit 33 of the monitoring apparatus 10 sends an image distribution request to the omnidirectional camera CA via the communication unit 31 (T3). The omnidirectional camera CA starts imaging processing according to power-on in accordance with the image distribution request sent from the monitoring device 10 (T4). Further, the signal processing unit 33 of the monitoring apparatus 10 sends a sound distribution request to the microphone array MA via the communication unit 31 (T5). The microphone array MA starts sound collection processing in response to power-on in accordance with the sound distribution request sent from the monitoring device 10 (T6).

The PTZ camera CZ transmits PTZ captured image data (for example, a still image and a moving image) obtained by imaging to the monitoring apparatus 10 via the network NW (T7). The output control unit 35 of the monitoring apparatus 10 converts the PTZ captured image data sent from the PTZ camera CZ into display data such as NTSC, and outputs it to the monitor MN to send an instruction to display the PTZ captured image data together with the display data. (T8). The monitor MN displays the data (see FIG. 16) of the PTZ image IMG2 captured by the PTZ camera CZ on the display screen according to the instruction sent from the monitoring device 10 (T9).

The omnidirectional camera CA transmits omnidirectional captured image data (for example, a still image and a moving image) obtained by imaging to the monitoring apparatus 10 via the network NW (T10). The output control unit 35 of the monitoring device 10 converts the omnidirectional captured image data sent from the omnidirectional camera CA into display data such as NTSC, and outputs it to the monitor MN to display an instruction to display the omnidirectional captured image data. It is sent together with the data (T11). The monitor MN displays data (see FIG. 16) of the omnidirectional image IMG1 taken by the omnidirectional camera CA on the display screen according to the instruction sent from the monitoring device 10 (T9).

The microphone array MA encodes the sound data of the monitoring area 8 obtained by sound collection through the network NW and transmits the encoded sound data to the monitoring device 10 (T12).

The sound source direction detection unit 34 of the monitoring apparatus 10 executes various sound control processes (see FIG. 10) including the sound source detection process in the monitoring area 8 using the sound data sent in step T12 (T13). Details of the sound control process in step T13 will be described later with reference to FIG.

The output control unit 35 of the monitoring device 10 uses the sound pressure level for each block constituting the omnidirectional captured image data calculated by the sound source direction detection unit 34 in step T13, and for each block constituting the omnidirectional captured image data. Then, a sound pressure map in which the calculated value of the sound pressure level is assigned to the position of the corresponding block is generated. The output control unit 35 performs a color conversion process on the sound pressure level for each block of the generated sound pressure map to a visual image (for example, a colored image) so that it is visually and easily discriminable for the user. 16 is generated (T14).

The output control unit 35 of the monitoring device 10 converts the sound pressure heat map data generated in step T14 into display data such as NTSC, and outputs it to the monitor MN to output the sound pressure heat map onto the omnidirectional image data. Is sent together with the display data (T15). The monitor MN displays the sound pressure heat map superimposed on the omnidirectional captured image data on the display screen according to the instruction sent from the monitoring device 10 (see FIG. 16, T9).

The signal processing unit 33 of the monitoring device 10 uses the sound data sent from the microphone array MA in step T12 to form directivity while sequentially scanning the directivity direction within the monitoring area 8, thereby making it possible to perform each directivity direction. The drone DN is detected and determined (T16). Details of the detection determination process for the unmanned air vehicle dn will be described later with reference to FIGS. 14 and 15.

If the drone DN is not detected in step T16, the process of the flying object detection system 5 according to the first embodiment returns to step T7, and a series of processes from step T7 to step T16 is repeated. On the other hand, when the drone DN is detected in step T16, the process of the flying object detection system 5 according to the first embodiment proceeds to the process after step T17.

When the drone DN is detected as a result of the drone DN detection determination process in step T16, the output control unit 35 of the monitoring device 10 displays the omnidirectional captured image IMG1 displayed on the display screen of the monitor MN in step T16. An instruction to superimpose and display an identification mark (not shown) indicating the drone DN existing in the pointing direction detected in (T17). In accordance with the instruction sent from the monitoring device 10, the monitor MN synthesizes (superimposes) an identification mark (not shown) indicating the drone DN and displays it on the omnidirectional image IMG1 (T18).

The output control unit 35 of the monitoring apparatus 10 provides information regarding the detection direction of the drone DN detected in step T16 (in other words, the direction set when the drone DN is detected) to the PTZ camera CZ, and the PTZ camera CZ. A PTZ control request for changing the optical axis direction (in other words, the imaging direction) and the zoom magnification is sent in association with each other (T19).

In accordance with the PTZ control request sent from the monitoring device 10, the PTZ camera CZ drives the lens driving motor 59 in the imaging direction control unit 58 based on information on the directivity direction, for example, and the optical axis of the imaging lens of the PTZ camera CZ L2 is changed, and the imaging direction is changed to the pointing direction (T20). At the same time, the imaging direction control unit 58 changes the zoom magnification of the imaging lens 55a of the PTZ camera CZ to a preset value, a value corresponding to the proportion of the captured image of the drone DN, or the like.

Thereby, the PTZ camera CZ can adjust (that is, direct) the optical axis direction in the detection direction (that is, the directing direction) of the drone DN detected in step T16, and can further determine the details of the drone DN. The zoom magnification can be changed so that the PTZ captured image IMG2 can be captured. Thereafter, when the PTZ captured image IMG2 captured by the PTZ camera CZ is sent to the monitoring device 10, the monitoring device 10 detects the omnidirectional captured image IMG1 in which the wide area of the monitoring area 8 is known and the drone PTZ detected in the monitoring area 8. The PTZ captured image IMG2 whose details are known can be displayed side by side on the monitor MN1 (see FIG. 16).

FIG. 16 is a diagram illustrating a display screen example in which the omnidirectional captured image IMG1 and the PTZ captured image IMG2 are displayed side by side when the drone DN is detected. As described with reference to FIG. 9, in the first embodiment, if the sound pressure level for each block calculated by the sound source direction detection unit 34 is equal to or less than the third threshold th3, the sound pressure level is displayed in a colorless manner. If it is greater than the third threshold th3 and less than or equal to the second threshold th2, it is displayed in blue, and if the sound pressure level is greater than the second threshold th2 and less than or equal to the first threshold th1, it is displayed in pink, and the sound pressure level is If it is larger than one threshold th1, it is displayed in red.

As shown in FIG. 16, the omnidirectional captured image IMG1 shows a sound pressure heat map in which the presence or absence of a sound source can be determined in a wide area together with the appearance of the monitoring area 8. For example, since the sound pressure level is higher than the first threshold th1 near the rotor blades and the rotor around the housing center of the drone DN, red regions RD1, RD2, RD3, as an example of the first visual image showing the position of the drone DN It is drawn with RD4. Similarly, pink areas PD1, PD2, PD3, and PD4 indicating that the sound pressure value is next to the red area are drawn as an example of the first visual image indicating the position of the drone DN around the red area. Yes. Similarly, blue areas BD1, BD2, BD3, and BD4 indicating that the sound pressure value is next to the pink area are drawn as an example of the first visual image indicating the position of the drone DN around the pink area. ing.

On the other hand, in the PTZ captured image IMG2, the details of the appearance of the drone DN detected in the monitoring area 8 are shown to be clarified. Therefore, the user can browse the omnidirectional captured image IMG1 and the PTZ captured image IMG2 shown in FIG. 16 side by side to see what kind of appearance the drone DN is detected in and around the monitoring area 8. It can be grasped in detail.

Thereafter, the process of the flying object detection system 5 according to the first embodiment returns to step T7, and each process after step T7 is repeated until a predetermined event such as a power-off operation is detected. .

Next, details of the operation procedure of the sound control process in step T13 in FIG. 9 will be described with reference to FIGS. FIG. 10 is a flowchart showing an example of the operation procedure of the sound control process in step T13 of FIG. FIG. 12 is a diagram illustrating a display screen example in which a sound pressure heat map generated based on the sound pressure level that is not suppressed is superimposed on the omnidirectional captured image data in the monitoring area 8. FIG. 13 shows a display screen in which the sound pressure heat map generated based on the sound pressure level suppressed by detecting the drone DN and operating the PTZ camera CZ is superimposed on the omnidirectional captured image data in the monitoring area 8. It is a figure which shows an example.

In FIG. 10, the sound source direction detection unit 34 determines the omnidirectional captured image data in the monitoring area 8 based on the omnidirectional captured image data captured by the omnidirectional camera CA and the sound data collected by the microphone array MA. A sound parameter (for example, sound pressure level) is calculated for each block to be configured (S1). The sound source direction detector 34 detects the sound source in the monitoring area 8 using the calculated sound parameter (S1). The detected position of the sound source may be used as a reference position of the directivity range BF1 necessary for setting the initial directivity direction when the monitoring apparatus 10 detects the drone DN in step T16.

The sound source direction detection unit 34 constantly monitors and measures the noise level included in the sound data collected by the microphone array MA (S2). When it is determined that the noise level is less than a certain value (predetermined value) (S3, NO), the sound control process of the sound source direction detection unit 34 is terminated. Therefore, the sound source direction detection unit 34 does not execute the above-described frequency characteristic suppression process (see FIG. 11) of the sound pressure level (see the first condition described above), and the process of step T13 ends.

On the other hand, when the sound source direction detection unit 34 determines that the noise level measured in step S2 is equal to or higher than a predetermined value (predetermined value) (S3, YES), the memory 38 is referred to and the PTZ camera CZ is in operation. It is determined whether or not there is (S4). When the output control unit 35 of the monitoring apparatus 10 causes the PTZ camera CZ to execute PTZ control, the memory 38 temporarily holds, for example, a PTZ control request sent to the PTZ camera CZ. The sound source direction detection unit 34 can determine whether or not the current time is during the operation of the PTZ camera CZ based on, for example, whether or not the PTZ control request is held in the memory 38.

When the sound source direction detection unit 34 determines that the current time is not during the operation of the PTZ camera CZ (S4, NO), a large occurrence that occurs when the panning, tilting, or movement of the imaging lens 55a from the PTZ camera CZ is performed. Since no sound is generated, the frequency characteristic of the sound pressure level of the sound data collected by the microphone array MA is suppressed by a predetermined value QT (see FIG. 11, S5). After step S5, the sound control process of the sound source direction detection unit 34 ends, and the process of step T13 ends.

On the other hand, when the sound source direction detection unit 34 determines that the current time is during the operation of the PTZ camera CZ (S4, YES), the sound source direction detection unit 34 refers to the memory 38 and determines whether the drone DN is being detected (S6). ). For example, in the loop (LOOP) process of FIG. 9, when it is determined that the drone DN is detected at the time of executing the process of step T <b> 16 of the previous cycle, the detection direction of the drone DN determined at that time is stored in the memory 38. Information on (that is, the scanned directivity direction) is held. The sound source direction detection unit 34 can determine whether or not the current time is detecting the drone DN based on, for example, whether or not the information regarding the detection direction is held in the memory 38.

When the sound source direction detection unit 34 determines that the current time is that the drone DN is being detected (S6, YES), the sound source direction detection unit 34 ends the sound control processing. Accordingly, the sound source direction detection unit 34 does not execute the above-described suppression process of the frequency characteristic of the sound pressure level (see FIG. 11) (see the second condition described above), and the process of step T13 ends.

On the other hand, if the sound source direction detection unit 34 determines that the drone DN is not currently detected (S6, NO), the sound source direction is not picked up by the suppression process, so the sound is picked up by the microphone array MA. The frequency characteristic of the sound pressure level of the sound data is suppressed by the predetermined value QT (see FIG. 11, S5). After step S5, the sound control process of the sound source direction detection unit 34 ends, and the process of step T13 ends.

In FIG. 12, in the omnidirectional captured image IMG1 displayed on the monitor MN, for example, the noise level has a constant value (predetermined value) even though the substance of the drone DN is not detected in the monitoring area 8. An example of being detected as a ghost drone GDN1 under the influence of noise at that noise level is shown. Since the ghost drone GDN1 has not been subjected to suppression processing, for example, the virtual device erroneously detected by the monitoring device 10 based on two noise sources (specifically, sound sources located in the blue regions BD1 and BD4) is detected. A drone.

On the other hand, in FIG. 13, although the drone DN is detected and the PTZ camera CZ is operating, the process of step S5 shown in FIG. 10 (that is, the sound pressure level frequency characteristic suppression process) is executed. In this case, the presence of the already detected drone DN is not reflected in the sound pressure heat map, indicating that the drone DN has been missed (that is, lost). Therefore, the sound source direction detection unit 34 determines that the second condition is satisfied and the sound pressure level suppression processing when the surrounding noise level exceeds a certain value when the drone DN is detected and the PTZ camera CZ operates. Do not execute. As a result, the sound source direction detection unit 34 can suppress overlooking the drone DN entity through the suppression process, and thus can accurately detect the drone DN entity.

Next, details of the operation procedure of the detection determination process of the drone DN in step T16 of FIG. 9 will be described with reference to FIGS. FIG. 14 is a diagram illustrating an example of sequential scanning in the directivity direction performed in the monitoring area 8 when the drone DN is detected. FIG. 15 is a flowchart illustrating an example of an operation procedure of the drone DN detection determination process in step T <b> 16 of FIG. 9.

In the sound source detection unit UD, the directivity processing unit 63 sets, for example, the directivity range BF1 based on the sound source position estimated by the sound source direction detection unit 34 as the initial position of the directivity direction BF2 (S21). The initial position may not be limited to the directivity range BF1 based on the sound source position of the monitoring area 8 estimated by the sound source direction detection unit 34. That is, an arbitrary position designated by the user may be set as the initial position, and the monitoring area 8 may be sequentially scanned. Since the initial position is not limited, even when the sound source included in the directivity range BF1 based on the estimated sound source position is not the drone DN, a drone flying in another directivity direction can be detected at an early stage.

The directivity processing unit 63 determines whether or not the sound data signal collected by the microphone array MA and converted into a digital value by the A / D converters An1 to Aq is temporarily stored in the memory 38 (S22). ). When the sound data signal is not stored (S22, NO), the processing of the directivity processing unit 63 returns to step S21.

When the sound data signal collected by the microphone array MA is temporarily stored in the memory 38 (S22, YES), the directivity processing unit 63 moves in the arbitrary directivity direction BF2 in the directivity range BF1 of the monitoring area 8. On the other hand, beam forming is performed, and the sound data signal in the directivity direction BF2 is extracted (S23).

The frequency analysis unit 64 detects the frequency of the extracted sound data signal and its sound pressure level (S24).

The object detection unit 65 compares the detection sound pattern registered in the pattern memory of the memory 38 with the detection sound pattern obtained as a result of the frequency analysis process, and detects the drone DN (S25).

The detection result determination unit 66 notifies the output control unit 35 of the result of the comparison, and notifies the detection direction control unit 68 about the detection direction shift (S26).

For example, the object detection unit 65 compares the pattern of the detection sound obtained as a result of the frequency analysis process with the four frequencies f1, f2, f3, and f4 registered in the pattern memory of the memory 38. As a result of comparison, the object detection unit 65 has at least two of the same frequency in both detection sound patterns, and if the sound pressure level of these frequencies is greater than the first threshold th1, the detection sound patterns of both , And it is determined that the drone DN exists.

Here, it is assumed that at least two frequencies match, but the object detection unit 65 approximates when one frequency matches and the sound pressure level of this frequency is greater than the first threshold th1. You may determine that you are doing.

Further, the object detection unit 65 may set an error of an allowable frequency with respect to each frequency, and may determine the presence or absence of the approximation assuming that the frequencies within the error range are the same frequency.

Further, in addition to the comparison of the frequency and the sound pressure level, the object detection unit 65 may determine that the sound pressure level ratios of the sounds of the respective frequencies substantially match in addition to the determination condition. In this case, since the determination conditions become strict, the sound source detection unit UD can easily identify the detected drone DN as a pre-registered object, and the detection accuracy of the drone DN can be improved.

The detection result determination unit 66 determines whether or not the drone DN is present as a result of step S26 (S27).

When the drone DN exists (S27, YES), the detection result determination unit 66 notifies the output control unit 35 that the drone DN exists (that is, the detection result of the drone DN) (S28).

On the other hand, when the drone DN does not exist (S27, NO), the detection result determination unit 66 instructs the scanning control unit 67 to move the scanning direction BF2 in the monitoring area 8 to the next different direction. In response to the instruction from the detection result determination unit 66, the scanning control unit 67 moves the pointing direction BF2 to be scanned in the monitoring area 8 in the next different direction (S29). The notification of the detection result of the drone DN may be collectively performed after the omnidirectional scanning is completed, not at the timing when the detection processing of one directivity direction is completed.

The order in which the directing direction BF2 is sequentially moved in the monitoring area 8 is, for example, within the directivity range BF1 or the entire range of the monitoring area 8 so as to go from the outer circumference to the inner circumference, or the inner circle. A spiral (spiral) order may be used so as to go from the circumference to the outer circumference.

In addition, the detection direction control unit 68 does not scan the directivity direction continuously as in a single stroke, but sets a position in the monitoring area 8 in advance, and sets the directivity direction BF2 at each position in an arbitrary order. It may be moved. Thereby, the monitoring apparatus 10 can start the detection process from a position where the drone DN easily enters, for example, and can improve the efficiency of the detection process.

The scanning control unit 67 determines whether or not scanning in all directions in the monitoring area 8 has been completed (S30). When the omnidirectional scanning is not completed (S30, NO), the processing of the signal processing unit 33 returns to step S23, and the processing from steps S23 to S30 is repeated. That is, the directivity processing unit 63 performs beamforming in the directivity direction BF2 at the position moved in step S29, and extracts sound data in the directivity direction BF2. As a result, the sound source detection unit UD continues to detect other drones that may exist even if one drone DN is detected, so that a plurality of drones can be detected.

On the other hand, when the omnidirectional scanning is completed in step S30 (S30, YES), the directivity processing unit 63 erases the sound data collected by the microphone array MA temporarily stored in the memory 38 (S31). ). The directivity processing unit 63 may delete the sound data collected by the microphone array MA from the memory 38 and store it in the recorder RC.

After erasing the sound data, the signal processing unit 33 determines whether or not to end the drone DN detection process (S32). The detection process for the drone DN is terminated in response to a predetermined event. For example, the number of times that the drone DN has not been detected in step S26 is held in the memory 38, and when this number exceeds a predetermined number, the drone DN detection process may be terminated. Further, the signal processing unit 33 may end the detection process of the drone DN based on time-up by a timer or a user operation on a UI (User Interface) (not shown) of the operation unit 32. Moreover, you may complete | finish when the power supply of the monitoring apparatus 10 turns off.

In the process of step S24, the frequency analysis unit 64 analyzes the frequency and also measures the sound pressure at that frequency. The detection result determination unit 66 may determine that the drone DN is approaching the sound source detection unit UD when the sound pressure level measured by the frequency analysis unit 64 gradually increases with time. .

For example, if the sound pressure level of a predetermined frequency measured at a certain time is smaller than the sound pressure level of the same frequency measured at a time later than that time, the sound pressure increases with time, and the drone It may be determined that the DN is approaching. Further, the sound pressure level may be measured three times or more, and it may be determined that the drone DN is approaching based on the transition of statistical values (for example, variance value, average value, maximum value, minimum value, etc.).

In addition, when the measured sound pressure level is larger than a warning threshold that is a warning level, the detection result determination unit 66 may determine that the drone DN has entered the warning area. The warning threshold is a value larger than the first threshold th1 described above, for example. The warning area is, for example, the same area as the monitoring area 8 or an area included in the monitoring area 8 and narrower than the monitoring area 8. The alert area is an area where entry of the drone DN is restricted, for example. Moreover, the approach determination and the intrusion determination of the drone DN may be executed by the detection result determination unit 66.

As described above, in the flying object detection system 5 according to the first embodiment, the monitoring device 10 can adjust the optical axis direction and the sound of the PTZ camera CZ arranged so as to be able to image the monitoring area 8 or the sound of the monitoring area 8. The communication unit 31 communicates with the microphone array MA arranged so that sound can be collected. When the noise level included in the sound of the monitoring area 8 collected by the microphone array MA is equal to or higher than a predetermined value (predetermined value), the monitoring device 10 suppresses the sound parameter indicating the sound volume by a predetermined value QT. At the same time, based on the sound parameter (for example, sound pressure level) indicating the volume of sound collected by the microphone array MA or the sound parameter (for example, sound pressure level) subjected to suppression processing, the drone DN in the monitoring area 8 It has a signal processing unit 33 that detects presence or absence. In addition, when the PTZ camera CZ adjusts the optical axis direction in the detection direction in which the drone DN is detected, the monitoring apparatus 10 omits the execution of the sound parameter suppression process.

As a result, the monitoring apparatus 10 suppresses (that is, lowers) the frequency characteristic of the sound pressure level by a predetermined value QT in a state where a noise level higher than a certain value is generated. A frequency characteristic (see characteristic Cv2) in which the peak value of the sound pressure level is less than the first threshold is obtained, and erroneous detection of the presence of the ghost drone described above can be effectively suppressed. In addition, the monitoring device 10 detects the presence or absence of the drone DN using sound data collected under a situation where the suppression process is not necessary or sound data collected under a situation where the suppression process is necessary. As a result, it is possible to accurately detect the presence or absence of the drone DN. In addition, when the PTZ camera CZ adjusts the optical axis direction and the generated sound is generated with a noise level higher than a certain value (predetermined value), the monitoring apparatus 10 determines the predetermined value QT of the frequency characteristic of the sound pressure level. Therefore, the drone DN can be detected accurately without missing the substance of the drone DN. Therefore, according to the monitoring device 10, the erroneous detection of the drone DN can be accurately reduced according to the environment of the monitoring area 8, and the deterioration of the recognition accuracy of the drone DN can be suppressed.

In addition, when the drone DN is not detected, the monitoring device 10 executes a sound parameter (for example, sound pressure level) suppression process in the signal processing unit 33. As a result, the monitoring device 10 can obtain a frequency characteristic (see characteristic Cv2) in which the peak value of the sound pressure level is less than the first threshold value by the suppression process, so that even if the noise level is equal to or higher than a predetermined value (a constant value). The erroneous detection that the ghost drone mentioned above exists can be suppressed effectively.

Further, when the PTZ camera CZ does not adjust the optical axis direction, the monitoring device 10 executes a sound parameter (for example, sound pressure level) suppression process in the signal processing unit 33. As a result, the monitoring device 10 can obtain a frequency characteristic (see characteristic Cv2) in which the peak value of the sound pressure level is less than the first threshold value by the suppression process, so that even if the noise level is equal to or higher than a predetermined value (a constant value). The erroneous detection that the ghost drone mentioned above exists can be suppressed effectively.

In addition, the monitoring device 10 communicates with the omnidirectional camera CA arranged so that the monitoring area 8 can be imaged in the communication unit 31. When the drone DN is detected, the monitoring device 10 uses a first visual image (see, for example, FIG. 12) indicating the position of the drone DN in the signal processing unit 33 in the first monitoring area 8 captured by the omnidirectional camera CA. The image is superimposed on the captured image (for example, the omnidirectional captured image IMG1) and displayed on the monitor MN. Thereby, the user can visually grasp the position of the drone DN detected by the monitoring device 10 on the omnidirectional captured image IMG1 displayed on the monitor MN, and can easily specify the position of the drone DN specifically. .

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is obvious for those skilled in the art that various modifications, modifications, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure. In addition, the constituent elements in the various embodiments described above may be arbitrarily combined without departing from the spirit of the invention.

Note that this application is based on a Japanese patent application filed on May 24, 2018 (Japanese Patent Application No. 2018-099869), the contents of which are incorporated herein by reference.

The present disclosure is useful as a flying object detection device, a flying object detection method, and a flying object detection system that accurately reduce false detection of an unmanned air vehicle according to the environment of a monitoring area and suppress deterioration in recognition accuracy of the unmanned air vehicle. It is.

5 flying object detection system 10 monitoring device 31 communication unit 32 operation unit 33 signal processing unit 34 sound source direction detection unit 35 output control unit 37 SPK
38 memory 39 setting management unit 63 directivity processing unit 64 frequency analysis unit 65 object detection unit 66 detection result determination unit 67 scanning control unit 68 detection direction control unit CA omnidirectional camera CZ PTZ camera MA microphone array MN monitor NW network RC recorder UD sound source detection unit

Claims

A communication unit that communicates with a first camera that can adjust the optical axis direction and is arranged so as to be able to capture an image of the monitoring area, or a microphone array that is arranged so as to be able to pick up sound of the monitoring area;
When the noise level included in the sound of the monitoring area collected by the microphone array is equal to or higher than a predetermined value, the sound parameter indicating the loudness is suppressed by a predetermined value, and the sound is collected by the microphone array. A processor for detecting the presence or absence of an unmanned air vehicle in the monitoring area based on the sound parameter indicating the volume of the sound or the sound parameter subjected to the suppression process,
The processor is
When the first camera adjusts the optical axis direction in the detection direction in which the unmanned air vehicle is detected, the execution of the sound parameter suppression process is omitted.
Flying object detection device.
The processor is
When the unmanned air vehicle is not detected, the sound parameter suppression process is executed.
The flying object detection device according to claim 1.
The processor is
When the first camera does not adjust the optical axis direction, the sound parameter suppression processing is executed.
The flying object detection device according to claim 1.
The communication unit is
Communicating with the second camera arranged to be able to image the monitoring area,
The processor is
When the unmanned aerial vehicle is detected, a first visual image indicating the position of the unmanned aerial vehicle is superimposed on a first captured image of the monitoring area captured by the second camera and displayed on a monitor.
The flying object detection device according to any one of claims 1 to 3.
A flying object detection method in a flying object detection device connected to be communicable with a first camera capable of adjusting an optical axis direction or a microphone array,
When a noise level included in the sound of the monitoring area collected by the microphone array is equal to or higher than a predetermined value, a step of suppressing a sound parameter indicating the volume of the sound by a predetermined value;
Detecting the presence or absence of an unmanned air vehicle in the monitoring area based on a sound parameter indicating the volume of the sound collected by the microphone array or the sound parameter subjected to the suppression process,
In the step of detecting the presence or absence of the unmanned air vehicle,
When the first camera adjusts the optical axis direction in the detection direction in which the unmanned air vehicle is detected, execution of the sound parameter suppression processing is omitted, and the volume of the sound collected by the microphone array Detecting the presence or absence of the unmanned air vehicle based on a sound parameter indicating
Flying object detection method.
A first camera having an adjustable optical axis direction and capable of capturing an image of a monitoring area; a microphone array disposed to be able to pick up sound of the monitoring area; and the first camera or the microphone array. A flying object detection device that communicates between,
The flying object detection device is
When the noise level included in the sound of the monitoring area collected by the microphone array is equal to or higher than a predetermined value, the sound parameter indicating the loudness is suppressed by a predetermined value,
Based on the sound parameter indicating the volume of the sound collected by the microphone array or the suppressed sound parameter, the presence or absence of an unmanned air vehicle in the monitoring area is detected.
When the first camera adjusts the optical axis direction in the detection direction in which the unmanned air vehicle is detected, the execution of the sound parameter suppression process is omitted.
Flying object detection system.