JP2021097416A - Image processing apparatus, monitoring system, and image processing method - Google Patents

Abstract

To visually present to a user that, in what kind of place in an imaging area of a camera device, an unmanned flying body is detected and what kind of sound source exists in what other places in the same imaging area, without deteriorating visibility of the captured image of the camera device.SOLUTION: A sound source direction detection unit 34, on the basis of sound collected by a microphone array MAk, derives sound pressure that specifies magnitude of sound in an imaging area for each predetermined unit of pixels constituting a captured image in the imaging area. An output control section 35, in accordance with comparison between the sound pressure and a plurality of threshold values relating to the magnitude of sound, superimposes sound source visual information obtained by gradually converting the sound pressure into different visual images for each predetermined unit of the pixels constituting the captured image in the imaging area and displays them in a first monitor MN1. The sound source direction detection section 34, when any one of the sound source positions in the captured image in the imaging area is specified, derives the sound pressure for each value obtained by dividing a predetermined unit of pixels constituting a rectangular range including the sound source position by size ratio between the rectangular range and the captured image in the imaging area.SELECTED DRAWING: Figure 16

Description

The present invention relates to, for example, a monitoring system and a monitoring method for monitoring an imaging area of a camera device in which an unmanned vehicle is flying.

Conventionally, there is known a flying object monitoring device capable of detecting the existence of an object and the direction of arrival of the object by using a plurality of sound detection units that detect the sound generated in the monitoring area for each direction. (See, for example, Patent Document 1). When the processing device of this flying object monitoring device detects the flying object and its flying direction by sound detection by an omnidirectional microphone, the monitoring camera is pointed in the direction in which the flying object has flown. Further, the processing device displays the image captured by the surveillance camera on the display device.

Japanese Unexamined Patent Publication No. 2006-168421

In Patent Document 1, a surveillance camera capable of changing the shooting direction in an arbitrary direction within the surveillance area is provided, and when a flying object such as a helicopter or a Cessna is detected, the shooting direction of the surveillance camera is changed. It is disclosed to let you. In other words, it is disclosed to change the shooting direction of the surveillance camera in order to focus on and image the detected flying object.

However, Patent Document 1 does not consider displaying a wide range of captured images of the surroundings including an unmanned flying object detected within the angle of view of the camera device with respect to the imaging area. Therefore, the user visually presents what kind of sound source exists in what kind of place in the image pickup area of the camera device when the unmanned vehicle is detected and in what other place in the same image pickup area. There was a problem that it could not be done.

The present invention was devised in view of the above-mentioned conventional situation, in which an unmanned vehicle is detected in any place in the image pickup area of the camera device, and what kind of sound source is detected in any other place in the same image pickup area. It is an object of the present invention to provide a monitoring system and a monitoring method for visually presenting the existence to a user without deteriorating the visibility of a captured image of a camera device.

The present invention collects sound by a camera that captures an imaging area, a microphone array that collects sound in the imaging area, a monitor that displays an captured image of the imaging area captured by the camera, and the microphone array. A sound parameter derivation unit that derives a sound parameter that specifies the loudness of the sound in the imaging area for each predetermined unit of pixels constituting the image captured in the imaging area based on the sound, and a sound parameter derivation unit. According to the comparison between the derived sound parameter and the threshold for sound volume, the sound source visual image obtained by converting the sound parameter into a visual image for each predetermined unit of the pixel is the size of the captured image in the imaging area. A signal processing unit that generates a semi-transparent map of sound parameter maps connected so as to correspond to the above is provided, and the signal processing unit superimposes the semi-transparent map on an image captured in the imaging area and mounts the monitor on the monitor. Provide a monitoring system to display.

Further, the present invention is a monitoring method in a monitoring system having a camera and a microphone array, in which an image pickup area is imaged by the camera, sound of the image pickup area is picked up by the microphone array, and the sound of the image pickup area is picked up by the camera. The captured image of the imaging area is displayed on the monitor, and the sound parameter for specifying the loudness of the sound of the imaging area is set based on the sound picked up by the microphone array. A sound source visual that is derived for each predetermined unit of the constituent pixels and the sound parameter for each predetermined unit of the pixel is converted into a visual image according to the comparison between the derived sound parameter and the threshold value for the loudness of the sound. A semi-transparent map of a sound parameter map in which images are connected so as to correspond to the size of the captured image in the imaging area is generated, and the generated translucent map is superimposed on the captured image in the imaging area to monitor the monitor. Provides a monitoring method to display in.

According to the present invention, regardless of the loudness of the sound at the sound source position detected in the imaging area of the camera device, the loudness of the sound at the sound source position can be presented in detail and stepwise, and the sound at the sound source position can be presented. It can contribute to an accurate grasp of the size of the user.

The figure which shows an example of the schematic structure of the unmanned vehicle detection system of each embodiment. Diagram showing an example of the appearance of the sound source detection unit Block diagram showing in detail an example of the internal configuration of a microphone array A block diagram showing in detail an example of the internal configuration of an omnidirectional camera Block diagram showing in detail an example of the internal configuration of a PTZ camera Block diagram showing in detail an example of the internal configuration of the monitoring device Timing chart showing an example of the detection sound signal pattern of an unmanned aircraft registered in the memory Timing chart showing an example of frequency change of the detection sound signal obtained as a result of frequency analysis processing A sequence diagram illustrating an example of an operation procedure for detecting an unmanned flying object and displaying the detection result in the first embodiment. A flowchart illustrating a detailed example of the operation procedure of the unmanned vehicle detection determination in step S8 of FIG. A diagram showing an example of how an unmanned aircraft is detected by scanning the directivity directions in order in the monitoring area. The figure which shows the display screen example of the 1st monitor when the masking area is not set. Explanatory diagram showing a display example of the masking area in chronological order during automatic learning processing A sequence diagram illustrating an example of an operation procedure for setting a masking area in the first embodiment. The figure which shows the display screen example of the 1st monitor when a masking area is set Schematic diagram of the dynamic change of the display resolution of the sound pressure heat map in the second embodiment. A flowchart illustrating an example of an operation procedure for dynamically changing the display resolution of the sound pressure heat map in the second embodiment. Schematic explanatory view of the width adjustment of the threshold value according to the frequency distribution of the sound pressure value and the display result of the captured image accompanying the width adjustment in the second embodiment. (A), (B) Schematic explanatory view of changing the setting of the width between the threshold values that defines the use of the sound source visual image in the second embodiment. Schematic explanatory view of the display of the captured image due to the change of the width setting between the threshold values defining the use of the crimson image and the ultramarine blue image in the second embodiment. A flowchart illustrating an example of an operation procedure for changing the setting of the width between threshold values in the second embodiment. Schematic diagram of overlay display of omnidirectional image and translucent sound pressure heat map in the third embodiment A diagram showing an example of a display screen of the first monitor on which an omnidirectional image and a translucent sound pressure heat map are overlaid. A sequence diagram illustrating an example of an operation procedure of overlay display of an omnidirectional image and a translucent sound pressure heat map in a third embodiment.

Hereinafter, each embodiment in which the monitoring system and the monitoring method according to the present invention are specifically disclosed will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art. It should be noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

Hereinafter, as an example of the monitoring system according to the present invention, an unmanned aerial vehicle detection system for detecting an unmanned aerial vehicle (for example, a drone, a radio-controlled helicopter) to be monitored will be described as an example. Further, the present invention is not limited to the monitoring system, and can be defined as a monitoring method executed in the monitoring system.

Hereinafter, the user of the unmanned aircraft detection system (for example, an observer who looks around the monitoring area and guards) is simply referred to as a "user".

FIG. 1 is a diagram showing an example of a schematic configuration of the unmanned vehicle detection system 5 of each embodiment. The unmanned vehicle detection system 5 detects an unmanned vehicle dn (see, for example, FIG. 12) intended by the user as a detection target. The unmanned aerial vehicle dn is, for example, a drone that flies autonomously using a GPS (Global Positioning System) function, a radio-controlled helicopter that flies by radio control by a third party, and the like. Such an unmanned air vehicle dn is used, for example, for aerial photography of a target, surveillance, transportation of supplies, and the like.

In each embodiment, as the unmanned aerial vehicle dn, a multicopter type drone equipped with a plurality of rotors (in other words, rotary wings) is exemplified. In a multicopter type drone, generally, when the number of rotor blades is two, harmonics having a frequency twice as high as that of a specific frequency and further harmonics having a frequency obtained by multiplying the specific frequency are generated. Similarly, when the number of blades of the rotor is three, harmonics having a frequency three times higher than that of a specific frequency, and further harmonics having a frequency obtained by multiplying the specific frequency are generated. The same applies when the number of rotor blades is 4 or more.

The unmanned vehicle detection system 5 includes a plurality of sound source detection units UD1, ..., UDk, ..., UDn, a monitoring device 10, a first monitor MN1, a second monitor MN2, and a recorder RC. n is a natural number of 2 or more. The plurality of sound source detection units UD1, ..., UDk, ..., UDn are interconnected with the monitoring device 10 via the network NW. k is a natural number from 1 to n. Each sound source detection unit (for example, sound source detection unit UD1) has a configuration including a microphone array MA1, an omnidirectional camera CA1, and a PTZ camera CZ1, and other sound source detection units UDk also have the same configuration.

Unless it is necessary to distinguish each sound source detection unit, it is referred to as a sound source detection unit UDk or simply a sound source detection unit UD. Similarly, unless it is necessary to distinguish between individual microphone arrays, omnidirectional cameras, and PTZ cameras, microphone array MAk or microphone array MA (see FIG. 2), omnidirectional camera CAk or omnidirectional camera CA (FIG. 2). (See), PTZ camera CZk or PTZ camera CZ (see FIG. 2).

In the sound source detection unit UDk, the microphone array MAk collects omnidirectional sound in the monitored area (that is, the monitoring area) in which the own device is installed in an omnidirectional state. The microphone array MAk has a housing 15 (see FIG. 2) having a cylindrical opening having a predetermined width formed in the center. Sounds to be picked up by the microphone array MAk include, for example, mechanical operating sounds such as drones, sounds emitted by humans, and other sounds, and are limited to sounds in the audible frequency range (that is, 20 Hz to 23 kHz). However, low-frequency sounds lower than the audible frequency and ultrasonic sounds exceeding the audible frequency may be included.

The microphone array MAk includes a plurality of omnidirectional microphones M1 to Mq (see FIG. 3). q is a natural number of 2 or more. The microphones M1 to Mq are arranged concentrically at predetermined intervals (for example, uniform intervals) around the opening provided in the housing 15 along the circumferential direction. As the microphones M1 to Mq, for example, an electret condenser microphone (ECM: Electret Condenser Microphone) is used. The microphone array MAk transmits the voice data of the sound obtained by collecting the sounds of the microphones M1 to Mq to the monitoring device 10 via the network NW. The arrangement of the microphones M1 to Mq described above is an example and may be another arrangement (for example, a square arrangement or a rectangular arrangement), but the microphones M1 to Mq are arranged side by side at equal intervals. It is preferable to be done.

Further, the microphone array MAk has a plurality of microphones M1 to Mq (for example, q = 32) and a plurality of amplifiers (amplifiers) PA1 to PAq (see FIG. 3) for amplifying output signals of the plurality of microphones M1 to Mq. .. The analog signals output from each amplifier are converted into digital signals by the A / D converters A1 to Aq (see FIG. 3) described later. The number of microphones in the microphone array MAk is not limited to 32, and may be another number (for example, 16, 64, 128).

An omnidirectional camera CAk that substantially matches the volume of the opening is housed inside the opening formed in the center of the housing 15 (see FIG. 2) of the microphone array MAk. That is, the microphone array MAk and the omnidirectional camera CAk are integrally arranged so that the centers of the respective housings are in the coaxial direction (see FIG. 2). The omnidirectional camera CAk is a camera equipped with a fisheye lens 45a (see FIG. 4) capable of capturing an omnidirectional image of a monitoring area (see above) as an imaging area of the omnidirectional camera CAk. In each embodiment, the sound collecting area of the microphone array MAk and the imaging area of the omnidirectional camera CAk are described as a common monitoring area, but the spatial size (for example, volume) of the sound collecting area and the imaging area is the same. It does not have to be. For example, the volume of the sound collecting area may be larger or smaller than the volume of the imaging area. In short, it suffices if there is a common space between the sound collection area and the imaging area. The omnidirectional camera CAk functions as a surveillance camera capable of capturing an imaging area in which a sound source detection unit UDk is installed, for example. That is, the omnidirectional camera CAk has an angle of view of, for example, 180 ° in the vertical direction and 360 ° in the horizontal direction, and images a monitoring area 8 (see FIG. 11), which is a hemisphere, as an imaging area.

In each sound source detection unit UDk, the omnidirectional camera CAk is fitted inside the opening of the housing 15, so that the omnidirectional camera CA and the microphone array MA are arranged coaxially. In this way, by aligning the optical axis of the omnidirectional camera CAk with the central axis of the housing of the microphone array MAk, the imaging area and the sound collecting area in the axial circumferential direction (that is, the horizontal direction) become substantially the same. The position of the subject in the omnidirectional image captured by the omnidirectional camera CAk (in other words, the direction indicating the position of the subject as seen from the omnidirectional camera CAk) and the position of the sound source to be picked up by the microphone array MAk (in other words, in other words). The direction indicating the position of the sound source as seen from the microphone array MAk) can be expressed in the same coordinate system (for example, the coordinates indicated by (horizontal angle, vertical angle)). Each sound source detection unit UDk is attached so that, for example, the upward direction in the vertical direction is the sound collecting surface and the imaging surface in order to detect the unmanned flying object dn flying in the sky (see FIG. 2). ..

The monitoring device 10 is configured by using, for example, a PC (Personal Computer) or a server. The monitoring device 10 forms directivity (that is, beamforming) with an arbitrary direction as the main beam direction based on the user's operation with respect to the omnidirectional sound picked up by the microphone array MAk, and the directivity is formed. The sound of the direction can be emphasized. A technique relating to directivity control processing of sound data for beamforming the sound picked up by the microphone array MAk is a known technique, for example, as shown in Reference Patent Documents 1 and 2.

(Reference Patent Document 1) Japanese Patent Application Laid-Open No. 2014-143678 (Reference Patent Document 2) Japanese Patent Application Laid-Open No. 2015-029241

The monitoring device 10 processes the captured image to generate an omnidirectional image by using the image captured by the omnidirectional camera CAk (hereinafter, may be abbreviated as “captured image”). The omnidirectional image may be generated by the omnidirectional camera CAk instead of the monitoring device 10.

The monitoring device 10 uses a sound pressure heat map based on a calculated value of a sound parameter (for example, a sound pressure described later) that specifies the loudness of the sound picked up by the microphone array MAk in the captured image captured by the omnidirectional camera CAk. (See FIG. 15) is superimposed and output to the first monitor MN1 or the like for display.

Further, the monitoring device 10 provides an omnidirectional image IMG1 with a visual image (for example, an identification mark (not shown)) that makes it easy for the user to visually distinguish the detected unmanned vehicle dn from the unmanned vehicle dn of the first monitor MN1. It may be displayed at the position of. The visual image means, for example, in the omnidirectional image IMG1, when the user looks at the omnidirectional image IMG1, the information is represented to such an extent that it can be clearly distinguished from other subjects, and the same applies hereinafter. ..

The first monitor MN1 displays the omnidirectional image IMG1 captured by the omnidirectional camera CAk. The second monitor MN2 displays the omnidirectional image IMG1 captured by the omnidirectional camera CAk. Further, the first monitor MN1 generates and displays a composite image in which the above-mentioned identification mark (not shown) is superimposed on the omnidirectional image IMG1. In FIG. 1, the first monitor MN1 and the second monitor MN2 and two monitors are connected to the monitoring device 10, but for example, only the first monitor MN1 may be connected. Further, the first monitor MN1 or the second monitor MN2, or the first monitor MN1 and the second monitor MN2 may be configured as an integral device with the monitoring device 10.

The recorder RC is configured by using a semiconductor memory such as a hard disk drive or a flash memory, and is transmitted from various image data (see below) generated by the monitoring device 10 and from each sound source detection unit UDk. Saves various omnidirectional image and audio data. The recorder RC may be configured as a device integrated with the monitoring device 10, or may be omitted from the configuration of the unmanned flying object detection system 5.

In FIG. 1, a plurality of sound source detection units UDk and a monitoring device 10 have a communication interface and are connected to each other via a network NW so as to be capable of data communication. The network NW may be a wired network (for example, an intranet, the Internet, a wired LAN (Local Area Network)), or a wireless network (for example, a wireless LAN). The sound source detection unit UDk and the monitoring device 10 are connected via the network NW. The monitoring device 10, the first monitor MN1, the second monitor MN2, and the recorder RC are all installed in the monitoring room RM where the user is resident at the time of monitoring.

FIG. 2 is a diagram showing an example of the appearance of the sound source detection unit UD. The sound source detection unit UD includes the microphone array MA, the omnidirectional camera CA, and the PTZ camera CZ described above, as well as a support base 70 that mechanically supports them. The support base 70 includes a tripod 71, two rails 72 fixed to the top plate 71a of the tripod 71, and a first mounting plate 73 and a second mounting plate 74 attached to both ends of the two rails 72, respectively. Has a structure in combination with.

The first mounting plate 73 and the second mounting plate 74 are mounted so as to straddle the two rails 72, and have substantially the same plane surface. Further, the first mounting plate 73 and the second mounting plate 74 are slidable on the two rails 72, and are adjusted and fixed at positions separated or close to each other.

The first mounting plate 73 is a disk-shaped plate material. An opening 73a is formed in the center of the first mounting plate 73. The housing 15 of the microphone array MA is housed and fixed in the opening 73a. On the other hand, the second mounting plate 74 is a substantially rectangular plate material. An opening 74a is formed in a portion near the outside of the second mounting plate 74. The PTZ camera CZ is accommodated and fixed in the opening 74a.

As shown in FIG. 2, the optical axis L1 of the omnidirectional camera CA housed in the housing 15 of the microphone array MA and the optical axis L2 of the PTZ camera CZ mounted on the second mounting plate 74 are in the initial installation state. Is set to be parallel to each other.

The tripod 71 is supported by the ground contact surface with three legs 71b, and the position of the top plate 71a can be moved in the direction perpendicular to the ground contact surface by manual operation, and the top plate 71a can be moved in the pan direction and the tilt direction. The orientation of the plate 71a can be adjusted. Thereby, the sound collecting area of the microphone array MA (in other words, the imaging area of the omnidirectional camera CA or the monitoring area of the unmanned vehicle detection system 5) can be set in any direction.

FIG. 3 is a block diagram showing in detail an example of the internal configuration of the microphone array MAk. The microphone array MAk shown in FIG. 3 includes a plurality of microphones M1 to Mq (for example, q = 32), a plurality of amplifiers (amplifiers) PA1 to PAq for amplifying output signals of the plurality of microphones M1 to Mq, and each amplifier PA1 to PAq. The configuration includes a plurality of A / D converters A1 to Aq, a compression processing unit 25, and a transmission unit 26, which convert analog signals output from the above into digital signals, respectively.

The compression processing unit 25 generates a packet of voice data based on the digital voice signals output from the A / D converters A1 to An. The transmission unit 26 transmits the voice data packet generated by the compression processing unit 25 to the monitoring device 10 via the network NW.

In this way, the microphone array MAk amplifies the output signals of the microphones M1 to Mq by the amplifiers PA1 to PAq and converts them into digital audio signals by the A / D converters A1 to Aq. After that, the microphone array MAk generates a voice data packet in the compression processing unit 25, and transmits the voice data packet to the monitoring device 10 via the network NW.

FIG. 4 is a block diagram showing in detail an example of the internal configuration of the omnidirectional camera CAk. The omnidirectional camera CAk shown in FIG. 4 has a configuration including a CPU 41, a communication unit 42, a power supply management unit 44, an image sensor 45, a fisheye lens 45a, a memory 46, and a network connector 47.

The CPU 41 performs signal processing for controlling the operation of each part of the omnidirectional camera CAk, data input / output processing with and from other parts, data calculation processing, and data storage processing. Instead of the CPU 41, a processor such as an MPU (Micro Processing Unit) or a DSP (Digital Signal Processor) may be provided.

For example, the CPU 41 generates two-dimensional panoramic image data (that is, two-dimensional panoramic converted image data) obtained by cutting out an image in a specific range (direction) from the omnidirectional image data according to a user's designation to operate the monitoring device 10. And save it in the memory 46.

The image sensor 45 is configured by using, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor, and captures an optical image of subject light from an imaging area focused by a fisheye lens 45a on a light receiving surface. By processing, omnidirectional image data of the imaging area is acquired.

The fisheye lens 45a incidents and collects subject light from all directions in the imaging area, and forms an optical image of the subject light on a light receiving surface (not shown) of the image sensor 45.

The memory 46 stores the ROM 46z in which the data of the program and the set value for defining the operation of the omnidirectional camera CAk is stored, and the omnidirectional image data or the cut-out image data or work data obtained by cutting out a part of the omnidirectional image data. It has a RAM 46y to be used, and a memory card 46x that is freely inserted and removed from the omnidirectional camera CAk and stores various data.

The communication unit 42 is a network interface (I / F) that controls data communication with the network NW connected via the network connector 47.

The power management unit 44 supplies DC power to each unit of the omnidirectional camera CAk. Further, the power management unit 44 may supply DC power to a device connected to the network NW via the network connector 47.

The network connector 47 is a connector capable of transmitting omnidirectional image data or two-dimensional panoramic image data to the monitoring device 10 via the network NW and supplying power via the network cable.

FIG. 5 is a block diagram showing in detail an example of the internal configuration of the PTZ camera CZk. The description of each part similar to the omnidirectional camera CAk will be omitted by assigning a reference numeral corresponding to each part in FIG. The PTZ camera CZk is a camera whose optical axis direction (sometimes referred to as an imaging direction) can be adjusted by an instruction to change the angle of view from the monitoring device 10.

Like the omnidirectional camera CAk, the PTZ camera CZk has a CPU 51, a communication unit 52, a power supply management unit 54, an image sensor 55, an image pickup lens 55a, a memory 56, and a network connector 57, as well as an image pickup direction control unit 58 and a lens drive motor. Has 59. When the monitoring device 10 receives an angle-of-view change instruction, the CPU 51 notifies the imaging direction control unit 58 of the angle-of-view change instruction.

The image pickup direction control unit 58 controls at least one of the pan direction and the tilt direction for the image pickup direction of the PTZ camera CZ according to the angle of view change instruction notified from the CPU 51, and further changes the zoom magnification as necessary. The control signal for this is output to the lens drive motor 59. The lens drive motor 59 drives the image pickup lens 55a according to this control signal, changes the image pickup direction (direction of the optical axis L2 shown in FIG. 2), and adjusts the focal length of the image pickup lens 55a to increase the zoom magnification. change.

The imaging lens 55a is configured by using one or more lenses. In the image pickup lens 55a, the optical axis directions of the pan rotation and the tilt rotation are changed by driving the lens drive motor 59 in response to the control signal from the image pickup direction control unit 58.

FIG. 6 is a block diagram showing in detail an example of the internal configuration of the monitoring device 10. The monitoring device 10 shown in FIG. 6 has a configuration including at least a communication unit 31, an operation unit 32, a signal processing unit 33, a speaker device (SPK) 37, a memory 38, and a setting management unit 39.

The communication unit 31 receives the omnidirectional image data or the two-dimensional panoramic image data transmitted by the omnidirectional camera CAk and the audio data transmitted by the microphone array MAk and outputs them to the signal processing unit 33.

The operation unit 32 is a user interface (UI: User Interface) for notifying the signal processing unit 33 of the content of the user's input operation, and is composed of a pointing device such as a mouse or a keyboard, for example. Further, the operation unit 32 is arranged, for example, corresponding to each screen of the first monitor MN1 and the second monitor MN2, and is configured by using a touch panel or a touch pad that can be directly input with a user's finger or a stylus pen. May be good.

The operation unit 32 is a red region of a sound pressure heat map (see FIG. 15) displayed superimposed on an image (omnidirectional image IMG1) captured by any of the omnidirectional cameras CAk on the first monitor MN1 and the second monitor MN2. When the RD1 is designated by the user, the coordinate data indicating the designated position is acquired and output to the signal processing unit 33. The signal processing unit 33 reads the sound data collected by the microphone array MAk corresponding to the omnidirectional camera CAk from the memory 38, and directs the sound data from the microphone array MAk in the direction toward the actual sound source position corresponding to the designated position. It is output from the speaker device 37 after forming the property. As a result, the user can clearly confirm not only the unmanned flying object dn but also the sound emphasized at the position specified by the user himself / herself on the captured image (omnidirectional image IMG1).

The signal processing unit 33 is configured by using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor), and is a control process for controlling the operation of each unit of the monitoring device 10. , Performs data input / output processing with other parts, data calculation (calculation) processing, and data storage processing. The signal processing unit 33 includes a sound source direction detection unit 34, an output control unit 35, a directivity processing unit 63, a frequency analysis unit 64, an object detection unit 65, a detection result determination unit 66, a scanning control unit 67, and a detection direction control unit 68. , Masking area setting unit 69a and threshold adjustment unit 69b are included. Further, the monitoring device 10 is connected to the first monitor MN1 and the second monitor MN2.

The sound source direction detection unit 34 uses, for example, the sound data of the sound of the monitoring area 8 picked up by the microphone array MAk according to a known whitening cross-correlation method (CSP (Cross-power Spectrum Phase analysis) method) to position the sound source. To estimate. In the CSP method, the sound source direction detection unit 34 divides the monitoring area 8 shown in FIG. 11 into a plurality of blocks, and when sound is picked up by the microphone array MAk, the sound pressure or volume threshold value is exceeded for each block. By determining whether or not there is sound, the position of the sound source in the monitoring area 8 can be roughly estimated.

Further, the sound source direction detection unit 34 as a sound parameter derivation unit is an omnidirectional image of the monitoring area 8 based on the omnidirectional image data captured by the omnidirectional camera CAk and the sound data collected by the microphone array MAk. The sound pressure as a sound parameter is calculated for each pixel constituting the data. The sound source direction detection unit 34 outputs the calculated value, which is the calculation result of the sound pressure, to the output control unit 35. This sound pressure calculation process is a known technique, and detailed description of the process is omitted.

The setting management unit 39 has a coordinate conversion formula related to coordinate conversion of a position designated by the user with respect to the screen of the first monitor MN1 on which the omnidirectional image data captured by the omnidirectional camera CAk is displayed. This coordinate conversion formula is specified by the user on the omnidirectional image data, for example, based on the physical distance difference between the installation position of the omnidirectional camera CAk (see FIG. 2) and the installation position of the PTZ camera CZk (see FIG. 2). This is a mathematical formula for converting the coordinates of the position (that is, (horizontal angle, vertical angle)) into the coordinates in the direction seen from the PTZ camera CZk.

Using the coordinate conversion formula held by the setting management unit 39, the signal processing unit 33 uses the position specified by the user from the installation position of the PTZ camera CZk with reference to the installation position of the PTZ camera CZk (see FIG. 2). Coordinates (θMAh, θMAv) indicating the direction of direction toward the actual sound source position corresponding to are calculated. θMAh is a horizontal angle in the direction toward the actual sound source position corresponding to the position specified by the user when viewed from the installation position of the PTZ camera CZk. θMAv is a vertical angle in the direction toward the actual sound source position corresponding to the position specified by the user when viewed from the installation position of the PTZ camera CZk. As shown in FIG. 2, since the distance between the omnidirectional camera CAk and the PTZ camera CZk is known and the respective optical axes L1 and L2 are parallel, the calculation process of the above coordinate conversion formula is known, for example. It can be realized by geometric calculation. The sound source position is an actual sound source position corresponding to the position designated by the operation unit 32 by the operation of the user's finger or the stylus pen with respect to the video data displayed on the first monitor MN1 and the second monitor MN2.

As shown in FIG. 2, the omnidirectional camera CAk and the microphone array MAk are arranged so that the optical axis direction of the omnidirectional camera CAk and the central axis of the housing of the microphone array MAk are coaxial with each other in each embodiment. Has been done. Therefore, the coordinates of the user-designated position derived by the omnidirectional camera CAk according to the user's designation for the first monitor MN1 on which the omnidirectional image data is displayed are the sound enhancement direction (also the directional direction) as seen from the microphone array MAk. Can be regarded as the same as). In other words, when the monitoring device 10 specifies the user for the first monitor MN1 (which may be the second monitor MN2) on which the omnidirectional image data is displayed, the monitoring device 10 performs all the coordinates of the specified position on the omnidirectional image data. It is transmitted to the directional camera CAk. As a result, the omnidirectional camera CAk uses the coordinates of the designated position transmitted from the monitoring device 10 to indicate the direction of the sound source position corresponding to the user-designated position as seen from the omnidirectional camera CAk (horizontal angle, vertical). Angle) is calculated. Since the calculation process in the omnidirectional camera CAk is a known technique, the description thereof is omitted. The omnidirectional camera CAk transmits the calculation result of the coordinates indicating the direction of the sound source position to the monitoring device 10. The monitoring device 10 can use the coordinates (horizontal angle, vertical angle) calculated by the omnidirectional camera CAk as the coordinates (horizontal angle, vertical angle) indicating the direction of the sound source position as seen from the microphone array MAk.

However, when the omnidirectional camera CAk and the microphone array MAk are not arranged coaxially, the setting management unit 39 derives the omnidirectional camera CAk according to the method described in, for example, Japanese Patent Application Laid-Open No. 2015-209241. It is necessary to convert the coordinates to the coordinates in the direction viewed from the microphone array MAk.

Further, the setting management unit 39 has a first threshold value th1 and a second threshold value th2 and which are compared with the sound pressure p for each pixel constituting the omnidirectional image data or the two-dimensional panoramic image data calculated by the sound source direction detection unit 34. The third threshold th3 (see, for example, FIG. 9) is held. Here, the sound pressure p is used as an example of a sound parameter related to a sound source, represents the loudness of the sound picked up by the microphone array MA, and represents the loudness of the sound output from the speaker device 37. It is distinguished from the volume. The first threshold value th1, the second threshold value th2, and the third threshold value th3 are values to be compared with the sound pressure of the sound generated in the monitoring area 8, and are, for example, predetermined values for determining the sound emitted by the unmanned flying object dn. Is set to. Further, a plurality of threshold values can be set in addition to the above-mentioned first threshold value th1, second threshold value th2, and third threshold value th3, and here, for the sake of brief explanation, for example, the first threshold value th1 and a value larger than this. The second threshold value th2 is set, and the third threshold value th3, which is a larger value, is set (first threshold value th1 <second threshold value th2 <third threshold value th3).

Further, as will be described later, in the sound pressure heat map generated by the output control unit 35, the red region RD1 (see FIG. 15) of the pixel in which the sound pressure larger than the third threshold value th3 is obtained is the omnidirectional image data. On the displayed first monitor MN1, for example, it is drawn in red. Further, the pink region PD1 of the pixel in which the sound pressure larger than the second threshold value th2 and equal to or lower than the third threshold value th3 is obtained is drawn in pink, for example, on the first monitor MN1 on which the omnidirectional image data is displayed. The blue region BD1 of the pixel from which the sound pressure larger than the first threshold value th1 and equal to or lower than the second threshold value th2 is obtained is drawn in blue, for example, on the first monitor MN1 on which the omnidirectional image data is displayed. Further, the sound pressure region N1 of the pixels of the first threshold value th1 or less is drawn in, for example, colorless on the first monitor MN1 on which the omnidirectional image data is displayed, that is, it is not different from the display color of the omnidirectional image data. ..

The speaker device 37 outputs audio data collected by the microphone array MAk or audio data collected by the microphone array MAk and formed with directivity by the signal processing unit 33. The speaker device 37 may be configured as a device separate from the monitoring device 10.

The memory 38 is configured by using, for example, a ROM or RAM, and holds, for example, various data including sound data in a certain section, setting information, a program, and the like. Further, the memory 38 has a pattern memory (see FIG. 7) in which a sound pattern peculiar to each unmanned vehicle dn is registered. Further, the memory 38 stores the sound pressure heat map data generated by the output control unit 35. Further, in the memory 38, an identification mark (not shown) schematically indicating the position of the unmanned flying object dn is registered. The identification mark used here is, for example, a star-shaped symbol. The identification mark is not limited to a star shape, but may be a circle, a quadrangle, or a symbol or character such as a "swastika" shape reminiscent of an unmanned flying object dn. Further, the display mode of the identification mark may be changed between daytime and nighttime. For example, it may be a star shape in the daytime and a quadrangle not to be mistaken for a star at nighttime. Moreover, the identification mark may be changed dynamically. For example, the star-shaped symbol may be blinked or rotated to further alert the user.

FIG. 7 is a timing chart showing an example of the detection sound pattern of the unmanned aircraft dn registered in the memory 38. The detection sound pattern shown in FIG. 7 is a combination of frequency patterns, and sounds of four frequencies f1, f2, f3, and f4 generated by rotation of four rotors mounted on the multicopter type unmanned aircraft dn. Including. The signals of each frequency are, for example, signals of different sound frequencies generated by the rotation of a plurality of blades pivotally supported by each rotor.

In FIG. 7, the frequency domain indicated by the diagonal line is the region where the sound pressure is high. The detected sound pattern may include not only the number and sound pressure of sounds having a plurality of frequencies but also other sound information. For example, a sound pressure ratio representing the ratio of sound pressure at each frequency can be mentioned. Here, as an example, the detection of the unmanned flying object dn is determined by whether or not the sound pressure of each frequency included in the detection sound pattern exceeds the threshold value.

The directivity processing unit 63 uses the sound signal (also referred to as sound data) picked up by the omnidirectional microphones M1 to Mq and the setting result of the masking area setting unit 69a to perform the directivity forming process (beamforming) described above. ) Is performed, and sound data extraction processing is performed in which the directivity direction is the direction of an area other than the masking area set by the masking area setting unit 69a. Further, the directivity processing unit 63 can also perform sound data extraction processing in which a range in the direction of other areas other than the masking area set by the masking area setting unit 69a is set as the directivity range. Here, the directivity range is a range including a plurality of adjacent directivity directions, and is intended to include a certain extent of the directivity direction as compared with the directivity directions.

The frequency analysis unit 64 performs frequency analysis processing on the sound data extracted in the directivity direction by the directivity processing unit 63. In this frequency analysis process, the frequency included in the sound data in the directivity direction and the sound pressure thereof are detected.

FIG. 8 is a timing chart showing an example of the frequency change of the detection sound signal obtained as a result of the frequency analysis process. In FIG. 8, four frequencies f11, f12, f13, f14 and sound pressures of each frequency are obtained as detection sound signals (that is, detection sound data). In the figure, the irregularly changing frequency fluctuations are caused by, for example, the rotational fluctuations of the rotor (rotor blade) that slightly change when the unmanned vehicle dn controls the attitude of the unmanned vehicle dn itself.

The object detection unit 65 as the detection unit performs the detection processing of the unmanned flying object dn by using the frequency analysis processing result of the frequency analysis unit 64. Specifically, in the detection process of the unmanned vehicle dn, the object detection unit 65 detects the detection sound obtained as a result of the frequency analysis process in the areas other than the masking area set by the masking area setting unit 69a. (Refer to FIG. 8) (frequency f11 to f14) and the pattern of the detection sound registered in advance in the pattern memory of the memory 38 (see FIG. 7) (frequency f1 to f4) are compared. The object detection unit 65 determines whether or not the patterns of the detection sounds of both are similar.

Whether or not the two patterns are similar is determined as follows, for example. When the sound pressures of at least two frequencies included in the detection sound data among the four frequencies f1, f2, f3, and f4 each exceed the threshold value, it is assumed that the sound patterns are similar, and the object detection unit 65 is unmanned. Detects the flying object dn. The unmanned flying object dn may be detected when other conditions are satisfied.

When it is determined that the unmanned vehicle dn does not exist, the detection result determination unit 66 instructs the detection direction control unit 68 to shift to the detection of the unmanned vehicle dn in the next directivity direction. When the detection result determination unit 66 determines that the unmanned air vehicle dn exists as a result of scanning in the directivity direction, the detection result determination unit 66 notifies the output control unit 35 of the detection result of the unmanned air vehicle dn. The detection result includes information on the detected unmanned aircraft dn. The information of the unmanned air vehicle dn includes, for example, identification information of the unmanned air vehicle dn and position information (for example, direction information) of the unmanned air vehicle dn in the sound collecting space.

The detection direction control unit 68 controls the direction for detecting the unmanned flying object dn in the sound collecting space based on the instruction from the detection result determination unit 66. For example, the detection direction control unit 68 detects an arbitrary direction of the direction range BF1 (see FIG. 11) including the sound source position estimated by the sound source direction detection unit 34 in the entire sound collection space (that is, the monitoring area 8). Set as the direction.

The scanning control unit 67 instructs the directivity processing unit 63 to perform beamforming with the detection direction set by the detection direction control unit 68 as the directivity direction.

The directivity processing unit 63 performs beamforming in the directivity direction instructed by the scanning control unit 67. In the initial setting, the directivity processing unit 63 sets the initial position in the directivity range BF1 (see FIG. 11) including the sound source position estimated by the sound source direction detection unit 34 as the directivity direction BF2. The directivity direction BF2 is set one after another in the directivity range BF1 by the detection direction control unit 68.

The masking area setting unit 69a is based on the omnidirectional image data or the two-dimensional panoramic image data of the monitoring area 8 captured by the omnidirectional camera CAk and the audio data of the monitoring area 8 picked up by the microphone array MAk. A masking area is set to exclude the detection of the unmanned flying object dn appearing in the orientation image or the two-dimensional panoramic image (that is, the captured image). Details of the masking area setting will be described later with reference to FIGS. 13 and 14.

The output control unit 35 controls each operation of the first monitor MN1, the second monitor MN2, and the speaker device 37, and transmits the omnidirectional image data or the two-dimensional panoramic image data transmitted from the omnidirectional camera CAk to the first monitor MN1. , The second monitor MN2 is output and displayed, and the audio data transmitted from the microphone array MA is output to the speaker device 37. Further, when the unmanned vehicle dn is detected, the output control unit 35 displays the identification mark (not shown) representing the unmanned vehicle dn by superimposing it on the omnidirectional image. 2 Monitor MN2 is also acceptable).

Further, the output control unit 35 uses the sound data collected by the microphone array MAk and the coordinates indicating the direction of the sound source position derived by the omnidirectional camera CAk to obtain the sound data collected by the microphone array MAk. By performing the directivity formation process, the sound data in the directivity direction is emphasized. The directivity forming process of voice data is a known technique described in, for example, Japanese Patent Application Laid-Open No. 2015-209241.

Further, the output control unit 35 uses the sound pressure value for each pixel constituting the omnidirectional image data or the two-dimensional panoramic image data calculated by the sound source direction detection unit 34, and uses the omnidirectional image data or the two-dimensional panoramic image data. A sound pressure map is generated in which the calculated sound pressure values are assigned to the positions of the corresponding pixels for each of the pixels constituting the above. Further, the output control unit 35 performs color conversion processing on the sound pressure value for each pixel of the generated sound pressure map into a visual image (for example, a colored image) so that the user can easily visually distinguish the sound pressure value. , Generate a sound pressure heat map as shown in FIG.

Although it has been explained that the output control unit 35 generates a sound pressure map or a sound pressure heat map in which the sound pressure value calculated for each pixel is assigned to the position of the corresponding pixel, the sound pressure is generated for each pixel. Instead of calculating, the average value of the sound pressure value is calculated for each pixel block consisting of a predetermined number of pixels (for example, 2 × 2, 4 × 4), and the average of the sound pressure values corresponding to the corresponding predetermined number of pixels is calculated. A sound pressure map or a sound pressure heat map may be generated by assigning a value.

The details of the threshold value adjusting unit 69b will be described in the second embodiment described later, and detailed description thereof will be omitted here.

Next, the operation of the unmanned vehicle detection system 5 of the present embodiment will be described in detail.

FIG. 9 is a sequence diagram illustrating an example of the operation of detecting the unmanned aircraft dn and displaying the detection result in the first embodiment. When each device of the unmanned vehicle detection system 5 (for example, the first monitor MN1, the monitoring device 10, the omnidirectional camera CAk, and the microphone array MAk) is turned on, the unmanned vehicle detection system 5 starts operation. Further, as a premise of the explanation of FIG. 9, it is assumed that the masking area for excluding the detection of the unmanned aircraft dn is already set, and the information indicating the masking area is registered in the memory 38.

In the initial operation, the monitoring device 10 makes an image distribution request to the omnidirectional camera CAk (S1). The omnidirectional camera CAk starts the imaging process in response to the power-on according to this image distribution request. Further, the monitoring device 10 makes a voice distribution request to the microphone array MAk (S2). In response to this audio distribution request, the microphone array MA starts sound collection processing in response to power-on.

When the initial operation is completed, the omnidirectional camera CAk transmits the data of the omnidirectional image (for example, still image, moving image) obtained by imaging to the monitoring device 10 via the network NW (S3). Although it is described in FIG. 9 that the omnidirectional image data is transmitted from the omnidirectional camera CAk for the sake of simplicity, the two-dimensional panoramic image data may be transmitted, and the same applies to FIG. Is. The monitoring device 10 converts the omnidirectional image data transmitted from the omnidirectional camera CAk into display data such as NTSC, outputs the data to the first monitor MN1, and instructs the display of the omnidirectional image data (S4). When the first monitor MN1 inputs the display data transmitted from the monitoring device 10, the first monitor MN1 displays the data of the omnidirectional image IMG1 (see FIGS. 12 and 15) by the omnidirectional camera CAk on the screen.

Further, the microphone array MAk encodes the voice data of the monitoring area 8 obtained by sound collection and transmits it to the monitoring device 10 via the network NW (S5). In the monitoring device 10, the sound source direction detection unit 34 configures the omnidirectional image data of the monitoring area 8 based on the omnidirectional image data captured by the omnidirectional camera CAk and the sound data picked up by the microphone array MAk. The sound pressure as a sound parameter is calculated for each pixel, and the sound source position in the monitoring area 8 is estimated (S6). This estimated sound source position is used as a reference position of the directivity range BF1 required for setting the initial directivity direction when the monitoring device 10 detects the unmanned vehicle dn.

Further, in the monitoring device 10, the output control unit 35 uses the sound pressure value for each pixel that constitutes the omnidirectional image data calculated by the sound source direction detection unit 34, and each one constitutes the omnidirectional image data. For each pixel, a sound pressure map is generated in which the calculated value of the sound pressure is assigned to the position of the corresponding pixel. Further, the output control unit 35 performs color conversion processing on the sound pressure value for each pixel of the generated sound pressure map into a visual image (for example, a colored image) so that the user can easily visually distinguish the sound pressure value. , A sound pressure heat map as shown in FIG. 15 is generated (S7).

Further, in the monitoring device 10, the signal processing unit 33 sequentially determines the directivity outside the area of the masking area set by the masking area setting unit 69a by using the voice data transmitted from the microphone array MAk in step S5. By forming the vehicle, the detection and determination of the unmanned air vehicle dn is performed for each direction direction in which the directivity is formed (S8). Details of the detection and determination process of the unmanned aircraft dn will be described later with reference to FIGS. 10 and 11.

When the unmanned flying object dn is detected as a result of the detection determination process, the output control unit 35 in the monitoring device 10 displays the sound pressure generated in step S7 on the omnidirectional image IMG1 displayed on the screen of the first monitor MN1. It is instructed to superimpose and display the heat map and the identification mark (not shown) representing the unmanned aircraft dn existing in the directional direction detected in step S8 (S9).

The first monitor MN1 synthesizes (superimposes) a sound pressure heat map on the omnidirectional image IMG1 and displays it according to the instruction from the monitoring device 10, and also synthesizes an identification mark (not shown) representing the unmanned aircraft dn (not shown). Superimpose) and display (S10). After that, the process of the unmanned vehicle detection system 5 returns to step S3, and each process of steps S3 to S10 is repeated until a predetermined event such as, for example, the power is turned off is detected.

FIG. 10 is a flowchart illustrating a detailed example of the operation procedure of the unmanned vehicle detection determination in step S8 of FIG. In the sound source detection unit UDk, the directivity processing unit 63 uses the information of the masking area set by the masking area setting unit 69a and is based on the sound source position outside the masking area and estimated by the sound source direction detection unit 34. The directivity range BF1 is set as the initial position of the directivity direction BF2 (S21). The masking area information is the coordinates in the direction facing the masking area as seen from the microphone array MAk.

FIG. 11 is a diagram showing an example of how the directivity direction BF2 is sequentially scanned in the monitoring area 8 and the unmanned flying object dn is detected. The initial position may be outside the masking area set by the masking area setting unit 69a, and may not be limited to the directional range BF1 based on the sound source position of the monitoring area 8 estimated by the sound source direction detection unit 34. .. That is, if it is outside the masking area set by the masking area setting unit 69a, an arbitrary position designated by the user may be set as the initial position, and the inside of the monitoring area 8 may be sequentially scanned. Since the initial position is not limited, even if the sound source included in the direction range BF1 based on the estimated sound source position is not an unmanned air vehicle, it is possible to detect an unmanned air vehicle flying in another direction at an early stage. It becomes.

The directivity processing unit 63 determines whether or not the sound data collected by the microphone array MAk and converted into digital values by the A / D converters An1 to Aq is temporarily stored in the memory 38 (S22). .. If it is not stored (S22, NO), the process of the directivity processing unit 63 returns to step S21.

When the sound data collected by the microphone array MA is temporarily stored in the memory 38 (S22, YES), the directivity processing unit 63 is outside the masking area set by the masking area setting unit 69a. In addition, beamforming is performed on an arbitrary directivity direction BF2 in the directivity range BF1 of the monitoring area 8, and sound data in the directivity direction BF2 is extracted and processed (S23).

The frequency analysis unit 64 detects the frequency of the extracted sound data and the sound pressure thereof (S24).

The object detection unit 65 compares the pattern of the detection sound registered in the pattern memory of the memory 38 with the pattern of the detection sound obtained as a result of the frequency analysis process, and detects the unmanned flying object (S25).

The detection result determination unit 66 notifies the output control unit 35 of the result of this comparison, and also notifies the detection direction control unit 68 of the detection direction shift (S26).

For example, the object detection unit 65 compares the detection sound pattern obtained as a result of the frequency analysis process with the four frequencies f1, f2, f3, and f4 registered in the pattern memory of the memory 38. As a result of comparison, when the object detection unit 65 has at least two identical frequencies in the patterns of both detected sounds and the sound pressures of these frequencies are larger than the first threshold value th1, the patterns of both detected sounds are different. Approximately, it is determined that the unmanned air vehicle dn exists.

Here, it is assumed that at least two frequencies match, but the object detection unit 65 approximates when one frequency matches and the sound pressure at this frequency is greater than the first threshold value th1. It may be determined that the frequency is correct.

Further, the object detection unit 65 may set an error of an allowable frequency for each frequency, and may determine the presence or absence of the above approximation on the assumption that the frequencies within this error range are the same frequency.

Further, the object detection unit 65 may determine that the sound pressure ratios of the sounds of the respective frequencies are substantially the same in addition to the determination condition, in addition to the comparison of the frequencies and the sound pressures. In this case, since the determination condition becomes strict, the sound source detection unit UDk can easily identify the detected unmanned flying object dn as a pre-registered object, and can improve the detection accuracy of the unmanned flying object dn.

As a result of step S26, the detection result determination unit 66 determines whether or not the unmanned flying object dn exists (S27).

When the unmanned air vehicle dn exists (S27, YES), the detection result determination unit 66 notifies the output control unit 35 that the unmanned air vehicle dn exists (detection result of the unmanned air vehicle dn) (S28).

On the other hand, when the unmanned flying object dn does not exist (S27, NO), the detection result determination unit 66 instructs the scanning control unit 67 to move the directivity direction BF2 of the scanning target in the monitoring area 8 in the next different direction. To do. The scanning control unit 67 moves the directivity direction BF2 of the scanning target in the monitoring area 8 in the next different direction in response to an instruction from the detection result determination unit 66 (S29). It should be noted that the notification of the detection result of the unmanned vehicle dn may be collectively performed after the omnidirectional scanning is completed, not at the timing when the detection process of one directivity direction is completed.

Further, the order in which the directivity direction BF2 is sequentially moved in the monitoring area 8 is outside the masking area set by the masking area setting unit 69a, for example, within the directivity range BF1 of the monitoring area 8 or within the entire range of the monitoring area 8. Then, the order may be spiral (spiral) so as to go from the outer circumference to the inner circumference, or from the inner circumference to the outer circumference.

Further, the detection direction control unit 68 does not continuously scan the directivity direction as in a single stroke, but sets a position in the monitoring area 8 in advance, and the masking area set by the masking area setting unit 69a. If it is outside, the directivity direction BF2 may be moved to each position in any order. As a result, the monitoring device 10 can start the detection process from a position where, for example, the unmanned vehicle dn easily invades, and the detection process can be made more efficient.

The scanning control unit 67 determines whether or not scanning in all directions in the monitoring area 8 has been completed (S30). When the scanning in all directions is not completed (S30, NO), the processing of the signal processing unit 33 returns to step S23, and the processing of steps S23 to S30 is repeated. That is, the directivity processing unit 63 beamforms the directivity direction BF2 at the position moved in step S29 outside the masking area set by the masking area setting unit 69a, and extracts and processes the sound data in the directivity direction BF2. .. As a result, even if one unmanned vehicle dn is detected, the sound source detection unit UDk continues to detect the unmanned vehicle dn that may exist in the other, so that the detection of a plurality of unmanned vehicle dn can be performed. It is possible.

On the other hand, when the scanning in all directions is completed in step S30 (S30, YES), the directivity processing unit 63 erases the sound data temporarily stored in the memory 38 and collected by the microphone array MAk (S31). ). Further, the directivity processing unit 63 may erase the sound data collected by the microphone array MAk from the memory 38 and save the sound data in the recorder RC.

After erasing the sound data, the signal processing unit 33 determines whether or not to end the detection process of the unmanned aircraft dn (S32). The end of the detection process of the unmanned aircraft dn is performed according to a predetermined event. For example, the number of times the unmanned air vehicle dn is not detected in step S26 may be stored in the memory 38, and when this number of times exceeds a predetermined number of times, the detection process of the unmanned air vehicle dn may be terminated. Further, the signal processing unit 33 may end the detection process of the unmanned aircraft dn based on the time-up by the timer or the user operation for the UI (User Interface) (not shown) included in the operation unit 32. Further, it may be terminated when the power of the monitoring device 10 is turned off.

In the process of step S24, the frequency analysis unit 64 analyzes the frequency and also measures the sound pressure of that frequency. The detection result determination unit 66 determines that the unmanned aircraft dn is approaching the sound source detection unit UDk when the sound pressure level measured by the frequency analysis unit 64 gradually increases with the passage of time. May be good.

For example, when the sound pressure level of a predetermined frequency measured at time t11 is smaller than the sound pressure level of the same frequency measured at time t12 after time t11, the sound pressure increases with the passage of time. It may be determined that the unmanned air vehicle dn is approaching. Further, even if the sound pressure level is measured three times or more and it is determined that the unmanned air vehicle dn is approaching based on the transition of statistical values (for example, variance value, average value, maximum value, minimum value, etc.). Good.

Further, when the measured sound pressure level is larger than the alert threshold value which is the alert level, the detection result determination unit 66 may determine that the unmanned vehicle dn has entered the alert area.

The alert threshold is, for example, a value larger than the above-mentioned third threshold th3. The caution area is, for example, the same area as the monitoring area 8, or an area included in the monitoring area 8 and narrower than the monitoring area 8. The caution area is, for example, an area where the invasion of the unmanned aircraft dn is restricted. Further, the approach determination and the intrusion determination of the unmanned aircraft dn may be executed by the detection result determination unit 66.

FIG. 12 is a diagram showing an example of a display screen of the first monitor MN1 when the masking area MSK3 is not set. In FIG. 12, the unmanned aircraft dn detected by the monitoring device 10 is shown on the upper right side of the paper of FIG. 12 of the omnidirectional image IMG1 on the first monitor MN1. Further, sound pressure heat maps corresponding to sound sources generated within the range of the imaging angle of view of the unmanned flying object dn and the omnidirectional camera CAk are superimposed and displayed.

As described with reference to FIG. 9, in the present embodiment, if the sound pressure value for each pixel calculated by the sound source direction detection unit 34 is equal to or less than the first threshold value th1, the sound pressure value is displayed as colorless and the sound pressure value is the first. If it is larger than the first threshold value th1 and equal to or less than the second threshold value th2, it is displayed in blue, and if the sound pressure value is larger than the second threshold value th2 and less than or equal to the third threshold value th3, it is displayed in pink, and the sound pressure value is the third. If it is larger than the threshold value th3, it is displayed in red.

In FIG. 12, for example, since the sound pressure value is larger than the third threshold value th3 in the vicinity of the rotor and the rotor around the center of the housing of the unmanned flying object dn, the sound pressure values are drawn in the red regions RD1, RD2, RD3, and RD4. Similarly, around the red region, pink regions PD1, PD2, PD3, PD4 indicating that the sound pressure value is the second largest after the red region are drawn. Similarly, around the pink region, blue regions BD1, BD2, BD3, and BD4 indicating that the sound pressure value is the second largest after the pink region are drawn.

Further, FIG. 12 shows that a sound source also exists in an office building, and the pixel or a pixel whose sound pressure value for each pixel calculated by the sound source direction detection unit 34 exceeds the third threshold value th3. Red regions R1, R2, R3, R4, R5, R6, and R7 are drawn for the set. Similarly, around the red area of the office building, pink areas P1, P2, P3, P4, P5 indicating that the sound pressure value is the second largest after the red area are drawn. Similarly, around the pink area of the office building, blue areas B1 and B2 indicating that the sound pressure value is the second largest after the pink area are drawn. In the other regions of the omnidirectional image IMG1, since the sound pressure value is equal to or less than the first threshold value th1, the images are drawn in the colorless region N1 and the visibility of the background omnidirectional image IMG1 is not deteriorated.

Next, the details of setting the masking area in the present embodiment will be described with reference to FIGS. 13, 14 and 15. FIG. 13 is an explanatory diagram showing a display example of the masking area during the automatic learning process in chronological order. FIG. 14 is a sequence diagram illustrating an example of an operation procedure for setting the masking area in the first embodiment. FIG. 15 is a diagram showing an example of a display screen of the first monitor when a masking area is set. The sequence shown in FIG. 14 is a so-called initial setting that is executed before the start of the operation of the sequence shown in FIG.

In FIG. 14, the user instructs the monitoring device 10 to start the automatic learning process of the masking area by using, for example, the operation unit 32 (T1). The monitoring device 10 makes an image distribution request to the omnidirectional camera CAk (T2). The omnidirectional camera CAk starts the imaging process in response to the power-on according to this image distribution request. The omnidirectional camera CAk transmits the data of the omnidirectional image (for example, still image, moving image) obtained by imaging to the monitoring device 10 via the network NW (T3). The monitoring device 10 converts the omnidirectional image data transmitted from the omnidirectional camera CAk into display data such as NTSC, outputs the data to the first monitor MN1, and instructs the display of the omnidirectional image data (T4). As a result, when the first monitor MN1 inputs the display data transmitted from the monitoring device 10, the first monitor MN1 displays the data of the omnidirectional image IMG1 by the omnidirectional camera CAk on the screen (see the upper left of FIG. 13).

Further, the monitoring device 10 makes a voice distribution request to the microphone array MAk (T5). In response to this audio distribution request, the microphone array MA starts sound collection processing in response to power-on. The microphone array MAk encodes the voice data of the monitoring area 8 obtained by sound collection and transmits it to the monitoring device 10 via the network NW (T6). In the monitoring device 10, the sound source direction detection unit 34 configures the omnidirectional image data of the monitoring area 8 based on the omnidirectional image data captured by the omnidirectional camera CAk and the sound data picked up by the microphone array MAk. The sound pressure as a sound parameter is calculated for each pixel.

Further, the masking area setting unit 69a determines a pixel or a set thereof in which the calculated value of the sound pressure by the sound source direction detection unit 34 is equal to or higher than a predetermined masking area threshold value (for example, the third threshold value th3 described above). The masking area setting unit 69a stores and registers the information indicating the determined pixels or a set thereof in the memory 38 as the information indicating the masking area (T7). The information indicating the masking area is, specifically, coordinates on an omnidirectional image that specifies the position of a pixel whose calculated sound pressure value is equal to or greater than the masking area threshold value. The masking area setting unit 69a sets the information indicating the masking area and the masking area (that is, pixels whose calculated sound pressure value is equal to or higher than the masking area threshold value or a set thereof) in a predetermined color (for example, red) via the output control unit 35. ) Is output to the first monitor MN1 (T8). As a result, the first monitor MN1 performs a process of filling the position of the coordinates corresponding to the masking area MSK1 on the omnidirectional image IMG with a predetermined color according to the instruction transmitted from the monitoring device 10 (see the upper right corner of FIG. 13). .. In the omnidirectional image IMG1 on the upper right of the paper in FIG. 13, the masking area MSK1 shows the entire area filled with a predetermined color.

Similarly, the microphone array MA encodes the voice data of the monitoring area 8 obtained by the continuous sound collection via the network NW in accordance with the voice distribution request from the monitoring device 10 to the monitoring device 10. Send (T9). In the monitoring device 10, the sound source direction detection unit 34 configures the omnidirectional image data of the monitoring area 8 based on the omnidirectional image data captured by the omnidirectional camera CAk and the sound data picked up by the microphone array MAk. The sound pressure as a sound parameter is calculated for each pixel.

Further, the masking area setting unit 69a determines a pixel or a set thereof whose sound pressure calculated by the sound source direction detection unit 34 is equal to or greater than the masking area threshold value. The masking area setting unit 69a stores and registers the information indicating the determined pixels or a set thereof in the memory 38 as the information indicating the masking area (T10). The masking area setting unit 69a outputs information indicating the masking area and an instruction to fill the masking area with a predetermined color (for example, red) to the first monitor MN1 via the output control unit 35 (T11). As a result, the first monitor MN1 fills the position of the coordinates corresponding to the masking area MSK2 accumulated with respect to the masking area MSK1 on the omnidirectional image IMG with a predetermined color according to the instruction transmitted from the monitoring device 10. (See the lower right corner of Fig. 13). In the omnidirectional image IMG1 at the lower right of the paper, the masking area MSK2 shows the entire area filled with a predetermined color.

Here, the monitoring device 10 is instructed to end the automatic learning process of the masking area by a user operation using the operation unit 32 (T12). In response to this instruction, the monitoring device 10 transmits a voice distribution stop request to the microphone array MAk (T13). As a result, the microphone array MAk stops the distribution (transmission) of the audio data of the monitoring area 8 obtained by collecting the sound to the monitoring device 10.

Further, by a user operation using the operation unit 32, the operation of modifying the masking area (that is, adding or deleting the masking area) can be performed on the first monitor MN1 in which the masking area MSK2 at the lower right of FIG. 13 is shown. When this is done (T14), in the monitoring device 10, the masking area setting unit 69a adds or deletes the position on the designated omnidirectional image IMG1 as the masking area by the user operation, and then adds the position on the masking area. The information indicating the modified masking area MSK3 is stored in the memory 38 and registered (T15).

For example, in step T14, an operation for deleting unnecessary areas GOM1, GOM2, and GOM3 as masking areas at the user's discretion with respect to the first monitor MN1 in which the masking area MSK2 at the lower right of FIG. 13 is shown ( For example, a range designation operation) is performed, or an operation (that is, a drawing operation) is performed to add the entire area of the office building in the background covered by the masking area MSK2 as a masking area.

The masking area setting unit 69a outputs information indicating the masking area and an instruction to fill the masking area with a predetermined color (for example, red) to the first monitor MN1 via the output control unit 35 (T16). As a result, the first monitor MN1 sets the position of the coordinates corresponding to the masking areas MSK3 accumulated with respect to the masking areas MSK1 and MSK2 on the omnidirectional image IMG in a predetermined color according to the instruction transmitted from the monitoring device 10. Perform the filling process (see the lower left of the page in FIG. 13). In the omnidirectional image IMG1 at the lower left of the paper, the masking area MSK3 shows the entire area filled with a predetermined color. Therefore, the sky is excluded from the masking area MSK3.

As a result, according to the sequence shown in FIG. 14, the unmanned flying object dn as the detection target of the present embodiment is in the sky where there is almost no sound source in the surroundings within the range of the imaging angle of view of the omnidirectional camera CAk. In view of the tendency to fly, the sound source that generated the sound pressure above the masking area threshold was not found in the sky by analyzing the voice data from the microphone array MAk, so the sky was set as the detection target of the unmanned flying object dn. It will be possible. On the other hand, by setting the area around the office building where many sound sources exist as a masking area, it is possible to avoid erroneous detection of sound sources other than the unmanned air vehicle dn as the unmanned air vehicle dn that is originally desired to be detected. This makes it possible to improve the detection accuracy and detection processing speed of the unmanned aircraft dn.

In other words, as shown in FIG. 15, in the omnidirectional image IMG1, the unmanned flying object dn is detected only in another area outside the masking area MSK3. As a result, compared to the omnidirectional image IMG1 shown in FIG. 12, the sound pressure value of the sound generated at the sound source position at the sound source position around the unmanned flying object dn detected in the area other than the masking area MSK3. The sound pressure heat map converted into a visual image is superimposed and displayed. On the other hand, around other sound sources that are not unmanned flying objects (for example, the yelling of a person in an office building) detected in the masking area MSK3, the sound pressure value of the sound generated at that sound source position is displayed in a visual image. The superimposed display of the converted sound pressure heat map is omitted.

In the present embodiment, when the masking area is set by the masking area setting unit 69a, as shown in FIG. 15, the sound pressure value of the sound generated at the sound source position is visually observed around the unmanned flying object dn. The sound pressure heat map converted into an image is superimposed and displayed. However, when the masking area is not set by the masking area setting unit 69a, as shown in FIG. 12, the sound pressure heat map corresponding to the sound pressure value of the detected sound source is superimposed on the omnidirectional image IMG1. May be displayed.

As described above, in the unmanned flight object detection system 5 of the present embodiment, the monitoring device 10 uses the sound data collected by the microphone array MAk to perform unmanned flight appearing in the captured image (omnidirectional image IMG1) of the monitoring area 8. A masking area for excluding the detection of the body dn is set by the masking area setting unit 69a. The monitoring device 10 detects an unmanned flying object dn in an area other than the masking area by using the voice data collected by the microphone array MAk and the information indicating the masking area. Further, when the monitoring device 10 detects the unmanned flying object dn outside the masking area, the sound source visual image (that is, the sound source visual image showing the loudness of the sound at the sound source position at the sound source position of the unmanned flying object dn in the omnidirectional image IMG1 (that is,). , Red region RD1, pink region PD1, blue region BD1, ... Visual images) are superimposed and displayed on the first monitor MN1.

As a result, the unmanned flying object detection system 5 automatically sets a masking area for excluding the detection processing of the unmanned flying object dn to be detected with respect to the monitoring area 8 to be imaged by the omnidirectional camera CAk. Therefore, the probability of erroneously detecting an object at the sound source position in the masking area as an unmanned flying object dn can be reduced, and deterioration of the detection accuracy of the unmanned flying object dn can be suppressed. Further, the unmanned flying object detection system 5 does not need to detect the unmanned flying object dn over the imaging angle of view of the omnidirectional camera CAk (that is, the entire area of the omnidirectional image IMG1), and is unmanned for an area omitting the masking area. Since the detection determination of the air vehicle dn may be performed, the detection process of the unmanned air vehicle dn can be further improved.

Further, in the unmanned vehicle detection system 5, the sound source direction detection unit 34 determines the sound pressure for specifying the loudness of the monitoring area 8 based on the sound data collected by the microphone array MAk, in the omnidirectional image IMG1. It is calculated for each predetermined unit of the pixels constituting the above. The masking area setting unit 69a superimposes and displays the position of the sound source whose calculated sound pressure value is equal to or higher than the masking area threshold value related to the loudness or the area including the position on the first monitor MN1, and further confirms the user. By the operation, the sound source area displayed on the first monitor MN1 is set as the masking area. As a result, the user has a masking area for omitting a place where the unmanned air vehicle dn is unlikely to fly but where other sound sources (for example, the yelling of a person) are likely to occur from the area to be detected by the unmanned air vehicle dn. Therefore, it can be easily set while visually checking the first monitor MN1.

Further, the masking area setting unit 69a masks the sound source area after the user addition operation by the user addition operation for further adding the sound source area (that is, the area that is a candidate for the masking area) displayed on the first monitor MN1. Set as an area. As a result, the user visually confirms the portion painted in a predetermined color as a candidate for the masking area by the monitoring device 10 on the first monitor MN1, and further adds the portion to the masking area at his / her own discretion. The masking area can be set while easily specifying it, improving user-friendliness.

Further, the masking area setting unit 69a performs a user deletion operation for deleting at least a part of the sound source area (that is, a region that is a candidate for the masking area) displayed on the first monitor MN1, and the sound source after the user deletion operation. Set the area as a masking area. As a result, the user visually confirms the portion painted in the predetermined color as a candidate for the masking area by the monitoring device 10 on the first monitor MN1, and the portion painted in the predetermined color at his / her own discretion. The masking area can be set while easily specifying some parts that you want to exclude as the masking area from, improving the usability of the user.

Further, the output control unit 35 uses the monitoring area 8 to generate a sound source visual image in which the sound pressure is gradually converted into a different visual image according to the comparison between the calculated sound pressure value and the plurality of threshold values relating to the loudness of the sound. The omnidirectional image IMG1 is superimposed on each predetermined unit of the pixels constituting the IMG1 and displayed on the first monitor MN1. As a result, by looking at the first monitor MN1, the user can know a wide range of the monitoring area 8 as an omnidirectional image in the omnidirectional image IMG1 of the monitoring area 8 captured by the omnidirectional camera CAk. Not only that, the location of the sound source (for example, the unmanned flying object dn) generated outside the masking area of the monitoring area 8 and the loudness of the sound can be easily confirmed as a visual image.

(Background to the second embodiment)
In Patent Document 1 described above, by comparing the sound pressure at a frequency peculiar to a flying object such as a helicopter or Cessna with a predetermined set level, if it is above the set level, it can be determined that the flying object is to be monitored. It is disclosed.

However, Patent Document 1 described above does not take into consideration that the measured sound pressure quantitatively indicates which stage of the sound pressures defined by the plurality of stages is. Therefore, when some kind of sound is detected in the imaging area of the camera device, the loudness of the sound is specified by detailed visual information of the sound regardless of the loudness of the sound at the detected sound source position. There was a problem that it could not be presented as a target.

Therefore, in the second embodiment, regardless of the loudness of the sound at the sound source position detected in the imaging area of the camera device, the loudness at the sound source position is presented in detail and stepwise, and the sound source position is presented. An example of a monitoring system that contributes to an accurate grasp of the loudness of the sound source in the above will be described.

(Second embodiment)
In the second embodiment, the internal configurations of the devices constituting the unmanned flying object detection system 5 are the same because the internal configurations of the devices constituting the unmanned flying object detection system 5 of the first embodiment are the same. The contents are designated by the same reference numerals, the description thereof will be omitted, and different contents will be described.

In the second embodiment, the monitoring device 10 generates the sound pressure heat map described in the first embodiment, displays it on the first monitor MN1, and then operates the operation unit 32 of the user (see below). The sound pressure heat map is analyzed and displayed in detail according to the relationship between the calculated value of the sound pressure required for generating the sound pressure heat map and a plurality of threshold values (see below). Hereinafter, three types of analysis methods will be described.

(First analysis method)
In the first analysis method, the monitoring device 10 superimposes the sound pressure heat map corresponding to the omnidirectional image IMG2 on the omnidirectional image IMG2 and displays it on the first monitor MN1, and then the user partially displays the omnidirectional image IMG2. When the range of is specified, the display resolution of the sound pressure heat map in the specified range is changed so as to be the same as the display resolution of the entire omnidirectional image IMG2. An operation example of this first analysis method will be described with reference to FIGS. 16 and 17. FIG. 16 is a schematic explanatory view of the dynamic change of the display resolution of the sound pressure heat map in the second embodiment. FIG. 17 is a flowchart illustrating an example of an operation procedure for dynamically changing the display resolution of the sound pressure heat map in the second embodiment.

In FIG. 17, the monitoring device 10 inputs the audio data of the monitoring area 8 transmitted from the microphone array MAk (S41). It is determined by the user operation using the operation unit 32 whether or not a part of the cutout range of the omnidirectional image IMG2 (see the upper left of the paper in FIG. 16) with respect to the first monitor MN1 is specified (S42).

Here, in the upper left of the paper of FIG. 16, the omnidirectional image IMG2 displayed on the first monitor MN1 is a case where the display size of the omnidirectional image IMG2 in the X direction is "Wmax" and the display size in the Y direction is "Hmax". Has a display resolution of "Wmax x Hmax". Further, the coordinates of the end points of the cutout range that is a part of the omnidirectional image IMG2 specified by the user operation are (X1, Y1), (X1, Y2), (X2, Y1), (X2, Y2). It is a rectangle. In FIG. 16, only (X1, Y1) and (X2, Y2) existing on the diagonal line of the same rectangle are shown.

If a rectangle having four points (X1, Y1), (X1, Y2), (X2, Y1), and (X2, Y2) as end points is not specified by the user operation (S42, NO), monitoring The sound source direction detection unit 34 of the device 10 sets (X, Y) = (0,0) to generate a sound pressure map for the entire omnidirectional image IMG2 (S43), and coordinates (X, Y). The sound pressure P (X, Y) at = (0,0) is calculated (S45).

Further, when the X coordinate of the omnidirectional image IMG2 does not match the maximum value Wmax (S46, NO), the sound source direction detection unit 34 generates a sound pressure map for the entire omnidirectional image IMG2. , The X coordinate is incremented by 1 (S47), and the sound pressure P (X, Y) is calculated at (X, Y) after the increment.

When the X coordinate of the omnidirectional image IMG2 matches the maximum value Wmax (S46, YES), the sound source direction detection unit 34 does not match the Y coordinate of the omnidirectional image IMG2 with the maximum value Hmax (S46, YES). S48, NO), in order to generate a sound pressure map for the entire omnidirectional image IMG2, the X coordinate is returned to 0 and the Y coordinate is incremented by 1 (S49), and the sound pressure P is increased (X, Y). Calculate (X, Y). The sound source direction detection unit 34 repeats each process of steps 45 to S49 until the Y coordinate of the omnidirectional image IMG2 matches the maximum value Hmax. A sound pressure map can be generated, and the sound pressure map is saved and registered in the memory 38 (S50).

On the other hand, when a rectangle having four points (X1, Y1), (X1, Y2), (X2, Y1), and (X2, Y2) as end points is specified by the user operation (S42, YES), The sound source direction detection unit 34 of the monitoring device 10 sets (X, Y) = (X1, Y1) in order to generate a sound pressure map for a range specified by the user operation in the omnidirectional image IMG2 (S44). ), The sound pressure P (X, Y) at the coordinates (X, Y) = (X1, Y1) is calculated (S45).

Further, when the X coordinate of the omnidirectional image IMG2 does not match the maximum value Wmax (S46, NO), the sound source direction detection unit 34 makes a sound for a range specified by the user operation in the omnidirectional image IMG2. In order to generate a pressure map, the X coordinate is increased by (X2-X1) / Wmax (S47), and the sound pressure P (X, Y) is calculated at (X, Y) after the increase.

When the X coordinate of the omnidirectional image IMG2 matches the maximum value Wmax (S46, YES), the sound source direction detection unit 34 does not match the Y coordinate of the omnidirectional image IMG2 with the maximum value Hmax (S46, YES). S48, NO), in order to generate a sound pressure map for the range specified by the user operation in the omnidirectional image IMG2, the X coordinate is returned to X1 and the Y coordinate is increased by (Y2-Y1) / Hmax (S49). ), The sound pressure P (X, Y) is calculated at (X, Y) after the increase. The sound source direction detection unit 34 repeats each process of steps 45 to S49 until the Y coordinate of the omnidirectional image IMG2 matches the maximum value Hmax, so that the sound pressure for the range specified by the user operation in the omnidirectional image IMG2 is reached. A map can be generated, and the map is saved and registered in the memory 38 (S50). After step S50, the process of the monitoring device 10 returns to step S41, and the processes of steps S42 to S50 are repeated for the voice data input in step S41.

Therefore, as shown in the upper right of the paper of FIG. 16, the monitoring device 10 simply cuts out a range designated by the user operation and generates an audio heat map according to the sound pressure value for each pixel constituting the range. When superimposed on the 1-monitor MN1, only an image having a low display resolution (that is, a coarse image) can be displayed.

However, according to the first analysis method of the present embodiment, the monitoring device 10 displays the display resolution of the cutout range of a part of the cutout range of the omnidirectional image IMG2 designated by the user operation for the entire omnidirectional image IMG2. The sound pressure P (X, Y) is calculated in fine units (that is, (X2-X1) / Wmax in the X direction and (Y2-Y1) / Hmax in the Y direction) so as to be the same as the resolution. To do. As a result, as shown in the lower right of FIG. 16 paper, the monitoring device 10 has a finer resolution than the display resolution in the sound pressure heat map when the sound pressure heat map in the cutout range specified by the user operation is simply cut out. The unit) can be displayed on the first monitor MN1 with high accuracy, and the user can more accurately grasp the details of the distribution of the sound source in the cutout range specified by the user operation.

(Second analysis method)
Before explaining the second analysis method, as a matter common to FIGS. 18 to 20, in the present embodiment, the widths of a plurality of threshold values that gradually define the loudness of the sound will be described. FIG. 18 is a schematic explanatory view of the width adjustment of the threshold value according to the frequency distribution of the sound pressure value and the display result of the captured image accompanying the width adjustment in the second embodiment. In FIGS. 18 to 20, as the sound pressure value moves from the lower limit to the upper limit, the sound source visual images corresponding to the thresholds are ultramarine blue image, indigo blue image, blue image, light blue image, blue green image, yellow green image, and the like. It is stipulated that yellow images, orange images, red images, and crimson images are used. When the sound pressure value is the minimum (that is, the lower limit), the ultramarine blue image is used, and when the sound pressure value is the maximum (that is, the upper limit), the crimson image is used.

For example, as shown on the left side of FIG. 18 paper, a total of 10 threshold values are defined so as to correspond to the magnitude of the sound pressure value. For example, in the example on the left side of FIG. 18 paper, the threshold values are provided on the respective sound pressure value axes. Each scale corresponds to the threshold value. Therefore, in the example on the left side of the page of FIG. 18, when the sound pressure value is included between a certain threshold value (for example, 5) and the next largest threshold value (for example, 6), the sound pressure heat map is generated when the sound pressure heat map is generated. A color image (for example, a yellow-green image) corresponding to the threshold value including the sound pressure value is used as a sound source visual image for visually indicating the sound pressure value.

In the second analysis method, when the output control unit 35 generates the sound pressure heat map corresponding to the omnidirectional image, the threshold adjustment unit 69b of the monitoring device 10 is used for each pixel (a predetermined unit of pixels) constituting the omnidirectional image. Based on the occurrence frequency (in other words, frequency distribution) of the sound pressure value calculated in each case, the total of 10 threshold values or the width between the threshold values is dynamically changed. That is, the threshold value adjusting unit 69b dynamically changes the setting of the correspondence between the plurality of threshold values and the sound source visual image according to the omnidirectional image. For example, referring to FIG. 18, the threshold value adjusting unit 69b holds the setting of using the yellow-green image as the sound source visual image when the sound pressure value is between the threshold value 5 and the threshold value 6, but the sound pressure value is the threshold value. When the appearance frequency of 5 to threshold 6 is high among all the pixels of the omnidirectional image, the width LG1 between the thresholds for using the yellow-green image (for example, threshold 5 to threshold 6) becomes narrower. The width is changed to LG2 (for example, threshold value 4.5 to threshold value 4.8).

Therefore, when the sound pressure value for each pixel constituting the omnidirectional image is concentrated in the width AR1 between the thresholds for using a specific color image, the sound pressure is as shown in the lower left of FIG. As the heat map VMP1, the same color image or a visually similar color image (for example, a yellow-green image and a blue-green image) is used as a sound source visual image, and the distribution of sound sources appearing in the omnidirectional image is finely tuned. Difficult to present to the user.

However, according to the second analysis method of the present embodiment, the monitoring device 10 uses the threshold value adjusting unit 69b to generate a sound source according to the appearance frequency (frequency distribution) of the sound pressure value for each pixel constituting the omnidirectional image. The setting of the correspondence relationship between the visual image and the plurality of threshold values is dynamically changed, and the sound pressure heat map VMP2 corresponding to the omnidirectional image is displayed on the first monitor MN1 after reflecting the change. As a result, the monitoring device 10 can present a detailed sound pressure distribution to the user as the sound pressure heat map VMP2 as shown in the lower right of the paper of FIG. 18, so that the user can accurately grasp the distribution of the sound source position. Can be done.

(Third analysis method)
In the third analysis method, the monitoring device 10 arbitrarily sets the upper limit, the lower limit, or both of the respective threshold values for defining the use of the sound source visual image (that is, the color image) by the user operation using the operation unit 32. Can be specified for. 19 (A) and 19 (B) are schematic explanatory views of changing the setting of the width between the threshold values defining the use of the sound source visual image in the second embodiment.

For example, in FIG. 19A, the width between thresholds (upper threshold) for defining the use of a sound source visual image (ie, a crimson image) indicating that the sound pressure value is the highest (ie, the upper limit) is the threshold. To specify the use of a sound source visual image (ie, an ultramarine image) that is changed from 9 to threshold 10 to threshold 6 to threshold 10 and further indicates that the sound pressure value is the lowest (ie, the lower limit). The width between the threshold values (lower end threshold value) is changed from the threshold value 0 to the threshold value 1 to the threshold value 0 to the threshold value 2. As a result, the monitoring device 10 dynamically changes the remaining eight threshold values so that the widths between the threshold values are evenly spaced, which is different from the width between the two threshold values changed by the user operation. The width between the thresholds can be changed as described above.

Further, for example, in FIG. 19 (B), the use of a sound source visual image (that is, a crimson image) showing that the sound pressure value is the highest (that is, the upper limit) after being changed as shown in FIG. 19 (A). The width between the threshold values (upper limit threshold value) for defining is changed from the threshold value 6 to the threshold value 10 to the threshold value 9.3 to the threshold value 10, and further indicates that the sound pressure value is the lowest (that is, the lower limit). The width between the thresholds (bottom threshold) for defining the use of the sound source visual image (that is, the ultramarine blue image) is changed from the threshold 0 to the threshold 2 to the threshold 0 to the threshold 5. As a result, the monitoring device 10 similarly has the remaining eight thresholds different from the example shown in FIG. 19A, but the widths between the remaining eight thresholds are evenly spaced. It can be changed dynamically and the width between the thresholds can be changed so as to be different from the width between the two thresholds changed by the user operation.

Further, in FIGS. 19A and 19B, between threshold values (upper threshold values) for defining the use of a sound source visual image (that is, a crimson image) indicating that the sound pressure value is the highest (that is, the upper limit). Changed both the width of and the width between the thresholds (bottom thresholds) to define the use of the sound source visual image (ie, the ultramarine image) indicating the lowest (ie, the lower limit) of the sound pressure value. Although the example is described, the same applies to the change of only one of them.

That is, if only the width between the thresholds (upper thresholds) for defining the use of the sound source visual image (ie, the crimson image) indicating that the sound pressure value is the highest (ie, the upper limit) is changed. Similarly, the monitoring device 10 dynamically changes the width between the remaining nine thresholds so as to be evenly spaced, and the width between the thresholds is different from the width between one threshold changed by the user operation. The width can be changed.

Also, when only the width between the thresholds (bottom threshold) for defining the use of the sound source visual image (that is, the ultramarine image) indicating that the sound pressure value is the lowest (that is, the lower limit) is changed. Similarly, the monitoring device 10 dynamically changes the width between the remaining nine thresholds so as to be evenly spaced, and the width between the thresholds is different from the width between one threshold changed by the user operation. You can change the width of.

Here, the operation related to the setting change of the threshold width by the third analysis method will be described with reference to FIGS. 20 and 21. FIG. 20 is a schematic explanatory view of the display of the captured image due to the change of the width setting between the threshold values defining the use of the crimson image and the ultramarine blue image in the second embodiment. FIG. 21 is a flowchart illustrating an example of an operation procedure for changing the setting of the width between the threshold values in the second embodiment.

As shown in FIG. 20, a sound source visual image (that is, a crimson image) indicating that the sound pressure value is the highest (that is, the upper limit) and a sound pressure value that is the lowest (that is, the lower limit) are shown. It is assumed that the width between the thresholds that define each use with the sound source visual image (that is, the ultramarine image) is changed. In this case, in the monitoring device 10, as shown in FIG. 21, the threshold value adjusting unit 69b is a correspondence table (not shown, correspondence relationship) between the sound source visual image and the threshold value that defines the use of the sound source visual image or the width between the threshold values. Dynamically change the settings of.

Therefore, in the example shown in FIG. 20, in the monitoring device 10, the sound pressure heat map VMP2 superimposed on the omnidirectional image (that is, the captured image) is only a crimson image, but a fine color type can be obtained by user operation. A sound pressure heat map VMP2A using a sound source visual image is generated, and the sound pressure heat map VMP2A is superimposed on an omnidirectional image (that is, an captured image) and displayed on the first monitor MN1. The user operation is, for example, an input operation for the threshold value or the width between the threshold values displayed on the first monitor MN1 (not shown), or a drag on the display screen for the width between the threshold values displayed on the first monitor MN1. Operations apply, but are not limited to these operations.

In FIG. 21, the threshold adjustment unit 69b defines the use of a sound source visual image (that is, a crimson image) indicating that the sound pressure value is the highest (that is, the upper limit) by the user operation using the operation unit 32. It is determined whether or not the width between the thresholds has been changed (S61). When the width between the thresholds defining the use of the sound source visual image (that is, the crimson image) indicating that the sound pressure value is the highest (that is, the upper limit) is changed (S61, YES), the threshold adjusting unit 69b , It is determined whether or not the width between the thresholds defining the use of the sound source visual image (that is, the ultramarine image) indicating that the sound pressure value is the lowest (that is, the lower limit) has been changed (S62).

The threshold value adjusting unit 69b changes the width between the threshold values indicating the use of the sound source visual image (that is, the ultramarine blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) (S62, YES), the corresponding table of the threshold value or the width between the threshold values that defines the use of the sound source visual image and the sound source visual image is modified according to the change result (S63). For example, the threshold value adjusting unit 69b dynamically changes the remaining eight threshold values so that the widths between the threshold values are evenly spaced, and is different from the width between the two threshold values changed by the user operation. Change the width between thresholds to. As a result, the monitoring device 10 obtains a large sound pressure value between the threshold value that regulates the use of the crimson image and the threshold value that regulates the use of the ultramarine blue image, for example, the width between the threshold values by the user. By adjusting, the distribution around the pixel where many sound pressure values are concentrated can be displayed in detail as a sound pressure heat map.

The threshold adjustment unit 69b does not change the width between the thresholds that define the use of the sound source visual image (that is, the crimson image) indicating that the sound pressure value is the lowest (that is, the upper limit) (S62, NO). , The correspondence table between the sound source visual image and the threshold value that defines the use of the sound source visual image or the width between the threshold values is modified according to the change result (S64). For example, the threshold value adjusting unit 69b dynamically changes the remaining nine threshold values so that the widths between the threshold values are evenly spaced, and is different from the width between one threshold value changed by the user operation. Change the width between thresholds to. As a result, when the monitoring device 10 obtains many sound pressure values below the threshold value that regulates the use of the crimson image, for example, a large number of sound pressure values are concentrated by the user adjusting the width between the threshold values. The distribution around the pixels can be displayed in detail as a sound pressure heat map.

On the other hand, in step S61, when the width between the thresholds defining the use of the sound source visual image (that is, the crimson image) indicating that the sound pressure value is the highest (that is, the upper limit) is not changed (S61, NO). The threshold adjustment unit 69b determines whether or not the width between the thresholds defining the use of the sound source visual image (that is, the ultramarine image) indicating that the sound pressure value is the lowest (that is, the lower limit) has been changed. (S65). If the width between the threshold values that specify the use of the sound source visual image (that is, the ultramarine blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) is not changed (S65, NO), the threshold value. The process of the adjusting unit 69b returns to step S61.

The threshold value adjusting unit 69b changes the width between the threshold values indicating the use of the sound source visual image (that is, the ultramarine blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) (S65, YES), the corresponding table of the threshold value or the width between the threshold values that defines the use of the sound source visual image and the sound source visual image is modified according to the change result (S66). For example, the threshold value adjusting unit 69b dynamically changes the remaining nine threshold values so that the widths between the threshold values are evenly spaced, and is different from the width between one threshold value changed by the user operation. Change the width between thresholds to. As a result, when the monitoring device 10 obtains a large number of sound pressure values that are equal to or higher than the threshold value that defines the use of the ultramarine image, for example, the user can adjust the width between the threshold values to obtain a large number of sound pressure values. The distribution around the concentrated pixels can be displayed in detail as a sound pressure heat map.

As described above, in the unmanned flying object detection system 5 of the present embodiment, the monitoring device 10 monitors the sound pressure for specifying the loudness of the monitoring area 8 by using the sound data collected by the microphone array MAk. It is calculated for each predetermined unit of the pixels constituting the captured image (omnidirectional image IMG2) of the area 8. The monitoring device 10 uses the sound source visual image in which the sound pressure is gradually converted into different visual images according to the comparison between the calculated sound pressure value and a plurality of threshold values relating to the loudness of the pixels constituting the captured image. It is superimposed for each predetermined unit and displayed on the first monitor MN1. When any sound source position is specified in the captured image on which the sound source visual image is superimposed, the monitoring device 10 sets a predetermined unit of pixels constituting the rectangular range including the sound source position between the captured image and the rectangular range. The sound pressure is calculated for each value divided by the size ratio.

As a result, the monitoring device 10 has a higher resolution (unit) than the display resolution in the sound pressure heat map when the sound pressure heat map in the rectangular range (that is, the cutout range) specified by the user operation is simply cut out. It can be displayed on the first monitor MN1 with high accuracy, and the user can more accurately grasp the details of the distribution of the sound source in the cutout range specified by the user operation. In other words, the monitoring device 10 presents the loudness of the sound at the sound source position in a detailed and stepwise manner regardless of the loudness of the sound at the sound source position detected in the monitoring area 8 of the omnidirectional camera CAk. It can contribute to the accurate grasp of the loudness of the sound at the sound source position to the user.

Further, in the unmanned vehicle detection system 5 of the present embodiment, the monitoring device 10 uses the sound data collected by the microphone array MAk to determine the sound pressure for specifying the loudness of the monitoring area 8 in the monitoring area. It is calculated for each predetermined unit of the pixels constituting the captured image (omnidirectional image IMG2) of 8. The monitoring device 10 corresponds to each threshold value of a plurality of threshold values that gradually defines the loudness of the sound and a sound source visual image in which the sound pressure is gradually converted into a visual image having a different sound pressure according to the comparison with each threshold value. The relationship setting is dynamically changed according to the captured image (that is, the omnidirectional image) of the monitoring area 8. The monitoring device 10 uses a sound source visual image corresponding to the calculated sound pressure value as the captured image for each predetermined unit of the pixels constituting the captured image based on the calculated sound pressure value and the changed correspondence relationship setting. It is superimposed and displayed on the first monitor MN1.

As a result, the monitoring device 10 provides a fine sound pressure as a sound pressure heat map for visually indicating the position of the sound source picked up in the monitoring area 8 according to the captured image captured by the omnidirectional camera CAk. The distribution can be presented to the user, and the distribution of the sound source position can be grasped more accurately by the user.

Further, the threshold value adjusting unit 69b of the monitoring device 10 changes the width between each threshold value that defines the sound source visual image based on the appearance frequency of the sound pressure for each predetermined unit of the pixels constituting the captured image of the monitoring area 8. .. As a result, the monitoring device 10 increases the types of sound source visual images for the pixels corresponding to the sound pressure calculated values that appear frequently, while the sound source visual images are added to the pixels corresponding to the sound pressure calculated values that appear less frequently. Since the sound pressure heat map is generated after reducing the types of sounds, it is possible to present the sound pressure distribution of various shades in detail instead of the monotonous hue, and it is possible for the user to accurately grasp the distribution of the sound source position.

Further, the threshold value adjusting unit 69b of the monitoring device 10 is changed in width according to the operation of changing the width between the threshold values that defines the use of the sound source visual image (that is, the crimson image) corresponding to the upper limit value of the sound pressure. After that, the width between all the other thresholds except the width between the thresholds is changed evenly. As a result, when the monitoring device 10 obtains many sound pressure values below the threshold value that regulates the use of the crimson image, for example, a large number of sound pressure values are concentrated by the user adjusting the width between the threshold values. The distribution around the pixels can be displayed in detail as a sound pressure heat map.

Further, the threshold value adjusting unit 69b of the monitoring device 10 changes the width according to the operation of changing the width between the threshold values that defines the use of the sound source visual image (that is, the ultramarine blue image) corresponding to the lower limit value of the sound pressure. The width between all the other thresholds except the width between the thresholds after being set is changed evenly. As a result, when the monitoring device 10 obtains a large number of sound pressure values that are equal to or higher than the threshold value that defines the use of the ultramarine image, for example, the user can adjust the width between the threshold values to obtain a large number of sound pressure values. The distribution around the concentrated pixels can be displayed in detail as a sound pressure heat map.

Further, the threshold value adjusting unit 69b of the monitoring device 10 has a sound source visual image corresponding to the width between the threshold values and the lower limit value of the sound pressure that define the use of the sound source visual image (that is, the crimson image) corresponding to the upper limit value of the sound pressure. In response to the change of width between thresholds that defines the use of (ie, ultramarine images), the width between all other thresholds is evenly changed except the width between the thresholds after those widths have been changed. To do. As a result, the monitoring device 10 obtains a large sound pressure value between the threshold value that regulates the use of the crimson image and the threshold value that regulates the use of the ultramarine blue image, for example, the width between the threshold values by the user. By adjusting, the distribution around the pixel where many sound pressure values are concentrated can be displayed in detail as a sound pressure heat map.

(Background to the third embodiment)
In Patent Document 1 described above, a surveillance camera capable of changing the shooting direction in an arbitrary direction within the surveillance area is provided, and when a flying object such as a helicopter or a Cessna is detected, the shooting direction of this surveillance camera is changed. The conversion is disclosed. In other words, it is disclosed to change the shooting direction of the surveillance camera in order to focus on and image the detected flying object.

However, Patent Document 1 described above does not consider displaying a wide range of captured images of the surroundings including an unmanned flying object detected within the angle of view of the camera device with respect to the imaging area. Therefore, the user visually presents what kind of sound source exists in what kind of place in the image pickup area of the camera device when the unmanned vehicle is detected and in what other place in the same image pickup area. There was a problem that it could not be done.

Therefore, in the third embodiment, the camera detects what kind of sound source exists in what kind of place in the image pickup area of the camera device, and in what other place in the same image pickup area. An example of a monitoring system that visually presents to the user without deteriorating the visibility of the captured image of the device will be described.

(Third Embodiment)
In the third embodiment, the internal configurations of the devices constituting the unmanned flying object detection system 5 are the same because the internal configurations of the devices constituting the unmanned flying object detection system 5 of the first embodiment are the same. The contents are designated by the same reference numerals, the description thereof will be omitted, and different contents will be described.

In the third embodiment, the monitoring device 10 generates a sound pressure heat map (sound parameter map) described in the first embodiment, and then becomes a semi-transparent image (semi-transparent map) of the sound pressure heat map. A semi-transparent sound pressure heat map is generated, and this semi-transparent sound pressure heat map is superimposed on an omnidirectional image and displayed on the first monitor MN1 (see FIG. 22). FIG. 22 is a schematic explanatory view of an overlay display of an omnidirectional image and a translucent sound pressure heat map in the third embodiment.

In the present embodiment, as shown in FIG. 22, the monitoring device 10 displays the omnidirectional image IMG1 captured by the omnidirectional camera CAk on the first monitor MN1. Further, the monitoring device 10 uses a sound source visual image in which the sound pressure value calculated for each pixel constituting the omnidirectional image IMG1 or for each predetermined unit of the pixel is gradually converted into a different visual image, and the omnidirectional image IGM1 is used. A sound pressure heat map corresponding to the above is generated, and further, a semi-transparent sound pressure heat map TRL1 obtained by converting this sound pressure heat map into a semi-transparent image is generated and displayed on the second monitor MN2.

In FIG. 22, the monitoring device 10 displays the omnidirectional image IMG1 on the first monitor MN1 and the translucent sound pressure heat map TRL1 on separate monitors from the second monitor MN2. For example, the omnidirectional image IMG1 may be displayed in a window, and the translucent sound pressure heat map TRL1 may be displayed in another window.

Further, the monitoring device 10 displays the omnidirectional image IMG1A on which the translucent sound pressure heat map TRL1 is superimposed on the omnidirectional image IMG1 on the first monitor MN1 (see FIG. 23). FIG. 23 is a diagram showing an example of a display screen of the first monitor MN1 on which the omnidirectional image IMG1 and the translucent sound pressure heat map TRL1 are overlaid.

In FIG. 23, the unmanned flying object dn detected by the monitoring device 10 is shown on the upper right side of the paper of FIG. 23 of the omnidirectional image IMG1A on the first monitor MN1. Further, a semi-transparent sound pressure heat map corresponding to each of the unmanned flying object dn and the sound source generated within the range of the imaging angle of view of the omnidirectional camera CAk is superimposed and displayed.

As will be described with reference to FIG. 24, in the present embodiment, if the sound pressure value for each pixel calculated by the sound source direction detection unit 34 is equal to or less than the first threshold value th1, a colorless translucent color (that is, colorless). If the sound pressure value is greater than the first threshold value th1 and less than or equal to the second threshold value th2, it is displayed in a translucent blue color, and if the sound pressure value is greater than the second threshold value th2 and less than or equal to the third threshold value th3. For example, it is displayed in a pink semi-transparent color, and if the sound pressure value is larger than the third threshold value th3, it is displayed in a red semi-transparent color.

In FIG. 23, for example, since the sound pressure value is larger than the third threshold value th3 in the vicinity of the rotor and the rotor around the center of the housing of the unmanned flying object dn, it is drawn in the translucent red regions RD1A, RD2A, RD3A, and RD4A. .. Similarly, around the translucent red region, semitransparent pink regions PD1A, PD2A, PD3A, and PD4A indicating that the sound pressure value is the second largest after the translucent red region are drawn. Similarly, around the translucent pink region, semitransparent blue regions BD1A, BD2A, BD3A, and BD4A indicating that the sound pressure value is the second largest after the translucent pink region are drawn.

Further, FIG. 23 shows that a sound source also exists in an office building, and the pixel or a pixel whose sound pressure value for each pixel calculated by the sound source direction detection unit 34 exceeds the third threshold value th3. Semi-transparent red regions R1A, R2A, R3A, R4A, R5A, R6A, and R7A are drawn for the set. Similarly, around the translucent red region of the office building, translucent pink regions P1A, P2A, P3A, P4A, and P5A indicating that the sound pressure value is the second largest after the translucent red region are drawn. Similarly, around the translucent pink region of the office building, translucent blue regions B1A and B2A indicating that the sound pressure value is the second largest after the translucent pink region are drawn. In the other regions of the omnidirectional image IMG1A, since the sound pressure value is equal to or less than the first threshold value th1, the images are drawn in the colorless region.

As described above, in the present embodiment, the monitoring device 10 superimposes the translucent sound pressure heat map TRL1 different from the first embodiment on the omnidirectional image IMG1 and displays it on the first monitor MN1. The position of the sound source appearing in the IMG1 and the loudness of the sound at that position can be visually discriminated by the user, and the visibility of the omnidirectional image IMG1 can not be deteriorated.

Next, the operation of the unmanned vehicle detection system 5 of the present embodiment will be described in detail with reference to FIG. 24.

FIG. 24 is a sequence diagram illustrating an example of an operation procedure of overlay display of the omnidirectional image IMG1 and the translucent sound pressure heat map TRL1 in the third embodiment. When each device of the unmanned vehicle detection system 5 (for example, the first monitor MN1, the monitoring device 10, the omnidirectional camera CAk, and the microphone array MAk) is turned on, the unmanned vehicle detection system 5 starts operation. Further, in the description of FIG. 24, the masking area described in the first embodiment may or may not be used. In FIG. 24, for example, a case where a masking area is used will be described as an example. When the masking area is used, it is assumed that the information indicating the masking area is registered in the memory 38.

The monitoring device 10 makes an image distribution request to the omnidirectional camera CAk (S71). The omnidirectional camera CAk starts the imaging process in response to the power-on according to this image distribution request. Further, the monitoring device 10 makes a voice distribution request to the microphone array MAk (S72). In response to this audio distribution request, the microphone array MA starts sound collection processing in response to power-on.

When the initial operation is completed, the omnidirectional camera CAk transmits the data of the omnidirectional image (for example, still image, moving image) obtained by imaging to the monitoring device 10 via the network NW (S73). Although it is described in FIG. 24 that the omnidirectional image data is transmitted from the omnidirectional camera CAk for the sake of simplicity, the two-dimensional panoramic image data may be transmitted. The monitoring device 10 converts the omnidirectional image data transmitted from the omnidirectional camera CAk into display data such as NTSC, outputs the data to the first monitor MN1, and instructs the display of the omnidirectional image data (S74). When the first monitor MN1 inputs the display data transmitted from the monitoring device 10, the first monitor MN1 displays the data of the omnidirectional image IMG1 by the omnidirectional camera CAk (see the upper left of FIG. 22) on the screen.

Further, the microphone array MAk encodes the voice data of the monitoring area 8 obtained by sound collection and transmits it to the monitoring device 10 via the network NW (S75). In the monitoring device 10, the sound source direction detection unit 34 configures the omnidirectional image data of the monitoring area 8 based on the omnidirectional image data captured by the omnidirectional camera CAk and the sound data picked up by the microphone array MAk. The sound pressure as a sound parameter is calculated for each pixel, and the sound source position in the monitoring area 8 is estimated (S76). This estimated sound source position is used as a reference position of the directivity range BF1 required for setting the initial directivity direction when the monitoring device 10 detects the unmanned vehicle dn.

Further, in the monitoring device 10, the output control unit 35 uses the sound pressure value for each pixel that constitutes the omnidirectional image data calculated by the sound source direction detection unit 34, and each one constitutes the omnidirectional image data. For each pixel, a sound pressure map is generated in which the calculated value of the sound pressure is assigned to the position of the corresponding pixel. Further, the output control unit 35 performs color conversion processing on the sound pressure value for each pixel of the generated sound pressure map into a visual image (for example, a colored image) so that the user can easily visually distinguish the sound pressure value. , A semi-transparent sound pressure heat map as shown in the upper right corner of FIG. 22 is generated (S77). As for the method of generating the semi-transparent sound pressure heat map, for example, the output control unit 35 first generates the sound pressure heat map (see step S7 in FIG. 9), and secondly generates the semi-transparent sound pressure heat map. The procedure is to generate a translucent sound pressure heat map by performing transparent processing.

Further, in the monitoring device 10, the signal processing unit 33 sequentially determines the directivity outside the area of the masking area set by the masking area setting unit 69a by using the voice data transmitted from the microphone array MAk in step S75. By forming the vehicle, the detection and determination of the unmanned air vehicle dn is performed for each direction direction in which the directivity is formed (S78). Since the details of the detection determination process of the unmanned vehicle dn have been described with reference to FIGS. 10 and 11, the description thereof will be omitted here.

When the unmanned flying object dn is detected as a result of the detection determination process, the output control unit 35 in the monitoring device 10 displays the omnidirectional image IMG1 displayed on the screen of the first monitor MN1 on the translucent image generated in step S77. It is instructed to superimpose and display the sound pressure heat map and the identification mark (not shown) representing the unmanned aircraft dn existing in the directional direction detected in step S78 (S79).

The first monitor MN1 synthesizes (superimposes) a translucent sound pressure heat map on the omnidirectional image IMG1 and displays it according to the instruction from the monitoring device 10, and also displays an identification mark (not shown) indicating the unmanned aircraft dn. It is combined (superimposed) and displayed (S80). After that, the process of the unmanned vehicle detection system 5 returns to step S73, and each process of steps S73 to S80 is repeated until a predetermined event such as, for example, the power is turned off is detected.

As described above, in the unmanned flying object detection system 5 of the present embodiment, the monitoring device 10 monitors the sound pressure for specifying the loudness of the monitoring area 8 by using the sound data collected by the microphone array MAk. It is calculated for each predetermined unit of the pixels constituting the captured image (omnidirectional image IMG1) of the area 8. The monitoring device 10 converts the sound pressure into a visual image for each predetermined unit of the pixel according to the comparison between the calculated value of the sound pressure and the threshold value for the loudness of the sound source, and displays the sound source visual image in all directions of the monitoring area 8. Generate a translucent sound pressure heat map that is concatenated to correspond to the size of the image. The monitoring device 10 superimposes the translucent sound pressure heat map on the captured image of the monitoring area 8 and displays it on the first monitor MN1.

As a result, in the unmanned flying object detection system 5, the unmanned flying object is detected in any place in the monitoring area 8 of the omnidirectional camera CAk, and what kind of sound source exists in any other place in the same monitoring area 8. This can be visually presented to the user without deteriorating the visibility of the captured image of the omnidirectional camera CAk.

Further, a plurality of threshold values relating to the loudness of sound are provided, and the monitoring device 10 gradually converts the sound pressure into different visual images for each predetermined unit of the pixel according to the comparison between the sound pressure and the plurality of threshold values. Using the sound source visual image, a translucent sound pressure heat map having a plurality of types of sound source visual images is generated. As a result, the monitoring device 10 uses a visual sound source visual image to detect the presence of sound pressure having a plurality of types of stages defined by a plurality of threshold values in the omnidirectional image captured by the omnidirectional camera CAk. It is possible to make the user more explicitly discriminate.

Further, also in the present embodiment, the monitoring device 10 sets the masking area described in the first embodiment, and uses the voice data picked up by the microphone array MAk and the information indicating the masking area to use the masking area. Detects unmanned aircraft outside. When the monitoring device 10 detects an unmanned vehicle outside the masking area, the sound emitted from the unmanned vehicle around the unmanned vehicle (in other words, the sound source position of the unmanned vehicle) in the omnidirectional image. The sound source visual image showing the size is translucently displayed on the first monitor MN1. As a result, the monitoring device 10 can exclude the inside of the masking area from the detection target of the unmanned vehicle, so that deterioration of the detection accuracy of the masking area can be suppressed and the detection speed of the unmanned vehicle can be improved. Further, the monitoring device 10 sets the level of the loudness of the sound emitted from the unmanned flying object at the sound source position of the unmanned flying object dn detected outside the masking area by the translucent image of the sound source visual image. Therefore, the visibility of the surrounding captured image can not be deteriorated.

Further, also in the present embodiment, the monitoring device 10 sets the correspondence relationship between each threshold value of the plurality of threshold values for stepwisely defining the loudness of the sound and the visual image of the plurality of types of sound sources in the captured image of the imaging area. Change accordingly. The monitoring device 10 connects the sound source visual image for each predetermined unit of the pixel so as to correspond to the size of the captured image in the imaging area based on the calculated value of the sound pressure and the changed correspondence relationship. Generate a heat map. As a result, the monitoring device 10 calculates the sound pressure obtained for each pixel constituting the captured image or for each predetermined unit of the pixel according to the content of the omnidirectional image (captured image) captured by the omnidirectional camera CAk. The correspondence between the value and the sound source visual image corresponding to the calculated value of the sound pressure can be made variable. Therefore, for example, in a place where specific sound pressure calculation values are concentrated, the monitoring device 10 uses a sound source visual image of a plurality of types of colors instead of a sound source visual image of a single color in that area. It is possible for the user to grasp the distribution of the loudness of the sound source in the sound source appearing in the captured image clearly and finely in detail.

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can come up with various modifications or modifications within the scope of the claims, which naturally belong to the technical scope of the present invention. Understood.

According to the present invention, an unmanned vehicle is detected in what place in the image pickup area of the camera device, and what kind of sound source exists in what other place in the same image pickup area. It is useful as a monitoring system and monitoring method that visually presents to the user without deteriorating visibility.

5 Unmanned flying object detection system 10 Monitoring device 31 Communication unit 32 Operation unit 33 Signal processing unit 34 Sound source direction detection unit 35 Output control unit 37 Speaker device (SPK)
38 Memory 39 Setting management unit 63 Directivity processing unit 64 Frequency analysis unit 65 Object detection unit 66 Detection result determination unit 67 Scan control unit 68 Detection direction control unit 69a Masking area setting unit 69b Threshold adjustment unit CA1, CAk, CAN Omnidirectional Camera CZ1, CZk, CZn PTZ Camera MA1, MAk, Man Microphone array MN1 1st monitor MN2 2nd monitor NW network RC recorder UD1, UDn Sound source detection unit

The present invention relates to, for example, an image processing device, a monitoring system, and an image processing method for monitoring an imaging area of a camera device in which an unmanned vehicle is flying.

The present invention includes a sound parameter deriving unit that derives a sound parameter that specifies the loudness of sound picked up by the sound collecting device in the imaging area imaged by the imaging device in association with the captured image of the imaging area. By comparing the sound parameter with the threshold value related to the loudness of the sound, a non-detection area setting unit that sets a region having a sound parameter exceeding the threshold value as a non-detection region with respect to the imaging region of the imaging area. Provided is an image processing apparatus, wherein the sound parameter is updated at a predetermined timing, and the non-detection region is cumulatively set each time the sound parameter is updated.

Further, the present invention includes a step of deriving a sound parameter that specifies the loudness of the sound picked up by the sound collecting device in the image pickup area imaged by the image pickup device in association with the captured image of the image pickup area. A step of setting a region having a sound parameter exceeding a threshold value as a non-detection region with respect to the imaging region of the imaging area is provided, the sound parameter is updated at a predetermined timing, and the non-detection region has the sound parameter. It provides an image processing method that is cumulatively set each time it is updated.

Claims (5)

A camera that captures the imaging area and
A microphone array that collects sound from the imaging area, and
A monitor that displays a captured image of the imaging area captured by the camera, and
Based on the sound picked up by the microphone array, a sound parameter deriving unit that derives a sound parameter for specifying the loudness of the sound in the imaging area for each predetermined unit of pixels constituting the image captured in the imaging area. ,
According to the comparison between the sound parameter derived by the sound parameter derivation unit and the threshold value related to the loudness of the sound, a sound source visual image obtained by converting the sound parameter into a visual image for each predetermined unit of the pixel is captured. It is equipped with a signal processing unit that generates a semi-transparent map of sound parameter maps that are connected so as to correspond to the size of the captured image of the area.
The signal processing unit superimposes the translucent map on the captured image of the imaging area and displays it on the monitor.
Monitoring system. The monitoring system according to claim 1.
A plurality of threshold values regarding the loudness of the sound are provided.
The signal processing unit uses a sound source visual image in which the sound parameter is stepwise converted into a different visual image for each predetermined unit of the pixel according to the comparison between the sound parameter and the plurality of threshold values. Generate a translucent map of the sound parameter map with the type of sound source visual image.
Monitoring system. The monitoring system according to claim 1.
A masking area setting unit that sets a masking area for excluding the detection of an unmanned vehicle appearing in the captured image of the imaging area based on the sound picked up by the microphone array.
A detection unit that detects the unmanned vehicle based on the sound picked up by the microphone array and the masking area set by the masking area setting unit is further provided.
When the unmanned vehicle is detected outside the masking area, the signal processing unit indicates the loudness of the sound of the unmanned vehicle around the unmanned vehicle in the captured image of the imaging area. Make the sound source visual image semi-transparent and display it on the monitor.
Monitoring system. The monitoring system according to claim 2.
Further provided with a threshold value adjusting unit that changes the setting of the correspondence between each threshold value of a plurality of threshold values for stepwisely defining the loudness of the sound source and the plurality of types of the sound source visual images according to the captured image of the imaging area. ,
The signal processing unit captures a sound source visual image for each predetermined unit of the pixel in the imaging area based on the sound parameter derived by the sound parameter deriving unit and the corresponding relationship changed by the threshold value adjusting unit. Generate a translucent map of sound parameter maps linked to correspond to the size of the captured image,
Monitoring system. A monitoring method in a monitoring system having a camera and a microphone array.
The imaging area is imaged by the camera.
The microphone array collects the sound in the imaging area and collects the sound.
The captured image of the imaging area captured by the camera is displayed on the monitor, and the image is displayed on the monitor.
Based on the sound picked up by the microphone array, a sound parameter for specifying the loudness of the sound in the imaging area is derived for each predetermined unit of pixels constituting the image captured in the imaging area.
According to the comparison between the derived sound parameter and the threshold value related to the loudness, the sound source visual image obtained by converting the sound parameter into a visual image for each predetermined unit of the pixel is the size of the captured image in the imaging area. Generate a semi-transparent map of the sound parameter map concatenated to correspond to
The generated translucent map is superimposed on the captured image of the imaging area and displayed on the monitor.
Monitoring method.

Images