JP2017175475A - Monitoring system and monitoring method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To visually present to a user in what kind of place in an imaging area of a camera device an unmanned flying body is detected and what kind of sound source exists in what other places in the same imaging area, without deteriorating visibility of the captured image of the camera device.SOLUTION: A sound source direction detection section 34, on the basis of a sound collected by a microphone array, derives a sound pressure that specifies magnitude of sound in an imaging area for each predetermined unit of pixels constituting a captured image. An output control section 35, in accordance with comparison between the sound pressure and a plurality of threshold values relating to the magnitude of sound, superimposes sound source visual information obtained by gradually converting the sound pressure into different visual images for each predetermined unit of the pixels constituting the captured image in the imaging area and displays them in a first monitor. The sound source direction detection section, when any one of the sound source positions in the captured image in the imaging area is specified, derives the sound pressure for each value obtained by dividing a predetermined unit of pixels constituting a rectangular range including the sound source position by a size ratio between the rectangular range and the captured image in the imaging area.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a monitoring system and a monitoring method for monitoring an imaging area of a camera device in which an unmanned aerial vehicle is flying, for example.

  Conventionally, a flying object monitoring apparatus capable of detecting the presence of an object and detecting the flying direction of the object by using a plurality of sound detection units that detect sound generated in the monitoring area for each direction is known. (For example, refer to Patent Document 1). When the flying flight object monitoring apparatus detects the flight of a flying object and its flying direction by sound detection using a non-directional microphone, the monitoring camera is directed in the direction in which the flying object has come. Further, the processing device displays the video image taken by the monitoring camera on the display device.

JP 2006-168421 A

  In Patent Document 1, a monitoring camera capable of changing the shooting direction in an arbitrary direction within the monitoring area is provided. When a flying object such as a helicopter or a Cessna is detected, the shooting direction of the monitoring camera is changed. Has been disclosed. In other words, it has been disclosed to change the shooting direction of the surveillance camera in order to capture an image of a detected flying object.

  However, Patent Document 1 does not consider displaying a wide range of surrounding captured images including an unmanned aerial vehicle detected within the range of the angle of view of the camera device with respect to the imaging area. For this reason, the unmanned flying object is detected in the imaging area of the camera device, and what kind of sound source is present in other locations in the same imaging area is visually presented to the user. There was a problem that it was not possible.

  The present invention has been devised in view of the above-described conventional situation, and an unmanned air vehicle is detected at any location within the imaging area of the camera device, and any sound source at any other location within the same imaging area. It is an object of the present invention to provide a monitoring system and a monitoring method for visually presenting to a user whether or not they exist without degrading the visibility of a captured image of a camera device.

  The present invention provides a camera that captures an imaging area, a microphone array that collects sound in the imaging area, a monitor that displays a captured image of the imaging area captured by the camera, and a sound that is collected by the microphone array. A sound parameter deriving unit for deriving a sound parameter for specifying a loudness of the imaging area for each predetermined unit of a pixel constituting the captured image of the imaging area, and the sound parameter deriving unit A sound source visual image obtained by converting the sound parameter into a visual image for each predetermined unit of the pixel according to a comparison between the derived sound parameter and a threshold value related to a loudness is a size of a captured image of the imaging area. A signal processing unit that generates a semi-transparent map of sound parameter maps that are connected to correspond to each other, and the signal processing unit converts the translucent map into the semi-transparent map. Superimposed on the captured image of the image area displayed on the monitor, to provide a monitoring system.

  The present invention is also a monitoring method in a monitoring system having a camera and a microphone array, wherein an imaging area is imaged by the camera, sound of the imaging area is collected by the microphone array, and the camera is used. The captured image of the imaged area is displayed on a monitor, and based on the sound collected by the microphone array, a sound parameter for specifying the loudness of the imaged area, the captured image of the imaged area is displayed. A sound source visual image obtained by converting the sound parameter for each predetermined unit of the pixel into a visual image in accordance with a comparison between the derived sound parameter and a threshold value related to a loudness level. Generate a semi-transparent map of the sound parameter map that connects the images so as to correspond to the size of the captured image of the imaging area. Wherein the translucent map superimposed on the captured image of the imaging area displayed on the monitor, to provide a monitoring method.

  According to the present invention, the loudness of the sound at the sound source position can be presented in a fine stepwise manner regardless of the magnitude of the sound at the sound source position detected in the imaging area of the camera device. It can contribute to the accurate grasp to the user of size.

The figure which shows an example of schematic structure of the unmanned air vehicle detection system of each embodiment The figure which shows an example of the external appearance of a sound source detection unit Block diagram showing an example of the internal configuration of the microphone array in detail Block diagram showing an example of the internal configuration of an omnidirectional camera in detail Block diagram showing in detail an example of the internal configuration of a PTZ camera Block diagram showing an example of the internal configuration of the monitoring device in detail Timing chart showing an example of detection sound signal pattern of unmanned air vehicle registered in memory Timing chart showing an example of frequency change of detected sound signal obtained as a result of frequency analysis processing The sequence diagram explaining an example of the operation procedure of detection of an unmanned air vehicle and display of a detection result in a 1st embodiment. 9 is a flowchart for explaining an example of details of the operation procedure of unmanned air vehicle detection determination in step S8 of FIG. The figure which shows an example of a mode that a pointing direction is scanned in order within a monitoring area, and an unmanned air vehicle is detected The figure which shows the example of a display screen of the 1st monitor when the masking area is not set Explanatory drawing showing a display example of the masking area in time series during automatic learning processing The sequence diagram explaining an example of the operation | movement procedure of the setting of a masking area in 1st Embodiment The figure which shows the example of a display screen of the 1st monitor when a masking area is set Outline explanatory drawing of dynamic change of display resolution of sound pressure heat map in the second embodiment The flowchart explaining an example of the operation | movement procedure of the dynamic change of the display resolution of the sound pressure heat map in 2nd Embodiment. Outline explanatory drawing of the display result of the picked-up image accompanying the width adjustment of the threshold value according to the frequency distribution of the sound pressure value and the width adjustment in the second embodiment (A), (B) Outline explanatory diagram of setting change of width between thresholds for defining use of sound source visual image in second embodiment Outline explanatory drawing of display of picked-up image in connection with setting change of width between threshold values defining use of red image and group blue image in second embodiment The flowchart explaining an example of the operation | movement procedure of the setting change of the width | variety between threshold values in 2nd Embodiment. Outline explanatory drawing of overlay display of omnidirectional image and translucent sound pressure heat map in the third embodiment The figure which shows the example of a display screen of the 1st monitor on which the omnidirectional image and the translucent sound pressure heat map were displayed by overlay. The sequence diagram explaining an example of the operation | movement procedure of the overlay display of an omnidirectional image and a translucent sound pressure heat map in 3rd Embodiment

  Hereinafter, embodiments that specifically disclose a monitoring system and a monitoring method according to the present invention will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of already well-known matters and repeated descriptions for substantially the same configuration may be omitted. This is to avoid the following description from becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

  Hereinafter, as an example of the monitoring system according to the present invention, an unmanned air vehicle detection system for detecting an unmanned air vehicle to be monitored (for example, a drone, a radio control helicopter) will be described as an example. Moreover, this invention is not limited to a monitoring system, It can also be prescribed | regulated as a monitoring method performed in a monitoring system.

  Hereinafter, a user of an unmanned air vehicle detection system (for example, a monitor who looks around and guards a monitoring area) is simply referred to as a “user”.

  FIG. 1 is a diagram illustrating an example of a schematic configuration of an unmanned air vehicle detection system 5 according to each embodiment. The unmanned aerial vehicle detection system 5 detects an unmanned aerial vehicle dn (for example, see FIG. 12) as a user's target as a detection target. The unmanned air vehicle dn is, for example, a drone that flies autonomously using a GPS (Global Positioning System) function, a radio controlled helicopter that flies by a third party's radio control, and the like. Such an unmanned air vehicle dn is used, for example, for aerial photography of a target, monitoring, transportation of goods, and the like.

  In each embodiment, a multicopter type drone equipped with a plurality of rotors (in other words, rotor blades) is exemplified as the unmanned air vehicle dn. In a multi-copter type drone, generally, when the number of rotor wings is two, a harmonic having a frequency twice as high as a specific frequency and further a harmonic having a frequency multiplied by that frequency is generated. Similarly, when the number of rotor wings is three, harmonics having a frequency three times as high as a specific frequency and further harmonics having a frequency multiplied by the harmonics are generated. The same applies when the number of rotor wings is four or more.

  The unmanned air vehicle detection system 5 includes a plurality of sound source detection units UD1, ..., UDk, ..., UDn, a monitoring device 10, a first monitor MN1, a second monitor MN2, and a recorder RC. n is a natural number of 2 or more. The plurality of sound source detection units UD1, ..., UDk, ..., UDn are connected to the monitoring apparatus 10 via the network NW. k is a natural number of 1 to n. Each sound source detection unit (for example, the sound source detection unit UD1) has a configuration including a microphone array MA1, an omnidirectional camera CA1, and a PTZ camera CZ1, and the other sound source detection units UDk also have the same configuration.

  Note that the sound source detection unit UDk or simply the sound source detection unit UD is referred to unless the individual sound source detection units need to be particularly distinguished. Similarly, the microphone array MAk or microphone array MA (see FIG. 2), the omnidirectional camera CAk or the omnidirectional camera CA (FIG. 2) unless there is a particular need to distinguish between individual microphone arrays, omnidirectional cameras, and PTZ cameras. See PTZ camera CZk or PTZ camera CZ (see FIG. 2).

  In the sound source detection unit UDk, the microphone array MAk collects sound in all directions in the area to be monitored (that is, the monitoring area) where the apparatus is installed in a non-directional state. The microphone array MAk has a housing 15 (see FIG. 2) in which a cylindrical opening having a predetermined width is formed at the center. The sound to be picked up by the microphone array MAk includes, for example, a mechanical operation sound such as a drone, a sound emitted by a human being, and other sounds, and is limited to a sound in an audible frequency range (that is, 20 Hz to 23 kHz). Alternatively, a low frequency sound lower than the audible frequency or an ultrasonic sound exceeding the audible frequency may be included.

  Microphone array MAk includes a plurality of omnidirectional microphones M1 to Mq (see FIG. 3). q is a natural number of 2 or more. The microphones M <b> 1 to Mq are arranged around the opening provided in the housing 15 at a predetermined interval (for example, a uniform interval) concentrically along the circumferential direction. As the microphones M1 to Mq, for example, electret condenser microphones (ECM) are used. The microphone array MAk transmits sound data of sounds obtained by collecting the sounds of the microphones M1 to Mq to the monitoring device 10 via the network NW. The arrangement of the microphones M1 to Mq is an example, and other arrangements (for example, a square arrangement or a rectangular arrangement) may be used. However, the microphones M1 to Mq are arranged at equal intervals. It is preferred that

  The microphone array MAk includes a plurality of microphones M1 to Mq (for example, q = 32) and a plurality of amplifiers (amplifiers) PA1 to PAq (see FIG. 3) that amplify output signals of the plurality of microphones M1 to Mq, respectively. . Analog signals output from each amplifier are converted into digital signals by A / D converters A1 to Aq (see FIG. 3), which will be described later. Note that the number of microphones in the microphone array MAk is not limited to 32, but may be other numbers (for example, 16, 64, 128).

  An omnidirectional camera CAk that substantially matches the volume of the opening is accommodated inside the opening formed in the center of the casing 15 (see FIG. 2) of the microphone array MAk. That is, the microphone array MAk and the omnidirectional camera CAk are integrated and arranged so that the center of each case is in the coaxial direction (see FIG. 2). The omnidirectional camera CAk is a camera equipped with a fisheye lens 45a (see FIG. 4) capable of capturing an omnidirectional image of a monitoring area (see above) as an imaging area of the omnidirectional camera CAk. In each embodiment, the sound collection area of the microphone array MAk and the imaging area of the omnidirectional camera CAk are both described as a common monitoring area, but the spatial size (for example, volume) of the sound collection area and the imaging area is the same. Not necessarily. For example, the volume of the sound collection area may be larger or smaller than the volume of the imaging area. In short, the sound collection area and the imaging area only need to have a common space. The omnidirectional camera CAk functions as a monitoring camera capable of imaging an imaging area where the sound source detection unit UDk is installed, for example. That is, the omnidirectional camera CAk has an angle of view of, for example, vertical direction: 180 ° and horizontal direction: 360 °, and images the monitoring area 8 (see FIG. 11), which is a hemisphere, for example, as an imaging area.

  In each sound source detection unit UDk, the omnidirectional camera CAk and the microphone array MA are arranged coaxially by fitting the omnidirectional camera CAk inside the opening of the housing 15. As described above, the optical axis of the omnidirectional camera CAk and the central axis of the housing of the microphone array MAk coincide with each other, so that the imaging area and the sound collection area in the axial circumferential direction (that is, the horizontal direction) become substantially the same. The position of the subject in the omnidirectional image captured by the omnidirectional camera CAk (in other words, the direction indicating the position of the subject as viewed from the omnidirectional camera CAk) and the position of the sound source to be collected by the microphone array MAk (in other words, Can be expressed in the same coordinate system (for example, coordinates indicated by (horizontal angle, vertical angle)). Each sound source detection unit UDk is attached so that, for example, the upward direction in the top-and-bottom direction is the sound collection surface and the imaging surface in order to detect the unmanned air vehicle dn flying in the sky (see FIG. 2). .

  The monitoring device 10 is configured using, for example, a PC (Personal Computer) or a server. The monitoring apparatus 10 forms directivity with the arbitrary beam direction as the main beam direction (that is, beam forming) based on the user's operation with respect to the omnidirectional sound collected by the microphone array MAk. The sound of direction can be emphasized. In addition, the technique regarding the directivity control processing of the sound data for beam forming the sound collected by the microphone array MAk is a known technique as shown in Reference Patent Documents 1 and 2, for example.

(Reference Patent Document 1) JP-A-2014-143678 (Reference Patent Document 2) JP-A-2015-029241

  The monitoring device 10 generates an omnidirectional image by processing the captured image using an image captured by the omnidirectional camera CAk (hereinafter, sometimes abbreviated as “captured image”). The omnidirectional image may be generated not by the monitoring device 10 but by the omnidirectional camera CAk.

  The monitoring apparatus 10 includes a sound pressure heat map based on a calculated value of a sound parameter (for example, a sound pressure described later) that specifies the magnitude of sound collected by the microphone array MAk in a captured image captured by the omnidirectional camera CAk. The image (see FIG. 15) is superimposed and output to the first monitor MN1 or the like and displayed.

  In addition, the monitoring apparatus 10 displays a visual image (for example, an identification mark (not shown)) that is easy for the user to visually identify the detected unmanned air vehicle dn in the omnidirectional image IMG1, and the unmanned air vehicle dn of the first monitor MN1. It may be displayed at the position. The visual image means, for example, information that is represented in such a way that when the user views the omnidirectional image IMG1 in the omnidirectional image IMG1, it can be clearly distinguished from other subjects. .

  The first monitor MN1 displays an omnidirectional image IMG1 captured by the omnidirectional camera CAk. The second monitor MN2 displays an omnidirectional image IMG1 captured by the omnidirectional camera CAk. The first monitor MN1 generates and displays a composite image in which the above-described identification mark (not shown) is superimposed on the omnidirectional image IMG1. In FIG. 1, the first monitor MN1 and the second monitor MN2 and the two monitors are connected to the monitoring device 10, but only the first monitor MN1 may be connected, for example. Further, the first monitor MN1 or the second monitor MN2, or the first monitor MN1 and the second monitor MN2 may be configured as an apparatus integrated with the monitoring apparatus 10.

  The recorder RC is configured using, for example, a semiconductor memory such as a hard disk (Hard Disk Drive) or a flash memory, and is transmitted from various image data (see below) generated by the monitoring device 10 and from each sound source detection unit UDk. Save various omnidirectional images and audio data. Note that the recorder RC may be configured as an apparatus integrated with the monitoring device 10 or may be omitted from the configuration of the unmanned air vehicle detection system 5.

  In FIG. 1, the plurality of sound source detection units UDk and the monitoring device 10 have a communication interface and are connected to each other via a network NW so that data communication is possible. The network NW may be a wired network (for example, an intranet, the Internet, a wired LAN (Local Area Network), or a wireless network (for example, a wireless LAN) Note that the sound source detection unit UDk and the monitoring device 10 are connected via the network NW. In addition, the monitoring device 10, the first monitor MN1, the second monitor MN2, and the recorder RC are all installed in the monitoring room RM where the user resides during monitoring.

  FIG. 2 is a diagram illustrating an example of the appearance of the sound source detection unit UD. The sound source detection unit UD includes the above-described microphone array MA, omnidirectional camera CA, and PTZ camera CZ, and a support base 70 that mechanically supports them. The support base 70 includes a tripod 71, two rails 72 fixed to the top plate 71 a of the tripod 71, and a first mounting plate 73 and a second mounting plate 74 that are respectively attached to both ends of the two rails 72. And has a combined structure.

  The first attachment plate 73 and the second attachment plate 74 are attached so as to straddle the two rails 72 and have substantially the same plane. Further, the first mounting plate 73 and the second mounting plate 74 are slidable on the two rails 72, and are adjusted and fixed at positions separated or approached from each other.

  The first mounting plate 73 is a disk-shaped plate material. An opening 73 a is formed at the center of the first mounting plate 73. A housing 15 of the microphone array MA is accommodated and fixed in the opening 73a. On the other hand, the second mounting plate 74 is a substantially rectangular plate material. An opening 74 a is formed in a portion near the outside of the second mounting plate 74. The PTZ camera CZ is accommodated and fixed in the opening 74a.

  As shown in FIG. 2, the optical axis L1 of the omnidirectional camera CA housed in the casing 15 of the microphone array MA and the optical axis L2 of the PTZ camera CZ attached to the second mounting plate 74 are in the initial installation state. Are set to be parallel to each other.

  The tripod 71 is supported on the grounding surface by three legs 71b, and the position of the top plate 71a can be moved in the vertical direction with respect to the grounding surface by manual operation, and the top and bottom can be moved in the pan and tilt directions. The direction of the plate 71a can be adjusted. Thereby, the sound collection area of the microphone array MA (in other words, the imaging area of the omnidirectional camera CA or the monitoring area of the unmanned air vehicle detection system 5) can be set in an arbitrary direction.

  FIG. 3 is a block diagram showing in detail an example of the internal configuration of the microphone array MAk. The microphone array MAk shown in FIG. 3 includes a plurality of microphones M1 to Mq (for example, q = 32), a plurality of amplifiers (amplifiers) PA1 to PAq that amplify output signals of the plurality of microphones M1 to Mq, and amplifiers PA1 to PAq. Are configured to include a plurality of A / D converters A1 to Aq that convert each analog signal output from the digital signal into a digital signal, a compression processing unit 25, and a transmission unit 26.

  The compression processing unit 25 generates audio data packets based on the digital audio signals output from the A / D converters A1 to An. The transmission unit 26 transmits the audio data packet generated by the compression processing unit 25 to the monitoring apparatus 10 via the network NW.

  Thus, the microphone array MAk amplifies the output signals of the microphones M1 to Mq with the amplifiers PA1 to PAq, and converts them into digital audio signals with the A / D converters A1 to Aq. Thereafter, the microphone array MAk generates a voice data packet in the compression processing unit 25 and transmits the voice data packet to the monitoring apparatus 10 via the network NW.

  FIG. 4 is a block diagram showing in detail an example of the internal configuration of the omnidirectional camera CAk. The omnidirectional camera CAk shown in FIG. 4 includes a CPU 41, a communication unit 42, a power management unit 44, an image sensor 45, a fisheye lens 45a, a memory 46, and a network connector 47.

  The CPU 41 performs signal processing for overall control of operations of each unit of the omnidirectional camera CAk, data input / output processing with other units, data calculation processing, and data storage processing. Instead of the CPU 41, a processor such as an MPU (Micro Processing Unit) or a DSP (Digital Signal Processor) may be provided.

  For example, the CPU 41 generates two-dimensional panoramic image data (that is, image data obtained by two-dimensional panorama conversion) obtained by cutting out an image in a specific range (direction) from the omnidirectional image data according to the designation of the user who operates the monitoring device 10. And stored in the memory 46.

  The image sensor 45 is configured using, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor, and captures an optical image of subject light from the imaging area collected by the fisheye lens 45a on the light receiving surface. By processing, omnidirectional image data of the imaging area is acquired.

  The fish-eye lens 45a receives and collects subject light from all directions of the imaging area, and forms an optical image of the subject light on a light receiving surface (not shown) of the image sensor 45.

  The memory 46 stores a ROM 46z in which a program for defining the operation of the omnidirectional camera CAk and data of setting values are stored, and omnidirectional image data or cutout image data or work data obtained by cutting out a part of the range. And a memory card 46x that is detachably connected to the omnidirectional camera CAk and stores various data.

  The communication unit 42 is a network interface (I / F) that controls data communication with the network NW connected via the network connector 47.

  The power management unit 44 supplies DC power to each unit of the omnidirectional camera CAk. Further, the power management unit 44 may supply DC power to devices connected to the network NW via the network connector 47.

  The network connector 47 is a connector capable of transmitting omnidirectional image data or two-dimensional panoramic image data to the monitoring apparatus 10 via the network NW and supplying power via the network cable.

  FIG. 5 is a block diagram showing in detail an example of the internal configuration of the PTZ camera CZk. About each part similar to omnidirectional camera CAk, the code | symbol corresponding to each part of FIG. 4 is attached | subjected, and the description is abbreviate | omitted. The PTZ camera CZk is a camera that can adjust the optical axis direction (sometimes referred to as an imaging direction) in accordance with a view angle change instruction from the monitoring device 10.

  Similar to the omnidirectional camera CAk, the PTZ camera CZk includes a CPU 51, a communication unit 52, a power management unit 54, an image sensor 55, an imaging lens 55a, a memory 56, and a network connector 57, an imaging direction control unit 58, and a lens drive motor. 59. When there is a view angle change instruction of the monitoring device 10, the CPU 51 notifies the imaging direction control unit 58 of the view angle change instruction.

  The imaging direction control unit 58 controls the imaging direction of the PTZ camera CZ in at least one of the pan direction and the tilt direction according to the view angle change instruction notified from the CPU 51, and further changes the zoom magnification as necessary. The control signal for output is output to the lens drive motor 59. The lens drive motor 59 drives the imaging lens 55a according to this control signal, changes its imaging direction (direction of the optical axis L2 shown in FIG. 2), and adjusts the focal length of the imaging lens 55a to increase the zoom magnification. change.

  The imaging lens 55a is configured using one or two or more lenses. In the imaging lens 55a, the optical axis directions of pan rotation and tilt rotation are changed by driving the lens driving motor 59 according to the control signal from the imaging direction control unit 58.

  FIG. 6 is a block diagram illustrating an example of the internal configuration of the monitoring apparatus 10 in detail. 6 has a configuration including at least a communication unit 31, an operation unit 32, a signal processing unit 33, a speaker device (SPK) 37, a memory 38, and a setting management unit 39.

  The communication unit 31 receives omnidirectional image data or two-dimensional panoramic image data transmitted from the omnidirectional camera CAk and audio data transmitted from the microphone array MAk and outputs the received data to the signal processing unit 33.

  The operation unit 32 is a user interface (UI) for notifying the signal processing unit 33 of the content of a user input operation, and is configured by a pointing device such as a mouse or a keyboard. Further, the operation unit 32 is arranged using, for example, a touch panel or a touch pad that is arranged corresponding to each screen of the first monitor MN1 and the second monitor MN2 and can be directly input by a user's finger or stylus pen. Also good.

  The operation unit 32 displays a red region of a sound pressure heat map (see FIG. 15) displayed on the first monitor MN1 or the second monitor MN2 so as to be superimposed on the captured image (omnidirectional image IMG1) of any omnidirectional camera CAk. When RD1 is designated by the user, coordinate data indicating the designated position is acquired and output to the signal processing unit 33. The signal processing unit 33 reads out the sound data collected by the microphone array MAk corresponding to the omnidirectional camera CAk from the memory 38 and directs the sound data from the microphone array MAk toward the actual sound source position corresponding to the designated position. Output from the speaker device 37. Thus, the user can clearly check not only the unmanned air vehicle dn but also the sound at the position specified by the user on the captured image (omnidirectional image IMG1) is emphasized.

  The signal processing unit 33 is configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor), and performs control processing for controlling the operation of each unit of the monitoring apparatus 10 in an integrated manner. Data input / output processing, data calculation (calculation) processing, and data storage processing are performed with respect to other units. The signal processing unit 33 includes a sound source direction detection unit 34, an output control unit 35, a directivity processing unit 63, a frequency analysis unit 64, an object detection unit 65, a detection result determination unit 66, a scanning control unit 67, and a detection direction control unit 68. , A masking area setting unit 69a and a threshold adjustment unit 69b. The monitoring device 10 is connected to the first monitor MN1 and the second monitor MN2.

  The sound source direction detection unit 34 uses, for example, the sound data of the sound in the monitoring area 8 collected by the microphone array MAk according to a known whitening cross-correlation method (CSP (Cross-power Spectrum Phase analysis) method). Is estimated. In the CSP method, the sound source direction detection unit 34 divides the monitoring area 8 shown in FIG. 11 into a plurality of blocks, and when sound is collected by the microphone array MAk, the sound pressure or sound volume exceeds a threshold value for each block. By determining whether there is sound, the position of the sound source in the monitoring area 8 can be roughly estimated.

  The sound source direction detection unit 34 as a sound parameter derivation unit is also based on the omnidirectional image data captured by the omnidirectional camera CAk and the sound data collected by the microphone array MAk, and the omnidirectional image of the monitoring area 8. The sound pressure as the sound parameter is calculated for each pixel constituting the data. The sound source direction detection unit 34 outputs a calculated value that is a calculation result of the sound pressure to the output control unit 35. This sound pressure calculation process is a known technique, and a detailed description of the process is omitted.

  The setting management unit 39 has a coordinate conversion formula related to the coordinate conversion of the position specified by the user on the screen of the first monitor MN1 on which the omnidirectional image data captured by the omnidirectional camera CAk is displayed. This coordinate conversion formula is specified by the user on the omnidirectional image data based on the physical distance difference between the installation position of the omnidirectional camera CAk (see FIG. 2) and the installation position of the PTZ camera CZk (see FIG. 2), for example. This is a mathematical formula for converting the coordinates of the position (that is, (horizontal angle, vertical angle)) into the coordinates in the direction viewed from the PTZ camera CZk.

  The signal processing unit 33 uses the coordinate conversion formula held by the setting management unit 39 to determine the position designated by the user from the installation position of the PTZ camera CZk with reference to the installation position of the PTZ camera CZk (see FIG. 2). The coordinates (θMAh, θMAv) indicating the directivity direction toward the actual sound source position corresponding to is calculated. θMAh is a horizontal angle in a direction toward the actual sound source position corresponding to the position specified by the user as viewed from the installation position of the PTZ camera CZk. θMAv is a vertical angle in a direction toward the actual sound source position corresponding to the position specified by the user as viewed from the installation position of the PTZ camera CZk. As shown in FIG. 2, since the distance between the omnidirectional camera CAk and the PTZ camera CZk is known and the optical axes L1 and L2 are parallel to each other, the calculation processing of the coordinate conversion formula is, for example, It can be realized by geometric calculation. The sound source position is an actual sound source position corresponding to the position designated from the operation unit 32 by the operation of the user's finger or the stylus pen with respect to the video data displayed on the first monitor MN1 and the second monitor MN2.

  As shown in FIG. 2, in each embodiment, the omnidirectional camera CAk and the microphone array MAk are arranged so that the optical axis direction of the omnidirectional camera CAk and the central axis of the casing of the microphone array MAk are coaxial. Has been. For this reason, the coordinates of the user-specified position derived by the omnidirectional camera CAk in accordance with the user's specification for the first monitor MN1 on which the omnidirectional image data is displayed are the sound emphasizing direction viewed from the microphone array MAk (both the directional direction). Can be regarded as the same. In other words, when the user designates the first monitor MN1 (which may be the second monitor MN2) on which the omnidirectional image data is displayed, the monitoring apparatus 10 sets all the coordinates of the designated position on the omnidirectional image data. It transmits to the direction camera CAk. As a result, the omnidirectional camera CAk uses the coordinates of the designated position transmitted from the monitoring device 10, and the coordinates (horizontal angle, vertical) indicating the direction of the sound source position corresponding to the user designated position as seen from the omnidirectional camera CAk. Corner). Since the calculation process in the omnidirectional camera CAk is a known technique, the description thereof is omitted. The omnidirectional camera CAk transmits a calculation result of coordinates indicating the direction of the sound source position to the monitoring device 10. The monitoring apparatus 10 can use the coordinates (horizontal angle and vertical angle) calculated by the omnidirectional camera CAk as coordinates (horizontal angle and vertical angle) indicating the direction of the sound source position as viewed from the microphone array MAk.

  However, when the omnidirectional camera CAk and the microphone array MAk are not coaxially arranged, the setting management unit 39 derives the omnidirectional camera CAk according to a method described in, for example, Japanese Patent Laid-Open No. 2015-029241. It is necessary to convert the coordinates thus obtained into coordinates in the direction viewed from the microphone array MAk.

  In addition, the setting management unit 39 compares the first threshold th1, the second threshold th2, and the sound pressure p for each pixel constituting the omnidirectional image data or the two-dimensional panoramic image data calculated by the sound source direction detection unit 34. A third threshold th3 (see, for example, FIG. 9) is held. Here, the sound pressure p is used as an example of a sound parameter relating to the sound source, represents the loudness of sound collected by the microphone array MA, and represents the loudness output from the speaker device 37. It is distinguished from the volume. The first threshold th1, the second threshold th2, and the third threshold th3 are values that are compared with the sound pressure of the sound generated in the monitoring area 8, and are, for example, predetermined values for determining the sound emitted by the unmanned air vehicle dn. Set to A plurality of threshold values can be set in addition to the first threshold value th1, the second threshold value th2, and the third threshold value th3 described above. For the sake of simple explanation, for example, the first threshold value th1 and a larger value are set. Are set, and a third threshold value th3, which is a larger value (first threshold value th1 <second threshold value th2 <third threshold value th3).

  As will be described later, in the sound pressure heat map generated by the output control unit 35, the omnidirectional image data is stored in the red region RD1 (see FIG. 15) of the pixel in which the sound pressure greater than the third threshold th3 is obtained. On the displayed first monitor MN1, for example, it is drawn in red. Further, the pink area PD1 of the pixel in which the sound pressure greater than the second threshold th2 and less than or equal to the third threshold th3 is obtained is drawn, for example, in pink on the first monitor MN1 on which the omnidirectional image data is displayed. A blue region BD1 of a pixel in which a sound pressure greater than the first threshold th1 and less than or equal to the second threshold th2 is obtained is rendered, for example, in blue on the first monitor MN1 on which the omnidirectional image data is displayed. In addition, the sound pressure region N1 of the pixel equal to or less than the first threshold th1 is drawn, for example, in a colorless manner on the first monitor MN1 on which the omnidirectional image data is displayed, that is, the display color of the omnidirectional image data is not different. .

  The speaker device 37 outputs sound data picked up by the microphone array MAk or sound data picked up by the microphone array MAk and having directivity formed by the signal processing unit 33. Note that the speaker device 37 may be configured as a separate device from the monitoring device 10.

  The memory 38 is configured using, for example, a ROM or a RAM, and holds various data including sound data of a certain section, setting information, a program, and the like. The memory 38 has a pattern memory (see FIG. 7) in which a sound pattern unique to each unmanned air vehicle dn is registered. Further, the memory 38 stores sound pressure heat map data generated by the output control unit 35. In the memory 38, an identification mark (not shown) schematically representing the position of the unmanned air vehicle dn is registered. The identification mark used here is, for example, a star symbol. The identification mark is not limited to a star shape, but may be a circle or a rectangle, or a symbol or character such as a “卍” shape reminiscent of an unmanned air vehicle dn. Further, the display mode of the identification mark may be changed between daytime and nighttime. For example, it may have a star shape during the daytime and a quadrangle that is not mistaken for a star at nighttime. Further, the identification mark may be changed dynamically. For example, a star symbol may be blinked or rotated, and the user can be further alerted.

  FIG. 7 is a timing chart showing an example of a detection sound pattern of the unmanned air vehicle dn registered in the memory 38. The detected sound pattern shown in FIG. 7 is a combination of frequency patterns, and sounds of four frequencies f1, f2, f3, and f4 generated by the rotation of four rotors mounted on the multicopter type unmanned air vehicle dn. Including. The signals of the respective frequencies are signals having different sound frequencies that are generated, for example, with the rotation of a plurality of wings pivotally supported by each rotor.

  In FIG. 7, the frequency region indicated by the oblique lines is a region where the sound pressure is high. The detected sound pattern may include not only the number of sounds having a plurality of frequencies and the sound pressure but also other sound information. For example, a sound pressure ratio representing a ratio of sound pressures at each frequency can be used. Here, as an example, detection of the unmanned air vehicle dn is determined based on whether or not the sound pressure of each frequency included in the detection sound pattern exceeds a threshold value.

  The directivity processing unit 63 uses the sound signals (also referred to as sound data) collected by the non-directional microphones M1 to Mq and the setting result of the masking area setting unit 69a, and uses the directivity forming process (beam forming) described above. The sound data is extracted with the direction of the area other than the masking area set by the masking area setting unit 69a as the pointing direction. The directivity processing unit 63 can also perform sound data extraction processing in which the direction range of other areas excluding the masking area set by the masking area setting unit 69a is set as the directivity range. Here, the directivity range is a range including a plurality of adjacent directivity directions, and is intended to include a certain extent of directivity direction compared to the directivity direction.

  The frequency analysis unit 64 performs frequency analysis processing on the sound data extracted in the directivity direction by the directivity processing unit 63. In this frequency analysis process, the frequency and the sound pressure included in the sound data in the directivity direction are detected.

  FIG. 8 is a timing chart showing an example of the frequency change of the detected sound signal obtained as a result of the frequency analysis processing. In FIG. 8, four frequencies f11, f12, f13, f14 and sound pressures at the respective frequencies are obtained as detected sound signals (that is, detected sound data). In the figure, the fluctuation of each frequency that changes irregularly occurs, for example, due to the rotational fluctuation of the rotor (rotary blade) that slightly changes when the unmanned aerial vehicle dn controls the attitude of its own aircraft.

  The object detection unit 65 as the detection unit performs detection processing of the unmanned air vehicle dn using the frequency analysis processing result of the frequency analysis unit 64. Specifically, in the detection process of the unmanned air vehicle dn, the object detection unit 65 detects the detection sound obtained as a result of the frequency analysis process in other areas excluding the masking area set by the masking area setting unit 69a. (See FIG. 8) (frequency f11 to f14) and the detection sound pattern (see FIG. 7) (frequency f1 to f4) registered in advance in the pattern memory of the memory 38 are compared. The object detection unit 65 determines whether or not the patterns of both detection sounds are approximate.

  Whether or not both patterns are approximated is determined, for example, as follows. Of the four frequencies f1, f2, f3, and f4, if the sound pressures of at least two frequencies included in the detected sound data exceed the threshold values, the object detection unit 65 determines that the sound pattern is approximated, The flying object dn is detected. The unmanned air vehicle dn may be detected when other conditions are satisfied.

  When it is determined that the unmanned air vehicle dn does not exist, the detection result determination unit 66 instructs the detection direction control unit 68 to shift to detection of the unmanned air vehicle dn in the next pointing direction. The detection result determination unit 66 notifies the output control unit 35 of the detection result of the unmanned air vehicle dn when it is determined that the unmanned air vehicle dn exists as a result of the scanning in the pointing direction. The detection result includes information on the detected unmanned air vehicle dn. The information of the unmanned air vehicle dn includes, for example, identification information of the unmanned air vehicle dn and position information (for example, direction information) of the unmanned air vehicle dn in the sound collection space.

  The detection direction control unit 68 controls the direction for detecting the unmanned air vehicle dn in the sound collection space based on an instruction from the detection result determination unit 66. For example, the detection direction control unit 68 detects an arbitrary direction of the directivity range BF1 (see FIG. 11) including the sound source position estimated by the sound source direction detection unit 34 in the entire sound collection space (that is, the monitoring area 8). Set as direction.

  The scanning control unit 67 instructs the directivity processing unit 63 to perform beam forming with the detection direction set by the detection direction control unit 68 as the directivity direction.

  The directivity processing unit 63 performs beam forming with respect to the directivity direction designated by the scanning control unit 67. In the initial setting, the directivity processing unit 63 sets the initial position in the directivity range BF1 (see FIG. 11) including the sound source position estimated by the sound source direction detection unit 34 as the directivity direction BF2. The directivity direction BF2 is sequentially set by the detection direction control unit 68 within the directivity range BF1.

  The masking area setting unit 69a is based on the omnidirectional image data or two-dimensional panoramic image data of the monitoring area 8 captured by the omnidirectional camera CAk and the audio data of the monitoring area 8 collected by the microphone array MAk. A masking area for excluding detection of the unmanned air vehicle dn appearing in the orientation image or the two-dimensional panoramic image (that is, the captured image) is set. Details of the setting of the masking area will be described later with reference to FIGS.

  The output control unit 35 controls the operations of the first monitor MN1, the second monitor MN2, and the speaker device 37, and the omnidirectional image data or the two-dimensional panoramic image data transmitted from the omnidirectional camera CAk is transmitted to the first monitor MN1. , Output to the second monitor MN2 for display, and further output the audio data transmitted from the microphone array MA to the speaker device 37 as audio. In addition, when the unmanned air vehicle dn is detected, the output control unit 35 displays the identification mark (not shown) representing the unmanned air vehicle dn in a superimposed manner on the omnidirectional image and displays the first monitor MN1 (first 2 monitor MN2).

  In addition, the output control unit 35 uses the audio data collected by the microphone array MAk and the coordinates indicating the direction of the sound source position derived by the omnidirectional camera CAk to output the sound data collected by the microphone array MAk. By performing directivity formation processing, sound data in the directivity direction is emphasized. The directivity forming process for audio data is a known technique described in, for example, Japanese Patent Application Laid-Open No. 2015-029241.

  Further, the output control unit 35 uses the sound pressure value for each pixel constituting the omnidirectional image data or the two-dimensional panoramic image data calculated by the sound source direction detection unit 34 to use the omnidirectional image data or the two-dimensional panoramic image data. A sound pressure map in which a calculated value of sound pressure is assigned to the position of the corresponding pixel is generated for each pixel that constitutes. Furthermore, the output control unit 35 performs a color conversion process on the sound pressure value for each pixel of the generated sound pressure map to a visual image (for example, a colored image) so that the user can visually and easily discriminate. Then, a sound pressure heat map as shown in FIG. 15 is generated.

  The output control unit 35 has been described as generating a sound pressure map or a sound pressure heat map in which the sound pressure value calculated in units of pixels is assigned to the position of the corresponding pixel, but the sound pressure is calculated for each pixel. Without calculating, an average value of sound pressure values is calculated for each pixel block composed of a predetermined number (for example, 2 × 2, 4 × 4) of pixels, and an average of the sound pressure values corresponding to the corresponding predetermined number of pixels is calculated. A sound pressure map or a sound pressure heat map may be generated by assigning values.

  The details of the threshold adjustment unit 69b will be described in a second embodiment to be described later, and detailed description thereof is omitted here.

  Next, operation | movement of the unmanned air vehicle detection system 5 of this embodiment is demonstrated in detail.

  FIG. 9 is a sequence diagram for explaining an example of the operation of detecting the unmanned air vehicle dn and displaying the detection result in the first embodiment. When the power of each device (for example, the first monitor MN1, the monitoring device 10, the omnidirectional camera CAk, and the microphone array MAk) of the unmanned air vehicle detection system 5 is turned on, the unmanned air vehicle detection system 5 starts its operation. Further, as a premise of the description of FIG. 9, it is assumed that a masking area for excluding detection of the unmanned air vehicle dn is already set, and information indicating the masking area is registered in the memory 38.

  In the initial operation, the monitoring device 10 issues an image distribution request to the omnidirectional camera CAk (S1). The omnidirectional camera CAk starts imaging processing according to power-on in accordance with this image distribution request. Moreover, the monitoring apparatus 10 performs an audio | voice delivery request | requirement with respect to microphone array MAk (S2). The microphone array MA starts sound collection processing in response to power-on in accordance with the voice distribution request.

  When the initial operation ends, the omnidirectional camera CAk transmits data of omnidirectional images (for example, still images and moving images) obtained by imaging to the monitoring apparatus 10 via the network NW (S3). In FIG. 9, for the sake of simplicity, it is described that omnidirectional image data is transmitted from the omnidirectional camera CAk. However, two-dimensional panoramic image data may be transmitted, and the same applies to FIG. It is. The monitoring device 10 converts the omnidirectional image data transmitted from the omnidirectional camera CAk into display data such as NTSC, and outputs it to the first monitor MN1 to instruct the display of the omnidirectional image data (S4). When the display data transmitted from the monitoring apparatus 10 is input, the first monitor MN1 displays the omnidirectional image IMG1 data (see FIGS. 12 and 15) by the omnidirectional camera CAk on the screen.

  Further, the microphone array MAk encodes the audio data of the monitoring area 8 obtained by sound collection via the network NW and transmits the encoded audio data to the monitoring device 10 (S5). In the monitoring device 10, the sound source direction detection unit 34 configures omnidirectional image data in the monitoring area 8 based on the omnidirectional image data captured by the omnidirectional camera CAk and the audio data collected by the microphone array MAk. The sound pressure as the sound parameter is calculated for each pixel to be used, and the sound source position in the monitoring area 8 is estimated (S6). This estimated sound source position is used as a reference position of the directivity range BF1 necessary for setting the initial directivity direction when the monitoring apparatus 10 detects the unmanned air vehicle dn.

  Further, in the monitoring device 10, the output control unit 35 uses the sound pressure value for each pixel that configures the omnidirectional image data calculated by the sound source direction detection unit 34, to configure each omnidirectional image data. For each pixel, a sound pressure map in which a calculated value of sound pressure is assigned to the position of the corresponding pixel is generated. Furthermore, the output control unit 35 performs a color conversion process on the sound pressure value for each pixel of the generated sound pressure map to a visual image (for example, a colored image) so that the user can visually and easily discriminate. Then, a sound pressure heat map as shown in FIG. 15 is generated (S7).

  Further, in the monitoring device 10, the signal processing unit 33 uses the audio data transmitted from the microphone array MAk in step S5 to sequentially provide directivity to the outside of the masking area set by the masking area setting unit 69a. By forming, detection determination of the unmanned air vehicle dn is performed for each directivity direction in which directivity is formed (S8). Details of the detection determination process for the unmanned air vehicle dn will be described later with reference to FIGS. 10 and 11.

  When the unmanned air vehicle dn is detected as a result of the detection determination process, the output control unit 35 in the monitoring device 10 generates the sound pressure generated in step S7 on the omnidirectional image IMG1 displayed on the screen of the first monitor MN1. An instruction is given to superimpose and display the heat map and an identification mark (not shown) representing the unmanned air vehicle dn present in the pointing direction detected in step S8 (S9).

  The first monitor MN1 synthesizes (superimposes) and displays a sound pressure heat map on the omnidirectional image IMG1 in accordance with an instruction from the monitoring device 10 and synthesizes an identification mark (not shown) representing the unmanned air vehicle dn (not shown). (S10). Thereafter, the process of the unmanned air vehicle detection system 5 returns to step S3, and the processes of steps S3 to S10 are repeated until a predetermined event such as a power-off operation is detected.

  FIG. 10 is a flowchart for explaining an example of details of the operation procedure of the unmanned air vehicle detection determination in step S8 of FIG. In the sound source detection unit UDk, the directivity processing unit 63 uses the masking area information set by the masking area setting unit 69a and is based on the sound source position that is outside the masking area and estimated by the sound source direction detection unit 34. The directivity range BF1 is set as the initial position of the directivity direction BF2 (S21). The masking area information is coordinates in a direction facing the masking area as viewed from the microphone array MAk.

  FIG. 11 is a diagram illustrating an example of a state in which the pointing direction BF2 is sequentially scanned in the monitoring area 8 and the unmanned air vehicle dn is detected. Note that the initial position may be outside the masking area set by the masking area setting unit 69a, and may not be limited to the directivity range BF1 based on the sound source position of the monitoring area 8 estimated by the sound source direction detecting unit 34. . That is, if it is outside the masking area set by the masking area setting unit 69a, the monitoring area 8 may be sequentially scanned by setting an arbitrary position designated by the user as the initial position. Since the initial position is not limited, even if the sound source included in the directivity range BF1 based on the estimated sound source position is not an unmanned air vehicle, it is possible to detect an unmanned air vehicle flying in another directivity direction at an early stage. It becomes.

  The directivity processing unit 63 determines whether or not the sound data collected by the microphone array MAk and converted into digital values by the A / D converters An1 to Aq is temporarily stored in the memory 38 (S22). . If not stored (S22, NO), the processing of the directivity processing unit 63 returns to step S21.

  When the sound data collected by the microphone array MA is temporarily stored in the memory 38 (S22, YES), the directivity processing unit 63 is outside the masking area set by the masking area setting unit 69a. In addition, beam forming is performed for an arbitrary directivity direction BF2 in the directivity range BF1 of the monitoring area 8, and sound data in the directivity direction BF2 is extracted (S23).

  The frequency analysis unit 64 detects the frequency of the extracted sound data and the sound pressure thereof (S24).

  The object detection unit 65 compares the detection sound pattern registered in the pattern memory of the memory 38 with the detection sound pattern obtained as a result of the frequency analysis process, and detects the unmanned flying object (S25).

  The detection result determination unit 66 notifies the result of the comparison to the output control unit 35 and also notifies the detection direction control unit 68 of the detection direction shift (S26).

  For example, the object detection unit 65 compares the detected sound pattern obtained as a result of the frequency analysis processing with the four frequencies f1, f2, f3, and f4 registered in the pattern memory of the memory 38. As a result of the comparison, the object detection unit 65 has at least two of the same frequency in both detection sound patterns, and when the sound pressure of these frequencies is greater than the first threshold th1, the detection sound patterns of both are Approximate and determine that the unmanned air vehicle dn exists.

  Here, it is assumed that at least two frequencies match, but the object detection unit 65 approximates when one frequency matches and the sound pressure at this frequency is greater than the first threshold th1. It may be determined that

  In addition, the object detection unit 65 may set an error of an allowable frequency for each frequency, and may determine the presence or absence of the approximation assuming that the frequencies within the error range are the same frequency.

  Further, in addition to the comparison of the frequency and the sound pressure, the object detection unit 65 may determine that the sound pressure ratios of the sounds of the respective frequencies substantially match in addition to the determination condition. In this case, since the determination conditions become strict, the sound source detection unit UDk can easily identify the detected unmanned air vehicle dn as a previously registered object, and the detection accuracy of the unmanned air vehicle dn can be improved.

  The detection result determination unit 66 determines whether or not the unmanned air vehicle dn exists or not as a result of step S26 (S27).

  When the unmanned air vehicle dn exists (S27, YES), the detection result determination unit 66 notifies the output control unit 35 that the unmanned air vehicle dn exists (detection result of the unmanned air vehicle dn) (S28).

  On the other hand, when the unmanned air vehicle dn does not exist (S27, NO), the detection result determination unit 66 instructs the scanning control unit 67 to move the pointing direction BF2 to be scanned in the monitoring area 8 in the next different direction. To do. In response to the instruction from the detection result determination unit 66, the scanning control unit 67 moves the pointing direction BF2 to be scanned in the monitoring area 8 in the next different direction (S29). The notification of the detection result of the unmanned air vehicle dn may be performed collectively after the omnidirectional scanning is completed, not at the timing when the detection processing of one directivity direction is completed.

  Further, if the order in which the directing direction BF2 is sequentially moved in the monitoring area 8 is outside the masking area set by the masking area setting unit 69a, for example, within the directivity range BF1 of the monitoring area 8 or within the entire range of the monitoring area 8. Thus, a spiral (spiral) order may be employed so as to go from the outer circumference to the inner circumference or from the inner circumference to the outer circumference.

  In addition, the detection direction control unit 68 does not continuously scan the directivity direction as in a single stroke, but sets a position in the monitoring area 8 in advance, and sets the masking area set by the masking area setting unit 69a. If it is outside, the directing direction BF2 may be moved to each position in an arbitrary order. Thereby, the monitoring apparatus 10 can start a detection process from the position where the unmanned air vehicle dn easily enters, for example, and can improve the detection process.

  The scanning control unit 67 determines whether or not the omnidirectional scanning in the monitoring area 8 has been completed (S30). When the omnidirectional scanning has not been completed (S30, NO), the signal processing unit 33 returns to step S23, and the processes from step S23 to S30 are repeated. That is, the directivity processing unit 63 performs beam forming in the directivity direction BF2 at the position moved in step S29 outside the masking area set by the masking area setting unit 69a, and extracts sound data in the directivity direction BF2. . Thereby, even if one unmanned air vehicle dn is detected, the sound source detection unit UDk continues the detection of the unmanned air vehicles dn that may exist, so that the detection of a plurality of unmanned air vehicles dn is possible. Is possible.

  On the other hand, when the omnidirectional scanning is completed in step S30 (S30, YES), the directivity processing unit 63 deletes the sound data collected by the microphone array MAk temporarily stored in the memory 38 (S31). ). The directivity processing unit 63 may erase the sound data collected by the microphone array MAk from the memory 38 and store it in the recorder RC.

  After erasing the sound data, the signal processing unit 33 determines whether or not to end the detection process of the unmanned air vehicle dn (S32). The detection process of the unmanned air vehicle dn is terminated according to a predetermined event. For example, the number of times that the unmanned air vehicle dn was not detected in step S26 is held in the memory 38, and when this number exceeds a predetermined number, the detection process for the unmanned air vehicle dn may be terminated. Further, the signal processing unit 33 may end the detection process of the unmanned air vehicle dn based on time-up by a timer or a user operation on a UI (User Interface) (not shown) of the operation unit 32. Moreover, you may complete | finish when the power supply of the monitoring apparatus 10 turns off.

  In the process of step S24, the frequency analysis unit 64 analyzes the frequency and also measures the sound pressure at that frequency. The detection result determination unit 66 determines that the unmanned air vehicle dn is approaching the sound source detection unit UDk when the sound pressure level measured by the frequency analysis unit 64 gradually increases with time. Also good.

  For example, when the sound pressure level of a predetermined frequency measured at time t11 is smaller than the sound pressure level of the same frequency measured at time t12 after time t11, the sound pressure increases with time. It may be determined that the unmanned air vehicle dn is approaching. Even if the sound pressure level is measured three times or more and it is determined that the unmanned air vehicle dn is approaching based on the transition of statistical values (for example, dispersion value, average value, maximum value, minimum value, etc.). Good.

  In addition, when the measured sound pressure level is larger than a warning threshold that is a warning level, the detection result determination unit 66 may determine that the unmanned air vehicle dn has entered the warning area.

  The warning threshold is a value larger than the above-described third threshold th3, for example. The warning area is, for example, the same area as the monitoring area 8 or an area included in the monitoring area 8 and narrower than the monitoring area 8. The alert area is an area where intrusion of the unmanned air vehicle dn is restricted, for example. Moreover, the approach determination and the intrusion determination of the unmanned air vehicle dn may be executed by the detection result determination unit 66.

  FIG. 12 is a diagram showing a display screen example of the first monitor MN1 when the masking area MSK3 is not set. In FIG. 12, in the first monitor MN1, the unmanned air vehicle dn detected by the monitoring device 10 is shown on the upper right side of the paper of FIG. 12 of the omnidirectional image IMG1. Furthermore, sound pressure heat maps corresponding to the unmanned air vehicle dn and the sound source generated within the range of the imaging angle of view of the omnidirectional camera CAk are superimposed and displayed.

  As described with reference to FIG. 9, in this embodiment, if the sound pressure value for each pixel calculated by the sound source direction detection unit 34 is equal to or less than the first threshold th <b> 1, the display is colorless, and the sound pressure value is the first value. If it is greater than the first threshold th1 and less than or equal to the second threshold th2, it is displayed in blue, and if the sound pressure value is greater than the second threshold th2 and less than or equal to the third threshold th3, it is displayed in pink, and the sound pressure value is third. If it is larger than the threshold th3, it is displayed in red.

  In FIG. 12, for example, the sound pressure value is larger than the third threshold th3 in the vicinity of the rotor wing and the rotor around the center of the casing of the unmanned air vehicle dn, so that the red regions RD1, RD2, RD3, and RD4 are drawn. Similarly, pink areas PD1, PD2, PD3, and PD4 indicating that the sound pressure value is next to the red area are drawn around the red area. Similarly, blue areas BD1, BD2, BD3, and BD4 indicating that the sound pressure value is the second largest after the pink area are drawn around the pink area.

  In addition, FIG. 12 shows that a sound source also exists in an office building, and the sound pressure value for each pixel calculated by the sound source direction detecting unit 34 exceeds the third threshold th3 or the pixel Red regions R1, R2, R3, R4, R5, R6, and R7 are drawn for the set. Similarly, pink areas P1, P2, P3, P4, and P5 indicating that the sound pressure value is next to the red area are drawn around the red area of the office building. Similarly, blue areas B1 and B2 indicating that the sound pressure value is next to the pink area are drawn around the pink area of the office building. In the other areas of the omnidirectional image IMG1, since the sound pressure value is equal to or less than the first threshold th1, it is drawn in the colorless area N1, and the visibility of the omnidirectional image IMG1 serving as the background is not deteriorated.

  Next, details of the setting of the masking area in the present embodiment will be described with reference to FIGS. FIG. 13 is an explanatory diagram showing a display example of the masking area in time series in the automatic learning process. FIG. 14 is a sequence diagram illustrating an example of an operation procedure for setting a masking area in the first embodiment. FIG. 15 is a diagram illustrating a display screen example of the first monitor when a masking area is set. The sequence shown in FIG. 14 is a so-called initial setting executed before the operation of the sequence shown in FIG. 9 is started.

  In FIG. 14, the user instructs the monitoring apparatus 10 to start the automatic learning process of the masking area using the operation unit 32, for example (T1). The monitoring apparatus 10 makes an image distribution request to the omnidirectional camera CAk (T2). The omnidirectional camera CAk starts imaging processing according to power-on in accordance with this image distribution request. The omnidirectional camera CAk transmits data of omnidirectional images (for example, still images and moving images) obtained by imaging to the monitoring apparatus 10 via the network NW (T3). The monitoring device 10 converts the omnidirectional image data transmitted from the omnidirectional camera CAk into display data such as NTSC, and outputs it to the first monitor MN1 to instruct the display of the omnidirectional image data (T4). Thus, when the display data transmitted from the monitoring device 10 is input, the first monitor MN1 displays the data of the omnidirectional image IMG1 from the omnidirectional camera CAk on the screen (see the upper left of FIG. 13).

  In addition, the monitoring device 10 issues a voice distribution request to the microphone array MAk (T5). The microphone array MA starts sound collection processing in response to power-on in accordance with the voice distribution request. The microphone array MAk encodes the audio data of the monitoring area 8 obtained by sound collection via the network NW and transmits it to the monitoring device 10 (T6). In the monitoring device 10, the sound source direction detection unit 34 configures omnidirectional image data in the monitoring area 8 based on the omnidirectional image data captured by the omnidirectional camera CAk and the audio data collected by the microphone array MAk. A sound pressure as a sound parameter is calculated for each pixel.

  Further, the masking area setting unit 69a determines a pixel or a set of pixels whose calculated sound pressure by the sound source direction detection unit 34 is equal to or greater than a predetermined masking area threshold (for example, the above-described third threshold th3). The masking area setting unit 69a stores and registers information indicating the determined pixel or a set thereof in the memory 38 as information indicating the masking area (T7). Specifically, the information indicating the masking area is coordinates on the omnidirectional image that specify the position of the pixel at which the calculated sound pressure value is equal to or greater than the masking area threshold. The masking area setting unit 69a outputs, via the output control unit 35, information indicating the masking area and the masking area (that is, a pixel or a set of the sound pressure calculated values equal to or greater than the masking area threshold) to a predetermined color (for example, red). ) Is output to the first monitor MN1 (T8). Thereby, the first monitor MN1 performs a process of painting the position of the coordinates corresponding to the masking area MSK1 on the omnidirectional image IMG with a predetermined color according to the instruction transmitted from the monitoring device 10 (see the upper right in FIG. 13). . In the omnidirectional image IMG1 on the upper right side of FIG. 13, the masking area MSK1 indicates the entire area painted in a predetermined color.

  Similarly, the microphone array MA encodes the audio data of the monitoring area 8 obtained by the continuous sound collection via the network NW in accordance with the audio distribution request from the monitoring device 10 to the monitoring device 10. Transmit (T9). In the monitoring device 10, the sound source direction detection unit 34 configures omnidirectional image data in the monitoring area 8 based on the omnidirectional image data captured by the omnidirectional camera CAk and the audio data collected by the microphone array MAk. A sound pressure as a sound parameter is calculated for each pixel.

  Further, the masking area setting unit 69a determines a pixel or a set thereof in which the calculated sound pressure by the sound source direction detection unit 34 is equal to or greater than the masking area threshold value. The masking area setting unit 69a stores and registers information indicating the determined pixel or a set thereof in the memory 38 as information indicating the masking area (T10). The masking area setting unit 69a outputs, via the output control unit 35, information indicating the masking area and an instruction to fill the masking area with a predetermined color (for example, red) to the first monitor MN1 (T11). Thereby, the first monitor MN1 fills the position of the coordinates corresponding to the masking area MSK2 accumulated with respect to the masking area MSK1 on the omnidirectional image IMG with a predetermined color according to the instruction transmitted from the monitoring device 10. (Refer to the lower right of FIG. 13). In the omnidirectional image IMG1 on the lower right side of FIG. 13, the masking area MSK2 indicates the entire area painted with a predetermined color.

  Here, a user operation using the operation unit 32 instructs the monitoring device 10 to end the automatic learning process of the masking area (T12). In response to this instruction, the monitoring device 10 transmits a voice distribution stop request to the microphone array MAk (T13). Thereby, the microphone array MAk stops the delivery (transmission) of the audio data in the monitoring area 8 obtained by sound collection to the monitoring device 10.

  Further, by a user operation using the operation unit 32, an operation for correcting the masking area (that is, adding or deleting the masking area) is performed on the first monitor MN1 on which the masking area MSK2 on the lower right side of FIG. 13 is shown. When done (T14), in the monitoring apparatus 10, the masking area setting unit 69a adds or deletes the specified position on the omnidirectional image IMG1 as a masking area by the user operation, Information indicating the masking area MSK3 after correction is stored in the memory 38 and registered (T15).

  For example, in step T14, an operation for deleting unnecessary areas GOM1, GOM2, and GOM3 as masking areas (at the discretion of the user) with respect to the first monitor MN1 showing the masking area MSK2 at the lower right of FIG. For example, a range specifying operation) is performed, or an operation for adding a background office building area covered by the masking area MSK2 as a masking area (that is, a drawing operation) is performed.

  The masking area setting unit 69a outputs, via the output control unit 35, information indicating the masking area and an instruction to fill the masking area with a predetermined color (for example, red) to the first monitor MN1 (T16). As a result, the first monitor MN1 displays the position of the coordinates corresponding to the masking area MSK3 accumulated with respect to the masking areas MSK1 and MSK2 on the omnidirectional image IMG in a predetermined color according to the instruction transmitted from the monitoring device 10. A painting process is performed (see the lower left of FIG. 13). In the omnidirectional image IMG1 on the lower left side of FIG. 13, the masking area MSK3 indicates the entire area painted with a predetermined color. Accordingly, the setting is excluded from the masking area MSK3.

  Thereby, according to the sequence shown in FIG. 14, the unmanned air vehicle dn as the detection target of the present embodiment is in the sky where there is almost no sound source in the vicinity of the imaging field angle of the omnidirectional camera CAk. In view of the tendency to fly, since no sound source with a sound pressure exceeding the masking area threshold was found in the sky by analysis of the sound data from the microphone array MAk, the sky was set as a detection target for the unmanned air vehicle dn. It becomes possible. On the other hand, by setting an office building with many sound sources in the surrounding area as a masking area, it is possible to avoid erroneous detection of a sound source other than the unmanned air vehicle dn as an unmanned air vehicle dn originally intended to be detected. It is possible to improve the detection accuracy and the detection processing speed of the unmanned air vehicle dn.

  In other words, as shown in FIG. 15, in the omnidirectional image IMG1, the unmanned air vehicle dn is detected only in another area outside the masking area MSK3. As a result, compared with the omnidirectional image IMG1 shown in FIG. 12, the sound pressure value of the sound generated at the sound source position at the sound source position around the unmanned air vehicle dn detected in the other area except the masking area MSK3. Is superimposed and displayed with a sound pressure heat map converted into a visual image. On the other hand, the sound pressure value of the sound generated at the sound source position is displayed in the visual image around other sound sources that are not unmanned air vehicles detected in the masking area MSK3 (for example, a yelling voice of a person in an office building). The superimposed display of the converted sound pressure heat map is omitted.

  In the present embodiment, when the masking area is set by the masking area setting unit 69a, the sound pressure value of the sound generated at the sound source position is visually displayed around the unmanned air vehicle dn as shown in FIG. The sound pressure heat map converted into an image is displayed superimposed. However, when the masking area is not set by the masking area setting unit 69a, a sound pressure heat map corresponding to the detected sound pressure value of the sound source is superimposed on the omnidirectional image IMG1 as shown in FIG. May be displayed.

  As described above, in the unmanned air vehicle detection system 5 of the present embodiment, the monitoring device 10 uses the voice data collected by the microphone array MAk to perform unmanned flight that appears in the captured image (omnidirectional image IMG1) of the monitoring area 8. A masking area for excluding detection of the body dn is set by the masking area setting unit 69a. The monitoring device 10 detects the unmanned air vehicle dn in other areas excluding the masking area using the voice data collected by the microphone array MAk and the information indicating the masking area. When the monitoring device 10 detects the unmanned air vehicle dn outside the masking area, the monitoring device 10 displays the sound source visual image (that is, the sound level at the sound source position) at the sound source position of the unmanned air vehicle dn in the omnidirectional image IMG1. , Red region RD1, pink region PD1, blue region BD1,...) Are superimposed and displayed on first monitor MN1.

  Thereby, the unmanned air vehicle detection system 5 automatically sets a masking area for excluding the detection process of the unmanned air vehicle dn to be detected for the monitoring area 8 to be imaged by the omnidirectional camera CAk. Therefore, the probability of erroneous detection of an object at the sound source position in the masking area as the unmanned air vehicle dn can be reduced, and deterioration in detection accuracy of the unmanned air vehicle dn can be suppressed. Further, the unmanned air vehicle detection system 5 does not need to detect the unmanned air vehicle dn over the imaging angle of view of the omnidirectional camera CAk (that is, the entire area of the omnidirectional image IMG1), and is unmanned for an area where the masking area is omitted. Since the detection determination of the flying object dn may be performed, the detection process of the unmanned flying object dn can be further improved.

  In the unmanned aerial vehicle detection system 5, the sound source direction detection unit 34 determines the sound pressure for specifying the loudness of the monitoring area 8 based on the sound data collected by the microphone array MAk as an omnidirectional image IMG1. Is calculated for each predetermined unit of the pixels constituting the. The masking area setting unit 69a displays the position of the sound source where the calculated value of the sound pressure is equal to or greater than the masking area threshold related to the loudness or a region including the position superimposed on the first monitor MN1, and further confirms the user. By the operation, the sound source area displayed on the first monitor MN1 is set as a masking area. Thereby, the user can reduce the possibility that the unmanned air vehicle dn will fly, but the masking area for omitting the place where the other sound source (for example, a person's yell) is likely to occur from the detection target area of the unmanned air vehicle dn. The first monitor MN1 can be easily set while visually confirming.

  Further, the masking area setting unit 69a masks the sound source area after the user addition operation by the user addition operation for further adding the sound source area displayed on the first monitor MN1 (that is, the area as a masking area candidate). Set as area. As a result, the user automatically confirms a portion of the first monitor MN1 that is painted in a predetermined color as a masking area candidate by the monitoring device 10 while visually checking the portion that the user desires to add to the masking area. The masking area can be set while specifying easily, improving user convenience.

  Further, the masking area setting unit 69a performs the sound source after the user deletion operation by the user deletion operation for deleting at least a part of the sound source region (that is, the masking area candidate region) displayed on the first monitor MN1. Set the area as a masking area. As a result, the user can visually check on the first monitor MN1 the portion that has been painted in a predetermined color as a masking area candidate by the monitoring device 10 and is painted in the predetermined color by his / her own judgment. Therefore, it is possible to set a masking area while easily specifying a part to be excluded as a masking area.

  Further, the output control unit 35 converts the sound source visual image obtained by converting the sound pressure into different visual images in a stepwise manner in the monitoring area 8 according to the comparison between the calculated sound pressure value and a plurality of threshold values related to the sound volume. The omnidirectional image IMG1 is displayed on the first monitor MN1 in a superimposed manner for each predetermined unit of pixels constituting the omnidirectional image IMG1. Thereby, the user can know a wide state of the monitoring area 8 as an omnidirectional image in the omnidirectional image IMG1 of the monitoring area 8 captured by the omnidirectional camera CAk by looking at the first monitor MN1. In addition, the location of the sound source (eg, unmanned air vehicle dn) generated outside the masking area of the monitoring area 8 and the magnitude of the sound can be easily confirmed as a visual image.

(Background to the second embodiment)
In Patent Document 1 described above, by comparing the sound pressure at a frequency peculiar to a flying object such as a helicopter or Cessna with a predetermined setting level, if it is equal to or higher than the setting level, it is determined that the flying object is a monitoring object. It is disclosed.

  However, Patent Document 1 described above does not take into account that the measured sound pressure quantitatively indicates which of the sound pressures in which a plurality of stages are defined. For this reason, when any sound is detected in the imaging area of the camera device, the sound level is specified by detailed visual information regardless of the size of the sound at the detected sound source position. There is a problem that it cannot be presented.

  Therefore, in the second embodiment, regardless of the magnitude of the sound at the sound source position detected in the imaging area of the camera device, the loudness of the sound at the sound source position is presented in a fine stepwise manner. An example of a monitoring system that contributes to an accurate grasp of the loudness of sound in the user will be described.

(Second Embodiment)
In the second embodiment, the internal configuration of each device constituting the unmanned air vehicle detection system 5 is the same as the internal configuration of each device constituting the unmanned air vehicle detection system 5 of the first embodiment. The contents are denoted by the same reference numerals, description thereof is omitted, and different contents are described.

  In the second embodiment, the monitoring device 10 generates the sound pressure heat map described in the first embodiment and displays the heat pressure heat map on the first monitor MN1, and then the user operates the operation unit 32 (see below). The sound pressure heat map is finely analyzed and displayed according to the relationship between the calculated value of the sound pressure necessary for generating the sound pressure heat map and a plurality of threshold values (see later). Hereinafter, three analysis methods will be described.

(First analysis method)
In the first analysis method, the monitoring apparatus 10 superimposes the sound pressure heat map corresponding to the omnidirectional image IMG2 on the omnidirectional image IMG2 and displays the sound map on the first monitor MN1, and then the user performs a part of the omnidirectional image IMG2. When the range is designated, the display resolution of the sound pressure heat map in the designated range is changed to be the same as the display resolution of the entire omnidirectional image IMG2. An example of the operation of the first analysis method will be described with reference to FIGS. FIG. 16 is a schematic explanatory diagram of dynamic change of the display resolution of the sound pressure heat map in the second embodiment. FIG. 17 is a flowchart for explaining an example of an operation procedure for dynamically changing the display resolution of the sound pressure heat map in the second embodiment.

  In FIG. 17, the monitoring apparatus 10 has input the audio data of the monitoring area 8 transmitted from the microphone array MAk (S41). It is determined whether or not a partial clipping range of the omnidirectional image IMG2 (see the upper left of FIG. 16) on the first monitor MN1 has been designated by a user operation using the operation unit 32 (S42).

  Here, in the upper left of FIG. 16, the omnidirectional image IMG2 displayed on the first monitor MN1 has a display size in the X direction “Wmax” and a display size in the Y direction “Hmax” of the omnidirectional image IMG2. Furthermore, it has a display resolution of “Wmax × Hmax”. In addition, the coordinates of the end points of the cutout range, which are specified by the user operation and become a part of the omnidirectional image IMG2, are (X1, Y1), (X1, Y2), (X2, Y1), (X2, Y2). It is a rectangle. In FIG. 16, only (X1, Y1) and (X2, Y2) existing on the diagonal of the same rectangle are shown.

  If a rectangle with four endpoints (X1, Y1), (X1, Y2), (X2, Y1), (X2, Y2) as end points is not specified by the user operation (S42, NO), monitoring The sound source direction detection unit 34 of the apparatus 10 sets (X, Y) = (0, 0) (S43) and generates coordinates (X, Y) in order to generate a sound pressure map for the entire omnidirectional image IMG2. = Sound pressure P (X, Y) at (0,0) is calculated (S45).

  Further, when the X coordinate of the omnidirectional image IMG2 does not coincide with the maximum value Wmax (S46, NO), the sound source direction detection unit 34 generates a sound pressure map for the entire omnidirectional image IMG2. The X coordinate is incremented by 1 (S47), and the sound pressure P (X, Y) is calculated at the incremented (X, Y).

  When the X coordinate of the omnidirectional image IMG2 matches the maximum value Wmax (S46, YES), the sound source direction detection unit 34 (when the Y coordinate of the omnidirectional image IMG2 does not match the maximum value Hmax) ( S48, NO), in order to generate a sound pressure map for the entire omnidirectional image IMG2, the X coordinate is returned to 0 and the Y coordinate is incremented by 1 (S49), and the sound pressure P is increased at (X, Y) after the increment. (X, Y) is calculated. The sound source direction detection unit 34 repeats the processes in steps 45 to S49 until the Y coordinate of the omnidirectional image IMG2 coincides with the maximum value Hmax, so that the entire omnidirectional image IMG2 is processed as in the first embodiment. A sound pressure map can be generated and stored in the memory 38 and registered (S50).

  On the other hand, when a rectangle having four end points (X1, Y1), (X1, Y2), (X2, Y1), (X2, Y2) is designated by a user operation (S42, YES), The sound source direction detection unit 34 of the monitoring device 10 sets (X, Y) = (X1, Y1) in order to generate a sound pressure map for the range specified by the user operation in the omnidirectional image IMG2 (S44). ), The sound pressure P (X, Y) at the coordinates (X, Y) = (X1, Y1) is calculated (S45).

  Furthermore, when the X coordinate of the omnidirectional image IMG2 does not match the maximum value Wmax (S46, NO), the sound source direction detection unit 34 generates sound for the range designated by the user operation in the omnidirectional image IMG2. In order to generate the pressure map, the X coordinate is increased by (X2-X1) / Wmax (S47), and the sound pressure P (X, Y) is calculated at (X, Y) after the increase.

  When the X coordinate of the omnidirectional image IMG2 matches the maximum value Wmax (S46, YES), the sound source direction detection unit 34 (when the Y coordinate of the omnidirectional image IMG2 does not match the maximum value Hmax) ( (S48, NO), in order to generate a sound pressure map for the range designated by the user operation in the omnidirectional image IMG2, the X coordinate is returned to X1, and the Y coordinate is increased by (Y2-Y1) / Hmax (S49). ), The sound pressure P (X, Y) is calculated at (X, Y) after the increase. The sound source direction detection unit 34 repeats the processes in steps 45 to S49 until the Y coordinate of the omnidirectional image IMG2 coincides with the maximum value Hmax, so that the sound pressure with respect to the range designated by the user operation in the omnidirectional image IMG2 A map can be generated and stored in the memory 38 and registered (S50). After step S50, the process of the monitoring apparatus 10 returns to step S41, and the processes of steps S42 to S50 are repeated for the voice data input in step S41.

  Therefore, as shown in the upper right of FIG. 16, the monitoring device 10 simply cuts out a range specified by a user operation and generates an audio heat map according to the sound pressure value for each pixel constituting the range. When superimposed on one monitor MN1, it is only possible to display an image with a low display resolution (that is, a coarse image).

  However, according to the first analysis method of the present embodiment, the monitoring apparatus 10 displays the display resolution of the cutout range for the partial cutout range of the omnidirectional image IMG2 designated by the user operation as the entire omnidirectional image IMG2. Sound pressure P (X, Y) is calculated in fine units (that is, every (X2-X1) / Wmax in the X direction and every (Y2-Y1) / Hmax in the Y direction) so as to be the same as the resolution. To do. As a result, as shown in the lower right corner of FIG. 16, the monitoring apparatus 10 has a finer resolution than the display resolution in the sound pressure heat map when the sound pressure heat map in the cutout range specified by the user operation is simply cut out ( (Unit) can be displayed on the first monitor MN1 with high accuracy, and the user can more accurately grasp the details of the distribution of the sound source in the clipping range specified by the user operation.

(Second analysis method)
Before describing the second analysis method, as a matter common to FIGS. 18 to 20, in the present embodiment, a plurality of threshold widths that define the volume of sound stepwise will be described. FIG. 18 is a schematic explanatory diagram of threshold width adjustment according to the frequency distribution of sound pressure values and a captured image display result associated with the width adjustment in the second embodiment. In FIG. 18 to FIG. 20, sound source visual images corresponding to the threshold values from the lower limit toward the upper limit are a group blue image, an indigo image, a blue image, a light blue image, a blue-green image, a yellow-green image, It is specified that a yellow image, an orange image, a red image, and a red image are used. When the sound pressure value is minimum (that is, lower limit), the group blue image is used, and when the sound pressure value is maximum (that is, upper limit), the red image is used.

  For example, as shown on the left side of FIG. 18, a total of ten threshold values are defined so as to correspond to the magnitude of the sound pressure value. For example, in the example on the left side of FIG. Each scale corresponds to a threshold value. For this reason, in the example on the left side of FIG. 18, when the sound pressure value is included between a certain threshold value (for example, 5) and the next largest threshold value (for example, 6), A color image (for example, a yellow-green image) corresponding to a threshold value including a sound pressure value is used as a sound source visual image for visually indicating the sound pressure value.

  In the second analysis method, when the output control unit 35 generates the sound pressure heat map corresponding to the omnidirectional image, the threshold adjustment unit 69b of the monitoring device 10 is configured for each pixel constituting the omnidirectional image (a predetermined unit of pixels). Based on the frequency of occurrence of sound pressure values calculated in other ways (in other words, frequency distribution), a total of 10 threshold values or the width between the threshold values is dynamically changed. That is, the threshold adjustment unit 69b dynamically changes the setting of the correspondence between the plurality of thresholds and the sound source visual image according to the omnidirectional image. For example, referring to FIG. 18, if the sound pressure value is between the threshold value 5 and the threshold value 6, the threshold adjustment unit 69 b holds the setting for using the yellow-green image as the sound source visual image. When the appearance frequency of 5 to threshold 6 is high among all the pixels of the omnidirectional image, the width LG1 (for example, threshold 5 to threshold 6) between the thresholds for using the yellow-green image becomes narrower. Width LG2 (for example, threshold value 4.5 to threshold value 4.8).

  Therefore, when the sound pressure values for each pixel constituting the omnidirectional image are concentrated in the width AR1 between the thresholds for using a specific color image, as shown in the lower left of FIG. As the heat map VMP1, the same color image or visually similar color images (for example, a yellow-green image and a blue-green image) are used as the sound source visual image, and the distribution of the sound source appearing in the omnidirectional image is finely defined. It is difficult to present to the user.

  However, according to the second analysis method of the present embodiment, the monitoring apparatus 10 uses the threshold adjustment unit 69b to generate a sound source according to the appearance frequency (frequency distribution) of the sound pressure value for each pixel constituting the omnidirectional image. The setting of the correspondence relationship between the visual image and the plurality of threshold values is dynamically changed, and the sound pressure heat map VMP2 corresponding to the omnidirectional image is displayed on the first monitor MN1 after reflecting the change. As a result, the monitoring device 10 can present a fine sound pressure distribution to the user as the sound pressure heat map VMP2 as shown in the lower right of FIG. 18, and the user can accurately grasp the sound source position distribution. Can do.

(Third analysis method)
In the third analysis method, the monitoring apparatus 10 arbitrarily sets an upper limit, a lower limit, or both of the respective thresholds for defining the use of the sound source visual image (that is, the color image) by a user operation using the operation unit 32. Can be specified. FIGS. 19A and 19B are schematic explanatory diagrams of the setting change of the width between the thresholds defining the use of the sound source visual image in the second embodiment.

  For example, in FIG. 19A, the width between thresholds (upper threshold) for defining the use of a sound source visual image (that is, a red image) indicating that the sound pressure value is the highest (that is, the upper limit) is the threshold. 9 to threshold value 10 to threshold value 6 to threshold value 10, and further to specify the use of a sound source visual image (that is, an ultramarine image) indicating that the sound pressure value is the lowest (that is, the lower limit). The width between the thresholds (lower threshold) is changed from threshold 0 to threshold 1 to threshold 0 to threshold 2. Accordingly, the monitoring device 10 dynamically changes the remaining eight threshold values so that the widths between the threshold values are equal, and is different from the width between the two threshold values changed by the user operation. Thus, the width between the threshold values can be changed.

  Further, for example, in FIG. 19B, use of a sound source visual image (that is, a red image) indicating that the sound pressure value is the highest (that is, the upper limit) after being changed as illustrated in FIG. The threshold value for defining the threshold value (upper threshold value) is changed from threshold value 6 to threshold value 10 to threshold value 9.3 to threshold value 10, and further indicates that the sound pressure value is the lowest (that is, the lower limit). The width between threshold values (lower threshold values) for defining the use of the sound source visual image (that is, the group blue image) is changed from threshold value 0 to threshold value 2 to threshold value 0 to threshold value 5. As a result, the monitoring device 10 similarly sets the remaining eight threshold values so that the respective threshold values are different from the example shown in FIG. 19A, but the widths between the remaining eight threshold values are equal. The width between the threshold values can be changed so as to be different from the width between the two threshold values that are changed dynamically by a user operation.

  In FIGS. 19A and 19B, between the thresholds (upper thresholds) for defining the use of the sound source visual image (that is, the red image) indicating that the sound pressure value is the highest (that is, the upper limit). And the width between the thresholds (lower threshold) to specify the use of the sound source visual image (that is, the group blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) is changed. Although the example to be described was demonstrated, it is the same also in the change of either one.

  In other words, when only the width between the thresholds (upper threshold) for defining the use of the sound source visual image (that is, the red image) indicating the highest sound pressure value (that is, the upper limit) is changed Similarly, the monitoring device 10 dynamically changes so that the width between the remaining nine threshold values becomes equal, and the threshold value is different from the width between the single threshold values changed by the user operation. The width can be changed.

  Also, when only the width between the thresholds (lower threshold) for defining the use of the sound source visual image (that is, the group blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) is changed. Similarly, the monitoring device 10 dynamically changes the widths between the remaining nine thresholds so that they are equally spaced, and the threshold values are different from the widths between the single thresholds changed by the user operation. The width of can be changed.

  Here, the operation related to the threshold width setting change according to the third analysis method will be described with reference to FIGS. FIG. 20 is a schematic explanatory diagram of display of a captured image in accordance with a setting change of a width between thresholds that defines use of a red color image and a group blue image in the second embodiment. FIG. 21 is a flowchart for explaining an example of an operation procedure for changing the setting of the width between thresholds in the second embodiment.

  As shown in FIG. 20, the sound source visual image (that is, the red image) indicating that the sound pressure value is the highest (that is, the upper limit) and the sound pressure value that is the lowest (that is, the lower limit) are indicated. Assume that both the widths between the thresholds that define the use of the sound source visual image (that is, the group blue image) are changed. In this case, in the monitoring device 10, as shown in FIG. 21, the threshold adjustment unit 69b has a correspondence table (not shown, correspondence relationship) between the sound source visual image and the threshold value that defines the use of the sound source visual image or the width between the threshold values. Change the setting dynamically.

  Therefore, in the example shown in FIG. 20, the monitoring device 10 has only a red image in the sound pressure heat map VMP2 superimposed on the omnidirectional image (that is, the captured image). A sound pressure heat map VMP2A using the sound source visual image is generated, and the sound pressure heat map VMP2A is superimposed on the omnidirectional image (that is, the captured image) and displayed on the first monitor MN1. The user operation is, for example, an input operation on an input screen (not shown) of a threshold value or a width between threshold values displayed on the first monitor MN1, or a drag on a display screen of a width between threshold values displayed on the first monitor MN1. Although applicable, it is not limited to these operations.

  In FIG. 21, the threshold adjustment unit 69 b defines the use of a sound source visual image (that is, a red image) indicating that the sound pressure value is the highest (that is, the upper limit) by a user operation using the operation unit 32. It is determined whether or not the width between the thresholds has been changed (S61). When the width between the thresholds that defines the use of the sound source visual image (that is, the red image) indicating that the sound pressure value is the highest (that is, the upper limit) is changed (S61, YES), the threshold adjustment unit 69b Then, it is determined whether or not the width between the thresholds for defining the use of the sound source visual image (that is, the group blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) is changed (S62).

  The threshold adjustment unit 69b changes the width between the thresholds that regulates the use of the sound source visual image (that is, the group blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) (S62, YES), in accordance with the change result, the correspondence table between the sound source visual image and the threshold value defining the use of the sound source visual image or the width between the threshold values is corrected (S63). For example, the threshold value adjustment unit 69b dynamically changes the remaining eight threshold values so that the widths between the threshold values are equal to each other, and is different from the width between the two threshold values changed by the user operation. Change the width between thresholds. Thereby, for example, when a large number of sound pressure values between the threshold value that defines the use of the red color image and the threshold value that defines the use of the ultramarine image are obtained, the monitoring apparatus 10 can obtain a width between the threshold values by the user. Through the adjustment operation, the distribution around the pixel where many sound pressure values are concentrated can be displayed as a fine sound pressure heat map.

  The threshold adjustment unit 69b determines that the width between the thresholds defining the use of the sound source visual image (that is, the red image) indicating that the sound pressure value is the lowest (that is, the upper limit) is not changed (S62, NO). Depending on the change result, the correspondence table between the sound source visual image and the threshold value that defines the use of the sound source visual image or the width between the threshold values is corrected (S64). For example, the threshold adjustment unit 69b dynamically changes the remaining nine thresholds so that the widths between the thresholds are equally spaced, so that the width between the single thresholds changed by the user operation is different. Change the width between thresholds. Thereby, for example, when many sound pressure values less than or equal to a threshold value that regulates the use of a red image are obtained, the monitoring apparatus 10 concentrates a large number of sound pressure values by adjusting the width between the threshold values by the user. It is possible to display the distribution around the pixel in detail as a sound pressure heat map.

  On the other hand, in step S61, when the width between the thresholds that regulates the use of the sound source visual image (that is, the red image) indicating that the sound pressure value is the highest (that is, the upper limit) is not changed (S61, NO), The threshold adjustment unit 69b determines whether or not the width between the thresholds defining use of the sound source visual image (that is, the group blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) has been changed. (S65). In addition, when the width between the threshold values defining use of the sound source visual image (that is, the group blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) is not changed (S65, NO), the threshold value is changed. The process of the adjustment unit 69b returns to step S61.

  When the width between the thresholds that regulates the use of the sound source visual image (that is, the group blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) is changed (S65, YES), in accordance with the change result, the correspondence table between the sound source visual image and the threshold value defining the use of the sound source visual image or the width between the threshold values is corrected (S66). For example, the threshold adjustment unit 69b dynamically changes the remaining nine thresholds so that the widths between the thresholds are equally spaced, so that the width between the single thresholds changed by the user operation is different. Change the width between thresholds. Thereby, for example, when many sound pressure values greater than or equal to a threshold value that regulates the use of the group blue image are obtained, the monitoring apparatus 10 can obtain a large number of sound pressure values by adjusting the width between the threshold values by the user. The distribution around the concentrated pixels can be displayed as a fine sound pressure heat map.

  As described above, in the unmanned air vehicle detection system 5 of the present embodiment, the monitoring device 10 monitors the sound pressure that specifies the sound level of the monitoring area 8 using the sound data collected by the microphone array MAk. Calculation is performed for each predetermined unit of pixels constituting the captured image (omnidirectional image IMG2) of area 8. According to the comparison between the calculated sound pressure value and a plurality of threshold values relating to the sound volume, the monitoring device 10 converts the sound source visual image obtained by stepwise conversion of the sound pressure into different visual images of the pixels constituting the captured image. The information is displayed on the first monitor MN1 in a superimposed manner for each predetermined unit. When any sound source position is specified in the captured image on which the sound source visual image is superimposed, the monitoring device 10 determines a predetermined unit of pixels constituting the rectangular range including the sound source position between the captured image and the rectangular range. Sound pressure is calculated for each value divided by the size ratio.

  As a result, the monitoring apparatus 10 can increase the sound pressure heat map in the rectangular range (that is, the cutout range) designated by the user operation with a finer resolution (unit) than the display resolution in the sound pressure heat map when the cutout is simply cut out. The information can be displayed on the first monitor MN1 with high accuracy, and the user can be made to grasp the details of the distribution of the sound source in the clipping range designated by the user operation more accurately. In other words, the monitoring device 10 presents the sound level at the sound source position in a fine and stepwise manner regardless of the sound level at the sound source position detected in the monitoring area 8 of the omnidirectional camera CAk. This can contribute to the accurate grasp of the sound volume at the sound source position to the user.

  Further, in the unmanned air vehicle detection system 5 of the present embodiment, the monitoring device 10 uses the audio data collected by the microphone array MAk to determine the sound pressure that specifies the loudness of the monitoring area 8. It calculates for every predetermined unit of the pixel which comprises the eight captured images (omnidirectional image IMG2). The monitoring apparatus 10 corresponds to each threshold value of a plurality of threshold values that prescribes the volume of sound in a stepwise manner and a sound source visual image that is converted stepwise into visual images having different sound pressures in accordance with comparison with each threshold value. The setting of the relationship is dynamically changed according to the captured image (that is, the omnidirectional image) of the monitoring area 8. The monitoring apparatus 10 uses the sound source visual image corresponding to the calculated sound pressure value as the captured image for each predetermined unit of the pixels constituting the captured image based on the calculated sound pressure value and the changed setting of the correspondence relationship. The information is superimposed and displayed on the first monitor MN1.

  As a result, the monitoring device 10 uses a fine sound pressure as a sound pressure heat map for visually indicating the position of the sound source picked up in the monitoring area 8 according to the captured image captured by the omnidirectional camera CAk. The distribution can be presented to the user, and the distribution of the sound source position can be accurately grasped by the user.

  Further, the threshold adjustment unit 69b of the monitoring device 10 changes the width between the thresholds that define the sound source visual image based on the appearance frequency of the sound pressure for each predetermined unit of the pixels constituting the captured image of the monitoring area 8. . As a result, the monitoring device 10 increases the type of the sound source visual image for the pixel corresponding to the sound pressure calculation value having a high appearance frequency, while the sound source visual image is applied to the pixel corresponding to the sound pressure calculation value having a low appearance frequency. Since the sound pressure heat map is generated after the number of types is reduced, it is possible to present sound pressure distributions of fine and diverse shades instead of monotonous shades, and the distribution of sound source positions can be accurately grasped by the user.

  Further, the threshold adjustment unit 69b of the monitoring device 10 changes its width in accordance with an operation for changing the width between the thresholds that regulates the use of the sound source visual image (that is, the red image) corresponding to the upper limit value of the sound pressure. The widths between all other thresholds except for the width between the thresholds after are changed equally. Thereby, for example, when many sound pressure values less than or equal to a threshold value that regulates the use of a red image are obtained, the monitoring apparatus 10 concentrates a large number of sound pressure values by adjusting the width between the threshold values by the user. It is possible to display the distribution around the pixel in detail as a sound pressure heat map.

  In addition, the threshold adjustment unit 69b of the monitoring device 10 changes its width according to an operation of changing the width between thresholds that regulates the use of the sound source visual image (that is, the group blue image) corresponding to the lower limit value of the sound pressure. The widths between all other thresholds except for the width between the thresholds after being changed are uniformly changed. Thereby, for example, when many sound pressure values greater than or equal to a threshold value that regulates the use of the group blue image are obtained, the monitoring apparatus 10 can obtain a large number of sound pressure values by adjusting the width between the threshold values by the user. The distribution around the concentrated pixels can be displayed as a fine sound pressure heat map.

  Further, the threshold adjustment unit 69b of the monitoring device 10 has a sound source visual image corresponding to the width between the thresholds that defines the use of the sound source visual image (that is, the red image) corresponding to the upper limit value of the sound pressure and the lower limit value of the sound pressure. (In other words, according to the operation of changing the width between the thresholds that regulates the use of the ultramarine image), the width between all other thresholds is changed equally except for the width between the thresholds after the width is changed. To do. Thereby, for example, when a large number of sound pressure values between the threshold value that defines the use of the red color image and the threshold value that defines the use of the ultramarine image are obtained, the monitoring apparatus 10 can obtain a width between the threshold values by the user. Through the adjustment operation, the distribution around the pixel where many sound pressure values are concentrated can be displayed as a fine sound pressure heat map.

(Background to the third embodiment)
In Patent Document 1 described above, a monitoring camera capable of changing the shooting direction in an arbitrary direction within the monitoring area is provided. When a flying object such as a helicopter or a Cessna is detected, the shooting direction of the monitoring camera is changed. Turning is disclosed. In other words, it has been disclosed to change the shooting direction of the surveillance camera in order to capture an image of a detected flying object.

  However, the above-described Patent Document 1 does not consider displaying a wide range of captured images including unmanned aerial vehicles detected within the range of the angle of view of the camera device with respect to the imaging area. For this reason, the unmanned flying object is detected in the imaging area of the camera device, and what kind of sound source is present in other locations in the same imaging area is visually presented to the user. There was a problem that it was not possible.

  Therefore, in the third embodiment, the unmanned flying object is detected in the imaging area of the camera device, and what kind of sound source exists elsewhere in the same imaging area. An example of a monitoring system that visually presents to a user without degrading the visibility of a captured image of the apparatus will be described.

(Third embodiment)
In 3rd Embodiment, since the internal structure of each apparatus which comprises the unmanned air vehicle detection system 5 of 1st Embodiment is the same as that of each apparatus which comprises the unmanned air vehicle detection system 5 of 1st Embodiment, it is the same. The contents are denoted by the same reference numerals, description thereof is omitted, and different contents are described.

  In the third embodiment, the monitoring device 10 generates the sound pressure heat map (sound parameter map) described in the first embodiment, and then becomes a translucent image (semi-transparent map) of the sound pressure heat map. A translucent sound pressure heat map is generated, and this translucent sound pressure heat map is superimposed on the omnidirectional image and displayed on the first monitor MN1 (see FIG. 22). FIG. 22 is a schematic explanatory diagram of overlay display of an omnidirectional image and a translucent sound pressure heat map in the third embodiment.

  In the present embodiment, as shown in FIG. 22, the monitoring apparatus 10 displays an omnidirectional image IMG1 captured by the omnidirectional camera CAk on the first monitor MN1. Further, the monitoring apparatus 10 uses the sound source visual image obtained by stepwise converting the sound pressure value calculated for each pixel constituting the omnidirectional image IMG1 or for each predetermined unit of the pixel into different visual images, and thereby using the omnidirectional image IGM1. And a translucent sound pressure heat map TRL1 obtained by converting the sound pressure heat map into a translucent image is generated and displayed on the second monitor MN2.

  In FIG. 22, the monitoring device 10 displays the omnidirectional image IMG1 on the first monitor MN1 and the translucent sound pressure heat map TRL1 on a separate monitor from the second monitor MN2, but in the omnidirectional image IMG1, For example, the omnidirectional image IMG1 may be displayed in the window, and the translucent sound pressure heat map TRL1 may be displayed in another window.

  Furthermore, the monitoring apparatus 10 displays an omnidirectional image IMG1A in which the translucent sound pressure heat map TRL1 is superimposed on the omnidirectional image IMG1 on the first monitor MN1 (see FIG. 23). FIG. 23 is a diagram illustrating a display screen example of the first monitor MN1 in which the omnidirectional image IMG1 and the translucent sound pressure heat map TRL1 are displayed in an overlay manner.

  In FIG. 23, in the first monitor MN1, the unmanned air vehicle dn detected by the monitoring device 10 is shown on the upper right side of the omnidirectional image IMG1A in FIG. Further, semi-transparent sound pressure heat maps corresponding to the unmanned air vehicle dn and the sound source generated within the range of the imaging field angle of the omnidirectional camera CAk are superimposed and displayed.

  As described with reference to FIG. 24, in this embodiment, if the sound pressure value for each pixel calculated by the sound source direction detection unit 34 is equal to or less than the first threshold th1, a colorless translucent color (that is, colorless) is used. If the sound pressure value is greater than the first threshold th1 and less than or equal to the second threshold th2, it is displayed in a blue translucent color, and the sound pressure value is greater than the second threshold th2 and less than or equal to the third threshold th3. If the sound pressure value is larger than the third threshold th3, it is displayed in a red translucent color.

  In FIG. 23, for example, the sound pressure value is larger than the third threshold th3 in the vicinity of the rotor wing and the rotor around the housing center of the unmanned air vehicle dn, so that the unillustrated aircraft dn is drawn in the translucent red regions RD1A, RD2A, RD3A, and RD4A. . Similarly, semi-transparent pink areas PD1A, PD2A, PD3A, and PD4A indicating that the sound pressure value is next to the semi-transparent red area are drawn around the semi-transparent red area. Similarly, semi-transparent blue areas BD1A, BD2A, BD3A, and BD4A are drawn around the semi-transparent pink area and indicate that the sound pressure value is the second highest after the semi-transparent pink area.

  FIG. 23 also shows that a sound source exists in an office building, and the sound pressure value for each pixel calculated by the sound source direction detecting unit 34 exceeds the third threshold th3 or its pixel. Translucent red regions R1A, R2A, R3A, R4A, R5A, R6A, and R7A are drawn for the set. Similarly, semi-transparent pink areas P1A, P2A, P3A, P4A, and P5A indicating that the sound pressure value is next to the semi-transparent red area are drawn around the semi-transparent red area of the office building. Similarly, semi-transparent blue areas B1A and B2A indicating that the sound pressure value is next to the semi-transparent pink area are drawn around the semi-transparent pink area of the office building. In the other areas of the omnidirectional image IMG1A, since the sound pressure value is equal to or less than the first threshold th1, the area is drawn in a colorless area.

  As described above, in the present embodiment, the monitoring device 10 displays the translucent sound pressure heat map TRL1 different from that in the first embodiment on the omnidirectional image IMG1 and displays it on the first monitor MN1. The user can visually determine the position of the sound source that appears in the IMG 1 and the volume of the sound at that position, and the visibility of the omnidirectional image IMG 1 can be prevented from being deteriorated.

  Next, operation | movement of the unmanned air vehicle detection system 5 of this embodiment is demonstrated in detail with reference to FIG.

  FIG. 24 is a sequence diagram illustrating an example of an operation procedure for overlay display of the omnidirectional image IMG1 and the translucent sound pressure heat map TRL1 in the third embodiment. When the power of each device (for example, the first monitor MN1, the monitoring device 10, the omnidirectional camera CAk, and the microphone array MAk) of the unmanned air vehicle detection system 5 is turned on, the unmanned air vehicle detection system 5 starts its operation. In the description of FIG. 24, the masking area described in the first embodiment may or may not be used. For example, FIG. 24 illustrates a case where a masking area is used. When a masking area is used, it is assumed that information indicating the masking area is registered in the memory 38.

  The monitoring apparatus 10 makes an image distribution request to the omnidirectional camera CAk (S71). The omnidirectional camera CAk starts imaging processing according to power-on in accordance with this image distribution request. In addition, the monitoring apparatus 10 makes a voice distribution request to the microphone array MAk (S72). The microphone array MA starts sound collection processing in response to power-on in accordance with the voice distribution request.

  When the initial operation ends, the omnidirectional camera CAk transmits data of omnidirectional images (for example, still images and moving images) obtained by imaging to the monitoring apparatus 10 via the network NW (S73). In FIG. 24, for the sake of simplicity of explanation, it is assumed that omnidirectional image data is transmitted from the omnidirectional camera CAk, but two-dimensional panoramic image data may be transmitted. The monitoring device 10 converts the omnidirectional image data transmitted from the omnidirectional camera CAk into display data such as NTSC, and outputs it to the first monitor MN1 to instruct the display of the omnidirectional image data (S74). When the display data transmitted from the monitoring apparatus 10 is input, the first monitor MN1 displays the omnidirectional image IMG1 data (see the upper left of FIG. 22) from the omnidirectional camera CAk on the screen.

  Further, the microphone array MAk encodes the audio data of the monitoring area 8 obtained by sound collection via the network NW and transmits the encoded audio data to the monitoring device 10 (S75). In the monitoring device 10, the sound source direction detection unit 34 configures omnidirectional image data in the monitoring area 8 based on the omnidirectional image data captured by the omnidirectional camera CAk and the audio data collected by the microphone array MAk. The sound pressure as the sound parameter is calculated for each pixel to be calculated, and the sound source position in the monitoring area 8 is estimated (S76). This estimated sound source position is used as a reference position of the directivity range BF1 necessary for setting the initial directivity direction when the monitoring apparatus 10 detects the unmanned air vehicle dn.

  Further, in the monitoring device 10, the output control unit 35 uses the sound pressure value for each pixel that configures the omnidirectional image data calculated by the sound source direction detection unit 34, to configure each omnidirectional image data. For each pixel, a sound pressure map in which a calculated value of sound pressure is assigned to the position of the corresponding pixel is generated. Furthermore, the output control unit 35 performs a color conversion process on the sound pressure value for each pixel of the generated sound pressure map to a visual image (for example, a colored image) so that the user can visually and easily discriminate. A translucent sound pressure heat map as shown in the upper right of FIG. 22 is generated (S77). For example, the output control unit 35 firstly generates a sound pressure heat map (see step S7 in FIG. 9) and secondarily generates the sound pressure heat map semi-transparently. This is a procedure of generating a translucent sound pressure heat map by performing transparent processing.

  Further, in the monitoring apparatus 10, the signal processing unit 33 uses the audio data transmitted from the microphone array MAk in step S75 to sequentially provide directivity to the outside of the masking area set by the masking area setting unit 69a. By forming, detection determination of the unmanned air vehicle dn is performed for each directivity direction in which directivity is formed (S78). Since the details of the detection determination process for the unmanned air vehicle dn have been described with reference to FIGS. 10 and 11, the description thereof is omitted here.

  As a result of the detection determination process, when the unmanned air vehicle dn is detected, the output control unit 35 in the monitoring device 10 generates the translucent image generated in step S77 in the omnidirectional image IMG1 displayed on the screen of the first monitor MN1. An instruction is given to superimpose the sound pressure heat map and an identification mark (not shown) representing the unmanned air vehicle dn present in the pointing direction detected in step S78 (S79).

  The first monitor MN1 synthesizes (superimposes) and displays a translucent sound pressure heat map on the omnidirectional image IMG1 according to an instruction from the monitoring device 10, and also displays an identification mark (not shown) representing the unmanned air vehicle dn. Combined (superimposed) and displayed (S80). Thereafter, the process of the unmanned air vehicle detection system 5 returns to step S73, and the processes of steps S73 to S80 are repeated until a predetermined event is detected, for example, the power is turned off.

  As described above, in the unmanned air vehicle detection system 5 of the present embodiment, the monitoring device 10 monitors the sound pressure that specifies the sound level of the monitoring area 8 using the sound data collected by the microphone array MAk. Calculation is performed for each predetermined unit of pixels constituting the captured image (omnidirectional image IMG1) of area 8. The monitoring device 10 converts the sound source visual image obtained by converting the sound pressure into a visual image for each predetermined unit of pixels in all directions of the monitoring area 8 according to the comparison between the calculated value of the sound pressure and the threshold value related to the sound volume. A translucent sound pressure heat map connected so as to correspond to the size of the image is generated. The monitoring device 10 superimposes the translucent sound pressure heat map on the captured image of the monitoring area 8 and displays it on the first monitor MN1.

  Thereby, in the unmanned air vehicle detection system 5, the unmanned air vehicle is detected at any location within the monitoring area 8 of the omnidirectional camera CAk, and any other sound source is present at any other location within the same monitoring area 8. This can be visually presented to the user without degrading the visibility of the captured image of the omnidirectional camera CAk.

  Further, a plurality of threshold values relating to the volume of sound are provided, and the monitoring device 10 converts the sound pressure for each predetermined unit of pixels into different visual images step by step according to the comparison between the sound pressure and the plurality of threshold values. Using the sound source visual image, a translucent sound pressure heat map having a plurality of types of sound source visual images is generated. As a result, the monitoring device 10 uses the visual sound source visual image to indicate the presence of sound pressure having a plurality of types of stages defined by a plurality of thresholds in the omnidirectional image captured by the omnidirectional camera CAk. It is possible to make the user more explicitly discriminate.

  Also in this embodiment, the monitoring apparatus 10 sets the masking area described in the first embodiment, and uses the audio data collected by the microphone array MAk and the information indicating the masking area to use the masking area. Detect unmanned air vehicles outside. When the monitoring device 10 detects an unmanned aerial vehicle outside the masking area, the sound of the sound emitted from the unmanned aerial vehicle around the unmanned air vehicle in the omnidirectional image (in other words, the sound source position of the unmanned air vehicle) is detected. The sound source visual image indicating the size is made translucent and displayed on the first monitor MN1. Thereby, since the monitoring apparatus 10 can exclude the inside of a masking area from the detection target of an unmanned air vehicle, it can suppress the deterioration of the detection accuracy of a masking area, and can also improve the detection speed of an unmanned air vehicle. In addition, the monitoring device 10 can detect the level of the sound level emitted from the unmanned air vehicle at the sound source position of the unmanned air vehicle dn detected outside the masking area, using the translucent image of the sound source visual image. In addition, the visibility of surrounding captured images can be prevented from being deteriorated.

  Also in the present embodiment, the monitoring device 10 sets the correspondence relationship between each threshold value of a plurality of threshold values that defines the volume of sound step by step and a plurality of types of sound source visual images in the captured image of the imaging area. Change accordingly. Based on the calculated sound pressure value and the changed correspondence, the monitoring apparatus 10 connects the sound source visual image for each predetermined unit of the pixel so as to correspond to the size of the captured image in the imaging area. Generate a heat map. Thereby, the monitoring apparatus 10 calculates the sound pressure obtained for each pixel or for each predetermined unit of the pixel constituting the captured image according to the content of the omnidirectional image (captured image) captured by the omnidirectional camera CAk. The correspondence between the value and the sound source visual image corresponding to the calculated value of the sound pressure can be made variable. Therefore, for example, in a place where specific sound pressure calculation values are concentrated, the monitoring device 10 uses a plurality of types of sound source visual images instead of a single color sound source visual image as a sound source visual image in the vicinity. It is possible to allow the user to grasp in detail the distribution of the loudness of the sound source that appears in the captured image clearly and finely.

  While various embodiments have been described above with reference to the drawings, it goes without saying that the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

  According to the present invention, an unmanned air vehicle is detected at which location in the imaging area of the camera device, and what kind of sound source exists at what other location in the same imaging area. The present invention is useful as a monitoring system and a monitoring method for visually presenting to a user without deteriorating visibility.

5 Unmanned flying object detection system 10 Monitoring device 31 Communication unit 32 Operation unit 33 Signal processing unit 34 Sound source direction detection unit 35 Output control unit 37 Speaker device (SPK)
38 Memory 39 Setting management unit 63 Directivity processing unit 64 Frequency analysis unit 65 Object detection unit 66 Detection result determination unit 67 Scan control unit 68 Detection direction control unit 69a Masking area setting unit 69b Threshold adjustment units CA1, CAk, CAn Omnidirectional Camera CZ1, CZk, CZn PTZ camera MA1, MAk, MAn Microphone array MN1 First monitor MN2 Second monitor NW Network RC recorder UD1, UDn Sound source detection unit

Claims (5)

A camera for imaging the imaging area;
A microphone array for picking up sound in the imaging area;
A monitor for displaying a captured image of the imaging area captured by the camera;
A sound parameter deriving unit for deriving a sound parameter for specifying the volume of sound in the imaging area for each predetermined unit of pixels constituting the captured image in the imaging area based on the sound collected by the microphone array; ,
The sound source visual image obtained by converting the sound parameter into a visual image for each predetermined unit of the pixel according to the comparison between the sound parameter derived by the sound parameter deriving unit and a threshold value related to sound volume is captured. A signal processing unit that generates a semi-transparent map of a sound parameter map connected so as to correspond to the size of a captured image of the area,
The signal processing unit superimposes the semi-transparent map on a captured image of the imaging area and displays it on the monitor.
Monitoring system. The monitoring system according to claim 1,
A plurality of thresholds related to the volume of the sound are provided,
The signal processing unit uses a sound source visual image obtained by gradually converting the sound parameter into different visual images for each predetermined unit of the pixel according to the comparison between the sound parameter and the plurality of threshold values. Generating a translucent map of the sound parameter map having the sound source visual image of a type;
Monitoring system. The monitoring system according to claim 1,
Based on the sound collected by the microphone array, a masking area setting unit for setting a masking area for excluding detection of an unmanned air vehicle appearing in a captured image of the imaging area;
A detection unit for detecting the unmanned air vehicle based on the voice collected by the microphone array and the masking area set by the masking area setting unit;
When the unmanned air vehicle is detected outside the masking area, the signal processing unit indicates the loudness of the unmanned air vehicle around the unmanned air vehicle in the captured image of the imaging area. The sound source visual image is made translucent and displayed on the monitor.
Monitoring system. The monitoring system according to claim 2,
A threshold adjustment unit that changes the setting of the correspondence between each of the plurality of thresholds that define the volume of sound in stages and the plurality of types of sound source visual images according to the captured images of the imaging area; ,
The signal processing unit, based on the sound parameter derived by the sound parameter deriving unit and the correspondence changed by the threshold adjustment unit, generates a sound source visual image for each predetermined unit of the pixel in the imaging area. Generate a semi-transparent map of the sound parameter map connected to correspond to the size of the captured image,
Monitoring system. A monitoring method in a monitoring system having a camera and a microphone array,
The imaging area is imaged by the camera,
The microphone array picks up the sound of the imaging area,
Displaying a captured image of the imaging area captured by the camera on a monitor;
Based on the sound collected by the microphone array, a sound parameter for specifying the sound volume of the imaging area is derived for each predetermined unit of pixels constituting the captured image of the imaging area,
A sound source visual image obtained by converting the sound parameter into a visual image for each predetermined unit of the pixel according to a comparison between the derived sound parameter and a threshold value related to a loudness is a size of a captured image of the imaging area. Generate a semi-transparent map of sound parameter maps connected to correspond to
The generated translucent map is superimposed on the captured image of the imaging area and displayed on the monitor.
Monitoring method.