JP6664119B2 - Monitoring system and monitoring method - Google Patents

Description

  The present invention relates to a monitoring system and a monitoring method for monitoring an imaging area of a camera device in which an unmanned aerial vehicle is flying, for example.

  2. Description of the Related Art Conventionally, there is known a flying object monitoring apparatus capable of detecting the presence of an object and detecting the direction in which the object is flying by using a plurality of sound detectors for detecting sounds generated in a monitoring area for each direction. (See, for example, Patent Document 1). When the processing device of the flying object monitoring apparatus detects the flying of the flying object and the direction of the flying object by sound detection using a non-directional microphone, the processing device turns the monitoring camera in the direction in which the flying object has flew. Further, the processing device displays an image captured by the monitoring camera on a display device.

JP 2006-168421 A

  Patent Document 1 discloses that a sound pressure at a frequency specific to a flying object such as a helicopter or a Cessna is compared with a predetermined setting level, and if the sound pressure is equal to or higher than the setting level, it is determined that the flying object is a monitoring target. ing.

  However, Patent Literature 1 does not take into account that the measured sound pressure quantitatively indicates which of a plurality of specified sound pressures. For this reason, when any sound is detected in the imaging area of the camera device, the loudness of the sound is determined by detailed visual information of fine sound regardless of the loudness of the sound at the detected sound source position. There was a problem that it could not be presented in a typical way.

  The present invention has been devised in view of the above-described conventional situation, and irrespective of the magnitude of the sound volume at the sound source position detected in the imaging area of the camera device, the sound volume at the sound source position is finely and stepwisely adjusted. And a monitoring system and a monitoring method that contribute to accurate grasp of the loudness of the sound at the sound source position to the user.

The present invention provides a camera that captures an image of an imaging area, a microphone array that collects sound of the imaging area, a monitor that displays a captured image of the imaging area captured by the camera, and a captured image of the imaging area. a predetermined number of pixels constituting a sound parameter derivation unit of the imaging area, based on sound collected by the microphone array, sound parameters specifying the loudness of the imaging area, the sound of the image pickup area A sound parameter deriving unit for deriving for each parameter deriving unit, and stepping the sound parameter into different visual images according to a comparison between the sound parameter derived by the sound parameter deriving unit and a plurality of thresholds relating to a loudness. specifically the converted sound visual images, a signal processing unit to be displayed on the monitor superimposed on each sound parameter derivation unit of the imaging area, the For example, the sound parameter deriving unit, a part of the rectangular area in the captured image of the imaging area of the sound source visual image is superimposed is designated, the sound parameter derivation unit of the imaging area of the imaging area Provided is a monitoring system that derives sound parameters of the rectangular range for each value divided by a size ratio between a captured image and the rectangular range.

Further, the present invention provides a camera for capturing an image of an imaging area, a microphone array for picking up sound of the imaging area, a monitor for displaying a captured image of the imaging area captured by the camera, and an imaging of the imaging area. Using a predetermined number of pixels constituting an image as a unit for deriving a sound parameter of the imaging area, a sound parameter for specifying a loudness of the sound in the imaging area based on the sound collected by the microphone array, A sound parameter deriving unit that derives for each sound parameter deriving unit, each of a plurality of thresholds that defines the loudness of the sound in stages, and a visual image in which the sound parameters are different according to a comparison with each of the thresholds. The setting of the correspondence relationship with the sound source visual image that is stepwise converted is derived by the threshold adjustment unit that changes according to the captured image of the imaging area and the sound parameter derivation unit. Based on said correspondence relation changed by the sound parameter and the threshold adjustment unit is, for each sound parameter derivation unit of the imaging area, the sound source visual image corresponding to the sound parameters captured image of the imaging area And a signal processing unit for superimposing the signal on the monitor and displaying the signal on the monitor.

Further, the present invention is a monitoring method in a monitoring system having a camera and a microphone array, wherein the camera captures an image of an imaging area, the microphone array collects sound of the imaging area, and the camera A captured image of the imaged area is displayed on a monitor, and a predetermined number of pixels constituting the imaged image of the imaged area are used as sound parameter derivation units of the imaged area, based on sound collected by the microphone array. A sound parameter specifying the loudness of the sound in the imaging area is derived for each sound parameter deriving unit of the shooting area, and the derived sound parameter is compared with a plurality of thresholds relating to the loudness of the sound. Te, the source visual images stepwise convert the sound parameters to different visual images, sound parameters guiding the imaging area Superimposed on each unit is characterized in that displayed on the monitor, further, a part of the rectangular area in the captured image of the image pickup area source visual image is superimposed is designated, the imaging area A monitoring method is provided for deriving a sound parameter of the rectangular range for each value obtained by dividing a sound parameter deriving unit by a size ratio between a captured image of the imaging area and the rectangular range.

Further, the present invention is a monitoring method in a monitoring system having a camera and a microphone array, wherein the camera captures an image of an imaging area, the microphone array collects sound of the imaging area, and the camera A captured image of the imaged area is displayed on a monitor, and a predetermined number of pixels constituting the imaged image of the imaged area are used as sound parameter derivation units of the imaged area, based on sound collected by the microphone array. A sound parameter that specifies the loudness of the sound in the imaging area is derived for each sound parameter deriving unit of the shooting area, and each of a plurality of thresholds that defines the loudness of the sound in a stepwise manner. Setting the correspondence with a sound source visual image that is stepwise converted to a visual image in which the sound parameters are different according to a comparison with a threshold value, Changed according to the rear of the captured image, the derived based on said correspondence has been changed to the sound parameters for each sound parameter derivation unit of the imaging area, the sound source visual image corresponding to the sound parameter A monitoring method is provided, which superimposes on a captured image of the imaging area and displays the image on the monitor.

  According to the present invention, regardless of the loudness of the sound at the sound source position detected in the imaging area of the camera device, the loudness of the sound at the sound source position can be finely presented in a stepwise manner. It is possible to contribute to accurate understanding of the size by the user.

FIG. 1 is a diagram illustrating an example of a schematic configuration of an unmanned aerial vehicle detection system of each embodiment. Diagram showing an example of the appearance of a sound source detection unit Block diagram showing an example of the internal configuration of the microphone array in detail Block diagram showing an example of the internal configuration of the omnidirectional camera in detail The block diagram which shows an example of the internal structure of a PTZ camera in detail. Block diagram showing an example of the internal configuration of the monitoring device in detail A timing chart showing an example of a pattern of a detection sound signal of the unmanned aerial vehicle registered in the memory A timing chart showing an example of a frequency change of a detected sound signal obtained as a result of the frequency analysis processing Sequence diagram illustrating an example of an operation procedure of detecting an unmanned aerial vehicle and displaying a detection result in the first embodiment. 9 is a flowchart for explaining an example of the details of the operation procedure for unmanned flying object detection determination in step S8 in FIG. 9. The figure which shows an example of a mode that a directional direction is scanned in order within a monitoring area, and an unmanned aerial vehicle is detected. The figure which shows the example of a display screen of the 1st monitor when a masking area is not set. Explanatory diagram showing a display example of a masking area in a time series during an automatic learning process Sequence diagram illustrating an example of an operation procedure for setting a masking area according to the first embodiment. The figure which shows the example of the display screen of the 1st monitor when a masking area is set. Schematic description of dynamic change of display resolution of sound pressure heat map in the second embodiment 9 is a flowchart illustrating an example of an operation procedure of dynamically changing a display resolution of a sound pressure heat map according to the second embodiment. Schematic explanatory diagram of a threshold width adjustment according to the frequency distribution of sound pressure values and a display result of a captured image accompanying the width adjustment in the second embodiment. (A), (B) Schematic explanatory diagram of a setting change of a width between thresholds defining use of a sound source visual image in the second embodiment. FIG. 11 is a schematic explanatory diagram of a display of a captured image according to a change in setting of a width between thresholds that defines use of a red image and a group of blue images in the second embodiment. 9 is a flowchart for explaining an example of an operation procedure for changing the setting of the width between threshold values in the second embodiment. FIG. 9 is a schematic explanatory diagram of an overlay display of an omnidirectional image and a translucent sound pressure heat map in the third embodiment. The figure which shows the example of the display screen of the 1st monitor on which the omnidirectional image and the translucent sound pressure heat map were overlaid. FIG. 7 is a sequence diagram illustrating an example of an operation procedure of overlay display of an omnidirectional image and a translucent sound pressure heat map in the third embodiment.

  Hereinafter, each embodiment which specifically discloses a monitoring system and a monitoring method according to the present invention will be described in detail with reference to the drawings as appropriate. However, an unnecessary detailed description may be omitted. For example, a detailed description of a well-known item or a redundant description of substantially the same configuration may be omitted. This is to prevent the following description from being unnecessarily redundant and to facilitate understanding of those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the claimed subject matter.

  Hereinafter, as an example of a monitoring system according to the present invention, an unmanned aerial vehicle detection system for detecting an unmanned aerial vehicle (for example, a drone or a radio control helicopter) to be monitored will be described as an example. Further, the present invention is not limited to the monitoring system, but can be defined as a monitoring method executed in the monitoring system.

  Hereinafter, the user of the unmanned aerial vehicle detection system (for example, a guard who looks around the guard area and guards) is simply referred to as a “user”.

  FIG. 1 is a diagram illustrating an example of a schematic configuration of an unmanned aerial vehicle detection system 5 of each embodiment. The unmanned aerial vehicle detection system 5 detects an unmanned aerial vehicle dn (for example, see FIG. 12) that is intended by the user as a detection target. The unmanned aerial vehicle dn is, for example, a drone that flies autonomously using a GPS (Global Positioning System) function, a radio-controlled helicopter that flies by radio control of a third party, or the like. Such an unmanned aerial vehicle dn is used, for example, for aerial photography of a target, monitoring, transportation of goods, and the like.

  In each embodiment, a multi-copter type drone equipped with a plurality of rotors (in other words, rotating wings) is illustrated as an unmanned aerial vehicle dn. In a multi-copter type drone, when the number of blades of the rotor is two, a harmonic having a frequency twice as large as a specific frequency, and a harmonic having a frequency multiplied by the specific frequency are generally generated. Similarly, when the number of blades of the rotor is three, a harmonic having a frequency three times as high as the specific frequency and a harmonic having a frequency multiplied by three times are generated. The same applies when the number of rotor blades is four or more.

  The unmanned air vehicle detection system 5 is configured to include a plurality of sound source detection units UD1,..., UDk,..., UDn, a monitoring device 10, a first monitor MN1, a second monitor MN2, and a recorder RC. n is a natural number of 2 or more. , UDk,..., UDn are mutually connected to the monitoring device 10 via the network NW. k is a natural number of 1 to n. Each sound source detection unit (for example, sound source detection unit UD1) has a configuration including a microphone array MA1, an omnidirectional camera CA1, and a PTZ camera CZ1, and the other sound source detection units UDk have the same configuration.

  Note that the sound source detection units UDk or simply the sound source detection units UD, unless it is necessary to distinguish the individual sound source detection units. Similarly, the microphone array MAk or the microphone array MA (see FIG. 2), the omnidirectional camera CAk or the omnidirectional camera CA (see FIG. 2) unless the individual microphone array, omnidirectional camera, and PTZ camera need to be particularly distinguished. ), PTZ camera CZk or PTZ camera CZ (see FIG. 2).

  In the sound source detection unit UDk, the microphone array MAk collects sound in all directions in the area to be monitored (that is, the monitoring area) in which the own apparatus is installed in a non-directional state. The microphone array MAk has a housing 15 (see FIG. 2) in which a cylindrical opening having a predetermined width is formed in the center. The sound to be picked up by the microphone array MAk includes, for example, mechanical operation sounds such as drones, sounds emitted by human beings, and other sounds, and is limited to sounds in the audible frequency range (that is, 20 Hz to 23 kHz). Alternatively, a low-frequency sound lower than the audio frequency or an ultrasonic sound higher than the audio frequency may be included.

  The microphone array MAk includes a plurality of omnidirectional microphones M1 to Mq (see FIG. 3). q is a natural number of 2 or more. The microphones M1 to Mq are arranged at predetermined intervals (for example, uniform intervals) concentrically around the opening provided in the housing 15 along the circumferential direction. As the microphones M1 to Mq, for example, an electret condenser microphone (ECM) is used. The microphone array MAk transmits, to the monitoring device 10 via the network NW, sound data of sounds obtained by collecting the microphones M1 to Mq. Note that the arrangement of the microphones M1 to Mq is merely an example, and other arrangements (for example, a square arrangement or a rectangular arrangement) may be used. However, the microphones M1 to Mq are arranged at equal intervals. Is preferably performed.

  The microphone array MAk includes a plurality of microphones M1 to Mq (for example, q = 32) and a plurality of amplifiers (amplifiers) PA1 to PAq (see FIG. 3) for amplifying output signals of the plurality of microphones M1 to Mq, respectively. . Analog signals output from the amplifiers are converted into digital signals by A / D converters A1 to Aq (see FIG. 3) described later. Note that the number of microphones in the microphone array MAk is not limited to 32, but may be another number (for example, 16, 64, or 128).

  An omnidirectional camera CAk substantially matching the volume of the opening is accommodated inside the opening formed in the center of the housing 15 (see FIG. 2) of the microphone array MAk. That is, the microphone array MAk and the omnidirectional camera CAk are disposed integrally and with their respective housing centers coaxial with each other (see FIG. 2). The omnidirectional camera CAk is a camera equipped with a fisheye lens 45a (see FIG. 4) capable of capturing an omnidirectional image of a monitoring area (see above) as an imaging area of the omnidirectional camera CAk. In each embodiment, the sound collection area of the microphone array MAk and the imaging area of the omnidirectional camera CAk are both described as a common monitoring area, but the sound collection area and the imaging area have the same spatial size (for example, volume). It does not have to be. For example, the volume of the sound collection area may be larger or smaller than the volume of the imaging area. In short, it is sufficient that the sound collection area and the imaging area have a common space portion. The omnidirectional camera CAk functions as, for example, a surveillance camera that can image an imaging area in which the sound source detection unit UDk is installed. That is, the omnidirectional camera CAk has an angle of view of, for example, 180 ° in the vertical direction and 360 ° in the horizontal direction, and captures an image of the monitoring area 8 (see FIG. 11), which is, for example, a semi-celestial sphere, as an imaging area.

  In each sound source detection unit UDk, the omnidirectional camera CAk is fitted inside the opening of the housing 15 so that the omnidirectional camera CA and the microphone array MA are coaxially arranged. As described above, since the optical axis of the omnidirectional camera CAk and the central axis of the housing of the microphone array MAk match, the imaging area and the sound collection area in the axial circumferential direction (that is, the horizontal direction) become substantially the same, The position of the subject in the omnidirectional image captured by the omnidirectional camera CAk (in other words, the direction indicating the position of the subject viewed from the omnidirectional camera CAk) and the position of the sound source to be picked up by the microphone array MAk (in other words, The direction indicating the position of the sound source viewed from the microphone array MAk) can be expressed in the same coordinate system (for example, coordinates indicated by (horizontal angle, vertical angle)). Each sound source detection unit UDk is attached so that, for example, the sound pickup surface and the image pickup surface are directed upward in the vertical direction in order to detect the unmanned aerial vehicle dn flying in the sky (see FIG. 2). .

  The monitoring device 10 is configured using, for example, a PC (Personal Computer) or a server. The monitoring apparatus 10 forms directivity (that is, beam forming) in which an arbitrary direction is set as a main beam direction based on a user's operation, with respect to the omnidirectional sound collected by the microphone array MAk. The sound in the direction can be emphasized. The technology related to the directivity control processing of the sound data for beamforming the sound collected by the microphone array MAk is a known technology, for example, as shown in Reference Documents 1 and 2.

(Reference Patent Document 1) JP-A-2014-143678 (Reference Patent Document 2) JP-A-2015-029241

  The monitoring device 10 processes the captured image using an image captured by the omnidirectional camera CAk (hereinafter, may be abbreviated as “captured image”) to generate an omnidirectional image. The omnidirectional image may be generated by the omnidirectional camera CAk instead of the monitoring device 10.

  The monitoring device 10 adds a sound pressure heat map to a captured image captured by the omnidirectional camera CAk based on a calculated value of a sound parameter (for example, a sound pressure to be described later) that specifies the volume of the sound collected by the microphone array MAk. (See FIG. 15) is superimposed and output to the first monitor MN1 or the like for display.

  In addition, the monitoring device 10 displays, on the omnidirectional image IMG1, a visual image (for example, an identification mark (not shown)) that allows the user to visually distinguish the detected unmanned aerial vehicle dn from the unmanned aerial vehicle dn of the first monitor MN1. May be displayed at the position. The visual image means, for example, information that is displayed in the omnidirectional image IMG1 to such an extent that it can be clearly distinguished from other subjects when the user views the omnidirectional image IMG1, and so on. .

  The first monitor MN1 displays an omnidirectional image IMG1 captured by the omnidirectional camera CAk. The second monitor MN2 displays an omnidirectional image IMG1 captured by the omnidirectional camera CAk. Further, the first monitor MN1 generates and displays a composite image in which the above-described identification mark (not shown) is superimposed on the omnidirectional image IMG1. In FIG. 1, two monitors, the first monitor MN1 and the second monitor MN2, are connected to the monitoring device 10, but, for example, only the first monitor MN1 may be connected. Further, the first monitor MN1 or the second monitor MN2, or the first monitor MN1 and the second monitor MN2 may be configured as an integrated device with the monitoring device 10.

  The recorder RC is configured using a semiconductor memory such as a hard disk (Hard Disk Drive) or a flash memory, and is transmitted from various image data (see below) generated by the monitoring device 10 and the respective sound source detection units UDk. Saves various omnidirectional image and audio data. Note that the recorder RC may be configured as an integrated device with the monitoring device 10 or may be omitted from the configuration of the unmanned flying object detection system 5.

  In FIG. 1, the plurality of sound source detection units UDk and the monitoring device 10 have a communication interface and are connected to each other via a network NW so that data communication is possible. The network NW may be a wired network (for example, an intranet, the Internet, a wired LAN (Local Area Network), or a wireless network (for example, a wireless LAN). The sound source detection unit UDk and the monitoring device 10 are connected via the network NW. The monitoring device 10, the first monitor MN1, the second monitor MN2, and the recorder RC are all installed in a monitoring room RM where the user is resident at the time of monitoring.

  FIG. 2 is a diagram illustrating an example of the appearance of the sound source detection unit UD. The sound source detection unit UD includes the microphone array MA, the omnidirectional camera CA, the PTZ camera CZ, and a support base 70 that mechanically supports these units. The support base 70 includes a tripod 71, two rails 72 fixed to a top plate 71 a of the tripod 71, and a first mounting plate 73 and a second mounting plate 74 respectively mounted on both ends of the two rails 72. Has a combined structure.

  The first mounting plate 73 and the second mounting plate 74 are mounted so as to straddle the two rails 72 and have substantially the same plane. Further, the first mounting plate 73 and the second mounting plate 74 are slidable on the two rails 72, and are adjusted and fixed to positions separated or approached from each other.

  The first mounting plate 73 is a disk-shaped plate. An opening 73a is formed at the center of the first mounting plate 73. The housing 15 of the microphone array MA is accommodated and fixed in the opening 73a. On the other hand, the second mounting plate 74 is a substantially rectangular plate material. An opening 74a is formed in a portion near the outside of the second mounting plate 74. The PTZ camera CZ is housed and fixed in the opening 74a.

  As shown in FIG. 2, the optical axis L1 of the omnidirectional camera CA housed in the housing 15 of the microphone array MA and the optical axis L2 of the PTZ camera CZ mounted on the second mounting plate 74 are in the initial installation state. Are set to be parallel to each other.

  The tripod 71 is supported on the ground surface by three legs 71b. The position of the top plate 71a can be freely moved in the vertical direction with respect to the ground surface by manual operation, and the tripod 71 can be moved in the pan and tilt directions. The direction of the plate 71a can be adjusted. Accordingly, the sound collection area of the microphone array MA (in other words, the imaging area of the omnidirectional camera CA or the monitoring area of the unmanned aerial vehicle detection system 5) can be set to an arbitrary direction.

  FIG. 3 is a block diagram showing an example of the internal configuration of the microphone array MAk in detail. The microphone array MAk illustrated in FIG. 3 includes a plurality of microphones M1 to Mq (for example, q = 32), a plurality of amplifiers (amplifiers) PA1 to PAq that respectively amplify output signals of the plurality of microphones M1 to Mq, and each of the amplifiers PA1 to PAq. Are configured to include a plurality of A / D converters A1 to Aq for converting analog signals output from the A / D converter into digital signals, a compression processing unit 25, and a transmission unit 26, respectively.

  The compression processing unit 25 generates audio data packets based on the digital audio signals output from the A / D converters A1 to An. The transmitting unit 26 transmits the packet of the audio data generated by the compression processing unit 25 to the monitoring device 10 via the network NW.

  As described above, the microphone array MAk amplifies the output signals of the microphones M1 to Mq with the amplifiers PA1 to PAq, and converts the output signals into digital audio signals with the A / D converters A1 to Aq. Thereafter, the microphone array MAk generates a packet of audio data in the compression processing unit 25, and transmits the packet of the audio data to the monitoring device 10 via the network NW.

  FIG. 4 is a block diagram showing an example of the internal configuration of the omnidirectional camera CAk in detail. The omnidirectional camera CAk shown in FIG. 4 has a configuration including a CPU 41, a communication unit 42, a power management unit 44, an image sensor 45, a fisheye lens 45a, a memory 46, and a network connector 47.

  The CPU 41 performs signal processing for integrally controlling the operation of each unit of the omnidirectional camera CAk, data input / output processing with other units, data calculation processing, and data storage processing. Instead of the CPU 41, a processor such as an MPU (Micro Processing Unit) or a DSP (Digital Signal Processor) may be provided.

  For example, the CPU 41 generates two-dimensional panoramic image data (that is, two-dimensional panorama-converted image data) obtained by cutting out an image in a specific range (direction) from the omnidirectional image data according to a specification of a user who operates the monitoring device 10. And store it in the memory 46.

  The image sensor 45 is configured using, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor, and captures an optical image of subject light from an imaging area condensed by a fisheye lens 45a on a light receiving surface. By performing the processing, omnidirectional image data of the imaging area is obtained.

  The fish-eye lens 45a receives and converges subject light from all directions in the imaging area, and forms an optical image of the subject light on a light receiving surface (not shown) of the image sensor 45.

  The memory 46 stores a ROM 46z in which a program for defining the operation of the omnidirectional camera CAk and data of set values are stored, and omnidirectional image data or cutout image data or work data in which a part of the range is cut out. And a memory card 46x which is connected to the omnidirectional camera CAk so as to be freely inserted and removed and stores various data.

  The communication unit 42 is a network interface (I / F) that controls data communication with a network NW connected via the network connector 47.

  The power management unit 44 supplies DC power to each unit of the omnidirectional camera CAk. Further, the power management unit 44 may supply DC power to devices connected to the network NW via the network connector 47.

  The network connector 47 is a connector that can transmit omnidirectional image data or two-dimensional panoramic image data to the monitoring device 10 via the network NW and can supply power via a network cable.

  FIG. 5 is a block diagram showing an example of the internal configuration of the PTZ camera CZk in detail. The same parts as those in the omnidirectional camera CAk are denoted by the same reference numerals as those in FIG. 4 and the description thereof is omitted. The PTZ camera CZk is a camera that can adjust the optical axis direction (also referred to as an imaging direction) in response to an angle-of-view change instruction from the monitoring device 10.

  Like the omnidirectional camera CAk, the PTZ camera CZk includes a CPU 51, a communication unit 52, a power management unit 54, an image sensor 55, an imaging lens 55a, a memory 56, a network connector 57, an imaging direction control unit 58, and a lens driving motor. 59. When there is an instruction to change the angle of view of the monitoring device 10, the CPU 51 notifies the imaging direction control unit 58 of the instruction to change the angle of view.

  The imaging direction control unit 58 controls at least one of the pan direction and the tilt direction for the imaging direction of the PTZ camera CZ according to the angle-of-view change instruction notified from the CPU 51, and further changes the zoom magnification as necessary. Is output to the lens drive motor 59. The lens drive motor 59 drives the imaging lens 55a according to the control signal, changes the imaging direction (the direction of the optical axis L2 shown in FIG. 2), and adjusts the focal length of the imaging lens 55a to increase the zoom magnification. change.

  The imaging lens 55a is configured using one or two or more lenses. In the imaging lens 55a, the optical axis directions of the pan rotation and the tilt rotation are changed by driving the lens driving motor 59 according to the control signal from the imaging direction control unit 58.

  FIG. 6 is a block diagram illustrating an example of the internal configuration of the monitoring device 10 in detail. The monitoring device 10 illustrated in FIG. 6 has a configuration including at least a communication unit 31, an operation unit 32, a signal processing unit 33, a speaker device (SPK) 37, a memory 38, and a setting management unit 39.

  The communication unit 31 receives the omnidirectional image data or the two-dimensional panoramic image data transmitted by the omnidirectional camera CAk and the audio data transmitted by the microphone array MAk, and outputs the audio data to the signal processing unit 33.

  The operation unit 32 is a user interface (UI) for notifying the content of the user's input operation to the signal processing unit 33, and includes a pointing device such as a mouse and a keyboard. The operation unit 32 is arranged, for example, in correspondence with each screen of the first monitor MN1 and the second monitor MN2, and is configured using a touch panel or a touch pad that can be directly operated by a user's finger or a stylus pen. Is also good.

  The operation unit 32 is a red area of the sound pressure heat map (see FIG. 15) displayed on the first monitor MN1 and the second monitor MN2 so as to be superimposed on the captured image (omnidirectional image IMG1) of one of the omnidirectional cameras CAk. When RD <b> 1 is specified by the user, coordinate data indicating the specified position is obtained and output to the signal processing unit 33. The signal processing unit 33 reads the sound data collected by the microphone array MAk corresponding to the omnidirectional camera CAk from the memory 38, and directs the sound data from the microphone array MAk toward the actual sound source position corresponding to the designated position. After the characteristics are formed, the signal is output from the speaker device 37. Thereby, the user can clearly check not only the unmanned aerial vehicle dn but also the sound at the position designated by the user himself on the captured image (omnidirectional image IMG1) in an emphasized state.

  The signal processing unit 33 is configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor), and performs control processing for integrally controlling the operation of each unit of the monitoring device 10. It performs data input / output processing with other units, data calculation (calculation) processing, and data storage processing. The signal processing unit 33 includes a sound source direction detection unit 34, an output control unit 35, a directivity processing unit 63, a frequency analysis unit 64, an object detection unit 65, a detection result determination unit 66, a scan control unit 67, and a detection direction control unit 68. , A masking area setting unit 69a and a threshold adjustment unit 69b. The monitoring device 10 is connected to the first monitor MN1 and the second monitor MN2.

  The sound source direction detection unit 34 uses the sound data of the sound in the monitoring area 8 collected by the microphone array MAk in accordance with, for example, a known whitening cross-correlation method (CSP (Cross-power Spectrum Phase analysis) method). Is estimated. In the CSP method, the sound source direction detection unit 34 divides the monitoring area 8 shown in FIG. 11 into a plurality of blocks, and when a sound is collected by the microphone array MAk, the sound pressure or the volume exceeds a threshold value for each block. By determining whether or not there is a sound, the position of the sound source in the monitoring area 8 can be roughly estimated.

  In addition, the sound source direction detection unit 34 as a sound parameter derivation unit is configured to generate an omnidirectional image of the monitoring area 8 based on omnidirectional image data captured by the omnidirectional camera CAk and audio data collected by the microphone array MAk. The sound pressure as a sound parameter is calculated for each pixel constituting the data. The sound source direction detection unit 34 outputs a calculated value, which is a calculation result of the sound pressure, to the output control unit 35. The process of calculating the sound pressure is a known technique, and a detailed description of the process is omitted.

  The setting management unit 39 has a coordinate conversion formula for coordinate conversion of a position designated by the user on the screen of the first monitor MN1 on which the omnidirectional image data captured by the omnidirectional camera CAk is displayed. This coordinate conversion formula is based on, for example, a physical distance difference between the installation position of the omnidirectional camera CAk (see FIG. 2) and the installation position of the PTZ camera CZk (see FIG. 2). This is a mathematical expression for converting the coordinates of the position (that is, (horizontal angle, vertical angle)) into coordinates in the direction viewed from the PTZ camera CZk.

  The signal processing unit 33 uses the above-mentioned coordinate conversion formula held by the setting management unit 39 to set the position specified by the user from the installation position of the PTZ camera CZk with reference to the installation position of the PTZ camera CZk (see FIG. 2). Are calculated (θMAh, θMAv) indicating the directivity direction toward the actual sound source position corresponding to. θMAh is a horizontal angle in the direction toward the actual sound source position corresponding to the position specified by the user when viewed from the installation position of the PTZ camera CZk. θMAv is the vertical angle in the direction toward the actual sound source position corresponding to the position specified by the user, as viewed from the installation position of the PTZ camera CZk. As shown in FIG. 2, the distance between the omnidirectional camera CAk and the PTZ camera CZk is known, and the respective optical axes L1 and L2 are parallel. It can be realized by geometric calculation. The sound source position is an actual sound source position corresponding to the position specified by the operation unit 32 by operating the user's finger or stylus pen with respect to the video data displayed on the first monitor MN1 and the second monitor MN2.

  As shown in FIG. 2, in each embodiment, the omnidirectional camera CAk and the microphone array MAk are respectively arranged so that the optical axis direction of the omnidirectional camera CAk and the center axis of the housing of the microphone array MAk are coaxial. Have been. For this reason, the coordinates of the user-specified position derived by the omnidirectional camera CAk in accordance with the user's specification on the first monitor MN1 on which the omnidirectional image data is displayed are determined according to the sound enhancement direction (both directions) from the microphone array MAk. The same). In other words, when the user specifies the first monitor MN1 on which the omnidirectional image data is displayed (or the second monitor MN2 is also acceptable), the monitoring device 10 changes the coordinates of the specified position on the omnidirectional image data to all. Transmit to azimuth camera CAk. Thereby, the omnidirectional camera CAk uses the coordinates of the designated position transmitted from the monitoring device 10 to indicate the coordinates (horizontal angle, vertical angle) indicating the direction of the sound source position corresponding to the user-specified position as viewed from the omnidirectional camera CAk. Angle). The calculation processing in the omnidirectional camera CAk is a known technique, and thus the description is omitted. The omnidirectional camera CAk transmits the calculation result of the coordinates indicating the direction of the sound source position to the monitoring device 10. The monitoring device 10 can use the coordinates (horizontal angle, vertical angle) calculated by the omnidirectional camera CAk as coordinates (horizontal angle, vertical angle) indicating the direction of the sound source position viewed from the microphone array MAk.

  However, when the omnidirectional camera CAk and the microphone array MAk are not coaxially arranged, the setting management unit 39 derives the omnidirectional camera CAk according to a method described in, for example, JP-A-2015-029241. It is necessary to convert the obtained coordinates into coordinates in the direction viewed from the microphone array MAk.

  In addition, the setting management unit 39 includes a first threshold value th1, a second threshold value th2, and a first threshold value th which are compared with the sound pressure p of each pixel constituting the omnidirectional image data or the two-dimensional panoramic image data calculated by the sound source direction detection unit. The third threshold value th3 (for example, see FIG. 9) is held. Here, the sound pressure p is used as an example of a sound parameter relating to a sound source, represents a loudness of a sound collected by the microphone array MA, and represents a loudness of a sound output from the speaker device 37. It is distinguished from volume. The first threshold value th1, the second threshold value th2, and the third threshold value th3 are values to be compared with the sound pressure of the sound generated in the monitoring area 8, and are, for example, predetermined values for determining the sound emitted from the unmanned aerial vehicle dn. Is set to In addition, a plurality of thresholds can be set in addition to the first threshold th1, the second threshold th2, and the third threshold th3 described above. For simplicity, here, for example, the first threshold th1 and a larger threshold are set. Are set, and a third threshold th3, which is a larger value, is set (first threshold th1 <second threshold th2 <third threshold th3).

  Further, as described later, in the sound pressure heat map generated by the output control unit 35, the omnidirectional image data of the red region RD1 (see FIG. 15) of the pixel in which the sound pressure greater than the third threshold th3 is obtained. On the displayed first monitor MN1, for example, it is drawn in red. In addition, the pink area PD1 of the pixel at which the sound pressure is greater than the second threshold th2 and equal to or less than the third threshold th3 is drawn in, for example, pink on the first monitor MN1 on which the omnidirectional image data is displayed. The blue region BD1 of the pixel in which the sound pressure is larger than the first threshold th1 and equal to or smaller than the second threshold th2 is drawn, for example, in blue on the first monitor MN1 on which the omnidirectional image data is displayed. The sound pressure area N1 of the pixel equal to or smaller than the first threshold value th1 is rendered colorless on the first monitor MN1 on which the omnidirectional image data is displayed, that is, the display color of the omnidirectional image data is not different at all. .

  The speaker device 37 outputs sound data picked up by the microphone array MAk, or sound data picked up by the microphone array MAk and having directivity formed by the signal processing unit 33. Note that the speaker device 37 may be configured as a device separate from the monitoring device 10.

  The memory 38 is configured using, for example, a ROM or a RAM, and stores various data including sound data of a certain section, setting information, a program, and the like. The memory 38 has a pattern memory (see FIG. 7) in which a sound pattern unique to each unmanned aerial vehicle dn is registered. Further, the memory 38 stores sound pressure heat map data generated by the output control unit 35. In the memory 38, an identification mark (not shown) schematically representing the position of the unmanned aerial vehicle dn is registered. The identification mark used here is, for example, a star symbol. The identification mark is not limited to a star shape, but may be a circle or a square, or a symbol or a character such as a “swastika” shape reminiscent of an unmanned aerial vehicle dn. In addition, the display mode of the identification mark may be changed between daytime and nighttime. For example, it may be a star shape in daytime and a square shape not to be mistaken for a star at nighttime. Further, the identification mark may be dynamically changed. For example, a star-shaped symbol may be displayed blinking or rotated, thereby further raising the user's attention.

  FIG. 7 is a timing chart showing an example of a detection sound pattern of the unmanned aerial vehicle dn registered in the memory 38. The pattern of the detection sound shown in FIG. 7 is a combination of frequency patterns, and sounds of four frequencies f1, f2, f3, and f4 generated by rotation of four rotors mounted on the multi-copter type unmanned aerial vehicle dn, and the like. Including. The signals of the respective frequencies are, for example, signals of different sound frequencies generated with the rotation of a plurality of blades supported by the respective rotors.

  In FIG. 7, the frequency region indicated by oblique lines is a region where the sound pressure is high. Note that the detected sound pattern may include not only the number and sound pressure of sounds of a plurality of frequencies but also other sound information. For example, a sound pressure ratio or the like representing a ratio of sound pressures at respective frequencies can be used. Here, as an example, the detection of the unmanned aerial vehicle dn is determined based on whether or not the sound pressure of each frequency included in the detection sound pattern exceeds a threshold.

  The directivity processing unit 63 uses the sound signals (also referred to as sound data) collected by the non-directional microphones M1 to Mq and the setting result of the masking area setting unit 69a to perform the above-described directivity forming process (beam forming). ) To perform sound data extraction processing in which the direction of the other area other than the masking area set by the masking area setting unit 69a is set as the directional direction. Further, the directivity processing unit 63 can also perform sound data extraction processing in which the directivity range is a range in the direction of another area other than the masking area set by the masking area setting unit 69a. Here, the directional range is a range including a plurality of adjacent directional directions, and is intended to include a certain degree of spread of the directional directions as compared with the directional directions.

  The frequency analysis unit 64 performs a frequency analysis process on the sound data extracted in the directivity direction by the directivity processing unit 63. In this frequency analysis processing, the frequency included in the sound data in the directional direction and its sound pressure are detected.

  FIG. 8 is a timing chart showing an example of a frequency change of the detected sound signal obtained as a result of the frequency analysis processing. In FIG. 8, four frequencies f11, f12, f13, f14 and sound pressures at the respective frequencies are obtained as the detected sound signal (that is, detected sound data). In the figure, the fluctuations of each frequency that change irregularly occur, for example, due to the rotation fluctuations of a rotor (rotor wing) that slightly changes when the unmanned aerial vehicle dn controls the attitude of the unmanned aerial vehicle dn itself.

  The target object detection unit 65 as a detection unit performs detection processing of the unmanned aerial vehicle dn using the frequency analysis processing result of the frequency analysis unit 64. Specifically, in the detection processing of the unmanned aerial vehicle dn, the target object detection unit 65 detects the detection sound obtained as a result of the frequency analysis processing in the area other than the masking area set by the masking area setting unit 69a. (See FIGS. 8A and 8B) (frequency f11 to f14) and the detected sound pattern (see FIG. 7) (frequency f1 to f4) registered in the pattern memory of the memory 38 in advance. The object detection unit 65 determines whether or not the patterns of the two detected sounds are similar.

  Whether or not both patterns are similar is determined, for example, as follows. When the sound pressures of at least two frequencies included in the detected sound data among the four frequencies f1, f2, f3, and f4 each exceed the threshold value, the object detection unit 65 determines that the sound patterns are similar, and The flying object dn is detected. Note that the unmanned aerial vehicle dn may be detected when other conditions are satisfied.

  When it is determined that the unmanned aerial vehicle dn does not exist, the detection result determination unit 66 instructs the detection direction control unit 68 to shift to detection of the unmanned aerial vehicle dn in the next directional direction. The detection result determination unit 66 notifies the output control unit 35 of the detection result of the unmanned aerial vehicle dn when it is determined that the unmanned aerial vehicle dn exists as a result of the scanning in the directional direction. The detection result includes information on the detected unmanned aerial vehicle dn. The information on the unmanned aerial vehicle dn includes, for example, identification information of the unmanned aerial vehicle dn and position information (for example, direction information) of the unmanned aerial vehicle dn in the sound collection space.

  The detection direction control unit 68 controls a direction for detecting the unmanned aerial vehicle dn in the sound collection space based on an instruction from the detection result determination unit 66. For example, the detection direction control unit 68 detects an arbitrary direction of the directivity range BF1 (see FIG. 11) including the sound source position estimated by the sound source direction detection unit 34 in the entire sound collection space (that is, the monitoring area 8). Set as direction.

  The scanning control unit 67 instructs the directivity processing unit 63 to perform beamforming using the detection direction set by the detection direction control unit 68 as the directivity direction.

  The directivity processing unit 63 performs beamforming in the directivity direction specified by the scanning control unit 67. In the initial setting, the directivity processing unit 63 sets the initial position in the directivity range BF1 (see FIG. 11) including the sound source position estimated by the sound source direction detection unit 34 as the directivity direction BF2. The pointing direction BF2 is set one after another in the pointing range BF1 by the detection direction control unit 68.

  The masking area setting unit 69a is configured to perform omnidirectional image data or two-dimensional panoramic image data of the monitoring area 8 captured by the omnidirectional camera CAk and audio data of the monitoring area 8 collected by the microphone array MAk. A masking area for excluding detection of the unmanned aerial vehicle dn appearing in the azimuth image or the two-dimensional panoramic image (that is, the captured image) is set. Details of the setting of the masking area will be described later with reference to FIGS.

  The output control unit 35 controls the operations of the first monitor MN1, the second monitor MN2, and the speaker device 37, and outputs the omnidirectional image data or the two-dimensional panoramic image data transmitted from the omnidirectional camera CAk to the first monitor MN1. , And outputs the same to the second monitor MN2 for display, and further outputs the sound data transmitted from the microphone array MA to the speaker device 37 as sound. Further, when the unmanned aerial vehicle dn is detected, the output controller 35 superimposes an identification mark (not shown) representing the unmanned aerial vehicle dn on the omnidirectional image and displays the identification mark on the first monitor MN1 (the 2 monitor MN2).

  Further, the output control unit 35 uses the audio data collected by the microphone array MAk and the coordinates indicating the direction of the sound source position derived by the omnidirectional camera CAk to generate the sound data collected by the microphone array MAk. By performing the directivity forming process, the sound data in the directivity direction is emphasized. The processing for forming the directivity of audio data is a known technique described in, for example, Japanese Patent Application Laid-Open No. 2015-029241.

  The output control unit 35 uses the omnidirectional image data or the sound pressure value of each pixel constituting the two-dimensional panoramic image data calculated by the sound source direction detecting unit 34 to generate omnidirectional image data or two-dimensional panoramic image data. A sound pressure map in which the calculated value of the sound pressure is assigned to the position of the corresponding pixel for each pixel that constitutes the above. Furthermore, the output control unit 35 performs a color conversion process on the generated sound pressure map for each pixel to a visual image (for example, a colored image) so that the user can visually recognize and easily determine the sound pressure value. And a sound pressure heat map as shown in FIG.

  The output control unit 35 has been described to generate a sound pressure map or a sound pressure heat map in which the sound pressure values calculated in pixel units are assigned to the positions of the corresponding pixels, but the sound pressure value is calculated for each pixel. Without calculating, the average value of the sound pressure values is calculated for each pixel block composed of a predetermined number (for example, 2 × 2, 4 × 4) of pixels, and the average of the sound pressure values corresponding to the corresponding predetermined number of pixels is calculated. A sound pressure map or a sound pressure heat map may be generated by assigning a value.

  Note that details of the threshold adjustment unit 69b will be described in a second embodiment described later, and a detailed description thereof will be omitted here.

  Next, the operation of the unmanned aerial vehicle detection system 5 of the present embodiment will be described in detail.

  FIG. 9 is a sequence diagram illustrating an example of the operation of detecting the unmanned aerial vehicle dn and displaying the detection result in the first embodiment. When power is supplied to each device (for example, the first monitor MN1, the monitoring device 10, the omnidirectional camera CAk, and the microphone array MAk) of the unmanned vehicle detection system 5, the unmanned vehicle detection system 5 starts operating. 9, it is assumed that a masking area for excluding detection of the unmanned aerial vehicle dn has already been set, and information indicating the masking area has been registered in the memory 38.

  In the initial operation, the monitoring device 10 issues an image distribution request to the omnidirectional camera CAk (S1). The omnidirectional camera CAk starts the imaging process according to the power-on in accordance with the image distribution request. Further, the monitoring device 10 makes a voice distribution request to the microphone array MAk (S2). The microphone array MA starts a sound collection process according to the power-on according to the voice distribution request.

  When the initial operation ends, the omnidirectional camera CAk transmits data of an omnidirectional image (for example, a still image or a moving image) obtained by imaging to the monitoring device 10 via the network NW (S3). In FIG. 9, for simplicity of description, omnidirectional image data is transmitted from the omnidirectional camera CAk. However, two-dimensional panoramic image data may be transmitted, and the same applies to FIG. 14. It is. The monitoring device 10 converts the omnidirectional image data transmitted from the omnidirectional camera CAk into display data such as NTSC, outputs the data to the first monitor MN1, and instructs the display of the omnidirectional image data (S4). When the display data transmitted from the monitoring device 10 is input, the first monitor MN1 displays data of the omnidirectional image IMG1 from the omnidirectional camera CAk (see FIGS. 12 and 15) on the screen.

  Further, the microphone array MAk encodes the audio data of the monitoring area 8 obtained by the sound pickup via the network NW and transmits the encoded audio data to the monitoring device 10 (S5). In the monitoring device 10, the sound source direction detection unit 34 forms omnidirectional image data of the monitoring area 8 based on omnidirectional image data captured by the omnidirectional camera CAk and audio data collected by the microphone array MAk. The sound pressure as a sound parameter is calculated for each pixel to be processed, and the sound source position in the monitoring area 8 is estimated (S6). The estimated sound source position is used as a reference position of the directivity range BF1 necessary for setting the initial directivity when the monitoring device 10 detects the unmanned aerial vehicle dn.

  Further, in the monitoring device 10, the output control unit 35 uses the sound pressure values of each pixel constituting the omnidirectional image data calculated by the sound source direction detecting unit 34 to configure each omnidirectional image data. For each pixel, a sound pressure map in which the calculated value of the sound pressure is assigned to the position of the pixel is generated. Furthermore, the output control unit 35 performs a color conversion process on the generated sound pressure map for each pixel to a visual image (for example, a colored image) so that the user can visually recognize and easily determine the sound pressure value. Then, a sound pressure heat map as shown in FIG. 15 is generated (S7).

  Further, in the monitoring device 10, the signal processing unit 33 uses the audio data transmitted from the microphone array MAk in step S5 to sequentially change the directivity to the outside of the masking area set by the masking area setting unit 69a. Thus, the detection and determination of the unmanned aerial vehicle dn is performed for each directional direction in which the directivity is formed (S8). The details of the detection determination processing of the unmanned aerial vehicle dn will be described later with reference to FIGS.

  When the unmanned aerial vehicle dn is detected as a result of the detection determination process, the output control unit 35 in the monitoring device 10 adds the sound pressure generated in step S7 to the omnidirectional image IMG1 displayed on the screen of the first monitor MN1. An instruction is given to superimpose and display the heat map and an identification mark (not shown) indicating the unmanned aerial vehicle dn existing in the directional direction detected in step S8 (S9).

  The first monitor MN1 composes (superimposes) the sound pressure heat map on the omnidirectional image IMG1 according to the instruction from the monitoring device 10, and also composes an identification mark (not shown) representing the unmanned flying object dn (not shown). (S10). Thereafter, the process of the unmanned aerial vehicle detection system 5 returns to step S3, and the processes of steps S3 to S10 are repeated until a predetermined event such as a power-off operation is detected.

  FIG. 10 is a flowchart illustrating an example of the details of the operation procedure for unmanned flying object detection determination in step S8 in FIG. In the sound source detection unit UDk, the directivity processing unit 63 uses the information of the masking area set by the masking area setting unit 69a and is based on the sound source position outside the masking area and estimated by the sound source direction detection unit 34. The pointing range BF1 is set as an initial position of the pointing direction BF2 (S21). The information of the masking area is coordinates in a direction toward the masking area as viewed from the microphone array MAk.

  FIG. 11 is a diagram illustrating an example of a state in which the directional direction BF2 is sequentially scanned in the monitoring area 8 and the unmanned aerial vehicle dn is detected. Note that the initial position may be outside the masking area set by the masking area setting unit 69a, and need not be limited to the directivity range BF1 based on the sound source position of the monitoring area 8 estimated by the sound source direction detection unit 34. . That is, if the position is outside the masking area set by the masking area setting unit 69a, an arbitrary position specified by the user may be set as the initial position, and the inside of the monitoring area 8 may be sequentially scanned. Since the initial position is not limited, even if the sound source included in the directivity range BF1 based on the estimated sound source position is not an unmanned aerial vehicle, it is possible to early detect an unmanned aerial vehicle flying in another directional direction. Becomes

  The directivity processing unit 63 determines whether sound data collected by the microphone array MAk and converted into digital values by the A / D converters An1 to Aq is temporarily stored in the memory 38 (S22). . If not stored (S22, NO), the processing of the directivity processing unit 63 returns to step S21.

  If the sound data picked up by the microphone array MA is temporarily stored in the memory 38 (S22, YES), the directivity processing unit 63 is outside the masking area set by the masking area setting unit 69a. The beamforming is performed on an arbitrary directional direction BF2 in the directional range BF1 of the monitoring area 8, and sound data in the directional direction BF2 is extracted (S23).

  The frequency analysis unit 64 detects the frequency of the extracted sound data and its sound pressure (S24).

  The target object detection unit 65 compares the detection sound pattern registered in the pattern memory of the memory 38 with the detection sound pattern obtained as a result of the frequency analysis processing, and detects an unmanned aerial vehicle (S25).

  The detection result determination unit 66 notifies the output control unit 35 of the result of the comparison, and also notifies the detection direction control unit 68 of the shift in the detection direction (S26).

  For example, the object detection unit 65 compares the detected sound pattern obtained as a result of the frequency analysis processing with four frequencies f1, f2, f3, and f4 registered in the pattern memory of the memory 38. As a result of the comparison, the target object detection unit 65 has at least two same frequencies in both detection sound patterns, and when the sound pressure of these frequencies is greater than the first threshold th1, the two detection sound patterns Approximately, it is determined that the unmanned aerial vehicle dn exists.

  Here, it is assumed that at least two frequencies match, but the object detection unit 65 approximates when one frequency matches and the sound pressure of this frequency is larger than the first threshold th1. May be determined.

  Further, the target object detection unit 65 may set an allowable frequency error for each frequency, and determine whether or not the approximation is performed assuming that the frequencies within the error range are the same frequency.

  In addition to the comparison between the frequency and the sound pressure, the target object detection unit 65 may determine that the sound pressure ratios of the sounds of the respective frequencies are substantially the same, in addition to the determination condition. In this case, since the determination condition becomes strict, the sound source detection unit UDk can easily identify the detected unmanned aerial vehicle dn as a pre-registered target object, and can improve the detection accuracy of the unmanned aerial vehicle dn.

  The detection result determination unit 66 determines whether or not the unmanned aerial vehicle dn exists as a result of Step S26 (S27).

  When the unmanned vehicle dn exists (S27, YES), the detection result determination unit 66 notifies the output control unit 35 that the unmanned vehicle dn exists (the detection result of the unmanned vehicle dn) (S28).

  On the other hand, when the unmanned aerial vehicle dn does not exist (S27, NO), the detection result determination unit 66 instructs the scan control unit 67 to move the directional direction BF2 of the scan target in the monitoring area 8 to the next different direction. I do. The scanning control unit 67 moves the directional direction BF2 of the scanning target in the monitoring area 8 in the next different direction according to the instruction from the detection result determination unit 66 (S29). The notification of the detection result of the unmanned aerial vehicle dn may be collectively performed after the completion of the omnidirectional scan, not at the timing when the detection processing of one directional direction is completed.

  The order in which the directional directions BF2 are sequentially moved in the monitoring area 8 is, for example, within the directional range BF1 of the monitoring area 8 or within the entire range of the monitoring area 8 if it is outside the masking area set by the masking area setting section 69a. The order of spiral (spiral) may be such that the direction goes from the outer circumference to the inner circumference, or from the inner circumference to the outer circumference.

  Also, the detection direction control unit 68 sets a position in the monitoring area 8 in advance instead of continuously scanning the directivity direction like a single stroke, and sets the masking area set by the masking area setting unit 69a. Otherwise, the directional direction BF2 may be moved to each position in any order. Thereby, the monitoring apparatus 10 can start the detection processing from a position where the unmanned aerial vehicle dn easily enters, for example, and can improve the efficiency of the detection processing.

  The scanning control unit 67 determines whether the omnidirectional scanning in the monitoring area 8 has been completed (S30). When the omnidirectional scanning is not completed (S30, NO), the process of the signal processing unit 33 returns to Step S23, and the processes of Steps S23 to S30 are repeated. That is, the directivity processing unit 63 performs beamforming in the directivity direction BF2 of the position moved in step S29 outside the masking area set by the masking area setting unit 69a, and extracts sound data in the directivity direction BF2. . With this, even when one unmanned aerial vehicle dn is detected, the sound source detection unit UDk continues to detect other unmanned aerial vehicles dn, so that detection of a plurality of unmanned aerial vehicles dn can be performed. It is possible.

  On the other hand, when the omnidirectional scanning is completed in step S30 (S30, YES), the directivity processing unit 63 deletes the sound data temporarily stored in the memory 38 and collected by the microphone array MAk (S31). ). Further, the directivity processing unit 63 may delete the sound data collected by the microphone array MAk from the memory 38 and store the sound data in the recorder RC.

  After the erasure of the sound data, the signal processing unit 33 determines whether or not to terminate the detection processing of the unmanned aerial vehicle dn (S32). The process of detecting the unmanned aerial vehicle dn ends in response to a predetermined event. For example, the number of times that the unmanned aerial vehicle dn is not detected in step S26 may be stored in the memory 38, and when the number of times is equal to or more than a predetermined number, the detection processing of the unmanned aerial vehicle dn may be terminated. Further, the signal processing unit 33 may end the detection processing of the unmanned aerial vehicle dn based on a time-up by a timer or a user operation on a UI (User Interface) (not shown) of the operation unit 32. Further, when the power of the monitoring apparatus 10 is turned off, the processing may be ended.

  In the process of step S24, the frequency analysis unit 64 analyzes the frequency and measures the sound pressure of the frequency. The detection result determination unit 66 determines that the unmanned aerial vehicle dn is approaching the sound source detection unit UDk when the sound pressure level measured by the frequency analysis unit 64 gradually increases with time. Is also good.

  For example, when the sound pressure level of the predetermined frequency measured at time t11 is smaller than the sound pressure level of the same frequency measured at time t12 after time t11, the sound pressure increases with time, It may be determined that the unmanned aerial vehicle dn is approaching. Further, even if the sound pressure level is measured three or more times and it is determined that the unmanned aerial vehicle dn is approaching based on the transition of the statistical value (for example, variance value, average value, maximum value, minimum value, etc.). Good.

  Further, when the measured sound pressure level is larger than the alert threshold, which is the alert level, the detection result determination unit 66 may determine that the unmanned aerial vehicle dn has entered the alert area.

  Note that the alert threshold is a value larger than, for example, the third threshold th3 described above. The guard area is, for example, the same area as the monitoring area 8 or an area included in the monitoring area 8 and smaller than the monitoring area 8. The alert area is, for example, an area where entry of the unmanned aerial vehicle dn is restricted. In addition, the approach determination and the intrusion determination of the unmanned aerial vehicle dn may be executed by the detection result determination unit 66.

  FIG. 12 is a diagram illustrating an example of a display screen of the first monitor MN1 when the masking area MSK3 is not set. In FIG. 12, the unmanned aerial vehicle dn detected by the monitoring device 10 is shown on the first monitor MN1 on the upper right side of the omnidirectional image IMG1 in FIG. 12. Further, sound pressure heat maps respectively corresponding to sound sources generated within the range of the angle of view of the unmanned aerial vehicle dn and the omnidirectional camera CAk are superimposed and displayed.

  As described with reference to FIG. 9, in the present embodiment, if the sound pressure value of each pixel calculated by the sound source direction detection unit 34 is equal to or less than the first threshold th <b> 1, the sound pressure value is displayed in colorless. If the sound pressure value is larger than the first threshold value th1 and equal to or less than the second threshold value th2, the sound pressure value is displayed in blue. If the sound pressure value is larger than the second threshold value th2 and equal to or less than the third threshold value th3, the sound pressure value is displayed in pink. If it is larger than the threshold th3, it is displayed in red.

  In FIG. 12, for example, in the vicinity of the rotor and the rotor around the center of the housing of the unmanned aerial vehicle dn, since the sound pressure value is larger than the third threshold th3, it is drawn in the red regions RD1, RD2, RD3, and RD4. Similarly, pink areas PD1, PD2, PD3, and PD4 indicating that the sound pressure value is the second largest after the red area are drawn around the red area. Similarly, blue areas BD1, BD2, BD3, and BD4 indicating that the sound pressure value is the second largest after the pink area are drawn around the pink area.

  FIG. 12 also shows that a sound source is also present in the office building, and the sound pressure value for each pixel calculated by the sound source direction detection unit 34 exceeds the third threshold th3 or a pixel whose sound pressure value exceeds the third threshold th3. Red regions R1, R2, R3, R4, R5, R6, and R7 are drawn for the set. Similarly, around the red area of the office building, pink areas P1, P2, P3, P4, and P5 indicating that the sound pressure value is the second largest after the red area are drawn. Similarly, blue regions B1 and B2 indicating that the sound pressure value is the second largest after the pink region are drawn around the pink region of the office building. In other regions of the omnidirectional image IMG1, since the sound pressure value is equal to or less than the first threshold th1, it is drawn in the colorless region N1, and the visibility of the background omnidirectional image IMG1 is not deteriorated.

  Next, details of the setting of the masking area in the present embodiment will be described with reference to FIGS. 13, 14, and 15. FIG. FIG. 13 is an explanatory diagram showing a display example of the masking area in the time series in the automatic learning process. FIG. 14 is a sequence diagram illustrating an example of an operation procedure for setting a masking area according to the first embodiment. FIG. 15 is a diagram illustrating an example of a display screen of the first monitor when a masking area is set. The sequence illustrated in FIG. 14 is a so-called initial setting that is executed before the operation of the sequence illustrated in FIG. 9 starts.

  In FIG. 14, the user instructs the monitoring device 10 to start the automatic learning process of the masking area by using, for example, the operation unit 32 (T1). The monitoring device 10 makes an image distribution request to the omnidirectional camera CAk (T2). The omnidirectional camera CAk starts the imaging process according to the power-on in accordance with the image distribution request. The omnidirectional camera CAk transmits data of an omnidirectional image (for example, a still image or a moving image) obtained by imaging to the monitoring device 10 via the network NW (T3). The monitoring device 10 converts the omnidirectional image data transmitted from the omnidirectional camera CAk into display data such as NTSC, outputs the data to the first monitor MN1, and instructs display of the omnidirectional image data (T4). As a result, when the display data transmitted from the monitoring device 10 is input, the first monitor MN1 displays the data of the omnidirectional image IMG1 from the omnidirectional camera CAk on the screen (see the upper left of FIG. 13).

  Further, the monitoring device 10 makes a voice distribution request to the microphone array MAk (T5). The microphone array MA starts a sound collection process according to the power-on according to the voice distribution request. The microphone array MAk encodes the audio data of the monitoring area 8 obtained by the sound collection via the network NW and transmits the encoded audio data to the monitoring device 10 (T6). In the monitoring device 10, the sound source direction detection unit 34 forms omnidirectional image data of the monitoring area 8 based on omnidirectional image data captured by the omnidirectional camera CAk and audio data collected by the microphone array MAk. The sound pressure as a sound parameter is calculated for each pixel to be processed.

  Further, the masking area setting unit 69a determines a pixel or a set of pixels for which the calculated value of the sound pressure by the sound source direction detection unit 34 is equal to or larger than a predetermined masking area threshold (for example, the above-described third threshold th3). The masking area setting section 69a stores and registers the information indicating the determined pixel or the set thereof in the memory 38 as the information indicating the masking area (T7). The information indicating the masking area is, specifically, coordinates on an omnidirectional image that specifies the position of the pixel whose calculated value of the sound pressure is equal to or larger than the masking area threshold. The masking area setting unit 69a, via the output control unit 35, sets the information indicating the masking area and the masking area (that is, a pixel or a set of pixels whose sound pressure calculated value is equal to or larger than the masking area threshold) in a predetermined color (for example, red). ) Is output to the first monitor MN1 (T8). Thereby, the first monitor MN1 performs a process of painting the position of the coordinates corresponding to the masking area MSK1 on the omnidirectional image IMG with a predetermined color in accordance with the instruction transmitted from the monitoring device 10 (see the upper right of FIG. 13). . In the omnidirectional image IMG1 on the upper right side of FIG. 13, the masking area MSK1 indicates the entire area painted in a predetermined color.

  Similarly, the microphone array MA encodes the audio data of the monitoring area 8 obtained by the continuous sound pickup via the network NW in accordance with the audio distribution request from the monitoring device 10 and sends the encoded audio data to the monitoring device 10. Transmit (T9). In the monitoring device 10, the sound source direction detection unit 34 forms omnidirectional image data of the monitoring area 8 based on omnidirectional image data captured by the omnidirectional camera CAk and audio data collected by the microphone array MAk. The sound pressure as a sound parameter is calculated for each pixel to be processed.

  Further, the masking area setting unit 69a determines a pixel or a set of pixels for which the calculated value of the sound pressure by the sound source direction detection unit 34 is equal to or larger than the masking area threshold. The masking area setting unit 69a stores information indicating the determined pixel or the set thereof in the memory 38 as information indicating the masking area and registers the information (T10). The masking area setting unit 69a outputs information indicating the masking area and an instruction to paint the masking area with a predetermined color (for example, red) to the first monitor MN1 via the output control unit 35 (T11). Thereby, the first monitor MN1 paints the position of the coordinates corresponding to the masking area MSK2 accumulated with respect to the masking area MSK1 on the omnidirectional image IMG in a predetermined color according to the instruction transmitted from the monitoring device 10. (See the lower right of FIG. 13). In the omnidirectional image IMG1 on the lower right side of FIG. 13, the masking area MSK2 indicates the entire area painted with a predetermined color.

  Here, a user operation using the operation unit 32 instructs the monitoring apparatus 10 to end the automatic learning process of the masking area (T12). In response to this instruction, the monitoring device 10 transmits a voice distribution stop request to the microphone array MAk (T13). As a result, the microphone array MAk stops distribution (transmission) of the audio data of the monitoring area 8 obtained by the sound collection to the monitoring device 10.

  In addition, a user operation using the operation unit 32 causes an operation of correcting the masking area (that is, adding or deleting a masking area) to the first monitor MN1 indicating the masking area MSK2 at the lower right of FIG. When this is performed (T14), in the monitoring device 10, the masking area setting unit 69a adds a position on the designated omnidirectional image IMG1 as a masking area or deletes the position from the masking area by a user operation. Information indicating the corrected masking area MSK3 is stored and registered in the memory 38 (T15).

  For example, in step T14, an operation for deleting the unnecessary areas GOM1, GOM2, and GOM3 as masking areas under the judgment of the user is performed on the first monitor MN1 indicating the masking area MSK2 at the lower right of FIG. For example, a range designation operation is performed, or an operation (that is, a drawing operation) for adding an entire office building in the background covered by the masking area MSK2 as a masking area is performed.

  The masking area setting unit 69a outputs information indicating the masking area and an instruction to paint the masking area with a predetermined color (for example, red) to the first monitor MN1 via the output control unit 35 (T16). Accordingly, the first monitor MN1 displays the position of the coordinates corresponding to the masking area MSK3 accumulated with respect to the masking areas MSK1 and MSK2 on the omnidirectional image IMG in a predetermined color according to the instruction transmitted from the monitoring device 10. A painting process is performed (see the lower left of FIG. 13). In the omnidirectional image IMG1 at the lower left of FIG. 13, the masking area MSK3 indicates the entire area painted in a predetermined color. Therefore, the sky is excluded from the masking area MSK3.

  Accordingly, according to the sequence illustrated in FIG. 14, the unmanned aerial vehicle dn as the detection target of the present embodiment is positioned above the sky where almost no sound source exists within the range of the angle of view of the omnidirectional camera CAk. In view of the tendency to fly, analysis of the audio data from the microphone array MAk revealed that no sound source with a sound pressure higher than the masking area threshold was found in the sky, so the sky was set as the detection target of the unmanned aerial vehicle dn. It becomes possible. On the other hand, by setting an entire office building where a large number of sound sources are present in the surrounding area as a masking area, it is possible to avoid erroneous detection of sound sources other than the unmanned aerial vehicle dn as the unmanned aerial vehicle dn that should originally be detected. It is possible to improve the detection accuracy and the detection processing speed of the unmanned aerial vehicle dn.

  In other words, as shown in FIG. 15, in the omnidirectional image IMG1, detection of the unmanned aerial vehicle dn is performed only in another area outside the masking area MSK3. As a result, as compared with the omnidirectional image IMG1 shown in FIG. 12, the sound pressure value of the sound generated at the sound source position at the sound source position around the unmanned aerial vehicle dn detected in other areas except the masking area MSK3 Are superimposed and displayed on a sound pressure heat map converted into a visual image. On the other hand, the sound pressure value of the sound generated at the position of the sound source is displayed in a visual image around another sound source that is not an unmanned aerial vehicle detected in the masking area MSK3 (for example, a bellow of a person in an office building). The superimposed display of the converted sound pressure heat map is omitted.

  In the present embodiment, when the masking area is set by the masking area setting unit 69a, as shown in FIG. 15, the sound pressure value of the sound generated at the sound source position around the unmanned air vehicle dn is visually recognized. The sound pressure heat map converted to the image is displayed in a superimposed manner. However, when the masking area is not set by the masking area setting unit 69a, the sound pressure heat map corresponding to the sound pressure value of the detected sound source is superimposed on the omnidirectional image IMG1, as shown in FIG. May be displayed.

  As described above, in the unmanned aerial vehicle detection system 5 of the present embodiment, the monitoring device 10 uses the voice data collected by the microphone array MAk to perform an unmanned flight appearing in the captured image (omnidirectional image IMG1) of the monitoring area 8. A masking area for excluding the detection of the body dn is set by the masking area setting unit 69a. The monitoring device 10 detects the unmanned aerial vehicle dn in an area other than the masking area using the audio data collected by the microphone array MAk and the information indicating the masking area. Further, when detecting the unmanned aerial vehicle dn outside the masking area, the monitoring device 10 displays a sound source visual image indicating the loudness at the sound source position on the sound source position of the unmanned aerial vehicle dn in the omnidirectional image IMG1 (that is, , Red area RD1, pink area PD1, blue area BD1,...) Are superimposed and displayed on first monitor MN1.

  Accordingly, the unmanned aerial vehicle detection system 5 automatically sets a masking area for excluding the detection processing of the unmanned aerial vehicle dn to be detected from the monitoring area 8 to be imaged by the omnidirectional camera CAk. Therefore, the probability of erroneously detecting the object at the sound source position in the masking area as the unmanned aerial vehicle dn can be reduced, and deterioration in the detection accuracy of the unmanned aerial vehicle dn can be suppressed. Further, the unmanned aerial vehicle detection system 5 does not need to detect the unmanned aerial vehicle dn over the imaging angle of view of the omnidirectional camera CAk (that is, the entire region of the omnidirectional image IMG1), Since the detection determination of the flying object dn may be performed, the detection processing of the unmanned flying object dn can be further improved.

  Further, in the unmanned aerial vehicle detection system 5, the sound source direction detection unit 34 uses the omnidirectional image IMG1 to determine the sound pressure that specifies the loudness of the sound in the monitoring area 8 based on the sound data collected by the microphone array MAk. Is calculated for each predetermined unit of the pixels that constitute. The masking area setting unit 69a superimposes and displays on the first monitor MN1 the position of the sound source at which the calculated value of the sound pressure is equal to or greater than the masking area threshold relating to the loudness on the first monitor MN1, and further confirms the user. By the operation, the sound source area displayed on the first monitor MN1 is set as a masking area. Thereby, the user can mask the area where the unmanned aerial vehicle dn is unlikely to fly but other sound sources (for example, a bellow of a person) are likely to be generated from the detection target area of the unmanned aerial vehicle dn. As a result, the setting can be easily performed while visually checking the first monitor MN1.

  Further, the masking area setting unit 69a masks the sound source area after the user addition operation by performing a user addition operation for further adding the sound source area displayed on the first monitor MN1 (that is, a candidate area of the masking area). Set as an area. Thus, the user can visually confirm, on the first monitor MN1, a portion painted in a predetermined color as a masking area candidate automatically by the monitoring device 10, and determine a portion to be further added to the masking area by his / her own judgment. The masking area can be set while easily specifying, thereby improving the usability of the user.

  Further, the masking area setting unit 69a performs the user deletion operation for deleting at least a part of the sound source area displayed on the first monitor MN1 (that is, the area that is a candidate for the masking area). Set the area as a masking area. This allows the user to visually check, on the first monitor MN1, a portion painted in a predetermined color as a candidate for a masking area by the monitoring device 10 while checking the portion painted in the predetermined color by his own judgment. The masking area can be set while easily specifying a part to be excluded as a masking area from, and the usability for the user is improved.

  Further, the output control unit 35 converts the sound source visual image in which the sound pressure is stepwise converted into a different visual image in accordance with the comparison between the calculated value of the sound pressure and a plurality of thresholds regarding the loudness of the sound in the monitoring area 8. The pixels constituting the omnidirectional image IMG1 are superimposed on a predetermined unit basis and displayed on the first monitor MN1. Thus, the user can see a wide state of the monitoring area 8 as an omnidirectional image in the omnidirectional image IMG1 of the monitoring area 8 captured by the omnidirectional camera CAk by looking at the first monitor MN1. Not only can the location of the sound source (for example, the unmanned aerial vehicle dn) generated outside the masking area of the monitoring area 8 and the loudness of the sound can be easily confirmed as a visual image.

(History leading to the second embodiment)
According to Patent Document 1 described above, by comparing the sound pressure at a frequency specific to a flying object such as a helicopter or a Cessna with a predetermined set level, if the sound pressure is equal to or higher than the set level, it is determined that the flying object is a monitoring target It has been disclosed.

  However, Patent Literature 1 described above does not consider quantitatively indicating which of the measured sound pressures is the sound pressure in which a plurality of steps are defined. For this reason, when any sound is detected in the imaging area of the camera device, the loudness of the sound is determined by detailed visual information of fine sound regardless of the loudness of the sound at the detected sound source position. There was a problem that it could not be presented in a typical way.

  Therefore, in the second embodiment, regardless of the magnitude of the sound volume at the sound source position detected in the imaging area of the camera device, the loudness of the sound at the sound source position is presented in a fine-grained manner in a stepwise manner. An example of a monitoring system that contributes to accurate understanding of the loudness of a sound by a user will be described.

(Second embodiment)
In the second embodiment, the internal configuration of each device forming the unmanned aerial vehicle detection system 5 is the same as the internal configuration of each device forming the unmanned aerial vehicle detection system 5 of the first embodiment. The contents are denoted by the same reference numerals, description thereof will be omitted, and different contents will be described.

  In the second embodiment, the monitoring device 10 generates the sound pressure heat map described in the first embodiment, displays the heat map on the first monitor MN1, and then operates the operation unit 32 by a user (see below). The sound pressure heat map is finely analyzed and displayed in accordance with the relationship between the calculated value of the sound pressure required for generating the sound pressure heat map and a plurality of thresholds (see below). Hereinafter, three analysis methods will be described.

(First analysis method)
In the first analysis method, the monitoring device 10 superimposes the sound pressure heat map corresponding to the omnidirectional image IMG2 on the omnidirectional image IMG2 and displays it on the first monitor MN1. Is specified, the display resolution of the sound pressure heat map in the specified range is changed to be the same as the display resolution of the entire omnidirectional image IMG2. An operation example of the first analysis method will be described with reference to FIGS. FIG. 16 is a schematic explanatory diagram of dynamically changing the display resolution of the sound pressure heat map in the second embodiment. FIG. 17 is a flowchart illustrating an example of an operation procedure for dynamically changing the display resolution of the sound pressure heat map according to the second embodiment.

  In FIG. 17, the monitoring device 10 has input the audio data of the monitoring area 8 transmitted from the microphone array MAk (S41). It is determined whether or not a partial cutout range of the omnidirectional image IMG2 (see the upper left of FIG. 16) on the first monitor MN1 has been designated by a user operation using the operation unit 32 (S42).

  Here, in the upper left of FIG. 16, the omnidirectional image IMG2 displayed on the first monitor MN1 has the omnidirectional image IMG2 whose display size in the X direction is “Wmax” and whose display size in the Y direction is “Hmax”. Has a display resolution of “Wmax × Hmax”. Also, the coordinates of the end points of the cut-out range that is specified by the user operation and become a part of the omnidirectional image IMG2 are (X1, Y1), (X1, Y2), (X2, Y1), (X2, Y2). It is a rectangle. FIG. 16 shows only (X1, Y1) and (X2, Y2) existing on the diagonal line of the same rectangle.

  If the user does not specify a rectangle having four points (X1, Y1), (X1, Y2), (X2, Y1), and (X2, Y2) as end points (S42, NO), monitoring is performed. The sound source direction detection unit 34 of the device 10 sets (X, Y) = (0, 0) (S43) to generate a sound pressure map for the entire omnidirectional image IMG2 (S43), and coordinates (X, Y). = Sound pressure P (X, Y) at (0,0) is calculated (S45).

  Further, when the X coordinate of the omnidirectional image IMG2 does not match the maximum value Wmax (S46, NO), the sound source direction detection unit 34 generates a sound pressure map for the entire omnidirectional image IMG2. , The X coordinate is incremented by 1 (S47), and the sound pressure P (X, Y) is calculated at (X, Y) after the increment.

  When the X coordinate of the omnidirectional image IMG2 matches the maximum value Wmax (S46, YES), the sound source direction detection unit 34 determines that the Y coordinate of the omnidirectional image IMG2 does not match the maximum value Hmax (S46). (S48, NO), in order to generate a sound pressure map for the entire omnidirectional image IMG2, the X coordinate is returned to 0 and the Y coordinate is incremented by 1 (S49), and the sound pressure P is incremented at (X, Y) after the increment. (X, Y) is calculated. The sound source direction detection unit 34 repeats the processes of steps 45 to S49 until the Y coordinate of the omnidirectional image IMG2 matches the maximum value Hmax, thereby performing the same processing on the entire omnidirectional image IMG2 as in the first embodiment. A sound pressure map can be generated, and stored and registered in the memory 38 (S50).

  On the other hand, if a rectangle having four points (X1, Y1), (X1, Y2), (X2, Y1), and (X2, Y2) as end points is designated by a user operation (S42, YES), The sound source direction detection unit 34 of the monitoring device 10 sets (X, Y) = (X1, Y1) to generate a sound pressure map for a range specified by a user operation in the omnidirectional image IMG2 (S44). ), The sound pressure P (X, Y) at the coordinates (X, Y) = (X1, Y1) is calculated (S45).

  Further, when the X coordinate of the omnidirectional image IMG2 does not match the maximum value Wmax (S46, NO), the sound source direction detection unit 34 determines the sound of the omnidirectional image IMG2 for the range specified by the user operation. In order to generate the pressure map, the X coordinate is increased by (X2−X1) / Wmax (S47), and the sound pressure P (X, Y) is calculated at (X, Y) after the increase.

  When the X coordinate of the omnidirectional image IMG2 matches the maximum value Wmax (S46, YES), the sound source direction detection unit 34 determines that the Y coordinate of the omnidirectional image IMG2 does not match the maximum value Hmax (S46). (S48, NO), the X coordinate is returned to X1 and the Y coordinate is increased by (Y2-Y1) / Hmax in order to generate a sound pressure map for the range specified by the user operation in the omnidirectional image IMG2 (S49). ), And calculate the sound pressure P (X, Y) at (X, Y) after the increase. The sound source direction detection unit 34 repeats the processing of steps 45 to S49 until the Y coordinate of the omnidirectional image IMG2 matches the maximum value Hmax, thereby setting the sound pressure for the range of the omnidirectional image IMG2 specified by the user operation. A map can be generated and stored in the memory 38 and registered (S50). After step S50, the process of the monitoring device 10 returns to step S41, and the processes of steps S42 to S50 are repeated for the audio data input in step S41.

  Therefore, as shown in the upper right of FIG. 16, the monitoring device 10 simply cuts out the range specified by the user operation and generates an audio heat map according to the sound pressure value of each pixel constituting the range. When superimposed on one monitor MN1, only an image with a low display resolution (that is, coarse) can be displayed.

  However, according to the first analysis method of the present embodiment, the monitoring device 10 sets the display resolution of the cut-out range of a part of the cut-out range of the omnidirectional image IMG2 specified by the user operation to the display of the entire omnidirectional image IMG2. The sound pressure P (X, Y) is calculated in fine units (that is, every (X2−X1) / Wmax in the X direction, and every (Y2−Y1) / Hmax in the Y direction) so as to be the same as the resolution. I do. As a result, as shown in the lower right part of FIG. 16, the monitoring device 10 performs a resolution finer than the display resolution of the sound pressure heat map when the sound pressure heat map in the cutout range designated by the user operation is simply cut out (FIG. 16). (Unit), and can be displayed on the first monitor MN1 with high accuracy, and the user can more accurately grasp the details of the distribution of the sound sources in the cut-out range specified by the user operation.

(Second analysis method)
Before describing the second analysis method, as a matter common to FIGS. 18 to 20, in the present embodiment, a description will be given of the width of a plurality of thresholds that define the loudness of sound in a stepwise manner. FIG. 18 is a schematic explanatory diagram of a threshold width adjustment according to the frequency distribution of sound pressure values and a display result of a captured image accompanying the width adjustment in the second embodiment. In FIGS. 18 to 20, the sound source visual images corresponding to the sound pressure values from the lower limit to the upper limit and between the thresholds are group blue images, blue images, blue images, light blue images, blue green images, yellow green images, It is specified that a yellow image, an orange image, a red image, and a red image are used. When the sound pressure value is minimum (that is, the lower limit), an ultramarine image is used, and when the sound pressure value is maximum (that is, an upper limit), a red image is used.

  For example, as shown on the left side of FIG. 18, a total of ten thresholds are defined so as to correspond to the magnitude of the sound pressure value. For example, in the example on the left side of FIG. 18, each threshold is provided on the axis of each sound pressure value. Each scale corresponds to a threshold. For this reason, in the example on the left side of FIG. 18, when the sound pressure value is included between a certain threshold value (for example, 5) and the next largest threshold value (for example, 6), when the sound pressure heat map is generated, A color image (for example, a yellow-green image) corresponding to a portion between the thresholds including the sound pressure value is used as a sound source visual image for visually indicating the sound pressure value.

  In the second analysis method, when the output control unit 35 generates the sound pressure heat map corresponding to the omnidirectional image, the threshold value adjusting unit 69b of the monitoring device 10 sets each pixel (a predetermined unit of pixels) constituting the omnidirectional image. Based on the occurrence frequency (in other words, frequency distribution) of the sound pressure value calculated in the above manner, a total of ten thresholds or the width between the thresholds is dynamically changed. That is, the threshold adjustment unit 69b dynamically changes the setting of the correspondence between the plurality of thresholds and the sound source visual image according to the omnidirectional image. For example, referring to FIG. 18, the threshold adjustment unit 69b holds the setting of using the yellow-green image as the sound source visual image if the sound pressure value is between the threshold value 5 and the threshold value 6, but the sound pressure value is When the appearance frequency of 5 to the threshold 6 is high among all the pixels of the omnidirectional image, the width LG1 between the thresholds for using the yellow-green image (for example, the threshold 5 to the threshold 6) becomes narrower. To the width LG2 (for example, the threshold value 4.5 to the threshold value 4.8).

  Therefore, when the sound pressure value of each pixel constituting the omnidirectional image is concentrated in the width AR1 between the thresholds for using a specific color image, the sound pressure value is reduced as shown in the lower left of FIG. As the heat map VMP1, the same color image or a visually similar color image (for example, a yellow-green image and a blue-green image) is used as a sound source visual image, and the distribution of sound sources appearing in an omnidirectional image is finely divided. Difficult to present to the user.

  However, according to the second analysis method of the present embodiment, the monitoring device 10 uses the threshold adjustment unit 69b to generate the sound source according to the appearance frequency (frequency distribution) of the sound pressure value for each pixel forming the omnidirectional image. The setting of the correspondence between the visual image and the plurality of thresholds is dynamically changed, and after reflecting the change, the sound pressure heat map VMP2 corresponding to the omnidirectional image is displayed on the first monitor MN1. As a result, the monitoring apparatus 10 can present a fine sound pressure distribution to the user as the sound pressure heat map VMP2 as shown in the lower right part of FIG. 18 and allow the user to more accurately grasp the distribution of the sound source position. Can be.

(Third analysis method)
In the third analysis method, the monitoring device 10 sets the upper limit, the lower limit, or both of the thresholds for defining the use of the sound source visual image (that is, the color image) by user operation using the operation unit 32. Can be specified. FIGS. 19A and 19B are schematic explanatory diagrams of a change in the setting of the width between thresholds that defines the use of the sound source visual image in the second embodiment.

  For example, in FIG. 19A, the width between the thresholds (upper threshold) for defining the use of the sound source visual image (that is, the red image) indicating that the sound pressure value is the highest (that is, the upper limit) is equal to the threshold. The threshold value is changed from the threshold value 9 to the threshold value 10 to the threshold value 6 to the threshold value 10, and further defines the use of the sound source visual image (that is, the ultramarine blue image) indicating that the sound pressure value is the lowest (that is, the lower limit). The width between the thresholds (lower threshold) is changed from threshold 0 to threshold 1 to threshold 0 to threshold 2. Accordingly, the monitoring device 10 dynamically changes the remaining eight thresholds so that the width between the thresholds becomes equal, and is different from the width between the two thresholds changed by the user operation. Thus, the width between the threshold values can be changed.

  For example, in FIG. 19B, use of a sound source visual image (that is, a reddish image) indicating that the sound pressure value is the highest (that is, the upper limit) after being changed as shown in FIG. 19A is used. Is changed from the threshold value 6 to the threshold value 10 to the threshold value 9.3 to the threshold value 10, and further indicates that the sound pressure value is the lowest (that is, the lower limit). The width between the thresholds (lower threshold) for defining the use of the sound source visual image (that is, the group blue image) is changed from threshold 0 to threshold 2 to threshold 0 to threshold 5. Thus, the monitoring apparatus 10 similarly sets the width of the remaining eight thresholds to be equal, although the thresholds are different from the example illustrated in FIG. The width between the thresholds can be changed dynamically and different from the width between the two thresholds changed by the user operation.

  In FIGS. 19A and 19B, a threshold (upper threshold) for defining the use of a sound source visual image (that is, a reddish image) indicating that the sound pressure value is the highest (that is, the upper limit) is shown. And the width between the thresholds (lower threshold) for specifying the use of the sound source visual image (ie, the ultramarine blue image) indicating that the sound pressure value is the lowest (ie, the lower limit) is changed. Although the above example has been described, the same is true for only one of the changes.

  In other words, when only the width between the thresholds (upper threshold) for defining the use of the sound source visual image (that is, the red image) indicating that the sound pressure value is the highest (that is, the upper limit) is changed. Similarly, the monitoring device 10 dynamically changes the widths of the remaining nine thresholds so that they are equally spaced, and sets the thresholds so that the widths differ from one threshold changed by the user operation. The width can be changed.

  Also, when only the width between the thresholds (lower threshold) for defining the use of the sound source visual image indicating that the sound pressure value is the lowest (that is, the lower limit) (that is, the group blue image) is changed. In the same manner, the monitoring device 10 dynamically changes the width between the remaining nine thresholds so as to be equally spaced, and changes the threshold between the thresholds so as to be different from the width between one threshold changed by the user operation. Can be changed in width.

  Here, the operation related to the change of the setting of the threshold width by the third analysis method will be described with reference to FIGS. FIG. 20 is a schematic explanatory diagram of the display of the captured image according to the change of the setting of the width between the thresholds defining the use of the red image and the group blue image in the second embodiment. FIG. 21 is a flowchart illustrating an example of an operation procedure for changing the setting of the width between threshold values according to the second embodiment.

  As shown in FIG. 20, the sound source visual image (that is, a red image) indicating that the sound pressure value is the highest (that is, the upper limit) and the sound pressure value that is the lowest (that is, the lower limit) are shown. It is assumed that the width between the thresholds defining the use with the sound source visual image (that is, the group blue image) has both been changed. In this case, in the monitoring device 10, as shown in FIG. 21, the threshold adjustment unit 69b includes a correspondence table (not shown, correspondence relationship) between a sound source visual image and a threshold value defining use of the sound source visual image or a width between the threshold values. Dynamically change the settings of.

  Therefore, in the example illustrated in FIG. 20, the monitoring device 10 has the sound pressure heat map VMP2 superimposed on the omnidirectional image (that is, the captured image) is only the red color image. The sound pressure heat map VMP2A using the sound source visual image is generated, and the sound pressure heat map VMP2A is superimposed on the omnidirectional image (that is, the captured image) and displayed on the first monitor MN1. The user operation is, for example, an input operation on an input screen (not shown) of a threshold or a width between thresholds displayed on the first monitor MN1, or a drag on the display screen of a width between thresholds displayed on the first monitor MN1. Operations correspond, but are not limited to these operations.

  In FIG. 21, the threshold adjustment unit 69b regulates the use of a sound source visual image (that is, a reddish image) indicating that the sound pressure value is the highest (that is, the upper limit) by a user operation using the operation unit 32. It is determined whether the width between the thresholds has been changed (S61). When the width between the thresholds that defines the use of the sound source visual image (that is, the reddish image) indicating that the sound pressure value is the highest (that is, the upper limit) is changed (S61, YES), the threshold adjusting unit 69b Then, it is determined whether or not the width between the thresholds that defines the use of the sound source visual image (that is, the ultra-blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) is changed (S62).

  When the width between the thresholds defining the use of the sound source visual image (that is, the ultramarine blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) is changed (S62, YES), the correspondence table between the sound source visual image and the threshold value defining the use of the sound source visual image or the width between the threshold values is corrected according to the change result (S63). For example, the threshold adjustment unit 69b dynamically changes the remaining eight thresholds so that the width between the thresholds is equal, and differs from the width between the two thresholds changed by the user operation. To change the width between the thresholds. Accordingly, when a large sound pressure value is obtained between the threshold value specifying the use of the red image and the threshold value specifying the use of the ultra-blue image, for example, the monitoring device 10 sets the width between the thresholds by the user. By performing the adjustment operation, the distribution around the pixels where many sound pressure values are concentrated can be finely displayed as a sound pressure heat map.

  If the width between the thresholds defining the use of the sound source visual image (that is, the red image) indicating that the sound pressure value is the lowest (that is, the upper limit) is not changed (S62, NO) According to the result of the change, the correspondence table between the sound source visual image and the threshold value defining the use of the sound source visual image or the width between the threshold values is corrected (S64). For example, the threshold adjuster 69b dynamically changes the remaining nine thresholds so that the width between the thresholds is equal, and differs from the width between one threshold changed by the user operation. To change the width between the thresholds. Accordingly, when many sound pressure values equal to or less than the threshold value that specifies the use of the red image are obtained, the monitoring device 10 concentrates many sound pressure values by the user's operation of adjusting the width between the threshold values. The distribution around the pixel can be displayed as a sound pressure heat map.

  On the other hand, in Step S61, when the width between the thresholds that defines the use of the sound source visual image (that is, the red image) indicating that the sound pressure value is the highest (that is, the upper limit) is not changed (S61, NO), The threshold adjusting unit 69b determines whether or not the width between the thresholds defining the use of the sound source visual image (that is, the ultramarine blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) has been changed. (S65). If the width between the thresholds that defines the use of the sound source visual image (that is, the group blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) is not changed (S65, NO), the threshold The process of the adjusting unit 69b returns to Step S61.

  When the width between the thresholds defining the use of the sound source visual image (that is, the ultramarine blue image) indicating that the sound pressure value is the lowest (that is, the lower limit) is changed (S65, YES), the correspondence table between the sound source visual image and the threshold value defining the use of the sound source visual image or the width between the threshold values is corrected according to the change result (S66). For example, the threshold adjuster 69b dynamically changes the remaining nine thresholds so that the width between the thresholds is equal, and differs from the width between one threshold changed by the user operation. To change the width between the thresholds. Thereby, for example, when a large number of sound pressure values equal to or more than the threshold value that defines the use of the ultramarine blue image are obtained, the monitoring device 10 may increase the number of sound pressure values by adjusting the width between the threshold values by the user. The distribution around the concentrated pixels can be finely displayed as a sound pressure heat map.

  As described above, in the unmanned aerial vehicle detection system 5 of the present embodiment, the monitoring device 10 monitors the sound pressure that specifies the loudness of the sound in the monitoring area 8 using the sound data collected by the microphone array MAk. The calculation is performed for each predetermined unit of pixels constituting the captured image (omnidirectional image IMG2) of the area 8. The monitoring device 10 converts the sound source visual image in which the sound pressure is stepwise converted into a different visual image in accordance with the comparison between the calculated value of the sound pressure and the plurality of thresholds relating to the loudness of the pixels constituting the captured image. The information is superimposed for each predetermined unit and displayed on the first monitor MN1. When any one of the sound source positions is specified in the captured image on which the sound source visual image is superimposed, the monitoring device 10 determines a predetermined unit of a pixel forming a rectangular range including the sound source position by dividing the predetermined unit of the pixel between the captured image and the rectangular range. The sound pressure is calculated for each value divided by the size ratio.

  Accordingly, the monitoring apparatus 10 increases the sound pressure heat map of the rectangular range (that is, the cutout range) specified by the user operation with a finer resolution (unit) than the display resolution in the sound pressure heat map when the cutout is simply cut out. It can be displayed on the first monitor MN1 with high accuracy, and the user can more accurately grasp the details of the distribution of the sound sources in the cut-out range specified by the user operation. In other words, the monitoring device 10 presents the loudness of the sound at the sound source position finely and stepwise regardless of the loudness of the sound at the sound source position detected in the monitoring area 8 of the omnidirectional camera CAk. It is possible to contribute to the user's accurate grasp of the loudness at the sound source position.

  Further, in the unmanned aerial vehicle detection system 5 of the present embodiment, the monitoring device 10 uses the sound data collected by the microphone array MAk to determine the sound pressure for specifying the loudness of the sound in the monitoring area 8. The calculation is performed for each predetermined unit of pixels constituting the eight captured images (omnidirectional image IMG2). The monitoring device 10 corresponds to each of a plurality of thresholds that define the loudness of the sound stepwise, and a sound source visual image that is stepwise converted into a visual image having a different sound pressure according to a comparison with each threshold. The setting of the relationship is dynamically changed according to the captured image of the monitoring area 8 (that is, the omnidirectional image). The monitoring device 10 converts a sound source visual image corresponding to the calculated value of the sound pressure into the captured image for each predetermined unit of the pixels constituting the captured image based on the calculated value of the sound pressure and the setting of the changed correspondence relationship. The information is superimposed and displayed on the first monitor MN1.

  Accordingly, the monitoring device 10 can provide a detailed sound pressure heat map as a sound pressure heat map for visually indicating the position of the sound source collected in the monitoring area 8 according to the image captured by the omnidirectional camera CAk. The distribution can be presented to the user, and the distribution of the sound source position can be grasped more accurately by the user.

  In addition, the threshold adjusting unit 69b of the monitoring device 10 changes the width between the thresholds that define the sound source visual image based on the appearance frequency of the sound pressure of the pixels constituting the captured image of the monitoring area 8 for each predetermined unit. . Thereby, the monitoring apparatus 10 increases the type of the sound source visual image for the pixel corresponding to the sound pressure calculation value with a high appearance frequency, and adds the sound source visual image to the pixel corresponding to the sound pressure calculation value with a low appearance frequency. Since the sound pressure heat map is generated after reducing the number of types, sound pressure distributions of various shades can be presented in detail rather than monotonous shades, and the user can more accurately grasp the distribution of sound source positions.

  In addition, the threshold adjustment unit 69b of the monitoring device 10 changes the width in accordance with the operation of changing the width between the thresholds that specifies the use of the sound source visual image (that is, the red image) corresponding to the upper limit of the sound pressure. Except for the width between the thresholds after the threshold, the width between all the other thresholds is changed uniformly. Accordingly, when many sound pressure values equal to or less than the threshold value that specifies the use of the red image are obtained, the monitoring device 10 concentrates many sound pressure values by the user's operation of adjusting the width between the threshold values. The distribution around the pixel can be displayed as a sound pressure heat map.

  In addition, the threshold adjustment unit 69b of the monitoring device 10 changes the width in accordance with the operation of changing the width between the thresholds that specifies the use of the sound source visual image (that is, the group blue image) corresponding to the lower limit of the sound pressure. The width between all the thresholds except for the width between the thresholds after the change is uniformly changed. Thereby, for example, when a large number of sound pressure values equal to or more than the threshold value that defines the use of the ultramarine blue image are obtained, the monitoring device 10 may increase the number of sound pressure values by adjusting the width between the threshold values by the user. The distribution around the concentrated pixels can be finely displayed as a sound pressure heat map.

  In addition, the threshold adjustment unit 69b of the monitoring device 10 determines the width between the thresholds defining the use of the sound source visual image corresponding to the upper limit of the sound pressure (that is, the red image) and the sound source visual image corresponding to the lower limit of the sound pressure. In response to the operation of changing the width between the thresholds that regulates the use of (i.e., the ultramarine blue image), the width between all other thresholds except for the width between the thresholds after those widths are changed is changed equally. I do. Accordingly, when a large sound pressure value is obtained between the threshold value specifying the use of the red image and the threshold value specifying the use of the ultra-blue image, for example, the monitoring device 10 sets the width between the thresholds by the user. By performing the adjustment operation, the distribution around the pixels where many sound pressure values are concentrated can be finely displayed as a sound pressure heat map.

(Process leading to the third embodiment)
In Patent Literature 1 described above, a monitoring camera capable of changing a shooting direction in an arbitrary direction within a monitoring area is provided, and when a flying object such as a helicopter or a Cessna is detected, the shooting direction of the monitoring camera is changed. Diversion is disclosed. In other words, it is disclosed that the shooting direction of the surveillance camera is changed in order to focus and image the detected flying object.

  However, the above-mentioned Patent Document 1 does not consider displaying a wide range of captured images around the unmanned aerial vehicle detected within the range of the angle of view of the camera device with respect to the imaging area. For this reason, the unmanned aerial vehicle is detected in what place in the imaging area of the camera device, and what kind of sound source is present elsewhere in the same imaging area is visually presented to the user. There was a problem that it was not possible.

  Therefore, in the third embodiment, an unmanned aerial vehicle is detected at what location in the imaging area of the camera device, and what sound source is present at another location in the same imaging area. An example of a monitoring system that visually presents to a user without deteriorating the visibility of a captured image of the device will be described.

(Third embodiment)
In the third embodiment, the internal configuration of each device constituting the unmanned aerial vehicle detection system 5 is the same as the internal configuration of each device constituting the unmanned aerial vehicle detection system 5 of the first embodiment. The contents are denoted by the same reference numerals, description thereof will be omitted, and different contents will be described.

  In the third embodiment, after generating the sound pressure heat map (sound parameter map) described in the first embodiment, the monitoring device 10 becomes a translucent image (semi-transparent map) of the sound pressure heat map. The translucent sound pressure heat map is generated, and the translucent sound pressure heat map is superimposed on the omnidirectional image and displayed on the first monitor MN1 (see FIG. 22). FIG. 22 is a schematic explanatory diagram of an overlay display of an omnidirectional image and a translucent sound pressure heat map in the third embodiment.

  In the present embodiment, as illustrated in FIG. 22, the monitoring device 10 displays an omnidirectional image IMG1 captured by the omnidirectional camera CAk on the first monitor MN1. Further, the monitoring device 10 uses the sound source visual image obtained by converting the sound pressure value calculated for each pixel constituting the omnidirectional image IMG1 or for each predetermined unit of the pixel into a different visual image in a stepwise manner, using the omnidirectional image IGM1. Is generated, and a translucent sound pressure heat map TRL1 obtained by converting the sound pressure heat map into a translucent image is generated and displayed on the second monitor MN2.

  In FIG. 22, the monitoring device 10 displays the omnidirectional image IMG1 on the first monitor MN1 and the translucent sound pressure heat map TRL1 on a monitor separate from the second monitor MN2. For example, the omnidirectional image IMG1 may be displayed in a window, and the translucent sound pressure heat map TRL1 may be displayed in another window.

  Further, the monitoring device 10 displays an omnidirectional image IMG1A in which the translucent sound pressure heat map TRL1 is superimposed on the omnidirectional image IMG1 on the first monitor MN1 (see FIG. 23). FIG. 23 is a diagram showing an example of a display screen of the first monitor MN1 on which the omnidirectional image IMG1 and the translucent sound pressure heat map TRL1 are overlaid.

  In FIG. 23, the unmanned aerial vehicle dn detected by the monitoring device 10 is shown on the first monitor MN1 on the upper right side of the omnidirectional image IMG1A in FIG. 23. Further, translucent sound pressure heat maps corresponding to the unmanned aerial vehicles dn and sound sources generated within the range of the angle of view of the omnidirectional camera CAk are displayed in a superimposed manner.

  As described with reference to FIG. 24, in the present embodiment, if the sound pressure value for each pixel calculated by the sound source direction detection unit 34 is equal to or smaller than the first threshold th1, the colorless translucent color (that is, colorless). If the sound pressure value is greater than the first threshold value th1 and equal to or less than the second threshold value th2, the sound pressure value is displayed in a translucent blue color. If the sound pressure value is greater than the second threshold value th2 and equal to or less than the third threshold value th3. If the sound pressure value is larger than the third threshold th3, the color is displayed in a red translucent color.

  In FIG. 23, for example, in the vicinity of the rotor and the rotor around the center of the housing of the unmanned aerial vehicle dn, the sound pressure value is larger than the third threshold th3, and therefore, is drawn in the translucent red regions RD1A, RD2A, RD3A, RD4A. . Similarly, translucent pink areas PD1A, PD2A, PD3A, and PD4A indicating that the sound pressure value is the second largest after the translucent red area are drawn around the translucent red area. Similarly, semi-transparent blue regions BD1A, BD2A, BD3A, and BD4A indicating that the sound pressure value is the second largest after the translucent pink region are drawn around the translucent pink region.

  Also, FIG. 23 shows that a sound source also exists in the office building, and the sound pressure value of each pixel calculated by the sound source direction detection unit 34 exceeds the third threshold th3 or the pixel thereof. Translucent red areas R1A, R2A, R3A, R4A, R5A, R6A, and R7A are drawn for the set. Similarly, translucent pink areas P1A, P2A, P3A, P4A, and P5A indicating that the sound pressure value is the second largest after the translucent red area are drawn around the translucent red area of the office building. Similarly, around the translucent pink area of the office building, translucent blue areas B1A and B2A indicating that the sound pressure value is the second largest after the translucent pink area are drawn. In other areas of the omnidirectional image IMG1A, since the sound pressure value is equal to or less than the first threshold th1, the area is drawn as a colorless area.

  As described above, in the present embodiment, the monitoring device 10 superimposes the translucent sound pressure heat map TRL1 different from that of the first embodiment on the omnidirectional image IMG1 and displays it on the first monitor MN1. The user can visually determine the position of the sound source appearing in the IMG1 and the loudness of the sound at that position, and the visibility of the omnidirectional image IMG1 can be prevented from deteriorating.

  Next, the operation of the unmanned aerial vehicle detection system 5 of the present embodiment will be described in detail with reference to FIG.

  FIG. 24 is a sequence diagram illustrating an example of an operation procedure of overlay display of the omnidirectional image IMG1 and the translucent sound pressure heat map TRL1 in the third embodiment. When power is supplied to each device (for example, the first monitor MN1, the monitoring device 10, the omnidirectional camera CAk, and the microphone array MAk) of the unmanned vehicle detection system 5, the unmanned vehicle detection system 5 starts operating. In the description of FIG. 24, the masking area described in the first embodiment may or may not be used. FIG. 24 illustrates an example in which a masking area is used. When a masking area is used, it is assumed that information indicating the masking area is registered in the memory 38.

  The monitoring device 10 makes an image distribution request to the omnidirectional camera CAk (S71). The omnidirectional camera CAk starts the imaging process according to the power-on in accordance with the image distribution request. Further, the monitoring device 10 makes a voice distribution request to the microphone array MAk (S72). The microphone array MA starts a sound collection process according to the power-on according to the voice distribution request.

  When the initial operation is completed, the omnidirectional camera CAk transmits data of an omnidirectional image (for example, a still image or a moving image) obtained by imaging to the monitoring device 10 via the network NW (S73). Note that, in FIG. 24, for simplicity of description, it is described that omnidirectional image data is transmitted from the omnidirectional camera CAk, but two-dimensional panoramic image data may be transmitted. The monitoring device 10 converts the omnidirectional image data transmitted from the omnidirectional camera CAk to display data such as NTSC, outputs the data to the first monitor MN1, and instructs the display of the omnidirectional image data (S74). When the display data transmitted from the monitoring device 10 is input, the first monitor MN1 displays the data of the omnidirectional image IMG1 from the omnidirectional camera CAk (see the upper left of FIG. 22) on the screen.

  Further, the microphone array MAk encodes the audio data of the monitoring area 8 obtained by the sound pickup via the network NW and transmits the encoded audio data to the monitoring device 10 (S75). In the monitoring device 10, the sound source direction detection unit 34 forms omnidirectional image data of the monitoring area 8 based on omnidirectional image data captured by the omnidirectional camera CAk and audio data collected by the microphone array MAk. The sound pressure as a sound parameter is calculated for each pixel to be processed, and the sound source position in the monitoring area 8 is estimated (S76). The estimated sound source position is used as a reference position of the directivity range BF1 necessary for setting the initial directivity when the monitoring device 10 detects the unmanned aerial vehicle dn.

  Further, in the monitoring device 10, the output control unit 35 uses the sound pressure values of each pixel constituting the omnidirectional image data calculated by the sound source direction detecting unit 34 to configure each omnidirectional image data. For each pixel, a sound pressure map in which the calculated value of the sound pressure is assigned to the position of the pixel is generated. Furthermore, the output control unit 35 performs a color conversion process on the generated sound pressure map for each pixel to a visual image (for example, a colored image) so that the user can visually recognize and easily determine the sound pressure value. Then, a translucent sound pressure heat map as shown in the upper right of FIG. 22 is generated (S77). The method of generating the translucent sound pressure heat map is as follows. For example, the output control unit 35 first generates a sound pressure heat map (see step S7 in FIG. 9), and secondarily generates a semi-transparent sound pressure heat map. This is a procedure of generating a translucent sound pressure heat map by performing a transparent process.

  Further, in the monitoring device 10, the signal processing unit 33 uses the audio data transmitted from the microphone array MAk in step S75 to sequentially change the directivity to the outside of the masking area set by the masking area setting unit 69a. Thus, the detection and determination of the unmanned aerial vehicle dn is performed for each directional direction in which the directivity is formed (S78). The details of the detection and determination processing of the unmanned aerial vehicle dn have been described with reference to FIGS. 10 and 11, and a description thereof will be omitted.

  When the unmanned aerial vehicle dn is detected as a result of the detection determination process, the output control unit 35 in the monitoring device 10 adds the semi-transparent image generated in step S77 to the omnidirectional image IMG1 displayed on the screen of the first monitor MN1. An instruction is given to superimpose and display the sound pressure heat map and an identification mark (not shown) indicating the unmanned aerial vehicle dn existing in the directional direction detected in step S78 (S79).

  The first monitor MN1 synthesizes (superimposes) a translucent sound pressure heat map on the omnidirectional image IMG1 and displays the same on the omnidirectional image IMG1, and displays an identification mark (not shown) indicating the unmanned aerial vehicle dn. The images are combined (superimposed) and displayed (S80). Thereafter, the process of the unmanned aerial vehicle detection system 5 returns to step S73, and the processes of steps S73 to S80 are repeated until a predetermined event such as a power-off operation is detected.

  As described above, in the unmanned aerial vehicle detection system 5 of the present embodiment, the monitoring device 10 monitors the sound pressure that specifies the loudness of the sound in the monitoring area 8 using the sound data collected by the microphone array MAk. The calculation is performed for each predetermined unit of the pixels constituting the captured image (omnidirectional image IMG1) of the area 8. The monitoring apparatus 10 converts a sound source visual image obtained by converting a sound pressure into a visual image for each predetermined unit of pixels in accordance with a comparison between the calculated value of the sound pressure and a threshold regarding the loudness of the sound, in all directions of the monitoring area 8. A semi-transparent sound pressure heat map linked to correspond to the size of the image is generated. The monitoring device 10 superimposes the translucent sound pressure heat map on the captured image of the monitoring area 8 and displays the same on the first monitor MN1.

  As a result, in the unmanned aerial vehicle detection system 5, an unmanned aerial vehicle is detected at any place in the monitoring area 8 of the omnidirectional camera CAk, and at any other location in the same monitoring area 8, any sound source exists. This can be visually presented to the user as to whether or not the image is captured, without deteriorating the visibility of the image captured by the omnidirectional camera CAk.

  In addition, a plurality of thresholds related to the loudness of the sound are provided, and the monitoring device 10 converts the sound pressure for each predetermined unit of pixels into different visual images in a stepwise manner according to the comparison between the sound pressure and the plurality of thresholds. A translucent sound pressure heat map having a plurality of types of sound source visual images is generated using the sound source visual images. Accordingly, the monitoring device 10 determines the presence of sound pressure having a plurality of types of stages defined by a plurality of thresholds in the omnidirectional image captured by the omnidirectional camera CAk by using a visual sound source visual image. The user can be more clearly determined.

  Also in the present embodiment, the monitoring apparatus 10 sets the masking area described in the first embodiment, and uses the audio data collected by the microphone array MAk and the information indicating the masking area to set the masking area. Detect unmanned aerial vehicles outside. When detecting the unmanned aerial vehicle outside the masking area, the monitoring device 10 detects the sound emitted from the unmanned aerial vehicle around the unmanned aerial vehicle in the omnidirectional image (in other words, the sound source position of the unmanned aerial vehicle). The sound source visual image indicating the size is made translucent and displayed on the first monitor MN1. Accordingly, since the monitoring device 10 can exclude the inside of the masking area from the detection target of the unmanned aerial vehicle, it is possible to suppress the deterioration of the detection accuracy of the masking area and improve the detection speed of the unmanned aerial vehicle. Further, the monitoring device 10 determines the level of the loudness of the sound emitted from the unmanned aerial vehicle at the sound source position of the unmanned aerial vehicle dn detected outside the masking area by using the translucent image of the sound source visual image alone. In addition, it is possible to prevent the visibility of the surrounding captured image from deteriorating.

  Also in the present embodiment, the monitoring apparatus 10 sets the correspondence between the plurality of thresholds that define the loudness of the sound in stages and the plurality of types of sound source visual images in the captured image of the imaging area. Change accordingly. The monitoring apparatus 10 transposes the sound source visual image of each predetermined unit of pixels so as to correspond to the size of the captured image in the imaging area based on the calculated value of the sound pressure and the changed correspondence. Generate a heatmap. Accordingly, the monitoring device 10 calculates the sound pressure obtained for each pixel constituting the captured image or for each predetermined unit of the pixel in accordance with the content of the omnidirectional image (captured image) captured by the omnidirectional camera CAk. The correspondence between the value and the sound source visual image corresponding to the calculated value of the sound pressure can be made variable. Therefore, for example, in a place where a specific sound pressure calculation value is concentrated, the monitoring device 10 uses a sound source visual image of a plurality of colors instead of a single color sound source visual image to generate a sound source visual image therearound. It is possible to allow the user to grasp in detail the distribution of the loudness of the sound in the sound source appearing in the captured image clearly and finely.

  Although various embodiments have been described with reference to the drawings, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive various changes or modifications within the scope of the claims, and the present invention naturally covers the matters combining the above-described embodiments. It is understood that it belongs to the technical range of.

  The present invention presents the loudness of a sound at the sound source position finely and stepwise, regardless of the loudness of the loudness at the sound source position detected in the imaging area of the camera device, and provides the loudness of the sound at the sound source position. It is useful as a monitoring system and a monitoring method contributing to accurate understanding of the user.

5 Unmanned aerial vehicle detection system 10 Monitoring device 31 Communication unit 32 Operation unit 33 Signal processing unit 34 Sound source direction detection unit 35 Output control unit 37 Speaker unit (SPK)
38 Memory 39 Setting management unit 63 Directivity processing unit 64 Frequency analysis unit 65 Object detection unit 66 Detection result determination unit 67 Scanning control unit 68 Detection direction control unit 69a Masking area setting unit 69b Threshold adjustment units CA1, CAk, CAn All directions Camera CZ1, CZk, CZn PTZ camera MA1, MAk, MAn Microphone array MN1 First monitor MN2 Second monitor NW Network RC recorder UD1, UDn Sound source detection unit

Claims (8)

A camera for imaging the imaging area,
A microphone array that picks up sound in the imaging area;
A monitor that displays a captured image of the imaging area captured by the camera;
A sound parameter that specifies the loudness of the sound in the imaging area based on the sound collected by the microphone array, using a predetermined number of pixels constituting the captured image of the imaging area as a sound parameter derivation unit of the imaging area. A sound parameter deriving unit that derives a sound parameter deriving unit of the imaging area,
According to a comparison between the sound parameter derived by the sound parameter deriving unit and a plurality of thresholds related to the loudness of the sound, a sound source visual image in which the sound parameter is stepwise converted into a different visual image, A signal processing unit that is superimposed on each sound parameter derivation unit and displayed on the monitor,
The sound parameter deriving unit, when a partial rectangular range is specified in the captured image of the imaging area on which the sound source visual image is superimposed , the sound parameter derivation unit of the imaging area is a captured image of the imaging area. Derive a sound parameter of the rectangular range for each value divided by a size ratio of the rectangular range and
Monitoring system. A camera for imaging the imaging area,
A microphone array that picks up sound in the imaging area;
A monitor that displays a captured image of the imaging area captured by the camera;
A sound parameter that specifies the loudness of the sound in the imaging area based on the sound collected by the microphone array, using a predetermined number of pixels constituting the captured image of the imaging area as a sound parameter derivation unit of the imaging area. A sound parameter deriving unit that derives a sound parameter deriving unit of the imaging area,
Setting of a correspondence relationship between each of a plurality of thresholds that define the volume of sound in a stepwise manner and a sound source visual image in which the sound parameters are gradually converted into different visual images in accordance with the comparison with each of the thresholds A threshold adjustment unit that changes according to the captured image of the imaging area,
On the basis of the sound parameter derived by the sound parameter deriving unit and the correspondence changed by the threshold adjusting unit, for each sound parameter deriving unit of the imaging area, the sound source visual image corresponding to the sound parameter And a signal processing unit that superimposes the captured image of the imaging area and displays the image on the monitor.
Monitoring system. The monitoring system according to claim 2, wherein
The threshold adjustment unit, based on the appearance frequency of the sound parameter for each sound parameter derivation unit of the imaging area, to change the width of each threshold that defines the sound source visual image,
Monitoring system. The monitoring system according to claim 2, wherein
The threshold adjustment unit, in response to a change operation of the width between the thresholds defining the use of the sound source visual image corresponding to the upper limit of the sound parameter, other than the width between the thresholds after the width is changed other Change the width between all thresholds equally,
Monitoring system. The monitoring system according to claim 2, wherein
The threshold adjustment unit, according to a change operation of the width between the thresholds defining the use of the sound source visual image corresponding to the lower limit of the sound parameter, other than the width between the threshold after the width is changed other Change the width between all thresholds equally,
Monitoring system. The monitoring system according to claim 2, wherein
The threshold adjustment unit, the width between the threshold that defines the use of the sound source visual image corresponding to the upper limit of the sound parameter and the width between the threshold that defines the use of the sound source visual image corresponding to the lower limit of the sound parameter. In accordance with the change operation, the width between all the thresholds except for the width between the upper threshold and the lower threshold after the width is changed is changed uniformly.
Monitoring system. A monitoring method in a monitoring system having a camera and a microphone array,
With the camera, the imaging area is imaged,
The microphone array collects sound in the imaging area,
Displaying a captured image of the imaging area captured by the camera on a monitor,
A sound parameter that specifies the loudness of the sound in the imaging area based on the sound collected by the microphone array, using a predetermined number of pixels constituting the captured image of the imaging area as a sound parameter derivation unit of the imaging area. Is derived for each sound parameter derivation unit of the imaging area,
According to a comparison between the derived sound parameter and a plurality of thresholds relating to the loudness of the sound, a sound source visual image in which the sound parameter is stepwise converted into a different visual image is provided for each sound parameter deriving unit of the imaging area. Being superimposed and displayed on the monitor,
Further, when a partial rectangular range is specified in the captured image of the imaging area on which the sound source visual image is superimposed, the sound parameter derivation unit of the imaging area is defined as the captured image of the imaging area and the rectangular range. Derive a sound parameter of the rectangular range for each value divided by the size ratio of
Monitoring method. A monitoring method in a monitoring system having a camera and a microphone array,
With the camera, the imaging area is imaged,
The microphone array collects sound in the imaging area,
Displaying a captured image of the imaging area captured by the camera on a monitor,
A sound parameter that specifies the loudness of the sound in the imaging area based on the sound collected by the microphone array, using a predetermined number of pixels constituting the captured image of the imaging area as a sound parameter derivation unit of the imaging area. Is derived for each sound parameter derivation unit of the imaging area,
Setting of a correspondence relationship between each of a plurality of thresholds that define the volume of sound in a stepwise manner and a sound source visual image in which the sound parameters are gradually converted into different visual images in accordance with the comparison with each of the thresholds Is changed according to the captured image of the imaging area,
Based on the derived sound parameter and the changed correspondence, based on the sound parameter derivation unit of the imaging area, the sound source visual image corresponding to the sound parameter is superimposed on the captured image of the imaging area. Displaying on the monitor,
Monitoring method.