JP4175180B2 - Monitoring and reporting system - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a monitoring and reporting system for reporting an emergency situation that has occurred in a plurality of reporting units to a supervisor.
[0002]
[Prior art]
Conventionally, as an emergency call device, techniques described in the following Patent Literature 1 to Patent Literature 4 are known. The technology described in Patent Document 1 below determines whether or not an abnormality has occurred when an emergency call button provided in an emergency call device is pressed or when a speaker's characteristic voice is recognized. Deciding.
[0003]
Moreover, in the technique described in the following patent document 2 and patent document 3, the direction of a sound source such as an intruding object is specified using a plurality of microphones arranged in different directions on a horizontal plane, and the specified direction is By pointing the video camera, images and sounds of intruding objects and the like are acquired.
[0004]
Further, as a monitoring notification system related to Patent Document 4 below, the applicant configures the system shown in FIG. Specifically, as shown in the figure, a plurality of reporting units 101A to 101C,..., A monitoring server 102 and a monitoring monitor 103 are connected by an optical network 104 and acquired by each reporting unit 101. The transmitted image and sound are configured to be transmitted. Here, each notification unit 101 removes noise from the image data captured by the omnidirectional camera 111 and the movable camera 112 by the environmental image noise removal unit 121 and stores it in the image storage unit 122, and collects sound by the microphone 113. The environmental sound removal unit 131 removes noise from the sound and stores it in the voice storage unit 132.
[0005]
In this monitoring and reporting system, a camera remote control signal is generated by the monitoring person operating the monitoring server 102, and the imaging direction of the movable camera 112 is controlled by the camera control unit 123. Also, in this monitoring and reporting system, when the call switch 114 is pressed by the user, the image data stored in the image storage unit 122 is transmitted via the media converter 115 and the router 116, or by the VOIP processing unit 133. The speaker unit 117 and the microphone 113 are used to realize a call between the supervisor and the user.
[0006]
[Patent Document 1]
JP 2000-348278 A
[0007]
[Patent Document 2]
JP 2002-344957 A
[0008]
[Patent Document 3]
Japanese Patent Laid-Open No. 7-284186
[0009]
[Patent Document 4]
JP 2002-288774 A
[0010]
[Problems to be solved by the invention]
However, the technology related to Patent Document 4 has a configuration in which image information acquired by a terminal on the client side is directly transmitted to a monitoring server or the like. Because of the limited monitoring tasks, it is difficult to accommodate a large number of clients. Even if the same technology is adopted and a system with a large-scale client can be constructed, the task for monitoring on the server side over the entire area where the client is installed becomes large. There was a problem of increasing the burden.
[0011]
Further, in order to solve such a problem, the inventors of the present application have proposed a solution means to be described later. In the past, environmental noise faced the problem of reducing recognition accuracy. More specifically, for example, when environmental noise similar to abnormal sound is emitted, the environmental noise may be misrecognized as abnormal sound and a false alarm may be generated. When there are many false alarms, there is a problem that the number of monitoring tasks on the server side is increased, and the effect of the solution may be lost.
[0012]
Therefore, the present invention has been proposed in view of the above-described circumstances, and reduces the amount of information transmitted from a client when transmitting audio and images acquired on a large-scale client side to the server side. The monitoring task on the side can be reduced, the response response to the emergency response during emergency can be improved, and there is no need to perform special operations on the whistleblower. The purpose is to provide a monitoring and reporting system with improved reporting reliability.
[0025]
[Means for Solving the Problems]
  In order to solve the above-described problem, another monitoring and reporting system according to the present invention includes a plurality of reporting units and a monitoring server that presents a monitor with sound and images generated by the reporting unit via a communication line. In each of the connected systems, each reporting unit captures an image of a monitoring target, generates image data, and collects sound around the imaging unit to generate sound data. And voice recognition means for recognizing the status of the monitoring target from the voice data generated by the sound collection means, at least image data and voice data are communicated with the monitoring server, and the monitoring is performed by the voice recognition means. When it is determined that an abnormality has occurred in the target situation, a communication unit that transmits an abnormality occurrence signal to the monitoring server, and a remote control signal from the monitoring server Imaging control means for controlling the image direction, wherein the monitoring server stores communication location for communicating with the reporting unit, installation location data indicating the installation location of the plurality of reporting units, and When a plurality of abnormality occurrence signals are received from the reporting unit, the installation location of the reporting unit where the abnormality has occurred is extracted with reference to the installation location data, and according to the order of the location of the reporting unit where the abnormality has occurred, The moving direction recognition means for recognizing the moving direction of the monitoring target, the reporting unit that has transmitted the abnormality occurrence signal, and the imaging means of the reporting unit that exists in the moving direction recognized by the moving direction recognition means from the reporting unit The direction estimation means for estimating the direction of the monitoring target, and the imaging means of each notification unit estimated by the direction estimation means. The communication unit is controlled to transmit a remote control signal indicating a direction to each reporting unit, and an image generated by the imaging unit of each reporting unit whose imaging direction is controlled by the imaging control unit according to the remote control signal Monitoring control means for receiving data and voice data generated by the sound collecting means.
[0026]
In such a monitoring and reporting system, in order to monitor the surrounding abnormality by the monitoring server, the monitoring server stores in advance the location data of each reporting unit, recognizes the moving direction, and according to the moving direction. Control the imaging direction of the imaging means of the reporting unit.
[0027]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0028]
The present invention is applied to, for example, a monitoring and reporting system configured as shown in FIG.
[0029]
[Configuration of monitoring and reporting system]
This monitoring and reporting system includes a plurality of reporting units 2A, 2B, 2C (hereinafter simply referred to as “reporting unit 2” when referred to collectively) on the optical transmission network 1 and each reporting unit 2. Are connected to a monitoring server 3 for managing
[0030]
The monitoring server 3 transmits audio data and image data from each reporting unit 2 via the optical transmission network 1, stores the received audio data in the audio storage unit, and stores the image data in the image storage unit. . In addition, the monitoring server 3 includes a plurality of monitoring monitors 11 for presenting images and sounds to a supervisor who is a user, and an abnormality determination display monitor 12 for causing the supervisor to determine an abnormality. Furthermore, a sound emission mechanism is provided that emits an audio signal corresponding to the image displayed on the monitoring monitor 11 and the abnormality determination display monitor 12. Thereby, in the monitoring server 3, it is possible to monitor the emergency which generate | occur | produced around the notification unit 2 by the supervisor.
[0031]
Furthermore, the monitoring server 3 includes an operation input mechanism that is operated by a supervisor, and a camera remote control signal, a voice recognition setting signal, and an image recognition control for the notification unit 2 in response to the operation input mechanism being operated. Send a signal to the reporting unit 2. Other processes performed by the monitoring server 3 will be described later.
[0032]
In this monitoring and reporting system, for example, in the vicinity of an intersection road, as shown in FIG. 2, the reporting units 2A to 2G are installed on a road in a certain area outside through an interval and connected to the optical transmission network 1. It is configured. As shown in FIG. 3, each notification unit 2 is provided with a microphone 26, an omnidirectional camera 41, and a movable camera 42 on a pole 2 a provided on the road. In this example, the reporting unit 2 is provided with a microphone 26 on the pole 2a at a substantially head position when the user is walking, and a movable camera 42 and an omnidirectional camera 41 are provided on the pole 2a above the user's head. Yes.
[0033]
The reporting unit 2 acquires sound and images around the installation location and transmits them to the monitoring server 3. The notification unit 2 includes a media converter 21 and a router 22 as a communication function for communicating with the monitoring server 3 via the optical transmission network 1.
[0034]
In the notification unit 2, when transmitting voice and images to the monitoring server 3, communication data with the destination set as the monitoring server 3 is created by the router 22, and various signal conversions are performed by the media converter 21 to generate an optical signal as an optical transmission network. 1 to send. Further, in the notification unit 2, the optical signal from the monitoring server 3 is received by the media converter 21 and the router 22 through the optical transmission network 1 and is output to the camera server 23 and the audio signal processing unit 24.
[0035]
"Voice data processing by the reporting unit 2"
Next, processing related to voice data by the reporting unit 2 and its functional configuration will be described.
[0036]
Further, the notification unit 2 has a configuration for transmitting audio data to the monitoring server 3 as an audio signal processing unit 24, a speaker unit 25, a plurality of microphones 26A, 26B, and 26C (hereinafter simply referred to as “microphone 26”). And a call switch 27 is provided.
[0037]
As shown in FIG. 4 when the pole 2a is viewed from above, the microphones 26A to 26C are arranged on the side surface of the pole 2a through a certain distance d without being arranged on the same straight line. When the microphone 26 receives sound around the reporting unit 2, the microphone 26 generates a sound signal from the sound and outputs the sound signal to the sound signal processing unit 24.
[0038]
When the audio signals from the microphones 26A to 26C are input, the audio signal processing unit 24 converts each audio signal into audio data that is digital data of a predetermined level by an amplifier and an A / D converter (not shown). Send to.
[0039]
When the sound data is sent from the microphones 26 </ b> A to 26 </ b> C, the environmental sound removing unit 31 performs a process for removing the environmental noise component on each sound data, and a plurality of sound data including the user's sound existing in the vicinity of the reporting unit 2. Is sent to the sound source direction estimation unit 32 and the speech recognition unit 33.
[0040]
When a plurality of sound data from the environmental sound removal unit 31 is sent, the sound source direction estimation unit 32 performs a correlation process using the plurality of sound data. At this time, the sound source direction estimation unit 32 adds the two different audio data whose sampling times are shifted, recognizes the combination of the audio data having the highest signal level, and detects the audio detection direction of the microphone 26 combined with the audio data. Is a sound source arrival direction candidate. More specifically, the sound source direction estimating unit 32 adds, for example, voice data detected by the microphone 26A and voice data detected by the microphone 26B, and calculates the voice level. The sound source direction estimation unit 32 also performs the same calculation for the combination of the microphone 26A and the microphone 26C and the combination of the microphone 26B and the microphone 26C, and recognizes the combination having the highest sound level among the three combinations. A sound source arrival direction candidate. Then, the sound source direction estimation unit 32 estimates direction estimation information θv that is the direction of the monitoring target from a plurality of sound source arrival direction candidates.
[0041]
Note that the sound source direction estimation unit 32 not only estimates the direction estimation information θv using only the microphones 26 attached to the same pole 2a, but also transmits the audio data generated by the plurality of notification units 2 to the optical transmission network. 1 may be used to estimate the direction estimation information θv.
[0042]
In addition, the sound source direction estimation unit 32 may estimate the direction of the monitoring target by obtaining the time difference or phase difference between the sound detection times of the microphones 26. Further, in the sound source direction estimation unit 32, each microphone 26 is a side surface of the pole 2a installed perpendicular to the ground, and is attached to another microphone 26 at a predetermined interval on a horizontal plane with respect to the ground. If there is, the arrival time difference or phase difference of the sound wave of each microphone 26 may be obtained to estimate the horizontal plane direction to be monitored. Furthermore, in the sound source direction estimation unit 32, each microphone 26 is a side surface of the pole 2a installed perpendicular to the ground, and is placed on a surface perpendicular to the ground with a predetermined distance from another microphone 26. When attached, the arrival time difference or phase difference of the sound wave of each microphone 26 may be obtained to estimate the vertical plane direction of the monitoring target. Thereby, in the sound source direction estimation part 32, the arrival direction of the abnormal sound from the monitoring object can be estimated accurately.
[0043]
When voice data is sent from the environmental sound removal unit 31, the voice recognition unit 33 extracts an acoustic feature vector from the voice data, and stores the extracted feature vector and the abnormal feature sound database storage unit 34 in advance. Speech recognition is performed by matching with word dictionary data or sentence dictionary data expressed by the feature vector. Here, the word dictionary and the sentence dictionary include, for example, a feature vector indicating a voice of “help” and a feature vector indicating a voice of “stop”, which are supposed to be uttered at the time of abnormality. .
[0044]
In addition, the abnormal feature sound database storage unit 34 stores not only audio data when an abnormality occurs when the monitoring target is a living body, but also an acoustic sound indicating a collision sound or destruction sound of an object when the monitoring target is an object. Data may be stored, and the presence or absence of abnormality of the monitoring target may be recognized using the acoustic data. Thereby, the voice recognition unit 33 can recognize a specific abnormal sound with reference to the contents of the abnormal feature sound database storage unit 34.
[0045]
Further, the abnormal feature sound database storage unit 34 stores at least one of the fundamental frequency, power spectrum, formant, cepstrum, and temporal displacement of the sound data or sound data as a feature vector (feature of sound data or sound data). (Amount) may be stored.
[0046]
Furthermore, when the voice data is given as the sampled time series data s (t), the abnormal feature sound database storage unit 34 performs Fourier transform such as FFT (Fast Fourier Transform) shown in the following equation. The voice data is converted into a frequency function, and the frequencies corresponding to the plurality of peaks f1 to f3 of the frequency function are approximately formant as shown in FIG.
[0047]
S (exp (−jwm)) = Σs (t) · exp (−jwm)
Then, the abnormal feature sound database storage unit 34 stores formants as feature values of the voice data.
[0048]
Further, the abnormal feature sound database storage unit 34 obtains a cepstrum C (n) by performing inverse Fourier transform on the logarithm of the frequency function S (exp (−jwm)) in the above equation as in the following equation, A cepstrum may be stored as a feature amount of audio data.
[0049]
C (n) = (1 / N) ΣlogY · exp (2πk / N)
Then, the voice recognition unit 33 selects a feature vector that is closest to the extracted feature vector from the feature vectors stored in the abnormal feature sound database storage unit 34. Then, the speech recognition unit 33 calculates the degree of abnormal sound (hereinafter referred to as speech abnormal similarity Av) based on the distance between the selected feature vector and the extracted feature vector. The voice recognition unit 33 stores the calculated voice abnormality similarity Av in the voice storage unit 35.
[0050]
The data stored in the abnormal feature sound database storage unit 34 is not limited to a human voice, and may be a feature vector of a car collision sound, for example. As a result, not only a human voice but also an abnormal sound such as a collision or destruction of an object can generate the sound abnormality similarity Av.
[0051]
In addition, the audio signal processing unit 24 includes an environmental feature sound database storage unit 36 in which a characteristic vector of a specific environmental sound is stored according to the location where the reporting unit 2 is installed. The environmental feature sound database storage unit 36 stores feature vectors of environmental sounds that are not abnormal sounds, such as train crossing sounds and pedestrian crossing warning sounds.
[0052]
The feature vector of the environmental sound is sent from the audio signal processing unit 24 to the monitoring server 3 via the router 22 and the media converter 21, and a voice recognition setting signal indicating that the monitoring server 3 determines that the sound is not an abnormal sound. Is registered in the environmental feature sound database storage unit 36. The monitoring server 3 may automatically perform a process for determining whether or not the environmental sound from the reporting unit 2 is an abnormal sound by using an environmental characteristic sound database prepared in advance. It may be done manually.
[0053]
The feature vector of the environmental sound stored in the environmental feature sound database storage unit 36 is read by the speech recognition unit 33 when the speech recognition unit 33 recognizes the abnormal sound, and determines whether or not it is an abnormal sound. Compared to the feature vector. The speech recognition unit 33 does not determine that the sound is abnormal when the feature vector for determining whether or not the sound is abnormal is close to the feature vector of the environmental sound. Thereby, the reporting unit 2 can improve the reliability of abnormality detection without erroneously recognizing environmental sound as abnormal sound.
[0054]
The sound storage unit 35 stores sound data previously detected by the microphone 26 and sound abnormality similarity Av corresponding to the sound data. The voice data and the voice abnormality similarity Av stored in the voice storage unit 35 can be downloaded by the monitoring server 3 via the optical transmission network 1 when the supervisor operates the monitoring server 3. ing.
[0055]
The voice abnormality similarity Av includes those set by a voice recognition setting signal from the monitoring server 3. That is, when the monitoring server 3 refers to the voice data stored in the voice storage unit 35 of each reporting unit 2 and determines that the voice data corresponds to an abnormal sound, the monitoring server 3 converts the voice data into the voice recognition unit 33. And the feature vector is added to the abnormal feature sound database storage unit 34 as an abnormal sound. As described above, in the reporting unit 2, the feature vector stored in the abnormal feature sound database storage unit 34 or the environmental feature sound database storage unit 36 is added by the monitoring server 3, so that abnormality detection is performed compared to the initial installation time. Reliability can be improved.
[0056]
The audio signal processing unit 24 includes a VOIP (Voice over IP (Internet Protocol)) processing unit 37 to which audio data is output from the environmental sound removal unit 31. The VOIP processing unit 37 sends the audio data from the environmental sound removal unit 31 to the monitoring server 3 when the call switch 27 is operated.
[0057]
"Image data processing by report unit 2"
Next, processing related to image data by the reporting unit 2 and its functional configuration will be described.
[0058]
The notification unit 2 includes a camera server 23, an omnidirectional camera 41, a movable camera 42, and an input sensor 43 as a configuration for transmitting an image signal to the monitoring server 3.
[0059]
The omnidirectional camera 41 collects light with a lens having a wide viewing angle, and generates an image signal with an internal CCD (Charge Coupled Device) image sensor. The movable camera 42 has a pan function, a tilt function, and a zoom function, and generates an image signal by a CCD image sensor. The omnidirectional camera 41 and the movable camera 42 are connected to the camera server 23 and output image signals to the camera server 23.
[0060]
In the camera server 23, when an image signal obtained by capturing the surrounding situation of the notification unit 2 is input from the omnidirectional camera 41, the image signal is A / D converted and sent to the environmental image noise removing unit 51 as image data. When the environmental image noise removing unit 51 receives image data, the environmental image noise removing unit 51 removes noise from the image data and sends it to the image recognition unit 52.
[0061]
The image recognition unit 52 has a functional configuration as shown in FIG. 6 and performs image recognition processing on image data from the omnidirectional camera 41 and the movable camera 42.
[0062]
In the image recognition unit 52, when image data is input, the object image extraction unit 61 divides the background image and other object images by performing image processing such as a background subtraction method. The object image is image data indicating, for example, a person or a car. In the image recognition unit 52, for example, when image data of a still image is input, the feature amount extraction unit 62 recognizes in-image position information, size information, color information, and the like of the object image, and each piece of the information is recognized. Convert to features for the object. In addition, when a moving image over a plurality of frames is input, the feature amount extraction unit 62 converts motion speed information of the object image as an image feature amount.
[0063]
In the image recognition unit 52, the movement direction estimation unit 63 obtains direction estimation information that is a candidate for the direction θi of the monitoring target (hereinafter referred to as an object) corresponding to the object image from the position information of the object image. To the integrated direction detection unit 53.
[0064]
In addition, the camera server 23 includes a feature image database storage unit 54 that accumulates and stores feature amounts of objects that are assumed to be abnormal. When the feature amount of the object is obtained, the image recognition unit 52 performs matching processing with the feature amount of the object stored in the feature image database storage unit 54, and calculates the feature amount of the obtained object and the feature amount of the accumulated object. Is used to calculate the distance between feature vectors. In the image recognition unit 52, the abnormal image similarity calculation unit 64 obtains an abnormal image similarity Aj indicating the degree of abnormality based on the distance between the feature vectors.
[0065]
Further, the image recognition unit 52 stores the image data from the environmental image noise removal unit 51 in the image storage unit 55.
[0066]
Further, the camera server 23 includes, as an input sensor 43, a plurality of human body detection sensors for detecting human bodies in different directions using, for example, infrared rays. This input sensor 43 specifies a human body detection sensor that detects a human body among a plurality of human body detection sensors when a person object exists around the reporting unit 2. And this input sensor 43 estimates the direction in which a human body object exists from the human body detection direction of the specified human body detection sensor, and sends it to the integrated direction detection part 53 as direction estimation information.
[0067]
In addition, as the input sensor 43, as shown in FIG. 7 when the notification unit 2 is viewed from above, a plurality of detection ranges that are different in the imaging region of the omnidirectional camera 41 or the movable camera 42 are set. You may provide distance sensor 44A-44D. The distance sensors 44 </ b> A to 44 </ b> D send direction estimation information including distance information to the integrated direction detection unit 53 when a monitoring target such as a human body is detected. Each distance sensor 44 may be an ultrasonic sensor or an optical sensor.
[0068]
Further, the input sensor 43 may be composed of a plurality of infrared sensors within the imaging area of the omnidirectional camera 41 or the movable camera 42 and having different detection ranges. As a result, each infrared sensor sends direction estimation information indicating an area where the monitoring target exists to the integrated direction detection unit 53 using the infrared information reflected and detected from the monitoring target.
[0069]
The integrated direction detection unit 53 determines the direction with respect to the reporting unit 2 where the object exists from each direction estimation information from the sound source direction estimation unit 32, the image recognition unit 52, and the input sensor 43, and sends it to the camera control unit 56.
[0070]
The camera control unit 56 sets the pan and tilt amount Δθ and the zoom amount ΔZ of the movable camera 42 from the object direction determined by the integrated direction detection unit 53. At this time, the camera control unit 56 uses the direction estimation angle θ of the movable camera 42 for setting the pan and tilt amount Δθ as the audio abnormality similarity Av, the image abnormality similarity Ai, and the direction estimation information θv using the audio. Using the direction estimation information θi using the image,
θ = (Av × θv + Ai × θi) / (Av + Ai)
Is obtained by performing the following calculation. That is, the camera control unit 56 uses the audio abnormality similarity Av and the image abnormality similarity Ai as the weighting coefficients of the direction estimation information θv and the direction estimation information θi. Then, the camera control unit 56 determines the pan and tilt amount Δθ from the current imaging direction of the movable camera 42 with respect to the obtained direction estimation angle θ.
[0071]
In addition, when the difference between the direction estimation information θv and the direction estimation information θi is small, the camera control unit 56 sets the zoom setting value Z for setting the zoom amount ΔZ to the direction estimation information θi and the direction estimation information θv. When the zoom setting value Z is increased and the difference between the direction estimation information θv and the direction estimation information θi is large, the object existence probability in the direction estimation information θi and the direction estimation information θv is low. Therefore, the zoom setting value Z is set to be small.
[0072]
At this time, the camera control unit 56 uses, for example, the direction estimation information θv and the direction estimation information θi,
Z = α / (θv−θi)
α: Constant
The zoom setting value Z is obtained by performing the following calculation. Then, the camera control unit 56 determines the zoom amount ΔZ from the current zoom setting value of the movable camera 42 with respect to the obtained zoom setting value Z.
[0073]
Then, the camera control unit 56 drives the movable camera 42 by the pan and tilt amount Δθ and the zoom amount ΔZ to cause the movable camera 42 to image an object, and image data captured by the movable camera 42 is image recognition unit. 52.
[0074]
"Abnormality judgment processing"
Next, processing related to abnormality determination by the reporting unit 2 and the monitoring server 3 and its functional configuration will be described.
[0075]
The image recognition unit 52 calculates the degree of abnormality A by integrating the above-described audio abnormality similarity Av and image abnormality similarity Ai when making an abnormality determination. At this time, the image recognition unit 52 uses, for example, the following equation:
A = α × Ai + β × Av
α, β: Constant
Perform the following operation. Accordingly, the image recognition unit 52 calculates the degree of abnormality A for the reporting unit 2 and sends it to the monitoring server 3 via the optical transmission network 1. Further, the image recognition unit 52 calculates an abnormal value A, and when the abnormal value A becomes higher than a preset threshold value, the image and sound captured by the movable camera 42 are monitored. It may be transmitted to the server 3.
[0076]
In response to this, the monitoring server 3 displays a plurality of images transmitted from the reporting unit 2 on the monitor 11 for monitoring or the monitor 12 for abnormality determination display, so that the abnormality degree A becomes equal to or greater than the threshold for the monitor. The image and sound around the reporting unit 2 is monitored.
[0077]
Moreover, in the monitoring server 3, as shown in FIG. 8, in the own monitoring area 71 where there are a plurality of reporting units 2 connected via the optical transmission network 1, as shown in FIG. When the degree of abnormality A of the reporting unit 2-1 is low, it is assumed that image data captured by any reporting unit 2-2, 2-3, 2-4 is monitored. When the abnormality degree A of the reporting unit 2-1 exceeds the threshold value, information to that effect is transmitted from the reporting unit 2-1 to the monitoring server 3.
[0078]
In response to this, the monitoring server 3 determines that an abnormality has occurred in the vicinity of the notification unit 2-1, and notifies the notification unit 2-11, the notification unit 2-12 in the vicinity of the notification unit 2-1. A control signal for transmitting image data and audio data captured by the reporting unit 2-13 is transmitted to each reporting unit 2. In addition, the monitoring server 3 transmits a camera remote control signal with the direction of the reporting unit 2-1 in the imaging direction to the reporting unit 2-11, the reporting unit 2-12, and the reporting unit 2-13. In addition, the monitoring server 3 transmits a control signal enabling voice communication by the VOIP processing unit 37 to the reporting unit 2-1 in which the degree of abnormality A is equal to or greater than the threshold, and the call switch 27 of the reporting unit 2-1 It is possible to make a call between the user who has operated and the monitor on the monitoring server 3 side. Thereby, in the reporting unit 2-1, the voice switch 27 is operated and the voice data detected by the microphone 26 is transmitted to the monitoring server 3.
[0079]
As a result, the reporting units 2-11, 12 and 2-13 image the direction according to the camera remote control signal by the movable camera 42, and transmit the image data and the audio data to the monitoring server 3, It becomes an active state that enables monitoring by a monitor. In the monitoring server 3, when image data and audio data are sent from the reporting unit 2-1, the reporting unit 2-11, the reporting unit 2-12, and the reporting unit 2-13, the monitoring monitor 11 and the abnormality determination display monitor 12 are sent. By confirming the image and sound, the monitoring by the supervisor is performed with a lot of information.
[0080]
In this monitoring and reporting system, when the degree of abnormality A becomes high, not only the reporting unit 2-1, but also the image data acquired by the reporting unit 2-11, the reporting unit 2-12, and the reporting unit 2-13, and By storing audio data, it is possible to analyze information on areas where abnormality has occurred from various angles.
[0081]
In such a monitoring and reporting system, since only the reporting unit 2 exceeding the abnormality degree A is monitored by the monitoring server 3, it is not necessary to monitor all the reporting units 2 on the monitoring server 3 side. Therefore, according to this monitoring notification system, monitoring tasks on the monitoring server 3 side can be reduced.
[0082]
Further, in this monitoring notification system, image data obtained by capturing with the omnidirectional camera 41 and the movable camera 42 is stored in the image storage unit 55, and in response to a download request from the monitoring server 3 to the camera server 23. Since image data can be transmitted, a past image can be downloaded and analyzed by a supervisor.
[0083]
"Abnormality monitoring process"
Next, in the monitoring notification system configured as described above, a processing procedure of the abnormality monitoring processing that can be realized in addition to the processing described above will be described with reference to FIGS.
[0084]
In the abnormality monitoring process of the reporting unit 2 shown in FIG. 9, first, the movable mechanism (not shown) is controlled by the camera control unit 56 so that the angle of the movable camera 42 is set to the initial position at the start of the process (step S1). The zoom function of the movable camera 42 is controlled by the camera control unit 56 so as to maximize the degree of out (step S2). As a result, the reporting unit 2 is in a state of generating wide-angle image data using the image signal captured by the movable camera 42.
[0085]
In such a state, the notification unit 2 uses the image data captured and acquired by the movable camera 42 to obtain at least the image abnormality similarity Ai by the image recognition unit 52 and calculates the abnormality A. Process. Then, in the reporting unit 2, when it is determined that the degree of abnormality A is lower than the predetermined threshold (step S3), the processing of step S1 and step S2 is repeated to determine that the degree of abnormality A is higher than the predetermined threshold. If so (step S3), the image recognition unit 52 performs a process of extracting a moving object to be monitored from wide-angle image data generated by imaging with the zoom function set in step S2 (step S3). Step S4). Then, the image recognizing unit 52 creates direction estimation information θi from the extracted position of the moving object and sends it to the integrated direction detecting unit 53.
[0086]
Next, in the reporting unit 2, the integrated direction detection unit 53 estimates the imaging direction of the movable camera 42 using at least the direction estimation information θi, and the movable camera 42 is set so that the moving object becomes the center of the image data. Control (step S5) and control the movable camera 42 to zoom in (step S6). As a result, the reporting unit 2 can generate image data including the moving object at the center of the image and store it in the image storage unit 55 or transmit it to the monitoring server 3.
[0087]
Next, in the abnormality monitoring process of the reporting unit 2 shown in FIG. 10, the processes of steps S1 to S3 are performed in the same manner as the abnormality monitoring process shown in FIG. 9, and the abnormality degree A is higher than a predetermined threshold value in step S3. When it is determined that the height is high, the image recognition unit 52 extracts a moving object using the omnidirectional image data generated by imaging with the omnidirectional camera 41 (step S11). Then, the image recognition unit 52 creates direction estimation information θi from the moving object extraction position and sends it to the integrated direction detection unit 53.
[0088]
In response to this, the notification unit 2 calculates the presence position of the moving object by using the integrated direction detection unit 53 using at least the direction estimation information θi (step S12), and is moved by the camera control unit 56 from the calculated value of the presence position. The imaging direction (angle) of the camera 42 is controlled (step S13), and the movable camera 42 is controlled to zoom in (step S6). As a result, the reporting unit 2 can generate image data including a moving object and store it in the image storage unit 55 or transmit it to the monitoring server 3.
[0089]
Next, in the abnormality monitoring process of the reporting unit 2 shown in FIG. 11, first, in step S21, the voice recognition unit 33 determines the direction of the voice from the monitoring target for the plurality of microphones 26A to 26C. Determine the time. Then, in the voice recognition unit 33, in step S22, the voice data (channel signal) detected and generated by each microphone 26A, microphone 26B, and microphone 26C is shifted and added by the delay time determined in step S21. , The delay sum S (λ) is obtained. At this time, the speech recognition unit 33 obtains a delay sum for the combination of the microphone 26A and the microphone 26B, the microphone 26A and the microphone 26C, and the combination of the microphone 26B and the microphone 26C.
[0090]
As a result, when it is determined in step S23 that the delay sum S (λ) has been calculated for all the combinations, that is, all angles, the speech recognition unit 33 obtains the delay sum S (λ) having the largest value. Is sent to the integrated direction detector 53 as direction estimation information θv (step S24). Further, the voice recognition unit 33 may obtain the direction estimation information θv by obtaining the time difference or phase difference of the voice detection time.
[0091]
Next, the integrated direction detection unit 53 obtains an estimated direction captured by the movable camera 42 from at least the direction estimation information θv, and the camera control unit 56 controls the imaging direction of the movable camera 42.
[0092]
Next, in the abnormality monitoring process of the monitoring server 3 shown in FIG. 12, the monitoring server 3 receives the voice data SA (t), which is time-series data from the reporting unit 2A (step S31), and receives from the reporting unit 2B. The voice data SB (t) that is time series data is received (step S32), and the voice data SC (t) that is time series data from the reporting unit 2C is received (step S33).
[0093]
Next, the monitoring server 3 calculates the correlation of the voice data between the reporting units 2 using each voice data SA (t), voice data SB (t), and voice data SC (t) (step S34). . Then, the monitoring server 3 estimates the arrival direction of the sound emitted by the monitoring target from the correlation calculation result, and uses the camera remote control signals for setting the arrival direction as the imaging direction as the notification unit 2A, the notification unit 2B, and the notification unit 2C. Create and send about. Thereby, the reporting unit 2A, the reporting unit 2B, and the reporting unit 2C can control the movable camera 42 so that the arrival direction estimated in step S35 is the imaging direction.
[0094]
According to the monitoring and reporting system that performs such abnormality monitoring processing, it is possible to obtain image data of an abnormal situation from a plurality of angles by directing a plurality of reporting units 2 with different installation locations to a single voice arrival location. , Can be analyzed from various abnormal situations.
[0095]
Next, in the abnormality monitoring process of the monitoring server 3 shown in FIG. 13, first, the monitoring unit 11 is installed in the notification unit 2A presenting an image and sound to the monitor, and in the vicinity of the notification unit 2A. The monitoring server 3 receives the voice data SA (λ), SB (λ), and SC (λ) acquired by the reporting unit 2B and the reporting unit 2C (step S41).
[0096]
In the monitoring server 3, when the abnormality degree A of the reporting unit 2A exceeds a predetermined threshold and an abnormality occurrence signal is received from the reporting unit 2A (step S42), the reporting unit 2B and the reporting unit around the reporting unit 2A 2C is set as a monitoring target (step S43), and the correlation between the audio data SA (λ), SB (λ), and SC (λ) received in step S41 is calculated (step S44). Then, the monitoring server 3 estimates the arrival direction of the sound emitted by the monitoring target from the correlation calculation result, and creates a camera remote control signal for the notification unit 2B and the notification unit 2C for setting the arrival direction as the imaging direction. Send. Thereby, the reporting unit 2B and the reporting unit 2C can control the movable camera 42 so that the arrival direction estimated in step S35 is the imaging direction.
[0097]
According to the monitoring and reporting system that performs such an abnormality monitoring process, the imaging direction of the reporting unit 2 around the reporting unit 2 in which the abnormality has occurred can be set as the voice arrival direction. Even if the place of occurrence moves, the surrounding situation can be grasped.
[0098]
Next, in the abnormality monitoring process of the monitoring server 3 shown in FIG. 14, installation location data indicating the installation locations of the plurality of notification units 2 is stored in the monitoring server 3 in advance, and the monitoring area 71 (the installation area of the notification unit 2) is stored. ), The reporting unit 2A having the highest degree of abnormality A is extracted (step S51). In this example, for example, when the degree of abnormality A exceeds a predetermined value, information indicating the degree of abnormality A is automatically transmitted from the reporting unit 2 to the monitoring server 3.
[0099]
Next, the monitoring server 3 determines whether or not the reporting unit 2 having the highest degree of abnormality A has been changed from the reporting unit 2A to the reporting unit 2B (step S52). Then, the monitoring server 3 extracts the installation locations of the reporting unit 2A and the reporting unit 2B from the installation location data, and arranges the installation location data in the order of occurrence of the abnormality to recognize the moving direction of the monitoring target. Thereby, in the monitoring server 3, the extending direction of the monitoring target moving direction (the reporting unit 2A to the reporting unit 2B) is set as the imaging direction of the movable camera 42 (step S53).
[0100]
According to the monitoring and reporting system that performs such abnormality monitoring processing, the imaging direction of the reporting unit 2 can be the moving direction of the monitoring target in which an abnormality has occurred, so that the monitoring target moves and the location of occurrence of the abnormality moves. Even if it is a case, the surrounding situation can be grasped.
[0101]
[Effect of the embodiment]
As described in detail above, according to the monitoring and reporting system to which the present invention is applied, when a user in the vicinity of the reporting unit 2 encounters a dangerous state, the monitoring server 3 is appropriately notified to the monitoring server 3. By responding quickly, a safer environment can be provided.
[0102]
In addition, according to this monitoring and reporting system, when a large number of reporting units 2 are set and audio and images acquired by a large-scale client are transmitted to the monitoring server 3, each reporting unit having an abnormality degree A equal to or greater than a threshold value. 2 and the image and sound are transmitted only from the surrounding reporting unit 2, so that the monitoring task of the monitoring server 3 is reduced by reducing the information transmission amount of each reporting unit 2, and the monitoring response response in emergency normal time is increased, Furthermore, it is possible to eliminate the need for special operations at the whistleblower.
[0103]
Furthermore, according to this monitoring and reporting system, when a crime or an accident occurs in the area where the reporting unit 2 is installed, a user or victim in the vicinity of the reporting unit 2 operates the reporting button of the reporting unit 2. Therefore, it is not necessary to manually switch the camera mechanism of the reporting unit 2 on the monitoring server 3 side and control the angle and zoom of the camera mechanism.
[0104]
Furthermore, according to this monitoring and reporting system, in addition to reporting by the user operating the call switch 27 and manual control of the movable camera 42 by the monitor of the monitoring server 3, the police and It is possible to make a report to a security company. For example, according to this monitoring and reporting system, when a walking user is attacked by a criminal such as snatching and screams, etc., the reporting unit 2 detects that an abnormal state has occurred, and the user or crime The user's voice and image can be reported.
[0105]
The above-described embodiment is an example of the present invention. For this reason, the present invention is not limited to the above-described embodiment, and various modifications can be made depending on the design and the like as long as the technical idea according to the present invention is not deviated from this embodiment. Of course, it is possible to change.
[0106]
【The invention's effect】
According to the present invention, when using a large-scale system that acquires images and sounds of a large number of places, when transmitting sounds and images to a monitoring server, the amount of information transmitted is reduced and the server side In addition to reducing the monitoring task, it is possible to increase the monitoring response response in the normal emergency, and to eliminate the need for special operation by the reporter.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a functional configuration of a monitoring notification system to which the present invention is applied.
FIG. 2 is a diagram for explaining an arrangement example of notification units in a monitoring notification system to which the present invention is applied.
FIG. 3 is a side view of a notification unit of a monitoring notification system to which the present invention is applied.
FIG. 4 is a top view showing an installation example of a microphone of a notification unit in the monitoring notification system to which the present invention is applied.
FIG. 5 is a diagram for explaining a formant as a feature amount of voice data stored in an abnormal feature sound database storage unit;
FIG. 6 is a block diagram illustrating a functional configuration of an image recognition unit.
FIG. 7 is an explanatory diagram when creating direction estimation information when a distance sensor is used as an input sensor.
FIG. 8A is a diagram for explaining active reporting units that are monitored by the monitoring server when the degree of abnormality is low when a large number of reporting units are set in the monitoring area; (B) is a diagram for explaining an active reporting unit that is monitored by the monitoring server when the degree of abnormality is high.
FIG. 9 is a flowchart showing an example of abnormality monitoring processing by a reporting unit.
FIG. 10 is a flowchart showing another example of the abnormality monitoring process by the reporting unit.
FIG. 11 is a flowchart showing still another example of abnormality monitoring processing by a reporting unit.
FIG. 12 is a flowchart illustrating an example of an abnormality monitoring process performed by a monitoring server.
FIG. 13 is a flowchart illustrating another example of the abnormality monitoring process performed by the monitoring server.
FIG. 14 is a flowchart showing still another example of abnormality monitoring processing by the monitoring server.
FIG. 15 is a block diagram illustrating a specific configuration example of a conventional monitoring notification system.
[Explanation of symbols]
1 Optical transmission network
2 reporting unit
3 Monitoring server
11 Monitor for monitoring
12 Abnormality judgment display monitor
21 Media Converter
22 routers
23 Camera server
24 Audio signal processor
25 Speaker section
26 microphone
27 Call switch
31 Environmental sound removal unit
32 Sound source direction estimation unit
33 Voice recognition unit
34 Abnormal feature sound database storage
35 Voice memory
36 Environmental characteristic sound database storage
37 VOIP processing section
41 Omnidirectional camera
42 Movable camera
43 Input sensor
51 Environmental image noise removal unit
52 Image recognition unit
53 Integrated direction detector
54 Feature Image Database Storage Unit
55 Image storage
56 Camera control unit
61 Object image extraction unit
62 Feature extraction unit
63 Movement direction estimation unit
64 Abnormal image similarity calculator
71 Monitoring area

Claims (1)

In a monitoring and reporting system in which a plurality of reporting units and a monitoring server that presents sound and images generated by the reporting unit to a monitor are connected via a communication line,
Each reporting unit is
Imaging means for imaging a monitoring target and generating image data;
Sound collecting means for collecting sound around the imaging means to generate sound data, sound recognition means for recognizing the status of the monitoring target from the sound data generated by the sound collecting means, at least image data, Communication means for communicating voice data with the monitoring server, and when the voice recognition means determines that an abnormality has occurred in the status of the monitoring target, a communication means for transmitting an abnormality occurrence signal to the monitoring server;
Imaging control means for controlling the imaging direction of the imaging means by a remote control signal from the monitoring server,
The monitoring server is
A communication means for communicating with the reporting unit;
The installation location data indicating the installation locations of the plurality of notification units is stored, and when an abnormality occurrence signal is received from the plurality of notification units, the installation of the plurality of notification units in which an abnormality has occurred with reference to the installation location data A moving direction recognition means for extracting the location and recognizing the moving direction of the monitoring target according to the order of the installation location of the reporting unit in which an abnormality has occurred;
A notification unit that transmits the abnormality occurrence signal, and a direction estimation unit that estimates a direction of the monitoring target with respect to the imaging unit of the notification unit that exists in the movement direction recognized by the movement direction recognition unit from the notification unit;
The communication unit is controlled to transmit a remote control signal indicating the imaging direction of the imaging unit of each notification unit estimated by the direction estimation unit to each notification unit, and imaging is performed by the imaging control unit according to the remote control signal. A monitoring and reporting system comprising: monitoring control means for receiving image data generated by the imaging means of each reporting unit whose direction is controlled, and audio data generated by the sound collecting means.

Images