JP5192414B2 - Audio information display system - Google Patents

Description

  The present invention relates to a video display system including a plurality of cameras and a plurality of microphones.

  With the widespread use of IP-capable imaging devices such as Web cameras, it has become possible to construct a wide-area and large-scale monitoring system using an IP network. In such a monitoring system, it is possible to acquire a large amount of monitoring images, and multi-faceted monitoring is possible. Recently, there are also applications in which images of 100 or more cameras are displayed side by side on the screen of a monitoring monitor terminal to perform monitoring work.

However, if a camera is to be installed in a wide area without a blind spot, it requires a great deal of cost and a large number of monitor devices, making it difficult to construct an efficient monitoring system. Therefore, by installing a camera that can control the shooting direction in places where there is a high need for monitoring, and directing the camera in the direction in which abnormal noise occurred in the monitored area, it is possible to minimize the blind spot of the camera for monitoring. Has been proposed (see Patent Document 1).
JP 2006-254277 A

  However, in the technique disclosed in Patent Document 1, when a suspicious person knows the operation of the camera, the camera is turned outside the suspicious person's action range by throwing an object and making a sound. There is a problem that the position of the person outside the shooting range.

  In addition, when abnormal sounds occur in a plurality of directions, there is a problem that the direction of the camera cannot be determined uniquely. In addition, if the camera is pointed in the direction in which the abnormal sound is generated, there is a risk that other important places cannot be photographed. Usually, the installation position of the camera is oriented in an important direction, and it is preferable that surrounding information can be grasped while shooting an image in that direction.

  For example, it is conceivable to install a microphone in order to grasp information on the blind spot of the camera. However, if the collected sound is directly listened to by the monitor, problems such as difficulty in grasping the situation when the sound is generated simultaneously by multiple microphones, or the content of the conversation can be grasped. There is a possibility of infringement.

  An object of the present invention is to grasp information on the blind spot of a camera without degrading the monitoring efficiency in consideration of not infringing on privacy while utilizing audio information.

In a typical embodiment of the present invention, an audio information display system including a camera, a microphone to which audio is input, and a display device that displays an image captured by the camera, the audio input to the microphone It includes a sound processing device for generating audio information, and a video processor for processing the images captured by the camera, sound type information including a type of the voice, a position where the microphone is installed on the basis of The microphone management information and the camera management information including the position where the camera is installed are managed, and the voice processing device analyzes the voice input to the microphone to thereby display the voice recognition result that can be displayed on the display device. acquires, on the basis of the sound type information, estimates the type of voice input to the microphone, on the basis of the microphone management information and the camera control information, The position of the sound source of the sound input to the microphone is estimated, and sound information including information indicating the acquired sound recognition result, the estimated sound type, and the position of the estimated sound source is generated. The generated audio information is transmitted to the video processing device, and the video processing device combines the audio information received from the audio processing device with the video shot by the camera, and the audio information is synthesized. The video is displayed on the display device, and when the sound processing device estimates the position of the sound source of the audio input to the microphone, the audio is captured and the audio information is combined with the audio information. When the distance to the camera is larger than a predetermined threshold, the direction in which the microphone to which the sound is input is set as information indicating the position of the sound source .

  According to one aspect of the present invention, it is possible to grasp the situation of the blind spot of the camera that is difficult to grasp only by the video by displaying the video in which the audio information is synthesized (superimposed).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram showing an example of an audio information display monitoring system according to an embodiment of the present invention.

  The audio information display monitoring system provides video information and audio information collected by a plurality of cameras 110 and a plurality of microphones 111 installed in a building or the like to an administrator or a monitor of the monitoring center 2.

  The audio information display monitoring system according to the embodiment of the present invention includes an input device group (1A, 1B) including a plurality of cameras 110 and a plurality of microphones 111, and a monitoring center 2 that controls the input device group and displays a monitoring video. Including.

  The monitoring center 2 includes an audio processing PC 3, a camera control PC 4, a display PC 5, and a video storage PC 6.

  The sound processing PC 3 acquires sound from the microphone 111 and generates sound information. The camera control PC 4 acquires the video imaged by the camera 110 and transmits the acquired video image to the display PC 5.

  The display PC 5 generates a monitoring video obtained by synthesizing (superimposing) the audio information generated by the audio processing PC 3 on the video transmitted from the camera control PC 4. When the audio information corresponding to the video is not transmitted from the audio processing PC 3, the video transmitted from the camera control PC 4 is set as the monitoring video. Note that the generated audio information includes information related to an event estimated by analyzing the acquired audio.

  A monitor device 50 that displays a monitoring video is connected to the display PC 5. The monitor device 50 can display the monitoring videos corresponding to the plurality of cameras 110 side by side for each camera. An administrator or a monitor monitors the video displayed on the monitor device 50. The video storage PC 6 stores the monitoring video in a storage device.

  As described above, when the display PC 5 receives the audio information from the audio processing PC 3, the monitor device 50 displays the monitoring video in which the event information is superimposed on the video of the camera 110 near the microphone in which the event corresponding to the audio information has occurred. To display. The microphone 111 is fixed at a predetermined position and is associated with a predetermined camera 110. The sound information (event information) obtained by processing the sound acquired by the microphone 111 is combined with the video photographed by the corresponding camera 110 and displayed on the monitor device 50, thereby generating the blind spot of the camera. It is possible to monitor changes that are difficult to notice in events and images. The display PC 5 can display the video of the camera 110 specified by the administrator with high definition.

  Details of each component will be described below.

<Configuration of input device group>
The input device group (1A, 1B) is managed by the monitoring center 2. Each input device group is set for each predetermined area (for example, each floor) such as a building or for each camera or microphone. The input device group includes a plurality of cameras 110 and a plurality of microphones 111. In the following description, since the configuration of each input device group is the same, the same reference numerals are assigned to the same elements, and redundant descriptions are omitted.

  The number of cameras 110 and the number of microphones 111 do not have to be the same and may be different. Furthermore, the camera 110 and the microphone 111 may be installed at the same position, or may be installed at different positions. The camera 110 and the microphone 111 may have an integrated configuration or may be a separate housing.

  The camera 110 installed at a predetermined position captures a moving image having a predetermined resolution at a predetermined frame rate. The captured moving image (video) is transmitted via the communication network 100. The communication network 100 is configured by a network cable or a dedicated cable.

  The camera 110 is connected to the camera control PC 4 via the communication network 100. The camera 110 is, for example, an IP camera that can be connected to a network. The camera 110 may be a camera that outputs video in response to an external request, or may be a camera that outputs video unilaterally from the camera side.

  The microphone 111 installed at a predetermined position is connected to the voice processing PC 3 via the communication network 100. Other communication means may be used as long as the sound can be transmitted from the microphone 111 to the sound processing PC 3. The microphone 111 may be a microphone that can specify the direction of the sound source by a single device, such as a microphone array.

  A plurality of cameras 110 and microphones 111 are installed for each base. The position information of the camera 110 is managed by the camera control PC 4. The position information of the microphone 111 is managed by the sound processing PC 3.

<Configuration of monitoring center>
Next, the components of the monitoring center 2 (audio processing PC 3, camera control PC 4, display PC 5 and video storage PC 6) will be described with reference to FIGS.

  FIG. 2 is a diagram illustrating an example of the configuration of the audio processing PC 3 according to the embodiment of this invention.

  As described above, the sound processing PC 3 generates sound information including event information by analyzing the sound acquired from the microphone 111, and transmits the generated sound information to the display PC 5. The voice processing PC 3 is a computer including a CPU 201, a memory 202, an input unit 203, a communication unit 204, and a storage unit 210.

  The CPU 201 executes various processes by processing a program stored in the memory 202. The memory 202 stores a program executed by the CPU 201 and data necessary for executing the program.

  The input unit 203 is an interface for accepting input of information from a user or the like. Specifically, an input device such as a keyboard or a mouse. The communication unit 204 is an interface for communicating with an external device. For example, a network interface card. The voice processing PC 3 receives input of voice information from the microphone 111 connected to the communication network 100 via the communication unit 204.

  The storage unit 210 stores programs and data necessary for audio processing. Specifically, the storage unit 210 includes an OS 211, microphone management information 212, abnormal voice data 213, abnormal conversation data 214, a voice acquisition program 215, a voice presence / absence determination program 216, a sound source direction estimation program 217, and a voice type determination program 218. A voice recognition program 219 and an event notification program 220 are stored.

  The OS 211 is an operating system necessary for executing various programs.

  The microphone management information 212 stores identification information, installation information, and the like of the installed microphone 111. Details of the microphone management information 212 will be described with reference to FIG.

  The abnormal sound data 213 is information for specifying the type of sound input to the microphone 111. For example, it is information for identifying that the sound input to the microphone 111 is a footstep or for identifying an abnormal sound such as a destructive sound. Details of the abnormal audio data 213 will be described with reference to FIG.

  The abnormal conversation data 214 stores information for determining whether or not a suspicious word is included in the conversation when the voice input to the microphone 111 is a conversation. It is determined whether or not there is a problem with an event that has occurred by determining whether or not a word or a word included in the conversation of a suspicious person or an emergency conversation is included in the conversation. Details of the abnormal conversation data 214 will be described with reference to FIG.

  The sound acquisition program 215 acquires sound from the microphone 111. The sound presence / absence determination program 216 determines whether or not sound is input to the microphone 111.

  The sound source direction estimation program 217 estimates the direction of the sound source of the sound based on the sound input to the microphone 111 and the microphone management information 212.

  The sound type determination program 218 determines the sound type based on the sound input to the microphone 111 and the abnormal sound data 213. The type of sound corresponds to the event that caused the sound to be emitted.

  The speech recognition program 219 determines whether or not a word having a high possibility of abnormality is included in the conversation based on the content of the conversation and the abnormal conversation data 214 when the type of the sound input to the microphone 111 is conversation. Determine.

  The event notification program 220 transmits audio information including event information generated by the audio acquisition program 215, the sound source direction estimation program 217, the audio type determination program 218, and the audio recognition program 219 to the display PC 5.

  FIG. 3 is a diagram showing an example of the configuration of the camera control PC 4 according to the embodiment of the present invention.

  As described above, the camera control PC 4 acquires a video from the camera 110 and transmits the video to the display PC 5. The camera control PC 4 is a computer that includes a CPU 301, a memory 302, an input unit 303, a communication unit 304, and a storage unit 310.

  The CPU 301 executes various processes by processing a program stored in the memory 302. The memory 302 stores a program executed by the CPU 301 and data necessary for executing the program.

  The input unit 303 is an interface for accepting input by an administrator or a monitor. Specifically, an input device such as a keyboard or a mouse. The communication unit 304 is an interface for communicating with an external device. For example, a network interface card. The camera control PC 4 receives video input from the camera 110 connected to the communication network 100 via the communication unit 304.

  The storage unit 310 stores programs and data necessary for acquiring or transmitting video. Specifically, the storage unit 310 stores an OS 311, camera management information 312, a camera video acquisition program 313, and a camera video transmission program 314.

  The OS 311 is an operating system necessary for executing various programs.

  The camera management information 312 stores identification information and installation information of the installed camera 110. Details of the camera management information 312 will be described with reference to FIG.

  The camera video acquisition program 313 acquires video captured by the designated camera 110. The camera video transmission program 314 transmits video to the display PC 5.

  FIG. 4 is a diagram showing an example of the configuration of the display PC 5 according to the embodiment of the present invention.

  The display PC 5 superimposes and displays the audio information transmitted from the audio processing PC 3 on the video imaged by the camera 110. The display PC 5 is a computer including a CPU 401, a memory 402, an input unit 403, a display unit 404, a communication unit 405, and a storage unit 410.

  The CPU 401 executes various processes by processing a program stored in the memory 402. The memory 402 stores a program executed by the CPU 401 and data necessary for executing the program.

  The input unit 403 is an interface for accepting input of information from a user or the like. Specifically, an input device such as a keyboard or a mouse. For example, the designation of the camera 110 that displays the video is accepted.

  The display unit 404 outputs a video signal in which the audio information transmitted from the audio processing PC 3 is superimposed on the video received from the camera control PC 4 and causes the monitor device 50 to display the video signal.

  The communication unit 405 is an interface for communicating with an external device. For example, a network interface card. The display PC 5 receives input of audio information and video transmitted from the audio processing PC 3 and the camera control PC 4 connected to the communication network 100 via the communication unit 405.

  The storage unit 410 stores programs and data necessary for displaying video. Specifically, the storage unit 410 stores an OS 411, an event acquisition program 412, a video processing program 413, a display control program 414, a video storage program 415, and a stored video acquisition program 416.

  The OS 411 is an operating system necessary for executing various programs.

  The event acquisition program 412 acquires audio information including event information transmitted by the audio processing PC 3. The video processing program 413 generates a video in which audio information is superimposed on video information received from the camera control PC 4.

  The display control program 414 controls display of video information. For example, videos taken by a plurality of cameras 110 are displayed simultaneously, or videos taken by a designated camera 110 are enlarged and displayed.

  The video storage program 415 transmits the video and audio information received by the display PC 5 to the video storage PC 6. The stored video acquisition program 416 acquires video and audio information stored in the video storage PC 6.

  FIG. 5 is a diagram showing an example of the configuration of the video storage PC 6 according to the embodiment of the present invention.

  The video storage PC 6 is a computer including a CPU 501, a memory 502, an input unit 503, a communication unit 504, and a storage unit 510. The video storage PC 6 stores the video displayed on the display PC 5. Furthermore, when audio information is superimposed on the acquired video, audio and audio information corresponding to the video is stored. The display PC 5 can acquire and browse video data and audio data stored in the video storage PC 6 as needed.

  The CPU 501 executes various processes by processing the program stored in the memory 502. The memory 502 stores a program executed by the CPU 501 and data necessary for executing the program.

  The input unit 503 is an interface for receiving an input from an administrator or a monitor. Specifically, an input device such as a keyboard or a mouse. The communication unit 504 is an interface for communicating with an external device. For example, a network interface card. The video storage PC 6 receives video data and audio information data from the display PC 5 connected to the communication network 100 via the communication unit 504.

  The storage unit 510 stores video data 512 and audio information data 513. Further, it stores programs and data necessary for accumulating and managing video and audio information. Specifically, the storage unit 510 stores an OS 511, a video accumulation program 514, a video search program 515, and a data management program 516.

  The OS 511 is an operating system necessary for executing various programs.

  The video data 512 is a video shot by the camera 110 and is transmitted to the video storage PC 6 via the camera control PC 4 and the display PC 5.

  The audio information data 513 is audio input to the microphone 111 and information generated based on the audio, and is transmitted from the audio processing PC 3 to the video storage PC 6 via the display PC 5.

  The video accumulation program 514 receives video and audio information transmitted from the display PC 5 and stores them in the storage unit 510. The video search program 515 searches for the specified video, acquires the video data 512 and the audio information data 513, and transmits them to the display PC 5.

  The data management program 516 is a program for managing the video data 512 and the audio information data 513. The data management program 516 may be a general-purpose program or a dedicated program specialized for managing the video data 512 and the audio information data 513.

  Note that the audio information display monitoring system may be configured by integrating the audio processing PC 3, the camera control PC 4, the display PC 5, and the video storage PC 6 into one or a plurality of PCs.

  FIG. 8 is a diagram illustrating an example of the microphone management information 212 according to the embodiment of this invention.

  The microphone management information 212 includes a microphone ID 801, an address 802, a base ID 803, an installation position 804, and a peripheral camera ID 805.

  The microphone ID 801 is an identifier of the microphone 111. Address 802 is access destination information of the microphone 111. In the embodiment of the present invention, an IP address is used as the access destination information.

  The site ID 803 is an identifier of the site where the microphone 111 is installed. The installation base corresponds to, for example, a building or a room where the input device group is installed. The installation position 804 is information indicating the installation position in the installation base. Specifically, it is represented by coordinates where the microphone 111 is installed. In the embodiment of the present invention, the installation angle of the microphone 111 is further included. The peripheral camera ID 805 is an identifier of the camera 110 installed around the microphone 111 identified by the microphone ID 801.

  FIG. 9 is a diagram illustrating an example of the camera management information 312 according to the embodiment of this invention.

  The camera management information 312 includes a camera ID 901, an address 902, a base ID 903, an installation position 904, and a peripheral microphone ID 905.

  The camera ID 901 is an identifier of the camera 110. An address 902 is access destination information of the camera 110. In the embodiment of the present invention, like the microphone 111, an IP address is used as access destination information.

  The site ID 903 is an identifier of the site where the camera 110 is installed. Similar to the microphone 111, the installation base corresponds to a building or a room where the input device group is installed. The installation position 904 is information indicating the installation position in the installation base. The installation position 904 is configured by coordinates and an installation angle similarly to the installation position 804 of the microphone 111. The peripheral microphone ID 905 is an identifier of the microphone 111 installed around the camera 110 identified by the camera ID 901.

  FIG. 10 is a diagram illustrating an example of the abnormal sound data 213 according to the embodiment of this invention.

  The abnormal voice data 213 includes a voice ID 1001, a voice type 1002, and an importance level 1003.

  The voice ID 1001 is a voice type identifier. The audio type 1002 is the type of audio input to the microphone 111. For example, it may be a voice that is uttered during normal times such as conversation or footsteps, or a voice that is highly likely to be uttered when an abnormality such as a scream or a hitting sound occurs.

  The importance 1003 is the importance of the voice type 1002, and a higher importance is set for a voice having a higher possibility of abnormality. In the embodiment of the present invention, values from 1 to 5 are set, but they may be further subdivided.

  FIG. 11 is a diagram illustrating an example of the abnormal conversation data 214 according to the embodiment of this invention.

  The abnormal conversation data 214 includes a conversation ID 1101, text 1102, reading 1103, and importance 1104.

  The conversation ID 1101 is an identifier for identifying a word or a word included in the conversation. Text 1102 is a word or word included in the conversation. Registered in the text 1102 are words (abnormal voices) that are recalled mainly by suspicious acts.

  Reading 1103 is information indicating reading of the text 1102. Abnormal speech is extracted by acquiring a word or word that matches the reading 1103 from the content of the conversation when the speech type collected from the microphone 111 is “conversation”.

  The importance 1104 is the importance of the conversation content, and the higher importance is set for the voice information having a higher possibility of abnormality. In the embodiment of the present invention, a value from 1 to 5 is set as in the case of the abnormal sound data 213.

<Processing procedure>
Next, the procedure of each process of the audio information display monitoring system according to the embodiment of this invention will be described. First, a procedure for processing voice input from the microphone 111 by the voice processing PC 3 will be described.

  FIG. 6 is a flowchart illustrating a procedure for generating voice information from the voice input to the microphone 111 according to the embodiment of this invention.

  The CPU 201 of the voice processing PC 3 first acquires voice data from each microphone 111 based on the microphone management information 212 by executing the voice acquisition program 215 (S601). Note that the audio acquisition process in step S601 may be a method of receiving audio unilaterally transmitted from the microphone 111, or a method of receiving audio from the microphone 111 in response to an audio acquisition request from the audio processing PC3. It may be.

  Next, the CPU 201 of the sound processing PC 3 determines whether or not sound is included in the sound data acquired in the process of step S601 by executing the sound presence / absence determination program 216 (S602).

  The CPU 201 of the sound processing PC 3 determines whether or not sound is included in the sound data based on the determination result of step S602 (S603). The determination of the presence / absence of sound is performed by acquiring sound at a predetermined time, and determining whether sound data at a predetermined time is included in the sound data at the acquired time. Also, different thresholds for sound pressure levels may be provided for each microphone 111.

  When the acquired audio data is a silent section (result of S603 is “N”), the CPU 201 of the audio processing PC 3 returns to the process of the audio acquisition step S601.

  On the other hand, the CPU 201 of the sound processing PC 3 acquires the sound by the microphone 111 by executing the sound source direction estimation program 217 when the acquired sound data includes sound (result of S603 is “Y”). The sound source direction of the voice is estimated (S604). The sound source direction may be estimated based on information obtained by a plurality of microphones, as disclosed in Japanese Patent Laid-Open No. 2008-92512.

  As another method for acquiring the direction of the sound source, a microphone having the maximum sound pressure is acquired, and the sound source direction is determined based on the microphone management information 212 shown in FIG. 8 and the camera management information 312 shown in FIG. The direction may be estimated.

  In the method of estimating the direction of the sound source using the positional information of the camera 110 and the microphone 111, a threshold distance X for determining whether or not the camera and the microphone are handled as the same place is set in advance. When the microphone 111 having a certain sound pressure or more is separated from the camera 110 by a distance X or more, it is difficult to specify the position of the sound source, and therefore the position of the microphone is set as the sound source position. This will be described in more detail with reference to FIG.

  FIG. 12 is a diagram illustrating an example of an installation layout of the camera 110 and the microphone 111 according to the embodiment of this invention.

  In the layout shown in FIG. 12, three cameras 110 (C01 to C03) and four microphones 111 (M01 to M04) are installed.

  Here, when the sound pressure of the microphone M04 is high, the camera C03 is first identified as a peripheral camera based on the microphone management information 212 shown in FIG. Further, the distance between the microphone M04 and the camera C03 is calculated based on the installation position 804 of the microphone M04 and the installation position 904 of the camera C03. If the calculated distance is greater than or equal to X, the direction of the sound source with respect to the camera C03 is set to a direct downward direction, that is, 270 °.

  Similarly, when the sound pressure of the microphone M03 is high, the camera C03 is identified as a peripheral camera. If the distance between the microphone M03 and the camera C03 is X or less, the direction of the sound source with respect to the camera C03 is the direction of the sound source estimated by the microphone M03.

  Even if the camera 110 and the microphone 111 are in different directions even if they are installed at the same position, the angle is corrected based on the installation position information registered in the microphone management information 212 and the camera management information 312. .

  Here, the description returns to the flowchart of the procedure for acquiring the audio information in FIG.

  When the estimation of the sound source direction is completed, the CPU 201 of the audio processing PC 3 executes the audio type determination program 218 to estimate the type of audio such as a destructive sound or a footstep based on the abnormal audio data 213 (S605). A known technique can be used for the process of estimating the voice type in step S605. For example, Japanese Patent Laid-Open No. 2001-31292 discloses a technique for acquiring the type of sound by using sound spectrum information.

  Further, the CPU 201 of the voice processing PC 3 determines whether or not the voice type is conversation (S606). If the speech type is determined to be conversation (the result of S606 is “Y”), the content of the conversation is determined by executing the speech recognition program 219 to determine whether a predetermined text of the conversation content is included. Is recognized (S607). The predetermined text is the text 1102 included in the abnormal conversation data 214 shown in FIG.

  For speech recognition, a method based on HMM (Hidden Markov Model) widely known in the field of speech recognition can be used. For example, Japanese Patent Application Laid-Open No. 2008-58503 discloses a method capable of performing speech recognition with a certain degree of reliability only in the utterance portion of a speaker in an environment where a conversation between a speaker who is a target of speech recognition and another person can always exist. Is disclosed.

  It is also possible to use publicly available open source speech recognition software Julius (http://julius.sourceforge.jp). In addition, as long as it is information indicating the content of speech, information other than words may be acquired such as acquiring the degree of conversation risk based on acoustic information such as intonation information.

  Next, the CPU 201 of the sound processing PC 3 transmits the sound information acquired in the processing of steps S604 to S607 to the display PC 5 by executing the event notification program 220 (S608). When notifying an event by transmitting audio information, an identifier for identifying the microphone 111 and an identifier of the corresponding camera 110 are also transmitted.

  Next, the CPU 201 of the voice processing PC 3 determines whether or not an end instruction has been received (S609). If an end instruction has been received (the result of S609 is “Y”), this process ends. If an end instruction has not been received (the result of S609 is “N”), the process returns to the process of step S601, and this process is continued.

  Next, a procedure for displaying the monitoring video to which the audio information is added on the monitor device 50 will be described.

  FIG. 7 is a flowchart illustrating a procedure for generating and displaying a monitoring video to which audio information is added according to the embodiment of this invention.

  First, the CPU 301 of the camera control PC 4 acquires a video from the camera 110 by referring to the camera management information 312 by executing the camera video acquisition program 313 (S701). Further, the camera video transmission program 314 is executed to transmit the video to the display PC 5. The video acquisition method may be a method of acquiring a video by transmitting a video acquisition request from the camera control PC 4 to the camera 110, or a video unilaterally transmitted from the camera 110 to the camera control PC 4. The acquisition method may be used.

  Next, the CPU 401 of the display PC 5 acquires the audio information transmitted by the audio processing PC 3 by executing the event acquisition program 412 (S702). Furthermore, the presence / absence of an event is determined based on whether or not audio information is received (S703, S704).

  In the embodiment of the present invention, the CPU 401 of the display PC 5 receives the monitoring video shot by the camera 110 from the camera control PC 4 and displays the monitoring video together with the identification information of the camera 110 that shot the monitoring video. If the CPU 401 of the display PC 5 determines that an event has occurred (the result of S704 is “Y”), it executes the video processing program 413 and further superimposes the audio information received from the audio processing PC 3 on the video data ( S705).

  The audio information to be superimposed on the video information is, for example, a symbol indicating the direction of the sound source such as an arrow and information indicating the type of audio. Furthermore, when a word specified by the abnormal conversation data 214 is extracted from the conversation, the extracted word together with the voice type information is superimposed on the monitoring video.

  Note that the period in which the audio information is superimposed on the monitoring video is within a preset time from the event occurrence. Moreover, according to importance, you may make it display audio | voice information by changing the magnitude | size or color of a character. Also, information indicating the sound properties such as the sound pressure level or the sound distribution for each frequency component may be displayed. In addition, when an event with a high level of importance occurs when multiple videos are displayed, draw the attention of the administrator by highlighting the frame of the area in which the video in which the event occurred is displayed. May be displayed as follows. Hereinafter, an example of the monitoring video displayed by the display PC 5 will be described with reference to FIG.

  FIG. 13 is a diagram illustrating an example of a screen displaying a monitoring video displayed by the display PC 5 according to the embodiment of the present invention.

  In the example of the screen shown in FIG. 13, the display control program 414 displays a plurality of videos in which videos of cameras to be displayed are arranged. The number of displayed images may be the number of cameras 110 installed. If the number of installed cameras 110 exceeds the number M of videos that can be displayed on the screen, the screen selected by the administrator or M videos that have been received immediately before the audio input event are displayed side by side. Also good.

  In each image, information indicating a shooting base such as “base A” and “base B” is displayed. Further, the detailed installation position of the camera 110 that is photographing may be displayed, or the identification information of the camera 110 may be displayed.

  Also, the audio information received from the audio processing PC 3 is displayed on the video 132 or the like. In the video 132, the footstep sound of the passerby is detected by the microphone 111, and the voice processing PC3 notifies the display PC5. As a result, the voice type “footstep” and an arrow indicating the direction of the passerby are displayed. .

  Also, in the video 133, a conversation “dangerous” in the right direction is detected at the location A relative to the screen. In addition, in the video 136 and the video 137, it is possible to confirm that, at the site B, an impact sound is generated at the blind spot of the camera 110 and an abnormality has occurred.

  In particular, when abnormal sound with high importance such as impact sound is detected, as shown in the video 136 and the video 137 in FIG. 13, the frame displaying the video is highlighted and displayed. To pull.

  Returning to the description of the flowchart showing the procedure for generating and displaying the monitoring video in FIG.

  The CPU 401 of the display PC 5 executes the display control program 414 to generate a display video in which video images captured by the camera 110 are arranged as shown in FIG. 13 (S706). Further, the display video generated on the display unit 404 (monitor device 50) is output (S707).

  Next, the CPU 401 of the display PC 5 transmits the video and audio information from the display PC 5 to the video storage PC 6 by executing the video storage program 415. The CPU 501 of the video storage PC 6 stores the video and audio information transmitted from the display PC 5 in the video data 512 and the audio information data 513 by executing the video storage program 514 (S708).

  Further, the CPU 401 of the display PC 5 can display the past monitoring video stored in the video storage PC 6 by executing the stored video acquisition program 416. The CPU 401 of the display PC 5 transmits to the video storage PC 6 the conditions such as the shooting date and time input in order to display the past monitoring video designation.

  The CPU 501 of the video storage PC 6 searches the video data 512 and the audio information data 513 that satisfy the specified conditions by executing the video search program 515, and transmits it to the display PC 5. The CPU 401 of the display PC 5 displays the requested video based on the received video data and audio data.

  The CPU 501 of the video storage PC 6 deletes old video and audio information so that the free capacity becomes equal to or greater than the threshold when the free capacity of the storage unit 510 is less than a predetermined threshold. To do.

  Finally, the CPU 401 of the display PC 5 determines whether an end instruction has been received from the administrator (S709). When an end instruction is received from the administrator (the result of S710 is “Y”), this process ends. If no termination instruction has been received from the administrator (the result of S710 is “N”), the process returns to the process of step S701, and this process is executed continuously from the process of acquiring video data.

  According to the embodiment of the present invention, it is possible to superimpose and display audio information on video captured by a large number of cameras. Therefore, since an abnormality that has occurred at the blind spot of the camera 110 and an abnormality that is difficult to grasp from the video can be visually displayed, the monitoring target can be efficiently monitored.

  Further, according to the embodiment of the present invention, by displaying the speech type and the recognition result of the abnormal conversation, it is possible to reduce the possibility of infringing privacy by displaying the specific contents of the conversation.

It is a block diagram which shows an example of the audio | voice information display monitoring system of embodiment of this invention. It is a figure which shows an example of a structure of the audio | voice processing PC of embodiment of this invention. It is a figure which shows an example of a structure of camera control PC of embodiment of this invention. It is a figure which shows an example of a structure of display PC of embodiment of this invention. It is a figure which shows an example of a structure of the image | video storage PC of embodiment of this invention. It is a flowchart which shows the procedure which produces | generates audio | voice information from the audio | voice input into the microphone of embodiment of this invention. It is a flowchart which shows the procedure which produces | generates and displays the monitoring image | video with which the audio | voice information of embodiment of this invention was provided. It is a figure which shows an example of the microphone management information of embodiment of this invention. It is a figure which shows an example of the camera management information of embodiment of this invention. It is a figure which shows an example of the abnormal audio | voice data of embodiment of this invention. It is a figure which shows an example of the abnormal conversation data of embodiment of this invention. It is a figure which shows an example of the installation layout of the camera and microphone of embodiment of this invention. It is a figure which shows an example of the screen which displayed the monitoring image | video displayed by the display PC of embodiment of this invention.

1 Input device group 2 Monitoring center 3 Voice processing PC
4 Camera control PC
5 Display PC
6 Video storage PC
50 Monitor 100 Communication Network 110 Camera 111 Microphone 201 CPU
202 Memory 203 Input unit 204 Communication unit 210 Storage unit 211 OS
212 Microphone management information 213 Abnormal voice data 214 Abnormal conversation data 215 Voice acquisition program 216 Voice presence / absence determination program 217 Sound source direction estimation program 218 Voice type determination program 219 Voice recognition program 220 Event notification program 301 CPU
302 Memory 303 Input Unit 304 Communication Unit 310 Storage Unit 311 OS
312 Camera management information 313 Camera image acquisition program 314 Camera image transmission program 401 CPU
402 Memory 403 Input unit 404 Display unit 405 Communication unit 410 Storage unit 411 OS
412 Event acquisition program 413 Video processing program 414 Display control program 415 Video storage program 416 Stored video acquisition program 501 CPU
502 Memory 503 Input unit 504 Communication unit 510 Storage unit 511 OS
512 Video data 513 Audio information data 514 Video storage program 515 Video search program 516 Data management program

Claims (7)

An audio information display system including a camera, a microphone to which audio is input, and a display device that displays an image captured by the camera,
An audio processing device that generates audio information based on audio input to the microphone;
A video processing device that processes video captured by the camera;
Managing audio type information including the type of audio , microphone management information including a position where the microphone is installed , and camera management information including a position where the camera is installed ;
The voice processing device
By analyzing the voice input to the microphone, a voice recognition result that can be displayed on the display device is obtained,
Based on the audio type information, estimate the type of audio input to the microphone,
Based on the microphone management information and the camera management information , estimate the position of the sound source of the audio input to the microphone,
Generating voice information including information indicating the acquired voice recognition result, the estimated voice type, and the position of the estimated sound source;
Transmitting the generated audio information to the video processing device;
The video processing device includes:
Synthesizing audio information received from the audio processing device with video captured by the camera;
Displaying the synthesized image of the audio information on the display device ;
When the sound processing apparatus estimates the position of the sound source of the sound input to the microphone, the distance between the microphone to which the sound is input and the camera that has captured the video in which the sound information is synthesized is a predetermined threshold value. If it is larger, the direction in which the microphone to which the sound is input is set as information indicating the position of the sound source . In the audio type information, importance is set for each type of audio,
The audio information display system according to claim 1, wherein the video processing device displays the audio information based on the importance. The voice information display system includes abnormal conversation information including a predesignated word,
The voice processing device
When the estimated voice type is conversation, by recognizing the voice input to the microphone, information indicating the content of the conversation is generated,
The voice information is generated based on the abnormal conversation information when a word included in the abnormal conversation information is included in information indicating the content of the conversation. Voice information display system. In the abnormal conversation information, importance is set for each word,
The audio information display system according to claim 3, wherein the video processing device displays the audio information based on the importance.   The audio information display system according to claim 1, wherein the audio information includes information indicating a sound pressure of the audio input to the microphone.   2. The audio information display system according to claim 1, wherein the video processing apparatus preferentially displays the video combined with the audio information when displaying a plurality of videos taken by the camera. 3. .   The audio information display system further includes a video storage device including a storage unit that stores video captured by the camera.
  The video processing device transmits a video to be displayed on the display device to the video storage device,
  The video storage device
  Storing the video transmitted from the video processing device in the storage unit;
  The audio information display system according to claim 1, wherein the requested video is provided when an acquisition request for the video stored in the storage unit is received.