JP2011238985A - Method and device for displaying sound source estimation image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a device capable of creating and displaying a sound source estimation image, which shows a sound source direction estimated when taking a image, while sound information and image information are being collected.SOLUTION: The device includes a sound/image collection unit 10 in which microphones M1 to M5 are integrated with a camera 12. In the method, sound pressure waveform data and image data, obtained by performing an A/D conversion on a sound pressure signal and an image signal collected by using the sound/image collection unit 10, are stored temporarily in a buffer 31. Predetermined sampling numbers of the sound pressure waveform data are extracted in order from the buffer 31, and a sound source direction is estimated. The image data corresponding to the sound pressure waveform data used to estimate the sound source direction are extracted from the buffer 31, and the sound source direction data and the image data are synthesized to generate the sound source estimation image G, on which a graphic to indicate the sound source direction is drawn, and be displayed on a display screen 40M of a display device 40. The sound pressure waveform data and the image data used for generating the sound source estimation image G are eliminated thereafter.

Description

  The present invention relates to a method and an apparatus for displaying an image for estimating a sound source using sound information collected by a plurality of microphones and video information photographed by a photographing means.

Conventionally, as a method of estimating the direction of sound arrival, a microphone array in which a large number of microphones are arranged at equal intervals is constructed, and the direction of the sound source, which is the direction of sound wave arrival, is estimated from the phase difference of each microphone relative to the reference microphone. A so-called acoustic technique has been devised (see, for example, Non-Patent Document 1).
On the other hand, not a phase difference between output signals of a plurality of microphones arranged at a measurement point, but a plurality of microphone pairs arranged in a straight line intersecting each other from the plurality of microphones, and the position between the two microphones constituting the pair There has been proposed a method for estimating the direction of a sound source from a ratio between an arrival time difference corresponding to a phase difference and an arrival time difference between two pairs of other microphones (see, for example, Patent Documents 1 to 3).

Specifically, as shown in FIG. 7, two microphone pairs (M1, M3) and microphone pairs (four microphones M1 to M4) are arranged at predetermined intervals on two straight lines orthogonal to each other. M2, M4) arranged so as to constitute a structure with the microphone pair (M1, M3) the arrival time difference D ₁₃ of sound pressure signals to be input to the microphone M1, M3 constituting the said microphone pairs (M2, M4) The horizontal angle θ between the measurement point and the position of the sound source is estimated from the ratio of the arrival time difference D ₂₄ between the sound pressure signals input to the microphones M2 and M4, and the fifth microphone M5 is formed by the microphones M1 to M4. 4 microphone pairs (M5, M1), (M5, M2), (M5, M3), (M5, M4) are arranged in positions not on the plane, and the front The elevation angle φ formed by the measurement point and the position of the sound source is estimated from the arrival time differences D ₁₃ , D ₂₄ and D _5j (j = 1 to 4) between the microphones constituting each microphone pair.
The sound source direction measured from the measurement point can be expressed by the horizontal angle θ and the elevation angle φ.

Thereby, compared with the case where the sound source direction is estimated using the microphone array, the sound source direction can be accurately estimated with a small number of microphones.
Also, at this time, a video sampling means such as a CCD camera is provided to take an image of the estimated sound source direction, and then the image data and the sound source direction data are combined to produce the estimated sound source direction in the video. If the sound source estimation image in which the sound pressure level is displayed in a graphic form is displayed on a display screen such as a display, the sound source can be visually grasped.
Simultaneously with the sound collection, the image collection means continuously shoots the image, and the sound pressure waveform data that is the sound information and the image data that is the image information are stored in the hard disk of the computer. After collecting the information and video information, the sound pressure waveform data is extracted from the hard disk to estimate the sound source direction, and the image data corresponding to the sound pressure waveform data used for the sound source direction estimation calculation is extracted from the hard disk. A method of displaying a sound source estimation image by combining the image data and sound source direction data is also performed.

Japanese Patent Laid-Open No. 2002-181913 JP 2006-324895 A JP 2008-224259 A

Toshiro Oga, Yoshio Yamazaki, Yutaka Kaneda; Acoustic system and digital processing, Corona, 1995

By the way, in the conventional method, since all of the measured sound information and video information are stored on the hard disk, not only must a large-capacity hard disk be used, but also the sound information and video. Since the calculation of the sound source direction and the creation of the sound source estimation image are performed after collecting the above information, it takes time to estimate the sound source.
Therefore, it is desired to develop a method and an apparatus that can easily estimate a sound source in the field by creating and displaying a sound source estimation image while collecting sound information and video information.

  The present invention has been made in view of conventional problems, and creates and displays a sound source estimation image displaying a sound source direction estimated in a video while collecting sound information and video information. It is an object of the present invention to provide a method and apparatus capable of performing the above.

The invention according to claim 1 of the present application collects a sound pressure signal, which is sound information, and a video signal, which is video information, using a sound / video sampling unit in which a plurality of microphones and photographing means are integrated. A method for displaying a sound source estimation image for estimating a sound source using the collected sound pressure signal data and image data, wherein the sound pressure is sound information using the sound / video unit. A step (a) of simultaneously collecting a signal and a video signal which is video information, and A / D converting the collected sound pressure signal and the video signal to temporarily store them in a buffer as sound pressure waveform data and image data, respectively. The sound pressure waveform data for the analysis time length used for the sound source estimation is extracted from the buffer and collected by the plurality of microphones using the extracted sound pressure waveform data. A step (c) of calculating a phase difference of the pressure signal to estimate a sound source direction, and a step (d) of extracting image data for the analysis time length used for estimating the sound source direction in the step (c) from the buffer ) And the sound source direction data estimated in the step (c) and the image data extracted in the step (d), and a sound source estimation image in which a figure showing the estimated sound source direction is drawn. A step (e) of generating a sound source estimation image, a step (f) of displaying the sound source estimation image on a display means, and a sound pressure waveform data and an image used to create the sound source estimation image after the generation of the sound source estimation image And (g) erasing data from the buffer.
As a result, sound source estimation images can be created and sequentially displayed while collecting sound information and video information, so that the operator can view the sound source estimation video displayed by the display means in the field. The sound source can be estimated by looking.
When data for identifying the sound source is actually required, a sound source estimation image is created by collecting the sound information and the video information from the sound source direction estimated by the worker in the field. And save it. At this time, since the operator has already estimated the sound source, not only the sound pressure waveform data and the image data stored in the storage means such as the hard disk can be reduced, but also the calculation time can be reduced. And can be estimated efficiently.

A second aspect of the present invention is the sound source estimation image display method according to the first aspect, wherein in the step (g), the size of the free area of the buffer is at least the analysis time length. When the waveform data and the image data have a size that can be stored, the sound pressure waveform data and the image data used to create the sound source estimation image are deleted from the buffer, and the step (a) A step (h) for issuing a measurement start command for starting estimation of the sound source direction is provided between the steps (b), and it is determined whether or not the estimation of the sound source direction is to be ended after the step (g). Step (i) is provided, and in the step (b), A / D conversion is performed on the collected sound pressure signal and the video signal after a measurement start command is issued, and as sound pressure waveform data and image data, respectively. If the sound source direction estimation is not terminated in step (i), the process returns to step (h). If the sound source direction estimation is terminated, the sound source direction is temporarily stored in the buffer. The stored sound pressure waveform data and image data are stored in storage means different from the buffer. The maximum value of the predetermined free area is preferably 1/5 or less of the storage capacity of the buffer. Note that the minimum value is a size that can store the sound pressure waveform data and the image data for the analysis time length.
By adopting such a configuration, sound pressure waveform data and image data for constructing a sound source estimation image of the sound source estimated by the operator when the worker can estimate the sound source by visually recognizing the display means. Therefore, the sound source estimation image can be easily reproduced.
In addition, sound pressure waveform data and image data effective for creating a sound source estimation image can be saved by appropriately performing the measurement start and measurement end timings of the sound source estimation, so that the sound source can be estimated efficiently. .

The invention described in claim 3 is the sound source estimation image display method according to claim 1 or 2, wherein the sound source direction is estimated between step (b) and step (c). Is provided with a step (j) for issuing a retrograde measurement command for measuring retroactively by a preset retrograde time length, and in step (c), a predetermined retrograde time length is traced from the time when the retrospective measurement command is issued. Sound pressure waveform data from the time until the time elapsed by the analysis time length is extracted from the buffer, and the sound source direction is estimated using the extracted sound pressure waveform data.
As a result, even if sudden sounds or intermittent sounds occur, it is possible to display sound source estimation images for estimating the direction of the sources of these sounds, thereby eliminating measurement failures. be able to.

According to a fourth aspect of the present invention, there is provided a microphone group having two pairs of microphones arranged at predetermined intervals on two mutually intersecting straight lines, a photographing means for photographing a sound source direction image, and a display means. An image in which a graphic showing the direction of the sound source is drawn from the sound pressure signal of the sound propagating from the sound source collected by the microphone group and the image signal of the direction of the sound source taken simultaneously with the collection of the sound pressure signal. A sound source estimation image display device for displaying a sound source estimation image on a display means, wherein the sound pressure signal collected by each microphone and the video signal photographed by the photographing means are each converted into a digital signal. A converter and the A / D converted sound pressure signal are temporarily stored as sound pressure waveform data, and the A / D converted video signal is temporarily stored as image data. An existing buffer and sound pressure waveform data for the analysis time length used for sound source estimation from the buffer, and frequency analysis of the extracted sound pressure waveform data is performed between the microphones constituting the two microphone pairs. Sound source direction estimation means for obtaining a phase difference and estimating the direction of the sound source from the obtained phase difference ratio of the two pairs of microphone pairs, and the sound used for the estimation calculation from the buffer. Image data extracting means for extracting image data of a video signal photographed when a sound pressure signal corresponding to pressure waveform data is sampled, and combining the estimated sound source direction data and the extracted image data Sound source estimation image creating means for creating a sound source estimation image that is an image in which a graphic showing the estimated sound source direction is drawn; and creation of the sound source estimation image Later, characterized in that the sound pressure waveform data and image data used to create the corresponding sound source estimation image and a data erasing means for erasing from said buffer.
By using the sound source estimation image display device having such a configuration, it is possible to create and display a sound source estimation image while collecting sound information and video information. It can be performed. When data that actually identifies the sound source is required, a separate storage means such as a hard disk is provided, and the sound pressure signal and video signal from the sound source estimated at the site are collected and stored in the storage means. Then, a sound source estimation image may be created using the stored data and the sound source may be estimated again.

According to a fifth aspect of the present invention, in the sound source estimation image display device according to the fourth aspect, the measurement command means for issuing the measurement start command and the measurement end command for the sound source direction, the data storage means, the data storage means, The A / D converter further outputs sound pressure waveform data and image data obtained by A / D converting the sound pressure signal and the video signal, respectively, when a sound source direction measurement start command is issued. The data erasing means stores the data in the buffer, and when the size of the empty space in the buffer becomes equal to a size capable of storing at least the analysis time length of sound pressure waveform data and image data, the sound source estimation The sound pressure waveform data and the image data used to create the image for use are erased from the buffer, and the data storage means sends the buffer when the measurement end command is issued and the sound source direction is estimated. Wherein the storing the sound pressure waveform data and image data temporarily stored in the data storage means.
As a result, the sound pressure waveform data and the image data for constructing the sound source estimation image of the sound source estimated by the operator can be saved, so that the sound source estimation image can be easily reproduced and the sound source estimation work can be efficiently performed. Can be done well.

According to a sixth aspect of the present invention, in the sound source estimation image display device according to the fifth aspect of the present invention, a measurement time is set which is a time from when a measurement start command is issued until the sound source direction estimation ends. A time setting unit is provided, and the measurement command unit issues a measurement end command after a lapse of the measurement time from the time when the measurement start command is issued.
As a result, the operator does not need to consider the timing of the end of measurement, and can concentrate on the estimation of the sound source. Therefore, the sound source can be reliably estimated.

The invention described in claim 7 is a retrospective measurement in which the sound source direction estimation is retroactively measured by a preset retrograde time length in the sound source estimation image display device according to any one of claims 4 to 6. A retrograde measurement command means for issuing a command, and the sound source direction estimating means, when the retrograde measurement command is issued, the time from the time when the retroactive measurement command is issued by a predetermined retrograde time length Sound pressure waveform data up to a time elapsed by the analysis time length is extracted from the buffer, and a sound source direction is estimated using the extracted sound pressure waveform data.
Thereby, it is also possible to display a sound source estimation image for estimating the direction of the source of the sudden sound or intermittent sound.

According to an eighth aspect of the present invention, in the sound source estimation image display device according to any one of the fourth to seventh aspects, the microphone group is not a plane that is formed on the plane formed by the two microphone pairs. In addition, the sound source direction estimation means includes a phase difference between the microphones constituting the two pairs of microphones, and four microphones constituting the fifth microphone and the two pairs of microphones. The sound source direction is estimated by using the phase difference between the microphones constituting the four pairs of microphones constituted by each.
As a result, the horizontal angle θ and the elevation angle φ of the sound source direction can be estimated efficiently and accurately with a small number of microphones, so that a sound source estimation image with high accuracy can be created.

  The summary of the invention does not list all necessary features of the present invention, and sub-combinations of these feature groups can also be the invention.

It is a functional block diagram which shows the structure of the image display system for sound source estimation which concerns on embodiment of this invention. It is a flowchart which shows the display method of the image for sound source estimation using the image display system for sound source estimation which concerns on this Embodiment. It is a figure which shows an example of the image for sound source estimation. It is a figure for demonstrating the timing of a process of data. It is a functional block diagram which shows the structure of the image display system for sound source estimation provided with retrograde mode. It is a flowchart which shows operation | movement of the image display system for sound source estimation provided with retrograde mode. It is a figure which shows the arrangement | sequence of the microphone in the sound source search method using the conventional microphone pair.

  Hereinafter, the present invention will be described in detail through embodiments, but the following embodiments do not limit the invention according to the claims, and all combinations of features described in the embodiments are included. It is not necessarily essential for the solution of the invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a functional block diagram showing a configuration of a sound source estimation image display system.
The sound source estimation image display system includes a sound / video sampling unit 10, a data processing device 20, a calculation device 30, a display device 40, and a storage device 50 as data storage means.
The sound / image collection unit 10 includes a sound collection unit 11, a CCD camera (hereinafter referred to as a camera) 12 as a video collection unit, a microphone fixing unit 13, a camera support 14, a column 15, and a turntable 16. The base 17 is provided.
The sound collection means 11 includes a plurality of microphones M1 to M5.
The arrangement of the microphones M1 to M5 is the same as that shown in FIG. 7, and two microphone pairs (M1, M1) in which four microphones M1 to M4 are arranged at predetermined intervals on two straight lines orthogonal to each other. M5) and the microphone pair (M2, M4) are arranged so that the fifth microphone M5 is not on the plane formed by the microphones M1 to M4. Specifically, a square formed by the microphones M1 to M4 is formed. It arranges at the position of the apex of the quadrangular pyramid as the bottom. Thereby, four pairs of microphones (M5, M1) to (M5, M4) are further configured.
In this example, the shooting direction of the camera 12 is set to a direction that passes through the intersection of the two orthogonal lines and forms approximately 45 ° with the two lines. Therefore, the direction of the sound / image collection unit 10 is the direction of the white arrow D in FIG. The camera 12 collects an image corresponding to the direction of the sound / image collection unit 10.

The microphones M1 to M5 are installed on the microphone fixing unit 13, the camera 12 is installed on the camera support base 14, and the microphone fixing unit 13 and the camera support base 14 are connected by three support columns 15. That is, the sound collection means 11 and the camera 12 are integrated. The microphones M1 to M5 are disposed on the upper part of the camera 12.
The base 17 is a support member composed of three legs, and the turntable 16 is installed on the base 17. The camera support 14 is mounted on the rotation member 16r of the turntable 16.
Therefore, the sound collecting means 11 and the camera 12 can be rotated together by rotating the rotating member 16r.
The microphones M1 to M5 each measure a sound pressure level that is a magnitude of a sound pressure signal of a sound propagated from a sound source (not shown).

The data processing device 20 includes a measurement switch 21 as a measurement command means, an amplifier 22, an A / D converter 23, and a video input / output means 24.
The arithmetic unit 30 includes a buffer 31, sound pressure waveform data extraction means 32, sound source direction estimation means 33, image data extraction means 34, sound source estimation image creation means 35, data erasure means 36, and data storage means 37. With. The arithmetic device 30 is configured by software of a personal computer, for example.
The display device 40 includes a display screen 40M that displays a sound source estimation image that is an image for estimating a sound source position, which will be described later.
The storage device 50 is a non-volatile memory configured from, for example, a hard disk of a personal computer.

The measurement switch 21 outputs a measurement start signal for instructing the estimation calculation of the sound source direction to the amplifier 22 and the video input / output means 24, and outputs a measurement end signal for instructing the end of the estimation calculation of the sound source direction to the video input to the amplifier 22. It outputs to the output means 24 and the data storage means 37 mentioned later.
In this example, when the measurement switch 21 is in the OFF state, the output of the CCD camera 12 is input to the display device 40. At this time, by adding a speaker to the display device 40 and inputting the output of the microphone M5 to the speaker, the image captured by the CCD camera 12 is directly displayed on the display device 40, and the sound propagating from the imaging direction is displayed on the speaker. You may make it reproduce in.
The amplifier 22 includes a low-pass filter. When a command signal is input, the amplifier 22 removes high frequency noise components from the sound pressure signal of the sound collected by the microphones M1 to M5, and amplifies each sound pressure signal to perform A / D conversion. Output to the device 23.
The A / D converter 23 creates sound pressure waveform data obtained by A / D converting the sound pressure signal, and sends the sound pressure waveform data to the sound data storage area 31 a of the buffer 31.
When a command signal is input, the video input / output means 24 inputs a video signal continuously captured by the camera 12 and performs A / D conversion. The A / D converted image data is converted into a predetermined time T _p. It is sent to the image data storage area 31b of the buffer 31 every time (for example, T _p = 1/30 seconds).
The buffer 31 includes a rewritable memory such as a RAM, and includes a sound data storage area 31a for sequentially storing sound pressure waveform data and an image data storage area 31b for sequentially storing image data.
The image data output at each predetermined time T _p is image data that constitutes one screen displayed on the display screen 40M of the display device 40, that is, a so-called “one frame” image of a moving image.
Hereinafter, the predetermined time T _p is referred to as a screen switching time T _p .
For analysis time length T _w and the screen switching time T _p, respectively, is assumed to be preset to a sound pressure waveform data extracting means 32 and the image data extracting means 34 to be described later.
Analysis time length T _w is usually screen switching time T _p of 1 / N; but is set to (N an integer), in this example, N = 1, i.e., set to T _w = T _p.

Sound pressure waveform data extracting means 32, the analysis time length T _w content of sound pressure waveform data for a sound source direction estimation calculation extracted from the sound data storage area 31a of the buffer 31, which sound file FA _k (k = 1, 2, 3, ...) and sequentially output to the sound source direction estimating means 33.
DOA estimation unit 33 obtains the phase difference between the microphones M1~M5 from the sound picked file FA _k, calculates a horizontal angle θ and elevation φ is the sound source direction from the thus determined phase differences, the The calculation result is output to the sound source estimation image creating means 35. Details of the method for estimating the sound source direction (θ, φ) will be described later.
Image data extracting means 34 from the image data storage area 31b of the buffer 31, extracts image data between the screen switching time T _p, and outputs this as a moving image file FB _k to the sound source estimation image generating unit 35. The extracted image data is image data of a video signal photographed when a sound pressure signal corresponding to the sound pressure waveform data used for the estimation calculation is collected. That is, in this example, since the screen switching time T _p and the analysis time length T _w are the same length, the image data constituting the moving image file FB _k is the sound pressure constituting the sound file FA _k used for estimating the sound source direction. Data corresponding to the waveform data.
The sound source estimation image creation means 35 synthesizes the horizontal angle θ _k and elevation angle φ _k which are data of the sound source direction estimated using the sound file FA _k and the moving image file FB _k extracted by the image data extraction means 34. Then, a sound source estimation image G _k in which a graphic indicating the direction of the sound source is drawn in the image is created and output to the display device 40.
The data erasing means 36 is used to create the sound source estimation image G _k from the sound pressure waveform data temporarily stored in the sound data storage area 31a of the buffer 31 and the image data temporarily stored in the image data storage area 31b. The sound pressure waveform data constituting the sound file FA _k and the image data constituting the moving image file FB _k are deleted, and the sound pressure waveform data is respectively stored in the sound data storage area 31a and the image data storage area 31b. And a new empty area 31v in which image data can be stored.
The data storage unit 37 stores the sound pressure waveform data and the image data temporarily stored in the buffer 31 in the storage device 50 when the measurement end instruction is issued and the estimation of the sound source direction is ended.

Next, a method of displaying a sound source estimation image using the sound source estimation image display system of this example will be described with reference to the flowchart of FIG.
First, after connecting the sound / video sampling unit 10, the data processing device 20, the arithmetic device 30, and the display device 40, the sound / video sampling unit 10 is set at a measurement point (step S10).
Then, with the shooting direction of the camera 12 directed to the planned measurement location, the display screen 40M is checked to confirm that the camera 12 is shooting the planned measurement location, and at the same time the sound is collected by the microphones M1 to M5. In step S11, an image of the place to be measured is collected.
If the field of view is larger than the image field of view when viewing the measurement planned location from the measurement point, the rotating table 16 is moved around the center of the measurement planned location at a slow speed of, for example, about 3 ° / sec. Sound and video may be collected while rotating back and forth. A rotation range of about ± 60 ° is appropriate. Note that the sound / video sampling unit 10 may be fixed and measured, and after the measurement is completed, the rotating table 16 may be appropriately rotated to change the photographing direction to the direction in which the sound source is estimated, and then measure again.

Next, it is determined whether or not the measurement switch 21 is in an ON state (step S12). If the measurement switch 21 is in an OFF state, the process returns to step S11 to collect sound data and video data.
On the other hand, when the measurement switch 21 is in the ON state, the process proceeds to step S13, the output signals of the microphones M1 to M5 are amplified and A / D converted, and the A / D converted sound pressure waveform data is converted into the sound data of the buffer 31. In addition to storing in the storage area 31 a, the video signal of the camera 12 is A / D converted, and the A / D converted image data is stored in the image data storage area 31 b of the buffer 31.
At step S14, sequentially extracts the sound data storage area 31a previously specified number of samples of the sound pressure waveform data from the buffer 31 (analysis time length T _w content of sound pressure waveform data), which sound file FA _k (k = 1, 2, 3,...) And output to the sound source direction estimating means 33.
Next, the sound source direction is estimated using the sound file FA _k and a sound pressure level which is the magnitude of the sound pressure signal of the propagated sound is calculated (step S15). The sound pressure level is calculated using sound pressure waveform data of the microphone M5.
Extraction of sound pressure waveform data, it is necessary to perform in a state where the sound data storage area 31a in the analysis time length T _w min or more data is stored, be carried out after a predetermined time (e.g. 5 seconds) has elapsed from the measurement preferable. Alternatively, separately provided with means for counting the number of data stored in the sound data storage area 31a of the buffer 31, even if the extraction of the sound pressure waveform data at the stage of the analysis time length T _w of data has been stored Good.
The sound source direction is estimated by performing frequency analysis on the sound pressure waveform data by FFT, obtaining the respective phase differences between the microphones M1 to M5 for each frequency, and determining the direction of the sound source for each frequency from the obtained phase difference. presume. In this example, the horizontal angle θ and the elevation angle φ are obtained using the arrival time difference D _ij which is a physical quantity proportional to the phase difference instead of the phase difference.
The calculation method of the horizontal angle θ and the elevation angle φ in step S15 will be described later.

Next, image data for the screen switching time T _p is extracted from the image data storage area 31b of the buffer 31, and this is used as a moving image file FB _k (k = 1, 2, 3,...) As a sound source estimation image. The sound source direction data (θ, φ) estimated by the sound source estimation image creation means 35 using the sound file FA _k and the moving image file FB _k outputted from the image data extraction means 34 are output to the creation means 35. And a sound source estimation image G _k in which a graphic indicating the direction of the sound source, such as a mesh pattern circle, is drawn in the synthesized image (step S16), and this is displayed on the display device. 40 is displayed on the display screen 40M (step S17).
Figure 3 is a diagram showing an example of displaying the image G _k sound source estimation on the display screen 40M, the horizontal axis represents the horizontal angle theta, the vertical axis represents the elevation angle phi, the circle of the mesh pattern shown by reference numeral S of FIG size Represents the sound pressure level. Therefore, by repeating the steps S12 to S17, the sound source estimation image _Gk is displayed as a moving image on the display screen 40M. Therefore, if the worker has some experience, this moving image is viewed. Thus, the sound source can be specified.
It is also possible to display the estimated sound source direction for each preset frequency band. In this case, what is necessary is just to set the color of the mesh-shaped circle mark S for every frequency band.
Next, the measurement switch 21 determines whether the ON state (step S18), and if it is ON, erase the sound pressure waveform data and image data used to create the image G _k for sound estimation Then, after the empty area 31v is generated in the buffer 31 (step S19), the process returns to step S13, and the process of storing the sound pressure waveform data and the image data in the buffer 31 is continued.
If there is an empty area of a predetermined size in the buffer 31, the process of deleting the sound pressure waveform data and the image data is omitted, and the process immediately returns to step S13. Thereby, the arithmetic processing can be speeded up.
The size of the predetermined empty space may be a size that can store at least the analysis time length T _w content of sound pressure waveform data and image data. It should be noted that the maximum value of the size of the predetermined free area is preferably 1/5 or less of the storage capacity of the buffer 31.
The magnitude of the predetermined free space, if large enough to store the analysis time length T _w content of sound pressure waveform data and image data, sound pressure waveform data and the data of the image data temporarily stored in the buffer 31 Since the number can be maximized, the number of data stored in the storage device 50 is also maximized.

In step S18, when the measurement switch 21 is changed to the OFF state, the sound pressure waveform data and the image data temporarily stored in the buffer 31 are stored in the storage device 50 (step S20), and then measurement and sound source estimation are performed. to end the creation of the use image G _k.
In this example, the operator performs the process of turning off the measurement switch 21 while watching the moving image of the sound source estimation image G _k displayed on the display screen 40M. Thus, if the operator turns off the measurement switch 21 after an appropriate time after the sound source confirmation, the data after confirming the sound source on the display screen 40M can be saved.
Specifically, the magnitude of the predetermined free space saving size precepts can store and analysis time length T _w content of sound pressure waveform data and image data, and a sound pressure waveform data and image data in the buffer 31 one o'clock If the time when the buffer 31 runs out of free space is 5 minutes when the operator visually recognizes the sound source (actually, the sound source estimation image G _k having many meshed circles S and concentrated) Then, the person turns off the measurement switch 21 after a predetermined time (for example, 2 minutes). As a result, the sound pressure waveform data and the image data for a total of 5 minutes for 3 minutes before the operator visually recognizes the sound source and for 2 minutes after the operator visually recognizes the sound can be stored in the storage device 50. Therefore, since the sound source estimation image G _k visually recognized by the operator can be reproduced using the sound pressure waveform data and the image data stored in the storage device 50, the sound source can be reliably estimated.

すなわち、互いに直交する２直線上にそれぞれ所定の間隔で配置された２組のマイクロフォン対（Ｍ１，Ｍ３）及びマイクロフォン対（Ｍ２，Ｍ４）を構成するマイクロフォンＭ１，Ｍ３に入力する音圧信号の到達時間差Ｄ₁₃と、前記マイクロフォン対（Ｍ２，Ｍ４）を構成するマイクロフォンＭ２，Ｍ４に入力する音圧信号の到達時間差Ｄ₂₄との比から、計測点と音源位置との水平角θを推定し、前記到達時間差Ｄ₁₃，Ｄ₂₄と、前記第５のマイクロフォンＭ５と他のマイクロフォンＭ１〜Ｍ４との到達時間差Ｄ_5j（ｊ＝１〜４）とから計測点と音源位置との成す仰角φを推定する。
なお、前記到達時間差Ｄ_ijは、２つのマイクロフォン対（Ｍ_ｉ，Ｍ_ｊ）に入力される信号のクロススペクトルＰ_ij（ｆ）を求め、更に、対象とする前記周波数ｆの位相角情報Ψ（ｒａｄ）を用いて、以下の式（３）を用いて算出される。

The calculation method of the horizontal angle θ and the elevation angle φ in step S15 is as follows.
When the arrival time difference between the microphone Mi and the microphone Mj of each microphone pair (Mi, Mj) is D _ij , the horizontal angle θ and the elevation angle φ, which are the sound incident directions, are expressed by the following equations (1) and (2). Therefore, by analyzing the frequency of the output signals of the microphones M1 to M5 using an FFT and calculating the arrival time difference D _ij between the microphones M _i and M _j at the target frequency f, the horizontal angle is calculated. θ and elevation angle φ can be obtained.

That is, arrival of sound pressure signals input to the microphones M1 and M3 constituting the two microphone pairs (M1, M3) and the microphone pairs (M2, M4) arranged at predetermined intervals on two orthogonal lines. The horizontal angle θ between the measurement point and the sound source position is estimated from the ratio between the time difference D ₁₃ and the arrival time difference D ₂₄ of the sound pressure signals input to the microphones M2 and M4 constituting the microphone pair (M2, M4). The elevation angle φ between the measurement point and the sound source position is estimated from the arrival time differences D ₁₃ and D ₂₄ and the arrival time differences D _5j (j = 1 to 4) between the fifth microphone M5 and the other microphones M1 to M4. To do.
The arrival time difference D _ij is obtained as a cross spectrum P _ij (f) of signals input to the two microphone pairs (M _i , M _j ), and further, the phase angle information ψ ( rad) and is calculated using the following equation (3).

Thus, in the present embodiment, the sound pressure obtained by A / D converting the sound pressure signal and the video signal collected using the sound / video collection unit 10 in which the plurality of microphones M1 to M5 and the camera 12 are integrated. The waveform data and the image data are temporarily stored in the sound data storage area 31a and the image data storage area 31b of the buffer 31, respectively, and the sound pressure waveform data (analysis time) of the number of samples specified in advance from the sound data storage area 31a. the length T _w content of sound pressure waveform data) sequentially taken out, after the sound source direction estimation is sent to the sound source direction estimating unit 33 so as sound files FA _k, from the image data storage area 31b, the sound source direction estimation sound file FA _k image data to sound pressure waveform data corresponding used for extraction as a moving image file FB _k, the sound source direction data (theta, phi) and video Airu by synthesizing the FB _k, create a sound source estimation image G _k a shape is drawn indicating the direction of the sound source, such as a circle S of netted displayed on the display screen 40M of the display device 40 in the image Thereafter, by repeating the operation of deleting the sound pressure waveform data and the image data used to create the sound source estimation image G _k and generating the empty area 31v in the buffer 31, the display on the display device 40 is repeated. Since the moving image of the sound source estimation image G _k can be displayed on the screen 40M, it is possible to provide the operator with a sound source estimation moving image while collecting sound information and video information on site. it can.
Further, after completion of the measurement, it is possible because so as to save the sound pressure waveform data and image data temporarily stored in the buffer 31 in the storage unit 50, to reproduce the image G _k sound source estimation. Therefore, the sound source can be estimated efficiently.

Further, by adopting such a configuration, the arithmetic device 30 performs a detailed series of processing such as sound pressure signal sampling and sound source direction estimation calculation, video signal sampling and moving image file extraction, and sound source estimation image creation. It can be separated and executed asynchronously. Therefore, it is possible to increase the processing speed and efficiency compared to a method of creating a sound source estimation image after storing data in the hard disk. Instead, the order in which the sound source estimation images are displayed is controlled so as to keep it.
Specifically, as shown in FIG. 4, the screen G (k) on which the sound source estimation image G _k displayed kth on the time axis is displayed on the display screen 40M is the (k−1) th screen G. Although it is displayed after the display of (k-1), it takes a long time to create an image when there are many figures indicating the direction of the sound source in which the sound source estimation image G _k is depicted. Therefore, in such a case, the display time of the screen G (k−1) is extended to the end of the creation of the sound source estimation image G _k and displayed. In addition, the size of the frame surrounded by the broken line in the figure indicates 1/30 second, which is the standard display time.
On the other hand, if there are few figures indicating the direction of the sound source in which the (k + 1) -th displayed sound source estimation image G _{k + 1} is depicted, and therefore the creation time of the image is short, the sound source estimation screen G displayed time (k) of the image G _k shortened to match the creation end time source estimation image G _{k + 1,} the sound source estimation image G _{k + 1} that is created as soon as the screen G (k + 1) It is displayed on the display screen 40M. As a result, the display time of the screen G (k) (k = 1, 2,...) Slightly increases or decreases from 1/30 second. However, in reality, the sound source estimation image G _k and the sound source estimation image G _{k + 1} that are adjacent to each other do not differ greatly in the depicted figure. There is no problem above.

In the above-described embodiment, when the buffer 31 has a vacant area of a predetermined size, the erasing process of the sound pressure waveform data and the image data is omitted, but it is used to create the sound source estimation image G _k. The obtained sound pressure waveform data and image data may be deleted after the sound source estimation image G _k is displayed, or may be deleted immediately after the generation of the sound source direction estimation image G _k . Alternatively, the sound pressure waveform data and image data for a plurality of times after the sound source estimation image G _k is displayed a plurality of times may be deleted, or the sound pressure waveform data used for estimating the sound source direction may be used as the sound source. It may be deleted after the direction estimation calculation, and the image data may be deleted after the sound source estimation image G _k is created. However, in this case, the storage capacity of the buffer 31 may be small, but data sufficient to reproduce the sound source estimation image G _k is not stored in the buffer 31, so it is necessary to perform sound source estimation again. However, since the operator has already visually recognized the sound source, the sound and video can be collected in a narrow range centered on the direction of the sound source visually recognized by the operator. It can be carried out.

In the above example, create an image G _k for the sound source estimated by sequentially extracts sound pressure waveform data for a specified number of samples used from the start of measurement to the sound source estimation calculation (analysis time length T _w content of sound pressure waveform data) However, it is also possible to provide a pre-trigger function in the image display system for sound source estimation according to the present invention.
Specifically, as shown in FIG. 5, instead of the measurement switch 21, a measurement switch 21Z having a switching function including a mode switch 21a for switching the measurement mode between a normal mode and a backward mode and a measurement switch 21b is provided. At the same time, the arithmetic unit 30 has a stored data number detecting means 38 for counting the number of samples of the sound pressure waveform data and image data stored in the buffer 31, a buffer for the sound pressure waveform data extracting means 32 and the image data extracting means 34. Extraction method instruction means 39 for instructing a method for extracting data from 31 is provided.
Normal mode creates a sequential extraction to the image sound source estimation data of an analysis time length T _w min from the command signal command signal a time measurement start signal has been input is for starting the measurement display.
Backward mode, measurement start signal data from only back time of analysis time length T _w min preset backward time than the time of the input are sequentially extracted in creating an image for a sound source estimation display.

FIG. 6 is a flowchart showing a sound source estimation image display method when the pre-trigger function is provided in the sound source estimation image display system.
First, after connecting the sound / video sampling unit 10, the data processing device 20, the arithmetic unit 30, and the display device 40, the sound / video sampling unit 10 is set at a measurement point (step S30), and then a measurement switch with a switching function is provided. The mode switch 21a of 21Z is set to either the normal mode or the backward mode (step S31).
Next, sound is collected by the microphones M1 to M5, and at the same time, an image of the place to be measured is collected by the camera 12 (step S32).
The measurement switch 21b may be turned on simultaneously with the collection of sound and video as in the above embodiment, or the operator may determine the measurement start timing and turn on the measurement switch 21b.
Next, it is determined whether or not the measurement switch 21b is in the ON state (step S33). If the measurement switch 21b is in the ON state, the process proceeds to step S34, and the output signals of the microphones M1 to M5 are amplified and A / D converted to perform this A. The / D converted sound pressure waveform data is stored in the sound data storage area 31 a of the buffer 31, the video signal of the camera 12 is A / D converted, and this A / D converted image data is stored in the image data storage area 31 b of the buffer 31. Save to.
It is determined after measurement starts, at the stage of the analysis time length T _w of data has been stored, the measurement mode or the normal mode or backward mode (step S35).

When the measurement mode is the normal mode, the processes of steps S36 to S40, which are the same contents as steps S14 to S18 shown in the flowchart of FIG. 2, are performed. In step S40, the measurement switch 21b is in the ON state. Is determined, the sound pressure waveform data and the image data used to create the sound source estimation image G _k are erased to generate a free space 31v in the buffer 31 (step S41), and then the step Returning to S34, the acquisition of the sound pressure waveform data and the image data is continued. Again, if there is a predetermined size of free space in the buffer 31, the process of erasing the sound pressure waveform data and the image data is omitted, and the process immediately returns to step S34.
If the measurement switch 21b is changed to the OFF state in step S40, the sound pressure waveform data and the image data temporarily stored in the buffer 31 are stored in the storage device 50 (step S42), and then the measurement and sound source are measured. The creation of the estimation image G _k is finished.

On the other hand, when the measurement mode is the retrograde mode, the process proceeds to step S51, and it is determined whether or not the number of data for the retrograde is stored in the buffer 31. This determination is made based on the number of stored data detected by the stored data number detecting means 38.
If the number of backward data is not stored in the buffer 31, the process returns to step S34 to continue storing the sound pressure waveform data and the image data.
In step S51, if it is determined that the number of backward data is stored in the buffer 31, the process proceeds to step S52, where the extraction method instruction unit 39 includes the sound pressure waveform data extraction unit 32 and the image data extraction unit 34. And an extraction method instruction signal for instructing a method for extracting data from the buffer 31, and a time that is set back from the sound data storage area 31a of the buffer 31 by a preset backward time from the time when the measurement command signal is issued. sequentially extracted in advance the specified number of samples of sound pressure waveform data (analysis time length T _w worth of sound pressure waveform data) from the sound source it as a sound file _{FA m (m = 1,2,3, ......} ) It outputs to the direction estimation means 33 (step S53).
The sound source direction estimation means 33 performs a sound source direction estimation calculation using the sound file FA _m and calculates a sound pressure level that is the magnitude of the sound pressure signal of the transmitted sound (step S54).

Next, image data for each screen switching time T _p is extracted from the image data storage area 31b of the buffer 31, and this is used as a moving image file FB _m (k = 1, 2, 3,...) To create a sound source estimation image. The sound source direction data (θ, φ) estimated by the sound source estimation image creating means 35 using the sound file FA _m and the moving image file FB _m extracted by the image data extracting means 34 are output to the means 35. A sound source estimation image G _m in which a graphic indicating the direction of the sound source, such as a mesh-shaped circle S, is drawn in the image that is the combined image is created (step S55), and this is displayed on the display device 40. Is displayed on the display screen 40M (step S56).
The image data constituting the moving image file FB _m is image data from a time that is back by a preset retroactive time from the time when the measurement command signal is issued.
After the display of the image, the process proceeds to step S40 to determine whether the measurement switch 21b is in an ON state or an OFF state. If the measurement switch 21b is determined to be ON state, generating a free area 31v to erase the sound pressure waveform data and image data used to create the sound source estimation image G _m in the buffer 31 (step After step S41), the process returns to step S34 to continue capturing the sound pressure waveform data and the image data. Again, if there is a predetermined size of free space in the buffer 31, the process of erasing the sound pressure waveform data and the image data is omitted, and the process immediately returns to step S34.
If the measurement switch 21b is changed to the OFF state in step S40, the sound pressure waveform data and the image data temporarily stored in the buffer 31 are stored in the storage device 50 (step S42), and then the measurement and sound source are measured. The creation of the estimation image G _k is finished.
As a result, even if sudden sounds or intermittent sounds occur, the sound source estimation image for estimating the direction of the source of these sounds can be displayed reliably, thus eliminating measurement failures. Can do.

In the above example, the process of the measurement switch 21 is turned OFF by the operator was performed, to set the measurement time T _m is the time from the measurement start command is issued until the end of the sound source direction estimation measurement provided with a time setting unit, the measurement switch 21, it may be added a function of issuing a measurement end command after the lapse of the measuring time T _m from the time emitted the measurement start command.
For example, if the measurement time _Tm is 5 minutes, the measurement switch 21 is automatically turned off 5 minutes after the operator turns on the measurement switch 21, and a measurement end command is issued. As a result, the operator does not need to consider the timing of the end of measurement, and can concentrate on the estimation of the sound source. Therefore, the sound source can be reliably estimated.
In the above example, the horizontal angle θ and the elevation angle φ formed by the measurement point and the sound source position are estimated using the five microphones M1 to M5. However, when the sound source position is sufficient, the horizontal angle θ is sufficient. The microphone M5 may be omitted, and only two pairs of microphones (M1, M3) and (M2, M4) arranged at predetermined intervals on two straight lines that intersect with each other may be used.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the embodiment. It is apparent from the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  As described above, according to the present invention, while collecting sound information and video information, a sound source estimation image in which the estimated sound source direction and sound pressure level are graphically displayed in the video is created. Therefore, the sound source can be estimated efficiently on site.

10 sound / video sampling unit, 11 sound sampling means, M1-M5 microphone,
12 CCD camera, 13 microphone fixing part, 14 camera support base, 15 struts, 16 turntable, 17 base,
20 data processing devices, 21 measurement switches, 22 amplifiers, 23 A / D converters,
24 video input / output means,
30 arithmetic unit, 31 buffer, 31a sound data storage area,
31b Image data storage area, 32 sound pressure waveform data extraction means,
33 sound source direction estimation means, 34 image data extraction means, 35 sound source estimation image creation means, 36 data erasure means, 37 data storage means,
40 display devices, 40M display screen, 50 storage devices.

Claims (8)

A step (a) of simultaneously collecting a sound pressure signal, which is sound information, and a video signal, which is video information, using a sound / video unit in which a plurality of microphones and photographing means are integrated;
A step (b) of A / D converting the collected sound pressure signal and video signal and temporarily storing them in a buffer as sound pressure waveform data and image data, respectively;
Sound pressure waveform data for an analysis time length used for sound source estimation is extracted from the buffer, and using the extracted sound pressure waveform data, a phase difference between sound pressure signals collected by the plurality of microphones is calculated. Estimating the sound source direction (c);
(D) extracting from the buffer image data for the analysis time length used for estimating the sound source direction in the step (c);
The sound source direction data estimated in the step (c) and the image data extracted in the step (d) are combined to create a sound source estimation image in which a graphic showing the estimated sound source direction is drawn. Step (e);
Displaying the sound source estimation image on a display means (f);
A step (g) of erasing the sound pressure waveform data and the image data used for creating the sound source estimation image from the buffer after creating the sound source estimation image; Display method. In the step (g), when the size of the free area of the buffer becomes equal to a size that can store at least the sound pressure waveform data and the image data for the analysis time length, the sound source estimation image is created. While deleting the used sound pressure waveform data and image data from the buffer,
Between the step (a) and the step (b), a step (h) of issuing a measurement start command for starting estimation of the sound source direction is provided,
A step (i) of determining whether or not to end the estimation of the sound source direction is provided after the step (g);
In the step (b), the collected sound pressure signal and the video signal are A / D converted after the measurement start command is issued, and temporarily stored in the buffer as sound pressure waveform data and image data, respectively.
If the estimation of the sound source direction is not completed in step (i), the process returns to step (h). If the estimation of the sound source direction is completed, the sound pressure waveform data and image data temporarily stored in the buffer are returned. The sound source estimation image display method according to claim 1, wherein: is stored in a storage unit different from the buffer. Between the step (b) and the step (c), there is provided a step (j) for issuing a backward measurement command for measuring the sound source direction retroactively by a predetermined backward time length;
In the step (c), the sound pressure waveform data from the time when the backward measurement command is issued to the time after the predetermined backward time length to the time after the analysis time length is extracted from the buffer. The sound source estimation image display method according to claim 1, wherein the sound source direction is estimated using the extracted sound pressure waveform data. Providing a microphone group having two pairs of microphones arranged at predetermined intervals on two intersecting straight lines, a photographing means for photographing a sound source direction image and a display means, and propagating from the sound source collected by the microphone group A sound source estimation image, which is an image in which a graphic showing the direction of the sound source is drawn, is displayed on the display means from the sound pressure signal of the sound to be played and the video signal in the direction of the sound source taken simultaneously with the collection of the sound pressure signal. A sound source estimation image display device comprising:
An A / D converter that converts the sound pressure signal collected by each microphone and the video signal photographed by the photographing means into digital signals,
A buffer for temporarily storing the A / D converted sound pressure signal as sound pressure waveform data and temporarily storing the A / D converted video signal as image data;
The sound pressure waveform data corresponding to the analysis time length used for estimation of the sound source is extracted from the buffer, and the phase difference between the microphones constituting the two microphone pairs is analyzed by frequency analysis of the extracted sound pressure waveform data. Sound source direction estimation means for performing an estimation operation for estimating the sound source direction from the ratio of the phase difference between the two pairs of microphones thus determined;
Image data extraction means for extracting image data of a video signal captured when the sound pressure signal corresponding to the sound pressure waveform data used for the estimation calculation is sampled from the buffer;
Sound source estimation image creating means for synthesizing the estimated sound source direction data and the extracted image data to create a sound source estimation image which is an image in which a figure showing the estimated sound source direction is drawn. When,
A sound source estimation image comprising: a data erasure unit that erases the sound pressure waveform data and the image data used to create the sound source estimation image from the buffer after the sound source estimation image is created. Display device. A measurement command means for issuing a measurement start command and a measurement end command in the direction of the sound source;
Data storage means;
Data storage means,
The A / D converter stores, in the buffer, sound pressure waveform data and image data obtained by A / D converting the sound pressure signal and the video signal, respectively, when a sound source direction measurement start command is issued. And
The data erasing means is used to create a sound source estimation image when the size of the free space in the buffer is at least equal to the size capable of storing the sound pressure waveform data and the image data for the analysis time length. Delete the sound pressure waveform data and the image data from the buffer,
The data storage means stores the sound pressure waveform data and image data temporarily stored in the buffer when the measurement end command is issued and the estimation of the sound source direction is completed, in the data storage means. The sound source estimation image display device according to claim 4. While providing a measurement time setting means for setting a measurement time, which is the time from when the measurement start command is issued until the estimation of the sound source direction is completed,
6. The sound source estimation image display device according to claim 5, wherein the measurement command means issues a measurement end command after the measurement time has elapsed from the time when the measurement start command is issued. While providing a retrograde measurement command means for issuing a retrograde measurement command for measuring the sound source direction by going back a preset retrograde time length,
The sound source direction estimating means, when the backward measurement command is issued, between the time when the backward measurement command is issued and the time when the analysis time length has elapsed from the time that is a predetermined backward time length. The sound pressure estimation image according to any one of claims 4 to 6, wherein the sound pressure waveform data is extracted from the buffer and the sound source direction is estimated using the extracted sound pressure waveform data. Display device.   A fifth microphone that is not on the plane formed by the two sets of microphone pairs is added to the microphone group, and the sound source direction estimation means includes a phase difference between the microphones constituting the two sets of microphone pairs, and A sound source direction is estimated using a fifth microphone and a phase difference between the microphones constituting the four pairs of microphones constituted by each of the four microphones constituting the two pairs of microphones. The sound source estimation image display device according to any one of claims 4 to 7.

Images