DE102008026657A1 - Method for imaged representation of three dimensional acoustic objects as measuring object, involves bringing images in relation to acoustic reference image of measuring object immediately or at time point - Google Patents

Abstract

The method involves assigning an optical camera array to a microphone array, where the microphone array consists of spiral arms (120) and microphones (121). Images are recorded by the camera array in temporally synchronized manner parallel to a data flow of microphone signals of the microphone array, where the images comprise spatial information of a measuring object. The images are brought in relation to an acoustic reference image of the measuring object immediately or at a time point, where the camera array has video cameras (140) continuously recording the information of the object. Independent claims are also included for the following: (1) a device for imaged representation of three dimensional acoustic objects (2) a computer program for performing a method for imaged representation of three dimensional acoustic objects as a measuring object (3) a computer program product with a program code for performing a method for imaged representation of three dimensional acoustic objects as a measuring object.

Description

The The invention relates to a method and a device for imaging Representation of acoustic objects according to the generic terms of independent claims.

object The invention also includes a computer program and a computer program product with a program code stored on a machine-readable carrier is stored for carrying out the method.

State of the art

From the DE 103 04 215 A1 For example, a method and an apparatus for the imaging of acoustic objects as well as a corresponding computer program product and a corresponding computer-readable storage medium have become known which comprise a microphone array with an integrated video camera. This method and device is used to detect and display acoustic emissions. The device, also referred to as an acoustic camera, makes it possible to take pictures of acoustic objects, that is to say objects that emit sound, in which case an acoustic map and an optical image are superimposed by object distance and camera opening angle defining an optical image field onto which the acoustic card is "calculated". Calculation-relevant parameters of the microphones and the camera of an array are stored unmistakably in a parameter file of the array. This makes it possible to recapitulate details of the acoustic measurement even after completion of the measurement at a later time.

With Help of this device and this method becomes automatic Each measurement is documented by records of camera shots unsolvable with the data sets of the microphone time function, with time synchronization signals, with scene information and parameter files from microphone array and data recorder merged into a data file be stored. For every point in the acoustic picture can thus time function, frequency function, sound pressure, Coordinates, sound or correlation with a known time function be retrieved and processed by means of a computer. The processing may also be available at a later date after completing the Measurement made possible and thus a convenient treatment the data. This method and the device allow however, no spatial assignment of the sound sources to the captured by the video camera image.

Of the Invention is therefore the object of a method and a Device for the imaging of acoustic objects further develop the generic type so that also spatial information of the acoustic object detected and can be processed.

Disclosure of the invention

Advantages of the invention

These The object is achieved by a method and a device having the features the independent claims 1 and 10 solved. The basic idea of the invention is to provide an optical camera array Assign microphone array and simultaneously to the data stream of the microphone signals of the microphone array spatial information of the DUT to record images in temporal synchronization using the camera array and these images in relation to the reference acoustic image of the measurement object bring to. Through a camera array, spatial Information of the DUT are recorded. This is a precise three-dimensional sound source localization by possible that the distance of the potential sound sources to the microphone can be determined without this distance by hand to have to measure.

The Camera array includes several offset cameras on the to be described in more detail below a spatial source mapping enable.

By those listed in the dependent claims Measures are each advantageous embodiments and Developments specified in the independent claim 1 Method and that specified in the independent claim 10 Device possible.

So provides an advantageous embodiment that the camera array includes at least two, preferably three video cameras, the time continuously synchronizes spatial information of the DUT during the acoustic recording via the microphone array record. These video cameras each provide three camera images, their shifting, overlaying and determination of the resulting match of several, preferably three camera images, a calculation of variable distance between the camera and thus the microphone array and any pixel allows. This can a spatially staggered source arrangement in a source mapping being represented.

A Distance determination is also possible with the aid of epipolar geometry.

Preferably, the optical axes of the video cameras are arranged parallel to each other, although it should be emphasized that a not parallel arrangement is purely conceivable in principle.

Of the Data stream of the microphone signals of the microphone array and the data stream of the optical camera array are advantageously used to create a acoustic three-dimensional image of the measurement object to each other based.

The Distance determination can be done point by point take place on the surface of the test object. hereby can the mentioned spatially staggered source arrangement be presented in a source mapping. By variation of the spatial distance between object to be measured and microphone array and a related scan plane of the microphone array can be an acoustic Surface of the DUT and determined in a display unit being represented. Acoustic surface means here for example, the representation of the sound level on the surface of the measurement object and thus the spatial representation of Areas which have, for example, a very intense sound radiation. This illustration is the acoustic three-dimensional image.

A very advantageous embodiment of the method allows a coherence or incoherence analysis. For this be done in real time by means of coherence or incoherence filtering the data stream, the microphone signals of the microphone array for amplification or used for damping sound sources.

The Coherence or incoherence filtering occurs the base of the frequency domain, preferably by means of Fourier transformation, transformed temporal microphone signals.

The inventive device is characterized from that the camera array is assigned to the microphone array. Especially the camera array is arranged in the center of the microphone array. The Microphone array can have any configuration. For example, the microphones can be in a linear Array can be arranged, with the camera array then in the center this linear array is located. Also conceivable is a stochastic one Arrangement of the microphones, in which case the camera array essentially arranged so that it is possible in every direction of one same number of microphones is surrounded. Also an arrangement of microphones on concentric rings is conceivable.

A preferred embodiment provides, the microphones spiral to arrange and position the camera array in the center.

The optical axes of the cameras, for example the three video cameras are preferably arranged parallel to each other.

Short description of the drawing

An embodiment of a device according to the invention is in the 1 and will be explained in more detail in the following description.

embodiments the invention

An inventive device 100 has a basic body 110 on, on the spiral arms 120 are attached. These spiral arms 120 each have a plurality of microphones 121 on that along the spiral arms 120 are arranged. For the sake of clarity, in 1 these microphones only with a spiral arm 120 characterized. Each of the spiral arms has corresponding microphones 121 on. At the center of these spiral arms are three video cameras 140 arranged, whose optical axes are parallel to each other.

For the imaging of acoustic objects, this device is directed to the measurement object (not shown) and simultaneously and synchronized in time, both the sound emitted by the measurement object and the measurement object are detected with the aid of the video cameras 140 recorded.

The data streams of the microphones 121 as well as the video cameras 140 are processed in a computing unit (not shown), a computer. The basic algorithm of the microphone array technology is the so-called beamforming or delay-and-sum method. In this case, the signals of a number of sensors without directional information are linked in such a way that the signal components arriving from a certain direction or emitted by a specific location are amplified and all other signal components are attenuated. This happens, for example, in the case of microphone arrays in that the signals of the individual microphones 121 delayed according to the geometric arrangement of the sensors and potential source locations and the resulting runtime and then superimposed. If the assumed source location coincides with an actual source location, the signals add in phase. In the other case, phase cancellation occurs. This process is repeated for all possible sources. The energy of the signal resulting from superimposition is shown. This provides a source mapping in which the locations are highlighted in color with great agreement with the real source.

In the from the DE 103 04 215 A1 As a result of these methods, this source mapping is superimposed on a video image in order to achieve an easy allocation to the source objects. The "sharpness" of the source mapping corresponds to the spatial Chen resolution of the microphone array and depends on a variety of different factors. The geometric arrangement of sensors and sources is of crucial importance. Aperture is the maximum distance between two sensors. The larger the aperture, the greater the spatial resolution at a constant frequency of the sound event. In a planar sensor arrangement, the aperture is perpendicular to the sensor plane equal to zero. Accordingly, the spatial resolution in this direction, ie the depth of field, is extremely low. With a three-dimensional sensor arrangement, for. B. on a ball, the depth of field can only be improved slightly. As a result, it is not possible to accurately determine the distance of a sound source to the array using the beamforming algorithm. However, the distance is crucial for accurate localization in the other spatial directions. If too large a distance is used for the calculation, the sources are recognized in the wrong place and with an incorrect level. If the distance is too small, the source will not be recognized at all.

As a general rule can be said that the beamforming algorithm has a low Dynamic range with respect to the representation of the source mapping has. In addition, the algorithm generates for each real existing source so-called ghost sources whose levels depending on the sensor arrangement a maximum of 14 dB below the level of the real one Source lie. Other real sources that are more than 14 dB quieter as the first source, are obscured by their ghost sources. With a wrong distance assumption, this limited dynamic range decreases further.

Around It is the distance of the microphones to the target to determine required in the known from the prior art method the To measure distance, for example, by hand and then as a single value enter into the evaluation software. The software then calculates a plane perpendicular to the array on the using the beamforming algorithm looking for sources. The result is a source mapping for this distance. For three-dimensional source arrangements This does not make it possible to source in a source map to represent correctly. For a correct representation is it requires a three dimensional instead of a planar scan plane Scan surface to use on which all potential Source locations lie. In principle, it is possible, all distances between the sensors and the potential sound sources to measure manually. However, this is not in industrial use practical. The integration of CAD data would also be purely in principle possible, but requires an accurate Alignment of the array to that described by the CAD data DUT. Again, this exact alignment is not always possible. Such an application would be complete only on metrics limited to existing CAD data.

• a) Durch ein manuelles Ändern des angenommenen Abstands mittels der Software kann nacheinander die Position der im Videobild aufgenommenen Objekte bestimmt werden. Synchron zur Änderung des Abstands wird auch die Scanebene für das Beamforming verschoben. Sobald ein Objekt korrekt fokussiert wurde, stimmen auch die Position und der Pegel der zugehörigen Schallquelle. Bei dieser Ausgestaltung des Verfahrens liegen alle Scanpunkte auf einer Ebene. In einer Quellkartierung können daher auch nur Quellen, die auf dieser Ebene liegen, korrekt dargestellt werden. Vorteil dieser Ausführungsform des Verfahrens ist es, dass ein manuelles Vermessen der Messobjekte entfällt. Da die Videobilder zusammen mit den Mikrofondaten abgespeichert werden, kann die Bestimmung der korrekten Entfernung auch nach Beendigung einer Messkampagne und Demontage der Messanordnung erfolgen. Das Verfahren ist darüber hinaus robust und einfach in der Anwendung. Eine softwaremäßig realisierte verschiebbare Bild-Lupe unterstützt den Anwender bei der Bestimmung der höchsten Bildschärfe.
• b) Das vorstehend beschriebene Verfahren zur Bestimmung eines manuellen Fokus kann angewendet werden, um nacheinander den Abstand zu allen interessierenden Objekten zu bestimmen. Durch Interaktion mit der Software werden diese Punkte gespeichert. Die Software erzeugt eine Oberfläche, auf der alle gespeicherten Punkte liegen, und interpoliert die Flächen dazwischen. Auf diese Weise entsteht sukzessive eine dreidimensionale Fläche, die dann beim Beamforming als Scanfläche verwendet wird. Diese Fläche kann beispielsweise dem Schallpegel zugeordnet werden, wodurch eine akustische Oberfläche des Messobjekts darstellbar ist. Hierdurch können alle Quellen in einer Quellkartierung korrekt dargestellt werden.
• c) Gemäß einer weiteren Ausgestaltung des Verfahrens wird eine Tiefenschätzung mit Hilfe der Videokameras 140 vorgenommen. Rein prinzipiell basiert dieses Verfahren auf der Auswertung von Punktkorrespondenzen in verschiedenen Videobildern. Eine andere Variante besteht darin, aus der geometrischen Anordnung der sich auf oben beschriebene Weise ergebenden entzerrten Videobildern eine diskrete Reihe möglicher Abstände zu berechnen. Jedes Bild wird dann in kleine überlappende Teilbereiche zerlegt. Für jeden Teilbereich wird die Ähnlichkeit der entsprechenden Teilbereiche der einzelnen Videobilder bestimmt und abgespeichert. Die Verfahren zur Bestimmung von Ähnlichkeitsmaßen sind aus dem Bereich der sogenannten Computer Vision an sich bekannt, z. B. die Bestimmung der Summe der paarweisen quadratischen Differenzen zwischen den Bildern der einzelnen Videokameras oder die Ermittlung des normierten Kreuzkorrelationskoeffizienten. Dieser Vorgang wird für alle in Frage kommenden Abstände wiederholt. Anschließend wird für jeden Teilbereich der Abstand bestimmt, für den die Übereinstimmung maximal ist. Dieser Ab stand ist der Abstand des abgebildeten Objektes zu den Videokameras 140 und damit zum Mikrofonarray 120, 121. Das Verfahren setzt zusätzliche Algorithmen zur Detektion von Bildpunkten mit hohem Kontrast voraus, da nur hier eine eindeutige Unterscheidung zwischen hoher und niedriger Übereinstimmung der Teilbereiche möglich ist. Dies kann beispielsweise mit Hilfe eines Algorithmus zur Kantendetektion gelöst werden. Dabei wird für jeden Teilbereich die quadratische Summe des Helligkeitsunterschieds zwischen den Bildpunkten und dem mittleren Helligkeitswert bestimmt – diese entspricht der Varianz des Teilbereichs. Über einen Schwellwert werden Bildpunkte mit hohem Kontrast bestimmt. Aus der Umgebung jedes dieser Bildpunkte wird ein Teilbereich für die anschließende Entfernungsbestimmung gebildet. Objekte mit deutlich ausgeprägter, sich wiederholender Struktur führen zu Fehldetektionen, da bei der Überlagerung für mehrere Abstände gleich hohe Werte der Übereinstimmung erreicht werden (Moire-Effekt). Dieses Problem wird dadurch gelöst, dass Bildpunkte mit wiederholten Maximalwerten in der Übereinstimmung als ungültig markiert werden. Abschließend wird aus allen gültigen Bildpunkten mittels Interpolation eine dreidimensionale Fläche bestimmt, die als Scanfläche für das Beamforming verwendet wird. Das Verfahren ist vollständig automatisierbar. Eine Interaktion des Benutzers ist nicht erforderlich. Bei diesem Verfahren ist auch eine korrekte Erfassung von sich bewegenden Objekten auf sehr vorteilhafte Weise möglich.

In the 1 The device shown and the method according to the invention described below now solve the problem of determining the exact distance between individual points of the measurement object and the microphone array by a combination of the spiral arms 120 and the microphones 121 formed microphone arrays with the video array formed from the video cameras 140 which are arranged in the center of the microphone array. These video cameras 140 record images of the current measuring arrangement in a continuous and synchronized manner parallel to the data stream of the microphone signals. The optical axes of the cameras 140 are arranged in parallel. Calibration minimizes aberrations in a manner known per se. An object which is at a given distance to the cameras 140 is located at different positions on the pictures of the cameras 140 displayed. Through precise knowledge of the geometric relationships between object and cameras 140 which can be determined for example by the so-called epipolar geometry, the camera images can be individually equalized so that the object is at the same position on the individual images. By overlaying the camera images you get a new image, on which the object appears sharp. An object which is located at a different distance than the one currently assumed, however, is not mapped to the same position in the camera images after the equalization. After the overlay, the object therefore appears blurred. In principle, the following methods are conceivable:

• a) By manually changing the assumed distance by means of the software, the position of the objects recorded in the video image can be determined one after the other. Synchronous with the change in the distance, the scanning plane for beamforming is also shifted. Once an object has been properly focused, the position and level of the associated sound source are also correct. In this embodiment of the method, all scan points lie on one level. In a source map, therefore, only sources that lie on this level can be displayed correctly. The advantage of this embodiment of the method is that a manual measurement of the measurement objects is eliminated. Since the video images are stored together with the microphone data, the determination of the correct distance can also be made after completion of a measurement campaign and disassembly of the measuring arrangement. The process is also robust and easy to use. A software implemented sliding image magnifying glass supports the Users in determining the highest image sharpness.
• b) The manual focus determination method described above can be used to sequentially determine the distance to all objects of interest. Interacting with the software saves these points. The software creates a surface on which all stored points lie and interpolates the surfaces in between. In this way, a three-dimensional surface is created successively, which is then used as a scanning surface during beamforming. This area can for example be assigned to the sound level, whereby an acoustic surface of the measurement object can be displayed. This allows all sources in a source map to be displayed correctly.
• c) According to another embodiment of the method is a depth estimate using the video cameras 140 performed. In principle, this method is based on the evaluation of point correspondences in different video images. Another variant is to calculate a discrete series of possible distances from the geometric arrangement of the equalized video images resulting as described above. Each image is then decomposed into small overlapping subregions. For each subarea, the similarity of the corresponding subregions of the individual video images is determined and stored. The methods for determining similarity measures are known from the field of so-called computer vision per se, for. Example, the determination of the sum of the pairwise squared differences between the images of the individual video cameras or the determination of the normalized cross-correlation coefficient. This process is repeated for all distances in question. Then, for each subarea, the distance for which the match is maximum is determined. This distance was the distance of the imaged object to the video cameras 140 and thus to the microphone array 120 . 121 , The method requires additional algorithms for the detection of pixels with high contrast, since only here a clear distinction between high and low match of the sub-areas is possible. This can be solved for example with the aid of an algorithm for edge detection. In this case, the quadratic sum of the brightness difference between the pixels and the average brightness value is determined for each subarea - this corresponds to the variance of the subarea. A threshold value is used to determine pixels with high contrast. From the environment of each of these pixels, a partial area is formed for the subsequent distance determination. Objects with a pronounced, repetitive structure lead to misdetections, since the overlay for several distances achieves equal values of agreement (moiré effect). This problem is resolved by marking pixels with repeated maximum values in the match as invalid. Finally, a three-dimensional surface is determined from all valid pixels by means of interpolation, which is used as a scanning surface for beamforming. The procedure is fully automatable. An interaction of the user is not required. In this method, a correct detection of moving objects in a very advantageous manner possible.

A advantageous embodiment of the method provides a coherence / incoherence filtering which opens up the possibility of coherences between the sources. Have coherent sources usually the same effect origin. The knowledge of this effect origin Helps with the reduction of noise or sound modification.

With Help of incoherence filtering it is possible a to dampen the dominant source in the source mapping, that quieter sources, previously obscured by the ghost sources were, become visible.

When coherently apply signals when passing through at any time a time-invariant filtering into each other converted can be. The coherence filtering takes place in usually in the frequency domain. Thereby the two time signals become disassembled into blocks. For each block the complex becomes Spectrum calculated for example by means of the Fourier transform. Now the product of the spectrum of the useful signal (microphone signals of the array) and the normalized conjugate complex spectrum of the reference signal educated. The resulting complex spectrum is over averaged all (as many) time blocks as possible. Signal components, which meet the above requirement will be included amplified, or other signal components are attenuated.

In accordance with the present invention, an arbitrary additional sensor (not shown), e.g. As a microphone, an accelerometer, a laser vibrometer or the like in the immediate vicinity of the source of interest. The system performs the above-described analysis using the array microphone signals and the reference microphone signal. The resulting complex spectra are then used for beamforming. All sources that are incoherent with the reference sensor are attenuated in the source mapping. The result of beamforming is an energy equivalent quantity for each scan point. In the system this size is parallel for the calculated unfiltered and coherence filtered spectra. By subtracting the two energy values, one obtains a source mapping in which the sources coherent to the reference sensor are now attenuated.

In summary, it can be said that the combination of a microphone array 120 . 121 with a system for determining the distance between microphone array and measurement objects, for example in the form of a camera array 140 the exact determination of the distance from sound sources to the microphone array 120 . 121 in industrial application. This allows accurate localization and quantification of sources. The optical distance determination eliminates the need to manually measure the distances. The distance determination can also - and this is a decisive advantage - subsequently (without the presence of the object to be measured), since the raw data of the video cameras 140 be stored with the microphone data. The ability to generate a three-dimensional scan surface also allows sources that are not in a plane to be mapped in a source map. The dynamic range of the source mapping can be fully exploited and increased. This reduces the computational effort for the analysis, since instead of multiple source maps (for each distance one) now only one must be determined. The procedure can be carried out completely automatically and does not require any assumptions about the measurement object. In addition, the method can be carried out during the measurement / evaluation in a short time.

The Coherence / incoherence filtering opens the Possibility to show coherence between the sources. This not only allows quieter sources to be shown but it can also be appropriate steps to Noise reduction or sound modification on the basis of coherent Sound sources are made.

One big advantage is that the process in real time and can be used online. The user can be active in this way interact with the measurement object and the effects of all modifications at the sources and the position of the reference sensor immediately Experienced.

QUOTES INCLUDE IN THE DESCRIPTION

This list The documents listed by the applicant have been automated generated and is solely for better information recorded by the reader. The list is not part of the German Patent or utility model application. The DPMA takes over no liability for any errors or omissions.

Cited patent literature

• - DE 10304215 A1 [0003, 0023]

Claims (16)

Method for the imaging of three-dimensional acoustic objects as a measurement object by recording an acoustic reference image of the measurement object via a microphone array ( 120 . 121 ), wherein the microphone array ( 120 . 121 ) an optical camera array ( 140 ), which simultaneously to the data stream of the microphone signals of the microphone array ( 120 . 121 ) recording temporally synchronized spatial information of the measurement object images that are brought directly or at any later time in relation to the acoustic reference image of the measurement object. A method according to claim 1, characterized in that the camera array at least two, preferably three video cameras ( 140 ), which continuously spatial information of the measurement object during the acoustic recording via the microphone array ( 120 . 121 ) record. Method according to claim 2, characterized in that the optical axes of the video cameras ( 140 ) are arranged parallel to each other. Method according to one of the preceding claims, characterized in that on the basis of the camera signals the distance between microphone array ( 120 . 121 ) and measured object is determined. Method according to claim 4, characterized in that the variable distance between microphone array ( 120 . 121 ) and measurement object is determined by moving, superimposing and determining the resulting match of a plurality, preferably three camera images. Method according to claim 4, characterized in that that the distance is determined by means of the epipolar geometry. Method according to one of the preceding claims, characterized in that the data stream of the microphone signals of the microphone array ( 120 . 121 ) and the data stream of the optical camera array ( 140 ) are related to each other to produce an acoustic three-dimensional image of the measurement object. Method according to one of claims 4 to 6, characterized in that an acoustic surface of the measuring object with a variable distance between the measuring object and microphone array ( 120 . 121 ) and a related scanning surface of the microphone array is determined and displayed in a display unit, so that all similar sources of the measurement object are displayed simultaneously in a pictorial representation. Method according to one of the preceding claims, characterized in that in real time by means of coherence or incoherence filtering of the data stream, the microphone signals of the microphone array ( 120 . 121 ) and at least one reference signal for amplifying or attenuating sound sources are used. Method according to claim 8, characterized in that that the incoherence or coherence filtering on the base of the frequency domain, preferably by means of Fourier transformation, transformed temporal microphone signals takes place. Device for the imaging of acoustic objects by recording an acoustic reference image of the measurement object via a microphone array ( 120 . 121 ), characterized by a microphone array ( 120 . 121 ) associated optical camera array ( 140 ), through which simultaneously to the data stream of the microphone signals of the microphone array ( 120 . 121 ) three-dimensional information of the measurement object having images in time synchronized recordable and evaluable in an evaluation device for creating an acoustic map of the measurement object. Apparatus according to claim 10, characterized in that the camera array at least two, preferably three video cameras ( 140 ) having. Apparatus according to claim 11, characterized in that the optical axes of the video cameras ( 140 ) are arranged parallel to each other. Device according to one of claims 10 to 12, characterized in that the video cameras ( 140 ) are arranged substantially in the center of the microphone array. Computer program that shows all the steps of a procedure according to one of claims 1 to 9 executes when it runs on a computing device. Computer program product with program code based on a machine-readable carrier is stored for execution of the method according to any one of claims 1 to 9, when the Program is running on a computer.