JP2022174479A - Data visualizing system in space - Google Patents

Abstract

To provide a sound visualizing system capable of measuring sound-pressure level distribution data by beam-forming, respectively in real time even at different positions in depth directions in real three-dimensional space.SOLUTION: A data visualizing system is characterized by: storing coordinate positions of X, Y, Z coordinates in real space by carrying out space mapping processing of real space; superposing a different video image subjectable to video image control in real space where the coordinate positions are stored; forming a detection surface by cutting out a cross sectional face in a prescribed position of the different video image subjectable to video image control and also forming a plurality of mesh faces for calculation on the detection surface; detecting acquired data on the formed mesh faces for calculation by detection means; deriving data values on the mesh faces for calculation by calculating the detected data by calculation means; generating a data distribution diagram by data distribution diagram creating means from the acquired data values; and visualizing the data distribution situation in real space.SELECTED DRAWING: Figure 1

Description

The present invention relates to a sound visualization system, for example, capable of measuring sound pressure levels from different positions in the depth direction in a real space, and displaying the measurement results of the respective sound pressure level distributions in real time. This is related to a system that converts and visualizes sounds that can be stereoscopically visualized in a dimensional space.

As a conventional sound visualization device, the invention disclosed in Japanese Patent Application Laid-Open No. 2015-219138 is known, which searches for the arrival direction of the sound by beamforming applying the minimum variance method based on the input sound signal and presents the search result. there is
Also known is a method of measuring sound intensity in a three-dimensional space using a three-dimensional intensity probe (Japanese Patent Application Laid-Open No. 2003-294822).

JP 2015-219138 A JP 2003-294822 A

However, the sound visualization device according to Patent Document 1 performs measurement by arbitrarily setting the distance from the cross section where the sound pressure level distribution is to be measured to the microphone, and displays the measurement results two-dimensionally on the display of the PC. only be done. Therefore, there is a problem that it is difficult to grasp sound pressure level information at different positions on the evaluation plane, for example, in the depth direction.

Moreover, the measurement method according to Patent Document 2 has the problem that the flow of sound cannot be visualized unless the sound is stationary, and fluctuating sound cannot be measured in real time.

Therefore, the present invention has been devised to solve the above-described conventional problems, and enables real-time measurement of sound pressure level distribution data obtained by beam forming even at different positions in the depth direction in an actual three-dimensional space, A color-coded sound pressure level distribution map can be generated from each measured sound pressure level distribution data so as to be displayed in a three-dimensional space, which is a real space, thereby providing sound pressure level information at different positions in the depth direction. The purpose is to provide a sound visualization system that can reliably grasp and recognize about.

The present invention
Space mapping processing is performed on the real space to store the coordinate positions of the X, Y, and Z coordinates in the real space, and another image that can be image-controlled is superimposed in the real space in which the coordinate positions are stored. In addition, a detection plane can be formed by cutting out a cross-section at a predetermined location of another video for which the video control can be performed, and a plurality of calculation mesh planes can be formed on the detection plane, and the formed plurality of calculation meshes can be formed. Data obtained on the surface is detected by the detection means, the detected data is calculated by the calculation means to obtain data values on the plurality of calculation mesh surfaces, and a data distribution diagram creation means is obtained from the obtained plurality of data values. generated a data distribution map and visualized the data distribution situation in the real space,
characterized by
or,
Space mapping processing is performed on the real space in which the detection means for collecting sound from the sound emitting unit is arranged, and the coordinate positions of the X, Y, and Z coordinates in the real space are stored, and the coordinate positions are recognized. In the actual space, the detection means is arranged, and another image that can be image-controlled is superimposed, and a cross section at a predetermined position of the another image that can be image-controlled is cut out to form a sound detection surface. a plurality of computational mesh planes can be formed on the sound detection plane, sound data on the formed plurality of computational mesh planes is detected by the detection means, the detected sound data is calculated by the calculation means, and the Obtaining sound data values on a plurality of calculation mesh surfaces, generating a sound data distribution map from the obtained plurality of sound data values by a sound data distribution map creating means, and visualizing the sound data distribution situation in the real space,
characterized by
or,
The sound data value is a sound pressure level intensity value, and the sound data distribution map is a sound pressure level intensity distribution map,
characterized by
or,
The distribution map can display the intensity of sound pressure levels in different colors,
characterized by
or,
The display unit is a light transmissive display,
It is characterized by

According to the present invention, sound pressure level distribution data obtained by beam forming can be measured in real time even at different positions in the depth direction in an actual three-dimensional space, and the measured sound pressure level distribution data can be color-coded by color display. It is possible to provide a sound visualization system that can generate a sound pressure level distribution map to be displayed in a three-dimensional space, which is an actual space, and that can reliably grasp and recognize sound pressure level information at different positions in the depth direction. excellent effect.

It is an explanatory view explaining a schematic structure of the present invention. 1 is an explanatory diagram (1) for explaining the schematic configuration of an MR device; FIG. FIG. 2 is an explanatory diagram (2) for explaining the schematic configuration of the MR device; The MR device captures another video object, recognizes the position and orientation of the microphone array in the video object, and combines the position and orientation of the microphone array in the previously read three-dimensional space to obtain coordinate data in the three-dimensional space. FIG. 10 is an explanatory diagram for explaining an operation for associating with . FIG. 10 is an explanatory diagram for explaining that even if the wearer of the MR device moves due to the linking operation, the image of the three-dimensional space in which the microphone array is arranged does not change. FIG. 10 is an explanatory diagram illustrating an operation of manually adjusting the position and orientation of the microphone array; FIG. 4 is an explanatory diagram for explaining a setting operation of the sound visualization system according to the present invention; It is explanatory drawing (1) explaining the state which visualized the sound by this invention. It is explanatory drawing (2) explaining the state which visualized the sound by this invention. It is explanatory drawing (3) explaining the state which visualized the sound by this invention. FIG. 10 is an explanatory diagram when a light transmissive display is used for the display section; FIG. 10 is an explanatory diagram when a display that is not a light transmissive display is used for the display section;

BEST MODE FOR CARRYING OUT THE INVENTION The present invention will be described below based on an embodiment shown in the drawings.
FIG. 1 is a schematic configuration explanatory diagram for explaining the schematic configuration of the present invention.
Reference numeral 1 denotes a 16-channel microphone array configured by arranging, for example, 16 microphones 15 at predetermined intervals, and corresponds to detection means described in the claims. However, the detection means is not limited to the microphone array 1 .

Information on the sound pressure level from a predetermined position in the front three-dimensional space collected by the microphone array 1 is sent to the arithmetic unit 2 . The sound pressure level passes through the band-pass filter 3 of the arithmetic unit 2 to obtain a sound pressure level of a predetermined frequency.

That is, sound pressure levels collected from the plurality of microphones 15 of the microphone array 1, passed through the band-pass filter 3 of the arithmetic unit 2, and converted to a predetermined frequency are beam-formed to generate sound pressure level distribution data. . The sound pressure level distribution data is, for example, data processed as data capable of showing the intensity of the sound pressure level (dB) as a distribution map.

Then, the sound pressure level distribution data is generated as a color-coded sound pressure level distribution diagram in the image visualization unit 6, and the image is visualized on the display unit 11 or the like.
The color-coded sound pressure level distribution map data is transmitted to the control device 5 such as a PC, and can be stored in the storage unit.
Here, the arithmetic unit 2 and the control device 5 such as a PC may be configured as an integrated device, or may be configured separately.

Further, the data of the sound pressure level distribution map is also sent to the MR device 7, processed by the control unit 12 of the MR device 7, and visualized as an image. The full spelling of "MR" in the MR device 7 is "Mixed Reality", and "Mixed Reality" is translated as mixed reality.

Here, the MR device 7 includes an image capturing means 8 such as an RGB camera, a detection section 9 such as a depth sensor, a display section 11 such as a light transmissive display, a control section 12, and an input section 18 (FIG. 2).

Although four MR devices 7 are installed in FIG. 1, the number is not limited to this. It does not matter if it is one or more than one. When a plurality of MR devices 7 are configured as spectacle-type devices, for example, sound emission states are visualized in the real space through the display unit 11, which is a spectacle-type light transmission display. Images on which the pressure level distribution map is superimposed can be viewed.

2 and 3 are schematic configuration explanatory diagrams of the MR device 7. As shown in FIG. 1, sound pressure level distribution data generated by beamforming processing is sent to the MR device 7 and The control unit 12 controls and generates a color-coded sound pressure level distribution map (color map 14) in which the sound pressure level distribution data (for example, binary data 13) can be visualized.

Here, the operation of the MR device 7 will be further described. Since the glasses-type MR device 7 has a display unit 11 such as a light transmissive display, the real space can be viewed with the naked eye.
However, it also has a photographing means 8 such as an RGB camera 8 and a detection unit 9 such as a depth sensor. It is also configured to be able to
Then, the three-dimensional real space that has been photographed is spatially mapped by the control unit 12 as a three-dimensional space in which the coordinate positions of the X, Y, and Z coordinates can be recognized (see FIG. 4).

By the spatial mapping processing, for example, it is also possible to recognize the position of the wearer of the glasses-type MR device 7 in the three-dimensional space, which is the real space (self-position estimation function). These mapping and self-position estimation processes are performed, for example, by a known SLAM (Simultaneous Localization and Mapping) method.

Here, in the MR device 7, another image object can also be superimposed and arranged in the actual real space where the coordinate positions of the X, Y, and Z coordinates subjected to the spatial mapping process can be recognized. Then, a virtual image such as 3DCG of the other image object can be added to the real space where the coordinate positions of the X, Y, and Z coordinates can be recognized.
Here, the virtual video is configured so that video editing control can be performed by the control unit 12 .

Therefore, when capturing another video object with the MR device 7, the video object contains the above-described A microphone array 1 similar to the microphone array 1 arranged in the real space subjected to spatial mapping is arranged.

Then, the position and orientation of the microphone array 1 in the real space that has undergone spatial mapping processing are matched with the position and orientation of the microphone array 1 captured in another video object, and the coordinates of the real space are obtained. An operation of associating the data with the coordinate data of another video object is performed.

That is, the MR device 7 spatially maps the actual real space, for example, the three-dimensional real space in which the microphone array 1 is arranged, as a three-dimensional space in which the coordinate positions of the X, Y, and Z coordinates can be recognized. , superimposing another image object added with a virtual image such as 3DCG that can be edited and controlled by the control unit 12 in the real space that has undergone the space mapping process, and displaying the glasses-type transmissive display of the MR device 7. It was assumed to be visible to the naked eye through the portion 11 .

In other words, in the real space, for example, the state of sound emission, which cannot be visualized as an image, can be displayed in 3DCG or the like, and the image of another image object can be superimposed and displayed for visualization.

For example, using another video object, we can determine the position and volume of the sound emitted in the real space, especially the sound emission situation at multiple positions in the depth direction of the real space. It was configured so that the intensity of the sound could be detected.

Furthermore, the detected sound emission state and sound strength state data are processed and edited as a video, which is configured to be displayed on the display unit 11 of the light transmissive display, so that it can be recognized by the naked eye in a real space. It was made visible by superimposing them.

As described above, the MR device 7 of the present invention also has the photographing means 8 composed of an RGB camera or the like. Then, the image of the physical space photographed by the photographing means 8 is spatially mapped as a three-dimensional space image in which the coordinate positions in the X, Y, and Z directions are recognized. Here, a detection unit 9 such as a depth sensor is also provided for recognizing coordinates in the depth direction.

The space mapping data obtained by spatially mapping the real space shape captured by the photographing means 8 and the detecting portion 9 as a three-dimensional space shape whose depth position can also be recognized is stored in the storage means.

Here, after the coordinate position in the real space is recognized and stored by the space mapping process, the operator who wears and operates the MR device 7 can locate the MR device 7 at any position in the real space. You can also recognize whether you are wearing and operating.

In addition, the MR device 7 can also generate another image object to be superimposed on the real space on which the space mapping processing has been performed, and the other image object can add images such as 3DCG images that can be edited and controlled.

As another video object, for example, the example shown in FIG. 4 can be cited. For this image object, an image including a microphone array 1 similar to the microphone array 1 already spatially mapped in the real space and arranged at a predetermined position where the X, Y, and Z coordinates can be recognized is adopted. be.

Furthermore, for example, a 3DCG image, which can be edited and controlled, can be incorporated into the image of the separate image object.
Therefore, with another image object, it is possible to cut out a space at an arbitrary position in the space in front of the microphone array 1 as a 3DCG image that can be edited and controlled as described above. 3DCG images can be generated. Furthermore, a 3DCG image can be generated by dividing the sound detection plane 16 into a plurality of calculation mesh planes 17 each having a substantially square shape.

In this way, the image of another image object has, for example, a sound detection plane 16 and a calculation mesh plane 17 that can process sound emission conditions and sound strength conditions, and has a 3DCG image that can be edited. can do

Note that the number of microphones 15 arranged in the microphone array 1 is not limited at all. In this embodiment, a 16ch microphone array 1 is shown, and 16 microphones 15 are used. Further, although the sound detection surface 16 is provided with a plurality of divided computational mesh surfaces 17, the number of computational mesh surfaces 17 is not limited at all. For example, it may be composed of 33 horizontal and 33 vertical.

In this way, as the image of the other image object, for example, the microphone array 1 arranged in the image and the cross section of the space in front of which the sound is emitted can be cut out, and the cut out cross section can be image-processed. A sound detection surface 16 composed of a plurality of computational mesh surfaces 17 as a 3DCG image is configured as an image, and beam forming processing, which will be described later, is performed using this image. The sound pressure level of the sound detection surface 16 at a predetermined depth position can be measured, the sound pressure level distribution data can be calculated, and the sound pressure level distribution map based on the sound pressure level distribution data can be displayed in color. be.

Since the position of the sound detection surface 16 for measuring the sound pressure level is determined based on the positions and orientations of the plurality of microphones 15 arranged in the microphone array 1, coordinate recognition for the actual real space is performed. The installation position and orientation of the microphone array 1 in the real space that has undergone spatial mapping processing must match the installation position and orientation of the microphone array 1 in the video of another video object. Therefore, it is necessary to set the matching in the MR device 7 or the like.

Therefore, in the present invention, the installation positions and orientations of the microphone array 1 in the video of another video object displayed by the MR device 7 are changed to the installation positions and orientations of the microphone array 1 in the real space subjected to spatial mapping processing. In order to match the orientation, setting is performed manually as shown in FIG.

By adjusting the installation position and orientation of the microphone array 1 as described above, it is possible to align the positional relationship between the actual real space image including the position and orientation of the microphone array 1 and the image of another image object. Furthermore, it is possible to display a color map of a sound pressure level distribution diagram based on the sound pressure level distribution data on the sound detection surface 16 at the correct position in the actual real space image.

It should be noted that even if the wearer wearing the glasses-type MR device 7 moves, spatial mapping processing is performed in advance for the real space, and a spatial image obtained by performing coordinate recognition and another image object superimposed on the spatial image are generated. The positional relationship of the positions where the sound pressure level distribution diagram such as the color map is detected is fixed on the spatial image, and the positional relationship does not go out of order.

Next, beam forming processing using the microphone array 1, the sound detection surface 16, and the calculation mesh surface 17 will be described.
Beamforming processing is a method of visualizing sound, and uses a microphone array 1 (a plurality of omnidirectional microphones arranged on a plane) to digitally process three-dimensional spatial images in the depth direction. This is a method of calculating the sound pressure level on the calculation mesh plane 17 in a certain cross section (sound detection plane 16).

Then, the sound pressure level distribution map from the calculated sound pressure level distribution data is displayed in color, and superimposed on the three-dimensional spatial image as an image of another object, so that any depth direction of the three-dimensional spatial image can be obtained. It becomes possible to visualize the sound position even at the point.

An outline of the beamforming processing method will be described.
When sound enters the microphone array 1 from a certain direction, the arrival time of the sound at each of the multiple microphones 15 slightly shifts. However, this arrival time difference can be calculated from the position of each microphone 15 and the speed of sound. That is, a plurality of computational mesh planes 17 are formed on the sound detection plane 16 to be visualized, the distance from the microphone array 1 is arbitrarily set, and the position of each computational mesh plane 17 is determined based on the positional relationship of the microphones 15 and the speed of sound. In other words, the time difference when the sound arriving from 1 is incident on each microphone 15 is obtained.

Next, by delaying the signals from the microphones 15 by the amount of this time difference to align the arrival times and adding them in phase, only the sound arriving from that direction can be emphasized. Next, similar processing is performed for a large number of directions to calculate the sound pressure level on each calculation mesh plane 17 in the cross section (sound pressure detection plane 16) to be visualized, and generate sound pressure level distribution data.

Then, the generated sound pressure level distribution data is made into a distribution map and displayed in a color map by the control unit 12, so that the sound information can be visually grasped.

8 and 9 show specific examples in which the sound pressure level distribution data is converted into a distribution map and displayed in a color map by the control unit 12. FIG. In the actual figure, the intensity of the sound pressure level is displayed using colors such as red, and the difference in color and the shade of the color. That is, dark red indicates a high sound pressure level and indicates a strong point, and light red, yellow, or green indicates a low sound pressure level and indicates a weak point. FIG. 8 shows a sound pressure level distribution map at a measurement distance of 1.5 m from the microphone array 1, and FIG. 9 shows a sound pressure level distribution map at a measurement distance of 3.0 m from the microphone array 1. It is. In the figure, since the distribution map has to be expressed in monochrome, dark-colored areas have high sound pressure levels, and light-colored areas have low sound pressure levels.
In both cases, a speaker 20, which is a sound emitting device, is arranged in the vicinity, and therefore the sound pressure level distribution diagrams are figures showing substantially the same distribution.

Also, unlike FIGS. 8 and 9, FIG. 10 shows a state in which the wearer of the MR device 7 views the sound detection surface 16 from the left side of the arrangement surface on which the microphone array 1 and the sound detection surface 16 are arranged. However, even when the sound detection surface 16 is viewed from this position, the color-mapped sound pressure level distribution diagram can be confirmed.

If processing control is performed at high speed, transient sounds and unsteady sounds can also be visualized. This is because the processing capability of the control unit 12 is recognized to be, for example, a maximum of 25 fps.

The color coding and brightness for color display of the sound pressure level distribution map can be changed by the control unit 12 of the MR device 7, etc., and the sound pressure level distribution map color-coded by the changed color display is color map. 14 is displayed on the display unit 11 and stored as the storage data 10 in the storage unit (see FIG. 3).

That is, in the sound visualization system of the present invention, in addition to being able to measure the sound pressure level distribution in real time, the measured sound pressure level distribution can be stored in the MR device 7 . Furthermore, the stored sound pressure level distribution can be reproduced (reproduced) by projecting it onto the real space as a color-displayed sound pressure level distribution diagram (color map 14).

Here, the sound pressure level distribution data is stored in the MR device 7 as numerical data together with the upper and lower limit values of the band-pass filter 3 at the time of storage and the values of the measured cross-sectional distance. Then, when reproducing the stored data, a color map 14 is created in the MR device 7 based on this numerical data, and projected onto the real space in the form of 3DCG.

The procedure for storing and reproducing sound pressure level measurement data will be described below.
First, the UI 19 is used to set the upper and lower limits of the band-pass filter 3, the cross-sectional distance to be measured, and the time (length) for storing the results. Next, measurement of the sound pressure level distribution is started.
Then, when a save start button on the UI 19 is pressed at an arbitrary timing, the measurement results for the preset time are saved. Next, when the saved data reproduction button of the UI 19 is pressed, the saved sound pressure level distribution is reproduced (reproduced) in the actual real space as the color map 14 via the display unit 11 such as a light transmission display. .

Furthermore, the outline of the processing flow of the present invention will be described.
For example, from sound information collected by a 16-channel microphone array, the calculation unit 2 uses the value of the band-pass filter set in advance on the UI (user interface) 19 and the position of the cross section (sound detection surface 16) for measuring the sound pressure level distribution. Beamforming processing is performed based on this information. By this processing, the sound pressure level on each calculation mesh plane 17 in a certain section (sound detection plane 16) is obtained, and this information is sent to the MR device 7, the control device 5 such as a PC.

Setting operations such as band-pass filter value setting on the UI (user interface) 19 are performed by displaying a setting screen on the display unit 11 and tapping the screen, as shown in FIG. can be entered.

From the control device 5, sound pressure level data and measurement setting information on the UI 19 (specific frequency settings for beamforming processing, detection distance settings for color map display, etc.) converted into byte array data are also sent to the MR device 7. The control unit 12 of the MR device 7 generates sound pressure level distribution map data from the sound pressure level distribution data based on the UI value. Then, the information of the generated sound pressure level distribution diagram data is reconverted into the display color data of the color map, and the color map 14 is created from this display color data.

In addition, if the display unit 11 such as a light transmissive display is used for the spectacles type MR device 7, the color map image of the sound pressure level distribution diagram, which is another object image, is superimposed on the actual spatial image viewed with the naked eye. can see.

That is, as described above, the MR device 7 is considered to be a spectacles-type device, and the spectacles-type MR device 7 employs the light-transmitting display section 11 . Therefore, the color map image of the sound pressure level distribution map can be superimposed and visualized on the real space viewed with the naked eye through the light-transmitting display unit 11, and the sound can be visualized on the image in a realistic manner. becomes. As described in FIG.

A tablet-type display unit 11 that is not a light-transmissive display can also be used as the display unit 11 of the MR device 7 . In this case, the real space photographed by the camera is displayed on the display unit 11 of the tablet, and the color map image of the sound pressure level distribution map is superimposed on this image for visualization. As described in FIG.

In addition, measurement change settings such as changing the position of the sound detection surface 16 by moving it in the depth direction, changing the detection distance, changing the sound pressure level measurement range (change of the detection frequency range), etc. It can be changed not only on the device 7 but also on the controller 5 . That is, the setting screen is selected, the screen is displayed, and the setting is changed by the input section 18 .

The setting change information is sent to the control device 5, for example, and is reflected in the beam forming process. If color map information for another sound detection surface 16 is desired, the previous color map display operation must be canceled.

The present invention contemplates embodiments such as the following. For example, exploration of external noise inflow points (sound insulation defects) in buildings, defect inspections by exploring abnormal noise locations in factory equipment and production lines, sound exploration of equipment at construction sites, factory buildings, etc. Exploration of sound leakage parts of noise generating facilities, etc. can be mentioned.

1 Microphone array 2 Arithmetic unit 3 Bandpass filter 4 Beamforming processing unit 5 Control device 6 Image visualization unit 7 MR device 8 Photographing means 9 Detection unit 10 Saved data 11 Display unit 12 Control unit 13 Binary data 14 Color map 15 Microphone 16 Sound Detection surface 17 Calculation mesh surface 18 Input unit 19 UI
20 speakers

Claims (5)

Space mapping processing is performed on the real space to store the coordinate positions of the X, Y, and Z coordinates in the real space, and another image that can be image-controlled is superimposed in the real space in which the coordinate positions are stored. In addition, a detection surface can be formed by cutting out a cross section at a predetermined location of another image that can be controlled, and a plurality of calculation mesh surfaces can be formed on the detection surface, and the formed plurality of calculation meshes Data obtained on the surface is detected by the detection means, the detected data is calculated by the calculation means to obtain data values on the plurality of calculation mesh surfaces, and data distribution diagram creation means is obtained from the obtained plurality of data values. generated a data distribution map and visualized the data distribution situation in the real space,
A data visualization system in space characterized by:
The real space in which the detection means for collecting the sound from the sound emitting unit is arranged is subjected to space mapping processing, the coordinate positions of the X, Y, and Z coordinates in the real space are stored, and the coordinate positions are recognized. In the actual space, the detection means is arranged, and another image that can be image controlled is superimposed, and a cross section at a predetermined position of the another image that can be image controlled is cut out to form a sound detection surface. a plurality of computational mesh planes can be formed on the sound detection plane, sound data on the formed plurality of computational mesh planes is detected by the detection means, the detected sound data is calculated by the calculation means, and the Obtaining sound data values on a plurality of calculation mesh surfaces, generating a sound data distribution map from the obtained plurality of sound data values by a sound data distribution map creating means, and visualizing the sound data distribution situation in the real space,
A sound data visualization system in space characterized by:
The sound data value is a sound pressure level intensity value, and the sound data distribution map is a sound pressure level intensity distribution map,
3. The system for visualizing sound data in space according to claim 2, characterized in that:
The distribution map can display the intensity of sound pressure levels in different colors,
4. The system for visualizing sound data in space according to claim 2 or 3, characterized in that:
The display unit is a light transmissive display,
5. The system for visualizing sound data in space according to claim 2, claim 3, or claim 4, characterized in that:

Images