JP7474175B2 - Sound image drawing device and sound image drawing method - Google Patents

Description

The present invention relates to a sound image drawing device and a sound image drawing method.

Conventionally, a method is known in which a sound image is generated based on voice data collected inside a vehicle and the generated sound image is superimposed on an image (see, for example, Patent Documents 1 and 2). In addition, a method is known in which, when evaluating the positioning of a sound image, an evaluator hand-draws the sound image perceived by the evaluator on an overhead view and a front view.

JP 2020-12704 A JP 2004-77277 A

However, conventional technical methods may not be able to efficiently visualize the sound image perceived by the evaluator. For example, the methods of automatically drawing sound images described in Patent Documents 1 and 2 cannot visualize the subjective sound image perceived by the evaluator. In addition, the method of manually drawing the sound image on the overhead view and front view increases the workload of the evaluator.

The present invention has been made in consideration of the above, and aims to provide a sound image drawing device and a sound image drawing method that can efficiently visualize the sound image perceived by an evaluator.

In order to solve the above-mentioned problems and achieve the object, the sound image drawing device according to the present invention has an audio output control unit, a tracking unit, an image display control unit, and a storage unit. The audio output control unit causes the audio output device to output audio according to the acoustic parameters. The tracking unit tracks the user's body movements. The image display control unit synthesizes an object generated based on the trajectory of the user's body movements obtained by the tracking unit with a captured image and displays it on the display device. The storage unit associates the acoustic parameters with information for generating the object and stores them in the memory unit.

The present invention makes it possible to efficiently visualize the sound image perceived by the evaluator.

FIG. 1 is a diagram showing the configuration of a sound image rendering system according to an embodiment. FIG. 2 is a functional block diagram showing the configuration of the sound image rendering device according to the embodiment. FIG. 3 is a diagram showing an example of an object. FIG. 4 is a diagram illustrating the selection of an object. FIG. 5 is a diagram illustrating the deformation of an object. FIG. 6 is a diagram showing an example of an object. FIG. 7 is a diagram illustrating the reproduction of an object. FIG. 8 is a flowchart showing the procedure of a process executed by the sound image drawing device according to the embodiment.

Below, embodiments of the sound image drawing device and sound image drawing method disclosed in the present application will be described in detail with reference to the attached drawings. Note that the present invention is not limited to the embodiments described below.

First, a sound image drawing system including a sound image drawing device according to an embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing the configuration of the sound image drawing system according to an embodiment.

In sound quality evaluation, the tallyer tallys up and analyzes the sound images drawn by the evaluators. User U in FIG. 1 is the evaluator. The sound image drawing system 1 can assist the evaluator in drawing the sound image. The tallyer and the evaluator may be rephrased as the analyst and the inputter, respectively.

As shown in FIG. 1, the sound image drawing system 1 includes a vehicle 10, an audio output device 20, a display device 30, a camera 40, and a sound image drawing device 50. For example, the vehicle 10 is an automobile.

The audio output device 20 is, for example, a speaker provided in the vehicle 10. The display device 30 is, for example, VR (Virtual Reality) goggles worn by the user U.

The sound image drawing device 50 causes the display device 30 to display an image related to the sound image. Note that the display device 30, the camera 40, and the sound image drawing device 50 may be integrated into a single device.

The sound image rendering device 50 may be a separate device from the display device 30. In that case, the sound image rendering device 50 may be realized by an ECU (Electronic Control Unit) mounted on the vehicle 10, or a server capable of data communication with the vehicle 10.

In the sound image rendering method according to the embodiment, the sound image rendering device 50 causes the audio output device 20 to output sound according to the acoustic parameters.

In the sound image drawing method according to the embodiment, the sound image drawing device 50 tracks the body movements of the user U.

In the sound image drawing method according to the embodiment, the sound image drawing device 50 synthesizes an object generated based on the trajectory of the user U's body movements obtained by tracking with a captured image and displays it on the display device 30.

In the sound image drawing method according to the embodiment, the sound image drawing device 50 stores the acoustic parameters and information for generating an object in a storage unit in association with each other.

Therefore, according to this embodiment, the sound image perceived by the evaluator can be efficiently visualized.

FIG. 2 is a functional block diagram showing the configuration of a sound image drawing device according to an embodiment. As shown in FIG. 2, the sound image drawing device 50 has an IF unit 51, a control unit 52, and a storage unit 53.

The IF unit 51 is an interface for connecting the sound image drawing device 50 to other devices. The sound image drawing device 50 is connected to the audio output device 20, the display device 30, and the camera 40 via the IF unit 51.

The control unit 52 and memory unit 53 of the sound image drawing device 50 are realized by, for example, a computer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), flash memory, input/output ports, etc., and various other circuits.

The computer's CPU functions as the audio output control unit 521, tracking unit 522, image display control unit 523, and storage unit 524 of the control unit 52, for example, by reading and executing a program stored in the ROM.

The storage unit 53 corresponds to a RAM or a flash memory. The RAM or the flash memory can store acoustic information 531, object information 532, and information on various programs, etc.

The sound image rendering device 50 may also acquire the above-mentioned programs and various information via other computers or portable recording media connected via a wired or wireless network.

The audio output control unit 521 causes the audio output device 20 to output audio according to the acoustic parameters. For example, the audio output control unit 521 causes the audio output device 20 provided in the vehicle 10 to output audio according to the acoustic parameters.

The acoustic information 531 includes acoustic parameters. The audio output control unit 521 outputs audio based on the acoustic parameters obtained from the acoustic information 531.

The acoustic parameters may be parameters for reproducing the waveform of the sound, or may be a pre-created sound file.

The tracking unit 522 tracks the body movements of the user U. For example, the tracking unit 522 recognizes a part of the user U, such as the palm of the hand, based on the image input from the camera 40, and tracks the part by continually calculating the position of the recognized part.

The display device 30, camera 40, and tracking unit 522 can realize an AR (Augmented Reality) device capable of motion tracking, horizontal plane detection, etc.

The image display control unit 523 synthesizes the object generated based on the trajectory of the user U's body movement obtained by the tracking unit 522 with the captured image and displays it on the display device 30.

Here, the user U expresses the sound field perceived by listening to the sound through body movements. In this case, the sound field expressed by the user U is rendered as an object by the sound image rendering device 50 and is called a sound image.

For example, the image display control unit 523 renders a three-dimensional object generated based on the trajectory of the user U's body movement obtained by the tracking unit 522 in a virtual space with the captured image as the background. This allows the user U to intuitively confirm the sound image.

Figure 3 is a diagram showing an example of an object. The screen in Figure 3 is an example of a screen that the image display control unit 523 causes the display device 30 to display.

The background of the screen in FIG. 3 is an image captured by the camera 40. In this way, the image display control unit 523 superimposes the object on an image that mimics the field of view of the user U and displays it on the goggle-type display device 30 worn by the user U.

The image display control unit 523 displays the object 301 on the screen. The object 301 is a three-dimensional object. Therefore, it is possible to express gloss and shadow according to the light source and the shooting position of the camera 40, etc.

For example, the image display control unit 523 generates an object that coincides with the space enclosed by the trajectory of the palm of the user U obtained by the tracking unit 522. The image display control unit 523 may also generate objects such as spheres, ellipsoids, and rectangular parallelepipeds based on the trajectory.

The storage unit 524 associates the acoustic parameters with information for generating an object and stores them in the memory unit 53. For example, the storage unit 524 stores features such as the center of gravity, depth, and height of the object 301 as object information 532 in the memory unit 53.

The storage unit 524 also stores in the memory unit 53 the positional relationship between the marker 11 and the object shown in the first image. In the example of FIG. 3, the marker 11 is printed on the pillar portion of the vehicle 10, but it may be printed in another position.

Figure 4 is a diagram explaining the selection of an object. When a user U performs a predetermined operation on a generated object, the image display control unit 523 places the object in a selected state.

For example, when a gesture such as holding a palm over the object 301 in FIG. 4 for a certain period of time, stroking quickly, or tapping a certain number of times is detected from the trajectory obtained by the tracking unit 522, the image display control unit 523 sets the object 301 to a selected state.

The image display control unit 523 moves, enlarges, reduces, or deforms the three-dimensional object according to the body movements of the user U obtained by the tracking unit 522. For example, the image display control unit 523 can deform an object that is in a selected state according to the body movements of the user U.

In this way, user U can perform operations such as transforming the sound image through intuitive operations.

Figure 5 is a diagram explaining the deformation of an object. For example, if the object 301 is an ellipsoid, and the user U makes a movement that compresses the major axis of the ellipsoid, the image display control unit 523 shrinks the object 301 in the major axis direction.

In addition, the image display control unit 523 can detect user U's actions such as pinching, pushing, pulling, cutting, crushing, stretching, and squeezing, and can deform the object 301.

The image display control unit 523 can display multiple objects as shown in FIG. 6. FIG. 6 shows an example of objects.

In addition, the image display control unit 523 draws objects associated with each of the elements that make up the sound field based on the acoustic parameters.

For example, the image display control unit 523 renders the sound field represented by the user U in each frequency band (high, mid, low) as an individual object. This makes it possible to represent a variety of sound images.

In addition, if the audio is playing a piece of music, the image display control unit 523 may draw an object for each part or each instrument.

For example, objects 302, 303, and 304 in FIG. 6 may be generated by user U based on audio from different types of sound sources.

The image display control unit 523 may also display the object selected by the user U from among the multiple objects shown in FIG. 6, and hide the unselected objects.

The sound image rendering device 50 can reproduce the objects stored in the memory unit 53 at different dates, times, and locations. This allows users U other than the evaluator to re-experience the sound field perceived by the evaluator.

For example, the image display control unit 523 displays an object generated based on information for generating the object stored in the storage unit 53 by combining the object with a second image different from the first image, which is a captured image. FIG. 7 is a diagram for explaining the reproduction of an object.

Figure 7 shows an example of an object generated in a vehicle 10 being reproduced in an office. As shown in Figure 7, a marker 11a having visual characteristics equivalent to those of marker 11 is printed in a specified location in the office.

The background image in Figure 6 is an example of a first image. The background image in Figure 7 is an example of a second image.

The positional relationships between objects 302a, 303a, and 304a and marker 11a in FIG. 7 are the same as the positional relationships between objects 302, 303, and 304 and marker 11 in FIG. 6.

In this way, the sound image rendering device 50 allows the stored object to be reproduced in another location, allowing the tally-up person to perform tallying and analysis in any location.

In addition, the image display control unit 523 composites the object into the second image and displays it so that the positional relationship between the marker shown in the second image and the object displayed in the second image is the same as the positional relationship stored in the memory unit 53.

This allows the positional relationship to be reproduced accurately.

The reproduction of the saved object is not limited to VR goggles, but may be performed on any device equipped with the function of rendering three-dimensional objects, such as a personal computer, smartphone, tablet terminal, etc.

The procedure of the process executed in the sound image drawing device 50 according to the embodiment will be described with reference to FIG. 8. FIG. 8 is a flowchart showing the procedure of the process executed by the sound image drawing device according to the embodiment.

As shown in FIG. 8, first, the sound image rendering device 50 causes the sound output device 20 to output sound in accordance with the acoustic parameters (step S101). The sound image rendering device 50 also tracks the body movements of the user U from the image captured by the camera 40 (step S102).

Here, the sound image drawing device 50 checks whether the body movement of the user U is a generation action for generating an object (step S103).

If the body movement of the user U is not a generating movement (step S103: No), the sound image rendering device 50 proceeds to step S105.

On the other hand, if the body movement of the user U is a generating movement (step S103: Yes), the sound image rendering device 50 generates an object based on the trajectory of the body movement (step S104) and proceeds to step S105.

Next, the sound image drawing device 50 checks whether the body movement of the user U is a selection action for selecting an object (step S105).

If the body movement of the user U is not a selection movement (step S105: No), the sound image rendering device 50 proceeds to step S107.

On the other hand, if the body movement of the user U is a selection motion (step S105: Yes), the sound image rendering device 50 selects an object (step S106) and proceeds to step S107.

Furthermore, the sound image drawing device 50 checks whether the user's body movement is a deformation action for deforming an object (step S107).

If the body movement of the user U is not a deformation movement (step S107: No), the sound image rendering device 50 proceeds to step S109.

On the other hand, if the body movement of the user U is a transformation movement (step S107: Yes), the sound image rendering device 50 transforms the selected object (step S108) and proceeds to step S109.

For example, the object creation, selection, and transformation operations correspond to the operations described in Figures 3, 4, and 5, respectively.

Then, the sound image drawing device 50 draws the object (step S109). At this time, the sound image drawing device 50 reflects the result of the generation, selection, or transformation of the object in the object to be drawn.

If user U's work is finished (step S110: Yes), the sound image rendering device 50 associates the acoustic parameters with the object information and stores them (step S111).

On the other hand, if user U's work has not been completed (step S110: No), the sound image rendering device 50 returns to step S101 and repeats the process.

In this way, the sound image drawing device 50 can change and display objects in real time by repeatedly generating, selecting, and transforming objects, and drawing the objects. This allows the user U to intuitively evaluate and input.

The sound image drawing device 50 can generate a sound image without interrupting the evaluator's thoughts, especially compared to drawing a sound image on a two-dimensional piece of paper.

The sound image drawing device 50 can determine whether the task is finished based on an explicit end operation by the user U and conditions such as the passage of time.

As described above, the sound image drawing device 50 according to the embodiment has an audio output control unit 521, a tracking unit 522, an image display control unit 523, and a storage unit 524. The audio output control unit 521 causes the audio output device 20 to output audio in accordance with the acoustic parameters. The tracking unit 522 tracks the body movements of the user U. The image display control unit 523 synthesizes an object generated based on the trajectory of the body movements of the user U obtained by the tracking unit 522 with a captured image and displays it on the display device 30. The storage unit 524 associates the acoustic parameters with information for generating the object and stores them in the memory unit 53.

In this way, the sound image drawing device 50 allows the evaluator to visualize the position and shape of the sound image he or she perceives through an intuitive and simple action of indicating the position and shape of the sound image through body movements in space. Furthermore, the sound image drawing device 50 stores the acoustic parameters and the three-dimensional information of the object representing the sound image in association with each other. Therefore, according to this embodiment, the sound image perceived by the evaluator can be efficiently visualized. Furthermore, as a result, the work efficiency of the sound quality evaluation by the evaluator and the tallying person is improved.

In addition, by using a goggle-type display device, the user U can more accurately experience a sound field while viewing the real space.

Further advantages and modifications may readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof.

REFERENCE SIGNS LIST U User 1 Sound image drawing system 10 Vehicle 11, 11a Marker 20 Sound output device 30 Display device 40 Camera 50 Sound image drawing device 51 IF section 52 Control section 53 Storage section 301, 302, 303, 304, 302a, 303a, 304a Object 521 Sound output control section 522 Tracking section 523 Image display control section 524 Storage section 531 Acoustic information 532 Object information

Claims (8)

an audio output control unit that causes an audio output device to output audio in accordance with the acoustic parameters;
A tracking unit that tracks the user's body movements;
an image display control unit that synthesizes an object generated based on the trajectory of the user's body movement obtained by the tracking unit with a captured image and displays the object on a display device;
a storage unit that stores the acoustic parameters and information for generating the object in a storage unit in association with each other;
13. A sound image drawing device comprising: The sound image drawing device according to claim 1, characterized in that the image display control unit draws a three-dimensional object generated based on the trajectory of the user's body movement obtained by the tracking unit in a virtual space with the captured image as the background. The sound image drawing device according to claim 2, characterized in that the image display control unit moves, enlarges, reduces, or deforms the three-dimensional object according to the body movement of the user obtained by the tracking unit. The sound image drawing device according to claim 2, characterized in that the image display control unit draws objects corresponding to each of the elements that constitute the sound field based on the acoustic parameters. The sound image drawing device according to claim 1, characterized in that the image display control unit displays an object generated based on information for generating the object stored in the storage unit by synthesizing the object with a second image different from the first image, which is the captured image. The storage unit stores a positional relationship between the marker and the object shown in the first image in the memory unit;
The sound image drawing device of claim 5, characterized in that the image display control unit synthesizes and displays the object into the second image so that the positional relationship between the marker shown in the second image and the object displayed in the second image is the same as the positional relationship stored in the memory unit. The audio output control unit outputs audio from an audio output device provided in the vehicle in accordance with the acoustic parameters;
The sound image drawing device according to claim 1, characterized in that the image display control unit superimposes the object on an image simulating the user's field of vision and displays it on a goggle-type display device worn by the user. A sound image drawing method executed by a sound image drawing device, comprising:
an audio output control step of causing an audio output device to output audio in accordance with the acoustic parameters;
A tracking step of tracking a user's body movements;
an image display control step of synthesizing an object generated based on the trajectory of the user's body movement obtained by the tracking step with a captured image and displaying the image on a display device;
a storing step of storing the acoustic parameters and information for generating the object in a storage unit in association with each other;
13. A sound image drawing method comprising:

Images