WO2022044342A1 - Head-mounted display and voice processing method therefor - Google Patents

Abstract

This HMD displays a video of a real space captured by an imaging unit 71 and an AR object generated by an AR object generating unit 17 superimposed on each other on a display unit 72. Correspondingly, voice generated in the real space or voice generated by the AR object is output from a voice output unit 82. At this time, a voice data processing unit 25, in accordance with the positional relationship between an object present in the real space and an AR object disposed in superimposition therewith, subjects voice data of at least one of the voice from the real space and the voice generated by the AR object to processing/treatment, taking the acoustic characteristics of a surrounding object or a surrounding AR object into consideration. An acoustic characteristics data saving unit 21 saves the acoustic characteristics of the AR object in patterned form using the shape of each constituent element as a parameter. In this way, it is possible to perform in real-time a realistic display of an AR object and voice output taking the arrangement of the AR object into consideration.

Description

Head-mounted display and its audio processing method

The present invention relates to a head-mounted display and a voice processing method thereof.

In recent years, head-mounted displays (hereinafter, HMD: Head Mounted Display) have been put into practical use. The HMD displays a virtual reality (VR) image (VR object) created by a computer or the like on a display screen in the form of glasses, allowing the user to experience the feeling of being in a virtual world. .. Furthermore, it is also possible to superimpose an image (AR object) of augmented reality (AR) generated by a computer on an image in real space and display the AR object as if it exists in real space. It is possible.

In the game using the VR object, Patent Document 1 defines a spherical object called a sound reflection object in order to improve the entertainment property of the virtual space, and inputs the sound reflection object to the microphone as the reflection is generated. A configuration is disclosed in which the user's sound is processed and output from the headphones.

Japanese Unexamined Patent Publication No. 2018-11193

In the case of a conventional general HMD, in the case of a video transmission type (video see-through type) in which a real space image can be captured by a shooting means (camera), the shot real space image and an AR object are superimposed on the display screen of the HMD. Is displayed. At that time, as the sound provided to the user (acoustic space including sound effects), the sound in the real space acquired by the voice input means (microphone) mounted on the HMD is output as it is regardless of the presence or absence of the AR object. .. Therefore, the voice heard by the user does not reflect the influence of the AR object visually recognized by the user, and may give a sense of discomfort to the user depending on the arrangement status of the AR object.

In this regard, use a method such as the Finite Difference Time Domain (FDTD) method, which assumes the existence of an AR object and simulates the effect of acoustic characteristics due to the existence of the AR object on a sound field by a computer. You can also. However, such a simulation has a large computational load, increases the processing time, and is difficult to process in real time.

The above-mentioned Patent Document 1 describes a method of using a non-transmissive HMD, arranging a VR object in a virtual space, and changing the sound effect by the presence of a sound reflection object. However, the influence on the sound generated by the sound source in the real space due to the arrangement of the VR object or the AR object other than the sound reflection object is not taken into consideration, and the discomfort given to the user cannot be eliminated. Further, since Patent Document 1 defines that the sound source to be processed for sound exists at the center of the sound reflection object, it is difficult to apply it when the positional relationship between the sound source position and the AR object changes.

An object of the present invention is to arrange an AR object in a real space, process a voice generated in the real space in real time in consideration of the arrangement of the AR object, and present the head-mounted display to the user and a voice processing method thereof. To provide.

In order to solve the above-mentioned problems, the head-mounted display of the present invention has an imaging unit that captures an image in real space, an AR object generation unit that generates an AR object, and an image in real space that has been captured, as image processing. It includes an AR object superimposing unit that superimposes the generated AR object, and a display unit that displays the superposed real space image and the AR object. Further, as voice processing, the positional relationship between the voice input unit for inputting the voice emitted in the real space, the object existing in the real space arranged by the AR object superimposing unit, and the AR object displayed superimposed on the object. A voice data processing unit that processes at least one of the voice data of the voice from the real space and the voice emitted by the AR object in consideration of the acoustic characteristics of the surrounding object or the surrounding AR object, and the voice data. It is provided with an audio output unit that outputs audio from a real space processed by the processing unit or audio of an AR object.

According to the present invention, it is possible to realize a realistic display of an AR object and an audio output in consideration of the arrangement of the AR object in real time.

The schematic diagram which shows an example of the voice processing treated in this invention. The schematic diagram which shows an example of the voice processing treated in this invention. The schematic diagram which shows the other example of the voice processing treated in this invention. The schematic diagram which shows the other example of the voice processing treated in this invention. The figure which shows the hardware structure of HMD in Example 1. FIG. The figure which shows the functional block composition of HMD in Example 1. FIG. The figure which shows the example of the notation of acoustic characteristics. A table that describes the impulse responses of the components of an AR object. The figure which shows the waveform of the impulse response schematically. The figure which shows the example of video / audio processing at the time of AR object superimposition display. The figure which shows the example of video / audio processing at the time of AR object superimposition display. The figure which shows the example of video / audio processing at the time of AR object superimposition display. The figure which shows the other example of video / audio processing at the time of AR object superimposition display. The figure which shows the other example of video / audio processing at the time of AR object superimposition display. A flowchart showing a video processing procedure at the time of superimposing an AR object. A flowchart showing a voice processing procedure at the time of superimposing an AR object. The figure which shows the functional block composition of HMD in Example 2. FIG. The figure which shows the example of AR object superimposition display. The figure which shows the example of AR object superimposition display. A flowchart showing a video processing procedure at the time of superimposed display. A flowchart showing a voice processing procedure at the time of superimposed display. The figure which shows the state which the user attached the HMD of Example 3. The figure which shows the functional block composition of HMD in Example 4. FIG. A flowchart showing a voice processing procedure at the time of superimposed display. The figure which shows the whole structure of the HMD system in Example 5.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

1A and 1B are schematic views showing an example of voice processing handled in the present invention. The user 10 is wearing a head-mounted display (HMD) 1 and is listening to a voice emitted from the mouth 301 of the person 30. The HMD1 is attached to the head of the user 10, and the view of the user 10 is covered with the display screen of the HMD1. A camera 70 is arranged in the center of the front of the HMD 1 to capture a real-space image in the front direction of the user 10. The HMD1 is a video transmission type HMD, and the captured image is displayed on the display screen of the HMD1.

Further, although not shown, a microphone (hereinafter, "microphone") for acquiring real-space sound is arranged in the vicinity of the camera 70. The sound acquired by the microphone is processed as described later as necessary, and is output from the headphone 80 to the user 10. Although the HMD 1 and the headphone 80 are electrically connected, they may be structurally integrated.

FIG. 1A shows a state in which there is no AR object on the line segment connecting the user 10 and the mouth 301 of the person 30. In this state, the user 10 can visually recognize the person 30 in front by displaying the image taken by the camera 70 of the HMD1. Further, by acquiring the voice emitted from the mouth 301 of the person 30 with the microphone of the HMD 1 and outputting it from the headphone 80 without processing as it is, the user 10 can hear the voice of the person 30 without discomfort.

FIG. 1B shows a state in which the AR object 35 is arranged on the line segment connecting the user 10 and the mouth 301 of the person 30. It is assumed that the AR object 35 here is an acoustic obstacle that attenuates the transmission of sound, such as a tsuitate. In this state, even if the person 30 can be photographed by the camera 70 of the HMD 1, the person 30 is hidden behind the AR object 35 and is not displayed on the display screen of the HMD 1, and the user 10 visually recognizes the person 30. Can not do it. On the other hand, the sound emitted from the mouth 301 of the person 30 can be acquired by the microphone of the HMD 1 and output from the headphones 80. However, as in FIG. 1A, when the acquired voice data is output from the headphone 80 without being processed as it is, the voice of the invisible person 30 can be heard without being affected by the AR object 35 (acoustic obstacle). As a result, the user 10 feels uncomfortable.

Therefore, in this embodiment, when the AR object 35 that does not exist in the real space is displayed, the sound emitted from the mouth 301 of the person 30 is generated as if the AR object 35 exists in the real space, taking into account the acoustic characteristics of the AR object 35. Process as follows. Specifically, according to the shape and material of the AR object 35, processing processing for attenuating the audio data is performed and output. As a result, it is possible to reduce the discomfort of the voice given to the user 10.

Further, FIGS. 1C and 1D are schematic views showing other examples of voice processing handled in the present invention. In this example, the user 10 wearing the HMD 1 is located behind the person 30 and is in a state of listening to the voice emitted from the mouth 301.

FIG. 1C shows a state in which there is no AR object on a straight line connecting the user 10 and the mouth 301 of the person 30. By acquiring the voice emitted from the mouth 301 of the person 30 with the microphone of the HMD 1 and outputting it from the headphone 80 without processing as it is, the user 10 can hear the voice of the person 30 without discomfort. However, in this case, since the user 10 is located behind the person 30, the audible volume is smaller than that in the case of FIG. 1A.

FIG. 1D shows a state in which the AR object 35'is arranged on the extension line connecting the user 10 and the mouth 301 of the person 30. It is assumed that the AR object 35'here is an acoustic reflector that reflects the transmission of sound, for example, a mirror surface. In this state, the voice emitted from the mouth 301 of the person 30 is reflected by the AR object 35'and reaches the user 10 (indicated by a chain line in the figure). Therefore, as in FIG. 1C, if the audio data acquired by the microphone is output from the headphone 80 without being processed as it is, the user 10 feels uncomfortable.

Therefore, in this embodiment, when the AR object 35'that does not exist in the real space is displayed, the sound emitted from the mouth 301 of the person 30 is processed in consideration of the acoustic characteristics (reflection characteristics) of the AR object 35'. Specifically, according to the shape and material of the AR object 35', processing processing for amplifying audio data is performed and output. As a result, it is possible to reduce the discomfort of the voice given to the user 10.

As described above, in this embodiment, when displaying an AR object that does not exist in the real space, the audio data is processed and output in consideration of the acoustic characteristics (attenuation characteristics / reflection characteristics) of the AR object. Then, as shown in FIG. 1B, the case where the AR object is an acoustic obstacle (attenuation characteristic) will be mainly described.
Hereinafter, the configuration and operation of the head-mounted display (HMD) of this embodiment will be described.

[HMD hardware configuration]
FIG. 2A is a diagram showing a hardware configuration of a head-mounted display (HMD) 1. The HMD 1 includes a main control unit 2, a system bus 3, a storage unit 4, a sensor unit 5, a communication processing unit 6, a video processing unit 7, an audio processing unit 8, and an operation input unit 9.

The main control unit 2 is a microprocessor unit that controls the entire HMD 1 according to a predetermined operation program. The system bus 3 is a data communication path for transmitting and receiving various commands and data between the main control unit 2 and each constituent block in the HMD 1.

The storage unit 4 stores various data such as a program unit 41 that stores a program for controlling the operation of the HMD 1, an operation setting value, a detection value from the sensor unit, an object including contents, and library information downloaded from the library. It has various data units 42 and a rewritable program function unit 43 such as a work area used for various program operations.

The storage unit 4 can store an operation program downloaded from the network, various data created by the operation program, and the like. In addition, it is possible to store contents such as moving images, still images, and sounds downloaded from the network. In addition, it is possible to store data such as moving images and still images taken by using the shooting function of the camera. Here, the storage unit 4 needs to hold the stored information even when the HMD 1 is not supplied with power from the outside. Therefore, for example, a device such as a semiconductor element memory such as a flash ROM or SSD (Solid State Drive), a magnetic disk drive such as an HDD (Hard Disc Drive), or the like is used. Each operation program stored in the storage unit 4 can be updated and expanded in function by a download process from each server device on the network.

The sensor unit 5 is a group of various sensors for detecting the state of the HMD1. The sensor group includes a GPS (Global Positioning System) receiving unit 51, a geomagnetic sensor unit 52, a three-dimensional sensor unit 53, an acceleration sensor unit 54, and a gyro sensor unit 55. These sensor groups detect the position, tilt, direction, movement, etc. of the HMD1. In addition to this, an illuminance sensor, an altitude sensor, a proximity sensor, and the like may be further provided.

Of these, the three-dimensional sensor unit 53 measures the distance to each point of the object. There are various methods for 3D sensors (3D scanners), such as non-contact light (lattice pattern) projection method, non-contact laser light cutting method, and non-contact phase difference method (phase shift method) using a laser. ) Etc. are possible. In this embodiment, any method may be used.

The communication processing unit 6 has a LAN (Local Area Network) communication unit 61 and a telephone network communication unit 62. The LAN communication unit 61 is connected to a network such as the Internet via an access point or the like, and transmits / receives data to / from each network server device on the network. The connection with the access point or the like may be made by a wireless connection such as Wi-Fi (registered trademark).

The telephone network communication unit 62 performs telephone communication (call) and data transmission / reception by wireless communication with a base station or the like of a mobile telephone communication network. Communication with base stations, etc. is performed by W-CDMA (Wideband Code Division Multiple Access) (registered trademark) method, GSM (Global System for Mobile communications) method, LTE (Long Term Evolution) method, or other communication methods. good.

The LAN communication unit 61 and the telephone network communication unit 62 each include a coding circuit, a decoding circuit, an antenna, and the like. Further, the communication processing unit 6 may further include other communication units such as an infrared communication unit and a Bluetooth (registered trademark) communication unit.

The video processing unit 7 has an imaging unit 71, a display unit 72, and a video calculation unit 73. The image pickup unit 71 uses an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor to convert the light input from the lens into an electric signal to convert video data of an external scene or an object. The camera 70 (FIGS. 1A to 1D) to be acquired.

The display unit 72 provides image data (AR object) and video data (camera-captured video) to the user 10 wearing the HMD 1. Here, since the HMD 1 is a video transmissive type that does not see through the real space, the display unit 72 is a display device such as a backlit type liquid crystal display or a self-luminous organic EL display.

The video calculation unit 73 performs arithmetic processing related to video, such as a process of generating an augmented reality (AR object) image by a computer and a process of superimposing a real space image acquired by the imaging unit 71 and an AR object.

The voice processing unit 8 has a voice input unit 81, a voice output unit 82, and a voice calculation unit 83. The voice input unit 81 is a microphone that converts real space sounds, user voices, and the like into voice data and inputs them. The audio output unit 82 is a headphone 80 (FIGS. 1A to 1D) that outputs audio information and the like necessary for the user.

The voice calculation unit 83 performs calculation processing related to voice, such as detection of the position of a sound source and processing of voice data based on the acoustic characteristics of an AR object (acoustic changes due to material, thickness, cross-sectional area, etc.).

The operation input unit 9 receives an input of an operation instruction to the HMD1. The operation input unit 9 is composed of operation keys and the like in which button switches and the like are arranged, but may further include other operation devices. Further, the communication processing unit 6 may be used to operate the HMD 1 by using a separate mobile terminal device connected by wired communication or wireless communication. Further, the HMD1 may be operated by analyzing the captured image of the image pickup unit 71 of the image processing unit 7 and performing an operation such as a gesture.

The hardware configuration example of HMD1 shown in FIG. 2A includes many configurations that are not essential to this embodiment, but even if the configuration is not provided with these, the effect of this embodiment may be impaired. do not have.

[HMD functional block configuration]
FIG. 2B is a diagram showing a functional block configuration of the head-mounted display (HMD) 1. Each functional block included in the control function unit 11 is mainly realized by the main control unit 2 in FIG. 2A, and the program unit 41 and the program function unit 43 of the storage unit 4. The function of each function block will be described.

The 3D sensor information acquisition unit 12 acquires information from the 3D sensor unit 53 of the sensor unit 5. The information from the three-dimensional sensor unit 53 includes distance information from the HMD 1 to each point of the object in the real space.

The 3D data processing unit 13 grasps the shape of the object based on the information acquired by the 3D sensor information acquisition unit 12 (distance information to each point of the object). The three-dimensional data storage unit 14 stores the three-dimensional data in the real space obtained by the three-dimensional data processing unit 13 in various data units 42 of the storage unit 4. In this embodiment, it is premised that the three-dimensional data in the real space is acquired in advance and stored by the three-dimensional data storage unit 14.

The shooting data acquisition unit 15 captures the real space by the imaging unit 71 (camera 70) of the video processing unit 7 and acquires the shooting data. With this function, it is possible to provide a real space image to a user 10 who wears a video transmissive HMD1 that cannot directly see the real space.

The AR object information storage unit 16 stores information about the AR object in various data units 42 of the storage unit 4. The information about the AR object includes information about the image of the AR object (shape, size, arrangement, etc.) and incidental information about the virtual elements (hereinafter, components) that make up the AR object (material and thickness of each component).・ Cross-sectional area, etc.) is included. Further, the AR object information and the AR object incidental information can be acquired from an external server connected to the Internet network via the LAN communication unit 61 of the communication processing unit 6.

The AR object generation unit 17 generates an AR object by using the video calculation unit 73 of the video processing unit 7 based on the information about the image of the AR object stored in the AR object information storage unit 16. The AR object to be generated determines display specifications such as its shape, size, and arrangement according to the shape information of the real space stored in the three-dimensional data storage unit 14.

The AR object superimposing unit 18 superimposes the real space image acquired by the shooting data acquisition unit 15 and the AR object generated by the AR object generation unit 17 by using the image calculation unit 73 of the image processing unit 7. Specifically, occlusion processing such as displaying the real space image located in front (front) of the AR object but not displaying the real space image located in the back (back) of the AR object is performed and superimposed processing is performed. I do.

The display data output unit 19 displays the real space image superimposed by the AR object overlay unit 18 and the image of the AR object on the display screen of the HMD1.

The acoustic characteristic data storage unit 21 stores acoustic characteristic data related to acoustic characteristics (attenuation characteristics / reflection characteristics) with parameters such as material, thickness, and cross-sectional area for each component of the AR object. Of course, it goes without saying that the acoustic characteristic data storage unit 21 also stores acoustic characteristic data related to materials other than the components of the AR object to be adopted. Further, it is also possible to acquire the acoustic characteristics related to the material of the component of the AR object from the external server connected to the Internet network via the LAN communication unit 61 of the communication processing unit 6.

The voice data acquisition unit 22 is the voice input unit 81 (microphone) of the voice processing unit 7, and acquires the voice in the real space. The sound source position specifying unit 23 detects the position of the sound source. For this purpose, the voice calculation unit 83 of the voice processing unit 8 is used from the arrangement information of the real space stored in the three-dimensional data storage unit 14 and the image of the real space acquired by the shooting data acquisition unit 15. Identify the location of the sound source.

The AR object shielding determination unit 24 uses the positional relationship between the sound source position of the audio data acquired by the sound source position specifying unit 23 and the display position of the AR object displayed by the display data output unit 19, and the sound source position of the audio data. It is determined whether or not the display position of the AR object exists on the line segment connecting the user 10 and the position of the user 10 (that is, the position of the HMD1). That is, from the viewpoint of the user 10 (HMD1), it is determined whether or not the sound source position of the audio data is shielded by the AR object.

The voice data processing unit 25 analyzes and processes the voice data acquired by the voice data acquisition unit 22 according to the determination result of the AR object shielding determination unit 24. Specifically, when it is determined that the sound source position of the audio data is shielded by the AR object, the information of the AR object component is selected from the acoustic characteristics of each material stored in the acoustic characteristic data storage unit 21. Select the corresponding acoustic characteristics based on. Further, the voice data is analyzed and processed by the voice calculation unit 83 of the voice processing unit 8 using the selected acoustic characteristics.

The voice data output unit 26 outputs the voice processed by the voice data processing unit 25 from the voice output unit 82 (headphone 80) of the voice processing unit 7 toward the user 10.

[Acoustic characteristics of AR objects]
Next, the acoustic characteristics for the components of the AR object stored in the acoustic characteristic data storage unit 21 will be described.

FIG. 3 is a diagram showing an example of notation for acoustic characteristics, and shows two examples (a) and (b) having different components. (A) is an example of a low-pass type characteristic, and (b) is an example of a band-pass type characteristic, and the change in amplitude (response characteristic) is indicated by taking time or frequency on the horizontal axis. If there is an actual AR object, the acoustic characteristics of the actual object can be measured by a stationary analysis method, a non-stationary analysis method such as an impulse response, and stored in the acoustic characteristic data storage unit 21. However, in this embodiment, in order to simplify the arithmetic processing, the impulse response according to the incidental information such as the material, thickness, and cross-sectional area of the constituent elements of the AR object is stored in a pattern. The impulse response is a time change of the output amplitude waveform when an impulse (a signal having a very short time) is input.

FIG. 4 is a table describing the impulse response to the components of the AR object. This table is stored in the acoustic characteristic data storage unit 21. The items in the table 450 are the acoustic characteristics (impulse response) of the component with respect to the parameters of the number 451 that identifies the component of the AR object, the material 452 of the component, the thickness 453 of the component, and the cross-sectional area 454 of the component. ) 455 is described in a pattern. The parameters of the material 452 are distinguished by metal, glass, plastic and the like. The thickness 453 is distinguished by two patterns of thick / thin, and the cross-sectional area 454 is distinguished by two patterns of large / small. The impulse response 455 of each parameter is given in a pattern with a response value 1 to a response value 4 or the like.

FIG. 5 is a diagram schematically showing an impulse response waveform, and the response values 1 to 4 in FIG. 4 are shown as an example. The impulse response is a time change of the output amplitude waveform when an impulse is input, and changes depending on the combination of parameters such as the material, thickness, and cross-sectional area of the component.

For example, the response value 1 is a case where the material is metal, the thickness is thick, and the cross-sectional area is large, and the response waveform has a large amplitude attenuation. On the other hand, the response value 2 is a case where the material is metal, the thickness is thick, and the cross-sectional area is small, and the response waveform has a small amplitude attenuation. Further, the response value 3 and the response value 4 are cases where the material is metal and the thickness is thin, and the difference in frequency characteristics appears as the difference in the shape of the response waveform. In this way, the impulse responses to various conditions of the components are patterned and expressed. Impulse response values for materials other than metal can be patterned in the same way.

In this embodiment, the corresponding impulse response value is selected by referring to the table according to the material, thickness, and cross-sectional area of the components of the AR object. If the AR object information storage unit 16 does not have information on the components of the AR object (material, thickness, cross-sectional area, etc.), the information on the image of the AR object (shape, size, surface texture, etc.) can be used. It is also possible to estimate. Since the impulse response values are patterned in this way, it is possible to simplify and speed up the arithmetic processing.

The table in which the acoustic characteristics of this component are patterned and the impulse response value may be stored in various data units 42 of the storage unit 4 of the HMD 1, but communication processing is performed when the material of the AR object is known. It is also possible to download and use only the necessary impulse response value from an external server connected via the LAN communication unit 61 of the unit 6.

Although the impulse response value is adopted as the acoustic characteristic used in this embodiment, as shown in FIG. 3, a filter characteristic (for example, a low-pass filter) that defines the frequency response can also be adopted.

Further, the acoustic characteristics used in this embodiment are described assuming the state of FIG. 1B and paying attention to the attenuation characteristics when the voice passes through the AR object 35. Furthermore, in order to correspond to the state of FIG. 1D, by considering the reflection characteristic when the voice is reflected by the AR object 35', more realistic voice processing becomes possible. In this case, the distance between the AR object and the sound source in the real space may be added as a parameter to the table of FIG.

[Example of superimposed display by HMD]
6A to 6C are diagrams showing an example of video / audio processing at the time of superimposing and displaying an AR object. First, video processing will be described.

FIG. 6A is a real-space landscape seen in front of the user, and is photographed by the image pickup unit 71 (camera 70) of the HMD1. In the real space in this example, the background 31 of the room and the person 32 sitting in the corner of the room (the mouth 321 which is the sound source emitted by the person 32) exist.

FIG. 6B is an AR object generated by the AR object generation unit 17 of the HMD1. In this example, an example of an AR object 36 of an automobile is shown as an AR object. Image information of various AR objects is stored in the AR object information storage unit 16, and is selected from the image information.

FIG. 6C is an image in which the photographed image of the real space of FIG. 6A (the image of the background 31 of the room and the image of the person 32) and the AR object 36 of FIG. 6B are superimposed, and the display screen is displayed by the display data output unit 19 of the HMD 1. It is displayed on (display unit 72). The user 10 wearing the HMD1 can visually recognize such a superposed image via the HMD1.

In the video transmissive HMD of this embodiment, a real space image is captured by the image pickup unit 71, the image data is processed, and the image data is superimposed and displayed on the display unit 72 together with the AR object 36. At that time, the AR object superimposing unit 18 considers the state of the real space (arrangement and size of the object, etc.) stored in the three-dimensional data storage unit 14, and the size and arrangement of the AR object is the real space. Process so that it does not conflict with the state. That is, the real space image (a part of the background 31) that is not shielded by the AR object 36 is displayed, and the real space image data (a part of the background 31 and the person 32) that is shielded by the AR object 36 is not displayed. Perform processing.

In the case of FIG. 6C, the AR object 36 (automobile) is present in front of the background 31 and the person 32, and the person 32 is hidden behind the AR object 36 (automobile). In this example, the person 32 is hidden behind the AR object 36 and cannot be seen, but the outline of the person 32 is shown by a broken line 32'in order to show the position.

Next, voice processing will be explained. When the person 32 emits sound from the mouth 321 in the state of FIG. 6A, the sound is collected by the voice input unit 81 (microphone) and output as it is from the voice output unit 82 (headphone 80). The user 10 wearing the HMD 1 does not feel any discomfort when listening to the voice. In this case, the sound source position is specified by the sound source position specifying unit 23 to be the mouth 321 of the person 32.

On the other hand, in the state of FIG. 6C, if the voice of the person 32'hidden by the AR object 36 (automobile) is output at the same volume as that of FIG. 6A, the user 10 does not know the corresponding sound source, and a sense of discomfort occurs. Therefore, in this embodiment, when the sound source position is in a positional relationship shielded by the AR object 36, the audio data is processed and output based on the acoustic characteristics of the components of the AR object 36.

Therefore, the AR object shielding determination unit 24 determines whether or not the sound source position of the audio data is shielded by the AR object as seen from the user 10. When it is determined that the sound source position is shielded, the voice data processing unit 25 processes the voice data into a voice that is slightly trapped (muffled), and the processed voice is processed from the voice output unit 82 (headphone 80). Output. As a result, the user 10 feels that the voice emitted by the person 32'is heard from behind, and can intuitively grasp that the voice is from the person 32'behind the AR object 36 (automobile). ..

The voice data processing unit 25 has acoustic characteristics corresponding to the components (for example, the material is metal, the thickness is thin, and the cross-sectional area is large) of the AR object (automobile) stored in the acoustic characteristic data storage unit 21. By using the impulse response values 3 (FIGS. 4 and 5), it is possible to process the voice data with an increased sense of reality.

7A and 7B are diagrams showing other examples of video / audio processing at the time of superimposed display of AR objects. Here, assuming a case where a person gets on an AR object (automobile), a case where the processing of voice data from the person is different depending on the open / closed state of the window is shown.

FIG. 7A shows a state in which the AR object 37 and the person 33 in the real space are superimposed and displayed. The person 33 adjusts the arrangement and size of the AR object 37 so as to be in the AR object 37 (automobile). Further, the window 371 of the AR object 37 is closed, and the person 33 is displayed semi-transparently through the window 371.

In this state, the voice emitted by the person 33 is processed according to the acoustic characteristics of the AR object 37 and presented to the user 10 wearing the HMD 1. Since the window 371 whose material is glass exists as a component of the AR object on the line segment connecting the mouth 331 of the person 33 and the user 10, the impulse response value of glass is used for the acoustic characteristics.

FIG. 7B shows a state in which the passenger seat window 371 of the AR object 37 is open. Since there is no component of the AR object on the line segment connecting the mouth 331 of the person 33 and the user 10, the voice emitted by the person 33 is output to the user 10 without any processing, as in the state of FIG. 6A. do. In this way, by making the voice presented to the user 10 different according to the change of the AR object (the open / closed state of the window), the voice heard by the user 10 becomes more realistic.

In this example, opening and closing of the window was explained as a change of the AR object, but it can be applied to any form as long as it is a change of the AR object. Needless to say, for example, it can be applied to the open / closed state of a door in a house displayed as an AR object, the attachment / detachment state of a mask attached to a person's mouth, and the like.

Further, in the above example, it is assumed that the AR object is stationary, but it goes without saying that the above voice processing can be applied even when the AR object moves. In that case, the voice heard by the user 10 may be changed over time depending on the position of the AR object passing in front of the sound source.

[Video / audio processing procedure for superimposed display]
FIG. 8 is a flowchart showing a video processing procedure for superimposing and displaying a real space video and an AR object. Here, a case where the real space image of FIG. 6A and the AR object of FIG. 6B are superimposed and displayed as superimposed as shown in FIG. 6C will be described. Further, the following processing is executed by each functional block shown in FIG. 2B.

When the superimposed display process is started (S411), the shooting data acquisition unit 15 acquires the shooting data in the real space (S412).

From the AR objects stored in the AR object information storage unit 16, information regarding the image of the AR object 36 to be displayed (information such as shape, size, arrangement, etc.) is obtained (S413).

The AR object generation unit 17 analyzes the information regarding the image of the acquired AR object 36 and generates the AR object 36 (S414). When generating an AR object, AR to be displayed by obtaining real space information (three-dimensional data of the real space acquired in advance) stored in the three-dimensional data storage unit 14 and considering the shape of the real space. The arrangement and size of the object 36 are determined.

The AR object superimposing unit 18 superimposes the shooting data (real space image) acquired in the shooting data acquisition process (S412) and the AR object 36 generated in the AR object generation process (S414) (S415). At that time, an occlusion process is performed in which the image (image) portion hidden in the shadow is not displayed according to the positional relationship between the image in the real space seen from the user 10 (HMD1) and the depth direction of the AR object.

The display data output unit 19 outputs the display data to which the AR object superimposition processing (S415) has been performed to the display screen of the HMD 1 (S416).

This completes the video processing in which the real space video (background 31 and person 32 in FIG. 6A) and the AR object 36 (FIG. 6B) are superimposed and displayed as shown in FIG. 6C (S417).

FIG. 9 is a flowchart showing a procedure for processing audio data at the time of superimposed display. This is a process of converting the voice emitted by the person 32 in the real space of FIG. 6A into voice data that can be heard by the user 10 with the AR object 36 of FIG. 6C superimposed, and outputting the voice data.

When the processing of the voice data is started (S431), the incidental information (material, thickness, cross-sectional area, etc. of the components) of the components of the AR object 36 stored in the AR object information storage unit 16 is obtained ( S432).

The voice data acquisition unit 22 determines whether or not the voice input unit 81 has a voice input (S433). As a result of the determination, if there is a voice input, the process proceeds to S434, and the voice data acquisition unit 22 acquires the input voice data. If there is no voice input, wait until there is voice input.

The sound source position specifying unit 23 specifies the sound source position (the position of the mouth 321 of the person 32) (S435). The position of the mouth 321 of the person 32, which is the sound source position, is determined by analyzing the real space arrangement information stored in the three-dimensional data storage unit 14 and the real space image acquired by the shooting data acquisition unit 15. Can be identified.

The AR object shielding determination unit 24 determines whether or not the sound source position is shielded by the AR object. That is, it is determined whether or not the AR object 36 exists on the line segment connecting the sound source (mouth 321 of the person 32) and the user 10 wearing the HMD 1 (S436).

As a result of the determination, if the sound source position is shielded by the AR object, the process proceeds to S437, and if the sound source position is not shielded by the AR object, the process proceeds to S439.

In S437, the component of the AR object 36 that shields the sound source is specified. Then, the acoustic characteristic data corresponding to the component of the AR object 36 is selected from the acoustic characteristic data stored in the acoustic characteristic data storage unit 21. In the case of FIG. 6C, the acoustic characteristic data of the AR object 36 adopts the impulse response value 3 corresponding to the material being metal, the thickness being thin, and the cross-sectional area being large.

The voice data processing unit 25 processes voice data according to the acoustic characteristics (response value 3) selected in S437 (S438).

In S439, the audio data output unit 26 outputs audio data from the audio output unit 81 (headphone 80).

With the above, in the superimposed display as shown in FIG. 6C, the process of converting the voice in the real space into the voice data that can be heard by the user 10 wearing the HMD 1 is completed (S440).

In the above AR object shielding determination process (S436), it was determined whether or not the sound source position 301 was shielded by the AR object 35, assuming the state of FIG. 1B. Further, in order to correspond to the state of FIG. 1D, it is determined whether or not the sound is reflected by the AR object 35'in front of the sound source position 301, and the sound characteristic (reflection characteristic) of the AR object 35'is determined. The audio data processing process (S438) may be performed.

Further, in the above explanation, only the voices emitted in the real space (voices emitted by the persons 32 and 33) are taken up as the sound source, but further, the voices emitted by the AR objects (automobiles) 36 and 37 (for example, engine sound and horn sound) are taken up. ) Can also be processed. In that case, in the voice data output processing (S439), the voice data in the real space and the voice data generated from the AR object are superimposed and the voice is output.

Further, in the audio data processing process (S438), when the AR object which is a sound source is shielded / reflected by an object in real space, or when the AR object which is a sound source is shielded / reflected by another AR object, the above It goes without saying that the processing of the audio data of the AR object is performed based on the acoustic characteristic data of the object to be shielded / reflected, as in the description.

In the above embodiment, the video processing and the audio processing in the superimposed display are performed by the video calculation unit 73 and the audio calculation unit 83, which are hardware elements, but it goes without saying that these can also be realized by software processing.

According to the first embodiment, the superimposed display of the AR object and the audio output in consideration of the arrangement of the AR object are realized, and the user can watch the video and audio without discomfort. Further, since the patterned acoustic characteristic data is used in the voice data processing by the voice calculation unit, the processing time can be shortened and the processing can be performed in real time.

In the first embodiment, when the AR objects are superimposed and displayed, the arrangement and size of the AR objects are determined so that there is no contradiction in the arrangement in the real space. On the other hand, in the second embodiment, the arrangement and size of the AR object are determined first, and the image in the real space is edited and superimposed according to this.

The basic configuration of the HMD in the second embodiment is the same as the configuration described in the first embodiment, but the function for editing the image in the real space is added to the functional block.

FIG. 10 is a diagram showing a functional block configuration of the HMD. A shooting data editing unit 27 and a sound source position moving unit 28 are added to the functional block configuration of FIG. 2B.

The shooting data editing unit 27 edits the shooting data and edits the size and arrangement of the shooting target. That is, the object to be photographed is edited according to the AR object generated by the AR object generation unit 17, and both are superimposed and displayed by the AR object superimposition unit 18.

In the video edited by the shooting data editing unit 27, the sound source position moving unit 28 processes the audio data so as to match the moved sound source position when the sound source moves. The sound source position can be arbitrarily set by adjusting the volume difference between the left and right headphones, the arrival time difference between the left and right headphones, and the like.

11A and 11B are diagrams showing an example of AR object superimposed display. This is a case where the image of the person 34, which is the shooting data in the real space, is edited and superimposed on the AR object 37 (automobile) to display the state in which the person 34 is on the AR object 37 (automobile).

FIG. 11A shows a case where the person 34 and the AR object 37 are superimposed without any image processing. In this case, the person 34 shown by the broken line exists in the front portion (engine portion) of the AR object 37 (automobile), and is in a state that cannot be realized in reality. In the first embodiment (FIG. 7), the arrangement of the AR object 37 (automobile) is changed to realize the state in which the person 34 is on board, but in this embodiment, the arrangement of the person 34 is changed to get on the automobile. Realize the state of being.

FIG. 11B shows a case where the image of the person 34 is edited and superimposed on the AR object 37. In this example, the position of the person 34 is moved in the direction of the arrow 39 and placed at the position of the window 371 of the AR object 37. The person 34'after placement is made to look translucent through the window 371, and the size of the person 34'is also adjusted according to the size of the AR object 37. As described above, in FIG. 11B, the shooting data editing unit 27 edits the shooting data (size and arrangement) of the person 34 existing in the real space, and the image is not unnatural even if it is superimposed on the AR object 37. Has been realized.

In the video processing of FIG. 11B, the position of the person 34 is moved in the direction of the arrow 39 and displayed as the person 34'. At that time, the sound source position of the voice emitted from the person 34 changes from the mouth 341 before the movement to the mouth 341'after the movement. In line with this, the sound source position moving unit 28 processes the voice data so that the voice of the person 34 is emitted from the moved sound source position 341'. As a result, the voice of the person 34'can be provided so as not to make the person feel unnatural.

FIG. 12 is a flowchart showing a video processing procedure at the time of superimposed display in this embodiment. This is based on the flowchart of the first embodiment (FIG. 8), and steps S421 to S422 for editing a real space image are added. Here, only the differences from the flowchart of FIG. 8 will be described.

After performing the process of superimposing the AR object on the shooting data in S415, in the newly added S421, it is determined whether or not there is any unnaturalness in the positional relationship between the shooting data and the AR object in the superimposition display. do. For example, in the superimposed display of FIG. 11A, the person 34 in the real space exists in the front portion (engine portion) of the AR object 37 (automobile), which cannot exist in reality, and is in an unnatural state.

If the result of the judgment is an unnatural state, proceed to S422 and edit the shooting data. If it is not unnatural, the process proceeds to S416 and the display data is output as it is.

In S422, the shooting data is edited by the shooting data editing unit 27. For example, in the case of FIG. 11B, the shooting data of the person 34 is edited, the person 34 is arranged at the position of the window 371 of the AR object 37, and the size of the person 34 is also adjusted according to the size of the AR object 37. After that, it is output as display data in S416.

FIG. 13 is a flowchart showing a procedure for processing audio data at the time of superimposed display. This is based on the flowchart of the first embodiment (FIG. 9), and steps S441 to S443 of voice processing accompanying the movement of the sound source position are added. Here, only the differences from the flowchart of FIG. 9 will be described.

After the sound source position (for example, the mouth 341 of the person 34 in FIG. 11A) is specified by the sound source position specifying unit 23 in S435, whether or not the shooting data is edited by the shooting data editing unit 27 in the newly added S441 is determined. judge. That is, in the video processing flowchart of FIG. 12, it is determined whether or not the shooting data editing process of S422 is executed.

As a result of the determination, if the shooting data is edited, the process proceeds to the added S442, and if the shooting data is not edited, the process proceeds to S436.

In S442, it is determined whether or not it is necessary to move the sound source position with the editing of the shooting data. For example, in editing the shooting data from FIG. 11A to FIG. 11B, it is determined whether or not the sound source position (mouth 341 of the person 34) needs to be moved. If it is necessary to move the sound source position, the process proceeds to S443, and if it is not necessary to move the sound source position, the process proceeds to S436.

In S443, the sound source position moving unit 28 processes the voice data so as to match the moved sound source position.

After that, in S436, the AR object shielding determination unit 24 determines whether or not the sound source position is shielded by the AR object. Of course, the sound source position at this time will be the sound source position after the movement if the sound source position has been moved in S443.

Furthermore, in S437 and later, audio data is processed and output according to the acoustic characteristics of the AR object to be shielded.

As a result, the voice emitted from the mouth 341 of the person 34 in FIG. 11A is output as if it was emitted from the mouth 341'in FIG. 11B and with the influence of the shielding of the window 371 added.

According to the second embodiment, even if the relationship between the real space image and the arrangement and size of the AR object is unnatural, by editing the real space image or processing the real space sound. , The unnaturalness can be resolved and presented.

Further, in the second embodiment, by editing the video in the real space, it is possible to display a display that cannot be realized in reality. For example, it is possible to realistically display the virtual state in which a person in real space is riding in a car floating in the air as an AR object. It is difficult for a person to take such a posture in the real space, but it can be easily realized by editing the image in the real space.

In the third embodiment, a case where the HMD is equipped with two cameras will be described.
FIG. 14 is a diagram showing a state in which the user 10 wears the HMD 1 of this embodiment. Two cameras 711 and 712 are mounted on the left and right ends as the image pickup unit 71 of the HMD 1, and by displaying the real space image taken by the two cameras as a stereo image, the user 10 can use the depth direction of the real space. Will be able to be recognized intuitively. The internal configuration of the HMD 1 and the video / audio processing are the same as in the case of the first embodiment, and duplicate description will be omitted.

Further, although not shown in FIG. 14, the sound field space expands the sound in the real space by mounting two microphones on the left and right ends of the HMD1 for the sound input unit 81 (microphone) as well as the camera. It can be acquired as stereo sound. Then, by outputting the stereo sound from the sound output unit 82 (left and right headphones 821, 822) of the HMD 1, it is possible to provide the user 10 with a sound having a sense of depth.

According to the third embodiment, the user 10 wearing the HMD 1 can intuitively recognize the depth direction of the image in the real space and listen to the sound in the real space as the stereo sound in which the sound field space expands. ..

In the above embodiment, the case where there is one sound source was assumed, but in the fourth embodiment, the case where there are a plurality of sound sources will be described. The basic configuration of the HMD in the fourth embodiment is the same as the configuration described in the first embodiment, but the function for separating a plurality of sound sources is added to the functional block.

FIG. 15 is a diagram showing a functional block configuration of the HMD. A sound source separation unit 29 is added to the functional block configuration of FIG. 2B.

The sound source separation unit 29 analyzes the audio data acquired by the audio data acquisition unit 22, and performs a process of separating each sound source into individual sound sources. As a general method for separating sound sources, there is a method in which a plurality of microphones are installed externally, and audio data at individual positions of the plurality of microphones are detected for phase difference, sound pressure difference, etc., and the position of the sound source is accurately obtained. On the other hand, in this embodiment, as described in the third embodiment, two microphones are mounted on the left and right ends of the HMD 1, and the sound emitted by one sound source reaches the left and right microphones due to the phase difference and the volume difference. Detects the direction of the sound source and simply separates the sound source. Further, the position of the sound source can be specified and separated from the arrangement information in the real space stored in the three-dimensional data storage unit 14 and the image in the real space acquired by the shooting data acquisition unit 15.

FIG. 16 is a flowchart showing a procedure for processing voice data. This is based on the flowchart of the first embodiment (FIG. 9), and steps S445 and S446 accompanying the sound source separation are added. Here, only the differences from the flowchart of FIG. 9 will be described.

After the sound source position is specified by the sound source position specifying unit 23 in S435, a plurality of sound sources are individually separated by the sound source separating unit 29 in the newly added S445. Then, in the next S436 or later, audio data processing is performed on the separated individual sound sources based on the acoustic characteristics of the components of the AR object to be shielded.

When a plurality of sound sources exist, a threshold value is set for the volume (sound pressure), frequency, etc., and the sound data processing of S436 to S438 is performed for the sound source having a low volume below the threshold value and the sound source having a high frequency exceeding the threshold value. It may be possible to narrow down the number of sound sources to be processed by voice processing. In S446, it is determined whether or not the processing has been completed for all the separated sound sources, and if so, the audio data is output in S439.

Needless to say, also in this embodiment, in the voice data output processing (S439), the voice from the AR object (for example, the engine sound of the car or the horn sound) can be superimposed and the voice output can be performed.

According to the fourth embodiment, even when a plurality of sound sources exist, it is possible to output sound for each sound source in consideration of the arrangement of AR objects.

In the fifth embodiment, an HMD system that connects to an external server and obtains information necessary for processing will be described.
FIG. 17 is a diagram showing the overall configuration of the HMD system. The HMD 1 worn by the user 10 is connected to the wireless router 65 and the network network 66 via the LAN communication unit 61 of the communication unit 6. A plurality of servers 67 to 69 are connected to the network network 66.

The first server 67 contains information on the image of the AR object (information on the shape, size, arrangement, etc.), incidental information on the components of the AR object (material, thickness, cross-sectional area, etc. of each component). Information about AR objects is stored and managed. By using the first server 67, it is possible to reduce the load on various data units 42 of the storage unit 4 required by the AR object information storage unit 16.

The second server 68 is stored and managed by patterning acoustic characteristic data with parameters of material, thickness, and cross-sectional area for each material. The material stored and managed by the second server 68 includes not only the material of the component of the AR object but also the acoustic property data regarding the material other than the component of the AR object. By using this second server, it is possible to reduce the load on various data units 42 of the storage unit 4 required by the acoustic characteristic data storage unit 21.

The third server 69 is a server that performs arithmetic processing performed by the video arithmetic unit 73 of the video processing unit 7 and the audio arithmetic unit 83 of the audio processing unit 8 at high speed. By using this third server, it is possible to reduce the load on the AR object generation unit 17, the AR object superimposition unit 18, the sound source position specifying unit 23, and the voice data processing unit 25.

In this embodiment, all the servers 67 to 69 are used in order to reduce the load on the HMD 1, but it goes without saying that the servers can be appropriately selected and used as needed. It should be noted that these servers can be aggregated without being independent of each other.

Although the embodiments of the present invention have been described above with reference to Examples 1 to 5, the configuration for realizing the technique of the present invention is not limited to the above-mentioned Examples, and can be applied to various modified examples. For example, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. That is, when the AR object is superimposed and displayed on the image in the real space by the HMD, it can be applied when at least one of the object in the real space and the AR object becomes a sound source. Then, the audio data from the sound source is processed and output in consideration of the acoustic characteristics (attenuation characteristics or reflection characteristics are used depending on the positional relationship) of the surrounding object or the surrounding AR object.

In the description of each embodiment, a video transmission type HMD is premised, but the terminal used by the user does not have to be in the form of an HMD. For example, the AR object may be displayed by using the camera and the display of the smartphone, and the sound processed by the acoustic processing shown in the present embodiment may be output to the headphones worn by the user.

The functions and the like described in each embodiment may be realized by hardware, for example, by designing a part or all of them by an integrated circuit. Further, it may be realized by software by interpreting and executing a program in which a microprocessor unit or the like realizes each function or the like. Hardware and software may be used together. The software at that time may be stored in the program unit 41 or the like of the HMD in advance at the time of product shipment, or may be acquired from various servers or the like on the Internet after the product is shipped. .. Further, the software provided by a memory card, an optical disk, or the like may be acquired.

In addition, the control lines and information lines shown in the figure indicate what is considered necessary for explanation, and do not necessarily indicate all the control lines and information lines on the product. In practice, it can be considered that almost all configurations are interconnected.

1 ... Head mount display (HMD), 2 ... Main control unit, 4 ... Storage unit, 5 ... Sensor unit, 6 ... Communication processing unit, 7 ... Video processing unit, 8 ... Audio processing unit, 9 ... Operation input unit, 10 ... User, 12 ... 3D sensor information acquisition unit, 13 ... 3D data processing unit, 14 ... 3D data storage unit, 15 ... Shooting data acquisition unit, 16 ... AR object information storage unit, 17 ... AR object generation unit, 18 ... AR object superimposition unit, 19 ... display data output unit, 21 ... acoustic characteristic data storage unit, 22 ... audio data acquisition unit, 23 ... sound source position identification unit, 24 ... AR object shielding determination unit, 25 ... audio data processing unit , 26 ... Audio data output unit, 27 ... Shooting data editing unit, 28 ... Sound source position moving unit, 29 ... Sound source separation unit, 32 to 34 ... People, 35 to 37 ... AR object, 42 ... Various data units, 53 ... 3 Dimensional sensor unit, 67-69 ... server, 70 ... camera, 71 ... imaging unit, 72 ... display unit, 73 ... video calculation unit, 80 ... headphones, 81 ... audio input unit, 82 ... audio output unit, 83 ... audio calculation Department.

Claims (11)

1. In a head-mounted display that the user wears on his head and displays an augmented reality object (hereinafter referred to as AR object).
   An image pickup unit that captures images in real space,
   The AR object generator that creates the AR object, and
   An AR object superimposing unit that superimposes the captured real space image and the generated AR object,
   A display unit that displays the superimposed real-space image and an AR object,
   A voice input unit that inputs voice emitted in real space,
   At least one of the audio data from the real space and the audio emitted by the AR object, depending on the positional relationship between the object existing in the real space arranged by the AR object superimposition unit and the AR object displayed superimposed on the object. With the audio data processing unit that processes the surrounding object or the surrounding AR object in consideration of the acoustic characteristics.
   An audio output unit that outputs audio from the real space processed by the audio data processing unit or audio of an AR object, and
   A head-mounted display characterized by being equipped with.
2. In the head-mounted display according to claim 1,
   It is provided with an AR object shielding determination unit that determines whether or not the sound source position in the real space is shielded by the AR object arranged by the AR object superimposing unit when viewed from the head-mounted display.
   When it is determined by the determination of the AR object shielding determination unit that the sound source position in the real space is shielded, the audio data processing unit adds the acoustic characteristics of the AR object that shields the audio data in the real space. A head-mounted display that is characterized by being processed.
3. In the head-mounted display according to claim 2,
   Each component of the AR object is equipped with an acoustic characteristic data storage unit that stores the acoustic characteristic data related to the acoustic characteristics in a pattern with the shape as a parameter.
   The audio data processing unit is a head-mounted display characterized by selecting acoustic characteristic data of a corresponding AR object from the acoustic characteristic data storage unit and processing audio data in real space.
4. In the head-mounted display according to claim 1,
   A shooting data editing unit that edits the size and arrangement of the shot real space image according to the generated AR object.
   When the sound source position moves in the real space image edited by the shooting data editing unit, the sound source position moving unit that processes the sound data in the real space so as to match the moved sound source position, and the sound source position moving unit.
   A head-mounted display characterized by being equipped with.
5. In the head-mounted display according to claim 1,
   It has a plurality of cameras as the image pickup unit and has a plurality of cameras.
   On the display unit, a real space image taken by the plurality of cameras is displayed as a stereo image, and at the same time, the image is displayed as a stereo image.
   It has a plurality of microphones as the voice input unit, and has a plurality of microphones.
   A head-mounted display characterized by outputting real-space audio input to the plurality of microphones as stereo audio from the audio output unit.
6. In the head-mounted display according to claim 1,
   It has a plurality of microphones as the voice input unit, and has a plurality of microphones.
   A sound source separation unit for specifying the position of each sound source by using the plurality of microphones for a plurality of sounds emitted in the real space is provided.
   The voice data processing unit is a head-mounted display characterized in that it processes voice data for a plurality of voices according to each of the specified sound source positions.
7. In the head-mounted display according to claim 1,
   Connect to at least one external server via the communication unit,
   Obtain information about the AR object generated by the AR object generation unit from the external server, and obtain information about the AR object.
   Alternatively, the acoustic characteristic data of the AR object used for processing the voice data in the voice data processing unit is obtained from the external server.
   Alternatively, a head-mounted display characterized in that at least one of the AR object generation unit, the AR object superimposition unit, and the voice data processing unit is processed by using the external server.
8. In the voice processing method when displaying an augmented reality object (hereinafter referred to as AR object) on a head-mounted display worn by the user on the head.
   According to the video processing step of superimposing the generated AR object on the shot real space video and displaying it,
   A voice input step for inputting voice emitted in real space,
   At least one of the audio data from the real space and the audio emitted by the AR object is generated according to the positional relationship between the object existing in the real space arranged by the video processing step and the AR object displayed superimposed on the object. , Audio data processing steps that take into account the acoustic characteristics of surrounding objects or AR objects, and
   The audio output step that outputs the audio from the processed real space or the audio of the AR object,
   A voice processing method characterized by comprising.
9. In the voice processing method according to claim 8,
   The AR object shielding determination step for determining whether or not the sound source position in the real space is shielded by the AR object arranged by the image processing step when viewed from the head-mounted display is provided.
   When it is determined that the sound source position in the real space is shielded by the determination in the AR object shielding determination step, the audio data in the real space is added to the acoustic characteristics of the shielding AR object in the audio data processing step. A voice processing method characterized by processing.
10. In the voice processing method according to claim 9,
    In advance, for each component of the AR object, the shape is used as a parameter, and the acoustic characteristic data related to the acoustic characteristic is patterned and saved.
    The voice data processing step is a voice processing method characterized in that the acoustic characteristic data of a corresponding AR object is selected from the stored acoustic characteristic data and the voice data in the real space is processed.
11. In the voice processing method according to claim 8,
    In the video processing step, the size and arrangement of the captured real space image are edited according to the generated AR object, and when the edited real space image is accompanied by the movement of the sound source position, the moved sound source position is used. Sound source position movement step that processes real space audio data to match
    A voice processing method characterized by comprising.

Images