JP6201332B2 - Sound processor - Google Patents

Description

  The present invention relates to a sound processing apparatus.

  A head-mounted display device (Head Mounted Display, HMD) that is a display device mounted on the head is known. The head-mounted display device, for example, generates image light representing an image using a liquid crystal display and a light source, and guides the generated image light to a user's eye using a projection optical system or a light guide plate This causes the user to visually recognize the virtual image. There are two types of head-mounted display devices: a transmission type in which the user can visually recognize the outside scene in addition to a virtual image, and a non-transmission type in which the user cannot visually recognize the outside scene. The transmissive head-mounted display device includes an optical transmissive type and a video transmissive type.

  In a transmissive head-mounted display device, a technique is known that allows a user to visually recognize a character image representing the sound by converting the sound. For example, in Patent Document 1, when a user of a head-mounted display device visually recognizes content recorded on a recording medium, the sound included in the content is classified into human voice and environmental sound, and environmental sound is distinguished. A technique is disclosed in which a text image representing the image is generated and the generated text image is visually recognized by a user. Further, Patent Document 2 specifies a choir member whose tempo and pitch of the song are shifted in chorus practice by a choir who sings a song and a conductor who conducts a choir equipped with a head-mounted display device. Thus, a technique for allowing a conductor to visually recognize an image showing a specified choir member is disclosed.

JP 2011-250100 A JP 2010-145878 A

  However, in the technique described in Patent Document 1, the user can adjust the way the external sound is heard, which is different from the sound of the content, by adjusting the volume of the content being viewed by the user's operation. If the volume of the content is not adjusted and the volume of the content is high, it may be difficult to hear external sound. In addition, there is a problem that the user wants to recognize an unexpected external sound that occurs suddenly, such as in an emergency, while the volume of the content is high. In Patent Document 2, the user can recognize a sound source in which a tempo of a song or a pitch difference has occurred, but it cannot adjust the volume of the sound being viewed by the user. The above-described problem is not limited to the head-mounted display device, and is a problem common to sound output devices that output sound.

SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.
A first aspect of the present invention is a sound processing device,
A sound output unit for outputting sound;
A sound acquisition unit for acquiring an external sound that is an external sound different from the output sound output by the sound output unit;
A sound control unit for controlling the output sound based on the external sound;
An image display unit that generates image light representing an image, allows a user to visually recognize the image light, and transmits an outside scene;
A conversion unit that converts the external sound into a character image represented as a character;
A display control unit that causes the image display unit to generate character image light that is the image light representing the character image;
With
The display control unit, with respect to the image display unit, except for a central portion in an image generation possible region, which is a region where the image display unit can generate the image light regardless of the position of the sound source of the external sound The character image light is generated in a part ,
Means for detecting a specific word contained in the external sound,
The sound control unit is a sound processing device that prevents the specific word from being viewed by a user by increasing an output volume when the external word includes the specific word .
A second aspect of the present invention is a sound processing device,
A sound output unit for outputting sound;
A sound acquisition unit for acquiring an external sound that is an external sound different from the output sound output by the sound output unit;
A sound control unit for controlling the output sound based on the external sound;
An image display unit that generates image light representing an image, allows a user to visually recognize the image light, and transmits an outside scene;
A conversion unit that converts the external sound into a character image represented as a character;
A display control unit that causes the image display unit to generate character image light that is the image light representing the character image;
With
The display control unit, with respect to the image display unit, except for a central portion in an image generation possible region, which is a region where the image display unit can generate the image light regardless of the position of the sound source of the external sound The character image light is generated in a part,
Means for detecting a specific word contained in the external sound,
The said conversion part is a sound processing apparatus which does not convert the said specific word into a character image, when the said specific word is contained in the said external sound.
A third aspect of the present invention is a sound processing device,
A sound output unit for outputting sound;
A sound acquisition unit for acquiring an external sound that is an external sound different from the output sound output by the sound output unit;
A sound control unit for controlling the output sound based on the external sound;
An image display unit that generates image light representing an image, allows a user to visually recognize the image light, and transmits an outside scene;
A conversion unit that converts the external sound into a character image represented as a character;
A display control unit that causes the image display unit to generate character image light that is the image light representing the character image;
With
The display control unit, with respect to the image display unit, except for a central portion in an image generation possible region, which is a region where the image display unit can generate the image light regardless of the position of the sound source of the external sound The character image light is generated in a part,
Means for detecting a specific word contained in the external sound,
The display control unit is a sound processing apparatus that does not display the specific word on the image display unit when the specific word is included in the external sound.
The present invention can also be realized as the following forms.

(1) According to one aspect of the present invention, a sound processing apparatus is provided. The sound processing apparatus includes: a sound output unit that outputs a sound; a sound acquisition unit that acquires an external sound that is an external sound different from the output sound output from the sound output unit; and based on the external sound, A sound control unit that controls the output sound. In the sound processing apparatus of this embodiment, the output sound is controlled based on the external sound. Therefore, the user can listen to the external sound while viewing the output sound of the supplied content or the like, and can quickly change the external sound. Can be recognized.

(2) In the sound processing apparatus according to the above aspect, further: an image display unit that generates image light representing an image and allows a user to visually recognize the image light and transmits an outside scene; and the external sound is represented as characters. A conversion unit that converts the character image into a character image; and a display control unit that causes the image display unit to generate character image light that is the image light representing the character image. In the sound processing apparatus of this form, the user can visually recognize the external sound acquired by the sound acquisition unit as a character image representing the external sound, and can make the user recognize the external sound in more detail.

(3) In the sound processing device of the above aspect, the display control unit may set the size of the character image light that the image display unit makes a user visually recognize based on the external sound. According to the sound processing device of this aspect, the character size of the character image is set based on the external sound, so that the user can watch the external sound and also the difference in the volume and frequency of the external sound is detected. The size of the character can be visually recognized, and the external sound can be recognized in more detail.

(4) In the sound processing apparatus according to the above aspect, the display control unit may be configured to have a periphery of the image display unit excluding a central portion in an image generation possible region that is a region where the image display unit can generate the image light. The character image light may be generated in a part. According to the sound processing device of this embodiment, since the character image representing the external sound is not displayed at the central portion in the image generation possible region, the user's visual field is not obstructed more than necessary, and the user can The user can visually recognize the outside scene and the character image, thereby improving the convenience for the user.

(5) In the sound processing device of the above aspect, the sound processing device includes a plurality of the sound acquisition units arranged at different positions; and the sound processing device further includes: a volume of the external sound acquired by the one sound acquisition unit; A sound source direction estimation unit that estimates a sound source direction that is a direction from the image display unit to the sound source of the external sound based on the volume of the external sound acquired by another sound acquisition unit; May cause the image display unit to generate the character image light in the image generation possible region, which is a region in which the image display unit can generate the image light, in association with the sound source direction. According to the sound processing apparatus of this aspect, the user can visually recognize a character image representing the external sound as a character at a position near the sound source, and can make the user recognize the position of the sound source.

(6) In the sound processing apparatus of the above aspect, the sound control unit measures the volume of the external sound, and the volume of the measured external sound has changed from less than a first threshold value to more than the first threshold value. In this case, the volume of the output sound may be reduced. According to the sound processing apparatus of this aspect, even when the volume of output sound such as content is high, the volume of output sound is lowered when the volume of external sound is high, so the volume of warning sound or the like is high. Even when an external sound is acquired, the external sound can be heard more clearly.

(7) In the sound processing device according to the above aspect, the sound processing device further includes: a specific sound detecting unit that detects a specific sound included in the external sound; and the sound control unit is configured to detect the specific sound detected by the specific sound detecting unit. The output sound may be controlled based on the above. According to the sound processing device of this form, when the user detects a specific sound that the user wants to detect, the volume of the output sound is lowered, so that the user can hear only the necessary sound included in the external sound, User convenience is improved.

(8) The sound processing apparatus according to the above aspect further includes: a motion detection unit that detects a change in the motion of the user's head; and the sound control unit is configured to detect the user's head detected by the motion detection unit. The output sound may be controlled based on a change in movement. According to the sound processing apparatus of this embodiment, the volume and frequency of the output sound are adjusted based on two indexes, that is, the change in the external sound and the change in the user's line of sight, so that the output is performed only by the change in the external sound. Compared with the case where the sound volume is adjusted, unnecessary adjustment is suppressed, and the convenience for the user is improved.

(9) In the sound processing device according to the above aspect, the sound control unit causes the sound output unit to output a sound in which the volume of the external sound is set as the output sound based on a change in the external sound. Also good. According to the sound processing device of this aspect, when a specific sound such as a warning sound included in the external sound is detected with a volume lower than the output sound, the sound is used as an output sound having a volume larger than the volume of the external sound. Can be viewed by the user, and the convenience for the user is improved.

(10) In the sound processing device of the above aspect, when the sound acquisition unit acquires a voice of a specific person, the sound control unit changes the frequency of the acquired voice of the specific person, The sound output unit may output the voice of the specific person whose frequency has been changed, and reduce the volume of sound other than the voice of the specific person. According to the sound processing device of this aspect, when a voice having a frequency that is difficult for a person to hear is acquired, the user can easily recognize the external sound because the acquired voice is easily viewed.

(11) In the sound processing device according to the above aspect, the sound control unit measures the volume of the external sound and controls the output sound based on the measured volume of the external sound and the volume of the output sound. May be. According to the sound processing device of this aspect, the volume of the output sound is adjusted based on the volume of the external sound and the volume of the output sound, so that the user can view the output sound at a volume that reflects the user's intention. This can improve the convenience for the user.

(12) In the sound processing device according to the above aspect, the sound output unit may allow the user to view the output sound in a state of being mounted on the user's head. According to the sound processing device of this aspect, since the sound output unit is attached to the user's head, the sound output unit is hardly detached from the user's ear, and the user can stably view the output sound.

(13) In the sound processing device according to the above aspect, the change in the movement of the head is at least one of an angular velocity and an amount of change in the head of the user; The volume of the output sound may be reduced when the change in is greater than or equal to the second threshold. According to the sound processing device of this aspect, the volume of the output sound is adjusted based on two indicators, that is, the change in the external sound and the change in the user's line-of-sight direction. Compared with the case where the adjustment is made, unnecessary adjustment is suppressed, and the convenience for the user is improved.

(14) In the sound processing device according to the above aspect, the sound processing device further includes: an analysis unit that analyzes an intensity for each frequency in the frequency spectrum of the external sound; and the sound control unit has a specific frequency spectrum analyzed by the analysis unit. The volume of the output sound may be controlled based on the frequency. According to this form of the sound processing apparatus, when an abnormal sound or the like having a different frequency is detected in the acquired external sound, the user can be made to recognize the abnormal sound or the like, and the convenience for the user is improved. .

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with another new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including one or more or all of the three elements of a sound output unit, a sound acquisition unit, and a sound control unit. That is, this apparatus may or may not have a sound output unit. The device may or may not have a sound acquisition unit. The device may or may not have a sound control unit. For example, the sound output unit may output sound. The sound acquisition unit may acquire an external sound that is an external sound different from the output sound output from the sound output unit, for example. For example, the sound control unit may control the output sound based on the external sound. Such a device can be realized as a sound processing device, for example, but can also be realized as a device other than the sound processing device. According to such a form, it is possible to solve at least one of various problems such as improvement of convenience for a user who uses the apparatus, integration of the apparatus, and improvement of usability. Any or all of the technical features of each form of the sound processing apparatus described above can be applied to this apparatus.

  The present invention can also be realized in various forms other than the sound processing apparatus. For example, an image display device, a head-mounted display device, a sound processing device control method, an image display device control method, a head-mounted display device control method, a sound processing system, an image display system, a head-mounted image It can be realized in the form of a display system or the like. A computer program for realizing the functions of a sound processing device system, an image display device system, and a head-mounted display device system, a recording medium on which the computer program is recorded, and the computer program including the computer program are embodied in a carrier wave. It can be realized in the form of a data signal or the like.

2 is an explanatory diagram showing an external configuration of a head-mounted display device 100. FIG. 3 is a block diagram functionally showing the configuration of the head-mounted display device 100. FIG. It is explanatory drawing which shows a mode that image light is inject | emitted by the image light production | generation part. It is explanatory drawing which shows the flow of an output volume adjustment process. It is explanatory drawing which shows an example of the visual field VR visually recognized by the user. It is explanatory drawing which shows an example of the change of the volume of the external audio | voice acquired. It is explanatory drawing shown about an example of the frequency of the external audio | voice acquired. It is explanatory drawing which shows an example of visual field VR 'visually recognized by the driver | operator of a motor vehicle.

Next, embodiments of the present invention will be described in the following order based on the embodiments.
A. First embodiment:
A-1. Configuration of head mounted display device:
A-2. Output volume adjustment processing:
B. Variations:

A. First embodiment:
A-1. Configuration of head mounted display device:
FIG. 1 is an explanatory diagram showing an external configuration of the head-mounted display device 100. The head-mounted display device 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The head-mounted display device 100 according to this embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. In this specification, a virtual image visually recognized by the user with the head-mounted display device 100 is also referred to as a “display image” for convenience. Moreover, emitting image light generated based on image data is also referred to as “displaying an image”.

  The head-mounted display device 100 includes an image display unit 20 that allows a user to visually recognize a virtual image when mounted on the user's head, and a control unit 10 (controller 10) that controls the image display unit 20. I have.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. And microphones 62 and 64. Each of the right optical image display unit 26 and the left optical image display unit 28 is disposed so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

  The display driving units 22 and 24 include liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 (see FIG. 2). Details of the configuration of the display driving units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 2) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display driving units 22 and 24 to the eyes of the user. The light control plate is a thin plate-like optical element, and is arranged so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. In addition, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The camera 61 is disposed at a position corresponding to the user's eyebrow when the user wears the image display unit 20. The camera 61 captures an outside scene that is an external scenery in a direction opposite to the user's eye side, and acquires an outside scene image. The camera 61 in the present embodiment is a monocular camera, but may be a stereo camera.

  The right microphone 62 is arranged on the opposite side of the right display drive unit 22 in the right holding unit 21. The left microphone 64 is disposed on the opposite side of the left display driving unit 24 in the left holding unit 23. Since the right microphone 62 and the left microphone 64 are arranged at different positions with respect to the image display unit 20, the external volume that is the volume of the external sound acquired by the right microphone 62 and the external volume that the left microphone 64 acquires are Is different. Note that “speech” in this specification represents the meaning of “sound” in a broad sense including not only human voice but also mechanical sound and the like. The right microphone 62 and the left microphone 64 correspond to the sound acquisition unit in the claims.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connection unit 40 includes a main body cord 48, a right cord 42, a left cord 44, and a connecting member 46 connected to the control unit 10. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24. The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. The right earphone 32 and the left earphone 34 correspond to a sound output unit in the claims.

  The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. A connector (not shown) that fits each other is provided at each of the end of the main body cord 48 opposite to the connecting member 46 and the control unit 10. By fitting / releasing the connector of the main body cord 48 and the connector of the control unit 10, the control unit 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control unit 10 is a device for controlling the head-mounted display device 100. The control unit 10 includes a determination key 11, a lighting unit 12, a display switching key 13, a track pad 14, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. Yes. The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control unit 10. The lighting unit 12 notifies the operation state of the head-mounted display device 100 by its light emission state. Examples of the operating state of the head-mounted display device 100 include power ON / OFF. For example, an LED (Light Emitting Diode) is used as the lighting unit 12. The display switching key 13 detects a pressing operation and outputs a signal for switching the display mode of the content video between 3D and 2D, for example. The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the head-mounted display device 100 by detecting a slide operation of the switch.

  FIG. 2 is a block diagram functionally showing the configuration of the head-mounted display device 100. As shown in FIG. 2, the control unit 10 includes a CPU 140, an operation unit 135, an input information acquisition unit 110, a storage unit 120, a power supply 130, an interface 180, a transmission unit 51 (Tx51), and a transmission unit 52. (Tx52). The operation unit 135 receives an operation by the user and includes an enter key 11, a display switch key 13, a track pad 14, a luminance switch key 15, a direction key 16, a menu key 17, and a power switch 18.

  The input information acquisition unit 110 acquires a signal corresponding to an operation input by the user. As a signal corresponding to the operation input, for example, there is an operation input to the track pad 14, the direction key 16, and the power switch 18. The power supply 130 supplies power to each part of the head-mounted display device 100. As the power supply 130, for example, a secondary battery can be used. The storage unit 120 stores various computer programs. The storage unit 120 is configured by a ROM, a RAM, or the like. The CPU 140 reads out and executes the computer program stored in the storage unit 120, thereby operating the operating system 150 (OS 150), the image processing unit 160, the sound processing unit 170, the conversion unit 185, the direction determination unit 161, and the display control unit. 190.

  The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). Thereafter, the image processing unit 160 stores the converted digital image signal as image data Data (RGB data) of the target image in the DRAM in the storage unit 120 for each frame. Note that the image processing unit 160 may execute image processing such as various tone correction processing such as resolution conversion processing, brightness and saturation adjustment, and keystone correction processing on the image data as necessary. .

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. To do. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control unit 10 and the image display unit 20.

  The sound processing unit 170 performs various processes on the external sound to be acquired and the output sound that is the sound to be output. The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker (not shown) connected to the connecting member 46 ( (Not shown). For example, when the Dolby (registered trademark) system is adopted, processing on the audio signal is performed, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, for example.

  In addition, the sound processing unit 170 specifies the external sound volume acquired by the microphones 62 and 64 and the output sound volume that is the sound volume of the output sound as a numerical value of decibel (dB), or is included in the external sound and stored in the storage unit 120 in advance. A specific sound such as a stored warning sound is detected. The detection of a specific voice included in the external voice is performed by collating the waveform of the frequency spectrum such as a warning sound stored in advance in the storage unit 120 with the waveform of the frequency spectrum of the external voice acquired during a predetermined period. It is done by doing. The voice processing unit 170 can also discriminate between human voices and other voices by collating waveforms of frequency spectra. In addition, the sound processing unit 170 analyzes the frequency spectrum of the acquired external sound and calculates the intensity for each frequency. In the present embodiment, the sound processing unit 170 acquires the external sound volume every second, and in the analysis of the frequency spectrum, analyzes the external sound acquired per second.

  Also, the sound processing unit 170 determines whether or not the external volume acquired at a certain time is equal to or higher than a predetermined volume, or the external volume acquired at a certain time and the external volume acquired one second before that. The output volume is adjusted based on the change. The sound processing unit 170 adjusts the output volume based on a change in the frequency spectrum of the external sound acquired in one second from a certain time point with respect to the frequency spectrum of the external sound acquired in the immediately preceding one second. In addition, the sound processing unit 170 adjusts the output volume based on specific sound included in the detected external sound.

  The sound processing unit 170 estimates the direction of the external sound from the image display unit 20 to the sound source based on the external volume acquired by the right microphone 62 and the external volume acquired by the left microphone 64. Specifically, in this embodiment, when the volume acquired by the right microphone 62 is larger than the volume acquired by the left microphone 64, the audio processing unit 170 estimates that there is a sound source in the right direction with the user at the center. . The voice processing unit 170 corresponds to a sound control unit, an analysis unit, a specific sound detection unit, and a sound source direction estimation unit in the claims.

  The conversion unit 185 converts the sound acquired by the microphones 62 and 64 into a character image in which the sound is represented by characters. The direction determination unit 161 specifies the angular velocity and the amount of change in the angle of the user's head based on the change in the movement of the image display unit 20 detected by the 9-axis sensor 66 described later.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 controls driving of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, and left LCD control unit according to control signals. The left LCD 242 driving ON / OFF by 212, the left backlight 222 driving ON / OFF by the left backlight control unit 202, and the like are individually controlled. Thus, the display control unit 190 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light. The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  Further, the display control unit 190 sets the character size of the character image converted by the direction determination unit 161 based on the external volume acquired by the microphones 62 and 64, and represents the character image in which the character size is set. It transmits to the image display part 20 as a control signal. The image display unit 20 generates image light representing a character image based on the transmitted control signal and emits it to the user's eyes, so that the user can visually recognize the sound as a character image.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control unit 10. Examples of the external device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. A 9-axis sensor 66, a right microphone 62, and a left microphone 64.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of the user's head is detected. Since the orientation of the image display unit 20 is known from the detected movement of the user's head, the direction determination unit 161 can estimate the user's line-of-sight direction. The direction determination unit 161 and the 9-axis sensor 66 correspond to a motion detection unit in claims. The microphones 62 and 64 transmit the acquired audio signal to the conversion unit 185 and the audio processing unit 170.

  The right display driving unit 22 includes a receiving unit 53 (Rx53), a right backlight control unit 201 (right BL control unit 201) and a right backlight 221 (right BL221) that function as a light source, and a right LCD that functions as a display element. A control unit 211, a right LCD 241 and a right projection optical system 251 are included. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as display elements. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimating lens that converts the image light emitted from the right LCD 241 into a light beam in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as “light guide unit”.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes a receiving unit 54 (Rx54), a left backlight control unit 202 (left BL control unit 202) and a left backlight 222 (left BL202) that function as a light source, and a left LCD that functions as a display element. A control unit 212 and a left LCD 242 and a left projection optical system 252 are included. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as display elements. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are also collectively referred to as “image light generation unit”. The left projection optical system 252 is configured by a collimating lens that converts the image light emitted from the left LCD 242 into a light beam in a parallel state. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”.

  FIG. 3 is an explanatory diagram illustrating a state in which image light is emitted by the image light generation unit. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby changing the illumination light IL emitted from the right backlight 221 into an image. Is modulated into an effective image light PL representing The same applies to the left side. As shown in FIG. 4, the backlight system is used in the present embodiment, but it may be configured to emit image light using a front light system or a reflection system.

A-2. Output volume adjustment processing:
FIG. 4 is an explanatory diagram showing the flow of output volume adjustment processing. The output volume adjustment process is a process for adjusting the output volume output by each unit of the control unit 10 via the earphones 32 and 34 based on the change in the external sound acquired by the microphones 62 and 64. First, the microphones 62 and 64 acquire external sound (step S310). The sound processing unit 170 performs sound volume acquisition per second, frequency spectrum analysis, and specific sound detection for the external sound acquired by the microphones 62 and 64.

  Next, the sound processing unit 170 monitors whether a specific sound such as a warning sound is detected in the external sound (step S320). When a specific sound is detected (step S320: YES), the sound processing unit 170 decreases the output volume output from the earphones 32 and 34 (step S360). In this embodiment, the audio processing unit 170 lowers the output volume to a preset volume, but in other embodiments, the volume to be lowered may be set freely by the user, or based on the external volume The volume may be set. Further, it may be set based on the output volume and the external volume.

  When the output volume is lowered, next, the display control unit 190 displays a text image representing the acquired external sound on the image display unit 20 (step S370). FIG. 5 is an explanatory diagram showing an example of the visual field VR visually recognized by the user. FIG. 5 shows a visual field VR visually recognized by the user and a maximum image display area PN that is an area where the image display unit 20 can display an image. As shown in FIG. 5, the user can visually recognize a teacher TE, a plurality of students ST receiving a lecture of the teacher TE, and a text image TX1 representing the acquired external voice as a character image as an outside scene. . The user views the voice acquired by the lecture microphone attached to the teacher TE as output voice from the earphones 32 and 34 via the interface 180, and visually recognizes the character written on the whiteboard WB by the teacher TE. In the present embodiment, in the process of step S330, an announcement sound “ping-pong pampong” flowing before the announcement is detected as a specific sound, and the conversion unit 185 converts the external sound acquired by the microphones 62 and 64 into the external sound. Is converted to a character image representing the character as a character. The display control unit 190 displays the converted character image as the text image TX1 in the maximum image display area PN. The volume of the external audio “Ping Pong Pampong” included in the announcement sound is larger than the volume of the external audio “Now the earthquake has occurred.” Therefore, as shown in FIG. 5, the display control unit 190 increases the size of each character in the character image representing “An earthquake has occurred” rather than the character image representing the sound of “ping-pong pampong”. .

  The announcement sound is output from the speaker SP shown in FIG. 5, and the left microphone 64 is closer to the position of the speaker SP than the right microphone 62, so that the external volume acquired is larger than that of the right microphone 62. Therefore, the audio processing unit 170 determines that the speaker SP, which is a sound source of external audio, is on the left side with respect to the user. In the maximum image display area PN, the display control unit 190 displays the text image TX1 in a peripheral part that excludes the central part and increases in the part that occupies the left part of the center. In the present embodiment, the maximum image display area PN has 960 horizontal pixels and 540 vertical pixels. The maximum image display area PN corresponds to an image generation possible area in claims. The peripheral portion excluding the central portion in the maximum image display region PN refers to a region excluding the region having the horizontal pixel number 320 and the vertical pixel number 180 located in the center, and the horizontal pixel number 480 located in the center. It is more preferable that the region is a region excluding the region of 270 vertical pixels.

  Next, the sound processing unit 170 monitors whether the detected specific sound has returned to the original state where it is not detected (step S380). When the specific sound is not detected (step S380: YES), the sound processing unit 170 returns the output volume reduced in the process of step S360 to the output volume before the decrease (step S390). Thereafter, the control unit 10 monitors an operation for ending the output audio adjustment process (step S400). When the operation for ending the output volume adjustment process is not detected (step S400: NO), the sound processing unit 170 again monitors whether a specific sound such as a warning sound is detected in the external sound (step S320). ). When an operation for ending the output volume adjustment processing is detected (step S400: YES), the control unit 10 hides the text image TX1 displayed in the maximum image display area PN by the display control unit 190, and the control unit 10 Then, the output volume adjustment process is terminated.

  In the process of step S380, when a specific sound is detected continuously (step S380: NO), the control unit 10 monitors an operation for ending the output sound adjustment process (step S410). When the operation for ending the output volume adjustment process is not detected (step S380: NO), the voice processing unit 170 continues to monitor whether the original state where the detected specific voice is not detected is restored (step S380). . When an operation for ending the output volume adjustment process is detected (step S410: YES), the control unit 10 ends the output volume adjustment process.

  If a specific sound is not detected in the process of step 320 (step S320: NO), the sound processing unit 170 monitors whether the acquired external sound volume is changed and the sound volume is equal to or higher than a predetermined threshold. (Step S330). FIG. 6 is an explanatory diagram illustrating an example of a change in the volume of the acquired external sound. FIG. 6 shows a transition Nout (dB) of the volume of the external sound acquired by the microphones 62 and 64 and a predetermined volume threshold Nth (dB). As shown in FIG. 6, the acquired external volume is a volume smaller than the volume threshold Nth until time t1, and is a volume equal to or higher than the volume threshold Nth at time t2 one second after time t1. In this case, the volume changes from the volume N1 at time t1 to the volume N2 at time t2, and the volume N2 is greater than or equal to the volume threshold Nth (step S330 in FIG. 4: YES). Each part of performs the process after step S360.

  In the process of step S330, if the external volume does not change or the volume is smaller than the volume threshold value Nth even if the external volume changes (step S330: NO), the audio processing unit 170 specifies the external audio. It is monitored whether the frequency is detected (step S340). FIG. 7 is an explanatory diagram showing an example of the frequency of the acquired external sound. FIG. 7 shows a frequency spectrum obtained by analyzing the external sound acquired in each of the period T1 and the period T2 after the period T1. As shown in FIG. 7, when a frequency of a specific frequency band fth is detected in the change from the frequency spectrum fT1 in the period T1 to the frequency spectrum fT2 in the period T2 (step S340: YES), the control unit 10 Each unit performs the processing after step S360. Note that it is generally known that sound having a frequency of 3000 hertz (Hz) is easy for humans to hear even when the volume is low. Therefore, in another embodiment, in the case of a sound having a frequency close to 20 Hz or 20 000 Hz out of 20 Hz to 20000 Hz, which is a frequency that humans can hear, the output sound is output even if the external sound has the same volume. It may be small.

  In the process of step S340, when a specific frequency is not detected in the external sound (step S340: NO), the direction determination unit 161 in a state where the external sound is changed and a human voice is detected in the external sound. Monitors the detection of the movement of the user's head (step S350). While the voice processing unit 170 detects a human voice in the external voice, the direction determination unit 161 detects a change amount equal to or greater than a predetermined threshold in at least one of the angular velocity of the head and the change amount of the angle. Then (step S350: YES), each unit of the control unit 10 performs the processing after step S360. When the human voice is not detected in the process of step S350, or when the amount of change in the head movement equal to or greater than the threshold is not detected (step S350: NO), the voice processing unit 170 again converts the voice into an external voice. It is monitored whether a specific sound such as a warning sound is detected (step S320).

  As described above, in the head-mounted display device 100 according to the present embodiment, the audio processing unit 170 outputs audio output from the earphones 32 and 34 based on the change in external audio acquired by the microphones 62 and 64. Adjust the volume. Therefore, in the head-mounted display device 100, the volume of the output sound is reduced when the volume of the external sound or the like changes, so that the user can watch the sound of the content supplied via the interface 180. , Can hear external audio and can recognize external changes quickly.

  Further, in the head-mounted display device 100 according to the present embodiment, the conversion unit 185 converts the sound acquired by the microphones 62 and 64 into a character image in which the sound is represented by characters. The display control unit 190 transmits a control signal representing a character image to the image display unit 20, and image light representing the character image is generated by the image display unit 20. Therefore, in the head-mounted display device 100, the user can visually recognize the external sound acquired by the microphones 62 and 64 as the text image TX1 representing the external sound, and can make the user recognize the external sound in more detail. .

  In the head-mounted display device 100 according to this embodiment, the display control unit 190 sets the character size of the character image based on the external volume acquired by the microphones 62 and 64, and sets the character size. A control signal representing the character image is transmitted to the image display unit 20. Therefore, in the head-mounted display device 100, the character size of the text image TX1 is set based on the volume of the external sound, so that the user can watch the external sound and the difference in the external sound volume is text. Even the character size of the image TX1 can be visually recognized, and the external sound can be recognized in more detail.

  In the head-mounted display device 100 according to the present embodiment, the display control unit 190 includes the text image TX1 in the peripheral portion excluding the central portion in the maximum image display region PN, which is a region where the image display unit 20 can display an image. Is displayed. Therefore, in the head-mounted display device 100, since the character image representing the external voice is not displayed at the central portion in the maximum image display area PN, the user's visual field VR is not unnecessarily disturbed, and the user can In addition, it is possible to visually recognize an outside scene that is transmitted through the image display unit 20 and to visually recognize a character image, thereby improving user convenience.

  In the head-mounted display device 100 according to the present embodiment, the right microphone 62 and the left microphone 64 are arranged on the image display unit 20 at different positions. The sound processing unit 170 estimates the direction of the external sound from the image display unit 20 to the sound source based on the external volume acquired by the right microphone 62 and the external volume acquired by the left microphone 64. The sound processing unit 170 estimates that the speaker SP is on the left side with respect to the user, and displays the text image TX1 in a portion occupying the left portion of the center in the maximum image display region PN. Therefore, in the head-mounted display device 100, in the user's visual field VR, it is possible to make the user recognize the position of the sound source by visually recognizing a character image representing the external sound as a character near the sound source.

  Further, in the head-mounted display device 100 of the present embodiment, the audio processing unit 170 changes the output volume when the acquired volume changes from the volume N1 less than the volume threshold Nth to the volume N2 greater than or equal to the volume threshold Nth. Lower. Therefore, in the head-mounted display device 100, even when the volume of the content sound is high, the output sound volume is lowered when the external sound volume is high. Even if it is, you can hear the external sound more clearly.

  Further, in the head-mounted display device 100 according to the present embodiment, the sound processing unit 170 detects specific sound such as a warning sound included in the acquired external sound, and specifies the specific sound included in the detected external sound. The output sound is adjusted based on the sound. Therefore, in this head-mounted display device 100, when the user detects a specific sound that the user wants to detect, the output volume is lowered, so that the user can hear only the necessary sound included in the external sound, User convenience is improved.

  Further, in the head-mounted display device 100 of the present embodiment, the 9-axis sensor 66 detects the movement of the user's head, and based on the change in the movement of the image display unit 20 detected by the direction determination unit 161. Then, the angular velocity and the amount of change in angle in the user's head are specified. The sound processing unit 170 reduces the output volume based on the human voice included in the external sound and the amount of change equal to or greater than a predetermined threshold in the movement of the user's head. For this reason, in this head-mounted display device 100, the output volume is adjusted based on two indicators, the change in the external sound and the change in the user's line of sight, so the output volume is adjusted only by the change in the external sound. Compared with the case where it is done, unnecessary adjustment is suppressed and the convenience of the user is improved.

  In the head-mounted display device 100 of the present embodiment, the sound processing unit 170 analyzes the frequency spectrum of the acquired external sound, calculates the intensity for each frequency, and calculates the frequency spectrum of the analyzed external sound. Adjust the output audio based on the change. Therefore, in the head-mounted display device 100, when an abnormal sound having a different frequency is detected in the acquired external sound, the user can recognize the abnormal sound or the like, which is convenient for the user. improves.

B. Variations:
In addition, this invention is not limited to the said embodiment, It can implement in a various aspect in the range which does not deviate from the summary, For example, the following deformation | transformation is also possible.

B1. Modification 1:
In the above embodiment, the head-mounted display device 100 including the sound processing unit 170 and the image display unit 20 has been described as an example. However, the device including the sound processing unit 170 and the image display unit 20 is described. Various modifications can be made. FIG. 8 is an explanatory diagram showing an example of a visual field VR ′ visually recognized by the driver of the automobile. In this modified example, as shown in FIG. 8, a head-up display (HUD) formed on a windshield of an automobile is used as an image display device that allows a driver driving the automobile to visually recognize a character image. As audio processing devices, car audio mounted on a car and microphones for acquiring external sound attached to the front, rear, left and right of the car are used.

  FIG. 8 shows a maximum image display area PN and a text image TX2 in which the acquired external sound is displayed as a character image in the maximum image display area PN. The driver is listening to music with the car audio installed in the car. The sound source of the external sound acquired by the microphone is an ambulance (not shown) estimated to be left front from the vehicle based on the sound volume acquired by the plurality of microphones. In this modified example, when the external sound volume changes from the state where the external sound volume is less than the sound volume threshold Nth to the sound volume emitted by the ambulance ("Peeper per po" shown in FIG. 8) or more than the sound volume threshold value Nth, the sound for adjusting the volume of the car audio The processing unit 170 ′ decreases the volume of the music generated by the car audio. In the maximum image display area PN, the display control unit 190 displays a text image TX2 that is a character image representing a sound emitted by an ambulance. The text image TX2 is displayed in the upper left peripheral part in the maximum image display area PN. Note that the text image TX2 may not be displayed, and only the volume generated by the car audio may be reduced. Therefore, in this modified example, when there is some abnormality outside the automobile, it is easy for the driver of the automobile to recognize the abnormality, and the convenience of the driver is improved.

  Moreover, in the said embodiment, although it was an aspect provided with the image display part which can display the acquired external audio | voice as a character image, an image display part is not an essential structural requirement, and can change variously about the aspect of an audio processing apparatus. It is. For example, the audio processing device may be a portable music playback device that can play music content. When a user of a portable music playback device views the output sound of music content with a sealed earphone, external sound is acquired by a microphone formed in the portable music playback device, and the change of the external sound Based on this, the volume of the output sound may be set.

  In the above embodiment, the audio processing unit 170 performs the adjustment for decreasing the output sound based on the acquired change in the external sound, but may perform the adjustment for increasing the output sound. For example, when the user can view both the output sound and the external sound, the sound processing unit 170 adjusts to increase or decrease the output sound volume based on the difference between the external sound volume and the output sound volume. May be. In this modification, since the output volume is adjusted based on the difference between the external volume and the output volume, the user can view the output sound at a volume that reflects the user's intention, which is convenient for the user. Improves.

B2. Modification 2:
In the above embodiment, the volume of the output sound is adjusted based on the change of the external sound. However, various modifications can be made to the adjustment of the output sound volume. For example, the sound processing unit 170 may set the volume of the external sound acquired by the microphones 62 and 64 and output it from the earphones 32 and 34 as output sound. In this modification, when a specific sound such as a warning sound included in the external sound is detected with a volume lower than the output sound, the user can view the output sound with a volume larger than the external volume. , User convenience is improved.

B3. Modification 3:
In the above embodiment, the display control unit 190 sets the position for displaying the text image TX1 in the maximum image display area PN in association with the direction from the image display unit 20 to the external sound source. The position where the text image TX1 is displayed in the display area PN can be variously modified. For example, the head-mounted display device 100 may be configured such that only one microphone is arranged and the direction of the external sound source is not specified. Even if the direction of the external sound source can be specified, the display control unit 190 may set the position where the text image TX1 is displayed in the maximum image display area PN regardless of the direction of the external sound source.

  Also, various changes can be made to the size of each character of the character image displayed in the maximum image display area PN and the method of setting the size. The display control unit 190 may display each character in the character image in the maximum image display area PN with a predetermined size regardless of the external volume. The size of each character in the character image displayed in the maximum image display area PN may be set based on the frequency, the type of sound, and the amount of change in the movement of the user's head. In addition, the font of the character may be changed regardless of the size of the character.

B4. Modification 4:
In the above embodiment, the character image representing the external sound displayed in the maximum image display area PN is displayed in a predetermined manner. However, the setting of the displayed character image can be variously modified. When the user is listening to the output sound, the character image representing the external sound may be displayed as a character image different from that before the operation is performed by performing a predetermined operation on the operation unit 135.

  In the above embodiment, the microphones 62 and 64 are arranged at positions near both ears when the user wears the image display unit 20. However, the arrangement and structure of the microphones 62 and 64 are not limited thereto. Various modifications are possible. For example, the microphones 62 and 64 may be microphones having a mechanical structure that can change the orientation with respect to the image display unit 20 and having directivity. When the direction from the image display unit 20 to the sound source of the external sound can be estimated, the sound processing unit 170 can acquire the external sound in more detail by changing the direction of the microphones 62 and 64, and the external sound can be acquired. It becomes easy to detect the change of.

  In the above embodiment, the sound processing unit 170 detects a change in the external sound based on the frequency of the external sound. Therefore, the detection of a specific sound included in the external sound can be variously modified. For example, even if the voice is an external voice including a plurality of voices, the voice processing unit 170 may recognize and identify each voice. In this case, the sound processing unit 170 may be configured to adjust the external sound volume when a predetermined specific voice of one person is detected.

  In the above embodiment, the specific sound included in the external sound is detected and the output sound volume is adjusted. However, the specific sound may be detected and the output sound volume may be adjusted. For example, from the viewpoint of privacy protection, a specific word may not be viewed by the user by increasing the output volume. In addition, by registering specific words such as personal names in advance, the registered words may not be converted into character images even when voice is acquired. The same applies to the aspect in which the acquired external sound is converted as a character image representing the external sound, and a specific word may not be displayed on the image display unit 20 as a character image.

B5. Modification 5:
Moreover, in the said embodiment, although the output audio | voice was controlled based on the change of an external audio | voice, an output audio | voice may be controlled based on an external audio | voice. For example, it is in a state in which almost no external sound is always acquired, and control such as decreasing the volume of the output sound may be performed even if there is no change in the volume or frequency of the external sound. In this modified example, since the output sound is controlled even if the acquired external sound does not change, the user can listen to the output sound having an appropriate volume and frequency.

  Further, when the camera 61 acquires a predetermined human voice, the audio processing unit 170 changes the frequency of the acquired human voice to the output voice of the earphones 32 and 34. A mode in which a human voice is output and an output volume other than the voice of the person whose frequency is changed may be reduced. In this modified example, when a voice having a frequency that is difficult for a person to hear is acquired, the user can easily recognize the external voice in order to easily view the acquired voice. Further, when external sounds having the same frequency are acquired from a plurality of sound sources, the frequencies of the external sounds acquired from several sound sources are changed, and the external sounds with the changed frequencies are output from the earphones 32 and 34. May be. In general, when a person acquires a plurality of sounds having frequencies close to each other, it is difficult to hear a sound having a low frequency due to frequency masking. Therefore, in this modification, it is easy to make the user recognize the sound acquired from a plurality of sound sources by suppressing the influence of frequency masking.

B6. Modification 6:
As the image display unit, instead of the image display unit 20 worn like glasses, another type of image display unit such as an image display unit worn like a hat may be adopted. Moreover, in the said embodiment, although LCD and a light source are utilized as a structure which produces | generates image light, it replaces with these and you may employ | adopt other display elements, such as an organic EL display. In the above embodiment, the 9-axis sensor 66 is used as a sensor for detecting the movement of the user's head. Instead, one or two of an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor are used. You may use the sensor comprised from. In the above embodiment, the head-mounted display device 100 is a binocular optical transmission type. However, the present invention can be applied to other types of head-mounted display such as a video transmission type and a monocular type. The same applies to the apparatus.

  In the above embodiment, the head-mounted display device 100 may guide image light representing the same image to the left and right eyes of the user so that the user can visually recognize the two-dimensional image. It is also possible to guide the user to visually recognize a three-dimensional image by guiding image light representing a different image to his eyes.

  In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. For example, in the above-described embodiment, the image processing unit 160 and the sound processing unit 170 are realized by the CPU 140 reading and executing a computer program, but these functional units may be realized by a hardware circuit. Good.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

  Moreover, in the said embodiment, as shown in FIG. 1 and FIG. 2, the control part 10 and the image display part 20 are formed as a separate structure, However, about the structure of the control part 10 and the image display part 20, about. However, the present invention is not limited to this, and various modifications are possible. For example, all of the components formed in the control unit 10 may be formed inside the image display unit 20 or a part thereof may be formed. Further, among the configurations formed in the control unit 10, only the operation unit 135 may be formed as a single user interface (UI), or the power source 130 in the above embodiment is formed independently and can be replaced. It may be. Further, the configuration formed in the control unit 10 may be formed in the image display unit 20 in an overlapping manner. For example, the CPU 140 shown in FIG. 2 may be formed in both the control unit 10 and the image display unit 20, or a function performed by the CPU 140 formed in the control unit 10 and the CPU formed in the image display unit 20. May be configured separately.

  In the above embodiment, the voice processing unit 170 and the conversion unit 185 are formed in the control unit 10, but the present invention can also be realized by a voice processing system configured independently as another device. . For example, output audio control and audio processing may be performed by cloud computing, a server, a host computer, or a smartphone via a wireless communication line.

  In the above embodiment, the microphones 62 and 64 are arranged on the image display unit 20 and the earphones 32 and 34 are connected to the control unit 10. However, the arrangement of the microphones 62 and 64 and the earphones 32 and 34 is described. Various modifications are possible. For example, the earphones 32 and 34 may be connected to the head-mounted display device 100 like a head-mounted audio processing device mounted on the head. Further, as the sound output device, a speaker may be used instead of the earphones 32 and 34. Also, the microphones 62 and 64 may be sound acquisition devices provided at specific locations in the facility without being arranged in the image display unit 20. The sound output device may be a hearing aid, for example, attached to the head. In this modification, since it is mounted on the user's head, the sound output device is unlikely to come off from the user's ear, and the user can stably view the output sound.

B7. Modification 7:
For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image generation unit may use LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. In the case of the laser retinal projection type, the “image generation possible region” can be defined as an image region recognized by the user's eyes.

  Further, for example, the head-mounted display may be a head-mounted display in which the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit does not completely cover the user's eye. Good. The head mounted display may be a so-called monocular type head mounted display.

  Further, the earphone may be an ear-hook type or a headband type, or may be omitted. Further, for example, it may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet.

  The present invention is not limited to the above-described embodiments and modifications, and can be realized with various configurations without departing from the spirit of the present invention. For example, the technical features in the embodiments and the modifications corresponding to the technical features in each form described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10 ... Control part 11 ... Decision key 12 ... Illumination part 13 ... Display switching key 14 ... Trackpad 15 ... Luminance switching key 16 ... Direction key 17 ... Menu key 18 ... Power switch 20 ... Image display part 21 ... Right holding part 22 ... Right display drive unit 23 ... Left holding unit 24 ... Left display drive unit 26 ... Right optical image display unit 28 ... Left optical image display unit 30 ... Earphone plug 32 ... Right earphone (sound output unit)
34 ... Left earphone (sound output part)
DESCRIPTION OF SYMBOLS 40 ... Connection part 42 ... Right cord 44 ... Left cord 46 ... Connecting member 48 ... Main body code 51 ... Transmission part 52 ... Transmission part 53 ... Reception part 54 ... Reception part 61 ... Camera 62 ... Right microphone (sound acquisition part)
64 ... Left microphone (sound acquisition unit)
66 ... 9-axis sensor 100 ... Head-mounted display device 110 ... Input information acquisition unit 120 ... Storage unit 130 ... Power source 135 ... Operation unit 140 ... CPU
DESCRIPTION OF SYMBOLS 150 ... Operating system 160 ... Image processing part 161 ... Direction determination part 170 ... Sound processing part (Sound control part, analysis part, specific sound detection part, sound source direction estimation part)
180 ... Interface 185 ... Conversion unit 190 ... Display control unit 201 ... Right backlight control unit 202 ... Left backlight control unit 211 ... Right LCD control unit 212 ... Left LCD control unit 221 ... Right backlight 222 ... Left backlight 241 ... Right LCD
242 ... Left LCD
251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate Nout ... Transition of volume t1, t2 ... Time T1, T2 ... Time period N1, N2 ... Volume
Nth ... Volume threshold (first threshold)
fT1, fT2 ... frequency spectrum fth ... frequency band TX1, TX2 ... text image (character image)
OA ... External device WB ... White board RE ... Right eye LE ... Left eye IL ... Illumination light PL ... Image light PN ... Maximum image display area (image generation possible area)
AP ... tip SP ... speaker VR ... field of view

Claims (11)

A sound processing device,
A sound output unit for outputting sound;
A sound acquisition unit for acquiring an external sound that is an external sound different from the output sound output by the sound output unit;
A sound control unit for controlling the output sound based on the external sound;
An image display unit that generates image light representing an image, allows a user to visually recognize the image light, and transmits an outside scene;
A conversion unit that converts the external sound into a character image represented as a character;
A display control unit that causes the image display unit to generate character image light that is the image light representing the character image;
With
The display control unit, with respect to the image display unit, except for a central portion in an image generation possible region, which is a region where the image display unit can generate the image light regardless of the position of the sound source of the external sound The character image light is generated in a part ,
Means for detecting a specific word contained in the external sound,
The sound control unit is a sound processing device that prevents the specific word from being viewed by a user by increasing an output volume when the specific word is included in the external sound . A sound processing device,
A sound output unit for outputting sound;
A sound acquisition unit for acquiring an external sound that is an external sound different from the output sound output by the sound output unit;
A sound control unit for controlling the output sound based on the external sound;
An image display unit that generates image light representing an image, allows a user to visually recognize the image light, and transmits an outside scene;
A conversion unit that converts the external sound into a character image represented as a character;
A display control unit that causes the image display unit to generate character image light that is the image light representing the character image;
With
The display control unit, with respect to the image display unit, except for a central portion in an image generation possible region, which is a region where the image display unit can generate the image light regardless of the position of the sound source of the external sound The character image light is generated in a part ,
Means for detecting a specific word contained in the external sound,
The said conversion part is a sound processing apparatus which does not convert the said specific word into a character image, when the said specific word is contained in the said external sound . A sound processing device,
A sound output unit for outputting sound;
A sound acquisition unit for acquiring an external sound that is an external sound different from the output sound output by the sound output unit;
A sound control unit for controlling the output sound based on the external sound;
An image display unit that generates image light representing an image, allows a user to visually recognize the image light, and transmits an outside scene;
A conversion unit that converts the external sound into a character image represented as a character;
A display control unit that causes the image display unit to generate character image light that is the image light representing the character image;
With
The display control unit, with respect to the image display unit, except for a central portion in an image generation possible region, which is a region where the image display unit can generate the image light regardless of the position of the sound source of the external sound The character image light is generated in a part ,
Means for detecting a specific word contained in the external sound,
The display processing unit is a sound processing device that does not display the specific word on the image display unit when the specific word is included in the external sound . The sound processing device according to any one of claims 1 to 3 , wherein
The said display control part is a sound processing apparatus which sets the magnitude | size of the said character image light which the said image display part makes a user visually recognize based on the said external sound. The sound processing device according to any one of claims 1 to 4 ,
A plurality of the sound acquisition units at different positions;
The sound processing device further includes:
A sound source that is in the direction from the image display unit to the sound source of the external sound, based on the volume of the external sound acquired by one sound acquisition unit and the volume of the external sound acquired by another sound acquisition unit A sound source direction estimation unit for estimating the direction,
The display control unit causes the image display unit to generate the character image light in the peripheral portion in association with the sound source direction. A sound processing device according to any one of claims 1 to 5 ,
The sound control unit measures the volume of the external sound, and reduces the volume of the output sound when the measured volume of the external sound changes from less than a first threshold value to more than the first threshold value. , Sound processing device. The sound processing device according to any one of claims 1 to 6 , further comprising:
A specific sound detection unit for detecting a specific sound included in the external sound;
The sound processing unit is a sound processing device that controls the output sound based on the specific sound detected by the specific sound detection unit. The sound processing apparatus according to any one of claims 1 to 7 , further comprising:
It has a motion detector that detects changes in the movement of the user's head,
The said sound control part is a sound processing apparatus which controls the said output sound based on the change of the motion of the user's head detected by the said motion detection part. The sound processing apparatus according to any one of claims 1 to 8 ,
The sound control unit causes the sound output unit to output, as the output sound, a sound in which a volume of the external sound is set based on a change in the external sound. The sound processing apparatus according to any one of claims 1 to 9 ,
When the sound acquisition unit acquires a voice of a specific person, the sound control unit changes the frequency of the acquired voice of the specific person and changes the frequency to the sound output unit. A sound processing apparatus for outputting a voice of a specific person and reducing a volume of a sound other than the voice of the specific person. The sound processing device according to any one of claims 1 to 10 ,
The sound output unit is a sound processing device that allows a user to view the output sound in a state of being mounted on a user's head.

Images