JP6467922B2 - Head-mounted display device, head-mounted display device control method, information system, and computer program - Google Patents

Description

  The present invention relates to a head-mounted display device.

  A head-mounted display device (Head Mounted Display, HMD) that is a display device mounted on the head is known. The head-mounted display device is used, for example, by generating image light using a liquid crystal display and a light source, and guiding the generated image light to a user's eye using a projection optical system or a light guide plate Make the person visually recognize the virtual image. There are two types of head-mounted display devices: a transmission type in which the user can visually recognize the outside scene in addition to a virtual image, and a non-transmission type in which the user cannot visually recognize the outside scene. The transmissive head-mounted display device includes an optical transmissive type and a video transmissive type. An information processing apparatus such as a head-mounted display apparatus may be equipped with a microphone that acquires external sound.

  For example, in Patent Document 1, when a user of a head-mounted display device visually recognizes content recorded on a recording medium, the sound included in the content is classified into human voice and environmental sound, and environmental sound is distinguished. A technique is disclosed in which a text image representing the image is generated and the generated text image is visually recognized by a user. Further, Patent Document 2 specifies the direction of a sound source when a user of the head-mounted display device is used as a reference, and displays the sound acquired from the sound source as a character image in association with the direction of the sound source. A part-mounted display device is disclosed.

JP 2011-250100 A JP 2014-120963 A

  However, in the technique described in Patent Document 1, the user can adjust the way the external sound is heard, which is different from the sound of the content, by adjusting the volume of the content being viewed by the user's operation. If the volume of the content is not adjusted and the volume of the content is high, it may be difficult to hear external sound. In the technique described in Patent Document 2, the direction of the sound source and the sound emitted from the sound source are associated with each other and visually recognized by the user as a character image. There was a request to make a user visually recognize a character image. In addition, in the conventional head-mounted display device, improvement in usability has been desired.

SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.
One embodiment of the present invention provides a transmissive head-mounted display device. The head-mounted display device includes a sound acquisition unit that acquires sound, a sound source specifying unit that specifies the sound source using identification information of the sound source that generates sound, and a conversion that converts the acquired sound into a character image. An image display unit capable of displaying an image, an image display unit capable of displaying an image and transmitting an outside scene, a corresponding image associated with the identified sound source, and the converted character image a display image setting unit to be displayed on, Ru comprising a. The sound source specifying unit determines whether or not the speaker's mouth as the sound source is moving, and when the speaker's mouth is moving, the conversion unit converts the speech into a character image, and When the person's mouth is not moving, the conversion unit does not convert the sound into a character image.

(1) According to one aspect of the present invention, a transmissive head-mounted display device is provided. According to the head-mounted display device of this aspect, a sound acquisition unit that acquires sound; a sound source specifying unit that specifies the sound source using identification information of a sound source that emits sound; and a character image obtained from the acquired sound An image display unit capable of displaying an image and transmitting an outside scene; a corresponding image associated with the identified sound source and the converted character image in association with each other A display image setting unit to be displayed on the image display unit. According to the head-mounted display device of this form, the user can not only listen to the external sound but also recognize the sound as visual information as a character image, which improves convenience for the user. Further, since the character image is displayed on the image display unit together with the identification information of the sound source, the convenience for the user is further improved.

(2) In the head-mounted display device according to the above aspect, the sound acquisition unit specifies a direction of the sound source; and the display image setting unit associates the character image with the corresponding image in association with the specified direction. May be displayed on the image display unit. According to the head-mounted display device of this aspect, the display image setting unit can display the character image and the corresponding image on the image display unit in association with the specified sound source, and the user can display the character image. Can be recognized in association with the sound source.

(3) In the head-mounted display device according to the above aspect, when there are a plurality of the sound sources, the display image setting unit, based on the identification information set for each of the sound sources, You may set at least one of the display mode and display position in the said image display part of at least one with an image. According to the head-mounted display device of this aspect, even when the sound of the plurality of sound sources is acquired by the sound acquisition unit, the sound generated by the plurality of sound sources is associated with each sound source and the character image is displayed to the user. Therefore, the convenience for the user is further improved.

(4) In the head-mounted display device of the above aspect, when there are a plurality of the sound sources, the identification information set for each of the plurality of sound sources, the respective sounds of the plurality of sound sources, May be provided. According to the head-mounted display device of this embodiment, since the sound acquired by the sound acquisition unit is simply stored, the sound and the sound source that emits the sound are distinguished and stored. Convenience is improved.

(5) In the head-mounted display device according to the above aspect, the display image setting unit converts the size of the character image displayed on the image display unit and the size of the character image converted based on the acquired sound volume. At least one of whether or not to display the character image on the display unit may be determined. According to the head-mounted display device of this aspect, since there is a relationship between the acquired sound volume and the character image displayed on the image display unit, the user can recognize external sound as a character image by associating it more. , User convenience is further improved.

(6) In the head-mounted display device according to the above aspect, further; an imaging unit that images an outside scene; and when the sound source is a human, the human being that is the sound source based on the captured image of the outside scene A line-of-sight detection unit that detects a line-of-sight direction; and the display image setting unit is configured to display at least one of the corresponding image and the character image on the image display unit based on the detected line-of-sight direction And / or display position may be set. According to this form of the head-mounted display device, the character image visually recognized by the user differs depending on whether the person who is the sound source is speaking to the user or not, and the person who is the sound source is the user. The content of the voice can be recognized in association with

(7) The head-mounted display device of the above aspect further includes: an acceleration detection unit that detects acceleration of the image display unit; and the display image setting unit corresponds to the detected acceleration of the image display unit. The display position of at least one of the corresponding image and the character image displayed on the image display unit may be fixed. According to the head-mounted display device of this aspect, even if the user of the head-mounted display device is walking, for example, the position of the image displayed on the image display unit 20 is not blurred. It is easier for the person to visually recognize the corresponding image and the character image.

(8) In the head-mounted display device according to the above aspect, further stores: a communication unit that transmits / receives information to / from another device; the acquired sound; and the identification information of the sound source that emits the acquired sound An acquired voice storage unit; and the display image setting unit may transmit and receive the stored voice and the identification information to another device via the communication unit. According to the head-mounted display device of this aspect, information such as a sound source and sound generated by the sound source can be shared as a whole system including the head-mounted display device.

(9) The head-mounted display device according to the above aspect further includes: a position specifying unit that specifies a position of the image display unit; and the communication unit from another device to another head-mounted display device. The display image setting unit may set the corresponding image based on the identified position of the image display unit and the received position information and display the corresponding image on the image display unit; Good. According to this form of the head-mounted display device, the positional relationship between the user and the user of the other head-mounted display device is indicated to the user by the corresponding image that the user of the head-mounted display device visually recognizes. The user's convenience is further improved.

(10) In the head-mounted display device according to the above aspect, the sound source specifying unit specifies the language of the acquired voice; the display image setting unit is configured to select the corresponding image based on the specified voice language. You may set at least one of the display mode and display position in the said image display part of at least one of the said character images. According to the head-mounted display device of this aspect, the character image displayed on the image display unit and the corresponding image change depending on the language specified by the sound source specifying unit, and thus the difference in the specified language is used as visual information. The user can be made aware and the convenience of the user is improved.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with another new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, an aspect of the present invention is an apparatus including one or more of five elements of a sound acquisition unit, a sound source identification unit, a conversion unit, an image display unit, and a display image setting unit. It is feasible. That is, this apparatus may or may not have a voice acquisition unit. In addition, the device may or may not have a sound source specifying unit. Moreover, the apparatus may or may not have the conversion unit. Further, the apparatus may or may not have an image display unit. The apparatus may or may not have a display image setting unit. For example, the sound source acquisition unit may acquire sound. The sound source specifying unit may specify the sound source using, for example, identification information of a sound source that emits sound. For example, the conversion unit may convert the acquired voice into a character image. For example, the image display unit may be capable of displaying an image and transmitting an outside scene. For example, the display image setting unit may display the corresponding image associated with the identified sound source and the converted character image on the image display unit in association with each other. Such a device can be realized as, for example, a head-mounted display device, but can also be realized as a device other than the head-mounted display device. According to such a form, it is possible to solve at least one of various problems such as improvement and simplification of the operability of the device, integration of the device, and improvement of convenience of the user who uses the device. it can. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device.

  The present invention can also be realized in various forms other than the head-mounted display device. For example, a control method for a head-mounted display device, an information system having a head-mounted display device, a control method for a head-mounted display device, a computer program for realizing the information system, and a record recording the computer program It can be realized in the form of a medium and a data signal including the computer program and embodied in a carrier wave.

It is explanatory drawing which shows the external appearance structure of a head mounting type display apparatus (HMD). It is a block diagram which shows the structure of HMD functionally. It is the schematic which shows an example of the identification information memorize | stored in the identification information storage part. It is explanatory drawing which shows a mode that image light is inject | emitted by the image light production | generation part. It is a flowchart of a character image display process. It is explanatory drawing which shows the visual field which a user visually recognizes when a corresponding | compatible image is displayed on the image display maximum area | region. It is explanatory drawing which shows the visual field which a user visually recognizes when the character image into which the audio | voice was converted is displayed on the image display maximum area | region. It is explanatory drawing which shows the visual field which a user visually recognizes when the character image into which the audio | voice in the modification was converted is displayed on the image display maximum area | region. It is explanatory drawing which shows the one part external appearance structure of the information system containing HMD of a modification. It is explanatory drawing which shows the external appearance structure of HMD in a modification.

A. Embodiment:
A-1. Configuration of head mounted display device:
FIG. 1 is an explanatory diagram showing an external configuration of a head-mounted display device 100 (HMD 100). The HMD 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The HMD 100 according to the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time visually recognize an outside scene. In this specification, a virtual image visually recognized by the user with the HMD 100 is also referred to as a “display image” for convenience.

  The HMD 100 includes an image display unit 20 that allows a user to visually recognize a virtual image while being mounted on the user's head, and a control unit 10 (controller 10) that controls the image display unit 20.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. And a microphone 63. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

  The display driving units 22 and 24 include liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 (see FIG. 2). Details of the configuration of the display driving units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 2) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display driving units 22 and 24 to the eyes of the user. The light control plate is a thin plate-like optical element, and is arranged so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. In addition, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The camera 61 is a stereo camera disposed on each of the end ER of the right optical image display unit 26 and the EL of the left optical image display unit 28. Although details will be described later, when an image of a QR code that is determined to be the same as the QR code (registered trademark) stored in advance in the control unit 10 is detected from a captured image captured by the camera 61, it is determined that the QR code is owned. The identified human is identified as the sound source of the speaker.

  The microphone 63 acquires external sound. The microphone 63 is disposed on the opposite side (outside) of the right display driving unit 22 when facing the user in the right display driving unit 22 when the user wears the image display unit 20. The voice acquired by the microphone 63 is subjected to various processes by the voice processing unit 170 of the control unit 10 to be described later.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connection unit 40 includes a main body cord 48, a right cord 42, a left cord 44, and a connecting member 46 that are connected to the control unit 10. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24. The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30.

  The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. A connector (not shown) that fits each other is provided at each of the end of the main body cord 48 opposite to the connecting member 46 and the control unit 10. By fitting / releasing the connector of the main body cord 48 and the connector of the control unit 10, the control unit 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control unit 10 is a device for controlling the HMD 100. The control unit 10 includes a determination key 11, a lighting unit 12, a display switching key 13, a track pad 14, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. Yes. The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control unit 10. The lighting unit 12 notifies the operation state of the HMD 100 according to the light emission state. The operation state of the HMD 100 includes, for example, power ON / OFF. For example, an LED is used as the lighting unit 12. The display switching key 13 detects a pressing operation and outputs a signal for switching the display mode of the content video between 3D and 2D, for example. The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the HMD 100 by detecting a slide operation of the switch.

  FIG. 2 is a block diagram functionally showing the configuration of the HMD 100. As shown in FIG. 2, the control unit 10 includes a storage unit 120, a power source 130, an operation unit 135, a wireless communication unit 132, a CPU 140, an interface 180, a transmission unit 51 (Tx51), and a transmission unit 52 ( Tx52). The operation unit 135 receives an operation by the user and includes an enter key 11, a display switch key 13, a track pad 14, a luminance switch key 15, a direction key 16, a menu key 17, and a power switch 18.

  The power supply 130 supplies power to each part of the HMD 100. As the power supply 130, for example, a secondary battery can be used. The wireless communication unit 132 performs wireless communication with other devices such as a content server, a television, and a personal computer in accordance with a predetermined wireless communication standard such as a wireless LAN or Bluetooth.

  The storage unit 120 includes a ROM that stores computer programs, a RAM that is used when the CPU 140 executes writing and reading of various computer programs, an audio recording unit 122, and an identification information storage unit 124. . The sound recording unit 122 stores the acquired sound as data when control for recording the external sound acquired by the microphone 63 is performed. The identification information storage unit 124 stores sound source identification information for specifying a sound source. In the present embodiment, the identification information storage unit 124 stores a name of a person who is a speaker as a sound source as a QR code as identification information stored in advance. The voice recording unit 122 corresponds to the acquired voice storage unit in the claims.

  FIG. 3 is a schematic diagram illustrating an example of identification information stored in the identification information storage unit 124. As shown in FIG. 3, the identification information storage unit 124 stores personal information associated with each QR code serving as identification information. The stored personal information includes the name of the speaker as the sound source, the gender, the affiliation, and the telephone number that can be contacted to the speaker for each code (Figure) as a QR code that is identification information. , An e-mail address from which the speaker can receive e-mails, and a face photo.

  The CPU 140 reads a computer program stored in the ROM of the storage unit 120 and writes and reads the computer program in the RAM of the storage unit 120, whereby the operating system 150 (OS 150), the display control unit 190, the audio processing unit 170, It functions as an image processing unit 160, a line-of-sight direction specifying unit 168, a sound source identification unit 161, a conversion unit 169, and an image setting unit 165.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 controls driving of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, and left LCD control unit according to control signals. The left LCD 242 driving ON / OFF by 212, the left backlight 222 driving ON / OFF by the left backlight control unit 202, and the like are individually controlled. Thus, the display control unit 190 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light. Note that the generation of image light by the image display unit 20 is also referred to as “displaying an image”.

  The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The image processing unit 160 acquires an image signal included in the content, and transmits the acquired image signal to the reception units 53 and 54 of the image display unit 20 via the transmission units 51 and 52. Note that the image processing unit 160 may execute image processing such as various tone correction processing such as resolution conversion processing, brightness and saturation adjustment, and keystone correction processing on the image data as necessary. .

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker (not shown) connected to the connecting member 46 ( (Not shown). For example, when the Dolby (registered trademark) system is adopted, processing on the audio signal is performed, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, for example. The audio processing unit 170 performs various processes on the external audio acquired by the microphone 63. The audio processing unit 170 transmits the acquired audio to the conversion unit 169 as an audio signal. The voice processing unit 170 corresponds to the voice identification unit in the claims, and the voice processing unit 170 and the microphone 63 correspond to the voice acquisition unit in the claims.

  The conversion unit 169 analyzes the voice waveform of the voice signal transmitted from the voice processing unit 170 and recognizes the voice, and then converts it into a character image corresponding to the transmitted voice signal. The conversion unit 169 can convert a character image by setting various fonts and sizes of the character image for each sound source described later.

  The sound source identification unit 161 detects a QR code image determined to be the same as the QR code stored in the identification information storage unit 124 from the captured image of the camera 61. The sound source identification unit 161 detects a QR code stored in the identification information storage unit 124 from the captured image by performing pattern matching or statistical identification on the captured image. The sound source identification unit 161 performs pattern matching on the captured image of the camera 61 using the face photograph stored in the identification information storage unit 124 associated with the detected QR code, and identifies the speaker by image recognition. To do. The sound source identification unit 161 specifies the speaker's human mouth as a sound source by performing pattern matching or the like on the image of the speaker in the captured image after specifying the speaker. Note that even if a QR code image corresponding to the QR code stored in the identification information storage unit 124 is detected from the captured image, the speaker's mouth is outside the imaging range, and the sound source may not be specified. In that case, in another embodiment, the position of the QR code detected from the captured image may be specified as the sound source. The sound source identifying unit 161 and the camera 61 correspond to a sound source specifying unit in claims.

  The line-of-sight direction specifying unit 168 specifies the line-of-sight direction of the speaker specified by the sound source identification unit 161 by performing pattern matching or the like on the captured image of the camera 61. In the present embodiment, the line-of-sight direction specifying unit 168 specifies the line-of-sight direction of the speaker from the image of the speaker's eye in the captured image, but in other embodiments, the line-of-sight direction is specified by the image of the speaker's head. The direction of the speaker's head may be specified as the line-of-sight direction. The line-of-sight direction specifying unit 168 and the camera 61 correspond to the line-of-sight direction detecting unit in the claims.

  The image setting unit 165 causes the image display unit 20 to display the name associated with the identified sound source and stored in the identification information storage unit 124 and the character image obtained by converting the acquired voice. The angle of view of the camera 61 is set in advance so as to be the same as the field of view when the line of sight of the user of the HMD 100 faces the front. The image setting unit 165 causes the image display unit 20 to display the character image obtained by converting the acquired external sound at the position of the sound source specified from the captured image and the name of the specified sound source as a character image. . The image setting unit 165 corresponds to the display image setting unit in the claims.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control unit 10. Examples of the external device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. And a microphone 63.

  The right display driving unit 22 includes a receiving unit 53 (Rx53), a right backlight control unit 201 (right BL control unit 201) and a right backlight 221 (right BL221) that function as a light source, and a right LCD that functions as a display element. A control unit 211, a right LCD 241 and a right projection optical system 251 are included. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as display elements. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as “light guide unit”.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes a receiving unit 54 (Rx54), a left backlight control unit 202 (left BL control unit 202) and a left backlight 222 (left BL222) that function as a light source, and a left LCD that functions as a display element. A control unit 212 and a left LCD 242 and a left projection optical system 252 are included. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as display elements. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are also collectively referred to as “image light generation unit”. The left projection optical system 252 is configured by a collimating lens that converts the image light emitted from the left LCD 242 into a light beam in a parallel state. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”.

  FIG. 4 is an explanatory diagram illustrating a state in which image light is emitted by the image light generation unit. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby changing the illumination light IL emitted from the right backlight 221 into an image. Is modulated into an effective image light PL representing The same applies to the left side. As shown in FIG. 4, the backlight system is adopted in the present embodiment, but it may be configured to emit image light using a front light system or a reflection system.

A-2. Character image display processing:
FIG. 5 is a flowchart of the character image display process. In the character image display process, the CPU 140 converts the voice uttered by the speaker as the sound source identified from the captured image into a character image, and the converted character image and the story stored in the identification information storage unit 124. An identification image corresponding to the identification information of the person is displayed on the image display unit 20.

  In the character image display process, first, the CPU 140 waits for reception of a predetermined operation in which the operation unit 135 starts the character image display process (step S11). When determining that the operation unit 135 has not received a predetermined operation (step S11: NO), the CPU 140 continues to wait for reception of the predetermined operation (step S11). When it is determined that the operation unit 135 has received a predetermined operation (step S11: YES), the CPU 140 images the outside scene using the camera 61 (step S13).

  The sound source identification unit 161 detects an image of a QR code determined to be the same as the QR code stored in the identification information storage unit 124 as identification information from the captured image of the camera 61 (step S15). When the sound source identification unit 161 does not detect the QR code image stored in the identification information storage unit 124 from the captured image (step S15: NO), the sound source identification unit 161 continues to detect the QR code image from the captured image. (Step S15). When the sound source identification unit 161 detects an image of the QR code stored in the identification information storage unit 124 from the captured image (step S15: YES), the sound source identification unit 161 is associated with the detected QR code and stored in the identification information storage unit 124. An image determined to be the same as the face photograph included in the stored personal information is specified from the captured images of the camera 61 (step S17). The image setting unit 165 is a character image representing a name included in personal information in the vicinity of a face photograph detected from a captured image in an image display maximum region PN, which is the maximum region in which the image display unit 20 can display an image. Is displayed as a corresponding image. In the present embodiment, if the sound source identification unit 161 cannot detect an image that is determined to be the same as the face photograph corresponding to the QR code, it is treated as having not detected the QR code, but in other embodiments, The QR code may be specified in place of the face photo.

  FIG. 6 is an explanatory diagram showing the visual field VR visually recognized by the user when the corresponding image is displayed in the image display maximum area PN. As shown in FIG. 6, the user visually recognizes the outside scene SC transmitted through the optical image display units 26 and 28 and the image IMG1 as the corresponding image displayed in the maximum image display area PN. The outside scene SC includes a teacher TE giving a lecture at school, a plurality of students ST listening to the lecture, and characters written on the white boat WB by the teacher TE. An ID card on which identification information QR1 (QR code) of the teacher TE is printed is attached to the chest part of the clothes of the teacher TE. The sound source identification unit 161 detects the identification information QR1 from the captured image, and the image setting unit 165 uses the personal information corresponding to the identification information QR1 stored in the identification information storage unit 124 as the name IMG1 as the name IMG1. Display in the maximum display area PN. The image setting unit 165 displays the image IMG1 in the vicinity of the mouth of the teacher TE who is a speaker in the maximum image display area PN.

  When the image IMG1 that is a corresponding image is displayed in the maximum image display area PN (step S19 in FIG. 5), the CPU 140 performs an operation of whether or not to record the sound acquired by the microphone 63 via the operation unit 135. Accept (step S21). When the CPU 140 receives a predetermined operation (step S21: YES), the voice of the teacher TE acquired by the voice processing unit 170 and the microphone 63 is associated with the identification information as data in the identification information storage unit 124. Recording is performed (step S23). When the voice is stored in the identification information storage unit 124 (step S23), the sound source identification unit 161 determines whether or not the mouth of the teacher TE that is the sound source is moving (step S25). Similarly, when a predetermined operation for storing voice is not accepted in the process of step S21 (step S25: NO), the sound source identification unit 161 determines whether the mouth of the teacher TE that is a sound source is moving. Is determined (step S25). When it is determined that the mouth of the teacher TE is not moving (step S25: NO), the conversion unit 169 determines that the speech is not the speech of the teacher TE that is the sound source even if the microphone 63 acquires the voice. The sound source identification unit 161 waits for detection of a state where the mouth of the teacher TE has moved without converting the voice into a character image (step S25). When it is determined that the mouth of the teacher TE, which is a sound source, is moving (step S25: YES), the conversion unit 169 converts the voice acquired by the microphone 63 into a character image (step S27).

  Next, the line-of-sight direction specifying unit 168 specifies the line-of-sight direction of the teacher TE in the captured image (step S29). The image setting unit 165 displays the character image in the display mode of the character image and the image display maximum area PN according to the specified line-of-sight direction when converting the speech of the teacher TE, which is a sound source, into a character image. A position is set (step S31).

  FIG. 7 is an explanatory diagram showing the visual field VR visually recognized by the user when the character image TX1 into which the voice has been converted is displayed in the image display maximum area PN. FIG. 7 shows a character image TX1 displayed in the image display maximum region PN when the line-of-sight direction of the teacher TE, which is a sound source, faces the user of the HMD 100. Since the line-of-sight direction of the identified teacher TE is directed to the user, the image setting unit 165 converts the character image TX1 obtained by converting the teacher TE's utterance “Start a lecture. The image is displayed near the center of the image display maximum area PN. Further, in the present embodiment, the image setting unit 165 makes the image display maximum region PN in proportion to the size of the voice (for example, decibel (dB)) acquired by the microphone 63 and the font size of the converted character image. To display. Therefore, in the character image TX1, the font of the character image obtained by converting “Lecture starts” obtained as a large sound is the character obtained by converting “For 1” as a small sound. Larger than the image font. The portion close to the center of the image display maximum region PN in the present embodiment refers to the left portion when the position of the teacher TE, which is the sound source in the captured image, is on the right side, and to the upper side. Means the lower part.

  When the character image TX1 is displayed in the image display maximum area PN (step S29 in FIG. 5), the operation unit 135 waits for reception of a predetermined operation for ending the character image display process (step S31). When the predetermined operation is accepted (step S31: YES), the CPU 140 ends the character image display process.

  If the predetermined operation is not accepted in the process of step S31 (step S31: NO), the sound source identification unit 161 determines whether or not the position of the teacher TE that is the identified sound source has changed (step S31). S33). In the present embodiment, the sound source identification unit 161 determines that the position of the teacher TE has changed when the position of the ID card that is the identification information QR1 of the predetermined teacher TE has changed in the captured image. If it is determined in step S33 that the position of the teacher TE that is the sound source has not been changed (step S33: NO), the CPU 140 repeats the processing from step S25. If it is determined in step S33 that the position of the teacher TE, which is a sound source, has changed, the CPU 140 repeats the processes in and after step S15.

  As described above, in the HMD 100 of the present embodiment, the sound source identification unit 161 detects the identification information QR1 from the captured image of the camera 61. The image setting unit 165 displays the name image IMG1 associated with the detected identification information QR1 and the character image TX1 into which the voice is converted by the conversion unit 169 in the image display maximum region PN of the image display unit. . Therefore, in the HMD 100 of the present embodiment, the user can not only listen to the external sound but also recognize the sound as the visual information as the character image TX1, thereby improving convenience for the user. In particular, when the user's ears are inconvenient, the user can be made to recognize voice as a character image, which is highly effective. Further, since the character image TX1 is displayed on the image display unit 20 together with the identification information of the speaker as the sound source, the convenience for the user is further improved.

  In the HMD 100 of the present embodiment, the sound source identification unit 161 identifies the direction from the user to the sound source by detecting the identification information QR1 from the captured image of the camera 61. Therefore, the image setting unit 165 can display the character image TX1 and the image IMG1 that is the corresponding image on the image display unit 20 in association with the specified sound source, and the user can display the character image TX1 using the sound source. Recognized in association.

  In the HMD 100 of the present embodiment, the voice recording unit 122 stores the voice of the teacher TE acquired by the voice processing unit 170 and the microphone 63 in association with the identification information. Therefore, compared with the case where the voice acquired by the microphone 63 is simply stored, the HMD 100 according to the present embodiment stores the voice and the speaker separately, so that the convenience for the user is improved.

  Further, in the HMD 100 of the present embodiment, the image setting unit 165 causes the image display unit 20 to display the audio volume acquired by the microphone 63 and the font size of the converted character image in proportion to each other. For this reason, in the HMD 100 of the present embodiment, since there is a relationship between the acquired sound volume and the character image displayed on the image display unit 20, the user can recognize external sound as a character image and recognize it. The convenience is further improved.

  In the HMD 100 of the present embodiment, the image setting unit 165 is displayed on the image display unit 20 according to the identified line-of-sight direction of the teacher TE when converting the speech of the teacher TE, which is a sound source, into a character image. Set the character image display position. Therefore, in the HMD 100 of the present embodiment, the character image visually recognized by the user differs depending on whether the speaker as the sound source is speaking to the user or not, and the intention of the speaker is associated with the user. The content of the utterance can be recognized.

B. Variation:
In addition, this invention is not limited to the said embodiment, It can implement in a various aspect in the range which does not deviate from the summary, For example, the following deformation | transformation is also possible.

B-1. Modification 1:
In the above embodiment, the teacher TE, which is one sound source, has been described. However, the present invention can also be applied to the case where sound is acquired from a plurality of sound sources. For example, the sound source identification unit 161 detects an image of a QR code stored in the identification information storage unit 124 as two different identification information from the captured image, and uses the two detected QR codes as an ID card on the chest. When the two speaker's mouths are identified, the image setting unit 165 may display the character image in the image display maximum area PN using the acquired speech as the speech of the speaker whose mouth is moving. Good. In this case, the image setting unit 165 displays the display position of the character image displayed in the image display maximum area PN in association with the positions of the two mouths specified, and the character image for each speaker. The font color may be set to different colors. That is, the image setting unit 165 sets the display mode and display position of the character image displayed in the image display maximum area PN for each detected voice of the speaker having different identification information. In this modification, even when the voices of a plurality of speakers are acquired by the microphone 63, the voices of the plurality of speakers are associated with the respective speakers and the user can visually recognize the character image. The convenience is further improved.

  In the above embodiment, the sound source identifying unit 161 identifies the identification information and the speaker's mouth as the sound source from the captured image of the camera 61 by image recognition such as pattern matching. The identification is not limited to this, and various modifications are possible. For example, the microphone 63 is a directional microphone, and the sound source identification unit 161 may specify the direction from the user to the sound source by specifying the direction of the acquired sound. In addition, the sound source identification unit 161 may specify a sound source by transmitting and receiving the location information of the speaker using a GPS module or the like. The sound source identification unit 161 may transmit and receive identification information by communicating with the HMD 100 and other devices worn by the speaker via the wireless communication unit 132. The wireless communication unit 132 corresponds to the communication unit in the claims.

  Further, as the identification of the acquired voice, for example, the language (Japanese or English) which is the type of voice may be identified, and the speaker and the voice by the speaker may be associated with each other. In this way, when sound as a plurality of types of languages is acquired by a plurality of speakers, the sound source identification unit 161 translates a plurality of types of languages as specific languages, and the image setting unit 165 May be displayed in the image display maximum area PN as a translated character image of a specific language. In this HMD 100, the character image displayed in the image display maximum area PN changes depending on the language specified by the sound source identification unit 161. Therefore, the character image translated into the language that is most easily understood by the user can be visually recognized and used. User convenience is improved. In addition, when the operation unit 135 is operated or sound related to the operation is acquired, the sound source identification unit 161 selects a specific language to be translated according to the operation, and the image setting unit 165 selects a plurality of languages. The translated character images in a plurality of languages may be displayed in the image display maximum area PN. The image setting unit 165 may display a character image translated into a specific language corresponding to the identification information of the user of the HMD 100 in the image display maximum area PN. Further, the image setting unit 165 may set not only the presence / absence of translation but also the position of the character image to be displayed in the image display maximum region PN according to the language specified by the sound source identification unit 161. For example, the conversion unit 169 does not perform translation according to the type of language, and the image setting unit 165 determines that the acquired voice is in the native language (for example, Japanese) of the user of the HMD 100. Display in a small font character image around the center of the image display maximum area PN, and if it is a language other than the native language, display it in a large font character image near the center of the image display maximum area PN. Also good. In addition, after the conversion unit 169 converts only a predetermined language (for example, English) speech into the native language, the image setting unit 165 displays only the converted character image of the acquired speech. The maximum display area PN may be displayed.

  In the above embodiment, the QR code has been described as an example of the identification information, but the identification information can be variously modified. For example, as a marker type such as QR code, other than QR code, barcode, AR marker, ID marker, NyID marker, DataMatrix marker, frame marker, division marker, add color to these identifiers to increase the amount of information Markers created by multi-dimensional codes, characters that can be read by OCR, or the like may be used. The identification information need not be provided in advance. For example, a speaker identified by the camera 61 of the HMD 100 may be imaged, and authentication based on the captured image data may be used instead of the identification information.

  FIG. 8 is an explanatory diagram showing a visual field VR visually recognized by the user when the character image TX2 into which the voice in the modified example is converted is displayed in the image display maximum area PN. FIG. 8 shows a character image TX2 displayed in the image display maximum area PN when the line-of-sight direction of the teacher TE, which is a sound source, does not face the user of the HMD 100 with respect to the visual field VR shown in FIG. ing. As shown in FIG. 8, the teacher TE is looking not at the user but at the white boat WB. In this case, the image setting unit 165 displays the voice of the teacher TE on the lower side in the maximum image display area PN. Let In addition, when the teacher TE is not looking at the user, the image setting unit 165 displays the character image of the same size font regardless of the volume of the acquired voice of the teacher TE. Display in the area PN. Note that the font of the character image and the display position of the character image that are changed according to the line-of-sight direction of the teacher TE, which is the sound source, can be variously modified. Regardless of the line of sight of the teacher TE, the CPU 140 can change the font size and color settings of the character image displayed in the image display maximum area PN when the operation unit 135 receives a predetermined operation.

B-2. Modification 2:
FIG. 9 is an explanatory diagram showing an external configuration of a part of an information system 500 including a modified HMD 100a. The information system 500 includes an HMD 100a and a server 300 that transmits / receives various information to / from the HMD 100a. In this modification, the server 300 includes an identification information storage unit 124, a conversion unit 169, and an image setting unit 165 of the storage unit 120 included in the HMD 100 of the above embodiment, instead of the HMD 100 of the above embodiment. The point provided is different from the above embodiment. Therefore, the information system 500 includes an identification information storage unit 324 that stores identification information, a CPU 340, and an information transmission / reception unit 332. The CPU 340 includes a conversion unit 369 that converts sound into a character image, and an image setting unit 365 that sets an image to be displayed on the image display unit 20a of the HMD 100a.

  In this modification, when the camera 61 of the HMD 100a captures an outside scene as in the above embodiment, information of the captured image is transmitted to the information transmitting / receiving unit 332 of the server 300 via the wireless communication unit 132 of the HMD 100a. The CPU 340 collates the identification information stored in the identification information storage unit 324, and transmits the collated information to the wireless communication unit 132 of the HMD 100a via the information transmission / reception unit 332, so that the sound source identification unit 161 selects the sound source. Identify. When the microphone 63 of the HMD 100 a acquires sound, the CPU 140 a transmits information on the sound source image and sound captured by the camera 61 to the server 300 via the wireless communication unit 132. The image setting unit 365 sets the display mode of the image to be displayed on the image display unit 20a of the HMD 100a based on the sound source image and sound received via the information transmitting / receiving unit 332, and sets the display mode of the set image. Information is transmitted to the HMD 100a via the information transmission / reception unit 332. The CPU 140a of the HMD 100a displays an image in the image display maximum area PN of the image display unit 20a based on the image display mode information received via the wireless communication unit 132.

  In this modification, a part of the functions of the HMD 100 in the above embodiment is mounted on the server 300. Therefore, the information system 500 as a whole can share information such as a sound source and sound generated by the sound source. Of the functions of the above-described embodiment, the functions mounted on the server 300 included in the information system 500 can be variously modified. For example, a camera 61 and a microphone 63 may be installed at the venue. In addition, a plurality of HMDs 100 and servers 300 included in the information system 500 may have the same function.

B-3. Modification 3:
In the above embodiment, the line-of-sight direction specifying unit 168 specifies the direction of the speaker by performing pattern matching on the captured image of the camera 61. However, the method for specifying the direction of the speaker can be variously modified. It is. For example, the line-of-sight direction specifying unit 168 specifies the position of a speaker carrying a device capable of transmitting and receiving information by using communication such as a GPS module or a LAN, and a gyro sensor or the like is mounted on the device carried by the speaker. Thus, the relationship between the position and orientation of the user of HMD 100 and the speaker may be specified. In addition, the line-of-sight direction specifying unit 168 specifies the position of the other HMD 100 on which the GPS module is mounted and the position of the HMD 100 on which the GPS module is mounted via the wireless communication unit 132, thereby You may specify the relative positional relationship with a user's position. The image setting unit 165 may set the corresponding image associated with the identification information of the other HMD 100 according to the specified relative positional relationship, and display the corresponding image set in the image display maximum area PN. . As the corresponding image set according to the relative positional relationship, for example, when the distance between the user and another user is a predetermined value or more, the image setting unit 165 causes the user to recognize the other user. In addition, an image or name of another user's face may be displayed as the corresponding image in the image display maximum area PN. When a gyro sensor or the like for specifying the orientation of the HMD 100 is mounted on the image display unit 20, the image setting unit 165 sets the image display maximum area PN in accordance with the orientation of the user and the orientation of other users. A corresponding image to be displayed may be set. For example, when it is determined that another user is near the user but there is another user behind the user, the image setting unit 165 causes the other user to move away by a predetermined value or more. As described above, the face image or name may be displayed on the corresponding image in the maximum image display area PN as described above. In this modification, the user can recognize the positional relationship between the user and other users by the corresponding image that is visually recognized by the user, and the convenience for the user is further improved.

  The setting of the display position at which the character image converted by the image setting unit 165 is displayed in the maximum image display area PN is not limited to the above-described embodiment, and various modifications can be made. The image setting unit 165 recognizes a character image described on the whiteboard WB, for example, by using pattern matching or the like on the captured image, and a character image at a position that does not overlap with the recognized character image. TX1 or the like may be displayed. The image setting unit 165 may display a character image at a position of the image display maximum region PN that does not overlap with a preset image regardless of the character image, or a specific image is included in the captured image. If included, the character image may not be displayed at all in the image display maximum area PN. In addition, as a pre-specified word, for example, a name related to privacy or a sentence with confidentiality duties is registered, so that the image setting unit 165 does not register even if it acquires the voice of the registered word. It is not necessary to display only the written words as character images in the image display maximum area PN. Conversely, when the sound of a specific word is acquired, the image setting unit 165 may display the specific word in the maximum image display area PN so as to emphasize the specific word. Further, the image setting unit 165 sets whether or not to display the character image of the acquired voice and the display mode by setting the security level between the HMD user and the speaker of the sound source registered in the identification information storage unit 124. May be displayed in the maximum image display area PN.

  The image setting unit 165 may set a character image to be displayed in the image display maximum area PN in association with the measured distance by measuring the distance to the speaker. For example, the image setting unit 165 may change the font size of the character image displayed in the maximum image display area PN according to the distance from the speaker. In addition, when the QR code or the speaker is not clearly recognized from the captured image of the camera 61 due to poor visibility, the sound source identification unit 161 identifies, for example, an optical beacon transmitted and received as optical information as identification information, and talks by infrared By specifying the distance to the person, the image setting unit 165 may display the acquired voice as a character image in the image display maximum area PN.

B-4. Modification 4:
Further, the HMD 100 may include an acceleration sensor that detects acceleration of the image display unit 20 and a gyro sensor that detects angular velocity. In this case, the image setting unit 165 may display the corresponding image or the character image in the image display maximum area PN so as to cancel the detected acceleration or angular velocity. That is, the image setting unit 165 can fix the display position of the corresponding image or the character image and display it in the image display maximum area PN using the detected acceleration or angular velocity. In this modified example, the image displayed on the image display unit 20 is fixed with respect to the captured image of the outside scene visually recognized by the user, so even if the user is walking, the image display maximum area PN Since the position of the displayed image is not blurred, it is easier for the user to visually recognize the corresponding image and the character image.

B-5. Modification 5:
In the above embodiment, the operation unit 135 is formed in the control unit 10, but the mode of the operation unit 135 can be variously modified. For example, a mode in which a user interface that is the operation unit 135 is provided separately from the control unit 10 may be used. In this case, the operation unit 135 is separate from the control unit 10 in which the power supply 130 and the like are formed. Therefore, the operation unit 135 can be reduced in size and user operability is improved. Further, although the camera 61 is disposed in the image display unit 20, the camera 61 may be configured separately from the image display unit 20 and can capture the outside scene SC. Alternatively, the HMD 100 in which the CPU 140 and the power supply 130 that are included in the control unit 10 are all mounted in the image display unit 20 may be used. The HMD 100 can be further downsized because there is no controller configured separately from the image display unit 20. Further, by mounting the CPU 140 on each of the control unit 10 and the image display unit 20, the control unit 10 may be used as a single controller and the image display unit 20 may be used as a single display device.

  For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image light generation unit may use LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like instead of the LCD. Further, for example, the present invention can be applied to the laser retinal projection type HMD 100.

  Further, for example, the HMD 100 may be a head mounted display in which the optical image display unit covers only a part of the user's eyes, in other words, the optical image display unit does not completely cover the user's eyes. The HMD 100 may be a so-called monocular type head mounted display. Although the HMD 100 is a binocular optical transmission type, the present invention can be similarly applied to other types of head-mounted display devices such as a video transmission type.

  Moreover, HMD100 may be used with the display apparatus only for displaying the image based on the image signal received from the other apparatus. Specifically, it is used as a display device corresponding to a desktop PC monitor. For example, by receiving an image signal from a desktop PC, an image is displayed in the maximum image display area PN of the image display unit 20. May be.

  Moreover, HMD100 may be used so that it may function as a part of system. For example, the HMD 100 may be used as a device for executing a part of functions of a system including an aircraft, and the system in which the HMD 100 is used is not limited to a system including an aircraft, but is a system including an automobile or a bicycle. There may be.

  The earphone may be an ear-hook type or a headband type, or may be omitted. Further, for example, it may be configured as a head-mounted display device mounted on a vehicle such as an automobile or an airplane. For example, it may be configured as a head-mounted display device built in a body protective device such as a helmet.

B-6. Modification 6:
The configuration of the HMD 100 in the above embodiment is merely an example and can be variously modified. For example, the direction key 16 provided in the control unit 10 may be omitted, or another operation interface such as an operation stick may be provided in addition to the direction key 16 and the track pad 14. Moreover, the control part 10 is a structure which can connect input devices, such as a keyboard and a mouse | mouth, and is good also as what receives an input from a keyboard or a mouse | mouth.

  As the image display unit, instead of the image display unit 20 worn like glasses, another type of image display unit such as an image display unit worn like a hat may be adopted. Further, the earphones 32 and 34 can be omitted as appropriate.

  FIG. 10 is an explanatory diagram showing an external configuration of the HMD in the modification. 10A, the difference from the HMD 100 shown in FIG. 1 is that the image display unit 20x includes a right optical image display unit 26x instead of the right optical image display unit 26, and the left optical image. It is a point provided with the left optical image display part 28x instead of the display part 28. The right optical image display unit 26x is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely above the right eye of the user when the HMD 100x is mounted. Similarly, the left optical image display unit 28x is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely above the left eye of the user when the HMD 100x is worn. 10B, the difference from the HMD 100 shown in FIG. 1 is that the image display unit 20y includes a right optical image display unit 26y instead of the right optical image display unit 26, and the left optical image. It is a point provided with the left optical image display part 28y instead of the display part 28. The right optical image display unit 26y is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely below the right eye of the user when the head mounted display is mounted. The left optical image display unit 28y is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely below the left eye of the user when the head mounted display is mounted. Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the user's eyes. The size of the optical member forming the optical image display unit is also arbitrary, and the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit completely covers the user's eye. It can be realized as an HMD 100 in an uncovered mode.

  In the above-described embodiment, the HMD 100 may guide the image light representing the same image to the left and right eyes of the user to make the user visually recognize the two-dimensional image, or may display different images on the left and right eyes of the user. The image light to be represented may be guided to make the user visually recognize the three-dimensional image.

  In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. For example, in the above-described embodiment, the image processing unit 160 and the sound processing unit 170 are realized by the CPU 140 reading and executing a computer program, but these functional units may be realized by a hardware circuit. Good.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

  Moreover, in the said embodiment, as shown to FIG. 1 and FIG. 4, although the control part 10 and the image display part 20 are formed as a separate structure, about the structure of the control part 10 and the image display part 20, about. However, the present invention is not limited to this, and various modifications are possible. For example, all of the components formed in the control unit 10 may be formed inside the image display unit 20 or a part thereof may be formed. In addition, the power source 130 in the above embodiment may be formed independently and replaceable, or the configuration formed in the control unit 10 may be overlapped and formed in the image display unit 20. For example, the CPU 140 shown in FIG. 2 may be formed in both the control unit 10 and the image display unit 20, or a function performed by the CPU 140 formed in the control unit 10 and the CPU formed in the image display unit 20. May be configured separately.

  The present invention is not limited to the above-described embodiments and modifications, and can be realized with various configurations without departing from the spirit of the present invention. For example, the technical features in the embodiments and the modifications corresponding to the technical features in each form described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10 ... Control part 11 ... Decision key 12 ... Illumination part 13 ... Display switch key 14 ... Track pad 15 ... Luminance switch key 16 ... Direction key 17 ... Menu key 18 ... Power switch 20 ... Image display part (image display part)
DESCRIPTION OF SYMBOLS 21 ... Right holding part 22 ... Right display drive part 23 ... Left holding part 24 ... Left display drive part 26 ... Right optical image display part 28 ... Left optical image display part 30 ... Earphone plug 32 ... Right earphone 34 ... Left earphone 40 ... Connection unit 42 ... Right cord 44 ... Left cord 46 ... Connecting member 48 ... Main body cord 51,52 ... Transmission unit 53,54 ... Reception unit 61 ... Camera (sound source identification unit, line-of-sight direction detection unit)
63 ... Microphone (voice acquisition unit)
100: Head-mounted display device (HMD)
120 ... Storage unit 122 ... Audio recording unit (acquired audio storage unit)
124 ... Identification information storage unit 130 ... Power source 132 ... Wireless communication unit (communication unit)
135: Operation unit 140 ... CPU
DESCRIPTION OF SYMBOLS 150 ... Operating system 160 ... Image processing part 161 ... Sound source identification part (sound source specific part)
165 ... Image setting unit (display image setting unit)
168 ... Gaze direction identification unit (gaze direction detection unit)
169: Conversion unit (conversion unit)
170: Voice processing unit (voice acquisition unit, voice identification unit)
180 ... interface 190 ... display control unit 201 ... right backlight control unit 202 ... left backlight control unit 211 ... right LCD control unit 212 ... left LCD control unit 221 ... right backlight 222 ... left backlight 241 ... right LCD
242 ... Left LCD
251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate 300 ... Server 324 ... Identification information storage unit 332 ... Information transmission / reception unit 340 ... CPU
365 ... Image setting unit 369 ... Conversion unit 500 ... Information system PCLK ... Clock signal VSync ... Vertical synchronization signal HSync ... Horizontal synchronization signal IMG1 ... Image (corresponding image)
OA ... External device WB ... White boat SC ... Outside view RE ... Right eye LE ... Left eye TE ... Teacher (sound source)
IL ... Illumination light PL ... Image light PN ... Maximum image display area VR ... Visual field ST ... Student QR1 ... Identification information (identification information)
TX1, TX2 ... Character image (character image)

Claims (18)

A transmissive head-mounted display device,
An audio acquisition unit for acquiring audio;
A sound source identifying unit that identifies the sound source using identification information of a sound source that emits sound;
A conversion unit for converting the acquired sound into a character image;
An image display unit capable of displaying an image and transmitting an outside scene;
A display image setting unit that displays the corresponding image associated with the identified sound source and the converted character image on the image display unit in association with each other ,
The sound source specifying unit determines whether or not the speaker's mouth as the sound source is moving, and when the speaker's mouth is moving, the conversion unit converts the speech into a character image, and A head-mounted display device in which the conversion unit does not convert the sound into a character image when a person's mouth is not moving . The head-mounted display device according to claim 1,
The voice acquisition unit identifies the direction of the sound source,
The display image setting unit is a head-mounted display device that displays the character image and the corresponding image on the image display unit in association with the specified direction. The head-mounted display device according to claim 1 or 2,
When there are a plurality of speakers as the sound source, the display image setting unit displays at least one of the corresponding image and the character image based on the identification information set for each speaker. A head-mounted display device that sets at least one of a display mode and a display position in the unit. The head-mounted display device according to any one of claims 1 to 3, further comprising:
A head-mounted device provided with an acquired sound storage unit that stores the identification information set for each of the plurality of sound sources and the sound of each of the plurality of sound sources in association with each other when there are a plurality of the sound sources. Type display device. The head-mounted display device according to any one of claims 1 to 4,
The display image setting unit displays the size of the character image that is converted and displayed on the image display unit based on the acquired sound volume and whether or not to display the character image on the image display unit A head-mounted display device that determines at least one of presence or absence. The head-mounted display device according to any one of claims 1 to 5, further comprising:
An imaging unit for imaging an outside scene;
A line-of-sight direction detection unit that detects a line-of-sight direction of the human being, which is the sound source, based on the captured image of the outside scene when the sound source is a human,
The display image setting unit sets at least one of a display mode and a display position in the image display unit of at least one of the corresponding image and the character image based on the detected line-of-sight direction. Type display device. The head-mounted display device according to any one of claims 1 to 6, further comprising:
An acceleration detection unit for detecting the acceleration of the image display unit;
The display image setting unit fixes a display position of at least one of the corresponding image and the character image displayed on the image display unit in correspondence with the detected acceleration of the image display unit. Type display device. The head-mounted display device according to any one of claims 1 to 7, further comprising:
A communication unit that transmits and receives information to and from other devices;
An acquired voice storage unit that stores the acquired voice and the identification information of the sound source that emits the acquired voice;
The display image setting unit is a head-mounted display device that transmits and receives the stored voice and the identification information to another device via the communication unit. The head-mounted display device according to claim 8, further comprising:
A position specifying unit for specifying the position of the image display unit;
The communication unit receives position information of another head-mounted display device from another device,
The head-mounted display device, wherein the display image setting unit sets the corresponding image based on the specified position of the image display unit and the received position information and displays the corresponding image on the image display unit. The head-mounted display device according to any one of claims 1 to 9,
The sound source specifying unit specifies the language of the acquired voice,
The display image setting unit sets at least one of a display mode and a display position in the image display unit of at least one of the corresponding image and the character image based on the specified language of the sound. Type display device.   The head-mounted display device according to claim 1,
  The head-mounted display device, wherein the display image setting unit causes the size of the character image displayed on the image display unit to be proportional to the size of the voice acquired by the voice acquisition unit.   The head-mounted display device according to claim 1,
  The head-mounted display device, wherein the display image setting unit causes the image display unit to display an image of the name of the speaker and the character image.   The head-mounted display device according to claim 1,
  The sound source specifying unit specifies a language of the speaker's voice,
  The conversion unit translates the voice of the speaker into a specific language and converts it into the character image,
  The display image setting unit displays the character image in a peripheral portion excluding the center of the image display unit when the language of the speaker's voice is the native language of the user of the head-mounted display device. The head-mounted display device displays the character image on the peripheral portion of the image display unit when the language of the speaker's voice is not the native language of the user of the head-mounted display device. The head-mounted display device according to claim 1,
The display image setting unit displays whether or not the character image is displayed on the image display unit according to the security level setting between the user of the head-mounted display device and the speaker, which is the identification information. A head-mounted display device that sets an aspect.   The head-mounted display device according to claim 1, further comprising:
  An imaging unit for imaging an outside scene;
  A line-of-sight direction detecting unit that detects the line-of-sight direction of the speaker based on the image of the captured outside scene
  The display image setting unit is a head-mounted display device that sets a display position of the character image on the image display unit in accordance with the line-of-sight direction of the speaker. A control method for a transmissive head-mounted display device having an audio acquisition unit for acquiring audio and an image display unit capable of displaying an image and transmitting an outside scene,
A sound source identifying step for identifying the sound source using identification information of a sound source that emits sound;
A conversion step of converting the acquired voice into a character image;
A process of associating the corresponding image associated with the identified sound source with the converted character image and causing the image display unit to display the associated image .
In the sound source identification step, it is determined whether or not the speaker's mouth as the sound source is moving. If the speaker's mouth is moving, the speech is converted into a character image in the conversion step, and the speech A control method in which the voice is not converted into a character image in the conversion step when the person's mouth is not moving . An information system including a transmissive head-mounted display device having an image display unit capable of displaying an image and transmitting an outside scene, and a communication unit transmitting and receiving information to and from other devices,
An audio acquisition unit for acquiring audio;
A sound source identifying unit that identifies the sound source using identification information of a sound source that emits sound;
A conversion unit that converts the acquired sound into a character image represented as an image;
An image information transmission unit that associates the correspondence image associated with the identified sound source and the converted character image and transmits the correspondence image to the communication unit ;
The sound source specifying unit determines whether or not the speaker's mouth as the sound source is moving, and when the speaker's mouth is moving, the conversion unit converts the speech into a character image, and An information system in which the conversion unit does not convert the sound into a character image when the person's mouth is not moving . A computer program for a transmissive head-mounted display device having an audio acquisition unit for acquiring audio and an image display unit capable of displaying an image and transmitting an outside scene,
A sound source identification function for identifying the sound source using identification information of a sound source that emits sound;
A conversion function that converts the acquired sound into a character image;
Causing the computer to realize a display image setting function for associating the corresponding image associated with the identified sound source and the converted character image and displaying them on the image display unit ,
The sound source identification function determines whether or not the speaker's mouth that is the sound source is moving. If the speaker's mouth is moving, the conversion function converts the speech into a character image, and A computer program in which the conversion function does not convert the sound into a character image when the person's mouth is not moving .

Images