JP2014027459A - Communication apparatus, communication method, and communication program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a communication apparatus, communication method, and communication program, having display means configured to be installed on the head of a user, capable of achieving communication using a voice, in view of the magnitude of the sound of a call environment.SOLUTION: The communication apparatus includes display means that is configured to be installed on the head of a user and outputs an image; communication means that exchanges data including sound information with another apparatus; and sound output means that converts the sound information received by the communication means into sound for output. A call device determines whether or not a call environment is a noise environment which is an environment having a possibility that noise may occur (S10). The communication apparatus, when the call environment is determined to be a noise environment (S65: YES), converts the sound which the sound information received by the communication means indicates into a text (S75). The communication apparatus displays an image indicating the text on the display means (S80).

Description

  The present invention relates to a communication device, a communication method, and a communication program that are configured to be wearable on a user's head, include display means for outputting an image, and that can transmit and receive data including sound information to and from other devices. .

  There is known a display device that is used while being worn on a user's head and allows the user to recognize various images. A head mounted display is mentioned as an example of a display device. Some display devices of this type have a function that allows the user to recognize sounds in consideration of surrounding sounds. For example, the head-mounted display device described in Patent Document 1 includes an air-conducting speaker and a bone-conducting sound speaker as means for surely perceiving the user, and uses both as necessary. On the other hand, various techniques for making a comfortable call in consideration of the volume of surrounding sounds have been studied. For example, the mobile phone described in Patent Document 2 includes a bone conduction speaker. When the ambient noise is high, the bone conduction speaker is used as a receiver for bone conduction sound. When the ambient noise is low, the bone conduction speaker is used. Used as a receiver with air conduction sound.

JP 2006-74798 A JP 2007-1998 A

  Development of a display device having a call function in consideration of ambient sounds is desired. In response to such a demand, it is conceivable to incorporate the mobile phone call technology described in Patent Document 2 in the head-mounted display device described in Patent Document 1. However, with the mobile phone call technology described in Patent Document 2, the voice of the other party may still be difficult to hear depending on the volume of sound around the listener or speaker.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a communication device, a communication method, and a communication including display means configured to be mounted on a user's head, capable of communication using voice in consideration of the volume of sound in a call environment. Is to provide a program.

  The communication device according to the first aspect is configured to be wearable on a user's head, and includes a display unit that outputs an image, a communication unit that transmits and receives data including sound information to and from another device, Sound output means for converting the sound information received by the communication means into sound, and determining means for determining whether or not the call environment is a noise environment that may generate noise; When the determination unit determines that the call environment is the noise environment, a conversion unit that converts voice represented by the sound information received by the communication unit into text, and the text generated by the conversion unit Display control means for causing the display means to display an image representing the above.

  According to the communication device according to the first aspect, even when the call environment is determined to be a noise environment, the user displays the matter represented by the sound information transmitted from the other device on the display unit. Can be recognized by looking at the text. Therefore, by using the communication device according to the first aspect, the user can perform communication using voice in consideration of the call environment.

  The communication method according to the second aspect is configured to be wearable on a user's head, and includes a display unit that outputs an image, a communication unit that performs transmission and reception of data including sound information with another device, A communication method executed by a communication device including sound output means for converting the sound information received by the communication means into vibration and outputting the vibration, wherein the call environment is an environment in which noise may occur. A determination step for determining whether or not the environment is a noise environment; and when the call environment is determined to be the noise environment in the determination step, the voice represented by the sound information received by the communication means is converted into text And a display control step for causing the display means to display an image representing the text generated in the conversion step. In the communication method according to the second aspect, the same steps and effects as those of the communication apparatus according to the first aspect are obtained by executing the above steps by the communication device.

  A communication program according to a third aspect is configured to be wearable on a user's head, and includes a display unit that outputs an image, a communication unit that performs transmission and reception of data including sound information with another device, A display program executed by a computer of a communication device comprising sound output means for converting the sound information received by the communication means into vibration and outputting the vibration, wherein the call environment may generate noise A determination step for determining whether or not the noise environment is, and when the call environment is determined to be the noise environment in the determination step, the voice represented by the sound information received by the communication means is converted into a text And a display control step for causing the display means to display an image representing the text generated in the conversion step. The communication program which concerns on a 3rd aspect can obtain the effect similar to the communication apparatus of a 1st aspect by making a computer perform the said step.

It is a perspective view which shows the external appearance of HMD1. 3 is a block diagram showing an electrical configuration of the mounting device 2 and the control device 3. FIG. It is a flowchart of a telephone call process. It is a flowchart of the telephone call environment judgment process performed by the telephone call process shown in FIG. It is explanatory drawing of the picked-up image 70, the display image 80, and the visual field 85. FIG.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The head mounted display 1 (hereinafter referred to as “HMD1”) shown in FIG. 1 has a function of transmitting and receiving data including sound information to and from other devices. The HMD 1 includes a mounting device 2 and a control device 3. The mounting device 2 is used by being mounted on the user's head. The mounting device 2 is detachably connected to the control device 3 via the harness 4. For example, the control device 3 is used by being mounted on a user's waist belt or the like. The control device 3 controls the mounting device 2. Hereinafter, each of the mounting device 2 and the control device 3 will be described in detail. The upward direction, downward direction, downward diagonal direction to the right, upward diagonal direction to the left, upward diagonal direction to the right, and downward diagonal direction to the left in FIG. 1 are the upward direction, downward direction, forward direction, backward direction, right direction, left direction, respectively. Direction.

  With reference to FIG. 1, the structure of the mounting apparatus 2 is demonstrated. The wearing device 2 includes glasses 11, a display device 12, a sound information input / output device 13, and an air conduction microphone 14. The glasses 11 have the same shape as normal glasses. In a state where the user wears the wearing device 2, the glasses 11 hold the display device 12, the sound information input / output device 13, and the air-conduction microphone 14 on the user's head. Inside the frame of the glasses 11, wiring that is electrically connected to the sound information input / output device 13 and the air conduction microphone 14 is provided. This wiring is for supplying power and transmitting / receiving various signals.

  The display device 12 is a retinal scanning display that scans laser light modulated in accordance with an image signal, emits the laser light as image light, and directly projects an image on the retina of at least one eye of the user. The display device 12 includes a housing 15 and a support unit 19. The housing 15 accommodates a configuration for outputting image light therein. The configuration for outputting image light includes an image display unit (not shown), an eyepiece optical system (not shown), and a half mirror 16 (hereinafter referred to as “HM16”). The image display unit includes a CPU 21 (see FIG. 2), a display control unit 29 (see FIG. 2), and a display 28 (see FIG. 2). The image display unit drives the display control unit 29 based on the video signal transmitted from the control device 3 via the harness 4 and displays an image on the display 28. The display 28 includes, for example, a liquid crystal display element and a backlight. The eyepiece optical system condenses image light indicating an image displayed on the display 28 so as to guide it to the eyes of the user. The HM 16 is provided on the left side of the display device 12 and reflects at least a part (for example, half) of the image light emitted from the eyepiece optical system so as to enter the left eyeball (not shown) of the user. Since the HM 16 transmits at least part of the light from the real image in the outside world, the user can visually recognize the image superimposed on the real image (outside landscape) in his field of view. The housing 15 further includes a camera 17 inside. The lens of the camera 17 is exposed from a small hole 18 provided on the front surface of the housing 15 and on the right side of the HM 16. The camera 17 generates image data representing an external landscape in front of the user's visual field. The support portion 19 is a member that is provided on the upper right portion of the display device 12 and attaches the display device 12 to the upper right portion of the glasses 11.

  The sound information input / output device 13 is a headphone type device that covers a user's ear (not shown), and includes a bone conduction microphone 31, an air conduction earphone 32, and a bone conduction earphone 33 shown in FIG. The air conducting microphone 14 is provided below the left front end of the glasses 11. When the user wears the wearing device 2, the air-conduction microphone 14 is disposed near the user's mouth (not shown).

  The configuration of the control device 3 will be described. The control device 3 is a substantially rectangular parallelepiped system box. The control device 3 includes an operation unit 5. The user can perform various operations such as starting a call with a user of another device via the operation unit 5. The control device 3 includes a connector 6. A communication cable can be connected to the connector 6.

  With reference to FIG. 2, the electrical configuration of the mounting device 2 will be described. The mounting device 2 includes a CPU 21, a ROM 23, a RAM 25, a display control unit 29, a camera 17, an air conduction microphone 14, a bone conduction microphone 31, an air conduction earphone 32, a bone conduction earphone 33, and a first wired communication unit 34. Prepare. The CPU 21 controls the entire mounting device 2 and is electrically connected to other electrical devices included in the mounting device 2. The ROM 23 stores various programs executed by the CPU 21. The RAM 25 temporarily stores various data.

  The display control unit 29 controls the display 28 in order to display an image on the display 28. Hereinafter, an image displayed on the display 28 is referred to as a display image. The camera 17 generates image data representing a still image or a moving image (hereinafter referred to as “captured image”). The air conduction type microphone 14 converts vibration of sound transmitted in the air into sound information. As the air conduction type microphone 14, for example, a known configuration such as a dynamic microphone and a condenser microphone can be used. Sound information generated based on vibration of sound transmitted in the air is referred to as first sound information. The bone conduction microphone 31 collects a user's bone conduction sound and converts it into sound information. The sound information generated by collecting the user's bone conduction sound is referred to as second sound information. The air-conduction type earphone 32 converts sound information into sound transmitted in the air and outputs the sound. The bone conduction earphone 33 converts sound information into bone conduction sound and outputs it. The first wired communication unit 34 performs wired communication with the control device 3 via the harness 4 (see FIG. 1). The CPU 21 controls the first sound information generated by the air conduction microphone 14 and the second sound information generated by the bone conduction microphone 31 in the call processing described later via the first wired communication unit 34 and the harness 4. 3 to send at any time.

  The electrical configuration of the control device 3 will be described. The control device 3 includes a CPU 41, a ROM 43, a RAM 45, a flash memory 47, a second wired communication unit 46, an operation unit 5, a wireless communication unit 53, and an external communication unit 55. The CPU 41 controls the entire control device 3 and is electrically connected to other electronic devices included in the control device 3. The ROM 43 stores various programs executed by the CPU 41. The RAM 45 temporarily stores various data. The flash memory 47 includes a registered image storage area 48, a setting storage area 49, and other storage areas 50. The registered image storage area 48 stores image data representing an image (hereinafter also referred to as “registered image”) used for determining whether or not the call environment is a noise environment. The registered image is a sign indicating an attribute of the surrounding environment (for example, a workplace, a play room, a factory, a laboratory, etc.). The registered image 51 represents a sign affixed to “WORK AREA” such as inside a factory. The setting storage area 49 stores various setting values including threshold values used in call processing to be described later. The other storage area 50 stores other information including an acoustic model, a language model, and a word dictionary for performing speech recognition. The flash memory 47 may store programs executed by the CPUs 21 and 41. The second wired communication unit 46 performs wired communication with the mounting device 2 via the harness 4. The wireless communication unit 53 performs wireless communication with other devices. The external communication unit 55 performs wired communication with other devices via a communication cable. Other devices that perform wireless or wired communication with the HMD 1 are, for example, other HMDs 1, PCs, mobile phones, and smartphones.

  An outline of call processing executed in the HMD 1 will be described. The HMD 1 can transmit and receive data including sound information to and from other devices via the wireless communication unit 53. Call processing is executed when a user of the HMD 1 uses this function to make a call with a user of another device. In the call processing, the HMD 1 switches the sound information input / output method according to whether or not the call environment is a noise environment. The call environment includes an environment around the HMD 1 (hereinafter also referred to as “own device 1”) that executes processing and an environment around another device that is a communication partner (hereinafter also referred to as “other device”). At least one of them. The call environment of the present embodiment refers to both the environment around the device itself and the environment around other devices. The noise environment is an environment that may generate noise. The noise environment of the present embodiment includes an environment in which noise is predicted to be generated during the process execution period, in addition to an environment in which noise is generated during the process execution. When it is determined that the call environment is not a noise environment, the HMD 1 transmits the first sound information generated by the air-conduction microphone 14 to another device and uses the air-conduction earphone 32 to monitor the sound information. Convert to sound and output. When it is determined that the call environment is a noise environment, the HMD 1 transmits the second sound information generated by the bone conduction microphone 31 to another device, and uses the bone conduction earphone 33 to transmit the sound information to the bone information. Convert to sound and output. Further, when it is determined that the call environment is a noise environment, the HMD 1 converts the voice represented by the sound information transmitted from the other device into text, and displays an image representing the generated text on the display 28.

  Call processing will be described with reference to FIGS. The call processing is started when the user of the HMD 1 operates the operation unit 5 to input an instruction to start communication with another device, or when a signal including an instruction to start communication is received from the other device. . The call process in FIG. 3 is executed by the CPU 21 and the CPU 41 in cooperation with each other according to the program stored in the ROM 43 in FIG. As an example, a case where communication is performed with another HMD 1 will be described.

  As shown in FIG. 3, the CPU 41 first establishes communication between the own apparatus 1 and another apparatus (S5). Next, the CPU 41 executes a call environment determination process (S10). With reference to FIG. 4, the call environment determination process will be described. As shown in FIG. 4, in the call environment determination process, first, the CPU 41 acquires the first sound information generated by the air-conduction microphone 14 as one aspect of the environment information (S15). The environment information is information representing the environment around the user of the HMD 1. The CPU 41 outputs an instruction to the CPU 21 and causes the control device 3 to start transmitting the first sound information and the second sound information. Based on the received instruction, the CPU 21 starts a process of transmitting the first sound information generated by the air conduction microphone 14 and the second sound information generated by the bone conduction microphone 31 to the control device 3 as needed. The first sound information acquired in S15 represents the sound of the environment around the device 1 itself. CPU41 specifies the sound level around the own apparatus 1 based on the 1st sound information acquired by S15, and transmits the data containing the specified sound level to another apparatus (S20). The sound level is represented by a well-known index that represents the degree of loudness of the surrounding sound. HMD1 of this embodiment represents a sound level using the sound pressure level (dB) estimated from the 1st sound information acquired by S15. Here, the 1st sound information acquired by S15 includes the information showing not only the surrounding sound but the voice which the user uttered, when the user uttered. Therefore, the HMD 1 of the present embodiment estimates the sound pressure level by excluding the sound in the frequency range corresponding to the sound from the sound represented by the first sound information. For this reason, the HMD 1 can eliminate the influence of the user's voice included in the first sound information, and more accurately predict the sound pressure level of the surrounding sound volume. Next, the CPU 41 receives data transmitted from the other device, and acquires the sound level of the other device from the received data (S25). As a specific example, it is assumed that the sound level of the own device 1 is 50 dB and the sound level transmitted from another device is 40 dB.

  CPU41 judges whether the sound level of the own apparatus 1 is below a threshold value (S30). The threshold is a value set in advance to determine whether or not the environment around the device 1 is a noise environment based on the sound level of the device 1 specified in S20. The threshold value of the process of S30 may be determined as appropriate according to the index representing the sound level, and may be a predetermined fixed value or a value set by the user. The threshold of this embodiment is preset as a sound pressure level of 60 dB. In a specific example, the CPU 41 determines that the sound level (50 dB) of the own device 1 is equal to or lower than the threshold (S30: YES), and determines whether the sound level of the other device is equal to or lower than the threshold (S35). The threshold value for the process of S35 may be determined as appropriate according to the index representing the sound level, and may be determined in advance or may be a value set by the user. The threshold for the process at S35 may be the same as or different from the threshold for the process at S30. The threshold of this embodiment is preset as a sound pressure level of 60 dB. In the specific example, the CPU 41 determines that the sound level (40 dB) of the other device is equal to or less than the threshold value (S35: YES), and acquires the image data of the own device 1 as one aspect of the environmental information (S40). In S40, the CPU 41 acquires the image data generated by the camera 17 as environment information different from the first sound information acquired in S15. In a specific example, a case is assumed where image data representing the captured image 70 illustrated in FIG. 5 is acquired.

  The CPU 41 determines whether or not the call environment is a noise environment based on the image data acquired in S40 (S45). In the present embodiment, the CPU 41 determines that the call environment is a noise environment when a registered image is included in the captured image represented by the image data acquired in S40. The process of determining whether or not a registered image is included in a captured image is executed by appropriately adopting a known technique. The registered image is represented by image data stored in the flash memory 47. For example, in S45, the determination process is executed according to the following procedure. First, the CPU 41 performs edge detection on the captured image and the registered image. As a method for detecting an edge, a known method such as a method for first-order differentiation of an image to detect a position where the gradient becomes maximum and a method for second-order differentiation of an image to detect a zero crossing position are used. Next, the CPU 41 extracts feature points (for example, vertex positions) from the detected edges. Next, the CPU 41 compares the feature point of the captured image with the feature point of the registered image by pattern matching. If the feature point of the captured image includes a pattern that matches the feature point of the registered image, the captured image It is determined that the registered image is included.

  As shown in FIG. 5, the captured image 70 of the specific example includes a sign 71. The sign 71 is stored in the flash memory 47 as the registered image 51 as shown in FIG. Therefore, the CPU 41 determines that the call environment is a noise environment (S45: YES), and turns on the flag (S55). The flag represents a determination result of whether or not the call environment is a noise environment. When the flag is ON, it indicates that the call environment is determined to be a noise environment. The CPU 41 also turns on the flag when the sound level of the own device 1 is larger than the threshold (S30: NO) and when the sound level of the other device is larger than the threshold (S35: NO) (S55). . On the other hand, if the CPU 41 determines that the call environment is not a noise environment based on the image data acquired in S40 (S45: NO), the CPU 41 turns off the flag (S50). If the flag is OFF, it indicates that the call environment is determined not to be a noise environment. After S50 or S55, the process returns to the call process of FIG.

  Following S10, the CPU 41 determines whether or not the sound of another device has been acquired (S60). The CPU 41 acquires sound information from data received from another device. CPU41 judges that the audio | voice of another apparatus was acquired when the information of the frequency band corresponding to an audio | voice is contained in sound information (S60: YES). In this case, when the flag set in the call environment determination process is ON as in the specific example (S65: YES), the sound information received from the other device by the mounting device 2 and the bone conduction earphone 33 are displayed. Is used to output instructions for converting sound information into sound. Based on the instruction received from the control device 3, the CPU 21 converts the sound information received from the other device into sound using the bone conduction earphone 33 and outputs the sound (S70). Next, the CPU 41 recognizes the sound information obtained in S60 and converts it into text (S75). A known method may be appropriately employed as the speech recognition method. For example, the CPU 41 analyzes sound information and extracts feature amounts. The CPU 41 matches the extracted feature quantity with the acoustic model and the language model stored in the flash memory 47 while referring to the word dictionary stored in the flash memory 47 as necessary. As a result, the likelihood is obtained for each sentence that can be accepted by the language model, and the sentence with the highest likelihood is obtained as the text of the recognition result. In a specific example, it is assumed that the voice represented by the sound information is converted into the text “Hello, this is ABC of X company.”.

  The CPU 41 generates image data representing text, and transmits the generated image data and an instruction for displaying a display image based on the image data to the mounting apparatus 2. Based on the instruction received from the control device 3, the CPU 21 controls the display control unit 29 to display the display image represented by the image data on the display 28 (S80). In the specific example, the display image 80 of FIG. 5 is displayed on the display 28 by the process of S80. When the real image is the same image as the photographed image 70 in the own field of view and the display image 80 is displayed on the display 28, the user superimposes the text on the real image in the own field of view 85 as shown in FIG. The displayed display image 80 can be visually recognized. On the other hand, when the flag set in the call environment determination processing is OFF (S65: NO), the CPU 41 attaches the sound information received from the other device and the instruction for outputting the sound information with the air conduction type earphone 32. Output to device 2. Based on the instruction received from the control device 3, the CPU 21 converts the sound information received from the other device into sound using the air conduction type earphone 32 and outputs the sound (S85).

  When the voice is not acquired from the other apparatus (S60: NO), the CPU 41 determines whether or not the voice of the own apparatus 1 is acquired after S80 or S85 (S90). The CPU 41 acquires sound information from the data acquired from the mounting device 2. CPU41 judges that the audio | voice of the own apparatus 1 was acquired when the information of the frequency band corresponding to an audio | voice is contained in the sound information transmitted from the mounting apparatus 2 (S90: YES). In this case, when the flag is ON as in the specific example (S95: YES), the CPU 41 obtains the second sound information generated by the bone conduction microphone 31 (S100). CPU41 outputs the data containing the acquired 2nd sound information to another apparatus (S105). On the other hand, when the flag is OFF (S95: NO), the CPU 41 obtains the first sound information generated by the air conduction microphone 14 from the mounting device 2 (S110). CPU41 outputs the data containing the acquired 1st sound information to another apparatus (S115). If the voice of the device 1 has not been acquired (S90: NO), if there is no instruction to end the call after S105 or S115 (S120: NO), the process returns to S60. If there is an instruction to end the call (S120: YES), the call process is ended.

  In the HMD 1 of the present embodiment, the display device 12 (display 28) corresponds to the display means of the present invention. The wireless communication unit 53 corresponds to communication means of the present invention. Each of the air conduction type earphone 32 and the bone conduction type earphone 33 corresponds to the sound output means of the present invention. Each of S30, S35, and S45 in FIG. 4 corresponds to a determination step of the present invention. The CPU 41 that executes each of the processes of S30, S35, and S45 functions as a determination unit of the present invention. S75 corresponds to the conversion step of the present invention. The CPU 41 that executes the process of S75 functions as the conversion means of the present invention. S80 corresponds to the display control step of the present invention. The CPU 41 that executes the process of S80 functions as display control means of the present invention. The CPU 41 that executes each of the processes of S15 and S40 functions as an acquisition unit of the present invention. The air conduction type microphone 14 corresponds to the first sound information generating means of the present invention. The CPU 41 that executes the process of S115 functions as a first communication control unit. The bone conduction microphone 31 corresponds to the second sound information generating means of the present invention. The CPU 41 that executes the process of S105 functions as the second communication control means of the present invention.

  According to HMD1, even when the call environment is determined to be a noise environment, the user can view the matter represented by the sound information transmitted from the other device by looking at the text displayed on the display 28. Can be recognized. The HMD 1 determines whether or not the call environment is a noise environment from three viewpoints. The first point of view is whether or not the surrounding environment of the own device 1 is an environment that generates noise at the time of processing execution, and the HMD 1 is a surrounding of the own device 1 that is an aspect of the environment information. Based on the first sound information representing the sound, it is determined whether or not it is a noise environment (S30). Since the first sound information generated by the air-conduction microphone 14 indicates whether or not noise is generated around the device 1, the call is based on information other than the first sound information generated by the air-conduction microphone 14. Compared with the case where it is determined whether the environment is a noise environment, the reliability of the determination result is high. When the sound level of the sound around the own device 1 is high, it is expected that the voice of the user of the other device is difficult to hear due to the influence of the sound around the own device 1 during the call processing. Even if it is difficult for the user of the HMD 1 to hear the sound represented by the sound information transmitted from the other device due to the influence of the sound around the device 1, the sound information is represented by viewing the text displayed on the display 28. Recognize matters.

  The second point of view is whether or not the surrounding environment of the device 1 may generate noise. The HMD 1 is based on image data that is one form of environmental information, and the call environment is a noise environment. It is automatically determined whether or not (S45). When a process for determining whether or not the call environment is a noise environment (determination process) is executed, the call process is being executed after the determination process is executed even if no noise is generated in the environment around the device 1. In addition, noise may be generated in the environment around the device 1. In consideration of such a case, in the HMD 1, information indicating that there is a possibility of generating noise is registered in advance as a registered image. Thereby, in addition to the environment in which noise is generated during the execution of the determination process, the environment in which noise is predicted to be generated during the process execution period can be included in the noise environment.

  The third viewpoint is whether or not the environment around the user of the other device may generate noise. The HMD 1 is based on data transmitted from the other device, and the call environment is a noise environment. It is determined whether or not there is (S35). When there is a possibility that the environment around the user of the other device may generate noise, it is assumed that the sound information transmitted from the other device includes noise and it is difficult to hear the voice of the user of the other device. On the other hand, the HMD 1 may determine whether or not the call environment is a noise environment based on data transmitted from another device, so that the environment around the user of the other device may generate noise. It can be determined whether or not there is. When it is determined that the call environment is a noise environment, the matter represented by the sound information transmitted from the other device is automatically displayed in text. For this reason, even when it is difficult to hear the voice of the user of the other apparatus due to the influence of the noise around the user of the other apparatus, the user of the own apparatus 1 can recognize the call content by looking at the text. Since the HMD 1 automatically acquires information for determining whether or not the call environment is a noise environment from the above three viewpoints, it is not necessary for the user to input information.

  Further, the HMD 1 transmits the first sound information generated by the air conduction microphone 14 to another device and uses the air conduction earphone 32 when it is determined that the call environment is not a noise environment. Sound information is converted into air conduction sound and output. When it is determined that the call environment is a noise environment, the HMD 1 transmits the second sound information generated by the bone conduction microphone 31 to another device, and uses the bone conduction earphone 33 to transmit the sound information to the bone information. Convert to sound and output. When the environment around the device 1 is a relatively noisy environment, generally, the first sound information includes a lot of information representing the surrounding noise, and therefore the second sound information is higher than the first sound information. The sound information more accurately represents the sound emitted by the user of the device 1. When it is determined that the environment around the user is a noise environment, the HMD 1 automatically transmits the second sound information to another device. Therefore, even when the user's own device 1 user is in a noisy environment, the user of the user's own device 1 can increase the possibility that the user of the user's device 1 will correctly recognize the user's voice. .

  When the surrounding environment is a relatively noisy environment, generally, the bone that reaches the auditory nerve by propagating to the cartilage near the inner wall of the ear canal rather than the air conduction sound that reaches the eardrum through the air. The user is easier to recognize the content of the sound when the sound is guided. In general, when sound information is converted into bone conduction sound, the user's voice is reproduced more accurately when the second sound information is converted into bone conduction sound than when the first sound information is converted into bone conduction sound. it can. Therefore, when the environment around the other device is a noise environment, it is estimated that the user of the other device can easily hear the bone conduction sound. On the other hand, when the HMD 1 determines that the environment around the other device is a noise environment, the HMD 1 automatically transmits to the other device the second sound information suitable for conversion to the bone conduction sound. . Therefore, the user of the own device 1 increases the possibility that the intention of the user of the own device 1 is accurately conveyed to the user of the other device by voice even when the environment around the user of the other device is determined to be a noise environment. Can do. When the own device 1 executes a call process with another HMD 1, when it is determined that the call environment is a noise environment by the same call process between the own device 1 and the other HMD 1, Two-sound information is transmitted / received, and the received second sound information is converted into a bone conduction sound. Therefore, users of the device 1 and the other HMD 1 can make a call by automatically switching to a bone-conducted sound that is hardly affected by noise even when the call environment is determined to be a noise environment.

  The communication device of the present invention is not limited to the HMD 1 of the above-described embodiment, and various modifications may be made without departing from the gist of the present invention. For example, the following modifications (A) to (F) may be added as appropriate.

  (A) The configuration, shape, design, and the like of the HMD 1 may be changed as appropriate. For example, the display device 12 of the HMD 1 may be attached to other wearing tools such as glasses, a helmet, and headphones that are used on a daily basis by the user. Further, for example, the mounting device 2 and the control device 3 may perform wireless communication instead of wired communication. For example, the mounting device 2 may have the function of the control device 3. Further, for example, the image display unit (not shown) included in the display device 12 is a retinal scanning display unit including a spatial modulation element such as a liquid crystal element and a light source, an organic EL (Electro Luminescent) element, a two-dimensional optical scanner, and a light source. It's okay. Further, for example, the bone conduction microphone 31, the air conduction earphone 32, and the bone conduction earphone 33 of the HMD 1 are built in the sound information input / output device 13, but may be provided separately from each other. Further, for example, the HMD 1 includes the air conduction microphone 14 and the bone conduction microphone 31 as the sound information generation device, but may include only one of them. Further, for example, the HMD 1 includes the air conduction type earphone 32 and the bone conduction type earphone 33 as the sound output device, but may include only one of them. Further, the HMD 1 may execute a call process with another apparatus using wired communication.

  (B) The method for determining whether the call environment is a noise environment may be changed as appropriate. For example, the modifications shown in (B-1) to (B-4) may be added as appropriate.

  (B-1) The call environment may be at least one of the environment around the device 1 and the environment around another device. Moreover, in the said embodiment, although the telephone call environment was judged whether it was a noise environment from three viewpoints as mentioned above, it is not limited to this. For example, based on one or more viewpoints selected from the above three viewpoints, it may be determined whether or not the call environment is a noise environment. Further, for example, the HMD 1 is determined to be an environment in which the call environment may generate noise from a predetermined number of viewpoints (for example, two or more viewpoints among the three viewpoints) among a plurality of viewpoints. In this case, it may be determined that the call environment is a noise environment.

  (B-2) The type and acquisition method of environmental information may be changed as appropriate. For example, when acquiring image data generated by the camera 17 as environment information, if a photographed image is analyzed and a pre-registered object or person is detected, it may be determined that the call environment is a noise environment. Good. More specifically, for example, when the machine 72 is detected in the captured image 70 shown in FIG. 5, it may be determined that the call environment is a noise environment. In addition, for example, when the environment information is information representing an attribute of the surrounding environment, the environment information may be acquired by RFID (Radio Frequency Identification). For example, information input by the user operating the operation unit 5 may be acquired as environment information. Further, for example, when it is known that the environment around the own device 1 (another device) becomes a noise environment only in a specific time zone, the CPU 41 acquires the time as environment information and makes a call based on the acquired time. It may be determined whether the environment is a noise environment. In this case, it is possible to appropriately determine whether the call environment is a noise environment according to the acquisition time. Further, for example, when a position and information indicating whether or not the position is a noise environment are stored in association with each other, the position of the device 1 may be acquired as environment information from a GPS or the like. In this case, when the acquired position of the own device 1 is a position registered in advance in association with the noise environment, the call environment may be determined to be the noise environment.

  Further, for example, the HMD 1 may determine whether or not the call environment is a noise environment based on the volume of the sound output device (the air conduction type earphone 32 or the bone conduction type earphone 33) of the HMD1. More specifically, when the volume of the sound output device is set to be relatively large, it is expected that the voice of the user of the other device is difficult to hear due to the call environment being a noise environment. Is done. For this reason, the HMD 1 acquires the volume setting from its own apparatus 1 or another apparatus, and the volume level set in 10 levels from 1 to 10 in the order of decreasing volume, for example, when the volume of the sound output apparatus is relatively large (for example, 9 or more), the call environment may be determined to be a noise environment. Further, for example, information related to content that is used in the device 1 or used within a predetermined time (for example, 10 minutes) may be acquired as environment information. More specifically, some contents (for example, images and sounds) that can be used in the HMD 1 are used in a specific environment. For example, among the contents displayed on the display 28 of the HMD 1, it is considered that a manual for explaining a method for operating a specific machine is used in a work environment such as a factory where the specific machine is arranged. When the HMD 1 stores content usage environments (for example, factories, research facilities, etc.) as supplementary information of content, the HMD 1 may use the supplementary information as environmental information. In this case, for example, the call process may be executed while executing the process of reproducing the content. In this case, for example, when the user of the device himself / herself wants to know the operation method of a specific machine, the manual explaining the operation method of the specific machine is reproduced and confirmed on the HMD 1 while the manual is unknown. It is possible to make inquiries about other points to users of other devices. Furthermore, if information on the content being used by the other device and image data representing the surroundings of the own device are transmitted, communication between the user of the own device and the user of the other device can be facilitated. The information related to the content includes, for example, information indicating the usage environment of the content, for example, information indicating the content content (for example, a manual for operating the machine), and information regarding the content usage target (for example, factory operations). Member, and researcher at the research institute).

  (B-3) The index indicating the sound level used for determining whether or not the call environment is a noise environment and the method for acquiring the index may be changed as appropriate. Moreover, the threshold value used for determining whether or not the call environment is a noise environment may be changed as appropriate according to an index representing the sound level. For example, in the process of S20, the CPU 41 may predict the sound pressure level without considering the influence of the voice uttered by the user based on the first sound information acquired in S15. In this case, it is preferable that the sound information is acquired at a timing estimated that the user does not utter. For example, when the HMD 1 includes a sound level meter, the sound level may be represented by a sound pressure level (dB) acquired from the sound level meter. Further, for example, the sound level may be represented by an S / N ratio (Signal to Noise ratio) in which the ratio of S (signal) and N (noise) included in the first sound information is expressed logarithmically. In this case, for example, the HMD 1 may use a vibration having a frequency representing sound as S (signal). For example, the threshold value may be a predetermined value or a value set by the user. In addition, when the threshold value is set to a predetermined value, for example, a specific value is selected from a plurality of threshold values according to conditions such as a user, a usage time zone, and an attribute (factory, laboratory, etc.) of the surrounding environment. A threshold value may be selected and used.

  (B-4) The method for determining whether or not the call environment is a noise environment based on data received from another device may be changed as appropriate. For example, the own device 1 may specify the sound level represented by the sound information based on the sound information included in the data. Further, for example, in another device, it may be determined whether the environment around the other device is a noise environment, and the determination result may be transmitted to the own device 1. In this case, the own device 1 may determine whether or not the call environment is a noise environment based on the determination result received from another device. Also, for example, when the data transmitted from the other device includes information representing the environment around the other device, whether or not the call environment is a noise environment based on the information included in the data received from the other device. May be judged. The information on the other device may be the same type of information as any of the environment information on the own device.

  (C) The HMD 1 switches between the air-conduction microphone 14 and the bone-conduction microphone 31 depending on whether the call environment is a noise environment, but is not limited to this. For example, when the own device 1 includes only the air conduction microphone 14 or the bone conduction microphone 31, the HMD 1 switches the sound information input device depending on whether or not the call environment is a noise environment. It does not have to be. Similarly, the HMD 1 switches between the air conduction type earphone 32 and the bone conduction type earphone 33 depending on whether or not the call environment is a noise environment, but is not limited thereto. For example, when the device 1 includes only the air conduction type earphone 32 or the bone conduction type earphone 33, the HMD 1 switches the sound information output device depending on whether or not the call environment is a noise environment. It does not have to be. Even when any one of the above modifications is made, if it is determined that the call environment is a noise environment, the user displays the matter represented by the sound information transmitted from the other device on the display 28. It can be recognized by looking at the text.

  (D) The arrangement, font, size, and the like of the text in the image representing the text may be changed as appropriate. The HM 16 may be configured not to transmit light from a real image of the outside world, and the user may be able to visually recognize only the image displayed on the display 28 in his field of view.

  (E) A program including an instruction for executing a call process may be stored in a storage device included in the HMD 1 until the HMD 1 executes the program, and a program acquisition method, an acquisition route, and a device that stores the program Each of these may be changed as appropriate. Therefore, the program executed by the CPU 41 may be received from another device via a communication cable or wireless communication and stored in a storage device such as the flash memory 47. Other devices include, for example, a PC and a server connected via a network. Similarly, the program executed by the CPU 21 may be received from another device via a communication cable or wireless communication, and stored in a storage device such as the RAM 25 and a nonvolatile medium. Other devices in this case include, for example, a control device 3, a PC, and a server connected via a network.

  (F) Each step of the communication process shown in FIG. 3 and FIG. 4 is not limited to the example executed by the CPU 41, and a part or all of the steps may be executed by another electronic device (for example, ASIC). Each step of the communication process may be distributed and processed by a plurality of electronic devices (for example, a plurality of CPUs). The steps of the communication process shown in FIGS. 3 and 4 can be changed in order, omitted, or added as necessary. For example, in the processing of S100 and S110 in the call processing of FIG. 3, the HMD 1 converts the user's voice of the user's device 1 input in S100 into text, and displays an image representing the generated text on the display 28. You may let them. In this way, the user of the user apparatus 1 can confirm how the user's voice of the user apparatus 1 is converted into text in the other apparatus by viewing the image displayed on the display 28. . Further, for example, when it is determined that the call environment is a noise environment (S95: YES), the first sound information acquired from the air conduction microphone 14 and the second sound information acquired from the bone conduction microphone 31 Both may be transmitted to other devices. In such a case, the other device can select the sound information that is easier to hear the sound from the first sound information and the second sound information, and convert the sound information into a sound. . Furthermore, based on an instruction from the CPU provided in the HMD 1, an operating system (OS) or the like operating on the HMD 1 performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing. Are also included within the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Head mounted display 2 Mounting apparatus 3 Control apparatus 12 Display apparatus 13 Sound information input / output apparatus 14 Air-conduction type microphone 21, 41 CPU
23, 43 ROM
25, 45 RAM
17 Camera 31 Bone Conduction Microphone 32 Air Conduction Earphone 33 Bone Conduction Earphone 34 First Wired Communication Unit 46 Second Wired Communication Unit 47 Flash Memory 53 Wireless Communication Unit

Claims (7)

Display means configured to be worn on the user's head and outputting an image;
A communication means for transmitting and receiving data including sound information to and from other devices;
Sound output means for converting the sound information received by the communication means into sound and outputting the sound;
A determination means for determining whether or not the call environment is a noise environment that may generate noise;
Conversion means for converting the voice represented by the sound information received by the communication means into text when the determination means determines that the call environment is the noise environment;
A communication apparatus comprising: display control means for causing the display means to display an image representing the text generated by the conversion means. An acquisition unit that acquires environment information representing an environment around the user;
The communication apparatus according to claim 1, wherein the determination unit determines whether or not the call environment is the noise environment based at least on the environment information acquired by the acquisition unit. First sound information generating means for converting vibration of sound transmitted through the air into first sound information;
A first communication control means for transmitting the first sound information generated by the first sound information generating means to the other device via the communication means;
The acquisition means acquires the first sound information generated by the first sound information generation means as at least a part of the environmental information,
The determination means determines whether or not the call environment is the noise environment based at least on a comparison result between the first sound information generated by the first sound information generation means and a threshold value. The communication device according to claim 2. Second sound information generating means for collecting the bone conduction sound of the user and converting it into second sound information;
When the determination unit determines that the call environment is the noise environment, the second sound information generated by the second sound information generation unit is transmitted to the other device via the communication unit. The communication apparatus according to claim 1, further comprising: a second communication control unit that performs the above-described operation.   5. The communication apparatus according to claim 1, wherein the determination unit determines whether or not the call environment is the noise environment based at least on the data received by the communication unit. . A display unit configured to be mounted on a user's head, outputting an image, a communication unit executing transmission / reception of data including sound information with another device, and the sound information received by the communication unit A communication method executed by a communication device including sound output means for converting to vibration and outputting,
A determination step of determining whether or not the call environment is a noise environment that is an environment that may generate noise;
A conversion step of converting the voice represented by the sound information received by the communication means into text when the call environment is determined to be the noise environment in the determination step;
A display control step of causing the display means to display an image representing the text generated in the conversion step. A display unit configured to be mounted on a user's head, outputting an image, a communication unit executing transmission / reception of data including sound information with another device, and the sound information received by the communication unit A communication program executed by a computer of a communication device including sound output means for converting to vibration and outputting,
A determination step of determining whether or not the call environment is a noise environment that is an environment that may generate noise;
A conversion step of converting the voice represented by the sound information received by the communication means into text when the call environment is determined to be the noise environment in the determination step;
A communication program for executing a display control step of causing the display means to display an image representing the text generated in the conversion step.

Images