JP7065730B2 - Wearable terminal and display system - Google Patents

Description

The present invention relates to a wearable terminal.

Patent Document 1 makes it possible to reduce visibility from the outside world when image light transmitted through a half mirror is transmitted to a head-mounted display provided with a half mirror that reflects image light and projects it onto the observer's eyes. A technique for providing a visibility-inhibiting member is disclosed.

Japanese Unexamined Patent Publication No. 2012-159680

By superimposing the screen of a personal computer or the like on the display surface of a head-mounted wearable terminal in a real space, for example, work can be easily performed in an open space where neither a display nor a partition is installed. In such an environment, it is normal that no telephone is installed, so communication using a mobile phone is also important.
Therefore, an object of the present invention is to utilize a wearable terminal in a call by mobile communication.

In order to achieve the above object, the present invention is a head-mounted wearable terminal, which corresponds to a transmissive display surface located in the field of view of the user wearing the terminal and an operation of the operation device by the user. The display control unit that superimposes the screen of the processing device that operates on the background and displays it on the display surface, and the communication means that performs mobile communication acquire the voice data received and release the received voice indicated by the voice data. Provided is a wearable terminal provided with a receiver output unit that outputs to a sound means.

Further, a generation unit that generates voice data indicating the voice of a sentence that is input using the operation device and displayed on the display surface, and an utterance output that outputs the generated voice data as utterance data of mobile communication. It may be provided with a part.

Further, the sentence data indicating the sentence input by using the operation device and displayed on the display surface is generated from the sentence data to generate the voice data indicating the voice of the sentence, and the voice data is used as the speech of the mobile communication. A text output unit that outputs data to a voice processing device that transmits data may be provided.

In addition, a specific unit that identifies the source user of the incoming call received by the communication means and a request unit that requests a connection to the destination of the user registered in the real-time communication service used by the specified user. And may be provided.

Further, it may be provided with a vibrating unit that vibrates when there is an incoming call to the communication means.

Further, the display control unit may display an image showing the incoming call on the display surface when there is an incoming call to the communication means.

Further, the display control unit does not display the image when the importance of the screen displayed on the display surface is equal to or higher than a predetermined reference when there is an incoming call to the communication means, and the vibration The unit may vibrate when the importance is at least the reference or higher.

Further, when the operation frequency to the operation device when there is an incoming call to the communication means is equal to or higher than the threshold value, the image is not displayed, and the vibrating unit is used when the operation frequency is at least the threshold value or higher. May vibrate.

Further, the present invention operates according to the wearable terminal, the operation device operated by the user wearing the wearable terminal, and the operation to the operation device, and the screen of the own device is displayed on the wearable terminal. A display system including a processing device is provided.

According to the present invention, the wearable terminal can be utilized in a call by mobile communication.

The figure which shows the whole structure of the ARPC system which concerns on Example. Diagram showing the hardware configuration of a personal computer Diagram showing the hardware configuration of wearable glasses Diagram showing the configuration of the projection device Diagram showing the functional configuration realized by the ARPC system Diagram showing an example of the displayed incoming image Diagram showing an example of the displayed input image The figure which shows an example of the operation procedure in a call processing Diagram showing the functional configuration realized in the modified example Diagram showing the functional configuration realized in the modified example Diagram showing an example of the displayed RTCS screen Diagram showing an example of importance table

[1] Example FIG. 1 shows the overall configuration of the ARPC system 1 according to the embodiment. The ARPC system 1 is a system for using a PC (Personal Computer: a personal computer (also referred to as a personal computer)) by using AR (Augmented Reality) technology.

The ARPC system 1 includes a personal computer 10 and a wearable glass 20. The personal computer 10 is a so-called personal computer, and includes an operation device such as a keyboard. The personal computer 10 of the present embodiment is integrated with the keyboard, and parts constituting the personal computer main body such as a processor are stored in the housing of the keyboard. The personal computer 10 performs operations such as execution of processing according to the operation on those operating devices. The personal computer 10 is an example of the "processing device" of the present invention.

The wearable glass 20 is a wearable terminal that is worn on the user's head and used (that is, a head-mounted type), and specifically, is a glasses-type wearable terminal. The wearable glass 20 is an example of the "wearable terminal" of the present invention. In this embodiment, the personal computer 10 and the wearable glass 20 are connected by wire as shown in FIG. The connection between the two devices may be wireless.

The wearable glass 20 has a display surface through which the background can be seen through, and the screen of the personal computer 10 is displayed on the display surface. The operation device included in the personal computer 10 is operated by the user who sees the displayed screen, that is, the user who wears the wearable glass 20. In this way, the personal computer 10 operates in response to an operation on the operating device, and the screen of the own device is displayed on the wearable glass 20.

FIG. 2 shows the hardware configuration of the personal computer 10. The personal computer 10 is a computer including a processor 11, a memory 12, a storage 13, a communication device 14, an input device 15, a bus 16, and a keyboard housing 17 for storing the devices. The word "device" here can be read as a circuit, a device, a unit, or the like. Further, each device may be included one or more, or some devices may not be included.

The processor 11 operates, for example, an operating system to control the entire computer. The processor 11 may be composed of a central processing unit (CPU) including an interface with a peripheral device, a control device, an arithmetic unit, a register, and the like. Further, the processor 11 reads a program (program code), a software module, data, and the like from the storage 13 and / or the communication device 14 into the memory 12, and executes various processes according to these.

The number of processors 11 that execute various processes may be one, two or more, and two or more processors 11 may execute various processes simultaneously or sequentially. Further, the processor 11 may be mounted on one or more chips. The program may be transmitted from the network over a telecommunication line.

The memory 12 is a computer-readable recording medium, and is composed of at least one such as a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and a RAM (Random Access Memory). May be done. The memory 12 may be referred to as a register, a cache, a main memory (main storage device), or the like. The memory 12 can store the above-mentioned program (program code), software module, data, and the like.

The storage 13 is a recording medium that can be read by a computer, and is, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, an optical magnetic disk (for example, a compact disk, a digital versatile disk, Blu-). It may consist of at least one such as a ray® disk), a smart card, a flash memory (eg, a card, stick, key drive), a floppy® disk, a magnetic strip, and the like. The storage 13 may be referred to as an auxiliary storage device.

The storage medium described above may be, for example, a database, server or other suitable medium including the memory 12 and / or the storage 13. The communication device 14 is hardware (transmission / reception device) for communicating between computers via a wired and / or wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, or the like. In this embodiment, the communication device 14 includes a mobile communication device 141. The mobile communication device 141 is hardware that performs mobile communication via a base station, and is an example of the "communication means" of the present invention.

The input device 15 is an operation device that accepts an operation for inputting information. In this embodiment, the input device 15 includes a keyboard 151 and a mouse 152 (which may be a touch pad) as operating devices. Each device such as the processor 11 and the memory 12 is accessible to each other via the bus 16 for communicating information. The bus 16 may be composed of a single bus or may be composed of different buses between the devices.

Further, the personal computer 10 or the like includes hardware such as a microprocessor, a digital signal processor (DSP: Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). It may be configured to include, and a part or all of each functional block may be realized by the hardware. For example, the processor 11 may be implemented on at least one of these hardware.

FIG. 3 shows the hardware configuration of the wearable glass 20. The wearable glass 20 is a computer including a processor 21, a memory 22, a storage 23, a communication device 24, an input device 25, a bus 26, a projection device 30, and a frame 40. The processor 21 to the communication device 24 are of the same type as the device of the same name shown in FIG. 2, although there are differences in performance and specifications.

The input device 25 includes, for example, a touch pad as an operation device that can be operated by a user wearing the wearable glass 20 without looking at his / her own hand. The projection device 30 projects the image indicated by the image data supplied from the external device onto the eyes of the user wearing the wearable glass 20. The projection device 30 includes a projection unit 31 and an optical path forming unit 32. The projection unit 31 projects light indicating the above-mentioned image. The optical path forming unit 32 forms an optical path from the projection unit 31 to the user's eye, and guides the light projected by the projection unit 31 to the user's eye.

FIG. 4 shows the configuration of the projection device 30. In FIG. 4, a projection unit 31L that projects light to the left eye and a projection unit 31R that projects light to the right eye are shown. Further, as members included in the optical path forming unit 32, a light guide plate 321, mirrors 322L and 322R (when they are not distinguished, they are referred to as "mirror 322"), and beam splitters 323L and 323R (when they are not distinguished, "beam splitter"). 323 ").

The light guide plate 321 is an elongated plate-shaped member. When the wearable glass 20 is attached to the user, the light guide plate 321 is located in front of the user's eyes, and the longitudinal direction of the light guide plate 321 is along the direction in which the user's eyes are lined up, that is, the left-right direction for the user. It has become. In the following, when the arrangement of each part is represented, it is represented by whether it is on the left or right side when viewed from the user wearing the wearable glass 20. For example, the projection unit 31L is arranged on the left side of the user's eye, and the projection unit 31R is arranged on the right side of the user's eye.

The mirror 322L is provided at the left end inside the light guide plate 321 and the mirror 322R is provided at the right end inside the light guide plate 321. The mirror 322L reflects the projected light projected from the projection unit 31L and incident on the light guide plate 321 toward the right side (the side where the beam splitter 323L is located). The mirror 322R reflects the projected light projected from the projection unit 31R and incident on the light guide plate 321 toward the left side (the side where the beam splitter 323R is located).

In FIG. 4 and the following figures, the optical path of the projected light is represented by using a line and an arrow (indicating the traveling direction of the projected light). The projected light reflected by both mirrors travels while reflecting inside the light guide plate 321. However, in order to make the figure easier to see, the optical path is represented by a simple straight line. The reflected projected light reaches the beam splitter 323 on both the left and right sides. The beam splitter 323 is a plate-shaped member (also referred to as a half mirror) that transmits a part of the incident light and reflects a part of the incident light.

The beam splitter 323L is arranged in the field of view of one eye (left eye) of the user wearing the own terminal. The beam splitter 323L has a reflecting surface 324L that reflects the projected light that has been reflected by the mirror 322L and has traveled through the light guide plate 321. On the opposite side of the reflecting surface 324L, a transmitting surface 325L that transmits light from the outside world is provided. Have. The projected light reflected by the reflecting surface 324L reaches the user's left eye, and the image indicated by the projected light is reflected on the user's left eye.

The beam splitter 323R is arranged in the field of view of one eye (right eye) of the user wearing the terminal. The beam splitter 323R has a reflecting surface 324R that reflects the projected light that has been reflected by the mirror 322R and has traveled through the light guide plate 321. On the opposite side of the reflecting surface 324R, a transmitting surface 325R that transmits light from the outside world is provided. Have. The projected light reflected by the reflecting surface 324R reaches the user's right eye, and the image indicated by the projected light is reflected on the user's right eye.

As described above, the reflecting surfaces 324L and 324R are surfaces on which the image indicated by the projected light is displayed. Further, when viewed from the user, the background beyond the reflective surfaces 324L and 324R can be seen through. That is, the reflective surfaces 324L and 324R are transmissive display surfaces located in the field of view of the user wearing the own terminal (wearable glass 20), and are an example of the "display surface" of the present invention. In the following, when these are not distinguished, they are referred to as "display surface 324".

The frame 40 is a component that supports the projection device 30 and fixes (attaches) the terminal to the user's head. The frame 40 is, for example, a metal or plastic part having a shape similar to the frame of eyeglasses. The frame 40 is provided with a vibration device 41 and a connector 42. The vibration device 41 is, for example, a device having a motor and a weight, and generating vibration by the motor rotating the weight (also referred to as a “vibrator”). The vibrating device 41 is an example of the "vibrating portion" of the present invention.

The connector 42 is a component for connecting a sound emitting means for outputting sound (for example, the earphone 50 in the figure). The earphone 50 converts an analog signal indicating sound supplied via the connector 42 into sound and outputs it, that is, emits sound. The earphone 50 has a cord having a length that reaches the ear of a user who wears the wearable glass 20 on the head while being connected to the connector 42.

Each device included in the ARPC system 1 stores a program provided by this system, and each processor executes the program to control each part, thereby realizing the function group described below.
FIG. 5 shows a functional configuration realized by the ARPC system 1. The personal computer 10 includes a screen data generation unit 101, a mobile communication reception processing unit 102, an utterance input reception unit 103, an utterance text data generation unit 104, and a mobile communication transmission processing unit 105. The wearable glass 20 includes a screen display control unit 201, a data acquisition unit 202, a reception vibration control unit 203, an received voice data output unit 204, an utterance voice data generation unit 205, and an utterance voice data output unit 206. Be prepared.

The screen display control unit 201 superimposes the screen of the processing device that operates in response to the operation of the operation device by the user on the background and displays it on the display surface 324 (reflection surfaces 324L and 324R shown in FIG. 4). The screen display control unit 201 is an example of the "display control unit" of the present invention. In this embodiment, the screen display control unit 201 displays the screen (hereinafter referred to as “personal computer screen”) of the personal computer 10 (a processing device that operates in response to the operation of the keyboard 151 and the mouse 152) on the display surface 324.

For example, when the user performs an operation to display the screen, the screen display control unit 201 transmits the request data requesting the screen data indicating the personal computer screen to the personal computer 10. The personal computer 10 supplies the received request data to the screen data generation unit 101. The screen data generation unit 101 generates screen data requested by the supplied request data, that is, screen data indicating a personal computer screen of the own device.

The screen data generation unit 101 transmits the generated screen data to the wearable glass 20. The screen display control unit 201 displays the personal computer screen indicated by the transmitted screen data on the display surface 324. So far, we have explained the functions related to the display of the personal computer screen. From here, the functions related to the call by mobile communication will be described.

The mobile communication reception processing unit 102 performs processing (reception processing) related to reception in a voice call in mobile communication performed by the mobile communication device 141 shown in FIG. For example, when there is an incoming call to the mobile communication device 141, the mobile communication reception processing unit 102 generates a telephone number of the caller and incoming data indicating that the incoming call has been received, and transmits the incoming data to the wearable glass 20. .. Further, when the call is started, the mobile communication reception processing unit 102 generates the received voice data indicating the received call voice (hereinafter referred to as “received voice”) and transmits it to the wearable glass 20.

The data acquisition unit 202 of the wearable glass 20 acquires the data transmitted from the personal computer 10. When the data acquisition unit 202 acquires the incoming data, the acquired incoming data is supplied to the screen display control unit 201 and the reception vibration control unit 203. The screen display control unit 201 causes the display surface 324 to display an image (incoming call image) indicating the incoming call when the incoming call data is supplied, that is, when there is an incoming call to the mobile communication device 141 (communication means). ..

FIG. 6 shows an example of the displayed incoming image. In FIG. 6A, the screen display control unit 201 displays the incoming image C1 as a part of the personal computer screen A1 by a two-dot chain line because the end of the display surface 324 is transparent and cannot be clearly seen by the user. It is displayed in). In FIG. 6B, the screen display control unit 201 is projected from the personal computer screen A1 to display the incoming image C2 at the end of the display surface 324.

If the display surface 324 is sufficiently wide, the screen display control unit 201 may display the incoming image C2 so as not to overlap the personal computer screen A1 at all. In the case of the incoming image C1, since it is displayed on the personal computer screen A1 that the user is paying attention to, it is easy to notice the incoming call, and in the case of the incoming image C2, it is less likely to interfere with viewing the personal computer screen A1. In addition, the telephone number of the caller can be transmitted to any incoming image.

The reception vibration control unit 203 controls the vibration device 41 shown in FIG. 4 so as to vibrate when incoming data is supplied, that is, when there is an incoming call to the mobile communication device 141 (communication means). When the vibrating device 41 vibrates, the frame 40 provided with the vibrating device 41 also vibrates, and the vibration is transmitted to the user's head. This makes it easier for the user to notice the incoming call than when only the incoming call image is displayed.

As shown in FIG. 6, the screen display control unit 201 displays the answer button B1 and the rejection button B2 together with the incoming image (these buttons are also shown in FIG. 6A as a part of the personal computer screen A1. In 6 (b), it is displayed regardless of the personal computer screen A1). When the rejection button B2 is operated, the mobile communication reception processing unit 102 ends the reception processing, and when the answer button B1 is operated, the mobile communication reception processing unit 102 starts a call and starts generating and transmitting the received voice data. .. When the data acquisition unit 202 acquires the received voice data, the data acquisition unit 202 supplies the acquired received voice data to the received voice data output unit 204.

The received voice data output unit 204 acquires the supplied received voice data, that is, the received voice data received by the mobile communication device 141 (communication means), and emits the received voice indicated by the received voice data (the sound receiving means (communication means). In this embodiment, the data is output to the earphone 50) shown in FIG. The received voice data output unit 204 is an example of the “received voice output unit” of the present invention. The received voice data output unit 204 converts the received voice data, which is digital data, into an analog signal and supplies it to the earphone 50 via the connector 42.

The earphone 50 emits the received voice indicated by the supplied analog signal. As a result, the user can hear the received voice by the incoming mobile communication. On the other hand, the reply from the user is made by inputting text, not by his own voice. When the response button B1 is operated, the screen display control unit 201 displays an input screen for inputting the utterance content (in this case, a reply) from the user.

FIG. 7 shows an example of the displayed input image. The screen display control unit 201 displays the character string "Please input the utterance content", the input field D1, and the confirmation button B3 as the utterance content input screen A2. When the user hears the received voice, he / she performs an operation of inputting a reply in the input field D1. The utterance input receiving unit 103 accepts the input of the utterance content on the utterance content input screen.

The utterance input receiving unit 103 receives an input using an operation device (keyboard 151 and mouse 152) included in the own device (personal computer 10). The utterance input receiving unit 103 does not accept voice input regardless of whether or not the own device is equipped with a microphone. This is because when the utterance content is uttered by voice, the content is known to those around.

The utterance input receiving unit 103 supplies the utterance content for which the input is received to the utterance text data generation unit 104. The utterance text data generation unit 104 generates the supplied utterance content, that is, text data indicating a sentence input by using the operation device (hereinafter referred to as “speech text data”). The utterance text data generation unit 104 transmits the generated utterance text data to the wearable glass 20.

The screen display control unit 201 of the wearable glass 20 causes the input field D1 to display the utterance content indicated by the transmitted utterance text data. In the example of FIG. 7, the input utterance content "Hello. It's hot." Is displayed. When the user presses the confirm button B3 in this state, the screen display control unit 201 supplies the utterance text data indicating the displayed utterance content to the utterance voice data generation unit 205.

The utterance voice data generation unit 205 generates utterance voice data indicating the utterance content indicated by the supplied utterance text data, that is, the voice of the sentence input by using the operation device. The utterance voice data generation unit 205 is an example of the “generation unit” of the present invention. The utterance voice data generation unit 205 generates utterance voice data by using a well-known voice synthesis technique, for example, by combining pre-stored phonetic data to generate voice data.

The utterance voice data generation unit 205 supplies the generated utterance voice data to the utterance voice data output unit 206. The utterance voice data output unit 206 outputs the supplied utterance voice data, that is, the utterance voice data generated by the utterance voice data generation unit 205 as the utterance data of the mobile communication. The utterance voice data output unit 206 is an example of the "utterance output unit" of the present invention.

The utterance data of the mobile communication referred to here is data indicating the utterance content to be transmitted to the other party in the mobile communication. In this embodiment, the input is performed by a character string (text), but the utterance voice data indicating the voice is output as the utterance data. The utterance voice data output unit 206 outputs the utterance voice data to the personal computer 10. The personal computer 10 supplies the output voice data to the mobile communication transmission processing unit 105.

The mobile communication transmission processing unit 105 performs processing (transmission processing) related to transmission in a voice call in mobile communication performed by the mobile communication device 141 shown in FIG. In the present embodiment, the mobile communication transmission processing unit 105 performs a process of transmitting the supplied voice data as it is to the mobile communication terminal of the call destination as a transmission process. In this case, the mobile communication terminal of the call destination emits the utterance voice indicated by the utterance voice data transmitted in the same manner as a normal voice call.

Based on the above configuration, the personal computer 10 and the wearable glass 20 perform call processing for making a call by mobile communication.
FIG. 8 shows an example of an operation procedure in call processing. This operation procedure is started, for example, when a user receives an incoming call while wearing the wearable glass 20 and displaying the screen of the personal computer 10.

First, the personal computer 10 (mobile communication reception processing unit 102) generates the caller's telephone number and incoming data indicating that there is an incoming call (step S11), and transmits the generated incoming data to the wearable glass 20 (step S11). Step S12). The wearable glass 20 (reception vibration control unit 203) controls to vibrate the vibration device 41 when incoming data is transmitted (step S13).

Next, the wearable glass 20 (screen display control unit 201) displays an incoming image indicating an incoming call on the display surface 324 (step S14). The operations of steps S13 and S14 may be performed in the opposite order or in parallel. Subsequently, when a response operation is performed on the screen on which the incoming image is displayed, the personal computer 10 accepts the operation (step S21). Then, the personal computer 10 (mobile communication reception processing unit 102) starts a call to generate the received voice data (step S22), and transmits the generated received voice data to the wearable glass 20 (step S23).

The wearable glass 20 (received voice data output unit 204) outputs the received voice indicated by the transmitted received voice data to the sound emitting means (earphone 50) (step S24). Next, the wearable glass 20 (screen display control unit 201) displays the utterance content input screen on the display surface 324 (step S31). The personal computer 10 (utterance input receiving unit 103) accepts the input of the utterance content on the utterance content input screen (step S32).

Next, the personal computer 10 (utterance text data generation unit 104) generates utterance text data indicating the input sentence (step S33) and transmits it to the wearable glass 20 (step S34). The wearable glass 20 (screen display control unit 201) displays the utterance content indicated by the transmitted utterance text data on the display surface 324 (step S35). The wearable glass 20 accepts a confirmation operation on the utterance content input screen (step S36).

Next, the wearable glass 20 (speech voice data generation unit 205) generates utterance voice data indicating the confirmed utterance content, that is, the voice of the sentence input by using the operation device (step S37). Subsequently, the wearable glass 20 (speech voice data output unit 206) outputs the generated utterance voice data to the personal computer 10 as utterance data for mobile communication (step S38).

The personal computer 10 (mobile communication transmission processing unit 105) performs a process of transmitting the transmitted utterance voice data as it is to the mobile communication terminal of the call destination (process related to transmission in a voice call in mobile communication). ) (Step S39). After that, the operations from steps S22 to S39 are repeated, and when the user performs an operation to end the call, this operation procedure ends.

In this embodiment, the voice of the call by the communication means (mobile communication device 141 of the personal computer 10) for performing mobile communication is output from the wearable glass 20. Further, the operation of rejecting or answering the call is also performed through the screen displayed on the wearable glass 20. As described above, according to the present embodiment, the wearable terminal (wearable glass 20) can be utilized in a call by mobile communication.

Further, in the present embodiment, the user who has started the mobile communication can respond by inputting the utterance content with the operation device (keyboard 151 and mouse 152) of the personal computer 10 without emitting a voice. As a result, it is possible to make a call by mobile communication without the contents being known to a third party around.

[2] Modifications The above-mentioned examples are merely examples of the implementation of the present invention, and may be modified as follows.

[2-1] Wearable terminal The wearable terminal of the present invention is not limited to the wearable glass 20. For example, as a method for forming an optical path, a well-known method other than the method shown in FIG. 4 (for example, a method using polarization) may be used. Further, although the wearable glass 20 is a glasses type, it may be a goggles type or a monocle type (in that case, the display surface becomes one). In short, any type of wearable terminal may be used as long as it is a head-mounted type and has a transmissive display surface located in the user's field of view.

[2-2] Processing device The processing device of the present invention is not limited to the personal computer 10. For example, it may be an ordinary desktop personal computer instead of a keyboard integrated type. Further, a processing device having a display surface such as a notebook computer, a smartphone or a tablet terminal may be used.

In that case, the display on the display surface of the processing device main body may be turned off, or the processing device main body may be put in a bag so that a third party cannot see the display surface of the processing device main body. .. The screen displayed on the wearable terminal may be any screen of the processing device, and is not limited to the screen of the personal computer. Any processing device may be any device as long as it operates in response to the operation of the operation device (in the case of a smartphone or tablet terminal, an external keyboard or the like) by the user.

[2-3] Communication means In the embodiment, the personal computer 10 as a processing device is provided with a communication means for mobile communication, but the present invention is not limited to this. For example, the above-mentioned other processing device (smartphone or the like) may be provided with a communication means, or the wearable glass 20 which is a wearable terminal may be provided. Regardless of which device is provided with the communication means, it is sufficient that the device and the wearable terminal can exchange data related to mobile communication.

[2-4] Sound Discharging Means The sound emitting means for the wearable terminal to output the received voice is not limited to that described in the embodiment (external earphone). For example, they may be headphones or speakers (bone conduction speakers, etc.), or they may be built into a wearable terminal. In short, any sound emitting means may be used as long as the user wearing the wearable terminal on his / her head can hear the emitted sound. However, in order to prevent information leakage, it is desirable that the emitted call voice is not heard by surrounding third parties.

[2-5] Spoken Voice Data Generation Device In the embodiment, the wearable glass 20 generates the utterance voice data, but another device may generate the utterance voice data.
FIG. 9 shows the functional configuration realized by this modification. In FIG. 9, a personal computer 10a provided with an utterance voice data generation unit 106 in addition to each unit shown in FIG. 5, and an utterance text data output unit 207 instead of the utterance voice data generation unit 205 and the utterance voice data output unit 206 shown in FIG. The wearable glass 20a provided with the above is represented.

When the confirmation button B3 is operated in the state shown in FIG. 7, the screen display control unit 201 of this modification supplies the utterance text data indicating the displayed utterance content to the utterance text data output unit 207. The utterance text data output unit 207 outputs the supplied utterance text data, that is, the text data indicating the text input by using the operation device and displayed on the display surface 324 to the personal computer 10a. The utterance text data output unit 207 is an example of the "text output unit" of the present invention.

The personal computer 10a supplies the output utterance text data (sentence data) to the utterance voice data generation unit 106. The utterance voice data generation unit 106 is the utterance voice data that indicates the voice of the supplied utterance text data, that is, the sentence (in the example of FIG. 7, the sentence "Hello. It's hot.") From the above-mentioned sentence data. To generate. Similar to the utterance voice data generation unit 205, a well-known voice synthesis technique is used to generate the utterance voice data.

The utterance voice data generation unit 106 supplies the generated utterance voice data to the mobile communication transmission processing unit 105. Similar to the embodiment, the mobile communication transmission processing unit 105 performs a process of transmitting the supplied utterance voice data as it is to the mobile communication terminal of the call destination as a transmission process. The personal computer 10a that performs processing on the spoken voice (generation processing and transmission processing of the spoken voice data) in this way is an example of the "speech processing device" of the present invention.

In this modification, as described above, the generation processing of the spoken voice data is performed by the processing device (personal computer 10a) instead of the wearable terminal. Generally, since a wearable terminal is made lightweight and thin, the processing capacity as an information processing device is suppressed to be lower than that of a normal personal computer or the like. In this modification, by performing the utterance voice data generation processing on the processing device having a higher processing capacity than the wearable terminal, the utterance voice is made higher quality than the case where this processing is performed on the wearable terminal. In addition, it is possible to reduce the time lag due to the processing of the spoken voice.

The voice processing device described above does not have to be a processing device whose screen is displayed on the wearable terminal, may be an external server device, or is a device other than a processing device such as a smartphone owned by the user. You may. Even in such cases, if the voice processing device has a higher processing capacity than the wearable terminal, the effect of the above-described modification can be obtained.

[2-6] Calls other than voice In each of the above examples, although the user does not speak, the call is made by the spoken voice data generated by the wearable terminal or the like, but the call is not limited to this.
FIG. 10 shows the functional configuration realized by this modification. In FIG. 10, a personal computer 10b including each unit except the mobile communication transmission processing unit 105 shown in FIG. 5 is shown.

Further, in FIG. 10, a wearable glass 20b including a source user specifying unit 208 and a service processing unit 209 instead of the utterance voice data generation unit 205 and the utterance voice data output unit 206 shown in FIG. 5 is shown. In this modification, the data acquisition unit 202 supplies the acquired incoming data to the source user identification unit 208. The sender user identification unit 208 identifies the sender user of the incoming call received by the mobile communication device 141 (communication means). The source user identification unit 208 is an example of the "specific unit" of the present invention.

The caller user identification unit 208 identifies the caller user by using, for example, a user table in which the caller's telephone number and the user are associated with each other, which is registered in advance by the user of the own device (wearable glass 20b). .. The source user specifying unit 208 may inquire of an external device having information such as a user table to specify the source user. The source user identification unit 208 supplies the service processing unit 209 with identification information that identifies the specified user.

The service processing unit 209 processes related to the real-time communication service (RTCS: Real Time Communication Service) used by the user indicated by the supplied identification information, that is, the user specified by the source user identification unit 208 (hereinafter, “service”). "Processing"). The real-time communication service (RTCS) is a service that enables real-time communication using a data communication line, and is sometimes called "chat" or the like.

The information exchanged by RTCS is mainly text data, but images, moving images, and the like may also be exchanged. However, voice exchange is excluded. For example, the service processing unit 209 performs a process of requesting a connection to the destination of the user (a user specified by the source user identification unit 208) registered in the RTCS as a service process. The service processing unit 209 is an example of the "request unit" of the present invention.

The service processing unit 209 is supplied by using, for example, a service table in which the RTCS used by another user registered in advance by the user of the own device and the user ID used in the RTCS are associated with each other. The RTCS and the user ID associated with the user indicated by the identification information are specified. This user ID is an example of the above-mentioned "destination of a user registered in RTCS".

It is assumed that the wearable glass 20b stores a program (browser program or application program) for using the main RTCS. The service processing unit 209 starts a program for using the specified RTCS and performs a process of requesting a connection to the specified user ID. Specifically, the service processing unit 209 performs, for example, a search process for searching the user ID on the activated RTCS screen as a process for requesting a connection.

Further, the service processing unit 209 performs a request process for requesting the start of chat with the user of the user ID found by searching as a process for requesting a connection. Note that these processes (search process and request process) are examples, and in addition to this, the process normally performed when starting communication with a user in the RTCS to be used may be performed as a process for requesting a connection. .. When the RTCS program is started, the screen display control unit 201 causes the corresponding RTCS screen to be displayed on the display surface 324.

FIG. 11 shows an example of the displayed RTCS screen. In FIG. 11A, the chat partner search screen A3 is displayed on the display surface 324 of the wearable glass 20b. On the chat partner search screen A3, the search key input field D2 and the search result E1 (that is, the result of the search process) are displayed. When the service processing unit 209 performs the process of selecting the search result E1 as the above-mentioned request process, the screen display control unit 201 displays the chat screen A4 with the searched user (“user A” in this example) in FIG. 11. Display as shown in (b).

On the chat screen A4, the input field D3 for the utterance content is displayed. When the user inputs the utterance content in the input field D3, the utterance input receiving unit 103 accepts the input of the utterance content in the input field D2. Then, when the user presses the confirm button, the utterance text data generation unit 104 generates utterance text data indicating the input sentence, and supplies the generated utterance text data to the screen display control unit 201.

The screen display control unit 201 displays the balloon image E2 including the character string "Hello. It's hot!", Which is the utterance content indicated by the supplied utterance text data, as shown in FIG. 11 (c). The utterance text data generation unit 104 also supplies the generated utterance text data to the service processing unit 209. The service processing unit 209 transmits the supplied utterance text data to the destination (destination registered in RTCS) of the user A specified as described above.

The utterance content indicated by the transmitted utterance text data is displayed on the display means of the terminal used by the user A. In FIG. 11D, the chat screen A5 thus displayed is shown. On the chat screen A5, an avatar E3 indicating the user (“user B” in this example) who is the chat partner, and a balloon image E4 including the character string “Hello. It's hot!” Which is the utterance content of the user B. Is displayed.

Further, since the chat screen A5 is a screen for chatting, the input field D4 for the utterance content is displayed as in the chat screen A4. For example, the user A may input the utterance content in the input field D4 and continue the conversation by chat. In this case, the connection by mobile communication may be terminated. Further, the user A may continue the voice conversation in the connected mobile communication without inputting to the input field D4. In that case, the user B talks while listening to the voice of the user A and inputting the utterance content in the input field D3.

In either case, the user B does not state his / her utterance by voice, and the displayed balloon image E2 is also displayed on the display surface 324 of the wearable glass 20b that only the user B can see. It is possible to eliminate the risk that the content of the utterance will be known to surrounding third parties. Further, since the voice of the user A is also heard only by the user B from the earphone 50, there is no possibility that the utterance content of the user A will be known to a third party. Further, since the chat connection with the user A is automatically performed as described above, the conversation using the RTCS can be smoothly started.

In the above example, the wearable glass 20b executes the program for using the RTCS, but the present invention is not limited to this, and the personal computer 10b may execute this program. In that case, the function corresponding to the service processing unit 209 is also realized in the personal computer 10b, and the service processing unit 209 may perform a process of requesting a connection via the function on the personal computer 10b side.

[2-7] Incoming Call Notification Method In the embodiment, when there is an incoming call, the user is notified by both vibration and the incoming call image, but the present invention is not limited to this, and for example, only one of them may be notified. .. At that time, which notification may be given may be different depending on the situation.

In this modification, the screen display control unit 201 displays the screen displayed on the display surface 324 when the incoming data is supplied, that is, when there is an incoming call to the mobile communication device 141 (communication means). Determine the importance. The screen display control unit 201 determines the importance according to, for example, the type of the program running on the displayed personal computer screen. The screen display control unit 201 makes a determination using, for example, an importance table in which the type of the program and the importance are associated with each other.

FIG. 12 shows an example of the importance table. In the example of FIG. 12, the importance of "for viewing files and reading emails" is "low", the importance of "for document creation" is "medium", and the importance of "for business use and video playback" is "high". Is associated with. The screen display control unit 201 determines the importance associated with the type of the program activated at the time of incoming call in the importance table as the importance of the screen.

For example, when the screen display control unit 201 determines the importance of "low", the incoming image described in the embodiment is displayed, and when the importance of "medium" and "high" is determined, the incoming image is displayed. Is not displayed, and the determination result is notified to the vibration control unit 203 at the time of reception. Upon receiving this notification, the receiving vibration control unit 203 controls and vibrates the vibration device 41 in the same manner as in the embodiment. As described above, in this modification, when the determined importance is equal to or higher than a predetermined reference (in this case, “medium”), the vibration device 41 vibrates and the screen display control unit 201 does not display the incoming image. It works like this.

When the incoming image is displayed, as shown in FIG. 6, the incoming image may overlap and part of the information displayed on the personal computer screen may not be visible. In this modification, by controlling the display of the incoming image as described above, when the screen with high importance is displayed, the incoming image does not get in the way, and the incoming image vibrates even if it is not displayed. Makes you aware of incoming calls.

The importance shown in FIG. 12 is merely an example, and is not limited to this. For example, a different importance may be determined based on the user's request, or the user may be able to set the importance. Further, when the importance of the importance is less than the predetermined standard is determined, the incoming image is displayed as described above, but the vibrating device 41 may or may not be vibrated together. In this modification, the vibrating device 41 may vibrate when the determined importance is at least a predetermined reference or higher.

The method of notifying an incoming call is not limited to this. For example, the notification method may differ depending on the frequency of operation to the operation device when an incoming call arrives. In this case, for example, the screen display control unit 201 operates the operation device (keyboard 151 and mouse 152) when the incoming data is supplied, that is, when there is an incoming call to the mobile communication device 141 (communication means). Calculate the frequency.

The screen display control unit 201 stores, for example, an operation time when an operation is performed on the operation device, and when incoming data is supplied, in a unit period (for example, 10 seconds, etc.) before that time. The number of stored operation times is calculated as the operation frequency. The screen display control unit 201 displays an incoming call image when the operation frequency is less than a predetermined threshold value, and receives the calculation result without displaying the incoming call image when the operation frequency is calculated above the threshold value. Notify the hour vibration control unit 203.

Upon receiving this notification, the receiving vibration control unit 203 controls and vibrates the vibration device 41 in the same manner as in the embodiment. As described above, in this modification, when the calculated operation frequency is equal to or higher than the threshold value, the vibration device 41 vibrates and the screen display control unit 201 operates so as not to display the incoming image.

In situations where the user is busy performing operations, he is often more focused on viewing the screen than in situations where there are few operations. In this modification, the incoming image is not disturbed in such a busy operation of the user, and the incoming image is noticed by vibration even if the incoming image is not displayed.

[2-8] Devices that realize each part The devices that realize each function shown in FIGS. 5 and the like are not limited to the devices shown in those figures. For example, the wearable glass 20 or another external device may realize the utterance text data generation unit 104 included in the personal computer 10. Further, the operation performed by one function may be shared by two or more functions. For example, the display control of the personal computer screen and the display control of the incoming image performed by the screen display control unit 201 may be shared by different functions. In short, it suffices if these functions are realized in the entire ARPC system.

[2-9] Category of Invention The present invention can be regarded as a processing device such as a personal computer and a wearable terminal such as a wearable glass, as well as a display system such as an ARPC system provided with these devices. Further, the present invention can be regarded as an information processing method for realizing the processing performed by each device, and also as a program for operating a computer that controls each device. This program may be provided in the form of a recording medium such as an optical disk that stores it, or may be provided in the form of being downloaded to a computer via a network such as the Internet and installed and made available. May be done.

[2-10] Processing Procedures, etc. The order of the processing procedures, sequences, flowcharts, etc. of each embodiment described in the present specification may be changed as long as there is no contradiction. For example, the methods described herein present elements of various steps in an exemplary order and are not limited to the particular order presented.

[2-11] Handling of input / output information and the like The input and output information and the like may be stored in a specific location (for example, a memory) or may be managed by a management table. Information to be input / output may be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.

[2-12] Software Software, whether referred to as software, firmware, middleware, microcode, hardware description language, or by any other name, is an instruction, instruction set, code, code segment, program code, program. , Subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc. should be broadly interpreted.

Further, software, instructions, and the like may be transmitted and received via a transmission medium. For example, the software may use wired technology such as coaxial cable, fiber optic cable, twist pair and digital subscriber line (DSL) and / or wireless technology such as infrared, wireless and microwave to website, server, or other. When transmitted from a remote source, these wired and / or wireless technologies are included within the definition of transmission medium.

[2-13] Information, Signals The information, signals and the like described herein may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.

[2-14] System, Network The terms "system" and "network" used herein are used interchangeably.

[2-15] Meaning of "based on" The term "based on" as used herein does not mean "based on" unless otherwise stated. In other words, the statement "based on" means both "based only" and "at least based on".

[2-16] "and", "or"
In the present specification, for configurations that can be implemented by either "A and B" or "A or B", the configuration described in one expression may be used as the configuration described in the other expression. For example, when "A and B" are described, they may be used as "A or B" as long as they are not inconsistent with other descriptions and can be carried out.

[2-17] Variations of Aspects, etc. Each of the examples described in the present specification may be used alone, in combination, or may be switched and used according to the execution. Further, the notification of predetermined information (for example, the notification of "being X") is not limited to the explicit one, but is performed implicitly (for example, the notification of the predetermined information is not performed). May be good.

Although the present invention has been described in detail above, it is clear to those skilled in the art that the present invention is not limited to the examples described herein. The present invention can be implemented as modifications and modifications without departing from the spirit and scope of the present invention as determined by the description of the scope of claims. Therefore, the description of the present specification is for the purpose of illustration and does not have any limiting meaning to the present invention.

1 ... ARPC system, 10 ... PC, 20 ... wearable glass, 30 ... projection device, 40 ... frame, 41 ... vibration device, 42 ... connector, 50 ... earphone, 101 ... screen data generator, 102 ... mobile communication reception processing Unit, 103 ... Speaking input reception unit, 104 ... Speaking text data generation unit, 105 ... Mobile communication transmission processing unit, 106 ... Speaking voice data generation unit, 141 ... Mobile communication device, 151 ... Keyboard, 152 ... Mouse, 201 ... screen display control unit, 202 ... data acquisition unit, 203 ... reception vibration control unit, 204 ... received voice data output unit, 205 ... spoken voice data generation unit, 206 ... spoken voice data output unit, 207 ... spoken text data output Unit, 208 ... Source user identification unit, 209 ... Service processing unit, 324 ... Display surface.

Claims (8)

It is a head-mounted wearable terminal,
A transparent display surface located in the field of view of the user wearing the terminal,
A display control unit that superimposes the screen of a processing device that operates according to the operation of the operation device by the user on the background and displays it on the display surface.
A receiving output unit that acquires voice data received by a communication means that performs mobile communication and outputs the received voice indicated by the voice data to the sound emitting means .
Sentence data indicating a sentence input using the operation device and displayed on the display surface is used to generate voice data indicating the voice of the sentence from the sentence data, and the voice data is used as speech data for mobile communication. With the text output unit that outputs to the voice processing device to be transmitted
Wearable terminal equipped with. A generation unit that generates voice data indicating the voice of a sentence input using the operation device and displayed on the display surface, and a generation unit.
The wearable terminal according to claim 1, further comprising an utterance output unit that outputs the generated voice data as utterance data for mobile communication. A specific unit that identifies the source user of the incoming call received by the communication means, and
The wearable terminal according to claim 1 or 2 , further comprising a requesting unit that requests a connection to the destination of the user registered in the real-time communication service used by the specified user. The wearable terminal according to any one of claims 1 to 3 , further comprising a vibrating unit that vibrates when there is an incoming call to the communication means. The wearable terminal according to claim 4 , wherein the display control unit displays an image indicating the incoming call on the display surface when there is an incoming call to the communication means. The display control unit does not display the image when the importance of the screen displayed on the display surface is equal to or higher than a predetermined standard when there is an incoming call to the communication means.
The wearable terminal according to claim 5 , wherein the vibrating unit vibrates when the importance is at least the reference or higher. When the operation frequency to the operation device when there is an incoming call to the communication means is equal to or more than the threshold value, the image is not displayed.
The wearable terminal according to claim 5 or 6 , wherein the vibrating unit vibrates when the operation frequency is at least the threshold value or higher. The wearable terminal according to any one of claims 1 to 7 .
An operating device operated by a user wearing the wearable terminal and
A display system including a processing device that operates in response to an operation on the operation device and displays the screen of the own device on the wearable terminal.

Images