JP5974008B2 - Wireless interface such as Bluetooth (registered trademark) with power management function for head mounted display - Google Patents

Description

This application is related to US Provisional Patent Application No. 61 / 384,419 filed on September 20, 2010 and US Provisional Patent Application No. 61 / 384,543 filed on September 20, 2010. Insist on the benefits of the issue. The entire teachings of the above application are incorporated herein by reference.

  The present disclosure relates to a head mounted display (HMD) system or other spectacle-type video display device having a wireless front end that controls activation of other system components.

  Recently developed microdisplays can provide large formats, high resolution color photos and streaming video in a minimal form factor. One use of such a display is in a head mounted display (HMD) device worn on the user's face or head as well as glasses or headphones. Electronic devices incorporated into such devices have become extremely sophisticated and are now integrated data processors, wireless interfaces and other input devices such as head tracking accelerometers, cameras, voice recognition circuits and software and other Components can be included.

  In a preferred environment, the head mounted display (HMD) system includes a high resolution micro display, microphone, speaker, wireless interface, wireless network interface, camera (visible light, infrared, etc.), audio device, keypad, game controller device. And auxiliary interfaces such as other peripheral devices.

  Wireless interfaces such as Bluetooth®, WiFi, cellular, etc. are the front end of the system and controls when starting other system components. When the HMD system is first switched on, it can operate with minimal functionality and provide only speaker and microphone functions, similar to a Bluetooth® headset. A paired device such as a Bluetooth® compatible mobile phone (ie, Apple iPhone or Google Android smartphone) can be used to transmit voice conversations in a hands-free mode. In this mode, other functions of the HMD system remain in hibernation mode until verbal commands that activate other system components and functionality are recognized by the headset.

  For example, when a Bluetooth front end detects a received data stream having an audio component, the audio component is passed to a speech recognition function embodied as a circuit or software. When the voice recognition function determines that the audio component is a verbal command to a system that requires video output, the front end causes the embedded data processor to turn on any other peripheral device such as a video display. When the video display is no longer needed, the processor and video driver circuit return to sleep mode.

  In another example, the user can speak a command that requires a higher functionality of the onboard processor. For example, the user can request to read a list of today's “to do” tasks. The list can be generated by an application program such as Microsoft Outlook running on the host computer. To execute this command, a text-to-speech conversion circuit that activates an embedded processor, accesses a local mass storage device, fetches list information to do, and generates a speech signal corresponding to spoken language Is also needed.

  In another example, a user may request to display an agenda document attached to a webmail message as a Microsoft Word document. This requires starting an embedded processor, starting a local microdisplay, and connecting to an external network, such as via Bluetooth®, wireless local area network (WLAN) circuitry and / or cellular modem circuitry. It may be. Those circuits are therefore only energized at that time. The embedded processor then contacts a host machine such as a personal computer to retrieve and display web information.

  To support this functionality, the wireless front-end not only performs speech-to-speech functions or converts gestures to text, but then interprets these speech or gesture commands, or verbal or gesture command A table or other logic is included therein that also allows a reference to the list of components that need to be energized for execution.

  The foregoing will become apparent from the following more specific description of exemplary embodiments of the invention, as illustrated in the accompanying drawings in which like reference numerals refer to like parts throughout the different views. The drawings are not necessarily drawn to scale, but instead focus on exemplifying embodiments of the invention.

1 is a head mounted display (HMD) system including a micro display and other peripheral interfaces. FIG. It is another perspective view which shows the HMD system with which the camera peripheral device was mounted | worn with the user's head. FIG. 2 is a high-level functional diagram showing the HMD and its use as a controller for a host processor such as a personal computer. FIG. 2 is a high level block diagram of components within an HMD assembly. FIG. 2 is a more detailed diagram illustrating electronic components used in one particular embodiment. 6 is a high-level flowchart illustrating an example of how the Bluetooth front end operates and controls as a system controller when various other HMD system components become active. A list of verbal commands, gesture commands or head movement commands and their activation parts of the subsystem.

  A description of exemplary embodiments follows.

  FIG. 1 generally illustrates a head mounted display (HMD) system 100 that includes a frame 1000, a strap 1002, a rear portion 1004, a speaker 1006, a cantilever or arm 1008 and a microdisplay subassembly 1010.

  The HMD 100 may be based on a Golden-i ™ hands-free communication and control device available from Kopin Corporation, located in Taunton, Massachusetts. One side of the device, specifically the opposite side of the microdisplay subassembly 1010, may include a “hot shoe” peripheral port 1020, as described further below, the “hot shoe” peripheral port 1020 corresponds to Connections can be provided to the accessory so that the user can remove various components attached to the HMD 100.

  FIG. 2 is an illustration of the HMD 100 worn on the user's head, with a camera accessory 1060 installed in the port 1020. The camera 1060 can include both audio and video detection and recording functions, and may be, for example, a small form factor similar to a “bullet cam”. As will be described further below, user commands, which may be head movements or voice commands, are provided by the user, thereby controlling display 1010, camera 1060 settings and performing other functions.

  The HMD system may function primarily as either a stand-alone personal computer or a “smartphone” device and / or as a remote control of the host personal computer or other device. For this purpose, the housing contains various electronic circuits and, as will be understood, includes a microcomputer, one or more wireless interfaces, associated memory or other storage, position, motion and / or gesture sensors. It is. Camera position, motion and / or gesture sensors are used to track the user's head or hand position, motion and / or gesture, and the microphone receives voice input commands.

  The wireless interface enables use of the HMD system 100 as a smartphone itself or as a remote control for the host computing device 200 (FIG. 3). For example, the host 200 may be a laptop, mobile phone, RIM® Blackberry (registered) with similar or greater computational complexity and similar or greater network connectivity than the remote control HMD system 100 itself. Trademark), Apple (R) iPhone (R), Google (R) Android (R) or other smartphone devices. The host 200 can further connect to other networks, such as through a wide area network (WAN) connection 210 to the Internet. The HMD system 100 and the host 200 are connected to each other through a suitable wireless connection, such as provided by a Bluetooth link or a wireless local area network (WLAN) 150.

  As shown in FIG. 3, the HMD system 100 includes voice input via a microphone, hand movement via a camera or other optical sensor, and / or an accelerometer or other head movement tracking circuit. Input from the user, such as head movement input via These are converted by the software in the system 100 (and / or in cooperation with the software in the host 200) into commands that the device 100 and / or the host 200 system can recognize. If these are commands for the HMD system 100, the HMD system 100 holds these commands, and if they are commands for the host 200, these commands are on the Bluetooth (registered trademark) and / or WiFi interface 150. Is transmitted to the host 200. The host 200 then interprets these translated commands and performs various functions according to its own operating system / application software.

Among these commands, there may be a command for fetching and displaying a document.
Among these commands, there may be a command for answering a call.
Among other commands, there may be commands that obtain information from the Internet and display it, such as email or web pages.

  Among other commands, there may be a command that selects a field of view within the virtual display and then returns the selection screen to the remote device. A virtual display area that is a maximal format may also be associated with application software running on or within an operating system running on the host 200, and the larger virtual display area within the field of view. It should also be understood that only a portion of is selected by various voice commands and / or head movements supplied to display 1010 on HMD 100.

  For more information on the functions that can be performed by the HMD 100 in response to voice input, movement and / or gesture detection, see “Remote Control of Host Application Motion and Voice Commands” filed May 5, 2010. No. 12 / 774,179 (Attorney Docket No. 0717.2098-001) and “Wireless Hands-Free Computing Headset Width Controls Controls” filed on Feb. 1, 2011. "By Motion, Body Gesture, or Vocal Commands" Country Patent Application No. 13 / 018,999 are described in (Attorney Docket No. 0717.2102-001), the entire contents of each are incorporated herein by reference.

  The HMD system 100 provides additional functionality to other accessories that outperform the camera, such as additional speakers, additional cameras, light sources, and any other electronic components with a built-in battery or hot shoe 1020. You can also

  FIG. 4 is a simplified high-level block diagram of a non-limiting exemplary embodiment of HMD system 100. As shown in FIG. 1, these components are integrated at various locations in the HMD housing. The HMD system 100 may include an ipod® assembly 4000, which includes the aforementioned microdisplay 4010 and one or more microphones 4020. One or more speakers 4030 are placed in a housing earphone near the user's ear (see item 1006 in FIG. 1). The system electronics can be placed on or within the frame at an appropriate location (such as the back 1004), with an Open Media Application Platform (OMAP) processor 4100, an output / audio companion chip 4102, a display driver 4104, a head A tracking circuit board 4106 and a wireless LAN / Bluetooth interface 4108 are included. A power source such as a lithium ion battery 4200 is also located in the housing.

  The head tracking circuit 4106 uses Hall effect sensors, MIM diodes, accelerometers, gyros and / or transducers to move laterally along and around the X, Y and Z axes. A circuit for detecting head movements and gestures such as a rotation gesture may be included.

  In addition, the HMD system 100 can receive input from an external input device such as a wireless mouse, a trackball, or a keyboard that can be wirelessly connected through the Bluetooth (registered trademark) interface 4108. WLAN / BT front end 4108, OMAP 4100 and / or software in the host can be used to interpret hand gestures detected by the camera. A camera board 4060 may additionally provide video input.

  The OMAP processor may include a central processing unit and on-chip memory such as random access memory (RAM) and / or read only memory (ROM), which may include non-volatile memory. OMAP is a Texas Instruments model OMAP 3530 processor or Texas Instruments, Inc. It may be a newer version using a multimedia processor sold by. In general, OMAP is a processor that is more powerful and consumes more power than the WLAN / BT interface 4108.

  In this example, a TPS65950 output / voice companion chip is also available from Texas Instruments, providing voice, USB, keypad control and battery charging functions to the system.

  The WLAN / BT interface 4108 may be a model LBEE 1W8 NEC interface circuit, or a newer wireless module with similar or greater capabilities.

  The display driver may be a model KCD-A910 display driver available from Kopin Corporation located in Westborough, Massachusetts.

  Microdisplay 4010 is also available from Kopin and may include model 113LV, 152LV, 230LV, WQVGA, or other manufacturer acceptable microdisplays.

  The NCS module 4400 processes the received microphone signal to provide a voice recognition function, and generates a voice signal in the voice companion chip 4102.

  FIG. 5 is a more detailed block diagram illustrating the connectivity between the various components of FIG.

  As can be more easily understood from this figure, the WLAN / BT front end 4108 can use a UART interface. This WLAN / BT front end 4108 provides power to the OMAP processor 4100 and correspondingly audio and output companion chip 4102 power on or power off states and other peripheral functions such as cameras, displays, WLAN modems, etc. Programmed to control.

  FIG. 6 is a flowchart illustrating some exemplary functions. Here, the WLAN / Bluetooth front end 4108 can control the application of power to other system components. FIG. 7 is an exemplary table that can be maintained by the front end 4108 to determine when various devices are powered on / off, see the discussion of FIG. 6 below. .

  In the first example, in state 6000, the system component may initially be in hibernation mode (hibernation), and only the Bluetooth headset function (eg, microphone 4020 and speaker 4030) is enabled. . This state 6000 can be entered when the HMD system 100 is first switched on. Thus, for example, in state 6100, if a call is received, the voice input / output function remains active in state 6110 (eg, microphone 4020 remains on and speaker 4030 function is enabled). Become). However, in state 6120, OMAP 4100, display 4010, camera 4060 and other components remain in hibernation mode. When the call ends, processing returns to state 6000.

  In another example, from hibernation state 6000, the voice circuit may detect a specific user voice command such as “read a list of things to do”. From state 6200 to state 6210, the OMAP function is enabled. Then, in state 6220, OMAP can access a local storage area, such as SDRAM memory, access files associated with the user's local task list, and obtain an itemized list of tasks to do today. In state 6240, what to do list data is returned from OMAP 4100 to the voice interface, and what to do list text is converted to voice and then listened on the speaker of headset 100. When this function is complete, state 6500 is entered, then the system returns to hibernation state 6000, and the system will then only function as a Bluetooth headset again and wait for another user command.

  In a further example, in state 6300, the user is www. NYTimes. com with a command indicating that he wants to browse the New York Times homepage. When this command is received, the OMAP function is enabled in state 6310. As described with reference to FIG. 3, the WLAN interface is activated, the OMAP can issue commands to the host processor, and the HTML content of the New York Times home page can be obtained. Upon entering state 6330, the function of the microdisplay 1010 is also enabled, thereby predicting that the user wishes to display a web page. New York Times. com's HTML web page is also fetched in other ways, for example, if the headset itself has the ability to access the Internet through on-board Bluetooth, WLAN circuitry (Wi-Fi), or through an auxiliary cellular modem can do. In any event, once the content is obtained, it is formatted in state 6350 and displayed on the microdisplay. At some point, the user can issue another command that disables the display function (or the command may time out), at which point the hibernation function 6000 is entered again.

  Thus, when the Bluetooth® and / or WiFi front end detects an incoming voice stream with a user voice command, the Bluetooth® and / or WiFi circuitry will automatically activate OMAP and other peripheral devices. Start only what is actually needed to run the command. The Bluetooth (R) and / or WiFi front end thus determines which peripheral devices and / or OMAP need to be energized for each execution of many different user commands, such as in the form of a table or database entry. Information indicating (if any) is retained.

  In yet another example (see FIG. 7), a user's hand gesture can be detected with a camera and an input can be provided to control the application of power to certain components of the system 100. In this state, the voice component function is initially enabled, but it detects verbal commands and reads and interprets hand gestures by activating at least the camera 4060 and OMAP 4100.

  In another example, the user can use head movements as input. In this mode, only the head movement tracking unit 4106 and the display are used, for example, when the user uses head movement to pan (move), zoom or otherwise navigate a large format virtual display area. It may be a related component that can be enabled.

  In this way, the functions of the HMD system are controlled via appropriate verbal commands, gesture commands or commands from the host computer, interpreted by the Bluetooth® and / or WiFi interface, and the subsystem electronics Activate only that part as needed. When the activity ends, another verbal command or timeout can return the HMD system to sleep mode. At this point, the Bluetooth® and / or WiFi circuitry will return to the hibernation state and wait for an external contact signal such as a call, another voice command, or a computer interface command over the WLAN.

  The teachings of all patents, published applications and references cited herein are incorporated by reference in their entirety.

While the invention has been particularly shown and described with reference to exemplary embodiments thereof, those skilled in the art will perceive without departing from the scope of the invention encompassed by the appended claims. It will be understood that various changes in form and detail may be made therein.
In addition, this invention contains the following content as an aspect.
[Aspect 1]
A method of controlling a head mounted display (HMD) system including a microdisplay, a user input device, a first processor and two or more other peripheral circuits, comprising:
Activating only the first processor and user input device in a first state;
Detecting user input; and
Interpreting the user input as a verbal command or a gesture command using the processor;
A step of entering a second state, wherein a selected peripheral device of the two or more other peripheral devices is activated and not selected in response to the verbal command or gesture command Is not activated, the process and
Including methods.
[Aspect 2]
The method of aspect 1, wherein the two or more other peripheral devices include a second processor.
[Aspect 3]
The method of aspect 2, wherein the second processor consumes more power than the first processor.
[Aspect 4]
The method of aspect 1, wherein the two or more other peripheral devices are selected from the group consisting of a wide area network interface, a second processor, a display, a camera, and a head motion tracker.
[Aspect 5]
Aspect 2 wherein the user input is a verbal command to access a document, and in the second state, power is applied to the second processor and microdisplay and then the document is displayed. The method described in 1.
[Aspect 6]
The user input is a verbal command to access a web page, and the second state further enables a wireless interface to connect to a network to fetch the web page. 5. The method according to 5.
[Aspect 7]
The peripheral device includes a camera and a second processor activated in the first state, the user input is a hand gesture, and the second processor is prior to entering the second state. A method according to aspect 2, wherein the hand gesture is interpreted.
[Aspect 8]
The method of aspect 2, wherein the peripheral device includes a wireless interface and the second state transmits the user command to a host over the wireless interface.
[Aspect 9]
8. The method of aspect 7, wherein the user command is a request to manipulate an aspect of the display, and the second state changes information display on the display.
[Aspect 10]
A micro-display that displays visual information;
Two or more peripheral devices that provide one or more peripheral input signals indicative of sensor input or one or more peripheral input signals indicative of at least one of user movement and / or voice input; ,
A first processor that converts the peripheral input signal into one or more detection commands;
A processor controller that selectively activates the microdisplay and the one or more peripherals in response to the one or more detection commands;
Headset mobile device with
[Aspect 11]
The one or more peripheral devices are:
Comprising one or more microphones for receiving audio signals from the user;
The first processor further includes:
The apparatus of aspect 10, comprising a voice recognition device that processes a voice signal to detect a voice command.
[Aspect 12]
The aspect 10, wherein the one or more peripheral devices are motion detectors, and the motion detector provides two or more motion inputs that indicate a user's motion along two or more axes. Equipment.
[Aspect 13]
The apparatus according to aspect 12, wherein the motion detector is a camera that detects gesture motion of the user's hand and / or body.
[Aspect 14]
The apparatus according to aspect 12, wherein the motion detector is a head motion tracking device that detects head motion of the user.
[Aspect 15]
The apparatus of aspect 10, wherein the peripheral device further comprises a communication interface that provides one or more wireless links between the headset portable device and another data processing device.
[Aspect 16]
The apparatus of aspect 10, wherein the user command is processed by the first processor to control an aspect of visual information display on the microdisplay.
[Aspect 17]
The apparatus according to aspect 16, wherein the user command controls a field of view.
[Aspect 18]
The apparatus of aspect 17, wherein the user command controls a zoom, pan or scale factor.
[Aspect 19]
The apparatus according to aspect 16, wherein the user command selects a hyperlink item for web page display.

Claims (17)

A method of controlling a head mounted display (HMD) system including a microdisplay, a user input device, a first processor and two or more other peripheral circuits, comprising:
By operating only the front Symbol first processor and a user input device, a step of entering the first state,
Detecting user input; and
Interpreting the user input as a verbal or gesture command using the first processor;
A step of entering the second state, on the basis of the command or gesture commands verbally, the two or more other of said first processor a command for operating the selected peripheral device of the peripheral device by but issuing, and by issuing a command to not operate the peripheral device is not selected the first processor, look including the step of entering said second state,
The method wherein the two or more other peripheral devices include a second processor, the second processor consuming more power than the first processor . Wherein the two or more other peripherals, a wide area network interface, the second processor, the display, is selected from the group consisting of a camera and a head movement tracker The method of claim 1. The user's input is a command in the verbal commands of the verbal is for accessing documents, and in the second state, power is applied to said second processor and microdisplay, then the document is displayed, the method according to claim 1. Wherein the user input is a command in the verbal commands of the verbal is for accessing the web page, the second state is further connected to a wireless interface to a network, the web page The method of claim 3 , wherein the method can be fetched. The peripheral device comprises a camera, the second processor is further operated in the first state, the user input is a gesture of the hand, the second processor enters the second state It said interpreting gestures hands before a method according to claim 1. The method of claim 1 , wherein the peripheral device includes a wireless interface, and wherein the second state transmits the user input to a host over the wireless interface. 6. The method of claim 5 , wherein the user input is a request to manipulate an aspect of the display and the second state changes information display on the display. A micro-display that displays visual information;
A user input device that provides user input;
Providing one or more peripheral input signals indicative of sensor input or one or more peripheral input signals indicative of at least one of user movement from the user and / or voice input from the user ; Two or more peripheral devices that are disabled in the first state ; and
A first processor that converts the user input into one or more detection commands , wherein only the first processor and the user input device are activated in the first state; When,
Based on said one or more detection command, the micro-display and the one or more peripheral devices, selectively activated and whether to deactivated, or selectively to or selectively deactivated actuate and a processor controller for a,
The headset portable device , wherein the two or more peripheral devices include a second processor, wherein the second processor consumes more power than the first processor . The one or more peripheral devices are:
Comprising one or more microphones for receiving audio signals from the user;
The first processor further includes:
9. The apparatus of claim 8 , comprising a voice recognition device that processes a voice signal to detect a voice command. Said one or more peripheral devices is a motion detector, the motion detector, the motion of the user provides shown to two or more motion input, the motion of the user in more than one axis that in line, apparatus according to claim 8. The apparatus of claim 10 , wherein the motion detector is a camera that detects gesture motions of the user's hand and / or body. The apparatus according to claim 10 , wherein the motion detector is a head movement tracking device that detects a head movement of the user. The peripheral device is between the headset mobile device and another data processing device further comprises a communication interface that provides one or more wireless links, according to claim 8. Before SL first processor processes the detection command, that controls the visual information display aspect of on the microdisplay apparatus of claim 8. The apparatus of claim 14 , wherein the detection command controls a field of view. The apparatus of claim 15 , wherein the detection command controls a zoom, pan or scale factor. The apparatus of claim 14 , wherein the detection command selects a hyperlink item for web page display.

Images