GB2495159A

GB2495159A - A head-mounted somatosensory control and display system based on a user's body action

Info

Publication number: GB2495159A
Application number: GB1200910.6A
Authority: GB
Inventors: Jung Ya Hsieh
Original assignee: GIXIA GROUP CO
Current assignee: GIXIA GROUP CO
Priority date: 2011-09-23
Filing date: 2012-01-19
Publication date: 2013-04-03
Also published as: GB201200910D0; CN103018905A

Abstract

A head-mounted somatosensory control and display system is suitable for use as a head-mounted movie theatre for use in an airplane cabin. It includes a base unit, an image pick-up device which may be a microcamera (32, figure 3), a display device which can be LCD display module (34, figure 3) and a processing device (36, figure 3). The display device may be mounted on the base unit at a position that is viewable by the user. The image pick-up device is mounted on the base unit at a position having a visual filed closely approximating to that of the user's eyes. The image pick-up device captures an image of the user's body action such as a specific hand gesture and the processing device analyzes the body action 503, 504, thereby overlaying the body action onto virtual images displayed on the display device and even allowing the user to immersively interact with virtual objects.

Description

HEAD-MOUNTED SOMATOSENSORY CONTROL AND DISPLAY

SYSTEM AND METHOD USING SAME

Field of the Invention

The invention relates to a head-mounted somatosensory control and display system, and more particularly to a head-mounted somatosensory control and display system with high operability, and a method using the same.

Description of Prior Art

A variety of head-mounted display devices have been available in the consumer market for years. They were previously expected to replace household display devices, such as TVs, taking advantage of their space effectiveness and capability of providing the visual pleasure as that which can be provided by a 60-or 70-inch display device. However, with the rapid innovation and development of projection televisions, it is now possible for consumers to accomplish the same visual effect simply by using a projector to project large-size images onto a wall or a projection screen, in contrast, head-mounted display devices have poor market share because they are too heavy to be worn on the head for a long time. Especially, the display devices, when being worn on the head, will block the sight of the wearers, making the wearers difficult to find out particular keys on controllers to tune the volume up or down or tune into a selected channel.

Considerable efforts were made by some manufacturers to deal with its market location by mounting an additional camera onto a head-mounted display device. As shown in FIG. 1, a conventional technique involves superimposing the real world images captured from the wearer's visua.l field by a head-mounted camera 1 2 over the virtual image data stored in an image storage unit 1 8 by virtue of a processor 1 6, in conjunction with monitoring of the wearer's horizontal position, vertical position, head position and body part position by a multi-functional sensor 10. The superimposed images are displayed on a head-mounted display device 14.

However, since the conventional head-mounted display device described above is of a non-transmissive type, the wearer can only observe the real-world a.ction indirectly by way of the head-mounted camera 12. Inherently, the shooting angle of the head-mounted camera would slightly deviate from the three-dimensional visual field of the wear's eyes. Unless motionless, the wearer would possibly misjudge the distance to a certain object and end up falling down due to such deviation in observation. Not to mention that the visual field from the camera is so restricted that it is much narrower than that from human eyes. In addition, the human eye can rapidly focus objects at different distances by using six muscles in the vicinity of the crystalline lens to move the eyeball and to quickly make the crystalline lens change shape. In contrast, an auto-focusing camera llas to take longer time to complete focusing operation as compared to human eyes, causing a delay in response time. A camera cannot replace the human eyes to observe the real world.

Another conventional design is disclosed in US 7,573,525. As illustrated in Fig. 2, a head-mounted camera assembly 2 is provided with a head-mounted display device 24 and an external controller 25 for setting a focal length of a camera 22 and displaying the image data captured by the camera 22 on the head-mounted display device 24. The external controller 25 is also adapted to move the image data displayed on the head-mounted display device 24 along the up, down, left and right directions and to magnify or minify the displayed images.

The head-mounted assembly disclosed in US 7,573,525 is provided with a transmissive-type display device, allowing the wearer to directly observe real-world objects through the display device while displaying the camera-captured images on the display device. However, the initiation of a displaying process, as well as the positions of the images thus displayed, have to be controlled by pressing some buttons on the external controller with the wearer's finger, causing inconvenience in operation. Especially, compared to the cameras now available in the market which are of high portability and are operable with a single hand, the conventional head-mounted assembly is quite cumbersome due to being equipped with a handheld controller and an entire set of head-mounted camera and display device.

Therefore, there exists a need for a more powerftil hea.d-mounted display device, which is ergonomically operable in response to user's body actions, thereby providing the user with enhanced operational convenience.

Summary of the Invention

An aspect of the invention is to provide a head-mounted somatosensory control and display system, which is capable of capturing an image of the user's body action in the real world and utilizing the body action as an operation instruction, thereby reducing the manufacture cost.

Another aspect of the invention is to provide a head-mounted soma.tosensory control and display system, which is capable of executing an j operation instruction upon identifying a specific body action, thereby achieving a simplified structure and a better portability.

A still another aspect of the invention is to provide a head-mounted soma.tosensory control and display system, which is capable of overlaying the user's specific body action onto the virtual objects displayed on the display device, thereby providing improved operability and ergonomic advances for user comfort and convenience.

A still another aspect of the invention is to provide a head-mounted somatosensory control and display method for automatically and rapidly capturing an image of the user's body action and analyzing and displaying the captured image.

A still another aspect of the invention is to provide a head-mounted somatosdnsory control and display method for executing an operation instruction upon identif,ing a corresponding body action, thereby achieving a better operability.

Therefore, the head-mounted sornatosensory control and display system disclosed herein is adapted for controlling image display based on a user's specific body action present within a predetermined image pick-up scope. The specific body action corresponds to a specific instruction. The system comprises a base unit mountable on the user's head; an image pick-up device mounted on the base unit and adapted for capturing an image of the user's body action within the predetermined image pick-up scope; a display device mounted on the base unit; and a processing device for receiving the captured image from the image pick-up device and assessing the presence or absence of the specific body action and, if identifying the presence of the specific body action, outputting the specific instruction corresponding to the specific body action to have an image displayed on the display device change to a different image.

The invention ftirther relates to a head-mounted somatosensory control and display method. The method is adapted for assessing whether a user performs a specific body action within a predetermined image pick-up scope. The specific body action is defined to serve as a specific operation instruction. The method comprises the steps of: a) placing a base unit on the user's head, wherein the base unit is provided with an image pick-up device and a display device at a position corresponding to the user's eyes; b) using the image pick-up device to capture an image within the predetermined image pick-up scope, and converting the captured image into electrical signals, and transmitting the electrical signals to a processing device; c) using the processing device to assess whether the user performs the specific body action; and d) executing the specific operation instruction corresponding to tile specific body action to have an image displayed on the display device change to a different image, if identifying the presence of the specific body action.

By using the image pick-up device to capturing images of the objects located within the predetermined image pick-up scope in front of the user, the processing device can subsequently analyze the captured images. If the processing device identifies that the user performs a predetermined body action, it would correlate the body action with a predetermined instruction and then execute the instruction. As a result, the invented system no longer requires a remote control, thereby ha.ving a simplified structure, a better portability and a reduced manufacture cost. Especially, the invented system and method enable an interaction of the captured images with a virtual object present in the displayed animation, such as a virtual swivel or driving lever. The invented system is more ergonomically friendly to a user who wants to adjust the displaying conditions of a film currently displayed on the display device. The invention achieves the objects described above accordingly.

Brief Description of the Drawings

Fig. 1 is a block diagram of a conventional head-mounted camera; Fig. 2 is a schematic diagram of a conventional head-mounted camera provided with a head-mounted display device; Fig. 3 is a schematic perspective view of a head-mounted somatosensory control and display system according to the first preferred embodiment of the invention; Fig. 4 is a block diagram of the head-mounted somatosensory control and display system shown in Fig. 3; Fig. 5 shows an operation chart of the head-mounted somatosensory control and display system shown in Fig. 3; Fig. 6 is a schematic diagram showing the overlay of a captured image of the wearer's hand onto an image currently displayed by the head-mounted somatosensory control and display system shown in Fig. 3, wherein the captured hand image corresponds to a "pause" instruction; Fig. 7 is another schematic diagram showing the overlay of a captured image of the wearer's hand onto an image currently displayed by the head-mounted somatosensory control and display system shown in Fig. 3, wherein the captured hand image corresponds to an instruction of "skip forward" to a given time point; Fig. 8 is another schematic diagram showing the overlay of a captured image of the wearer's hand onto an ima.ge currently displayed by the head-mounted somatosensory control and display system shown in Fig. 3, wherein the captured hand image corresponds to a volume adjustment instruction; Fig. 9 is a block diagram of a head-mounted somatosensory control and display system according to the second preferred embodiment of tile invention, showing that the video and audio data are stored in an external console and transmitted wirelessly by a. wireless Bluetooth module; Fig. 10 is a schematic diagram showing the overlay of a captured image of the wearer's hand onto an image currently displayed by the system shown in Fig. 9, wherein the wearer's finger is interacting with a virtual image; Fig. 11 is a schematic diagram showing that the image displayed by the system shown in Fig. 9 is changed as the electric gyro detects a rotation movement of the wearer's head; Fig. 12 is a schematic perspective view of a head-mounted somatosensory control and display system according to the third preferred embodiment of the invention; Fig. 13 is a schematic diagram of the wearer's both hands and a virtual keyboard during use of the head-mounted somatosensory control and display system shown in Fig. 12; Fig. 14 is a top view of the head-mounted somatosensory control and display system shown in Fig. 12, showing that the projected images are focused as virtual images at farther distances: and Fig. 15 is a schematic perspective view of a head-mounted somatosensory control and display system according to the fourth preferred embodiment of the invention.

Detailed Description of the Invention

Regarding previous description of the invention and other techniques, features and performance, it will be described more fully hereinafter with reference to the accompanying drawings of preferred embodiments for clear presentation.

Figs. 3 and 4 show a. head-mounted somatosensoiy control and display system according to the first preferred embodiment of the invention. For the purpose of illustration, the system according to this embodiment is described by way of example as a head-mounted movie theater for use in an airplane cabin, which comprises a base unit configured in the form of a frame 30, an image pick-up device in the form of a micro-camera 32, a liquid crystal display (LCD) module 34, a processing device 36, a storage device 38, a video and audio display device 37, and an audio player device in the form of a pair of earphones 33.

The LCD display module 34 is mounted on the frame 30 at a position that is viewable by the user's eyes. According to Step 501 shown in Fig. 5, when the user is going to use the head-mounted somatosensory control and display system disclosed herein, the user is required to wear the frame 30 on his/her head and place a compact disc 4 loaded with video/audio data into the video and audio display device 37. In Step 502, the video and audio display device 37

S

displays the video and audio data through tile LCD display module 34 and the earphones 33. The micro-camera 32 is mounted on the frame 30 at a position between tile user's eyes such that, in Step 503, the micro-camera 32 continuously captures images within a predetermined image pick-up scope.

The captured images are then converted into electrical signals, which are in turn transmitted to the processing device 36.

If the user wants to pause a film that is currently playing, the user could make a clenched-fist gesture 5 in front of his/her face, as shown in Fig. 6.

When the user's certain body action appears in the predetermined image pick-up scope of the micro-camera 32, the processing device 36 analyzes the actual images captured by the micro-camera 32 in Step 504, to assess whether the body action matches with a specific predetermined body action. If yes, a predetermined instruction corresponding to the predetermined built-in body action is retrieved from the storage device 38 in Step 505 for executing a prescribed operation. For example, the "clenched-fist gesture" described herein may correspond to a "pause" instruction for commending the video and audio display device 37 to temporarily stop displaying the film. On the contrary. the micro-camera 32 will keep capturing images and the film will be displayed continuously to its end without any interruption, if the "clenched-fist gesture" is never identified in Step 504.

As shown in Fig. 7, when the user, for example, stretches out the forefinger of his/her left hand, such a body action corresponds to an instruction of adjusting the frame rate according to this embodiment. Tn this case, the processing device 36 instructs the video and audio display device 37 to display a virtual frame-rate adjustment interface 382 on the LCD module 34 at a position corresponding to the location of the forefinget At the same time, the micro-camera 32 keeps capturing images of the forefinger, such that the processing device 36 can track the direction of the finger's movement and instruct the video and audio display device 37 to either skip backward or skip forward in accordance with the body action. In addition, the storage device 38 is stored with a variety of virtual control interface images. As shown in Fig. 8, when the user stretches out the forefinger of the right hand to present his/her wish to tune up or down the volume, the processing device 36, once recognizing the hand gesture, instructs the video and audio display device 37 to present a virtual volume adjustment interface 383 on the LCD module 34 a.t a position corresponding to the location of the forefinger, so that the user can a.djust the volume by moving the forefinger upward or downward.

It can be readily appreciated by those skilled in the art that the head-mounted soinatosensory control and display system disclosed herein is not limited to being useful in an airplane cabin but is adapted to substitute some of the in-home display devices, such as conventional somatosensory game consoles.

By virtue of connecting a camera to a television located in front of the user to capture images of the user's body actions, the user is allowed to interact with tile virtual game on the television screen by overlaying the user's images onto the displayed virtual pictures. According to the second preferred embodiment of the invention shown in Fig. 9, the video and audio data stored in an external console 5' are transnritted wirelessly by a wireless Bluetooth module 59' provided in the console 5' to a corresponding wireless Bluetooth module 39', from which the data are in turn transmitted to a portable processing device 36'.

The processing device 36' then instructs a display device to display a conesponding image shown in Fig. 10.

According to this embodiment, the display device comprises a transmissive-type display device 34', allowing the wearer to directly observe real-world objects through the display device while viewing the virtual images displayed on the screen. When a kid who wears the head-mounted somatosensory control and display system disclosed herein tries to touch a virtual dairy cattle shown on the screen with a single finger, the particular hand gesture is captured by a micro-camera 32' and converted into electrical signals, which are in turn transmitted to the processing device 36'. Once identifting the hand gesture, the processing device 36' renders the virtual dairy cattle to interact with the kid by instructing an audio player device to make a cow moo and then rendering the virtual dairy cattle to slowly move forward.

Once the virtual dairy cattle moves out from the screen as shown in Fig. 11, the kid might try to track the virtual dairy cattle by rotating his/her neck. In this case, an electric gyro 361' provided in the system is employed to assess the orientation of the kid's head, which is substantially equivalent to the direction that the transrnissive-type display device 34' and the micro-camera 32' face.

As a result, the images displayed on the transmissive-type display device 34' will move in accordance with the rotation movement of the wearer's head. For example, when the wearer who is viewing an image shown in Fig. 10 moves his/her head to the left, the electric gyro 361' detects a change in orientation and infons the processing device 36' of a new orientation that the wearer faces.

Such information is transmitted to the console 5' via the wireless Bluetooth module 39'. Upon receiving the information, the console 5' sends the corresponding image shown ill Fig. 11 back to the processing device 36', and the image is then displayed on the transrnissive-type display device 34'. In this embodiment, it is preferably that the castle and tree displayed on the transmissive-type display device 34' will also change their positions in accordance with the rotation movement of the wearer's head, so as to provide the wearer with a fully immersive simulation experience of virtual reality.

Alternatively, in the case where the wearer is playing a gun fighting game, a handgun gesture made by the wearer with his/her forefinger and thumb will be captured by the micro-camera 32', allowing the wearer to directly use the gesture as a virtual gun in the game. The head-mounted system disclosed herein has less peripherals and, thus, has better portability and operability.

While the displa.y device used in the embodiment above is of the transmissive-type, the display devices of other types may be used in other embodiments to keep the trend for portable electronics to be designed in a compact size. As shown in Figs. 12 and 14, the display device according to the third preferred embodiment of the invention comprises micro-projectors 43", micro-textured lenses 44" and an image pick-up device configured in the form of a pair of micro-cameras 42". Each of the micro-cameras 42" is mounted on the base unit 40" at a position right above one of the user's eyes, so as to have a solid visual angle closely approximating to that of the one eye.

The base unit 40" according to this embodiment is preferably configured in the form of conventional eye spectacles, which are normally worn in front of the user's eyes at an inappropriate distance for the user to perceive real images. As such, the display device is provided with two micro-projectors 43" at left and right temple arms, respectively. The micro-projectors 43' are connected to a portable wireless transceiver 46" via a transmission line 41". The wireless transceiver 46" is adapted to communicate with a host 45" in a wireless manner.

The micro-projectors 43' are adapted to project the image data provided by the host 45" to where the micro-textured lenses 440' reside, so that the user will view tile projected images as virtual images focused at farther distances due to deflection and reflection by the micro-textured lenses 440'. A three-dimensional virtual keyboard 50" is therefore presented as shown in Fig. 13 by virtue of difference in the images projected from the left and right micro-projectors 43".

When the micro-cameras 42" capture an image of the user's both hands 52", a corresponding virtual keyboard 50" reveals. The transmissivity of the lenses allows overlay of the actual hands 52" onto the virtual keyboard 50".

The cameras capture the motions of the user's hands, which are in turn transmitted to the processing device for analysis. Once being identified, the motions serve as instructions of typing characters on the virtual keyboard 50" and the input operation is completed upon recognizing the typing motions.

Moreover, according to the fourth preferred embodiment shown in Fig. I 5, the display device comprises a pair of lenses polarized orthogonally with each other. The optical fiber projectors mounted at two sides (not shown) are used to project images to the lenses, such that vertically polarized images are displayed on the lens at the left side and horizontally polarized images are displayed on the lens at the right side. The user who wears the device perceives three-di rnensional images accordingly.

By virtue of the structural arrangements and operation processes described above, the invented head-mounted somatosensory control and display system can capture a user's body action within the user's visual field and analyze the body action with a processing device, thereby overlaying the body action onto virtual images and even allowing the user to immersively interact with virtual objects. The processing device may further analyze the body action and, if identifying that it matches with a predetermined body action, generate an operation instruction corresponding thereto and even display corresponding image data, such as a virtual operation interface and the current status information, on the display device at a corresponding position. In contrast to the conventional techniques, the invented system can be easily manipulated through a user friendly interface in a. fascinating mannet The invention achieves the objects described above accordingly.

While the invention has been described with reference to the preferred embodiments above, it should be recognized that die preferred embodiments are given for the purpose of illustration only and are not intended to limit the scope of the present invention and that various modifications and changes, which will be apparent to those skilled in the relevant art, may be made without departing from the spirit and scope of the invention.

Claims

<claim-text>What is Claimed is: A head-mounted somatosensory control and display system, adapted for controlling image display based on a. user's specific body a.ction present within a predetennined image pick-up scope, wherein the specific body action corresponds to a specific instruction, the system comprising: a base unit mountable on the user's head; an image pick-up device mounted on the base unit and adapted for capturing an image of the user's body a.ction within the predetermined image pick-up scope: a display device mounted on the base unit; and a processing device for receiving the captured image from the image pick-up device and assessing the presence or absence of the specific body action and, if identifying the presence of the specific body action, outputting the specific instruction corresponding to the specific body action to have an image displayed on the display device change to a different image.</claim-text> <claim-text>2. The head-mounted somatosensory control and display system according to claim 1, wherein the specific body action comprises a hand gesture and the control and display system further comprises a storage device stored with an image data of a virtual operation interface corresponding to the hand gesture, wherein the image data of the virtual operation interface is retrievable by the processing device for being displayed on the display device.</claim-text> <claim-text>3. The head-mounted somatosensory control and display system according to claim 1, wherein the display device comprises a transrnissive-type display device.</claim-text> <claim-text>4. The head-mounted somatosensory control and display system according to claim 3. wherein the display device further comprises an audio player device.</claim-text> <claim-text>5. The head-mounted somatosensory control and display system according to anyone of claims 1-4, wherein the display device comprises a liquid crystal display module.</claim-text> <claim-text>6. The head-mounted somatosensory control and display system a.ccording to anyone of claims 1-4, wherein the display device comprises at least one micro-projector and at least one micro-textured lens to which the at least one micro-projector can project images.</claim-text> <claim-text>7. The head-mounted sornatosensory control and display system according to anyone of claims 1-4, wherein the display device comprises a pair of lenses polarized orthogonally with each other.</claim-text> <claim-text>8. The head-mounted somatosensory control and display system according to anyone of claims 1-4, wherein the image pick-up device comprises a micro-camera mounted in a manner corresponding to a position between the user's eyes.</claim-text> <claim-text>9. The head-mounted somatosensory control and display system according to anyone of claims 1-4, wherein the image pick-up device comprises two micro-cameras spaced apart from each other and mounted in a manner corresponding to the respective eyes of the user.</claim-text> <claim-text>10. A head-mounted somatosensory control and display method for assessing whether a user performs a specific body action within a predetermined image pick-up scope. wherein the specific body action is defined to serve as a specific operation instruction, the method comprising the steps of: a) placing a base unit on the user's head, wherein the base unit is provided with an image pick-up device and a display device at a position corresponding to the user's eyes; b) using the image pick-up device to capture an image within the predetermined image pick-up scope, and converting the captured image into electrical signals, and transmitting the electrical signals to a processing device; c) using the processing device to assess whether the user performs the specific body action; and d) executing the specific operation instruction corresponding to the specific body action to have an image displayed on the display device change to a different image, if identifying the presence of the specific body action.</claim-text>