CA2875261C

CA2875261C - Apparatus and method for a bioptic real time video system

Info

Publication number: CA2875261C
Application number: CA2875261A
Authority: CA
Inventors: Robert G. Hilkes
Original assignee: eSight Corp
Current assignee: eSight Corp
Priority date: 2012-06-01
Filing date: 2012-06-01
Publication date: 2019-05-21
Anticipated expiration: 2032-06-01
Also published as: CA3040218C; WO2013177654A1; EP2859399A1; EP2859399A4; CA2875261A1; CA3040218A1

Abstract

A method and apparatus of displaying an electronic video image using a head-worn near-to-eye display in a non-immersive fashion, such that the wearer can choose, through simple adjustments of their neck and eye angles, to either look at the displayed video image or their natural environment. The invention also relates to the incorporation of prescription lenses into the optical chain of the near-to-eye display. The invention also relates to the use of motion and position sensors incorporated into the head-worn device to enable automatic stabilization of the video image. The invention also relates to the use of motion and position sensors incorporated into the head-worn device to automatically adjust the vertical angle of either the camera or the electronic display or both, by sensing the vertical angular position of the user's head.

Description

Apparatus and Method for a Bioptic Real Time yideo Svstem Field of the Invention [0001] The invention relates generally to the field of wearable electronic displays and more specifically to the field of vision care.
Background of thenven 190021 There are numerous applications for lightweight head-worn near-to-eye displays.
These are commonly called Head Mounted Displays (1-1MD). HMDs display to the eye an electronically rendered image such that the wearer perceives that they are watching a sizeable electronic display at some distance in front of them.
The applications that use such 11MDs are numerous, including but not limited to virtual reality, electronic gaining, simulation environments such as for military simulations or flight simulators, medical applications such as for the enhancement of sight, and consumer applications such as the ability to view videos in a mobile setting.
100031 One of the fundamental challenges of HMDs is the tradeoff between the display's Field of View (FOV), being the size of the virtual display as perceived by the wearer, and pixel size. FOV is normally defined as the number of angular degrees subtended within the viewer's overall field of vision, horizontally, vertically, or on the diagonal. Horizontal FOV dimensions in the range of 20-degrees are typical, with larger dimensions being possible at significant expense.
Pixel size is similarly expressed as the number of angular arc minutes (1/60th of a degree) subtended by a single, typically square pixel element. As one might expect, to achieve a larger FOV with a given pixel resolution (number of pixels), results in a larger pixel size, and consequent loss of image detail.
100041 Peripheral vision is that portion of the human field of vision outside the center, say, 10-15 degrees FONT. Peripheral vision is extremely important in some 1-applications, especially those in which the wearer must maintain a connection with their natural environment to contextualize their situation, and enable way finding, orientation, and mobility. To provide significant peripheral vision via the electronic display requires an extremely large (and expensive) HMD.
Alternately HM:Ds which provide a significant natural peripheral vision external to the HMD housing, provide a very limited virtual electronic FOV, [00051 Many HMD applications can benefit from the incorporation of a live camera into the HMD, such that the wearer can not only view electronic data from a source, such as a video file, but also live video images of the world in front of them.
Image processing can be used to enhance the live camera image before it is presented to the eye, providing magnification, enhancement of brightness, or improved contrast for example.
(0006] In HMO systems that are to be used for multiple activities, different camera angles may be required for different tasks. For example, to observe an object a distance, the camera angle should be nearly horizontal relative to the horizon when the wearer's neck is straight and their gaze angled at the horizon. On the other hand, to view hand-held objects at a. close distance requires a camera that is angled downward, in order to avoid a highly exaggerated downward neck posture. In this manner, the angle of the camera mimics the angular movement of one's eyes in a non-HMD world.

2 100071 Finally, the angle of the display relative to the eyes is also dependent on the specific tasks of the wearer. In certain situations the wearer would like to look into the electronic display only temporarily, and by looking up at an angle higher than their habitual gaze. In other situations, the wearer would prefer a more immersive usage model, where the electronic display is directly in front of their normal line of gaze.
l0008) What is needed then is a general HMD device that is capable of providing significant unobstructed peripheral vision outside of the electronic display FOV, while simultaneously providing a high resolution video image. Further, the ability to adjust the angle of the display and the camera according to the wearer's specific activity would provide significant comfort and increased usability.
Summary of the Invention (00091 The ability to quickly alternate as required by the specific task, between viewing an image presented in the electronic display and viewing the world without the electronic display, enables many possible usage models for an Furthermore, in an MID with an integrated camera, the ability to adjust the vertical camera angle for different tasks, viewing objects distant and close for example, significantly increases the usability of such a device. Finally, an HMD
whereby the user is able to select the vertical position of the electronic display, in order to tradeoff a comfortable immersive video experience versus maintaining a broad natural field of view enables the HMO to be used in a variety of user applications and postures, 00101 The invention, in one aspect, relates to a method of orienting an electronic near-to-eye display such that the wearer views it slightly above their habitual line of sight for a given task. In this manner the wearer, through slight neck and eye

3

4 angle adjustments can, with minimal effort, alternate between the electronic display and their natural vision.
[0011] In one embodiment, the electronic display is mounted slightly above the wearer's habitual line of sight, so that by angling the neck slightly forward and directing their gaze slightly upward, they can look into the display. Alternately by angling the neck slightly back and directing their gaze slightly down, they can view below the electronic display using their natural vision, [0012] In another embodiment, the apparatus incorporates the wearer's prescription ophthalmic lenses, so that whether they gaze up into the electronic 1-IMD or down using their natural vision, they are able to do so through their prescription lenses.
This embodiment of the invention alleviates the need to switch between the HMD

device and the wearer's habitual spectacles.
1.00131 In another embodiment the apparatus incorporates a video camera, which provides the video information to the electronic display, the angle of the camera being adjustable as required by the specific task.
[0014] In any of the above embodiments the source of the video may come from a device other than the camera, in any standard electronic video format such as MPEG for example.
[0015] In another embodiment the apparatus may deploy motion sensing components in order to facilitate image stabilization for the electronic video image that is captured by the camera.
[00161 In another embodiment the motion sensing components could be used to determine the angular orientation of the apparatus, so that the vertical camera angle can be automatically adjusted based on head position.

Descrintion of the Drawings 100171 The invention is pointed out with particularity in the appended claims. The advantages of the invention described above, together with further advantages, may be better understood by referring to the following description taken in conjunction with the accompanying drawings. In the drawings, like reference characters generally refer to the same parts throughout the different views.
The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention.
100181 Figs. I through 4 are highly schematic diagrams of an embodiment of the system of the invention;
(0019] Fig. 5 is a more detailed view of an embodiment of the system for automatically adjusting the angle of the camera.
(0020] Fig. 6 is a more realistic rendering of a particular embodiment of the system of the invention.
100211 Figs. la through 7b depict three successive frames of simulated video, in order to show how motion vectors can be used for image stabilization.
100221 Fig. 8 shows an alternate method of altering the camera angle through defining a window on the image sensor rather than by physically altering the camera angle.
DescrIption of thek_referred Embodiment [00231 In brief overview and referring to Fig. 1, the system in one embodiment includes prescription lenses 6 mounted to an eyeglasses frame 1. The Head Mounted Display portion of the system comprises a housing 2, which can move relative to the eyeglasses frame 1, about a pivot point 3. The HMI) housing 2 incorporates HMD optics 5, a camera 9, and HMD electronics 8 (collectively, the "HMD").

[00241 In the orientation depicted in Fig 1., the wearer's heacUneck posture is angled slightly back 4, and he is viewing the world 7 through the prescription lens 6, without the use of the HMD optics 5 or camera 9.
[0025] In Fig. 2, the head/neck angle 4 is slightly forward, allowing the user to view the IIMD optics 5 through the same prescription lens 6 by directing their gaze at a slight upward angle 7. In this mode, video information viewed through the HMD
optics 5 can be provided from the video camera 9, oriented at an outward angle such that objects at a distance can be viewed in the video image. As discussed, the video could also come from other sources.
[00261 In Fig. 3 the head/ncck angle 4 is unchanged from Fig. 2, but the camera has been angled downward on a pivot point 11 so that the camera angle 10 is now aimed at a nearby object close at hand, perhaps in the wearer's hands.
100271 In Fig. 4 the slightly forward head/neck angle 4 remains unchanged, but the HMD angle has been significantly lowered by pivoting the HMD housing 2 relative to the eyeglasses frame I, around a pivot point 3. In this orientation the wearer is able to adopt a more comfortable viewing angle 7 for near-in tasks.
Furthermore, the camera angle 10 can be directed further downward because the camera pivot point 11 moves with the HMO housing 2.
[00281 In Fig_ 5, a method is shown whereby a linear motor 12 can be used to adjust the vertical angle 10 of the camera 9. The camera rotated around a pivot point 11 that is affixed to the HMD housing 2. With the adjustment of the camera angle automated thus, it is possible to use motion and position sensors embodied in the 1-1MD electronics 8, to provide positional data that can be used to control the linear motor 12.
[0029] Alternately, the angle 10 of the camera 9 can be adjusted manually by the user-[0030] Fig. 7 shows how the same motion and position sensors embodied in the HMD
electronics 8 can be used to enable an image stabilization function. Figs. 7a, 7b, and 7e show three consecutive frames of video as captured by the camera 9.
Because of the normal movement of the wearer's head, successive frames of video have a translational (up/down, left/right) and rotational (angular) position relative to the previous frame. The frame depicted in Fig. 7b for example has shifted to the right/down and rotated slightly counter-clockwise relative to the previous frame 7a. The translational vector Ai and rotational angle Oi can be determined from the motion and position sensors embodied in the HMD
electronics 8. By applying the opposite of the translational vector and the reverse rotational angle to the pixels in the video frame 7b, the differences between the two frames 7a and 7b can be removed. Figure 7c carries the example further, showing a frame of video that is now shifted left/down and clockwise relative to the previous frame 7b. A new vector 6,2 and rotational angle 02 can be applied, and so forth so that over time, small frame-to-frame variations caused by the wearer's head movement are removed from the displayed video. To distinguish between minor head movements, for which image stabilization should be applied, and gross head movements, which indicate the wearer's scene of interest has changed, requires that the image stabilization function only be applied to small translational and rotational vectors.
[0031] In 8a and 8b, the angle of the camera 9 is adjusted not by physically rotating the camera as previously discussed. Rather, an area of pixels, or a window 13, 14 can be defined on the image sensor so that the wearer perceives that the camera angle is physically altered. This technique requires a camera system wherein the usable image sensor area is larger than the video that is to be displayed to the wearer. Motion and position sensors embodied in the 1-1MD electronics 8 can be used to determine the wearer's head/neck angle 4 and, based on this information, define a capture window 13, 14 that gives the wearer the perception of a high or low camera angle 10.
[032] While the present invention has been described in terms of certain exemplary preferred embodiments, it will be readily understood and appreciated by one of ordinary skill in the art that it is not so limited, and that many additions, deletions and modifications to the preferred embodiments may be made within the scope of the invention as hereinafter claimed. Accordingly, the scope of the invention is limited only by the scope of the appended claims.
[033] What is claimed is:

Claims

Claims:

1. A head-worn device comprising:
a frame;
a binocular electronic near-to-eye display assembly pivotally attached to the frame;
a camera with vertical angle adjustment forming part of the electronic near-to-eye display;
and a pair of prescription lenses mounted within the frame;
whereby a wearer views their environment through their prescription lenses with their head in a first position such that the electronic near-to-eye display is pivoted upwards; views a video image presented on the electronic near-to-eye display through the same prescription lenses with their head in a second position such that the electronic near-to-eye display is pivoted down in front of their prescription lenses; and views the video image generated from the camera and presented on the electronic near-to-eye display through the same prescription lenses with their head in a third position, wherein the third position is such that the electronic near-to-eye display is pivoted further down in front of their prescription lenses than when their head was in the second position; and the vertical angle of the camera changes between a first angle and a second angle when the wearer's head is in the second and third positions respectively.

2. The head-worn device according to claim 1, wherein the vertical angle of the camera adjusts according to whether a user's head is in the first position or the second position.

3. The head-worn device according to claim 1, further comprising:
an orientation sensor; and a motion sensor, wherein the video image from at least one of the camera and display electronics is stabilized using motion information provided by the orientation and motion sensors.

4. The head-worn device according to claim 3, wherein the vertical angle of the camera is automatically adjusted for a given task using information provided by the orientation and motion sensors.

5. The head-worn device according to claim 3, wherein a perceived vertical angle of the camera is determined not by adjusting a physical angle of the camera, but by defining an area of pixels or window, on a camera sensor.

6. The head-worn device according to claim 5, wherein the vertical position of an area of pixels or window on the video camera is automatically adjusted for a given task using information from the orientation and motion sensors.

7. The head-worn device according to claim 1, wherein the vertical angular position of the electronic near-to-eye display can be adjusted based on a user's specific task and neck posture.

8. An apparatus for a head mounted electronic near-to-eye display module for use with prescription glasses having a pair of lenses, the electronic display comprising:
a camera adjustably attached to the display module;
a display electronics circuit in communication with the camera;
a head mounted display optics having a first surface adjacent the display electronics circuit and a second surface adjacent one of the pair of lenses of the prescription glasses; and a means for pivotally attaching the display module to the prescription glasses, whereby a wearer views their environment through their prescription lenses with their head in a first position such that the electronic near-to-eye display is pivoted upwards; views a video image presented on the electronic near-to-eye display through the same prescription lenses with their head in a second position such that the electronic near-to-eye display is pivoted down in front of their prescription lenses; and views the video image generated from the camera and presented on the electronic near-to-eye display through the same prescription lenses with their head in a third position, wherein the third position is such that the electronic near-to-eye display is pivoted further down in front of their prescription lenses than when their head was in the second position; and a vertical angle of the camera changes between a first angle and a second angle when the wearer's head is in the second and third positions respectively.

9. The apparatus for a head mounted electronic near-to-eye display module according to claim 8, wherein the vertical angle of the camera adjusts according to whether a user's head is in the first position or the second position.

10. The apparatus for a head mounted electronic near-to-eye display module according to claim 8, further comprising;
an orientation sensor; and a motion sensor, wherein the video image from at least one of the camera and display electronics is stabilized using motion information provided by the orientation and motion sensors.

11. The apparatus for a head mounted electronic near-to-eye display module according to claim 8, wherein the apparatus for the head mounted electronic near-to-eye display automatically pivots upwards and downwards based upon a movement of the wearer's head.

12. The head-worn device according to claim 1, wherein the electronic near-to-eye display assembly pivotally attached to the frame automatically pivots downwards based upon the movement of the wearer's head from the second position to the third position and upwards based upon a movement of the wearer's head from the third position to the second position.

13. The apparatus for a head mounted electronic near-to-eye display module according to claim 8, wherein the apparatus for the head mounted electronic near-to-eye display is pivotally attached to a frame and automatically pivots upwards and downwards based upon a movement of the wearer's head.

14. The apparatus for a head mounted electronic near-to-eye display module according to claim 8, wherein the display electronics circuit are mounted above the head mounted display optics; and the head mounted display optics further comprises a surface, the surface for reflecting an image generated by the display electronic circuit coupled to an entrance surface to an exit surface.

15. The head-worn device according to claim 1, wherein the binocular electronic near-to-eye display assembly pivotally attached to the frame automatically pivots upwards and downwards based upon a movement of the wearer's head.

16. A head-worn device comprising:
a frame;
a binocular electronic near-to-eye display assembly pivotally attached to the frame; and a camera with vertical angle adjustment forming part of the electronic near-to-eye display;
whereby a wearer views their environment directly with their head in a first position such that the electronic near-to-eye display is pivoted upwards; views a video image presented on the electronic near-to-eye display with their head in a second position such that the electronic near-to-eye display is pivoted down in front of their eyes; and views the video image generated from the camera and presented on the electronic near-to-eye display with their head in a third position, wherein the third position is such that the electronic near-to-eye display is pivoted further down in front of them than when their head was in the second position.

17. The head-worn device according to claim 16, wherein the vertical angle of the camera changes between a first angle and a second angle when the wearer's head is in the second and third positions respectively.

18. The head-worn device according to claim 16, wherein the electronic near-to-eye display automatically pivots into a different position based upon a user's head being in the first, second, and third positions.

19. The head-worn device according to claim 16, wherein the electronic near-to-eye display is manually pivoted into the different positions when a user's head is in the first, second, and third positions; and the vertical angle of the camera automatically changes between a first angle and a second angle when the wearer's head is in the second and third positions respectively.

20. The head-worn device according to claim 16, wherein the electronic near-to-eye display is manually pivoted into the different positions when a user's head is in the first, second, and third positions; and the vertical angle of the camera is manually adjustable between a first angle and a second angle when the wearer's head is in the second and third positions respectively.

21, The head-worn device according to claim 16, wherein the vertical angle of the camera adjusts according to whether a user's head is in the first position or the second position.

22. The head-worn device according to claim 16, further comprising:
an orientation sensor; and a motion sensor, wherein the video image from at least one of the camera and display electronics is stabilized using motion information provided by the orientation and motion sensors.

23. The head-worn device according to claim 22, wherein the vertical angle of the camera is automatically adjusted for a given task using information provided by the orientation and motion sensors.

24. The head-worn device according to claim 22, wherein a perceived vertical angle of the camera is determined not by adjusting a physical angle of the camera but by defining an area of pixels of a camera sensor presented to a user upon the electronic near-to-eye display.

25. The head-worn device according to claim 22, wherein the electronic near-to-eye display displays either an image acquired by the camera or an image coupled to the electronic near-to-eye display from another image source.

26. A head mounted electronic near-to-eye display module, the electronic near-to-eye display module comprising:
a camera adjustably attached to the display module;
a display electronics circuit in communication with the camera;

a head mounted display optics having a first surface adjacent the display electronics circuit and a second surface disposed towards a user's head; and a means for pivotally attaching the electronic near-to-eye display module to a frame worn by the user, whereby a wearer views their environment directly with their head in a first position such that the electronic near-to-eye display module is pivoted upwards; views a video image presented on the electronic near-to-eye display module with their head in a second position such that the electronic near-to-eye display module is pivoted down in front of their eyes; and views the video image generated from the camera and presented on the electronic near-to-eye display module with their head in a third position, wherein the third position is such that the electronic near-to-eye display module is pivoted further down in front of them than when their head was in the second position.

27. The head mounted electronic near-to-eye display module according to claim 26, wherein a vertical angle of the camera changes between a first angle and a second angle when the wearer's head is in the second and third positions respectively.

28. The head mounted electronic near-to-eye display module according to claim 26, wherein the electronic near-to-eye display automatically pivots into the different positions based upon the user's head being in the first, second, and third positions.

29. The head mounted electronic near-to-eye display module according to claim 26, wherein the electronic near-to-eye display is manually pivoted into a different position when the user's head is in the first, second, and third positions; and a vertical angle of the camera automatically changes between a first angle and a second angle when the wearer's head is in the second and third positions respectively.

30. The head mounted electronic near-to-eye display module according to claim 26, wherein the electronic near-to-eye display is manually pivoted into a different position when the user's head is in the first, second, and third positions; and a vertical angle of the camera is manually adjustable between a first angle and a second angle when the wearer's head is in the second and third positions respectively.

31. The head mounted electronic near-to-eye display module according to claim 26, wherein a vertical angle of the camera adjusts according to whether the user's head is in the first position or the second position.

32. The head mounted electronic near-to-eye display module according to claim 26, further comprising;
an orientation sensor; and a motion sensor, wherein the video image from at least one of the camera and display electronics is stabilized using motion information provided by the orientation and motion sensors.

33. The head mounted electronic near-to-eye display module according to claim 32, wherein a vertical angle of the camera is automatically adjusted for a given task using information provided by the orientation and motion sensors.

34. The head mounted electronic near-to-eye display module according to claim 32, wherein a perceived vertical angle of the camera is determined not by adjusting a physical angle of the camera but by defining an area of pixels of a camera sensor presented to the user upon the electronic near-to-eye display.

35. The head mounted electronic near-to-eye display module according to claim 32, wherein the electronic near-to-eye display displays either an image acquired by the camera or an image coupled to the electronic near-to-eye display from another image source.