ES2391112B2 - AUGMENTED REALITY SPACE PROJECTION SYSTEM FIXED TO THE HEAD - Google Patents

Abstract

The system object of the invention is an augmented reality projection device attached to the head, in such a way that the movements it performs are followed in solidarity. In addition to the fixing system, it consists of a video camera that records information from the environment, and a projector by means of which information generated by a computer is superimposed on it. Being in solidarity with the movement of the head, the relative position between the system and the user is known, allowing the projected elements to maintain spatial coherence with the environment from their point of view, in such a way that the projection can simulate 3D objects located in said environment, by varying the projection according to the movements registered; in this way, if the observer moves around the environment, the projection will vary correspondingly to simulate the presence of static objects.

Description

Head augmented reality spatial projection system

TECHNICAL SECTOR The invention is framed within augmented reality projection devices, combining characteristics of head-mounted devices and spatial projection devices.

BACKGROUND OF THE INVENTION Augmented reality is a discipline whose objective is the integration of information generated by a computer in the real environment that surrounds us, through the appropriate generation of graphics that are superimposed on said environment according to criteria set by each application. This integration can be carried out using different strategies, depending on whether they are aimed at an observer or the enrichment of an environment, and also depending on the technologies involved in their development. The factors common to most all integration solutions are: presence of one or more sensor devices, which capture information from the environment, a computer, which is responsible for generating the graphics that will later be superimposed on the environment, and a display device, which performs the final integration of the computer-generated elements with the environment. Proposals for different augmented reality element display devices have emerged from the different combinations of these factors. A possible classification of these is:

•

Indirect display, in which a display is used to visualize the result of the integration, such as a computer monitor or the screen of a handheld device (PDA, mobile phone, ...)

•

Direct visualization, in which some type of solution is used that performs the integration and generates two different views, one for each eye of the observer, so that a differentiated image is shown for

each of them with which a stereoscopic effect is achieved. This type of solution involves the use of a device attached to the head, so that the result of the integration is only visible to its user. Different solutions can be found within these devices:

o Head Mounted Displays (HMD's), for which the solution adopted consists of incorporating two small displays, mounted as glasses in glasses, in which the image that each eye should see is reproduced. These displays can be opaque (in which the image of the real elements and those generated by the computer are reproduced simultaneously), or transparent, in which only the elements generated by the computer are reproduced by allowing the passage of light from the environment.

o Virtual Retinal Displays (VRD's), for which the solution consists in projecting images generated by the computer directly onto the retina, so that they are superimposed directly on the observer's visual system.

o Head Mounted Projective Displays (HMPD's), in which a transparent screen is incorporated on which the elements generated by the computer are projected.

• Spatial projection, in which the elements generated by the computer are projected onto the environment itself (as a traditional projector would do on a screen), taking care that it is coherent with said environment but without this implying coherence with the point of view of any particular observer (the appearance of the projection does not change when the observer changes its position).

Therefore, it would be desirable to have a device capable of combining the advantages of each of the solutions presented, combining the fixed position at the head that allows simulating an integration of computer-generated 3D objects in the environment, with spatial projection, that allows the real enrichment of the environment through the direct integration on it of the elements generated by the computer. The result would be the possibility of having a device that would project computer-generated elements onto the environment, keeping spatial coherence with it in the eyes of the observer. To this end, the present invention combines fastening elements integral with the device at the observer's head, the presence of a camera mounted on said fastening element, and a pico-projector also mounted on it, so that all of them maintain their relative position and guarantee the possibility of fulfilling the objective of maintaining spatial coherence between the elements projected onto the environment and this in the eyes of the observer.

DESCRIPTION OF THE INVENTION The present invention consists of an augmented reality spatial projection device attached to the head and to the procedure for its correct operation as described in the claims. In particular, the invention has a camera configured to record information from the environment and coupled to a processing means that is configured to process the signal and extract the three-dimensional information from the environment. Furthermore, it comprises a projector configured to receive the three-dimensional information and generate a projection according to said three-dimensional information. It also has fixing means designed to fix the camera and the projector concerned to the head of a user so that they move jointly and severally without substantially varying the relative distance from the user's eyes.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 describes a possible embodiment of the system, in which a helmet (1) is used as a support element for the camera (2) and pico projector (3) and for the unit to be attached to the user's head. . Figure 2 shows how the configuration of the projection carried out by the pico projector can be modified when the system moves with respect to the environment: in order to make the projected element (5) appear immobile in the environment, a displacement of the system can cause the projection changes from a configuration like (4) to a configuration like (6), in such a way that the mentioned element (5) remains apparently immobile. Figures 3 and 4 show details of each projection configuration. Figure 6 shows in detail how to simulate the presence of a three-dimensional object (8) in the environment for the observer (7) it is necessary to make a projection adapted to the surface on which it is performed. In this case, the sensation of observing a three-dimensional object when it is projected on a flat and horizontal surface requires the picoprojector to generate an image like (9), which seen from the perspective of the observer (7) will give the sensation of contemplating the three-dimensional object (8) on said projection surface.

DETAILED DESCRIPTION OF THE INVENTION An element for fixing the assembly to the head (1); To this end, a rigidly attached rigid hat or cap can be used, or glasses fastened with temples or with an elastic band, in case the weight and volume of the other components allow it. Its fundamental task is not only to hold the assembly, but must guarantee that the relative positions between all the elements of the invention (camera-projector) and the eyes of the observer remain unchanged. This last characteristic is essential to allow the necessary calculations to be carried out for the correct operation of the system. If necessary, the measurement of these relative positions should be carried out through the appropriate system-wide calibration process.

• A video camera (2), fixed in solidarity to the fixing element described above. Its task is to capture images of the environment continuously (hence it is necessarily a video camera), images that will immediately be sent to a computer for processing. The result of this processing must be the position and three-dimensional orientation of the observer within the surrounding environment, or with respect to the most representative or necessary elements for the application considered in each case. In many applications, these representative elements are the so-called 'fiduciary marks' (5), whose shape and size are known, in such a way that through the perspective appreciated in the image the position and three-dimensional orientation of the observer, necessary to complete the application, can be easily removed. However, it is not necessary that the reference for the extraction of this information be these marks, and it may be any other object integrated into the environment.

• A pico projector (3), fixed in the same way on the fixing element described first in a solidary way, and guaranteeing a fixed relative position with respect to the camera, and both with respect to the user of the system (7). Its task is to project onto the environment the graphics generated by the computer (9) on the basis of the three-dimensional information extracted from the analysis of the images supplied by the camera. In this way, the graphics projected onto the environment maintain spatial coherence with it (8), giving the observer the feeling that the elements represented by said graphics are really 'there', perfectly integrated.

The system has a continuous operation, in such a way that the changes in position and / or orientation of the user with respect to the environment are immediately captured by the camera mounted on the system, analyzed by the computer and used to modify the projection.

The idea that the system is integrating into the environment a supposed statue, generated in this case by the computer, serves as an example; if the user of the system began to surround the position where said statue is being projected, as it would with a real one to appreciate it from all angles, the system would react by modifying the projection, in such a way that it would apparently remain static while the observer surrounds it, as it would happen with a real statue. Logically, said projection cannot be carried out in the air, so it must have a surface that allows this projection from all perspectives (cylindrical or rotating screen).

Spatial coherence is achieved by detecting and tracking known objects in the environment. Given the a priori dimensional knowledge of these objects, the image captured by the camera will present a certain deformation with respect to the known pattern, due to the perspective with which the scene is observed by the device; From this deformation, and using the appropriate mathematical tools (estimation of the inverse transformation of perspective from the image), it is possible to accurately determine the relative position between the camera and the reference object, placing both in space three-dimensional. With this information, the computer receives the information necessary to generate the graphics that will later be projected onto the environment in such a way that the user of the system perceives that the apparent position of the projected elements is consistent with the three-dimensional environment, so that the sensation is that they are perfectly integrated into it, in fixed spatial positions.

From the description of the components, it follows that the costs of the device object of the invention with respect to other devices described in the state of the art are clearly lower, fulfilling virtually the same functions, provided that a suitable surface is available to carry out the projection. Given the components described above and the way in which the role of each has been described, the logical operation of the system involves the following steps:

1. Capture of environment information: using the camera (2) incorporated in the system, images of the environment are collected continuously.

Since it is fixed to the user's head, the position relative to the user and to the projection system is known.

2.

Analysis of captured images: images captured by the camera are sent to a computer where they are analyzed using the appropriate algorithms for image processing and computer vision. As a result of this analysis, the position and 3D orientation of some or some key objects (5) that have been recognized in the images supplied by the camera are deduced.

3.

Generation of the graphics that will be projected onto the environment: a computer is also in charge of this task, for which it must take into account two factors:

•

Firstly, the positional information and three-dimensional orientation of the key objects obtained in the previous phase, since they will serve as the basis for achieving the necessary spatial coherence between the generated graphics and the position of the observer with respect to the environment; the effect that the projected elements are integrated 'spatially' in it must be achieved (8).

•

Secondly, the position and orientation of the area on which the projection must be carried out (5); given that the user may be moving or moving with respect to their environment, (4,6) it should be calculated where the projection should be made and with what relative size, making use of visual control algorithms applied to its position and scale, in order to compensate for any user movement in three-dimensional space. In this way, the generated graphics must dynamically modify their position and size within the area covered by the projector (i.e. the projected graphics do not always cover the entire projection area or remain static in it).

Four.

Finally, the graphics generated by the computer, according to the criteria previously exposed, are sent to the projector to be sent on the environment.

The system consists of the following fundamental parts: an element of head fixation, such as a helmet or glasses (1), which guarantee that the relative position between the different components of the system and the observer remains constant at all times; a camera (2), which collects the images of the environment to be analyzed; a picoprojector (3), which performs the projection of the images generated by the computer on the environment. The specific position in which these devices are fixed on the fastening element is not relevant, provided that the stationarity of their relative positions, as mentioned. Given the mobility of the user, the position of the area over which the project the computer generated image (5) into the covered area by the projector (4.6) may vary, so the position of the the image generated by the computer within it, to guarantee its correct overlay on target area (5). Since the system is designed to project the illusion on the observer (7) that there are three-dimensional objects (8) integrated into the environment, it must be make a projection according to the surface on which it is projected (9), so that in the eyes of the observer the apparent object actually has volume.

Example: A possible implementation of the system (although not limited to it) would consist of the following physical elements:

•

A helmet or goggles (1) as a support element and attachment to the head,

•

A pencil camera or sportcam (2) attached to the previous element, as an image capture device. This camera is connected to the computer through the appropriate interface elements (usually a USB connector for transmitting images to the computer) to supply the computer with the information necessary for it to perform the assigned calculations, that is, estimate the relative position of the computer. observer in the environment and with respect to the objects present in it, and the

subsequent generation of the graphics that will be projected, as

described in the following points.

• A computer for the analysis of the images supplied by the

camera, the result of which is the estimation of the three-dimensional position of the

5

user of the system within the environment, as well as its relative position

regarding the real integrated elements of said environment. East

computer, from this estimate of the three-dimensional position of the

observer, is also in charge of generating the graphics that

later they will be projected onto the environment, and they will simulate the

1 O

presence of static objects in it.

• A laser pico projector (3) fixed to the fixing and support elements,

that receives the images generated by the computer through the

suitable interface elements (usually a signal connector

video, such as a VGA, DVI or similar connector) and projects them

fifteen

about the environment.

INDUSTRIAL APPLICATION

The device described in the present invention can be used in many

of environments and applications given their mobility, versatility and low cost. Between

twenty

(although not limited to) them, are:

• Assistance to people with disabilities: suppose the case of people

with intellectual disabilities who want to give autonomy to

develop an independent life in your own home. The system

described can be used to provide instructions to the user

25

about housework, appliance management, schedule, etc., through

projection of the relevant instructions on any surface

close. In addition, by identifying household appliances

(washing machine, oven, dishwasher, ...) can be instructed in its handling

projecting on these devices the instructions (press a button

30

highlighted by projection, projection of arrows to indicate

movements or actions, etc.)

•

Enrichment of publications: through the provision of areas

marked in a publication, whether physical (book, magazine, children's story,

etc) or electronic (e-book reader, iPad, notebook, etc.), the system

can proceed to their identification and projection on these areas

5

of virtual objects 30, in such a way that move the media (paper or

apparatus), the projection is modified to give the feeling that said

object 30 remains static on the

support. The application could

serve so much

in the scope of entertainment how in publications

scientific, teaching material, etc.

10

• Applications home automation: the system can be used for perform the

control

of multiple appliances, to mode of I send to distance and

universal programmer,

through its adequate communication with the

central

computerized. Can to serve too how keyboard for the

phone,

of the alarm, adding to the characteristics described the

fifteen

detection of user movements (hands, fingers, etc.)

•

Entertainment - The system can be used to project games onto

those that can interact several people, when making this projection

about the environment (a board, characters from a game, etc). I also know

can use for visualization of future three-dimensional emissions

twenty

of TV.

Claims (1)

1.- Augmented reality spatial projection system comprising: - a camera (2) configured to capture images of the environment, said camera (2) coupled to processing means (10) configured to process the images and extract three-dimensional information , - a picoprojector (3) configured to receive the three-dimensional information and generate a projection according to said three-dimensional information, - fixing means (1) configured to fix the camera (2), the picoprojector ( 3) so that they move jointly in a joint manner without substantially varying the relative distance with the user's eyes, characterized in that the processing means (10) are also configured to estimate the user's three-dimensional position in the environment observed by means of the identification of fiduciary marks and known objects in the captured images. 2.- System according to claim 1, characterized in that the processing means (10) are further configured to estimate the three-dimensional position between the camera (2) and the pico projector (3) and to maintain in the 20 virtual images generated (7 ) by the pico projector the spatial coherence according to the point of view observed by the user. 3. System according to claim 2, where the virtual images generated (7) are objects whose dimensions are based on the information of the environment (8) captured by the camera (2) to simulate their integration with it. 4. System according to any one of claims 2 or 3, wherein the processing means (10) are configured to detect the user's movements by analyzing and comparing the images (8) captured by the camera at different times.