FR3043295A1 - SPACE ENHANCED REALITY DEVICE FOR OFFICE ENVIRONMENT - Google Patents

Abstract

The present invention relates to an ergonomic spatial augmented reality embedded system for an office environment. The device includes a first camera (103), which coupled to a pico projector (101) forms a stereo system configured to capture the geometry of the environment in a first phase. In a second phase, a second camera (104), of wide angle type, provides the position of the face of the user. The camera systems and the picoprojecteur are attached to a support (105) consisting of portions and joints, the modeling of the mechanism allows to establish the calibration. Since the calibration is known, the pico projector can adjust the projection image according to the position of the user and the captured geometry, so as to enrich the user's environment and give him a hologram feel. The device according to the invention is particularly intended for the video game industry.

Description

The present invention relates to projector / camera systems that enrich the real environment of the user by directly projecting digital information through a video projector. This type of system makes it possible to increase the environment of the user without encumbering him with a device that he carries on him, as it could be by resorting to augmented reality systems that rely on glasses or tablets. .

It is known from the state of the art a device which has been the subject of a patent application WO 2011106201 A2. This device is a spatial augmented reality system comprising a plurality of cameras, depth cameras and video projectors. The system includes a method of calibrating this multitude of devices. The main disadvantage of this device is that it is suitable only for room-scale environments, and not ergonomic for office-scale environments. Indeed this device is not embedded and relies on a method of calibration of a multitude of devices.

It is known from the state of the art a device which has been the subject of a patent application WO 2012112401 A2. This device is a spatial augmented reality system comprising a depth camera, a video camera and a video projector. The system uses the depth camera to obtain a three-dimensional view of the environment and the video camera to obtain the colors. The video projector then allows the projection of the image adapted to a defined surface according to its geometry and its position in space. This device has the main disadvantage of not being ergonomic for office-scale environments. Indeed this device is not embedded and requires at least one depth camera.

It is known from the state of the art a device that was the subject of a publication at the conference SIGCHI conference on human factors in computing systems in May 2012, written by H. Benko, R. Jota, A. Wilson, and entitled: "MirageTable: freehand interaction on a projected augmented reality tabletop". This device is a space augmented reality system that projects digital information onto a curved table through a video projector. The user's face is tracked through a depth camera, the user's environment on the curved table is captured, and the projected image is adjusted in accordance with the user's position and environment capture. This device has the main disadvantage of not being an embedded system and require at least one depth camera.

It is known from the state of the art a device which has been the subject of a patent application US 20130033484 A1. This device refers to a system comprising a pico projector and a camera system configured to receive spatial information from a surface. The system is configured to adapt the image projected by the video projector to the spatial orientation of the surface so as to allow preservation of the proportions of the image relative to the surface. This device has the main disadvantage of not integrating the information on the position of the face of the user and therefore can not project virtual 3D content in the real environment to be adapted to the point of view of the user. This device is intended to enrich paper documents, and can not enrich virtual content 3d compared to the user's view so as to give him the sensation of observing a hologram.

The present invention aims to overcome these disadvantages, and proposes an ergonomic spatial augmented reality device for an office environment. The device includes a first camera system configured to capture the geometry of the environment; a second camera system configured to track the position of the user's face in the space; a pico projector configured to project an image into the user's environment; a support for the picoprojector and the camera systems, consisting of a set of portions and joints; and a computer unit configured to manage the entire system. The modeling of the mechanism of the different portions and joints of the support makes it possible to establish a calibration of the first camera system with the second camera system and the pico projector. This device is indeed ergonomic for an office environment to the extent that it meets the criteria of size and minimal space for this type of environment. The joints allow the device to bend and unfold. The device is moreover an embedded system with its own battery, and is connected by a wireless connection to a computer unit.

This device is ergonomic for an office environment in that it meets functional criteria for this type of environment. The second camera system consists of at least one wide-angle camera, allowing the user's face position to be tracked in space with a large field of view. It is closer to the projection area, thus allowing the user looking at the projection area to face the second camera system and to facilitate the user's image tracking methods. .

This device is ergonomic for an office environment as it relies on an optimal solution for rebuilding the environment. Indeed, the picojet is used both to capture the geometry of the user's environment, and both to add virtual content to the user's actual environment.

The accompanying drawings illustrate the invention;

Figure 1 shows the device of the invention.

Figure 2 illustrates a scenario of the invention in office environment.

With reference to these drawings, the description of the system is organized in three stages in the following paragraphs. In a first section, the general operating principle of the invention as well as its characteristics are stated. In a second section, a hardware solution of the invention is detailed. Finally in a third section is explained a software solution to obtain all the features of the invention.

Figure 1 illustrates an interactive virtual content projection system on a surface. The system scans the projection surface with a camera system in a first phase. The system can capture any surface, regardless of its complexity. In a second phase, a second camera system tracks the position of the user's face in the space. A pico projector then projects in parallel virtual content in accordance with the position of the user's face and the captured environment, so as to give it a hologram feel.

The system is placed on the user's work environment (201, 202), such as a desk or table. The system is fixed, and the capture of the environment must be reset each time the user moves it. The user does not wear any device on it and is completely free. He is also free of his movements within the range of the system, i.e. the field of vision of the second camera system.

The projection zone is fixed, only the content of the projection varies according to the position of the user so as to give him the sensation of observing a hologram. In order to better highlight projected virtual 3D objects, the regions of the image that do not correspond to these 3d objects can be projected in black. The black projection emits indeed little light, and is not very visible, which makes it possible to bring out the 3d objects of interest.

The system includes: a pico projector (101) configured to project an image into the user's environment; a first camera system (102) configured to capture the geometry of the environment; 104) configured to track the position of the user's face in the space; ~ a support (105) to the picoprojector and the camera systems, consisting of a set of portions and joints; sensors that perform angle measurements on the joints of the support.

The picoprojecteur is slightly inclined, to obtain a sufficiently large projection surface for an office environment. The inclination, however, remains low so as to avoid the blur caused by an incorrect focal length. The picoprojector's field of view should be wide enough to provide a sufficiently large projection surface for an office environment.

The pico projector is reused to capture the geometry of the environment: coupled with a first camera (103), it forms a stereo system configured to capture the geometry of the environment. The pico projector is thus used both to add virtual content to the user's actual environment, and both to capture the geometry of the user's environment. This hardware solution is optimal in this sense. It allows the proposed system to overcome the depth cameras used by the solutions of the state of the art such as Kinect, replacing them with a RGB camera.

The picoprojector projects in a first phase a structured light on the stage, for example a structured light binary or for example a light coded in gray level or in color. The camera coupled to this pico projector observes and identifies structured light on the stage. The mapping of the pixels of the structured image projected on the stage and the pixels of the image captured by the camera makes it possible to reconstruct the geometry of the environment and to obtain a mesh of the environment.

The second camera system consists of at least one wide-angle camera, allowing the user's face position to be tracked in space with a large field of view. Several wide-angle cameras can be used for a very wide field of view and thus a very wide follow-up of the user. This camera system is closer to the projection area, which allows the user who is looking at the projection area to face him, and to facilitate the user's face tracking methods by the image.

The pico projector, the first camera system and the second camera system are attached to a support consisting of portions and joints. It provides a link between the systems, and the modeling of the link makes it possible to establish a calibration between the systems. Once calibrated, the pico projector can adjust the projection image according to the position of the user and the captured geometry of the environment, so as to give the user a hologram feeling.

The joints further enable the device to bend and expand (106). It is compact in its folded form, and deploys ergonomically for an office-wide environment.

The system is embedded, with its own computing unit, its own autonomy thanks to a battery, and works in real time. It is also connected and communicates by wireless communication with a computer unit.

The software solution consists of: - a brick for calculating the position of the face by the image; - a 3D reconstruction brick by the stereo system described in the previous section; "A brick of projective texturing.

There are many techniques to calculate the position of a user's face in space, this paragraph presents an example of a possible technique. Advanced leaming machine methods allow to obtain the remarkable points positions of the user's face in the image. A method has recently emerged and has emerged as the dominant approach. It is based on a regression of the shape of the face with a cascade approach. The mapping of the remarkable 2D points detected on the image by these methods with the associated 3D points obtained from a generic model of the human face makes it possible to know the pose of the face.

There are many techniques to reconstruct a 3D environment, this paragraph presents an example of a possible technique. The pico projector and the camera can be calibrated beforehand (the intrinsic parameters are calculated) and form a stereo system. The first step is to determine the map of disparities. For this, structured lights are emitted by the picoprojecteur The camera then seeks disparities on each epipolar line. From the map of disparities, the method realizes a 3d reconstruction knowing the intrinsic parameters of the camera and the pico-projector. Finally a triangulation of Delaunay or Voronoï makes it possible to establish a mesh of the surface.

A projective texturing brick synthesizes the image to be projected by the picoprojector so as to give the user a hologram feel. The method requires three rendering passes. In a first pass, the virtual content is rendered from the position of the user's face. In a second pass, the synthesized image at the first pass is projected from the position of the user's face on the environment captured by the stereo system. In a third passage, a rendering is made from a virtual camera placed at the position of the real pico projector. The image of this rendering is sent to the pico projector to be projected. The pipeline presented highlights the importance of properly calibrating the device and knowing the poses of the pico-projector relative to the poses of the camera systems.

The device according to the invention is particularly intended for the video game industry.

Claims (8)

A system, comprising: a first camera system configured to capture the geometry of the environment; a second camera system configured to track the position of the user's face in the space; a pico projector configured to project an image into the user's environment, adjusting the image in accordance with the position of the user and the geometry of the captured environment; characterized in that said pico projector, said first camera system and said second camera system are attached to an ergonomic device for an office-scale environment, including a set of portions and joints including the mechanism modeling allows a calibration of said first camera system with said second camera system and said pico projector. 2. System according to claim 1 characterized in that said device is deployable through said joints, allowing the device to be compact in its folded form, and to deploy ergonomically for the environment at the scale of an office . 3. System according to claim 2 characterized in that said first camera system is a stereo system consisting of a camera coupled to said pico-projector, to capture the geometry of the environment by an optimal solution to the extent that said pico-projector is used both to capture the geometry of the user's environment, and both to add virtual content to the user's actual environment. 4. System according to claim 3 characterized in that said second camera system consists of at least one wide angle camera, for tracking the position of the user's face in space with a large field of view. 5. The system of claim 4 characterized in that said device places said second camera system close to the projection area, allowing the user looking at the projection area to face said second camera system and to facilitate the method. Face tracking of the user by the image. 6. System according to claim 5 characterized in that said device is an embedded system. 7. System according to claim 6 characterized in that said device is a connected object, which communicates by wireless communication with a computer unit. 8. Method characterized in that a computer unit is configured to manage the system according to claims 1 to 7.