EP2119226A2

EP2119226A2 - Device and method for watching real-time augmented reality

Info

Publication number: EP2119226A2
Application number: EP08761752A
Authority: EP
Inventors: Valentin Lefevre; Nicolas Livet; Alan Savary
Original assignee: Total Immersion
Current assignee: Total Immersion
Priority date: 2007-01-12
Filing date: 2008-01-10
Publication date: 2009-11-18
Also published as: FR2911463A1; WO2008099092A3; WO2008099092A2; FR2911463B1; US20100045700A1; WO2008099092A8

Abstract

The invention relates to a real-time augmented-reality watching device (300), which comprises an image sensor (335) such as a PTZ camera, a visualisation system (320) and a control interface (310). In this device, the camera is controlled by a control interface operated by the user. Once the user has received (610) orientation information on the desired line of sight from the control interface, the orientation information on the line of sight is transmitted to the camera (615), the camera being powered and capable of movement. Then the camera transmits the orientation of its line of sight (620). The camera transmits a video stream in parallel. The position at which must be inserted data, such as the representation of a virtual three-dimensional object, is determined from the data resulting from the calibration and the orientation of the line of sight of the camera received from the latter (640). The data is further inserted in real time into the video stream that is transmitted to the visualisation system (645). A calibration of the camera is preferably carried out during the device set-up (600) in order to improve the device performance.

Description

Real-time augmented reality observation device and method for implementing such a device

The present invention relates to observation devices such as telescopes and more particularly augmented reality observation devices for adding real-time virtual objects to the observed image and methods for implementing such devices.

A large number of high-traffic sites such as tourist sites are equipped with telescopes or binoculars that allow their users to observe a panorama with a magnification factor, variable or not, giving the opportunity to better appreciate the view.

A telescope refers to a device located on many tourist sites offering an interesting point of view to observe. The principle of operation of the telescope is as follows: the user puts a coin in the device and then has a predetermined time to observe the offered point of view. The observation is similar to what would be done with the use of a pair of binoculars. The main function is therefore to provide a "magnification" of the panorama observed via a dedicated optical system. The latter offers a restricted field of view around the line of sight. The limitation of the field of view of the optical system is compensated by the two degrees of freedom offered by the apparatus by rotation of the line of sight in the horizontal plane, that is to say around the main axis of the device, and by rotation of the line of sight in the vertical plane, that is to say around an axis perpendicular to the main axis of the device. Thanks to these movements, the user is able to scan a large part of the observable panorama and thus observe in detail, using magnification, the areas that interest him. Figure 1 illustrates an example of such a telescope. However, the user of the system has no other information than that naturally present in the image. However, for the development of a site, it is often important to provide additional information. These For example, information may be of a cultural or technical nature but may also be of the advertising or economic type to indicate, for example, the presence of restaurants or hotels.

Techniques called "Augmented Reality" are adapted to the addition of such information to enrich the image observed. They make it possible to make appear in the same image, the real view and the added information. This information can then take the form of symbols such as arrows, logos, special signage, text but also three-dimensional virtual objects, animated or not. It is thus possible to make emerge, for example, an old building today disappeared above the ruin which the user would have had to be satisfied if it had only a conventional telescope.

There are augmented reality vision devices. For example, YDreams offers a solution called Virtual Sightseeing (Virtual Sightseeing is a registered trademark). This device has a degree of freedom. A disadvantage of this device is the quality of service drift over time. Indeed, the spatial and temporal synchronization depends not only on the accuracy of the encoders used to determine the motions but also on the accuracy and stability of the mechanical links used. In this respect, it should be noted that an error of one degree in the horizontal plane results in an error of more than four meters on the X coordinate of an object located two-fifty meters from the observation point. Any mechanical connection of this nature consisting of a displacement of heavy parts, as precise as it, evolves over time. This affects the accuracy of the synchronization between the virtual and the real which is degraded over time. Another device proposed by Trivisio Prototyping has, like conventional telescopes, two degrees of freedom. However, this device has the same drawback of quality of service drift over time. The invention solves at least one of the problems discussed above. The invention thus relates to a method for a real-time augmented reality observation device comprising an image sensor, a display system and a control interface, this method being characterized in that it comprises the following steps receiving a request comprising axis of view guidance information transmitted by said control interface;

transmitting said line of sight orientation information to said image sensor, said image sensor being mobile and motorized;

receiving from said image sensor the orientation of its line of sight; receiving at least one image originating from said image sensor;

determining in said received image the position at which at least one data item must be inserted, according to the orientation of the axis of view of said image sensor; and,

inserting in real time into said received image said at least one datum at said determined position.

The method according to the invention makes it possible to distinguish the control commands of the user from the commands of the image sensor and thus to improve the accuracy and reliability of the observation device, with a slight drift over time. The device is thus more robust in time and in the face of external aggressions. The camera brings the actual shooting of an area of the panorama to which a computer adds information elements precisely superimposed on the different elements of the image. This method makes it possible to improve the quality of the spatial and temporal synchronization between the real and the virtual which only then depends on the parameters inherent to the image sensor. The method according to the invention also offers possibilities of correction of disturbances. In addition, the method according to the invention allows great freedom as to the form that can take the control interface.

The method according to the invention advantageously comprises a calibration phase of said image sensor to take into account any imperfections thereof.

According to a particular embodiment, said calibration step comprises the calibration of at least one of the parameters included in the set of parameters including the radial distortion correction of said image sensor, the roll correction of said image sensor, the heading and pitch corrections of the line of sight of said image sensor and the offset between the image sensor; optical center and the center of rotation of said image sensor. Still according to a particular embodiment, said image sensor comprises a zoom function and the calibration of said at least one parameter is performed for a plurality of zoom factors.

The calibration of the image sensor and in particular the image sensor calibration for several zoom factors make it possible to consider the camera's defects as extrinsic parameters of the camera and thus optimize the calculations required on the images from this image sensor.

Still according to a particular embodiment, the method further comprises a co-location step of said image sensor and the scene observed by said image sensor to determine the laying of said at least one datum to be inserted into said image received in said observed scene.

Said at least one piece of data to be inserted in said received image is advantageously a representation of a virtual three-dimensional model animated or not.

According to a particular embodiment, said orientation of the line of sight is defined according to two degrees of freedom and said image sensor comprises a zoom function to enable the user to observe the point he wishes to see, with the desired magnification. The invention also relates to a computer program comprising instructions adapted to the implementation of each of the steps of the method described above.

The invention also relates to a means for storing information, removable or not, partially or completely readable by a computer or a microprocessor comprising code instructions of a computer program for the execution of each of the steps of the previously described method. The invention also relates to an augmented reality observation device comprising means of connection to an image sensor, a display system and a control interface, this device being characterized in that it comprises the following means means for receiving guidance line orientation information transmitted by said control interface;

means for controlling the orientation of the axis of view of said image sensor according to said orientation information received, said image sensor being mobile and motorized; means for receiving the orientation of the axis of view of said image sensor;

means for receiving at least one image from said image sensor;

means for determining in said received image the position at which at least one datum is to be inserted according to the orientation of the axis of view of said image sensor; and

means for inserting in real time into said received image said at least one datum at said determined position.

The device according to the invention makes it possible to separate the control commands of the user from the commands of the image sensor and thus to improve the accuracy and reliability of the observation device, with a small drift over time. The device is thus more robust in time and in the face of external aggressions. The camera brings the actual shooting of an area of the panorama to which a computer adds information elements precisely superimposed on the different elements of the image. The device makes it possible to improve the quality of the spatial and temporal synchronization between the real and the virtual which only then depends on the parameters inherent to the image sensor. The device according to the invention also offers possibilities of correction of disturbances. In addition, the device according to the invention allows great freedom as to the shape that can take the control interface.

The device according to the invention advantageously comprises means for transmitting said received image comprising said at least one given to allow the user to view the augmented images from said image sensor on a suitable device.

According to a particular embodiment, at least one of said image sensor and storage means is remote from said observation device. This embodiment allows the user of the observation device to benefit from a point of view that he can not reach and to protect the image sensor and / or the storage means against external aggressions such as vandalism.

Other advantages, aims and features of the present invention will emerge from the detailed description which follows, given by way of non-limiting example, with reference to the accompanying drawings in which:

- Figure 1 schematically shows a conventional telescope as used in tourist places;

FIG. 2 diagrammatically illustrates the observation device according to the invention;

FIG. 3, comprising FIGS. 3a, 3b and 3c, shows an example of a part of the observation device according to the invention comprising a sphere housing a motorized camera on which are mounted a control interface and a display system. ; FIG. 4 illustrates the observation device presented in FIG.

3, as a whole;

FIG. 5 illustrates an example of an apparatus that can be used to control the movements of the camera and to insert the virtual objects in the images coming from the camera; FIG. 6 schematically represents certain operating steps of the observation device according to the invention; and,

FIG. 7 shows another example of an observation device according to the invention in which the motorized camera is deported.

According to the invention, the control of the image capture device is indirect, that is to say that the movement of the camera is not physically or physically coupled to the movement made by the user.

The aiming control interface is separated from the positioning interface of the shooting module such as a camera. The coupling between the two interfaces is realized in a software way which also offers possibilities of correction of the disturbances. Thus, the movements of the user are reflected on the shooting module via the software interface. Since the virtual scene is co-located with the camera and not the hardware physically manipulated by the user, any loss of precision on the motion sensors of the control interface has no impact on the quality of the integration of the devices. virtual elements in the real image. The camera module, motorized, is preferably capable of producing lateral movements, also called cap or pan, and elevation movements, also called pitching or tilting. This type of shooting module can be introduced under the name PT (Pan, Tilt) camera. Advantageously, the shooting module also makes it possible to zoom, this type of shooting module can be called PTZ camera (Pan, Tilt, Zoom). The device according to the invention comprises a video acquisition system producing a video stream composed of images corresponding to the actual view observed, an image processing system for increasing the video stream by virtual elements inlaid in real time according to the direction of the line of sight of the device and a system for viewing the increased video stream. The line of sight is determined in real time by motion sensors. The accuracy of the position of the line of sight of the camera is essential to the proper functioning of the set, in order to insert the virtual objects in the appropriate locations, that is to say to spatially synchronize the real and virtual environments . The separation of the command to define the position of the line of sight as desired by the user, that is to say the user interface, means to achieve this position, that is to say say the internal mechanism of control of the movements of the camera, implies the setting up of a command interface allowing not only to define but also to transmit permanently the requested position of axis of sight. The use of a motorized camera makes it possible to move its axis of sight according to the request. This interface is preferably performed by the computer used to insert the virtual objects in the images from the camera. This computer receives the orientation request for the axis of view of the command interface, transmits the orders to the motorized camera to modify the line of sight and receives, from the camera, the exact position of the axis of view. referred. This is called indirect steering of the line of sight. The orientation information of the line of sight of the camera is received from it in the form of, for example, flow. Such a stream may transmit, for example, orientation and zoom data per second. When the position of the line of sight is known, the video stream is increased and restored.

The use of the position of the line of sight provided by the motorized camera and not the control interface makes it possible to improve the quality of the spatial and temporal synchronization between the real and the virtual, which depends only on the parameters inherent to the camera.

One of the objectives of indirect control is to limit the propagation of errors within the system. Thus, the durability of the synchronization between the virtual and the real over time relies entirely on the quality and reliability of the data received from the motorized camera. Even if the precision of the command interface deteriorates over time, the impact will be limited: the line of sight may not correspond exactly to the request sent by the interface but the real and the virtual will remain perfectly synchronized.

In addition, the separation between the command interface and the rest of the system allows great freedom as to the form that this interface can take. It may be for example a keyboard, a joystick type joystick or a dedicated pointing system such as a mechanical system with motion capture. Such a pointing system can mechanically couple the control interface and the display system. Figure 2 schematically illustrates the device 200 of the invention.

As illustrated, the device 200 comprises a control interface 205 which transmits a position request of the line of sight to the computer 210. The computer 210 in turn transmits a command, corresponding to the position request of the line of sight, to the motorized camera 215 having a line of sight 220. In parallel, the camera 215 transmits to the computer 210 the orientation of its line of sight, for example in the form of a data stream, thus allowing the processing of images from the camera during the period of movement. The camera 215 also transmits an image flow, preferably continuous, to the computer 210 which integrates in the images from this video stream the virtual objects according to the wishes of the user, pre-established scenarios and / or the context for forming the augmented video stream that is transmitted to a display system 225.

The quality of the system depends on the characteristics of the motorized camera which must be fast enough to respond to the requests from the control interface, sufficiently precise to best reach the requested positions and sufficiently enduring to maintain the quality of the preceding characteristics during the time.

The presented architecture concentrates the problem of the synchronization of the real and virtual environments on the camera, preferably a PT camera or a PTZ camera. Part of the coordinates of its line of sight is determined during the installation of the observation system while another part of the coordinates of its line of sight is determined in real time. The X, Y and Z coordinates as well as the roll (mi!) Along the line of sight are determined or calculated during the installation of the observation system while the heading and pitch (pan, tilt) are provided in real time. Similarly, if the zoom function on the line of sight is implemented, the zoom factor is determined in real time. To simplify calculations, the camera mark can be used as the reference mark. As a result, the X, Y and Z coordinates as well as the roll have zero values.

For reasons of robustness, the PTZ camera is preferably not accessible to the user. It is therefore necessary to integrate the latter within a suitable receptacle. Such a receptacle must advantageously protect the camera from external aggression, voluntary or otherwise, such as misuse, vandalism and bad weather. This receptacle presented by example a part without tint masking the camera to the user but allowing the latter to film the panorama without loss of light or deformation.

In order to avoid the problems of deformation related to the shape of this receptacle, the camera is here placed in the center of a sphere. Thus, whatever the position of the line of sight, the lens of the camera always remains equidistant from the receptacle.

In order to respect the practical aspect of the observation device, the point of rotation of the line of sight of the camera is preferably located at eye level. Alternatively, the camera can be deported to allow including a view that could not have the user.

According to a particular embodiment, the control interface, or PTZ interface, and the display system are arranged on the sphere used as a receptacle for the camera. The shape of the sphere is then used as a guide for the PTZ interface and for the display system, the guide being able to assume the appearance of a meridian-shaped movable rail.

FIG. 3, comprising FIGS. 3a, 3b and 3c, illustrates an example of a part of the observation device 300 comprising a sphere housing a PT camera or a PTZ camera, on which a control interface and a control system are mounted. viewing. Figure 3a shows a part of the device seen user side, Figure 3b shows a part of the device seen in profile and Figure 3c shows a part of the device seen camera side. The illustrated portion of the device comprises a sphere 305, a control interface 310 mounted on a guide 315 and a display system 320, here a binocular display. The sphere 305 preferably comprises two distinct parts, a first portion 325 transparent, semi transparent, or non-tain, located on the camera side and a second part 330 preferably opaque located on the user side. The control interface 310 advantageously comprises two handles allowing the user to move this control interface 310 on the opaque portion 330 of the sphere 305. For this purpose, the control interface 310 can move along the guide 315. , the guide 315 being rotatable about 180 degrees about the vertical axis of the sphere 305. It is of course possible to limit or extend the displacement of the control interface 310, in particular around the vertical axis of the sphere 305.

A motorized camera 335 is located in the center of the sphere 305 as shown in FIG. 3c. The movement of the camera 335 is controlled by the displacement of the interface 310. A movement along the vertical axis of the sphere of the control interface 310 causes a movement along the vertical axis of the camera 335 while a movement along the guide 315 of the control interface 310 causes a movement along an axis of the horizontal plane of the camera 335. The movement of the control interface 310 is detected by an optical encoder whose accuracy corresponds to that of the movements servo camera 335, or by a set of sensors. For example, a linear position sensor may be integrated with the control interface 310 to determine its position on the guide 315 while an angular sensor is placed on the link between the guide 315 and the sphere 305.

The computer used to control the movements of the camera 335 and to insert the virtual objects into the images from the camera 335 before they are transmitted to the display system 320 can be placed in the base (not shown) of the device 300. FIG. 4 illustrates the observation device 300 presented in FIG. 3. The observation device here comprises a foot 400 mounted on a base 405, a step 410 and a coin mechanism 415. It should be noted that the coin mechanism 415 is not necessary for the implementation of the invention.

By way of illustration, the camera 335 may be a camera equipped with a CCD (Charge-Coupled Device) sensor having a resolution of HD 108Oi, that is to say a resolution of 1080 lines with progressive scanning, with a 60 fps refresh rate and YUV-HD / HD-SDI type interface and RS-232 port to control camera movement and receive data related to its position. The binocular display 320 may comprise two Organic Light-Emitting Diode (OLED) displays, one for each eye, each having a resolution of 800 × 600 pixels (picture element), a resolution of 24 bits, a refresh rate of 60 images per second. second, a brightness greater than 50 cd / m ² , a contrast greater than 200: 1 and an interface type VGA (Video Graphics Array) and USB (Universal Serial Bus).

Figure 5 illustrates an example of an apparatus that can be used to control camera movements and to insert virtual objects into images from the camera. The device 500 is for example a microcomputer or a workstation.

The device 500 preferably comprises a communication bus 502 to which are connected: a central processing unit or microprocessor 504 (CPU,

Central Processing Unit);

a read-only memory 506 (ROM) that can include the operating system and programs such as "Prog";

a random access memory or cache 508 (Random Access Memory RAM) comprising registers adapted to record variables and parameters created and modified during the execution of the aforementioned programs;

a video capture card 510 connected to a camera 335 ';

an input / output card 514 connected to the camera 335 'and to a control interface 310'; and,

a graphics card 516 connected to a screen or a projector 320 '. Optionally, the apparatus 500 may also have the following elements:

a hard disk 520 which may comprise the aforementioned "Prog" programs and data processed or to be processed according to the invention;

a keyboard 522 and a mouse 524 or any other pointing device such as an optical pencil, a touch screen or a remote control allowing the user to interact with the programs according to the invention, in particular during the installation phases and / or initialization; a communication interface 526 connected to a distributed communication network 528, for example the Internet network, the interface being able to transmit and receive data; and, - A memory card reader (not shown) adapted to read or write processed or processed data according to the invention. In particular, the user can, according to a particular embodiment, insert a memory card so as to store images from the camera 335, real or augmented.

The communication bus allows communication and interoperability between the various elements included in the device 500 or connected to it. The representation of the bus is not limiting and, in particular, the central unit is capable of communicating instructions to any element of the apparatus 500 directly or via another element of the apparatus 500.

The executable code of each program enabling the programmable device to implement the processes according to the invention can be stored, for example, in the hard disk 520 or in the read-only memory 506. According to a variant, the executable code of the programs can be received via the communication network 528, via the interface 526, to be stored identically to that previously described.

More generally, the program or programs may be loaded into one of the storage means of the device 500 before being executed.

The central unit 504 will control and direct the execution of the instructions or portions of software code of the program or programs according to the invention, instructions which are stored in the hard disk 520 or in the read-only memory 506 or else in the other elements of aforementioned storage. When powering up, the program or programs that are stored in a non-volatile memory, for example the hard disk 520 or the read-only memory 506, are transferred into the random access memory 508 which then contains the executable code of the program or programs according to the invention, as well as registers for storing the variables and parameters necessary for the implementation of the invention.

It should be noted that the communication apparatus comprising the device according to the invention can also be a programmed apparatus. This device then contains the code of the computer program or programs for example frozen in a specific application integrated circuit (ASIC).

The apparatus 500 includes an augmented reality application such as Total Fusion's Fusion software (D'Fusion is a trademark of Total Immersion). The principle of real-time insertion of a virtual object in an image from a camera or other video acquisition means according to this software is described in the patent application WO 2004/012445.

FIG. 6 schematically illustrates certain operating steps of the observation device according to the invention. The operation of the observation device includes an installation and initialization phase (phase I) and a phase of use (phase II).

The installation and initialization phase comprises a step of calibrating the PTZ camera (step 600) and a step of loading the data used to enrich the real images (step 605). The loading of these data can be carried out during the installation of the observation device, when the device is switched on or at regular or programmed times.

During the use phase, the user's sighting movement information is received from the control interface (step 610) and used to control the line of sight of the camera (step 615). The camera then transmits the position of its line of sight and the zoom factor if the zoom function is implemented (step 620). The zoom factor then makes it possible to retrieve the intrinsic parameters of the camera and the distortion parameters (step 625) by comparison with the values established during the calibration of the camera during the initialization phase. In parallel, the heading and pitch data of the line of sight of the camera make it possible to retrieve the extrinsic data of the camera according to the current zoom level (step 630). A projection matrix is then determined from the data from steps 605, 625 and 630 (step 635). This projection matrix is used to determine the position of the elements, such as virtual objects, to be inserted into the image from the camera (step 640). These elements, for example a representation of virtual objects, are then inserted into the image from the camera (step 645) to form an augmented image. The augmented image is then presented to the user (step 650). Steps 610 to 650 are repeated for each image as indicated by the dotted arrow. It should be noted that steps 610 to 630 may not be repeated if there is no movement of the user.

The calibration of the camera (step 600) is intended to allow a good integration of the elements, such as virtual objects, in the images from the camera, by modeling the behavior of the camera for any type of environment in which it operates, that is to say by modeling the transformation of a point of space into a point of the image. The main calibration steps of the PTZ type camera are preferably the following,

- calibration of the radial distortion;

- Roll calibration of the shooting module (for example a CCD sensor);

- calibration of the field of vision;

- calibration of the distance between the optical center and the center of rotation of the camera; and,

- co-location of the real scene with respect to the camera. It is important to note that the focal length information given by PTZ cameras does not correspond to "metric" data. It is therefore necessary to precisely construct a correspondence table between the zoom value given by the PTZ camera and the intrinsic parameters found after calibration. It is first necessary to recall the projective geometry that makes it possible to express the relation between a point in space and its projected on the image plane.

The following notations are used in the rest of the description,

Where O is the position of the camera, and k is the line of sight; (θ, ïj, k) is the reference linked to the camera, in space; (D, u, v) is the landmark in the image; Where O 'is the center of the image, the coordinates of O' in the reference (D, u, v) are

(uo.vo);

Δ is the line perpendicular to the image plane and passing through the point O, Δ thus represents the optical axis of the camera; f is the focal length, that is to say the distance between the point O and the image plane; M is a point of space having coordinates (x, y, z) in the coordinate system

(θj, j, k); and, m is the projection of the point M in the image plane along line OM _1, the coordinates of m in the frame [θ, Tj, k) are (x'.y'.z ¹ ) with,

x '= f- z \ y' = f- z \ ï = f

The projection matrix P _r making it possible to go from a point M to the point m can be written in the following form,

and the matrix K of the affine transformation making it possible to pass from the reference [θj,], kj to the reference {D, u, v) can be written in the following form,

where (w ₀ v ₀ l) are the homogeneous coordinates of the point O in the frame (D, û, v), expressed in pixels, k _u is the horizontal scale factor and k _v is the vertical scale factor, expressed in pixels per millimeter.

The intrinsic parameters of the camera are the internal characteristics of the camera. The geometric model of the camera is expressed by the matrix product AT • /'.which gives the relation between the coordinates in the coordinate system (θj, j, k) of the point M (x, y, z) and the coordinates in the landmark (D, û, v) from the point q (u, v), projected from M into the image plane. The coordinates of the point q can therefore be expressed according to the following form,

u = k _u f + u ₀ ; v = kj - + v _o

Z Z

The matrix A intrinsic parameters can be expressed in the following form, a ,, 0 u _n

A = fK - P _r = 0 α ,, v _n with a _u = k _u f and a _v = kj

0 0 1 because it is possible to multiply all the coefficients of K - P _r by a factor f, the homogeneous coordinates being defined by a factor.

The intrinsic parameters must be related to other information more generally used in the world of video such as the resolution, in pixels, the size of the image sensor, in millimeters and the focal length according to the known relations.

The extrinsic parameters correspond to the rigid spatial transformation of the camera. The transformation matrix D, taking into account three degrees of freedom linked to the rotation R and three degrees of freedom linked to the translation T, can be written in the following form,

It is then possible to write the matrix of any prospective projection in the form of a 3x4 matrix defined only by a factor of approximately λ = f, where f is the focal length,

However, this is valid only for a so-called "pin-hole" camera, that is to say a theoretical camera that does not use a lens. Generally, cameras use at least one lens that induces Geometric and colorimetric distortions. Only geometric distortions are treated here. The geometric distortion introduced by a lens can be decomposed into two components, radial distortion and tangential distortion. The radial distortion is radially around the optical center in the image, in a negative manner (pincushion distortion, pincushion) or positive (barrel distortion, barrel). This distortion can be modeled polynomially, with monomials of even degrees. The tangential distortion can come from a misalignment of the lenses or imperfections thereof. By considering square pixels (α _v / α _u = 1), by placing at a focal distance of one pixel ("" = "" = 1) and starting from the hypothesis that the tangential distortion is zero (/? , = p ₂ = 0), the formulation of the distortion can be written in the following non-linear simplified form, u '= u - u ₀ V = v- v ₀ d ² = u' ² + V ² p - \ λ-α _x d ² + α ₂ d ⁴

U = M ₀ + u'p

V = v _Q + v 'p where (u, v) are the coordinates (in pixels) resulting from the perfect perspective projection, ai and α.2 are respectively the focal length (in pixels) horizontal and vertical, (uo, v _o ) are the coordinates (in pixels) of the optical center projected in the image and (U ₁ V) are the coordinates (in pixels) after distortion. This choice of simplification is explained by the fact that for a camera

PTZ, a single distortion is calculated for a set of focal values, the distortion parameters are then interpolated for the intermediate focal length values. It is therefore unnecessary to take this into account in the formulation. In order to obtain a sufficient precision while limiting the computation volume to allow a real-time execution of the observation device, the The solution is to consider the camera used as a simple ideal camera for which offsets are made outside the camera model. Thus the distortion is pre-compensated and other residual errors, such as the optical center, are not compensated at the projection but considering that these errors come from the positioning (position and orientation) of the camera. Even if this approach can theoretically prove to be wrong, it nevertheless allows to obtain good results.

In summary, all intrinsic and extrinsic parameter errors with respect to the perfect camera model are compensated for using the extrinsic parameters. For example, an optical decentering involving the intrinsic parameters (u _O) vo) is compensated for the extrinsic position and orientation parameters of the camera.

In addition, these compensations should not be considered constant because they may be a function of the variation of the user's aim control parameters. Thus, most of the compensations are calibrated for a set of focal values and interpolated when the focal length is between two focal values for which the compensations to be applied have been calculated. This approach makes it possible to correctly overlay the virtual objects on the images of a real scene by displaying the compensated images of the real scene and by rendering the virtual objects with a virtual camera simply offset and offset in position and orientation. The real-time calculation cost of the virtual PTZ camera model is then almost the same as that of a perfect virtual camera (pin-hole).

The approach for calibrating a PTZ type camera is however different than that set up for a conventional camera. Indeed, it is necessary to perform radial distortion calibration and field of view calibration steps for several zoom levels. The corresponding parameters are thus associated with a precise zoom level given in real time by the camera. Then, during the use of the device these parameters are interpolated according to the current zoom level of the camera.

Each calibration phase uses the compensations from the previous calibration phases. So, to be able to consider the camera as a pin-hole model, it is necessary to start by compensating the distortion. When the radial distortion is corrected, it is then possible to calibrate the roll of the image sensor which has the effect of having a rotation of the image around the center of the distortion. To this end, it is necessary to determine whether, during a course movement (horizontal rotation) the trajectory of the points in the image is a horizontal trajectory and if necessary, to compensate for this defect. When this compensation is established it is possible to measure the horizontal field of view simply by making a course movement. The focal length of the camera is thus calibrated and then a sufficiently complete camera model is available to compare the projection of a point theoretically on the optical axis and its actual position as a zoom is done.

After compensation for this decentering, it is finally possible to compensate for the distance between the optical center and the center of rotation of the camera. The previous formulation of the distortion shows that it is necessary to know the position of the optical center to estimate it. The calibration uses a pattern consisting of a set of coplanar points placed in a regular and known manner. In order to measure and estimate the distortion, one solution consists in comparing the theoretical projection of the set of points with the observed effective projection, which implies the knowledge of the extrinsic parameters of position and orientation of the target with respect to the camera as well as than the intrinsic parameters of horizontal and vertical focal lengths.

To be able to perform a calibration of the distortion, it is therefore theoretically necessary to know all the parameters of the camera (intrinsic and extrinsic) or to be placed in very precise conditions as to the placement of the test pattern with respect to the camera. This last approach being difficult to achieve, it is then advisable in theory to carry out a simultaneous calibration of all the parameters of the camera. The method used is to simultaneously calibrate all the parameters of the camera but keep only those related to the distortion, the others being estimated later by another method.

The distortion is estimated for several focal lengths of the camera. In addition, due to the mechanics of the PTZ camera, the distortion does not depend on the orientation of the camera which is preferably left in central orientation during the distortion calibration step. The first phase of the calibration, for a given focal length and expressed not by its metric focal length but by the value of the camera's encoder associated with the zoom factor, consists in placing a target in front of the camera lens. and to analyze the projection of the points of the test pattern into the image. The points of the chart are a set of coplanar points placed in a regular way (fixed and known horizontal and vertical spacings). The pattern is characterized by determining one of its points as a reference. This reference point defines the reference of the target from which all other points of the target can be expressed. The expression of the configuration of the test pattern with respect to the camera thus amounts to expressing the configuration (position and orientation) of this point with respect to the optical center of the camera. This configuration can be modeled by the data of a position T = (T _x , T _y , T _z ) which represents the position of the optical center with respect to the origin of the target expressed in the reference of the target, as well as of three Euler angles (γ, β, α) representing the rotations respectively around X, Y, Z successively applied in this order. Any point N of the test pattern can therefore be expressed, in the local coordinate system M of the test pattern, by the following relation,

Similarly, any point N of the test pattern can be expressed, in the C mark of the camera, by the following relation, P _{N l (} = Rot - (P _NIM -T) with, ca.cb ca.sb.sg - cg.sa ca.cg.sb + sa.sg

Rot = Rot, • Rot. • Rot _r = cb.sa ca.cg + sa.sb.sg cg.sa.sb - ca.sg

- sb cb.sg cb.cg where,

Rot _r 0 cos y cos = COS cos = s cos = S cos = sin /? ^ g = sin γ a being the angle of rotation around the x axis, β being the angle of rotation around the y axis and γ being the angle of rotation around the z axis.

According to the mathematical model, this point of the target projected on the image plane of the camera sees its image coordinates (u, v) generate an error (squared) E _n with respect to the measurement (u _n , v _n ) .

The calibration steps are as follows,

• choose a test pattern;

• determine the position of the points of the test pattern in the observed image;

• choose a reference P _{O /} M among these points of the image. This point P _{0 / M} therefore has (0,0,0) as coordinates in the frame of reference. All the other points have a position in the reference of the target which can be written in the form P _{N IM} = (x _N , Y _NO ) = (δ _w .i, δ _h .j, θ) where δ _w and δ _h are respectively the horizontal and vertical metric distance between two consecutive points on the test pattern;

• choose intrinsic parameters (horizontal focal point, vertical focal point, radial distortion, optical center) and extrinsic parameters (approximate position and orientation) for the camera. This choice represents the initialization of an iterative algorithm which will then estimate these parameters; and,

• apply the conjugate gradient technique by calculating the secant on the function representing the error between the model and the measure according to the relation fct = ΣE, At the end of the convergence of the conjugate gradient algorithm, all the parameters of the camera were estimated simultaneously. However, due to the poor numerical stability of certain parameters such as the focal length, it is preferable to keep only the two distortion parameters (ai, a2) which are associated with the value of the encoder of the zoom factor of the camera. This results in a map of the radial distortion as a function of the encoder values of the zoom factor. According to the formulation used, this distortion centers on the center of the image and not the optical center. However, since the camera is considered perfect, the optical center is at the center of the image. The error committed if the two centers are distinct is corrected beforehand by a rotation.

In practice, it is best to start with the lowest zoom factor and place the target in front of the camera, perpendicular to the optical axis, so that a maximum of points can be seen in the image without leave border. It is then necessary to freeze the image and to determine the position of the points of the pattern in the image by image analysis. When the points are matched, a calculation of the fitting (calculation by conjugate gradient) is launched and the result is saved. The zoom factor is then increased and the previous steps are repeated. As the zoom factor increases, it is necessary to move the sight away from the camera.

This calibration phase of the distortion can be automated. According to a first approach, the test pattern, controllable by computer, is placed on a mechanical rail so as to be able to move the sight away from the camera. According to a second approach, using the fact that the distortion is considered independent of the orientation of the camera, a set of sights is placed in front of the camera, each is placed at an angle and a distance to ensure that for each zoom factor chosen for the calibration, there is a heading configuration to observe at a good distance, a test pattern and only one.

After calibrating the distortion, it may be necessary to calibrate the roll of the image sensor. Indeed, it is possible that the sensor of the camera, for example a CCD sensor, is not perfectly oriented with respect to the movement mechanics of the camera. It should be noted that since the distortion is radial, the distortion compensation can not compensate for a roll defect of the image sensor. To compensate for the roll defect, it is necessary to virtually rotate the virtual camera around its optical axis by an angle θ which must be estimated. This angle is considered to be independent of the zoom factor.

To estimate the angle θ, it is necessary to determine a reference point situated in the observed scene, substantially in the center of the image coming from the camera, and to make a course movement of the camera. If the angle is zero, the reference point must remain permanently on the mid-height line, or horizon line, of the image coming from the camera otherwise, it passes below or above this line of halfway up. The angle θ is then estimated so that the reference point has a perfectly horizontal movement in the image. This step only needs to be performed for a single zoom value, for example the lowest zoom value.

In practice, this step can be automated by following a decorative point during a camera heading movement and calculating the compensation to be made so that the movement is horizontal in the image. For more robustness, this calculation can be performed on a set of points, the angle θ being evaluated according to an average.

As previously described, the zoom factor is controlled by indicating a value given to the encoder in charge of the zoom to calibrate the distortion. We therefore do not have the focal value directly. To calibrate the field of view, the principle is to create a map between the values of the zoom encoder and the field of view θ _u (Field Of View angle). The use of the field of view makes it possible to determine the value of the focal length in pixel according to the following relation,

FoW α, where FoW is the width of the field of view {Field of View Width) corresponding to the horizontal resolution in pixels (known). In order to map the zoom encoder values as a function of the field of view θ _u, a solution consists in aiming at a point or a vertical line of the decoration located at the center of the image when the heading is at zero and changing. the heading angle until the point or line is precisely on the right or left edge of the image. This makes it possible to measure the half-angle of the horizontal field of view and thus to calculate the value of the horizontal focal length. The value of the horizontal focal point makes it possible to deduce that of the vertical focal length from the known ratio of the focal lengths. This solution is applicable if the relationship between the heading encoder and heading is linear, known and symmetrical with respect to the center of course.

This procedure could be automated by following a point or line of scenery during a course move, until the tracked item disappears. The heading encoder then directly gives the value of the half horizontal field of view. When the optical center is not quite in the center of the image, the decentering can be corrected by a rotation combining heading and pitching to avoid inserting this decentering into the intrinsic parameters.

As mentioned before, it was made here the choice that the camera should be considered perfect. Thus, the heading and pitch errors with respect to the optical axis of the camera are compensated not in the form of a complexification of the model of the camera but by means of an offset on the position and the orientation of the camera. This pitch and pitch shift is preferably measured for all zoom factors. The measurement is made, for example, by aiming at a relatively far point of the scene while using the lowest minimum zoom factor. For each zoom factor increase, the pitch and pitch offset is set manually on the virtual camera so that the virtual point associated with the target point remains superimposed on it. Naturally, this procedure can be automated by following a point of the scene during a zooming motion and offsetting the pixel error by a shift in heading and pitch.

The calibration of the offset between the optical center and the center of rotation of the camera is the last phase of the calibration step. This Calibration of the shift between the optical center and the center of rotation of the camera is not necessary if the filmed scene and the position of the virtual objects in the filmed scene are still far from the camera. Indeed, in the other steps of the calibration, it was considered that the optical center and the center of physical rotation of the camera was in the same place. This can be false but has almost no impact as soon as the visual elements are a few tens of centimeters apart. However, for close points, the previous compensations are not sufficient and the taking into account of this shift of the optical center may be necessary. It is considered here that the offset exists only along the optical axis, which is consistent with the fact that the previous calibrations are intended to compensate for the offset, except on its optical axis component.

In order to determine the offset between the optical center and the center of rotation of the camera, one solution consists in aiming at a point of the scenery located physically close to the camera and whose physical distance with respect to the camera has previously been measured. This point is preferably chosen in the middle of the image when the line of sight of the camera is oriented with zero heading and pitch. For each zoom factor increase, the camera heading is changed until the target point is at the edge of the image. The offset is then adjusted manually so that the virtual point associated with the real point is superimposed on it. It is possible to automate the calibration of the shift between the optical center and the center of rotation of the camera by automatically following a point close to the scene by varying the zoom factor and compensating for the pixel error by a shift in translation. along the optical axis.

If the calibration of the camera is necessary to obtain a good integration of the virtual objects in the images coming from the camera, it is also necessary to know precisely the position and the orientation of the object with respect to the camera or in an equivalent way to know the configuration of the real camera in relation to the real world that it films. This calculation is called a "pose" calculation. To perform this pose calculation, it is necessary to relate an object of the three-dimensional scene with its two-dimensional image. For this, it is necessary to have the coordinates of several points of the object in the real scene (three-dimensional coordinates), all expressed with respect to a point of the object considered as object reference during the calculation of the pose. For this set of points, it is also necessary to have coordinates of their projection in the image (two-dimensional coordinates). A multi-level pose evaluation is then implemented.

A first level is to perform a quick calculation to obtain an approximate pose while a second level, by a longer calculation, uses the approximate pose to improve the pose estimate, iteratively.

The first-level estimate is based on a first-order approximation of the "weak perspective" perspective projection model. This method is fast and robust as long as the points chosen in the real space are distributed over the entire surface of the object and are not coplanar. In addition, to converge, it is necessary that the object is visible rather in the center of the image and located relatively far from the camera. This known method is presented, for example, in the article "Model-Based Object Pose in 25 Lines of Code" D. DeMenthon and L. S. Davis, International Journal of Computer Vision, 15, pp. 123-141, June 1995 and in the article "Object Pose: The Link between Weak Perspective, Paraperspective, and FuII Perspective", R. Horaud, F. Dornaika, Lamiroy B., S. Christy, International Journal of Computer Vision, Volume 22, No. 2, 1997.

This method performs a pose calculation by choosing one of the points of the object, in three-dimensional space, as a reference frame. However, it was noted that the quality of the pose estimate was not the same depending on the chosen reference point and sometimes it is useful to remove some points to obtain a better result, while keeping at least five points not coplanar. It is thus advantageous, if the object comprises r points in the three-dimensional space, to perform r. (R-5) pose calculations by taking each of the r points as a reference point and then to delete at each iteration, among the other points, the furthest point from the reference point. For each of these pose calculations, an average of the reprojection error in the image plane is calculated. The final pose is the one that corresponds to the smallest error.

The second-level estimation uses a stochastic exploration, guided by an error criterion, of the configuration space, that is to say the six-dimensional space corresponding to the set of pairs (position, orientation) close to the current pose. The idea is to start from the current pose then to move a small random shift in the configuration space. If the new configuration better meets the error criterion then it becomes the new reference or else another random offset is used. The steps of this method are as follows,

- select the pose determined according to the first level estimate;

- create random offsets of the current pose configuration;

for each of these random offsets, calculate the error generated by this pose. If the error is smaller than that obtained so far, then this pose is considered better and memorized;

if at the end of the estimation of these random shifts, none has improved the error of the current pose then a failure counter is increased, otherwise the failure storyteller is reset and the new one is reset. current configuration is the one memorized; and,

- If the failure counter reaches a predetermined threshold, the process is stopped otherwise the previous three steps are repeated.

The calculation of the error is preferably done by summing, for each point, the orthogonal distance to the square between the point of the object (P), in the three-dimensional space, in the considered pose (P) and the straight line, in the three-dimensional space resulting from the projection in the two-dimensional image associated with the point, in the three-dimensional space, and the projection center of the camera (defined by the center of the camera in the three-dimensional space and a unit vector u whose direction is defined by the center of the camera and the considered point, in three-dimensional space).

When the camera is calibrated and its configuration (position / orientation) relative to the environment is known, it is possible inserting virtual objects, or any other element such as a secondary video stream, into the images from the camera. The insertion of the virtual objects in the video stream can be realized in real time by the merger software of the company Total Immersion as indicated above. The choice of virtual objects to be inserted into the video stream can be done by geolocation by constituting a database. This database can be made manually or from existing databases via, for example, a connection to a network such as the Internet. Alternatively, the camera of the observation device can be deported as shown in FIG. 7. According to this embodiment, one or more cameras are installed at locations that are not directly accessible to the users such as at the top of a camera. building, in a cave or underwater. However, these cameras are connected to the control interfaces of the users so that they control the movements from a control and viewing platform. The use of cameras and possibly remote computers, also allows the cameras to be placed in inaccessible places to protect the equipment against vandalism or theft. FIG. 7 illustrates an observation device 300 'comprising a sphere 305' on which are movably attached a control interface 310 'and a display system 320''.The sphere 305' is preferably coupled to a foot 400 ' mounted on a base 405 'comprising a computer 500' which can also be deported A step 410 'allows the user to position themselves appropriately facing the control interface 310' and the display system 320 ". The camera 335 "is here deported to be fixed on a chimney 700 of a house 705.

Naturally, to meet specific needs, a person skilled in the field of the invention may apply modifications in the previous description, particularly with regard to the sighting movement control means and the shape of the observation device.

Claims

A method for a real-time augmented reality observation device (300) comprising an image sensor (335), a display system (320), and an indirect control interface (310) with a direct view, which method is characterized in that it comprises the following steps,

receiving a request comprising axis of view guidance information transmitted by said control interface (610);

transmitting said line of sight orientation information to said image sensor (615), said image sensor being mobile and motorized;

receiving from said image sensor the orientation of its line of sight (620); receiving at least one image originating from said image sensor;

determining in said received image the position to which at least one data item is to be inserted, according to the orientation of the axis of view of said image sensor (640); and,

inserting in real time into said received image said at least one datum at said determined position (645).

2. The method of claim 1 wherein said display system is movable and mechanically linked to at least one element of said control interface.

3. Method according to claim 1 or claim 2 characterized in that it further comprises a calibration phase of said image sensor (600).

4. Method according to claim 3 characterized in that said calibration step comprises the calibration of at least one of the parameters included in the set of parameters comprising the radial distortion correction of said image sensor, the roll correction. of said image sensor, corrections of heading and pitch of the axis of view of said sensor of images and the shift between the optical center and the center of rotation of said image sensor.

5. Method according to claim 4 characterized in that said image sensor comprises a zoom function and in that the calibration of said at least one parameter is performed for a plurality of zoom factors.

6. Method according to any one of the preceding claims, characterized in that it further comprises a co-location step of said image sensor and the scene observed by said image sensor to determine the laying of said at least one a data item to be inserted in said received image in said observed scene.

7. Method according to any one of the preceding claims, characterized in that said at least one piece of data to be inserted in said received image is dependent on the geographical position of said observation device.

8. Method according to any one of the preceding claims characterized in that said at least one piece of data to be inserted in said received image is a representation of a virtual three-dimensional model animated or not.

9. Method according to any one of the preceding claims characterized in that said orientation of the line of sight is defined according to two degrees of freedom and in that said image sensor comprises a zoom function.

10. Computer program comprising instructions adapted to the implementation of each of the steps of the method according to any one of the preceding claims.

An augmented reality observation device (300) comprising means for connection to an image sensor (335), a display system (320) and an indirect control interface (310) for direct viewing, which device is characterized in that it comprises the following means, - means for receiving guidance line orientation information transmitted by said control interface; means for controlling the orientation of the axis of view of said image sensor according to said orientation information received, said image sensor being mobile and motorized;

means for receiving the orientation of the axis of view of said image sensor;

means for receiving at least one image from said image sensor;

12. Device according to claim 11 wherein said display system is movable and mechanically linked to at least a portion of said control interface.

13. Device according to claim 11 or claim 12 characterized in that it further comprises means for transmitting said received image comprising said at least one datum.

14. Device according to any one of claims 11 and 13 characterized in that at least one of said image sensor and storage means is remote from said observation device.