WO2017077217A1

WO2017077217A1 - 3-d calibration of a video mapping system

Info

Publication number: WO2017077217A1
Application number: PCT/FR2016/052756
Authority: WO
Inventors: Julien BERTA
Original assignee: Smart Pixels
Priority date: 2015-11-04
Filing date: 2016-10-24
Publication date: 2017-05-11
Also published as: FR3043235A1; FR3043235B1

Abstract

The present invention relates to the 3-D calibration of a video mapping system comprising a camera (10), a video projector (20) and a flat surface (30) in the form of a substantially rectangular or square plane, mechanically connected to an object (O) intended to receive the projection of at least one image and having at least one marker (Ml, M2, M3, M4) at the corners of said surface, said calibration comprising: a pre-transformation (S5) using a pre-transform pt" of a plurality of second markers (A1, A2, A3, A4) the 2-D coordinates of which ({C"_A₁; C"_A₂; C"_A₃; C"_ A₄}) in the image frame of reference of the projector (R2) are known, said first pre-transform pt" being initialised during a first iteration as being equal to a non-degenerate homoographic matrix, and an iterative calculation of said pre-transform pt", as a function of homographies Hbc and Hep, in order to transform, in the image frame of reference (R2) of the projector, the 2-D coordinates of the second projected markers (Al, A2, A3, A4) in order to make same coincide with the 2-D coordinates ({C"_I1; C"_A2; C"_I3; C"_I4}) of third markers (I1, I2, I3, I4), referred to as ideal markers.

Description

3D CALIB RATION OF A VIDEO MAPPING SYSTEM

Technical field and prior art

The present invention relates to video mapping (or video mapping), also known as projection mapping.

The present invention relates more particularly to the 3D calibration of a video mapping system.

The video mapping, or projection mapping, consists mainly of projecting at least one image or a video stream on two-dimensional or three-dimensional objects; this makes it possible to recreate images or video on objects in particular to recreate virtual universes.

Video mapping is thus a projection technology that transforms objects, often of irregular shapes, into display surfaces for projecting images or video.

These objects can be complex architectural landscapes such as a building (cathedral, church, historical monument, etc.) as well as objects of smaller sizes such as for example consumer products such as clothing, shoes, handbags. , toys, models or prototypes.

The present invention thus has many advantageous applications in the marketing field for example for the design of a product or a packaging or for the choice and / or configuration of a vehicle in a car dealership, but also in the field of virtual reality or in the video game.

Of course, the present invention finds other advantageous applications in other fields such as, for example, in events, sound and light shows or street shows for projecting on large objects (for example facades). of buildings) images and / or video. In its simplest version, we can provide a static video mapping in which the images are processed beforehand to be projected by a video projector whose position is fixed.

In this case, the images are transformed so as to coincide with the object serving as a projection surface. We understand here that the video mapping is for single use; if you move the projector and / or the object, the images must be processed again so that the projector projects the images correctly onto the object. To remedy this, video mapping sometimes requires the use of a video projector coupled to a camera; the camera is able to detect the object on which the image or the video stream is to be projected so as to coincide with the contours, the shape and the relief of the object.

By camera is meant throughout the present description which follows any optical device for acquiring at least one image.

This approach therefore requires the use of calibration software to match the virtual space with the real environment.

The principle of this calibration is to map a two or three dimensional object into a virtual space that mimics the real environment.

In this way, the central unit of the computer knows precisely the locations of the object where the various information relating to the object to be simulated must be projected.

Thanks to this mapping of the object, the software can interact with the video projector to transform the image that one wishes to project on the surface of the object. Calibrating the camera and projector provides a projected image that perfectly matches the contours, shape and relief of the object.

A publication Hirooka known in the prior art ET AL "Calibration Free Display System Using Video Projector Onto Real Object Size" from ^January 1, 2006 which teaches a 2D calibration.

For this 2D calibration, it is proposed to calculate homographies.

However, this homography calculation is used here as a tool for simplifying the 3D space into a set of 2D subspaces.

In this publication, all calculations made to determine homographies to move from one space to another are thus performed in a 2D environment.

Such simplification then makes it possible to avoid working in the 3D space.

It should be noted here that this simplification does not make it possible to obtain an accurate 3D calibration of the system. Also known is a 3D calibration algorithm of a video mapping system proposed by AUDET et al in a 2009 scientific publication entitled "A user-friendly method to geometrically calibrated projector-camera System", IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshop.

In this publication, the proposed algorithm involves printed markers and projected markers.

The general concept of this calibration algorithm is to match the projected markers with predefined locations.

Once these markers are aligned, it is possible to calibrate the system and project the image correctly on the object.

The Applicant submits, however, that the proposed algorithm for system calibration is not fully satisfactory.

The proposed algorithm is indeed very sensitive to light variations; it is also not possible to calibrate the system during the projection (in "run-time"); this is called pre-calibration.

Several other disadvantages in AUDET et al will be highlighted in the following description.

Summary and object of the invention

The present invention aims to improve the situation described above.

The present invention is intended in particular to improve the approach proposed in AUDET et al.

For this purpose, the object of the invention relates according to a first aspect to a calibration method, implemented by computer means, for the calibration of a video mapping system comprising a camera and a video projector.

According to the invention, the video projector is capable of projecting at least one image onto an object positioned in a workspace; this workspace is called here world landmark.

Advantageously, the video mapping system comprises a flat surface; this flat surface is in the form of a plane having at least three corners. Preferably, this flat surface is substantially rectangular or square. The flat surface has in this case four corners. However, other embodiments may be provided in which the flat surface has more than four corners.

Preferably, this flat surface is mechanically linked to the object.

According to a variant, the object is placed, directly or indirectly, on this flat surface. In this case, the flat surface can also serve as support for the object.

According to another variant, the object comprises this flat surface.

According to the invention, the plane surface has, at at least three of its corners, a first marker whose spatial coordinates, called 3D, in the world coordinate system are known.

The flat surface therefore comprises at least three first markers positioned respectively in the corners of the surface.

These markers delimit the surface.

According to the invention, the method comprises a first calibration phase for estimating the pose of the camera and its optical parameters (so-called intrinsic).

Depending on the mathematical model of camera used, the intrinsic parameters of the camera include the focal length, the optical center and possible distortion parameters of the camera.

The pose of the camera, or extrinsic camera parameter, corresponds to the position and orientation of the camera relative to an object in the world landmark.

Advantageously, this first phase comprises a capture by the camera of an image, called the first image, comprising the first markers.

Following this capture, the first phase comprises a first detection step 1) during which the first markers are detected in the first image and then, by an image processing algorithm, the 2D coordinates of the first markers are determined in a first image. marker associated with the image captured by the camera; this mark is called the image mark of the camera.

The first phase then comprises a first determination step 2).

During this step, a first homography denoted H _{bc is determined} which allows the passage of the 3D coordinates of the first markers in the world coordinate system to the 2D coordinates of these markers in the image frame of the camera; it should be noted here that this homography H _bc is related to the pose and the optical parameters of the camera. It is understood that the markers are coplanar and that there is a world landmark where the coordinates of the markers can be expressed in a plane (that of the flat surface).

The homographic transformation (from a plane to a plane) is therefore applicable.

The determination of this first homography is characteristic of the present invention.

The method according to the present invention then comprises a second calibration phase for estimating the installation of the projector and its optical parameters (so-called intrinsic).

Depending on the mathematical model of the projector used, the intrinsic parameters of the projector include in particular the focal length, the optical center and any projector distortion parameters.

In the same way, the installation of the projector, or extrinsic parameter of the projector, corresponds to the position and the orientation of the projector relative to an object in the world landmark.

More particularly, this second phase comprises a pre-transformation step 3) during which a pre-transformed pt ", said first transformed, is applied to a plurality of second markers whose 2D coordinates in the reference associated with the projector, called the image reference mark. of the projector, are known.

Preferably, this first pre-transformed pt "is initialized during a first iteration as being equal to a non-degenerate homographic matrix.

Here we mean by non-degenerate transformation any geometric transformation of space that is invertible.

Then, there is provided a projection step 4) during which the video projector projects on the flat surface the second transformed markers.

Prior to this projection, provision is also made for a first filter including checking that the second transformed markers are included in the flat surface. This first filtering can be performed by comparing the 2D coordinates of the second markers transformed in the image frame of the projector with the 2D coordinates of the first markers.

This pre-transformation followed by the first filtering is characteristic of the present invention.

Positioning the first markers in the corners of the flat surface is very advantageous; this makes it possible to easily filter the pre-transformed markers which are outside the plane by comparing them with the coordinates of the first markers. This is not possible in AUDET et al.

Indeed, the algorithm proposed in AUDET et al does not guarantee that all the markers are projected on the flat surface.

A second filtering is then performed optionally to verify that the corners of the second transformed markers all belong to the image to be projected.

More particularly, this second filtering consists in verifying that the second transformed markers are included in the image to be projected by comparing their 2D coordinates of the second markers with the size of the image in the image reference frame of the projector.

This second filtering is characteristic of the invention. It thus ensures that no truncated marker (projecting beyond the frame of the projected image) is projected and then detected.

In AUDET et al, truncated markers introduce a significant amount of error into the result.

Following these filterings, the second calibration phase comprises a capture by the camera of a second image comprising the second projected markers (which are in the plane and in the image).

Following this capture, a second detection step 5) is provided during which the second markers are detected in the second image and then, by an image processing algorithm, the 2D coordinates of the second markers in the reference mark are determined. picture of the camera.

The second calibration phase then comprises a second determination step 6); during this step, a second homography H _cp is determined which is related to the pose and optical parameters of the projector and which allows the passage of the 2D coordinates of the second markers projected in the image frame of the camera to the 2D coordinates of these markers in the image mark of the projector.

The method according to the present invention finally comprises a treatment phase.

This treatment phase comprises a third calculation step 7) during which the first pre-transformed pt "is recalculated according to the first Hb _c and second H _cp homographies.

Here, we recalculate this pre-transformed pt "so as to transform, in the image frame of the projector, the 2D coordinates of the second projected markers to coincide with the 2D coordinates of third markers, called ideals (the 3D coordinates of these third markers in the world landmark are predetermined). According to the invention, the set of steps 3) to 7) is repeated, using for each iteration the first pre-transformed pt "calculated during the third calculation step of the previous iteration, until the pre -transformed pt "is substantially equal to an identity matrix.

According to the invention, it is collected at each iteration in a so-called calibration list the coordinates of the first markers in the world and camera landmarks and second markers in the world, camera and projector landmarks to estimate:

- the installation of the camera in the world reference and its optical parameters on the one hand, and

- the installation of the projector in the world reference and its optical parameters on the other hand. The treatment phase according to the present invention thus makes it possible to collect a list of points containing the information for the 3D calibration of the videomapping system: by this succession of technical steps, characteristic of the present invention, an iterative method is available which makes it possible to to build a list of points: at each iteration of the algorithm, the first markers (3D points of the world marker and 2D point of the camera marker) and the second markers (ideal 3D points of the world marker, 2D point of the camera marker and points 2D of the projector mark) are added to this calibration list.

It will furthermore be noted that, advantageously, unlike AUDET et al., The method proposed here in the context of the present invention is more robust and more reliable insofar as the second markers can not be projected outside the scope of the present invention. plan or picture.

The Applicant furthermore submits that the calibration method proposed here in the context of the present invention makes it possible to calibrate the running system (in "run-time").

In AUDET et al, it is not possible to apply the algorithm in "run-time"; the calibration in AUDET et al is comparable to a pre-calibration necessarily prior to the use of the system.

The Applicant furthermore submits that the calibration method proposed herein in the context of the present invention does not require the detection of the printed markers and the projected markers in the same camera image, which makes the detection robust and significantly less sensitive to the conditions. luminous ambient.

Advantageously, the treatment phase further comprises a first calculation step during which the 2D coordinates of the third ideal markers in the Camera image markers are calculated based on the first H _bc homography and 3D coordinates of the third markers in the world landmark.

According to a particular embodiment, the 2D coordinates of the third markers in the image frame of the camera are calculated according to the following formula:

/ '= H _cb I

in which :

Γ is the representative vector of the 2D coordinates of at least one of the third markers in the image frame of the camera, and

I is the representative vector of the 3D coordinates of at least one of the third markers in the world coordinate system.

In a particular embodiment, the processing phase comprises a second calculation step during which another pre-transform pt ', said second pre-transformed, is computed in order to transform, in the image reference frame of the camera, the coordinates 2D of the second projected markers to make them coincide with the 2D coordinates of the third markers

In this mode, the second pre-transformed pt 'is calculated according to the following formula:

/ '= pt'A'

in which :

- A 'is the representative vector of the 2D coordinates of at least one of the second markers in the image frame of the camera.

Preferably, the first transform pt '' is calculated according to the following formula:

pt "= Hcp ^ pt'Hcp

Preferably, during the first iteration, the 2D coordinates of the second markers in the image frame of the projector are determined randomly (provided that the second markers do not overlap with the first markers).

Preferably, during the first iteration, the first pre-transformed pt "is initialized as being equal to the identity matrix.

By choosing during the first iteration the identity matrix as the first pre-transformed pt ", the results are particularly advantageous. The calibration of the video mapping system is indeed very fast: the projected markers coincide with the ideal markers after only three iterations.

We then have:

A = I

In this way, the proposed algorithm makes it possible to accumulate at least 4 characteristic points (known in the three world, camera and projector markers) in only three iterations.

Accurate and robust calibration can therefore be achieved in a small number of iterations.

When the camera captures a first image, the projector projects a black image toward the flat surface.

The skilled person will understand here that, alternatively, it is possible to temporarily turn off the projector to have a black image on the flat surface or put the projector in "shutter" mode.

The projection of such a black image allows a calibration which is not sensitive to the variations of light, which is not found in AUDET et al which on the contrary is very sensitive to these variations: in AUDET et al, the least light variation makes calibration impossible; in AUDET et al, it is necessary to detect the printed markers and the projected markers in the same image.

As explained above, the set of steps 3) to 7) is reiterated in the method by using for each iteration the first pre-transformed pt "calculated during the third calculation step of the previous iteration and then pre-transforming the coordinates. 2D of the second markers in the image reference of the projector, this iteration is carried out until the pre-transformed pt "is substantially equal to an identity matrix to have:

pt "* A" = I "→ Id * A" = I "

let A "= 1" → A = I

The method therefore comprises an iterative loop which stops when the pre-transformed pt "is substantially equal to an identity matrix.

To reach this equality, we can calculate the distance between the first pretransformed pt "and the matrix identity: distance (pt", Id)

Preferably, it is sought that this distance is less than a predetermined threshold, noted for example ε: distance (pt ", Id) <ε

We understand here that we select a threshold ε sufficiently low to have a precise result.

The Applicant submits here that it is sufficient by correctly setting the threshold in question about three iterations to achieve the expected result and get a good calibration of the system.

Those skilled in the art will understand here that the distance can be calculated in different ways: it can be done by calculating a difference, the absolute value of a difference, a ratio, etc. It is understood here that the first markers are so-called physical markers.

Advantageously, a variant can be provided in which these first markers are obtained beforehand by printing the flat surface with an invisible ink of the infrared or ultraviolet ink type.

Preferably, the detection of the first markers in the first image comprises in this case a lighting of the surface with an infra-red or ultraviolet lamp.

Alternatively, it is also possible to provide a variant in which the first markers are obtained beforehand by etching the flat surface.

In this case, the detection of the first markers in the first image may preferably comprise the detection of the reliefs formed by the etching in the plane surface.

Correlatively, the subject of the present invention relates, according to a second aspect, to a computer program which comprises instructions adapted to the execution of the steps of the method as described above, this in particular when said computer program is executed by at least one processor.

Such a computer program can use any programming language, and be in the form of a source code, an object code, or an intermediate code between a source code and an object code, such as in a partially compiled form, or in any other desirable form. Similarly, according to a third aspect, the object of the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method as described above. . On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium may comprise storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit type ROM, or a magnetic recording means or a hard disk.

On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal can be conveyed via an electric or optical cable, by conventional radio or radio or by self-directed laser beam or by other ways. The computer program according to the invention can in particular be downloaded to an Internet type network.

Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the method in question. The subject of the present invention finally relates, according to a fourth aspect, to a video mapping system comprising a camera and a video projector capable of projecting at least one image onto an object positioned in the world reference.

The system comprises a flat surface in the form of a plane having at least three corners and being mechanically connected to the object, said plane surface having at at least three of its corners a first marker whose spatial coordinates , say 3D, in the world landmark are known.

According to the invention, the system comprises computer means configured for implementing the steps of the method described above.

More particularly, the system includes a first computer calibration module for estimating the pose of the camera.

This first module includes:

first image processing means configured to detect, in a first image captured by the camera, the markers, and to determine the 2D coordinates of the first markers in the image reference frame of the camera,

- First determination means configured to determine a first homography H _bc , related to the installation of the camera, allowing the passage of 3D coordinates of the first markers in the world coordinate system to 2D coordinates of the first markers in the image frame of the camera. It is understood that the markers are coplanar and that there is a world landmark where the coordinates of the markers can be expressed in a plane (that of the flat surface). The homographic transformation (from a plane to a plane) is therefore applicable.

The system further includes a second computer calibration module for estimating the placement of the projector.

This second module preferably comprises:

pre-transformation means configured to transform with a predetermined pre-transformation pt 2 of the second markers whose 2D coordinates in the image reference of the projector are known (this pre-transform pt "being initialized during a first iteration as being equal to a non-degenerate homographic matrix),

first filtering means configured to check that the second transformed markers are included in the plane surface by comparing the 2D coordinates of the second markers transformed in the image frame of the projector with the coordinates of the first markers (here, the filtering makes it possible to suppress the second markers that would not be theoretically projected on the map),

projector control means configured to project on the flat surface the second transformed markers,

second image processing means configured to detect, in a second image captured by the camera, the second transformed markers, and to determine the 2D coordinates of the second markers transformed in the image frame of the camera, and

second determination means configured to determine a second homography H _cp , related to the installation of the projector, for the passage of the 2D coordinates of the second markers in the image frame of the camera to the 2D coordinates of the markers in the image frame of the projector.

Finally, the system comprises a computer processing module which notably comprises a computer which is configured to recalculate, according to the first H _bc and second H _cp homographies, the first pre-transformed pt "to transform, in the image reference of the projector, the 2D coordinates of the second projected markers to coincide with the 2D coordinates of third markers whose 3D coordinates in the world coordinate are predetermined.

The computer processing module further comprises computer collection means configured to collect, at each iteration, the coordinates of the first markers in the world and camera markers and second markers in the world, camera and projector markers in a list of points, called the calibration list.

This list is advantageously used for the 3D calibration of the video mapping system by allowing the estimation of the pose of the camera in the world reference and its optical parameters, as well as the estimation of the installation of the projector in the reference world and of its optical parameters.

The system further comprises a processor configured to control all of said computer means above for said calculator to calculate the first pretransformed pt "until substantially equal to an identity matrix.

In a particular embodiment, the second module comprises second filtering means configured to verify that the second transformed markers are included in the image to be projected by comparing their 2D coordinates with the size of the image in the projector mark. Thus, the object of the present invention, by its various functional and structural aspects described above, allows a robust and reliable calibration of a video mapping system, regardless of the conditions of use of the system.

Brief description of the attached figures

Other features and advantages of the present invention will emerge from the description below, with reference to FIGS. 1 to 5, which illustrate an embodiment of this embodiment which is devoid of any limiting character and on which:

Figure 1 shows a flowchart illustrating the method of the invention according to an exemplary embodiment of the present invention;

FIG. 2 represents a schematic view of the video mapping system according to an exemplary embodiment of the present invention;

FIG. 3 diagrammatically represents the calibration principle of the video mapping system illustrating the position of the different markers on the flat surface;

FIG. 4 schematically represents the calibration of the completed system when the projected markers are merged with the ideal markers; FIG. 5 represents an example of a particular application of the invention in which the object is a car model placed on a flat surface serving as a support. Detailed description according to an advantageous exemplary embodiment

A method of calibrating a video mapping system will now be described in the following with reference to Figures 1 to 5.

Allowing the calibration of a video mapping system 100 so that the images projected by a video projector 20 respect the contours, the shape and the relief of an object O is one of the objectives of the present invention.

The example described here and illustrated in FIG. 5 relates to an application in which the video projector 20 is set to project on a car model O, or more broadly an object O, an image corresponding for example to the dressing options of a car ; this example of application is particularly interesting for car dealerships by allowing the buyer to select and visualize in real time the various options selected (colors, rim, sunroof, headlights, mirrors, etc.) for the car that he wants to buy.

It will be understood that this application example is one example among others, and that obviously other applications may be envisaged within the scope of the present invention.

For example, we could imagine an example of an application for people working in marketing and responsible for proposing to decision makers (customers or managers) one or more possible projects for the packaging of a packaging (colors, typography, name of product, etc.).

In this case, the video mapping tool makes it possible not to create a prototype each time. Just project the desired image.

There may also be other examples of applications in marketing by presenting for example in the window the different skins, patterns and accessories possible of a shoe, a garment or a handbag.

In the example described here and as illustrated in FIGS. 2 or 3, the system 100 therefore conventionally comprises a camera 10 and a video projector 20.

The system 100 further comprises a flat surface 30 which is in this example of rectangular or square shape.

It is understood that this surface may have other shapes. In any case, this surface has at least three corners. This surface 30 is used for the calibration of the system 100; it is therefore characteristic of the present invention.

As illustrated in FIG. 5, this flat surface 30 is mechanically linked to the object O.

It will be noted here that, in this example, the object O is placed directly on the surface 30. The skilled person will understand here that the object O can also be placed indirectly on this surface 30, for example via a base (not shown here).

It should be noted that the method described above also works when the object O itself incorporates such a surface 30.

In the example described here and as illustrated in FIG. 3, the surface 30 has at its four corners (see FIGS. 2 and 3) markers respectively denoted by M _1s M ₂ , M ₃ and M ₄ .

The Applicant submits here that the invention functions correctly with at least three markers.

It will be noted here that the spatial coordinates (or 3D) of its markers M _ls M ₂ , M ₃ , and

M ₄ in the world reference R0 are known; they are noted below {CJVIi _; C_M _2; C_M _3; C_M ₄ }. M may also be noted matrix representative of the coordinates of these markers M _ls M ₂ , M ₃ and M ₄ in the world reference R0. It is understood here that the markers are coplanar and that there is a world landmark where the coordinates of the markers can be expressed in a plane (that of the flat surface).

In the example described here, the method thus comprises a first phase of calibration PI implemented by a module 110 (for example an electronic circuit) whose object is to estimate the pose of the camera 10.

During this first phase PI, the camera 10 (which includes an image sensor) captures during a step S1 an image Iml.

It will be understood that this image Im1 comprises the markers M _1s M ₂ , M ₃ and M ₄ . Following this IF capture, there is provided a detection S2 in Iml image of these markers M _ls M _2, M ₃ and M _4.

In the example described here, this detection S2 is carried out using means 111 (for example a DSP) which implement an image processing algorithm detecting in the image Im1 specific objects (for example markers whose shape is predetermined: in Figure 3, a tag). Then, these means 111 determines during a step S3 the 2D coordinates, noted (C'_Mi _; C'_M _2; C'_M _3; C'_M ₄ }, of these first markers M _1s M ₂ , M ₃ and M ₄ in the image marker of the RI camera.

The vectors Mi representative of the 3D coordinates of the first markers M _ls M ₂ , M ₃ and M _{4 are thus known} in the world coordinate system RO, and M'i which are the vectors of the 2D coordinates {C'_Mi _; C'_M _2; C'_M _3; C'_M ₄ } markers M _ls M ₂ , M ₃ and M ₄ in the image frame of the camera RI.

The passage of points from one plane to another plane is associated with a homographic transformation.

We can therefore establish the following relationships:

Mi = H _bc * M'i <→ M'i = H _cb * Mi

with i an integer included in this example between 1 and N (where N is the number of markers, here), and

with H _bc ("board to camera") the homography allowing the passage of the 3D coordinates of the first markers M _ls M ₂ , M ₃ and M ₄ in the world coordinate RO to the 2D coordinates of the markers M _ls M ₂ , M ₃ and M ₄ in the image frame of the camera RI, and H _cb the homography allowing the inverse transformation.

In the example described here, it is therefore the means 112 (for example a calculator) which determine, during a step S4, the homography H _bc .

It will be noted here that this homography H _bc (or H _cb ) is related to the installation of the camera 10.

The method then comprises a calibration phase P2 implemented by a module 120 which makes it possible to estimate the installation of the video projector 20.

According to the invention, the calibration of the system 100 consists in that the projected markers A ₁ , A ₂ , A ₃ and A ₄ by the video projector 20, noted here the second markers, reach an ideal position; they must be confused with third markers, called ideals, I ₂ , 1 ₃ and I ₄ .

To calibrate the system correctly, you need a relation of the type:

A = I or A "/

Here we know the 2D coordinates {C "_Ai _; C" _A _2; C "_A _3; C" _A ₄ } second markers Ai, A ₂ , A ₃ and A ₄ in the image mark of the projector R2. Logically, the video projector 20 already knows the 2D coordinates of these second markers in the picture frame of the projector R2. The vectors of the 2D coordinates {C "_A _1; C "_A _2; C" _A _3; C "_A4} second markers in the image frame of the projector R2.

In the example described here, the means 121 (for example a processor) pre-transform, during a step S5, these second markers Ai, A ₂ , A ₃ and A4 using a pre-transform pt "so as to converge these markers with ideal markers.

This pre-transformed pt '' allows to coincide in the R2 mark the projected markers Ai, A ₂ , A ₃ and A4 with the third so-called ideal markers, noted Ii, I ₂ , 1 ₃ and I ₄ .

We then note Ι 'Ί the vectors of the 2D coordinates of the third markers in the image frame of the projector R2.

We thus have:

I "t = Pt" A "t

Those skilled in the art will understand here that this pre-transformed is also a homographic transformation.

During the first iteration, pt "is initialized to be equal to a non-degenerate homographic matrix, and it can also be initialized as being equal to the identity matrix, which provides a satisfactory result.

In the example described here, the means 122i compare the coordinates of the second markers transformed in the image frame of the projector R2 with the coordinates of the first markers in this same frame.

This comparison S6 makes it possible to filter the second transformed markers which are not projected in the plan; these markers are therefore removed later so as to keep only the projected markers that are relevant for the calibration. This comparison therefore makes it possible to carry out a first filtering.

A second filtering S6 'is then provided by the means 122 ₂ which makes it possible to filter the second markers transformed in the projector image mark which are not entirely inside the image to be projected; these markers are thus removed in order to keep only complete markers and not truncated by the edge of the image.

After the pre-transformation pt ", we know the coordinates of the corners of the second markers in the marker of the projector.

The technical characteristics of the projector also give us the resolution of the projected image. The projected pixels belong to the rectangle [(0,0), (Hres, Vres)] (where Hres and Vres are the horizontal and vertical resolutions of the projector).

It is then possible to exclude any marker whose at least one of the corners does not belong to this rectangle. Then, the means 123 which are able to control the projectors trigger during a step S6 "the projection of said markers Ai, A ₂ , A ₃ and A4 on the flat surface 30.

The method then provides an S7 capture by the camera 10 of an Im2 image which includes the projected markers A ₁ , A ₂ , A3, and A4.

In the example described here, the means 124 then detects in the picture Im2 in a step S8 markers Ai, _A2, A3, A4 and then determine at a step S9 2D {C'_Ai _coordinates; C'_A _2; C'_A _3; C'_A4} of these markers Ai, A ₂ , A3, and A4 in the image frame of the camera RI.

Thus A'i are known vectors representing the coordinates of the markers projected in the image frame of the camera RI.

Also knowing the vectors A "of the 2D coordinates of these second markers in the image marker of the projector R2, the means 125 determine in a step S 10 the homography H _cp , which is related to the installation of the projector 20, and which allows the passage of the 2D coordinates of the second markers projected in the image frame of the RI camera to the 2D coordinates of these markers in the image frame of the projector R2.

We have the following relationship:

A'i = H _cp * A "i ^ A" i = H _pc * A'i

We note here that H _pc is the homography allowing the inverse transformation.

We therefore know the following relations:

Mi = H _bc * M'i <→ M'i = H _cb * Mi

Ai = H _bc * A'i → A'i = H _cb * Ai

A'i = H _cp * A "i ^ A" i = H _pc * A'i

The relationships between the spatial coordinates of the points in the world reference R0 and in the marks (RI, R2) image of the camera and the projector as well as the homographies allowing to go from one marker to the other will now be used in this connection. which follows to estimate the calibration parameters of the camera 10 and the video projector 20. After estimating the homographies Hb _c and H _pc related to the respective poses of the camera 10 and the video projector 20, the method provides a treatment phase P3 with several calculation steps SI 1, S12 and S13 which are characteristic of the invention.

We then use markers, I ₂ , 13 and I ₄ ideals.

The underlying concept here is to match the coordinates of the projected markers with these ideal markers. Given the relationships above, we have here:

I'i = _Hcp * I "i ^ i" i = H _pc * the ili = H _bc * I'i ^ I'i = H _cb * / i with

I'i is a vector representative of the 2D coordinates of at least one of the third markers in the image frame of the IR camera, and

Ii is a vector representative of the 3D coordinates of at least one of the third markers in the world reference R0.

In the example described here, a calculator 131 will calculate in a step SU the 2D coordinates {C 'Ii _; C'_I _2; C 13; C'_I ₄ } third markers Ii, I ₂ , 13 and I ₄ in the image frame of the camera RI according to the homography H _bc .

In the example described here, the calculation S U is performed by a first computer 131 according to the following formula:

I'i = H _cb Ii

Then, the phase P3 comprises a calculation S 12 made by a second computer 132 which consists of calculating another pre-transform pt ', said second pre-transformed, to transform, in the image reference frame of the camera RI, the 2D coordinates { C 'Ai _; C'_A _2; C'_A _3; C'_A ₄ } second projected markers (Ai, A ₂ , A3, A ₄ ) in order to make them coincide with the 2D coordinates {C 'Ii _; C'_I _2; C'_I _3; It's ₄ } third markers.

This calculation S 12 is carried out according to the following formula:

I'i = pt'A'i

in which

- Γ is a vector of 2D coordinates {C 'Ii _; C'_I _2; C 13 _; C'_I ₄ } of at least one of the third markers in the image frame of the IR camera, and

- A 'is a vector of 2D coordinates {C'_A _1; C'_A _2; C'_A _3; C'_A ₄ } at least one of the second markers in the image frame of the camera RI.

With the steps S I 1 and S 12, it is then possible for the computer 133, said third computer, to calculate the first transform pt "according to the following formula:

pt "= Hcp ^ pt'Hcp

Indeed, starting from the relationship: I'i = pt'A'i, I'i _cp = H * I "i and A'i = H _cp * A" i, the processor 133 may deduce:

I'i = pt'A'i → H _cp I "i = pt'H _cp A" U "i = H _cp

Gold,

I "i = pt" A "i So we have :

pt "= H _cp ¹ pt'H _cp

If this pre-transformed equals the identity matrix, then it means that the projected markers coincide with the ideal markers, namely:

I "i = pt" A "i → I" i = Id * A "i, ie I" i = A "i

We also have: i = A'i and Ii = Ai.

In the example described here, the system 100 therefore comprises a processor 140 which is configured to control the various computer means above to repeat all the above steps from the pre-transformation S5 so that the pre-transformed pt "is equal to (or substantially equal to) an identity matrix Id.

This equality is reached when the distance between pt "and Id is less than a predetermined threshold, noted ε.

This convergence of the pre-transform pt "to the identity matrix Id makes it possible to adapt the image to be projected according to the position of the flat surface 30.

Once this equality is reached, the ideal case Ii = Ai is reached.

In this example, the computer module comprises collection means (not shown here) which are configured to collect at each iteration the first markers and the second markers in a list.

In this case, we have the calibration parameters stored and saved by the system 100 which are: {Μί, Μ'ί, Α "ί, Α'ί, Αί}.

This list of points {Mi, M 'i, A "ί, A'i, Ai} is advantageously used for the 3D calibration of the videomapping system by allowing the estimation of the pose of the camera and its optical parameters as well as placing the projector in the world landmark and its optical parameters.

The succession of the steps above thus makes it possible to obtain a list of parameters making it possible to calibrate a video mapping system quickly and efficiently.

With the appropriate parameters, it is possible in only a few iterations to calibrate the system 100, which is very efficient compared to the system proposed in the prior art, in particular AUDET et al.

It should be observed that this detailed description relates to a particular embodiment of the present invention, but in no case does this description take any limiting character to the subject of the invention; on the contrary, its purpose is to remove any imprecision or misinterpretation of the claims that follow.

It should also be observed that the reference signs in parentheses in the following claims are in no way limiting in nature; these signs are only intended to improve the intelligibility and understanding of the claims that follow as well as the scope of the protection sought.

It should also be observed that the number of markers, here limited to 4 {M1, M2, M3, M4}, {A1, A2, A3, A4} and {II, 12, 13, 14} is not limiting.

The algorithm operates from a marker and supports more than four markers per iteration.

For the sole purpose of improving intelligibility and understanding of the claims that follow as well as the scope of the protection sought, one can easily imagine a marker in each corner of the plane, but this number can be adjusted.

Claims

A method of calibrating a video mapping system (100) comprising a camera (10) and a video projector (20) capable of projecting at least one image onto an object (O) positioned in a working space called a world reference ( R0)

wherein said system (100) comprises a planar surface (30) in the form of a plane having at least three corners and being mechanically bonded to said object (O), said planar surface (30) having at least three of its corners a first marker (M _ls M ₂ , M ₃ , M ₄ ) whose spatial coordinates ({C_Mi _; C_M _2; C_M _3; C_M ₄ }), say 3D, in the world reference (R0) are known

said method, implemented by computer means, comprising:

a) a first calibration phase (PI) for estimating the pose and the optical parameters of the camera (10) comprising:

1) following a catch (SI) by said camera (10) a first image (Iml) comprising said first marker (M _ls M _2, M _3, M _4), a first detecting step (S2, S3) during which said first markers (M _ls M ₂ , M ₃ , M ₄ ) are detected (S 2) in said first image (Im ₁ ) and then (2D), using an image processing algorithm, the 2D coordinates are determined (S3) ({C'_M _1; C'_M _2; C'_M _3; C'_M ₄ }) of said first markers (M _ls M ₂ , M ₃ , M ₄ ) in a reference associated with the image captured by said camera (10), called the image marker of the camera (RI),

2) a first determination step (S4) in which a first Hb _c homography is determined, related to the pose and to the optical parameters of the camera (10), allowing the passage of 3D coordinates ({C_M _1; C_M _{2 ;} C_M _3; C_M _4}) of the first marker (M _ls M _2, M _3, M ₄₎ in the world coordinate system (R0) to the 2D coordinates _{({C'_Mi;} C'_M _2; C'_M _3; C 'M ₄ ') of said markers (M _ls M ₂ , M ₃ , M ₄ ) in the image frame of the camera (RI),

b) a second calibration phase (P2) for estimating the pose and optical parameters of the projector (20) comprising:

3) a pre-transformation step (S5) during which a pre-transform pt ", said first transformed, is applied to a plurality of second markers (A ₁ , A ₂ , A ₃ , A ₄ ) whose 2D coordinates ({C "_Ai _; C" _A _2; C "_A _3; C" _A ₄ }) in the mark associated with said projector (20), called the image mark of the projector (R2), are known, said first pre-transformed pt "being initialized during a first iteration as being equal to a non-degenerate homographic matrix,

4) a projection step (S6, S6 ', S6 ") comprising a first filtering (S6) of verifying that said second transformed markers (Ai,

A ₂ , A ₃ , A4) are included in the plane surface (30) by comparing the 2D coordinates ({C "_Ai _; C" _A _2; C "_A _3; C" _A4}) of said second markers transformed in the reference image of the projector (R2) with the coordinates ({C "_Mi _; C" _M _2; C "_M _3; C" _M ₄ }) of the first markers (M _ls M ₂ , M ₃ , M ₄ ), and a projection (S6 ") by said projector (20) said second transformed markers (Ai, A ₂ , A3, A4) on said flat surface (30),

5) following a capture (S7) by said camera (10) of a second image (12) comprising said second projected markers (Ai, A ₂ , A3, A4), a second detection step (S8, S9) at during which said second markers (Ai, A ₂ , A3, A4) are detected (S8) in said second image (12) and then determined

(S9), by an image processing algorithm, the 2D coordinates ({C 'Ai _; C'_A _2; C'_A _3; C'_A4}) of said second markers (Ai, A ₂ , A3, A4) in the image frame of the camera (RI), and

6) a second determination step (S 10) during which a second homography H _cp , related to the pose and to the optical parameters of the projector (20), is determined for the passage of the 2D coordinates ({C'_Ai _; C '_A _2; C'_A _3; C'_A4}) of the second projected markers (Ai, A ₂ , A3, A4) in the image frame of the camera (RI) at 2D coordinates ({C "_Ai _; C" _A _2; C "_A _3; C" _A4}) of said markers in the image marker of the projector (R2),

c) a treatment phase (P3) comprising:

7) a third calculation step (13) during which, according to the first Hb _c and second H _cp homographies, said pre-transformed pt "to transform, in the image reference frame of the projector (R2), the coordinates 2D ({C "_Ai _; C" _A _2; C "_A _3; C" _A4}) of the second projected markers (Ai, A ₂ , A3, A4) in order to make them coincide with the 2D coordinates ({C "_Ii _C "_A _2; I3 C;

C I4}) of said third markers (Ii, I ₂ , 13, 1 ₄ ), said ideals, whose 3D coordinates ({C Ii _; C_I _2; C I3 _; CL,}) in the world coordinate (R0) are predetermined , in which process all of the steps 3) to 7) are repeated, using for each iteration the first pre-transformed pt "calculated during the third calculation step (13) of the previous iteration, until said pre-transformed pt "is substantially equal to an identity matrix,

in which, at each iteration, the coordinates of the first markers (M _ls M ₂ , M ₃ , M ₄ ) are collected in a calibration list in the world and camera markers and second markers (A ₁ , A ₂ , A ₃ , A4) in the world, camera and projector landmarks.

A method according to claim 1, wherein the processing step (P3) comprises a first calculation step (SU) in which the 2D coordinates ({C 'Ii _; C'_I ₂ ; C'_I _3; C I4} ) of the third markers (Ii, I ₂ , I ₃ , I4) in the image frame of the camera (RI) are calculated according to the first homography Hb _c and said coordinates 3D ({C_Ii _; C_I _2; C_I _3; C I4}) and said third markers (Ii, I ₂ , I ₃ , 1 ₄ ) in the world coordinate system.

A method according to claim 2, wherein the 2D coordinates ({C'_Ii _; C'_I _2; C'_I _3; C I4}) of the third markers (Ii, I ₂ , I ₃ , I4) in the image mark of the camera (RI) are calculated according to the following formula:

/ '= H _cb I

in which

Γ is the representative matrix vector of the 2D coordinates ({C'Ji _; C'_I _2; C'_I _3; C I4}) of at least one of the third markers (Ii, I ₂ , 1 ₃ , 1 ₄ ) in the camera image reference (RI), and

I is the vector representative of the 3D coordinates ({C Ii _; C_I _2; C_I _3; C I4}) of at least one of the third markers (Ii, I ₂ , 1 ₃ , 1 ₄ ) in the world reference (R0) .

A method according to claim 2 or 3, wherein the processing step (P3) comprises a second calculation step (S 12) in which another pre-transformed pre-transformed pt 'pre-transform is calculated to transform, in the image frame of the camera (RI), the 2D coordinates ({C'_Ai _; C'_A _2; C'_A _3; C'_A4}) of the second projected markers (Ai, A ₂ , A3, A4) so to make them coincide with the 2D coordinates ({C'_Ii _; C'_I _2; C'_I _3; C'_I ₄ }) of the third markers (Ii, I ₂ , I3, I ₄ ), said second pre-transformed pt 'being calculated according to the following formula:

/ '= pt'A' in which

Γ is the representative vector of the 2D coordinates ({C'Ji _; C'_I _2; C I3; C'_I ₄ }) of at least one of the third markers (Ii, I ₂ ,, h) in the image reference frame of the camera (RI), and

- A 'is the representative vector of 2D coordinates ({C'_A _1; C'_A _2; C'_A _3;

C'_A ₄ }) of at least one of the second markers (Ai, A ₂ , A3, A ₄ ) in the image frame of the camera (RI).

The method of claim 4, wherein the first transform pt "is calculated according to the following formula:

pt "= Hcp ^ pt'Hcp

6. Method according to any one of the preceding claims, wherein, during the first iteration, the 2D coordinates ({C "_A _1; C" _A _2; C "_A _3; C" _A ₄ }) of the second markers. (Ai, A ₂ , A3, A ₄ ) in the image frame of the projector (R2) are determined randomly.

7. A method according to any one of the preceding claims, wherein, during the first iteration, the first pre-transformed pt '' is initialized to be equal to the identity matrix.

The method as claimed in any one of the preceding claims, wherein, during the capture (S1) by said camera (10) of a first image (Im1), the projector (20) projects towards said plane surface ( 30) a black image.

The method of any of the preceding claims, wherein the distance between the first pre-transform pt "and the identity matrix is less than a predetermined threshold.

The method according to any one of the preceding claims, wherein the first markers (M _ls M ₂ , M ₃ , M ₄ ) are previously obtained by printing the flat surface (30) with an invisible ink of the infra-red or ultraviolet ink type.

11. The method of claim 10, wherein the detection (S2) of the first markers (M _ls M ₂ , M ₃ , M ₄ ) in said first image (Im1) comprises a lighting of said surface with an infra-red lamp or ultraviolet.

12. Method according to any one of the preceding claims 1 to 9, wherein the first markers (M _ls M ₂ , M ₃ , M ₄ ) are previously obtained by etching said flat surface (30).

13. The method of claim 12, wherein the detection (S2) of the first markers (M _ls M ₂ , M ₃ , M ₄ ) in said first image (Im1) comprises detecting the reliefs formed by said etching in the flat surface. (30).

14. Method according to any one of the preceding claims, in which, following the first filtering (S6), a second filtering (S6 ') is performed consisting in verifying that the second transformed markers are included in the image to be projected by comparing their 2D coordinates of said second markers with the size of the image in the image frame of the projector (R2).

15. A computer program comprising instructions adapted for performing the steps of the method according to any one of claims 1 to 14 when said computer program is executed by at least one processor.

A computer-readable recording medium on which a computer program is recorded including instructions for performing the steps of the method according to any one of claims 1 to 14.

17. A video mapping system (100) comprising a camera (10) and a video projector (20) capable of projecting at least one image onto an object (O) positioned in a working space called world reference (R0),

wherein said system (100) comprises a plane surface (30) in the form of a plane having at least four corners and being mechanically connected to said object (O), said planar surface (30) having at at least three of its corners a first marker (M _ls M ₂ , M ₃ , M ₄ ) whose spatial coordinates ({C_Mi _; C_M _2; C_M _3; C_M ₄ }), say 3D, in the world reference (R0) are known,

said system (100) comprising:

1) a first calibration computer module (110) for estimating the pose and optical parameters of the camera (10) comprising:

first image processing means (111) configured to detect in a first image (II) captured by said camera (10) said markers (M _ls M ₂ , M ₃ , M ₄ ), and to determine the 2D coordinates ( {C'_Mi _; C'_M _2; C'_M _3; C'_M ₄ }) of said first markers (M _1s M ₂ , M ₃ , M ₄ ) in a coordinate system associated with the image captured by said camera (10 ), called camera image reference (RI),

first determining means (112) configured to determine a first homology Hb _c , related to the pose to the optical parameters of the camera (10), allowing the passage of 3D coordinates ({C_M _1; C_M _2; C_M _3; C_M ₄ }) of the first markers (M _ls M ₂ , M ₃ , M ₄ ) in the world coordinate (R0) to the 2D coordinates ({C'_Mi _; C'_M _2; C'_M _3; C'_M ₄ }) of said markers (Mi, M ₂ , M ₃ , M ₄ ) in the image frame of the camera (RI),

2) a second calibration computer module (120) for estimating the pose and optical parameters of the projector (20) comprising:

pre-transformation means (121) configured to transform with a pre-transform pt "second markers (Ai, A ₂ , A ₃ , A4) whose coordinates 2D ({C" _Ai _; C " _A _2; C "_A _3; C" _A4}) in the reference associated with said projector (20), referred to as image marker of the projector (R2), are known, said pretransformed pt "being initialized as being equal to a non-homographic matrix degenerate,

first filtering means (122i) configured to verify that the second transformed markers (Ai, A ₂ , A3, A4) are included in the planar surface (30) by comparing the 2D coordinates ({C "_Ai _; C" _A _{2 ;} C "_A _3; C" _A4}) of said second markers transformed in the image reference of the projector (R2) with the coordinates ({C "_Mi _; C" _M _2; C "_M _3; C" _M ₄ }) of the first markers (M _ls M ₂ , M ₃ , M ₄ ),

control means (123) of the projector (20) configured to project on said plane surface (30) the second transformed markers (Ai, A ₂ , A ₃ , A4),

second image processing means (124) configured to detect in a second image (12) captured by said camera (10) said second transformed markers (A ₁ , A ₂ , A3, A4), and to determine the 2D coordinates ( {C'_Ai _; C'_A _2; C'_A _3; C'_A4}) of said second transformed markers (Ai, A ₂ , A3, A4) in the image frame of the camera (RI), and

second determining means (125) configured to determine a second homography H _cp , related to the pose and to the optical parameters of the projector (20), for the passage of 2D coordinates ({C'_A _1; C'_A _2; C '_A _3; C'_A4}) of the second markers (Ai, A _2, A3, A4) in the image coordinate of the camera (RI) to the 2D coordinates ({C "_AI _C" _A _{2 C} "_A _{3 C} "_A4}) of said markers in the projector image reference frame (R2),

c) a computer processing module (130) comprising:

a computer (133) configured to recalculate, according to the first Hb _c and second H _cp homographies, the first pretransformed pt '' to transform, in the image frame of the projector (R2), the 2D coordinates ({C "_Ai _; C "_A _2; C" _A _3; C "_A4}) of the second projected markers (Ai, A ₂ , A3, A4) so as to coincide with the 2D coordinates ({C" _Ii _; C "_A _2; C" _I _{3 C} "_I _4}) of third markers (Ii, _I2, 13, 14), the 3D coordinates ({C _Ii; C_I _2; C _I3; CL,}) in the world coordinate system (R0) are predetermined , computer collection means configured to collect in a calibration list the coordinates of the first markers (Mi, M ₂ , M ₃ , M ₄ ) in the world and camera markers and second markers (A ₁ , A ₂ , A3, A4 ) in the world, camera and projector landmarks,

said system further comprising a processor (140) configured to control all of said computer means of said system so that said calculator (133) calculates the first pre-transformed pt "until it is substantially equal to an identity matrix.

18. System according to claim 17, comprising computer means configured for implementing the steps of the method according to any one of claims 2 to 14.