WO2010146289A2

WO2010146289A2 - System and method for locating a photographic image

Info

Publication number: WO2010146289A2
Application number: PCT/FR2010/051169
Authority: WO
Inventors: Livio De Luca; Matthieu Deveau
Original assignee: Centre National De La Recherche Scientifique - Cnrs -
Priority date: 2009-06-16
Filing date: 2010-06-11
Publication date: 2010-12-23
Also published as: WO2010146289A3; FR2946765A1

Abstract

The invention relates to a method and system for locating (2) a photographic image (22) for modeling a three-dimensional scene. The system (2) comprises: a referenced device (4) including a first imager (10) and targets (12₁, 12₂), the first imager (10) being capable of imaging a scene (20) for generating the photographic image to be located (22); a referencing device (6) including a second imager (26) capable of imaging a target (12₁) for generating a reference tag, said second imager (26) being capable of imaging the referenced device (4) for generating a photographic referencing image (30); and a computing unit (9) capable of angularly locating the photographic image to be located (22) from the reference tag and the photographic reference image (30).

Description

System and method for photographic image localization

The present invention relates to a system and method for locating a photographic image for modeling a 3-dimensional scene, as well as an associated tagging device.

Modeling from photographs consists of taking several shots from different angles of a building or scene using a camera, calculating the positions and orientations of the points of view of the different shots , to point out the various structuring elements of this building on the various shots, and to reconstruct a three-dimensional model of this building or this scene in a virtual space, from the positions pointed on the various shots.

In this process, the score comes in two distinct stages: - the calculation of the positions and orientations of the points of view of the different shots; and

- the seizure of structural elements of the building.

The pointing is done by a technician who manually selects pixels from shots using a pointer of a computer. This score requires several hours of work.

The object of the invention is in particular to reduce or eliminate the long and tedious step of pointing related to the calculation of the positions and orientations of the pictures.

For this purpose, the subject of the invention is a photographic image-locating system for modeling a 3-dimensional scene, the system comprising:

a marked device comprising a first imager and at least two targets distinct from each other; the first imager being able to image a scene to generate the photographic image to be located;

a locating device comprising a second imager able to image at least one target positioned at a known distance to generate a reference vignette, said second imager being able to image the device identified for generating a photographic registration image, said shooting of the identified device being simultaneous with the shooting of the scene; and

a calculation unit capable of angularly locating the photographic image to be located in relation to a reference frame from the reference vignette and from the photographic marker image.

According to particular embodiments, the location system comprises one or more of the following characteristics:

a measurement unit capable of measuring the position of the device identified with respect to the reference frame, said position measurement being simultaneous with the shooting of the scene; and wherein the computing unit is adapted to position the photographic image to be located from the measured position; said measuring unit comprising

a transmitter / receiver able to emit a beam towards a reflector and to receive the beam reflected by the reflector; and a reflector adapted to reflect the beam from the transmitter / receiver; the transmitter / receiver belonging to the tracking device, the reflector belonging to the device identified; the position of the device identified with respect to a reference corresponding to the position of the reflector relative to a reference frame;

- targets of different colors.

The invention also relates to a device identified for a photographic image locating system described above, said identified device comprising a first imager and at least two targets distinct from each other; the first imager being able to image a scene to generate the photographic image to locate.

The invention also relates to a method for locating at least one photographic image for modeling a 3-dimensional scene from a photographic image locating system; said locating system comprising a marked device comprising a first imager and at least two targets distinct from each other, and a locating device comprising a second imager and a calculating unit, characterized in that the method comprises the steps following: obtaining at least part of a photographic image, hereinafter referred to as a reference vignette, representing at least one target positioned at a known distance;

- shooting a scene by the first imager to generate the photographic image to locate;

- Shooting of the device identified by the second imager to generate a registration photographic image, said shooting of the identified device being simultaneous with the shooting of the scene;

angular location of the photographic image to be located relative to a reference frame from the reference vignette and the photographic marker image.

According to particular embodiments, the method comprises one or more of the following characteristics:

- from the location system described above further comprising a measurement unit comprising a transmitter / receiver integrated in the tracking device and a reflector attached to the device identified; the method further comprises the steps of:

measuring the distance and angles between the reflector and the second imager by the measurement unit, said measurement of distance and angles being simultaneous with the shooting of the scene; and positioning the photographic image to be located from said distance and angle measurement and the registration photographic image;

the step of obtaining the reference vignette is performed by shooting the target positioned at a distance known by the second imager;

the method further comprises the following steps: sampling of the reference sticker as a function of the known distance and the distance between the reflector and the second imager; and

determining the position of the targets in the registration photographic image from the sampled reference vignette;

from the location system described above further comprising a memory storing distances and angles between each target and the reflector, the method further comprises a step of determining the possible positions of each target in a virtual space, said possible positions being the points of intersection between a virtual sphere and a virtual line, said virtual sphere being centered on the position of the reflector, said virtual sphere having a radius equal to the distance between each target and the reflector; said virtual straight line passing through a perspective point of the registration photographic image and by the position of the centroid of each target in the registration photographic image; the method further comprises a step of determining the positions of the targets in the virtual space by similarity between, on the one hand, the possible positions of the targets and, on the other hand, the distances and the angles between each target and the target. reflector;

from the location system described above, in which the memory stores the distances and the angles between the reflector and the first imager, the method comprises the following steps:

calculating a rotation matrix representative of the rotation between the reflector and the first imager and a translational matrix representative of the translation between the reflector and the first imager; and

establishing a matrix representative of the orientation of the first imager relative to the reference frame, and of the matrix representative of the first imager position relative to the reference frame, from the positions of the targets, of the rotation matrix representative of the rotation. between the reflector and the first imager, and the translational matrix representative of the translation between the reflector and the first imager; and angular location of the photographic image to be located with respect to the reference frame from the matrix representative of the orientation of the device identified with respect to the reference frame, and of the matrix representative of the position of the device identified with respect to the reference frame.

Finally, the subject of the invention is also a computer program, characterized in that it comprises instructions for implementing the method for locating at least one photographic image for modeling a 3-dimensional scene from a system photographic image localization, when these instructions are executed by a processor.

The invention will be better understood on reading the description which follows, given solely by way of example and with reference to the drawings, in which: Figure 1 is a schematic view of the locating system according to the invention; FIG. 2 is a diagram of the steps of the method according to the invention; and Figures 3 and 4 are diagrams showing certain steps of the method according to the invention.

With reference to FIG. 1, the locating system 2 comprises a device marked 4 and a tracking device 6.

According to the embodiment illustrated in FIG. 1, the device marked 4 comprises a support 8, a camera hereinafter called first imager 10 and six targets 12 _! with i = 1, ... 6 carried by the support 8.

The support 8 is for example formed by a ring 16 on the outer periphery of which the targets 12 _! are fixed. The ring 16 is further provided with a bar 18 carrying the first imager 10.

In a variant, the support 8 has a different shape.

The first imager 10 is able to photograph a building or a scene

20 to generate a photographic image to be located 22.

The six H ₁ targets are spherical in shape. Each target has a different color.

Alternatively, the targets have different shapes.

Alternatively, the device 4 identified has only two targets distinct from each other.

The tracking device 6 comprises a second camera, hereinafter referred to as the second imager 26, capable of photographing the device marked 4 at the moment when the first imager 10 takes the scene 20 for photography. generating a photographic image hereinafter called a registration photographic image 30.

The locating system 2 further comprises a measurement unit 7 and a calculation unit 9.

The measurement unit 7 is able to measure the position of the device identified with respect to a fixed reference. It comprises a transmitter / receiver 24 integrated in the marking device 6 and a reflector 14 fixed on the inner periphery of the ring 16 of the device identified.

The transmitter / receiver 24 is able to emit a laser beam towards the reflector 14 and to receive the beam reflected by the reflector 14.

The transmitter / receiver 24 and reflector 14 is for example constituted by a rangefinder. In this case, the reflector 14 is a prism, and the rangefinder is provided with a prism optical tracking system.

The transmitter / receiver assembly 24, reflector 14 and second imager 26 is preferably constituted by a total station. In this case, the positions measured by the transmitter / receiver of the total station are corrected to define positions with respect to the center of view of the second imager 26. In the remainder of the description, it will be considered that the transmitter / transmitter assembly receiver 24, reflector 14 and second imager 26 is constituted by a total station and that the positions measured by the transmitter / receiver are corrected to define positions with respect to the center of view of the second imager 26.

In a variant, the transmitter / receiver 24 is able to emit an ultrasound beam.

A reference frame 25 is conventionally chosen at the center of view of the second imager 26. A reference frame 27 is conventionally selected at the center of the reflector 14. A reference frame 28 is conventionally chosen at the center of view of the first imager 10. For the purposes of the invention, the rotation matrices R, R 'and Rg, and the translation matrices T, T' and Tg are established.

The rotation matrix R is representative of the rotation between the reference 27 of the reflector 14 and the reference 28 of the first imager 10. The rotation matrix R 'is representative of the rotation between the reference 25 of the second imager 26 and the reference 27 of the Reflector 14. Finally, the matrix Rg is representative of the rotation between the reference frame 25 of the second imager 26 and the reference frame 28 of the first imager 10 or, in other words, the orientation of the first imager 10, with respect to the reference frame 25. In practice , Rg = R. R '.

The translation matrix T is representative of the translation between the reference 27 of the reflector 14 and the reference 28 of the first imager 10. The translation matrix T 'is representative of the translation between the reference frame 25 of the second imager 26 and the reference 27 of the Reflector 14. The translation matrix Tg is representative of the translation between the reference frame 25 of the second imager and the reference frame 28 of the first imager 10 or, in other words, the position of the first imager 10, with respect to the reference frame 25. In practice, Tg = T + T '.

The calculation unit 9 comprises a memory 32 able to store a computer program and initial data, and a processor 34 able to execute this computer program from the initial data and data provided by the measurement unit. 7, the first imager 10 and the second imager 26. For this purpose, the computing unit 9, the first imager 10 and the second imager 26 comprise unrepresented communication means.

The initial data comprise the distances and the angles between the targets Yl _x and the reflector 14, the distances and the angles between the reflector 14 and the first imager 10, as well as a reference vignette, as explained hereinafter. Referring to Figure 2, the locating method according to the invention begins with a step 36 during which a target is taken in photography at a known distance do. This target is called the first target \ 2 _\ below.

In a step 38, a portion of the photographic image photographed in step 36 is extracted and stored in the memory 32. This image portion represents the first target 1 _\ . This portion of the image is shown schematically in FIG. 3 and will be called hereinafter reference vignette 39.

During a step 40, the distances and the angles between the reflector 14, and more precisely the reference 27 of the reflector, and the first imager 10, or more precisely the reference 28 of the first imager, are measured and recorded in the memory 32. The processor 34 calculates the rotation matrix R and the corresponding translation matrix T and stores them in the memory 32.

During a step 42, the distances and the angles between the targets Yl _x and the reflector 14 are measured and recorded in the memory 32.

The distances and the angles between the targets Yl ₁ and the reflector 14 define the geometric model of the device identified.

Steps 36 to 42 can be done in the laboratory.

During a step 44, the first imager 10 takes a photograph of the scene 20 or the building to be photographed, and generates a photographic image to be located 22.

Simultaneously, the second imager 26 takes a photograph of the device marked 4 to generate a registration photographic image 30 and the measurement unit 7 measures the distance to the identified device 4 and, in particular, to the reflector 14 and the angles aiming the laser beam in the reference frame 25 of the second imager. step

44 is performed on site. With reference to FIG. 3, the processor 34 generates during a step 46, a first 45, a second 47 and a third 49 working vignettes by sampling of the reference vignette 39 with three different resampling factors. For this purpose, the processor 34 uses the distance between the second imager 26 and the reflector 14 measured during step 44.

The first resampling factor fi is given by the formula below:

In which: - D is the distance between the second imager 26 and the reflector 14.

- C is the distance between the first target 12 ₁ and the second imager 26 when shooting in step 36.

When the first resampling factor fi is applied to the reference vignette 39, the latter is resized so that the first target 12 ₁ has an apparent dimension equivalent to the dimension it would have if it had been photographed at a distance. D.

The second resampling factor f ₂ is given by the formula below:

(D ₊ R)

Λ = ^" d _n where R is the distance between the reflector 14 and the target Yl ₁ farthest from the reflector 14.

When the second resampling factor f ₂ is applied to the reference vignette 39, the reference vignette 39 is resized so that the first target 12i has an apparent size equivalent to the dimension it would have if it had been photographed at a distance equal to D + R.

The third resampling factor f ₃ is given by the formula below: (D - R)

Z ₃ = d _n

When the third resampling factor f ₃ is applied to the reference vignette 39, the latter is resized so that the first target 12 ₁ has an apparent dimension equivalent to the dimension it would have if it had been photographed at a distance DR.

In a step 48, a set of correlation scores is computed between the registration photographic image 30 and the first job thumbnail 45. For this purpose, the registration photographic image 30 is partitioned into a plurality of portions. image 51 of the same size as the first work vignette 45.

A correlation score is for example calculated according to the formula mentioned below:

In which :

- r is the correlation score,

- I ⁴⁵ a vector representative of the ordered values of gray _n x n pixels ⁴⁵ of the working vignette, - I ⁵¹ is a vector representative of ordered gray values x _n ^5ι of the image portion 51,

V (/ ⁵ ) is the variance of the gray values of the vector I ⁴⁵ ,

V (/ ^{5 /} ) is the variance of the gray values of the vector I ⁵¹ ,

- Cov (/ ⁵ , 1 ⁵¹ ) is the covariance of / ⁵ and I ⁵¹ ,

In which :

- n is the total number of pixels in the work vignette Λ ^ ⁵ is the average of the gray values of the pixels x _n ^A5 , M ⁵¹ is the average of the gray values of the pixels XJ ⁵ .

Correlation scores are also calculated between the registering photographic image 30 and the second 47 and third 49 job thumbnails to obtain three sets of correlation scores. Each set of correlation scores corresponds to the correlation scores calculated with a different work vignette. These three sets of correlation scores may for example be represented by three images 53, 55, 57 in three dimensions as illustrated in FIG.

During a step 50, the processor 34 searches among these three sets of correlation scores, the highest value of the correlation score, as well as the position (x ₁₅ V ₁ ) in one of the images 53, 55, 57 , corresponding to this value

During a step 52, a contour detecting process is performed in a subset of the photographic image tracking 30 centered on the position (x _l5 V ₁₎ determined in step 50.

During a step 54, a circle is estimated on the contour detected during step 52, and the position (x, y) of the center of this circle is determined. This position (x, y,) is comparable to the position of the center of the detected target. This target is any one of the six targets of the support 8 and will be referenced hereinafter target 12.

In a variant, when the target 12 is not spherical, a shape corresponding to the shape of the target is estimated on the detected contour and the position of the center of gravity of this shape is determined.

During a step 56, the target 12 of the support 8 corresponding to the target detected during steps 46 to 54 is recognized with respect to its color.

Steps 46 to 56 make it possible to determine the positions of the targets in the registration photographic image 30 from the reference sticker 39. During a step 58, the position of the reflector 14 is sought in a virtual space created by the processor 34, from the distance and angles measured by the measurement unit 7 during the step 44.

A virtual sphere 59 illustrated in FIG. 4 is generated around the position of the reflector 14. The radius of the virtual sphere 59 is equal to the distance between the reflector 14 and the determined target 12. This distance was recorded in the memory 32 during the step 42.

During a step 60, a virtual straight line 61 passing through the aiming center 63 of the second imager, also called perspective point of the photographic locating image 30, and by the position (x, y) of the center of the target 12 ,, is defined. The points of intersection between the virtual line 59 and the virtual sphere 61 define two positional hypotheses of the target 12, in the virtual space.

During a step 62, the correlation score having the second highest value is searched and steps 52 to 60 are repeated for this second correlation score.

Steps 52 to 62 are repeated a number of times corresponding to the number of targets on the device 4 identified.

When these steps have been repeated a times with a equal to the number of targets, during a step 64, the real positions of the targets are chosen from among all the hypotheses of position of targets by similarity with the geometric model of the device spotted. 4 stored in memory 32.

During a step 66, a rotation matrix R 'and a translation matrix T' are established between the reference frame 25 of the second imager and the reference frame 27 of the reflector by means of the position of the targets determined during the process. step 64. During a step 68, the matrix Rg representative of the orientation of the first imager 10 and the matrix Tg representative of the position of the first imager 10 relative to the reference frame 25, are determined from the matrices R and T calculated during of step 40, and matrices R 'and T' established during step 66, and by applying the formulas below:

Rg = R. R 'Tg = T + T'

During a step 70, the photographic image to be located 22 is located in the virtual space by implementing the matrices Rg and Tg in 3D modeling software of the type:

- "NUBES" developed by Mr. Livio De Luca, from the GAMSAU laboratory, and available at the following Internet address: http://www.map.archi.fr/nubes/introduction/projet.htm;

- "MicMac" developed by Mr. Marc Pierrot-Deseilligny of IGN and Yves Egels, ENSG and available at the following internet address: http://www.micmac.ign.fr/;

- ImageModeler from Realviz - Autodesk;

- "Photomodeler" from Eos System Inc.; and

- "Socet Set" by BAE Systems.

In a variant, the photographic image to be located 22 is located in the virtual space using the matrix Rg and the matrix Tg. The 3D modeling of the scene is carried out by pointing the structuring elements onto the photographic image. to locate, by automatic detection of structuring elements on the photographic image 22 and / or by the exploitation of any other morphological information on the surface to be modeled (one or more positioned and oriented images), a scatter point cloud obtained by scanning laser, an already existing 3D model) .. The rotation matrix Rg and the translation matrix Tg must be calculated for each new position of the device identified.

In a variant, the photographic images to be located are localized with respect to a cloud of points representative of the scene to be modeled obtained by laser scanning scanning.

In a variant, the device identified is mounted on a drone or hot-air balloon to take aerial photographs.

In a variant, the images to be located are localized with respect to a 3D model of the environment.

Claims

1. A photographic image locating system (22) for modeling a scene in three dimensions, characterized in that the system (2) comprises: - a marked device (4) comprising a first imager (10) and at least two targets (12i, 12 ₂ ) distinct from one another; the first imager (10) being able to image a scene (20) to generate the photographic image to be located (22);

a locating device (6) comprising a second imager (26) able to image at least one target (12i) positioned at a known distance (do) to generate a reference vignette (39), said second imager (26) being capable of imaging the device identified (4) to generate a registration photographic image (30), said shooting of the identified device (4) being simultaneous with the shooting of the scene (20); and

a calculation unit (9) capable of angularly locating the photographic image to be located (22) with respect to a reference frame (25) from the reference vignette (39) and the registration photographic image (30) .

2. Location system (2) according to claim 1, which comprises a measuring unit (7) capable of measuring the position of the device (4) identified with respect to the reference system.

(25), said position measurement being simultaneous with the shooting of the scene (20); and wherein the computing unit (9) is adapted to position the photographic image to be located (22) from the measured position.

3.- locating system (2) according to claim 2, wherein the measuring unit (7) comprises: - a transmitter / receiver (24) adapted to emit a beam to a reflector (14) and to receive the beam reflected by the reflector (14); and a reflector (14) adapted to reflect the beam from the transmitter / receiver (24); the transmitter / receiver (24) belonging to the tracking device (6), the reflector (14) belonging to the identified device (4); the position of the device marked (4) with respect to a reference frame (25) corresponding to the position of the reflector (14) relative to a reference frame (25).

4. Location system (2) according to any one of claims 1 to 3, wherein the targets (12 _ls 12 ₂ ) are of different colors.

5.- Spotted device (4) for a photographic image locating system (2) according to any one of claims 1 to 4, characterized in that the device identified (4) comprises a first imager (10). ) and at least two targets

(12i, 12 ₂ ) distinct from each other; the first imager (10) being able to image a scene (20) to generate the photographic image to be located (22).

A method of locating at least one photographic image (22) to model a 3-dimensional scene from a photographic image locating system (2); said locating system (2) comprising a marked device (4) comprising a first imager (10) and at least two targets (12 _ls 12 ₂ ) distinct from each other, and a locating device (6) comprising a second imager (26) and a computing unit (9), characterized in that the method comprises the following steps:

- obtaining (36) at least a portion of a photographic image, hereinafter referred to as a reference vignette (39), representing at least one target (12i) positioned at a known distance (do);

- Shooting (44) a scene (20) by the first imager (10) to generate the photographic image to be located (22);

- Shooting (44) of the device identified (4) by the second imager (26) to generate a registration photographic image (30), said shooting of the identified device (4) being simultaneous with the shooting of the stage (20);

angular localization (70) of the photographic image to be located (22) relative to a reference frame (25) from the reference vignette (39) and the registration photographic image (30).

7.- locating method according to claim 6, from the locating system (2) comprising a measurement unit (7) comprising a transmitter / receiver (24) integrated in the tracking device (6) and a reflector (14). attached to the identified device (4); and wherein the method further comprises the steps of: measuring (44) the distance and angles between the reflector (14) and the second imager (26) by the measuring unit (7), said distance and angle measurement being simultaneous with the shooting of the scene (20); and

- positioning (70) the photographic image to be located (22) from said distance and angle measurement, and the registration photographic image (30).

8. A locating method according to any one of claims 6 and 7, wherein the step of obtaining (36) the reference vignette (39) is performed by shooting (36) of the target (12i ) positioned at a known distance (do) by the second imager (26).

9. A locating method according to any one of claims 7 and 8, which further comprises the following steps:

- sampling (46) of the reference tile (39) according to the known distance (do) and the distance between the reflector (14) and the second imager (26); and

- determining (48, 50, 52, 54, 56) the position of the targets (12 _ls 12 ₂ ) in the registering photographic image (30) from the sampled reference vignette (39).

10. A locating method according to claim 9, from the location system (2) which further comprises a memory (32) storing distances and angles between each target (12 _ls 12 ₂ ) and the reflector (14). , the method comprising a step of determining (58, 60, 62) possible positions of each target (12 ^ 12 ₂ ) in a virtual space, said possible positions being the points of intersection between a virtual sphere (59) and a virtual right (61), said virtual sphere (59) being centered on the position of the reflector (14), said virtual sphere (59) having a radius equal to the distance between each target (12 ^ 12 ₂ ) and the reflector (14) ); said virtual straight line (61) passing through a perspective point (63) of the registration photographic image (22) and by the position of the centroid of each target (12 ^ 12 ₂ ) in the registration photographic image (30).

11. A locating method according to claim 10, wherein the method further comprises a step of determining (64) the positions of the targets (12 _ls 12 ₂ ) in the virtual space by similarity between, on the one hand, the possible positions of the targets (12i, 12 ₂ ) and, secondly, the distances and angles between each target (12 ^ 12 ₂ ) and the reflector (14).

12. A locating method according to claim 11, from the location system (2) wherein the memory (32) stores the distances and angles between the reflector (14) and the first imager (10), wherein the method comprises the following steps:

calculation (40) of a rotation matrix (R) representative of the rotation between the reflector (14) and the first imager (10) and of a translational matrix (T) representative of the translation between the reflector (14); ) and the first imager (10); and

setting (66, 68) a matrix (Rg) representative of the orientation of the first imager (10) relative to the reference frame (25), and a matrix (Tg) representative of the first imager position (10) relative to the reference frame (25), from the positions of the targets (12 _ls 12 ₂ ), the rotation matrix (R) representative of the rotation between the reflector (14) and the first imager (10), and the translation matrix (T) representative of the translation between the reflector (14) and the first imager (10); and

angular localization (70) of the photographic image to be located (22) relative to the reference frame (25) from the matrix (Rg) representative of the orientation of the device identified (4) with respect to the reference frame (25), and the matrix (Tg) representative of the position of the device identified (4) with respect to the reference system (25).

13. Computer program, characterized in that it comprises instructions for implementing the method according to any one of claims 6 to 12, when they are executed by a processor.