WO2013160489A1

WO2013160489A1 - Method and system for generating and applying three-dimensional reconstructions

Info

Publication number: WO2013160489A1
Application number: PCT/ES2012/000114
Authority: WO
Inventors: Jeser Zalba Olcoz; Mikel Arizaleta Arteaga; Amalia ORTIZ NICOLAS
Original assignee: Visiona Control Insdustrial S.L.
Priority date: 2012-04-27
Filing date: 2012-04-27
Publication date: 2013-10-31
Also published as: WO2013160489A8

Abstract

The invention relates to a method and system for generating and applying three-dimensional reconstructions. The method comprises the real-time generation of a reference three-dimensional reconstruction of a surface (S) to be reconstructed that may move, using artificial vision techniques, and (a) comparing the reference three-dimensional reconstruction with the position of different points of the surface (S) to be reconstructed and (b) moving the surface (S) to be reconstructed on the basis of the discrepancies obtained as a result of the comparison, in order to position said surface according to the reference three-dimensional reconstruction. The system has been adapted to carry out the method of the invention.

Description

Generation method and system v reconstruction application

three-dimensional

Technical sector

The present invention concerns, in general and in a first aspect, a method of generation and application of three-dimensional reconstructions based on the use of artificial vision techniques, and more particularly a method comprising displacing an object based on positional differences found with respect to a three-dimensional reference reconstruction thereof.

A second aspect of the invention concerns a three-dimensional reconstruction application system adapted to implement the method according to the first aspect of the invention.

The invention is particularly applicable to the prior positioning of a patient in order to obtain great precision in an area to be treated by radiotherapy.

Prior art

Various proposals are known for carrying out three-dimensional reconstructions of various kinds, of different objects, and for various purposes of application, using artificial vision techniques.

US7620209B2 uses a method for the dimensional measurement of the surface contour of an object. The method uses the controlled projection of a series of patterns on the object and the corresponding acquisition, by scanning with a digital camera, of images corresponding to each pattern projected on the object to determine the topographic characteristics of the object, and is applied to objects subject to variations of movement, such as those related to human respiration, to complex bending of mechanical or civil structures, or to those obtained in processes of testing deformation forces in an airplane, vehicle, beam, bridge or other structure.

The objective of the method is to obtain a three-dimensional map of the surface of the object in digital format, by means of the corresponding processing of the scanned images.

In US7620209B2 it is not indicated or suggested to apply the proposed method to objects subjected to movement for any purpose other than the one indicated above related to obtaining a three-dimensional map, and in particular it is not proposed to use the generated three-dimensional information to compare it with the object position and act on it, moving it, nor obtaining said three-dimensional map in real time.

US7844079B2 also proposes to use a 3D reconstruction method through the projection and subsequent capture and processing of projected patterns on the object. In order to overcome the inconveniences caused by high projection and capture speed (decrease in the signal-to-noise level ratio, synchronization between projection and critical feedback, etc.), a new technique is proposed that systematically combines multiple patterns of structured light in a single pattern that can thus be continuously projected without causing the aforementioned problems, minimizing the delay between pattern projections.

In US7844079B2 it is commented that thanks to the technique used, a high speed of 3D reconstruction is achieved, in real time, which allows its application on moving objects, but it is also not indicated in US7844079B2 to apply the proposed method to objects subjected to movement in order of acting on them, moving them, to position them properly.

WO2010036403A2 proposes a system that uses a structured light projector, which projects sinusoidal patterns, and through the phase they obtain 3D coordinates, for the reconstruction of the object. It is indicated that the minimum of patterns to be captured to perform the reconstruction is of three patterns, as in the proposal. In this case the system is applied to moving objects (such as a person's face or finger in the case of biometric applications), so that the acquisition of the patterns on the object is divided into a series of sub images - Windows whose processing allows 3D reconstruction of the moving object.

Although WO2010036403A2 proposes the use of mechanisms for guiding the movement of the object, these are used to limit the movement of the object to an area covered by the projection / capture system, so that its 3D reconstruction is possible, not proposed in said International request a positioning of the moving object acting on it, moving it, after having obtained 3D reconstruction of it.

US7653226B2 proposes a system and method for the flexible generation of digitally reconstructed radiography (DRR) images in a graphic processing unit (GPU) from three-dimensional information obtained, for example, by computerized tomography (CT), computerized tomography megavoltage (MVCT), 3D scanning of high-contrast objects using C-arms, etc., involving all these techniques the emission of radiation on the patient, to unlike artificial vision techniques, so it is not advisable to use them too much or too often on the same patient.

Although in US7653226B2 it is proposed to use the DRRs for the positioning of patients to receive radiotherapy treatment, as well as to show the information included in them in a graphical way using augmented reality techniques, it is not proposed to use three-dimensional information directly for these purposes obtained, nor get the latter in real time.

The positioning of patients that can be carried out by means of the system and method of US7653226B2 is mandatory punctual, that is to say for a moment of time, since if the patient moved, a tomography of the same should be done again and the subsequent obtaining of the DRRs from them, which makes this system unfeasible to make a positional follow-up of the patient and a corresponding positional correction of the same during a radiotherapy treatment session.

Explanation of the invention.

It seems necessary to offer an alternative to the state of the art that covers the gaps found therein, in particular those related to the absence of proposals applied to the correct positioning of an object that suffer from the known systems and methods of generating three-dimensional reconstructions by Artificial vision techniques of objects that can move.

For this, the present invention concerns, in a first aspect, a method of generation and application of three-dimensional reconstructions comprising generating at least a three-dimensional real-time reconstruction of a surface to be reconstructed using artificial vision techniques, where said surface to be reconstructed It is susceptible to movement.

Unlike the known methods, in the one proposed by the first aspect of the invention, characteristically, said three-dimensional reconstruction is a three-dimensional reference reconstruction and the method further comprises:

a) comparing said three-dimensional reference reconstruction with the position of different points of the surface to be reconstructed; Y

b) move the surface to be reconstructed based on the discrepancies obtained as a result of said comparison, to position it according to the three-dimensional reference reconstruction. Step b) of the method comprises, in general, displacing the surface to be reconstructed to position it adjusted in correspondence with the three-dimensional reference reconstruction.

According to an exemplary embodiment, said three-dimensional reference reconstruction is a previously registered three-dimensional reconstruction for the surface to be reconstructed.

For another embodiment, said three-dimensional reference reconstruction is a current three-dimensional reconstruction, also in real time, of the surface to be reconstructed.

For some embodiments, the method comprises calculating automatically, using the aforementioned discrepancies, some displacement values to be applied in step b) and applying them manually or automatically, realizing both said comparison of stage a) in real time. as the calculation of said displacement values.

According to some embodiments, the method comprises using an augmented reality technique to show the three-dimensional reference reconstruction superimposed directly on the surface to be reconstructed or on a live image of the surface to be reconstructed.

The use of said augmented reality technique comprises projecting the three-dimensional reference reconstruction by superimposing it directly on the surface to be reconstructed or displayed on display means, such as a screen, superimposed on the live image of the surface to be reconstructed.

The method comprises, according to a preferred embodiment, carrying out the comparison of step a) between the three-dimensional reference reconstruction and the surface to be reconstructed, observing the discrepancies in said superposition of the three-dimensional reference reconstruction on the surface itself. reconstruct or on said live image of it.

According to an example of embodiment, the method comprises performing a current three-dimensional reconstruction also in real time, of the surface to be reconstructed.

The method comprises showing said three-dimensional reference reconstruction and said current three-dimensional reconstruction on display means, independently or superimposed on each other, depending on the embodiment.

For an exemplary embodiment, the method comprises, in a complementary or alternative way to the comparison between the three-dimensional reconstruction of reference and the surface to rebuild itself, carry out the comparison of stage a) between the three-dimensional reference reconstruction and the current three-dimensional reconstruction, the points that form the latter being representative of the position of different points of the surface to be reconstructed .

Current three-dimensional reconstruction must be understood as a reconstruction that has been obtained immediately before performing said stage a), or at least not so far apart in time that the shape or dimensions of the surface to be reconstructed has changed with respect to the moment in which perform stages a) and b).

According to an exemplary embodiment, the method comprises determining the position and orientation of an operator, anamortically transforming said reference three-dimensional reconstruction and / or said current three-dimensional reconstruction taking into account said position and orientation determined, at each moment, so that the three-dimensional reconstruction corresponds to the point of view of the operator, and show the operator said three-dimensional reconstruction of reference and / or said current three-dimensional reconstruction transformed anamorphically, by projection or on display means.

For another example of embodiment, the method comprises monitoring the position of the surface to be reconstructed during a given period and at least notifying if there is a mismatch with respect to the three-dimensional reference reconstruction.

In order to carry out said monitoring of the position of the surface to be reconstructed, the method comprises, according to an exemplary embodiment, performing a plurality of current three-dimensional reconstructions, in real time, over said predetermined period, and performing the comparison of step a) between the three-dimensional reference reconstruction and each of the current three-dimensional reconstructions.

According to one embodiment, the method comprises obtaining the three-dimensional reference reconstruction and / or the current three-dimensional reconstruction or reconstructions by performing the following steps:

i) project a series of sinusoidal patterns on the surface to be reconstructed using structured light;

ii) capture, by at least one camera of known relative position, images corresponding to each pattern projected on the surface to be reconstructed; iii) calculate the three-dimensional coordinates of each projected point from the captured images and the relative position of the camera; Y iv) perform the three-dimensional reconstruction of the surface to be reconstructed using said captured images and the information regarding the calculated three-dimensional coordinates of each projected point.

Although the method proposed by the invention has various applications, in various fields, including those relating to positional correction of mechanical or other parts, for a preferred embodiment the method is applied to the medical field, the surface being reconstruct at least part of a patient that includes an area to be treated, including the three-dimensional reconstruction of reference to said area to be treated, and the method being applied to the positioning, in step b), of at least the area to be treated in correspondence with the three-dimensional reference reconstruction thereof, the method comprising, in step b), displacing the surface to be reconstructed by moving a support or a treatment table on which the patient is disposed.

According to a variant of said embodiment, the method of the first aspect of the invention comprises carrying out step b) for the positioning of at least said area to be treated in correspondence with the three-dimensional reference reconstruction thereof prior to the localized treatment of it by radiotherapy.

For an exemplary embodiment, the method comprises displaying on the surface to be reconstructed or on a live image of the surface to be reconstructed, using an augmented reality technique, virtual graphic information that complements the real information of the object, in a complementary or alternative way to the superposition of three-dimensional reconstruction.

The application of the method proposed by the first aspect of the invention, as well as the system of the second aspect that will be described below, to the positioning of the patient prior to the radiotherapeutic treatment, ensures the correct positioning of the patient and allows, for some of the examples described above, monitor their position throughout the entire treatment session.

In the patient positioning phase, before starting the radiotherapy session, the method will calculate accurately and automatically the displacements that should be applied to the treatment table to align the position of the area to be radiated with the position defined in the treatment.

On the other hand, in the initial phase of patient positioning, the method will allow visualizing the 3D reconstruction of the patient superimposed on the position defined in the treatment. In this way, the radiotherapy technician can see with his own eyes and naturally if there are variations and how to correct them. You will also have a tool that will allow you to detect possible physical variations of the patient throughout the treatment sessions and make it necessary to review the planning of said treatment. At present there is no method that warns of this need.

Once the patient is correctly positioned and the radiotherapy session is started, the method of the invention will allow the patient's position to be monitored throughout the duration of the treatment, warning of possible movements that make it necessary to interrupt the session and reposition the patient.

In this way, the method and system of the invention will ensure greater precision and control of the patient's daily positioning and monitoring during treatment, which will allow the reduction of the configuration margin, which in turn, associated with other techniques such as treatments with modulated intensity (IMRT) or image-guided radiation therapy (IGRT), will allow a dose increase in tumor volumes without exceeding the tolerances of risk organs, with the benefit for patients of a greater probability of tumor control keeping the probability of complications in healthy tissue acceptable.

A second aspect of the invention concerns a three-dimensional reconstruction generation and application system, comprising artificial vision means in connection with an electronic system arranged and configured to generate one or more three-dimensional real-time reconstructions of a surface to be reconstructed using techniques. of artificial vision.

Unlike the known systems, the one proposed by the second aspect of the invention, in a characteristic way, is adapted to implement the method of the first aspect, the electronic system being configured to perform the three-dimensional reference reconstruction, in collaboration with the means of artificial vision, to determine the position of different points of the surface to be reconstructed, to make the comparison of stage a) to obtain the said discrepancies, and to at least notify the discrepancies in order to at least collaborate in the displacement of the surface to be reconstructed from step b), whether it is carried out automatically, supplying the control signals of corresponding actuation means, or manually, this collaboration consisting in the notification to an operator of said operator discrepancies or the calculation and notification of the offset values to apply.

According to an embodiment, the electronic system is configured to also perform the current three-dimensional reconstruction of the surface to be reconstructed, in collaboration with the means of artificial vision, comparisons between the three-dimensional reference reconstruction and the current three-dimensional reconstruction or the surface to be reconstructed, the calculation of the displacement values, the superposition of the three-dimensional reference reconstruction on the current three-dimensional reconstruction or on the surface to be reconstructed (controlling suitable means for this purpose, for example formed by a projector), the determination of the position and orientation of an operator and the corresponding anamorphic transformation of the three-dimensional reference reconstruction and / or the current three-dimensional reconstruction, and its visualization, according to the different described embodiments of the method of the first aspect of the invention.

For an exemplary embodiment, the system comprises, in connection with the electronic system, a projector, for carrying out the projection of the three-dimensional reference reconstruction by superimposing it directly on the surface to be reconstructed, as described above for the corresponding example of realization of the method of the first aspect.

In order to implement steps i) to iv) of the above described example of embodiment of the method of the first aspect of the invention, the system of the second aspect of the invention comprises, in connection and under the control of the electronic system:

- a high speed structured light projector to perform stage i) for patterns supplied by the electronic system, and

- to the aforementioned artificial vision means formed by at least one high speed acquisition camera synchronized with the structured light projector, to perform stage ii) and supply the acquired images to the electronic system, which is responsible for performing the stages iii) and iv).

According to another embodiment, the system comprises detection means for, in collaboration with the electronic system to which they are connected, to determine the position and orientation of the operator.

The detection means comprise, according to an implementation of said embodiment example:

- reflective markers arranged on the operator;

- an infrared lighting system that emits at least infrared light on said reflective markers; Y

- a stereoscopic system consisting of two cameras calibrated with each other, arranged to receive the infrared light reflected by the reflective markers, and connected to the electronic system to supply the captured images so that This determines the position and orientation of the operator, at all times, based on the images received, and performs the aforementioned anamorphic transformation.

The above-described examples of embodiment of the method and system proposed by the present invention, related to the use of augmented reality techniques and all the variants associated therewith, including in particular the one related to the anamorphic transformation of three-dimensional reference reconstruction and / or of the reconstruction or current three-dimensional reconstructions, although they have been described and claimed as dependent aspects of the described positional correction, they can be subject to independent protection by means of a method or system that shows increased information by superimposing on the object or image of the same information virtual, with three-dimensional preference, relative to a three-dimensional or other reconstruction, that complements the real information of the object, and that adapts, through anamorphic transformation, the view of the virtual graphic information according to the position and orientation of a user Aryan.

Brief description of the drawings

The foregoing and other advantages and features will be more fully understood from the following detailed description of some embodiments with reference to the attached drawings, which should be taken by way of illustration and not limitation, in which:

Fig. 1 schematically shows a scenario of application of the method and system proposed by the invention, where a large part of the elements included in the system of the second aspect of the invention are illustrated, in particular those used to reconstruct three-dimensionally apart from the patient illustrated and those used to project the three-dimensional reconstruction on the patient; Y

Fig. 2 shows the same scenario of Fig. 1, from another point of view, illustrating other elements of the system of the second aspect, in particular those that allow determining the position and orientation of the illustrated operator, in this case a radiotherapy technician

Detailed description of some embodiments

The main features of the system proposed by the second aspect of the invention applied to the positioning of a patient prior to radiotherapy treatment, for a preferred embodiment, are the following:

· Positioning of the area to be treated with a precision of tenths of a millimeter. • Monitoring of the patient's position in real time, obtaining a measurement every 50 milliseconds, that is, at a speed of 20-30 reconstructions per second.

• The system is not invasive at all, since it does not use tattoos or markers of any kind, nor does it emit additional ionizing radiation.

• Calculates in real time and automatically the displacements that must be applied to the treatment table to optimize the patient's position.

• It allows to visualize at all times the current reconstruction of the patient together with the one registered in the initial planning session.

For this purpose, three modules have been designed both at hardware and software level and complementary to each other, with the aim of reconstructing patient P in three dimensions, processing that information to calculate and monitor deviations from a reference position and visualize the results in 3D in real time by the radiotherapy technician O.

3D reconstruction:

A diagram of the components that form the system proposed by the invention is shown in Fig. 1. The system responsible for reconstructing a scene in 3D in real time is based on the projection of a series of structured light patterns and their capture by one or several cameras, which in the illustrated embodiment are the two cameras indicated with the numerals 2 and 3. For this, one of the keys of the system resides in the light source used, which in Fig. 1 is a structured light projector 1, is capable of projecting patterns at very high speed, above the 200 images / second, and that is also perfectly synchronized with cameras 2, 3.

Also illustrated in Fig. 1 is the electronic system SE to which the projector 1 and the cameras 2, 3 are connected, and which is responsible for controlling them, in collaboration with them, to perform the reconstruction or three-dimensional reconstructions, to determine the position of different points on the surface to be reconstructed S, which in this case corresponds to part of the trunk of a patient P.

Projection System:

Commercial projectors are capable of projecting images at speeds, in the best case, of 50-60 images / second, which makes their use in the 3D real-time reconstruction system unfeasible. On the other hand, there is another great inconvenience, since said commercial projectors do not allow the possibility of generating a digital signal to be able to synchronize the projection of a pattern with the capture of the camera.

The light source used for the development of the 3D reconstruction system is based on the DLP (Digital Light Processing) technology of Texas Instruments and meets the following specifications:

• Project a series of patterns previously stored in memory.

• Projection speeds greater than 200 images / second.

• Possibility of adjusting projection times.

· Digital inputs / outputs to allow communication with PCs and cameras.

• Possibility of incorporating different lenses to vary the area and projection distance.

• Possibility of coupling different types of light (visible, infrared, etc.). The core of a projector based on DLP technology is its DMD (Digital

Micromirror Device) which is a set of microscopic mirrors, distributed in a rectangular matrix and corresponding to the pixels of the image to be projected. Each of these mirrors can be controlled electronically so that it is capable of rotating a certain angle, thus allowing the light that falls on it to be projected outward or not.

The proposed system uses an LED light source, although it can be used with any other type of light emitter, and allows to project both black and white and grayscale patterns with any bit depth. The bit depth of the projected patterns will determine the maximum projection speed, as shown in the following table:

Pattern Type # Bits # Different Gray Levels Speed (patterns / s)

Black and white 1 2 22,727

Gray scale 4 16 3,497

Grayscale 7 128 569

Grayscale 8 256 291 For the projection of grayscale patterns of N bits, the image to be projected into N images in black and white corresponding to the channel of each bit is decomposed, so that different time intervals are projected, that is, the channel of the LSB (Least Significant Bit) will be projected for a time 2 ^N_1 times lower than the time corresponding to the MSB (Most Significant Bit) channel. In this way, the total projection time T _p is calculated using equation 1:

T _p = ∑ t _P = t _LSB ^■ ∑? = ÍLSB · (2 - l) (1)

Similarly, from (1), the projection time of each channel t _p can be calculated based on the total projection time of the pattern from equation (2):

. . 2'

t _p = ÍLSB · 2 ^? = T _p · _N (2)

L

The projection system 1 has a digital input and a digital output, so that it can synchronize the projection of each pattern with other devices, in particular with cameras 2 and 3.

Image Capture:

Cameras 2, 3 have no special characteristics, they are simply both an industrial model capable of capturing at a relatively high speed of 90 images / second. It is very important to have the entire system calibrated, that is, to know the exact relative position of the cameras 2, 3 with respect to the projector 1, both in position and orientation.

Reconstruction Method:

In order to obtain the 3D coordinates of the scene or object, the system proposed by the invention is based, for the example of embodiment explained here, on the "Phase Shifting Interferometry" algorithms, also called "Phase Measuring Profilometry". These algorithms use methods based on the projection of several equal sinusoidal patterns but displaced in one axis with respect to each other, and are implemented in the electronic system SE illustrated in Fig. 1.

It is necessary to project a minimum of three sinusoidal patterns, 120 ° out of phase each one, to be able to extract information from the phase of each pixel in the image, from which its 3D coordinate can be calculated. From the images of the three projected patterns l _1t l ₂ on ₃ , captured by the camera we proceed to calculate the phase of each pixel, which is ideally:

To optimize the processing and minimize the time needed to generate the 3D reconstruction of the scene, the following approach is used to avoid calculating the tangent arc: φ = - · í 2 · round í——) + (-1) · —— 1 (4)

Where l _mm , ax the _mec¡ are the minimum, maximum and median values for each pixel, and K is an integer constant that varies between 0 and 5.

From any of the expressions (3) or (4) the value of the module phase 2π is obtained, so in general, provided that the projected patterns contain multiple periods of the sinusoidal signal, a next step is necessary to unwind phase and calculate the absolute value of the phase for each pixel. For this, the simplest and most immediate process is to sweep along the image integrating the calculated phase values, taking into account that in each discontinuity of 2π, multiples of 2π must be added or subtracted.

In the calculation of the absolute phase $ _, BS mechanisms are also incorporated to compensate both the non-linearity of the cameras2, 3 when capturing and that of the projector 1 when projecting sinusoidal patterns. Once the absolute phase _A BS of each pixel (x¡ _ms , and _img ) of the image is obtained, the coordinates (x _pmy , y _pm y) corresponding to the pixel of the projector that has projected the phase of each pixel of the image are calculated image, so it is necessary to have the camera-projector system perfectly calibrated. To perform said system calibration, it is assumed that the projector 1 can be modeled mathematically as if it were a camera, which "captures" the projected image. Therefore the process used to calibrate the system is analogous to the calibration of a two-chamber stereoscopic system, obtaining both the intrinsic parameters of the camera and the projector, as well as their relative position, both in distances and orientation.

Then, from the correspondences between the points in the camera image and the points on the projector, it is possible to calculate the corresponding triangulation i » ^■ '-

14

and thus obtain the 3D coordinate of each point, as is done in a two-chamber stereoscopic system.

Different reconstruction methods can be used that differ mainly in the number and type of projected patterns. The fastest of all those that have been practically implemented for the system of the second aspect of the invention, needs to project and capture three images, so that through its use a reconstruction in 20 milliseconds is obtained, since we also have all the processes of Capture and process perfectly parallel.

3D Information Processing

Once the 3D reconstruction of the Pse patient is obtained, 3D mesh alignment algorithms are used to compare the current reconstruction with the reference 3D model. In this way you can calculate both the deviation in translation and rotation or even obtain a map of deviations of each point with respect to the reference. All this also carried out by the SE electronic system, which implements the aforementioned algorithms through a corresponding software.

On the one hand, these results allow calculating in real time and automatically the displacements that should be applied to the treatment table 7 to optimize the position of the patient P, achieving a positioning of the area to be treated with a precision of tenths of a millimeter .

On the other hand, the position of patient P can also be monitored in real time, obtaining a measurement every 50 milliseconds, so as to ensure that patient P has not moved beyond a previously defined safety threshold. Display:

The extracted information can be displayed both on a monitor and directly on the patient P.

In the monitor of the control PC (not illustrated) of the electronic system SE, both the real-time reconstruction of the patient P, as well as the reference position, and both overlays can be displayed. Additionally, the deviations of each point with respect to the reference can be shown in the form of a color map, as well as the values of displacements and rotations that should be applied to the treatment table 7 to correct the position of the patient P.

In order to facilitate the task of the radiotherapy technician O, it is possible to project the results on the patient P himself, so that the technician O can observe the reference position superimposed on the real patient 7. In this way, you can naturally see if there is an area that deviates from the original position and manually reposition the patient P.

The visualization of the reference position is carried out using augmented reality techniques, that is, virtual objects are mixed with real objects, so that the information of the real object that is displayed is "increased". For this, the system is based on having the user's point of view, in this case the radiotherapy technician O, perfectly located in the space.

As illustrated in Fig. 2, a stereoscopic system is used to calculate the position of the radiotherapy technician O, in connection and under the control of the electronic system SE, which is composed of two chambers 5, calibrated together, and an infrared lighting system 8, where the cameras 5 capture the image of a minimum of three spherical reflective markers 6 placed on the radiotherapy technician O. From the images captured by each camera 5 the electronic system SE calculates the position and orientation of the person in question O.

The image is then generated from that point of view of the reconstructed 3D information and an anamorphic perspective transformation is performed. The image obtained is projected by the projector 4, shown in Fig. 1, and gives a feeling that the 3D object is totally real and is perfectly integrated with the real patient P, provided it is viewed from the point of view for the which has been calculated, which, as is the point of view calculated for the radiotherapy technician O, will allow you to correctly visualize the real patient 7 and the virtual reference position superimposed.

According to the radiotherapy technician O it moves, its position will be recalculated, and consequently this view will be regenerated, so you will have the sensation of seeing two real objects, one the real patient 7 and the other the virtual patient correctly positioned .

The projection of the results on the patient P can be done with the same projector that is used to reconstruct 1 or an auxiliary projector 4 can be used that does not require any special synchronization, the latter case being the one illustrated in Fig. one.

A person skilled in the art could introduce changes and modifications in the described embodiments without departing from the scope of the invention as defined in the appended claims.

Claims

1. - Method of generation and application of three-dimensional reconstructions, of the type that comprises generating at least one real-time three-dimensional reconstruction of a surface to be reconstructed (S) using artificial vision techniques, where said surface to be reconstructed (S) is susceptible to move, the method being characterized in that said three-dimensional reconstruction is a three-dimensional reference reconstruction and also includes:

a) comparing said three-dimensional reference reconstruction with the position of different points of the surface to be reconstructed (S); Y

b) displacing said surface to be reconstructed (S) based on the discrepancies obtained as a result of said comparison, to position it according to said three-dimensional reference reconstruction.

2. - Method according to claim 1, characterized in that said reference three-dimensional reconstruction is a previously registered three-dimensional reconstruction for the surface to be reconstructed (S) or a current three-dimensional reconstruction, also in real time, of the surface to be reconstructed (S).

3. - Method according to claim 1 or 2, characterized in that it comprises calculating automatically, using said discrepancies, displacement values to be applied in said step b) and applying them manually or automatically, realizing both said comparison of step a) as the calculation of said offset values.

4. - Method according to any one of the preceding claims, characterized in that it comprises using an augmented reality technique to show said three-dimensional reference reconstruction superimposed directly on said surface to be reconstructed (S) or on a live image of the surface to be reconstructed ( S).

5. - Method according to claim 4, characterized in that the use of said augmented reality technique comprises projecting the three-dimensional reconstruction of reference by superimposing it directly on the surface to be reconstructed (S) or displaying it on display means superimposed on said live image of the surface to rebuild (S).

6. - Method according to claim 4 or 5, characterized in that it comprises carrying out said comparison of step a) between the three-dimensional reference reconstruction and the surface to be reconstructed (S), observing the discrepancies in said superposition of the three-dimensional reference reconstruction on the surface to be reconstructed (S) or on said live image thereof.

7. - Method according to any one of the preceding claims, characterized in that it comprises performing a current three-dimensional reconstruction, also in real time, of the surface to be reconstructed (S).

8. - Method according to claim 7, characterized in that it comprises showing said three-dimensional reference reconstruction and said current three-dimensional reconstruction on display means, independently or superimposed on each other.

9. Method according to claim 8, characterized in that it comprises carrying out at least part of said comparison of said stage a) between the three-dimensional reference reconstruction and the current three-dimensional reconstruction, the points forming the latter representing the position from different points of the surface to rebuild (S).

10. Method according to claim 8 or 9, characterized in that it comprises determining the position and orientation of an operator (O), anamortically transforming said three-dimensional reference reconstruction and / or said current three-dimensional reconstruction taking into account said determined position and orientation, in each moment, so that the three-dimensional reconstruction corresponds to the point of view of the operator (O), and show the operator (O) said three-dimensional reconstruction of reference and / or said current three-dimensional reconstruction transformed anamorphically, by projection or on display means .

11. - Method according to any one of the preceding claims, characterized in that said step b) comprises displacing said surface to be reconstructed (S) to position it adjusted in correspondence with the three-dimensional reference reconstruction.

12. - Method according to any one of the preceding claims, characterized in that it comprises monitoring the position of the surface to be reconstructed (S) during a certain period and at least notifying if there is a mismatch with respect to the three-dimensional reference reconstruction.

13. - Method according to claim 12 when it depends on the 8, characterized in that it comprises carrying out said monitoring of the position of the surface to be reconstructed (S) by performing a plurality of current three-dimensional reconstructions, in real time, at over said predetermined period, and making said comparison of step a) between the three-dimensional reference reconstruction and each of the current three-dimensional reconstructions.

14. Method according to any one of the preceding claims, characterized in that it comprises obtaining said three-dimensional reference reconstruction and / or said current three-dimensional reconstruction by performing the following steps:

i) project a structured series of sinusoidal patterns on said surface to be reconstructed (S);

ii) capture, by at least one camera of known relative position, images corresponding to each pattern projected on said surface to be reconstructed (S);

iii) calculate the three-dimensional coordinates of each projected point from said captured images and the relative position of said camera; Y

iv) perform said three-dimensional reconstruction of the surface to be reconstructed (S) using said captured images and the information regarding the calculated three-dimensional coordinates of each projected point.

15. Method according to any one of the preceding claims, characterized in that said surface to be reconstructed (S) is at least part of a patient (P) that includes an area to be treated, because said three-dimensional reference reconstruction includes at least said area to be treat, because it is applied to the positioning, in said stage b), of at least said area to be treated in correspondence with the three-dimensional reference reconstruction thereof, and because it comprises, in stage b), move said surface to be reconstructed (S ) by displacing at least one support or a treatment table (7) on which said patient (P) is arranged.

16. - Method according to claim 15, characterized in that it comprises carrying out said step b) for the positioning of at least said area to be treated in correspondence with the three-dimensional reference reconstruction thereof prior to the localized treatment thereof by radiotherapy.

17. - Method according to any one of the preceding claims, characterized in that it comprises showing on said surface to be reconstructed (S) or on a live image of the surface to be reconstructed (S), using an augmented reality technique, virtual graphic information that complement the real information of the object.

18. - System for the generation and application of three-dimensional reconstructions, of the type comprising artificial vision means in connection with an electronic system (SE) arranged and configured to generate at least a three-dimensional real-time reconstruction of a surface to be reconstructed (S) using techniques of artificial vision, the system being characterized in that it is adapted to implement the method according to any one of the preceding claims, the electronic system (SE) being configured to perform said three-dimensional reference reconstruction, in collaboration with the artificial vision means, to determine the position of different points of the surface to be reconstructed (S), to make said comparison of step a) to obtain said discrepancies, and to at least notify said discrepancies in order to at least collaborate in the displacement of the surface to be reconstructed ( S) of stage b).

19. - System according to claim 18, characterized in that said electronic system (SE) is configured to also perform said current three-dimensional reconstruction of the surface to be reconstructed (S), in collaboration with the artificial vision means, said comparisons between the three-dimensional reconstruction reference and the current three-dimensional reconstruction or the surface to be reconstructed (S), said calculation of said displacement values, said superposition of the three-dimensional reference reconstruction on the current three-dimensional reconstruction or on the surface to be reconstructed (S), said determination of the position and orientation of an operator (O) and the corresponding anamorphic transformation of the three-dimensional reference reconstruction and / or of the current three-dimensional reconstruction, and its visualization, according to the method of any one of claims 1 to 17.

20. - System according to claim 19, characterized in that it comprises, in connection with the electronic system (SE), a projector (4) for carrying out said projection of the three-dimensional reference reconstruction by superimposing it directly on the surface to be reconstructed (S) , according to the method of claim 5.

21. - System according to claim 18 or 19, characterized in that it comprises, to implement the method according to claim 14, in connection and under the control of the electronic system (SE):

- a high speed structured light projector (1) to perform said stage i) for patterns supplied by the electronic system (SE), and

- to said artificial vision means formed by at least one high speed acquisition camera (2, 3) synchronized with said structured light projector (1) to perform said stage ii) and supply the acquired images to the electronic system (SE) , which is responsible for performing these stages iii) and iv).

22. - System according to claim 19, characterized in that it comprises detection means for, in collaboration with said electronic system (SE) to which they are connected, to determine the position and orientation of said operator (O).

23. - System according to claim 22, characterized in that said detection means comprise:

- reflective markers (6) arranged on said operator (O);

- an infrared lighting system (8) that emits infrared light at least on said reflective markers (6); Y

- a stereoscopic system consisting of two cameras (5) calibrated with each other, arranged to receive the infrared light reflected by the reflective markers (6), and connected to the electronic system (SE) to supply the captured images so that it determines said position and operator orientation (O), at any time, depending on the images received, and perform the aforementioned anamorphic transformation.