WO2022242863A1

WO2022242863A1 - Computer-supported method for a moving display of an image of a real surrounding area and at least two virtual objects on a screen

Info

Publication number: WO2022242863A1
Application number: PCT/EP2021/063500
Authority: WO
Inventors: Ralf Schumacher; Florian Coigny; Daniel XANDER
Original assignee: Medartis Holding Ag
Priority date: 2021-05-20
Filing date: 2021-05-20
Publication date: 2022-11-24

Abstract

The invention relates to a computer-supported method for a moving display of a real image of a surrounding area and at least two virtual objects (F, F') on a screen (10), having the steps of a) capturing a real image (17) of a surrounding area, b) displaying the surrounding area, in particular at least one real object (16), and the virtual objects (F, F') together on the screen (10) in a three-dimensional manner, c) selecting at least one of the virtual objects (F') in response to the activation of a gripping function, d) determining a rotational and/or translational movement of a manipulation device (11), and e) moving the image (17) of the surrounding area, in particular the real object (16), and the unselected virtual objects (11) relative to the selected virtual object (F') in a rotational and/or translational manner on the basis of the rotational and/or translational movement of the manipulation device (11) determined in step d) as long as the gripping function is activated. The invention can be used to plan a surgery on a fractured bone containing multiple fragments for example.

Description

Computer-aided method for the moving display of an image of a real environment and at least two virtual objects on a screen

The present invention deals with computer-aided methods for the moving display of an image of a real environment and at least two virtual objects on a screen. In particular, the method can be used to plan an operation on an anatomical structure, in particular to plan an operation on a fractured bone containing multiple fragments.

In order to plan an implant for a bone fracture, it is necessary to virtually recreate the bone situation, which is referred to as deposition. A generic method is known from WO 2019/036790 A1 and WO 2019/161477 A1. It discloses surgical systems that make use of "virtual reality" and "augmented reality".

The system contains "Virtual Reality" glasses and two hand-held controllers. The glasses can display virtual hands of the user, virtual tools and virtual bones. Virtual fragments of a bone can be moved independently of each other. Nevertheless, this movement Not very intuitive. In addition, glasses of this type are expensive, not least because of the high graphics performance required, which stands in the way of their spread. Furthermore, they limit mobility due to their weight and size. They also ensure visual isolation from the real environment, which can occasionally even make the user feel nauseous.

Other well-known "Augmented Reality" methods use gesture control. Although this can be operated quite intuitively, it is limited by the lack of spatial reference. It is therefore an object of the present invention to provide a computer-aided method that allows virtual objects to be moved on a screen in a simpler and more intuitive manner. This should be achieved in particular in the field of surgery, especially bone surgery, but also in other technical fields - for example in hostile environments such as nuclear power plants, in space, under water or in the case of very small or very large (component) parts which robot technology is used.

These and other tasks are solved by the computer-aided method according to the invention for the moving display of an image of a real environment and at least two virtual objects on a screen. The method includes the steps a) capturing an image of a real environment b) joint three-dimensional display of the image of the real environment, in particular the at least one real object, and at least one of the virtual objects on the screen, c) selection of at least one of the virtual objects in Response to the activation of a gripping function, d) determining a rotational and/or translational movement of a manipulation device, e) rotational and/or translational movement of a selected virtual object relative to images of the environment, in particular the real object, and not from selected virtual objects, depending on the rotational and / or translational determined in step d) see movement of the manipulation device as long as the gripping function is activated.

This method enables a particularly simple and intuitive rotary and/or translatory movement of the virtual objects relative to one another. For this purpose, only the rotational and/or translational movement of the manipulation device has to be determined. Depending on this, the virtual object is also moved on the screen.

The real environment contains at least one real object. The image of the real environment, in particular of the at least one real object, can be captured in step a) using an imaging device, for example using a camera. The real object in the environment is recorded and can be displayed in relation to the environment. In principle, the real object can be any reference in the environment.

The virtual objects can be read from an imaging device or from a memory, for example.

The three-dimensional display in step b) can contain the superimposition of a live image from a camera on the one hand and at least two virtual objects on the other. The representation of another virtual or a real object, for example an anatomical structure such as a bone, preferably overlays the originally selected position of the real object. If the manipulation device moves and the perspective of the image changes as a result, the position of the overlay in the image and the perspective of the 3D representation are updated in such a way that the viewer has the impression that the virtual objects displayed are fixed to the selected one position and in the selected orientation in the environment. This is referred to as "anchored" in the context of the invention no translation or rotation of other virtual or real objects relative to the environment takes place.

In a preceding method step, initially only a single virtual object can be displayed on the screen. In a further preceding method step, this single virtual object can be divided into at least two virtual objects, which can then be selected separately in step c).

In one possible variant, the anchoring can be rigid. This means that the real movement of the manipulation device is converted into the movement of the virtual objects as simultaneously as possible and on a scale of 1:1. However, it is also possible for translational movements to be implemented on a different scale, in which case the paths actually covered by the manipulation device are either increased or decreased. A reduction in size can be particularly useful if small virtual objects are to be moved, as is often necessary, for example, in surgery, in particular in bone surgery. As an alternative or in addition to this, the conversion of the real movement into the virtual movement can be filtered. For example, a tremor reduction known per se from surgical robotics can be carried out, in which small movements of the manipulation device are filtered out in a specific frequency range. It is also conceivable and is within the scope of the invention that not all six degrees of freedom of the movement of the manipulation device are converted into the movement of the objects on the screen. For example, only translations (in one, two or three spatial directions) or only rotations (about one, two or three axes) can be implemented.

The manipulation device is preferably a mobile device that can contain the said screen and will be explained further below. For example, it can be a tablet or a smartphone. However, it is also conceivable and is within the scope of the invention for the manipulation device whose movement is determined to be a separate controller. Furthermore, the camera can also be part of the manipulation device or be separate from it.

In a possible variant, the rotational and/or translational movement of the manipulation device in step d) can be determined on the basis of the images recorded by the camera. For this purpose, the movement of a real environment, in particular of a real object, in the image recorded by the camera can be determined with image processing methods that are known per se from "augmented reality" and converted into the movement of the virtual objects. Also conceivable and however, the invention also covers the fact that the rotational and/or translational movement is determined with the aid of acceleration sensors that are known per se.

It is particularly advantageous if the screen on which the objects are displayed is part of the manipulation device. In this way, the effect of the movement of the manipulation device can be read directly from it.

It is also particularly advantageous if a virtual selection mark is displayed on the image screen, in particular a screen of the manipulation device. When the gripping function is activated, that virtual object is selected to which the selection mark on the screen is directed. It is expedient for this if the virtual selection mark is displayed in a central area of the screen. This also simplifies operation. With further advantage, the screen can be a touchscreen. In this case, the gripping function can be activated when a user touches the touch screen. This touch does not necessarily have to be at the point of the selection mark. In a further alternative, the gripping function can be activated by pressing a physical button.

Also advantageously, the gripping function can remain activated as long as the user is touching the touch screen or holding down the physical button, and can be deactivated when the user stops touching the touch screen or is releasing the button.

It is even more preferred if the selected virtual object is left in that position and that orientation in relation to the other virtual objects on the screen, in particular the touchscreen, that it was in at the time the gripping function was deactivated. This creates a kind of drag-and-drop functionality. In addition, functions known per se, such as “undo” and/or “repeat” can be implemented.

The objects can be displayed on the screen in step b) and the rotary and/or translatory movement of the manipulation device can be determined in step d) using methods known per se from the prior art.

The method according to the invention can be used in particular for planning an operation on an anatomical structure, in particular on a fractured bone containing at least two fragments or also on another body tissue. In this application, images of the fragments are the virtual objects. This special method according to the invention therefore contains the following steps: a) capturing an image of a real environment, in particular at least one real object, b) joint three-dimensional display of the image of the real environment, in particular the at least one real object, and at least one virtual partial structure of the anatomical structure, in particular one of the virtual fragments of the bone, on a screen, c) selection of at least one of the displayed virtual partial structures, in particular at least one of the virtual fragments, in response to activation of a gripping function, d) determination of a rotational and/or translational movement of a manipulation device, e) rotational and/or translational Moving a selected virtual substructure relative to images of the environment, in particular the real object, and the unselected virtual substructures, in particular the unselected virtual fragments, depending on the rotational and/or translational movement of the manipulation device as long as the gripping function is activated.

On the basis of the knowledge gained in this method according to the invention, preparatory planning for an operation can be undertaken in a subsequent step f). For example, a suitable cutting plane can be defined in which a bone or bone fragment can be cut. In addition, virtual measurements can also be taken on the virtual bone fragments. In a further step g), at least one operation on a real bone and real bone fragments can be performed. This can, for example, be the application of a bone plate as well as the Bone screw placement and alignment included. However, even without these steps f) and g), the insights gained in steps a) to e) and possibly f) can be valuable. For example, the method with steps a) to e) and optionally f) and/or g) can also be used for training purposes.

In addition to the virtual fragments, a virtual implant can also be displayed on the screen. The distinction between virtual fragments and virtual implant can be made possible by visual parameters, for example by different colors and/or surface properties. In some situations it can prove to be advantageous if the (virtual) freedom of movement of the implant or part of it is restricted; for example, a bone screw can only be displaced within a long hole in a bone plate.

It goes without saying that the invention is not limited to such methods for planning an operation and that the virtual objects do not necessarily have to be fragments of a bone or other body tissue. For example, the method according to the invention can also be used in mechanical engineering. In this case, the virtual objects can be individual parts or entire assemblies of a device, for example, which can be moved relative to other individual parts or assemblies of the device.

In a further aspect, the invention also includes a computer program product which executes a method according to the invention when it runs on a device with a screen and a sensor of the device. The device is preferably a mobile device and the screen can be a touch screen. If this computer program product is executed, the advantages explained above result. The invention is explained in more detail using an exemplary embodiment from bone surgery. show it

Figure 1: an image of a real environment containing a real object in the form of a table and a tablet aligned with the table with a touch screen on which the table is displayed,

FIG. 2: the tablet according to FIG. 1, with several virtual fragments of a bone also being displayed on the screen,

Figure 3: the selection of one of the virtual fragments using a selection mark,

Figure 4: the positioning of the selected virtual

Fragments in a different position and orientation assumed by shifting and rotating the tablet;

FIG. 5: the tablet according to FIG. 1, with several virtual fragments of a bone and a virtual implant also being displayed on the screen;

FIG. 6: the selection of the virtual implant by means of a selection mark;

FIG. 7: the positioning of the virtual implant in a different position and orientation assumed by moving and rotating the tablet.

FIG. 1 shows a mobile device in the form of a tablet 14. The tablet 14 contains a touchscreen 10 on its front and a camera on its rear (not visible here).

The tablet 14 is positioned in FIG. 1 in such a way that an environment with a table 16, ie a real object, is mera is detected. The image 17 of the area with the table 16 taken by the camera is displayed on the touchscreen 14 .

The tablet 14 also forms a manipulation device 11 with which its own rotational and translational movements can be determined. For this purpose, the tablet 14 contains a known arithmetic unit, also not shown. This runs an algorithm (ie a computer program product) from which the position and orientation of the tablet 14 relative to the real table 16 are determined using an image processing method known per se. A virtual selection mark in the form of a crosshair 12 is also shown in a central area of the screen 10, the function of which will be explained in more detail below.

Alternatively, the computing unit can also be designed as a unit that is separate from the tablet 14 . The processing unit can receive and send back image data or pre-processed data from the tablet 14, for example with a cable or wirelessly.

In FIG. 2, virtual fragments F, F′ of a bone are additionally shown on the touchscreen 10—in this example, a human lower jaw. If the crosshairs 12 point to one of the virtual fragments F, this is represented visually, for example by the virtual fragment F being colored red. To select one of the virtual fragments F, the tablet 14 is oriented so that the crosshairs 12 are above the virtual Fragment F comes to rest as shown in FIG. When the touch screen 10 is then touched, a gripping function is activated.

If the tablet 14 is now moved in a rotational and/or translational manner, this movement can be determined by the processing unit of the tablet 14 as explained above. Dependent on this movement also moves the selected virtual fragment F in relation to the table 16 and the remaining fragments F' on the touchscreen 10, as visualized in FIG. It is conceivable, for example, that only translations in a particular horizontal plane and/or rotations about a defined axis are converted into movements on the touch screen 10 . The software can also enable individual degrees of freedom to be deactivated via an input interface depending on the situation.

As long as the gripping function is activated (ie as long as the touchscreen 10 remains touched), the selected virtual fragment F is anchored on the touchscreen 10. This means that with a further rotational and/or translational movement of the tablet 14, only the image of the table 16 and the unselected fragments F' on the touchscreen 10 are moved accordingly, but not the selected fragment F as well.

When the user stops touching the touch screen 10, the gripping function is disabled. The previously selected virtual fragment F is then left in the position and orientation relative to the table 16 and the remaining fragments F' on the touch screen 10 that it was in at the time the gripping function was deactivated. This is shown in FIG.

Compared to FIGS. 2 to 4, a virtual implant I is additionally shown on the touchscreen 10 in FIGS. If the crosshairs 12 point to the virtual implant I, this is represented optically, for example by the virtual implant I being colored red. In order to select the virtual implant I, the tablet 14 is oriented in such a way that the crosshairs 12 lie over it comes as shown in Figure 6. If the touch screen 10 then touches is activated, a gripping function is activated. If the tablet 14 is now moved in rotation and/or translation, this movement can be determined by the processing unit of the tablet 14 as explained above. Depending on this movement, the virtual implant I is also moved in relation to the table 16 and the fragments F on the touchscreen 10, as is visualized in FIG.

As long as the gripping function is activated (ie as long as the touchscreen 10 remains touched), the virtual implant is anchored on the touchscreen 10 . Analogously to the above, this means that if the tablet 14 undergoes a further rotational and/or translational movement, only the image of the table 16 and the fragments F on the touchscreen 10 are moved accordingly, but not the virtual implant I.

When the user stops touching the touch screen 10, the gripping function is disabled. The virtual implant I is then left in that position and that orientation relative to the table 16 and the fragments F on the touchscreen 10 that it was in at the time the gripping function was deactivated. This is shown in FIG.

The position and orientation of the tablet 14 in a suitable form of representation, for example as a 4×4 transformation matrix, can be determined using software that is known per se. In order to detect whether the crosshairs 12 or another selection mark is directed at a virtual object, methods known per se can be used to calculate whether the beam from the center of the camera along the optical axis (derived from a 4×4 camera matrix) has a Intersection with a surface model of the virtual object. If the optical axis intersects with several surfaces the object whose intersection point is closest to the camera is selected preferentially.

For gripping and anchoring, the transformation between the tablet 14 and the target object at the time of gripping can be calculated. To move the virtual object, this transformation is combined with the current camera matrix and applied to the virtual object. This results in rigid anchorage during gripping.

Both can be implemented with known vector and matrix mathematics as well as 3D geometry methods.

Claims

patent claims

1. Computer-aided method for the moving representation of a real image of an environment and at least two virtual objects (F, F') on a screen (10), containing the steps of a) capturing a real image (17) of an environment, b) joint three-dimensional Displaying the image (17) of the environment, in particular the at least one real object (16), and the virtual objects (F, F') on the screen (10), c) selecting at least one of the virtual objects (F ^f ) in response to the activation of a gripping function, d) determining a rotary and/or translatory movement of a manipulation device (11), e) rotary and/or translatory movement of the image (17) of the environment, in particular of the real object (16) , and the non-selected virtual objects (F) relative to the selected virtual object (F ^f ), as a function of the rotational and/or translatory movement of the manipulation device (11) determined in step d), s as long as the grab function is activated.

2. The method according to claim 1, wherein the manipulation device (11) contains a camera with which a real image of an environment, in particular at least one real object (16), is recorded, which is recorded together with the virtual objects (F, F' ) is displayed on the screen (10).

3. The method according to claim 2, wherein in step d) the movement of the manipulation device (11) from the camera ra recorded image (17) of the environment, in particular the at least one real object (16) is determined.

4. The method according to any one of the preceding claims, wherein the virtual objects (F, F') are displayed on a screen (10) of the manipulation device (11).

5. The method according to any one of the preceding claims, wherein on the screen (10), in particular a screen (10) of the manipulation device (11), a virtual le selection mark (12) is displayed, wherein when activating the gripping function that virtual object ( F ^f ) is selected, on which the selection mark (12) on the screen (10) is directed.

6. The method according to any one of the preceding claims, wherein an implant (I) is additionally shown on the screen (10).

7. The method according to any one of claims 5 and 6, wherein the virtual selection mark (12) is displayed in a central area of the screen (10).

8. The method according to any one of claims 5 to 7, wherein the screen (10) is a touch screen (10) and the gripping function is activated when a user touches the touch screen (10).

The method of claim 8, wherein the gripping function remains activated as long as the user is touching the touch screen (10) and is deactivated when the user stops touching the touch screen (10).

10. The method as claimed in one of the preceding claims, in which the selected virtual object (F ^f ) is left in that position and that orientation in relation to the other virtual objects (F) on the screen, in particular the touch screen (10). who it is for

time of deactivation of the gripping function.

11. Computer-supported method according to one of the preceding claims for planning an operation on an anatomical structure with at least two partial structures, in particular a fractured bone containing at least two fragments (F, F'), in which steps a) to e) are carried out , where the virtual objects (F, F') are images of the partial structures, in particular the fragments.

12. The method according to claim 11, wherein an implant (I) is additionally displayed on the screen (10).

13. Computer program product which executes a method according to egg nem of claims 1 to 12 when it runs on a device (14), in particular a mobile device (14), wherein the device (14) has a screen (10), in particular has a touchscreen (10), and wherein the mobile device (14) is designed to determine a rotational and/or translational movement of the device (14).