WO2019122440A1

WO2019122440A1 - System and method for capturing and visualizing a 3d scene in real-time

Info

Publication number: WO2019122440A1
Application number: PCT/EP2018/086810
Authority: WO
Inventors: Karl-Ludwig Blocher; Reinhard Klein; Patrick STOTKO; Stefan KRUMPEN; Michael Weinmann
Original assignee: Viimle Gmbh; Rheinische Friedrich-Wilhelms-Universität Bonn
Priority date: 2017-12-22
Filing date: 2018-12-21
Publication date: 2019-06-27
Also published as: DE102017131083A1

Abstract

The invention relates to a method and a system for capturing and visualizing a 3D scene in real- time. The system comprises a display device arranged for visualizing at least a part of said 3D scene based on display information. The system further comprises a processing unit arranged for receiving capturing information, incrementally constructing a 3D model based on said capturing information, and generating and sending display information to the display device using said model. The system further comprises a movable capturing device arranged for capturing information from at least a part of said 3D scene and sending capturing information to said processing unit.

Description

System and method for capturing and visualizing a 3D scene in real-time

DESCRIPTION

FIELD OF THE INVENTION

The invention relates to a system and a method for incrementally capturing and remote visualization of a quasi-static 3D scene.

BACKGROUND OF THE INVENTION

With the help of special glasses or head mounted display (HMD), it is possible to visualize a 3D scene for a user, for example in a 3D cinema, with a 3D television or a game console. In the latter case, it may be possible that the location and orientation of the HMD determines how the 3D scene is visualized to the user, giving her the impression that she can move within the 3D scene and explore it. Normally, the visualization is based on a model of the 3D scene that resides in a memory coupled with a processor and the HMD. The model may be constructed using images from multiple special 3D cameras. These special cameras need to be positioned at well-defined positions with respect to the 3D scene. Constructing the model from the images may take quite some time. Therefore, both deciding which 3D scenes are available and con- structing the model needs to be done before the user starts using the HMD. Thus, it is impossi- ble for the user to immerse immediately into a previously unknown and not yet reconstructed remote scene.

OBJECT OF THE INVENTION

It is, therefore, an object of the present invention to provide a system and a method for capturing and remotely visualizing a quasi-static 3D scene in real-time.

BRIEF SUMMARY OF THE INVENTION

This object is achieved by the invention as claimed in the independent claims. Advanta- geous embodiments are described in the dependent claims. The use of the singular form should not exclude a possible multiplicity and vice versa, except for cases where it is explicitly stated.

According to the invention, a system for incrementally capturing and remote visualization of a quasi-static 3D scene comprises a movable image capturing device, a processing unit, and a display device remote from said capturing device. Said movable image capturing device is arranged for incrementally capturing information from at least part of said quasi-static 3D scene by moving the image capturing device between successive captured images and for sending said captured information to said processing unit in real-time. Said processing unit is arranged for receiving said captured information from the movable image capturing device, for incremen- tally constructing a 3D model of the quasi-static 3D scene based on said captured information in real-time, and for generating display information in real-time and sending said display infor- mation to said display device using said 3D model. Said display device is arranged for receiving said display information from the processing unit and for visualizing in real-time at least part of said 3D scene based on said display information.

Because of the separation of capturing device, processing unit and display device, the capturing, the construction and the visualization can be performed asynchronously. E.g. captur- ing may be done with a frame rate by one or more capturing devices with 15 fps or 30 fps and the construction may be done with 30 fps. The display information may comprise information at a rate (or sent at a rate) of 20 fps and the visualisation may be done with 90 fps, which may be needed for rendering on the display device, which may be a Virtual Reality (VR) device. Thus, for each device, the device specific real time requirements can be met independently. This al- lows the capturing, the constructing and the visualizing in real time or near real time, in particu- lar when the 3D scene is captured in increments and the corresponding 3D model is construct- ed and developed from the captured increments. As a result, right after the first capturing, transmission, construction and visualization, the user can explore the already captured part of the 3D scene starting from any location and orientation she wants. It is therefore possible for the user of the display device of the proposed system to immediately explore the continuously grow- ing reconstruction of the 3D scene. This may create a sentiment of immersion to this user from the very beginning. Already with a partially reconstructed 3D-scene, the user can explore the 3D-scene independently from the position and orientation of the capturing device.

Said movable image capturing device may be arranged for incrementally capturing infor- mation from at least part of said quasi-static 3D scene, where an increment of capturing infor- mation from at least part of said quasi-static 3D scene involves a first step of moving the captur- ing device and a second step of capturing information from at least part of said quasi-static 3D scene after the movement of the capturing device, wherein said movable image capturing de- vice is arranged for sending said captured information to said processing unit.

This allows the operator of the movable capturing device to find the best position for cap- turing information about the 3D scene and, therefore, capture only relevant information and send it to the processing unit in a controlled way.

Said processing unit may be arranged for sending a capturing request to said movable capturing device, wherein said capturing request indicates capturing information that is needed to visualize at least part of said 3D scene.

Thus, the capturing of the 3D scene can be aided or even guided by the processing unit. This guiding can be implemented in ways comparable to the one that is implemented in smartphone apps where, for example, a user is guided to take several pictures that are subse- quently merged by the smartphone app to a panorama picture.

Said movable capturing device can be arranged for capturing information from at least part of said 3D scene based on said capturing request.

This allows for an efficient capturing of the 3D scene, facilitating both an efficient construc- tion of the 3D model and an enhanced sentiment of immersion for the user of the display de- vice.

Said capturing request can include information on the motion of the movable image cap- turing device that is needed for capturing the requested information.

The operator of the movable image capturing device can thus be guided to find the best position for capturing the 3D scene.

A motion of the movable image capturing device may include a translational motion.

Translational motions of the movable image capturing device facilitate the capturing and, finally, the visualization of the 3D scene from different perspectives.

The display device can be arranged for sending a display request to said processing unit, wherein said display request indicates the display information needed to visualize said at least part of said 3D scene. Furthermore, said processing unit can be arranged for receiving said dis- play request and for sending display information to said display device in response to said dis play request.

In this way, the visualization of the 3D scene can be adapted to the actions of the user of the display device in real-time.

The display device may be arranged for sending a display request to said processing unit, wherein said display request indicates the display information needed to visualize at least part of said 3D scene. Furthermore, said processing unit is arranged for receiving said display re- quest and for generating and sending a capturing request to the movable image capturing de- vice in order to capture information to fulfill said display request.

The user of the display device can thus trigger capturing requests of the processing unit that are used to aid and guide the capturing of the 3D scene according to the current needs of the user. The display device can comprise a head mounted display (HMD) and a tracking system of said HMD, wherein the display device is arranged for generating said display request based on tracking data relating to the position and orientation of the HMD from said tracking system.

Using such a system, the user may trigger the display and the subsequent capturing re- quests by moving the head mounted display. The moving may reflect natural movements of the user when watching the 3D scene from a different perspective or focusing on a different part of it. The sentiment of immersion can thus be further enhanced.

Alternatively, a usual 2D display with a 3D viewer can be used. The tracking can than be accomplished with the help of a (3D) mouse or other tracking tools like cursors.

A further display device may be arranged for sending a further display request to said pro- cessing unit, for receiving further display information from the processing unit and for visualizing at least part of said 3D scene based on said further display information. Said further display request indicates the further display information needed to visualize said at least part of said 3D scene. Accordingly, said processing unit can also be arranged for generating and sending fur- ther display information to said further display device using said model and in response to said further display request.

An advantage of a second or a further display device, which may be located at a different location than the first display device, is that it enables a second or a further user to inde- pendently explore the 3D scene, for example at the same time, using the same 3D model.

The captured image information may comprise data relating to red, blue, green and depth channels (RBG-D).

Such data may be generated by the capturing device and sent to the processing unit by the capturing device.

The image capturing device can be provided with a feedback display device, which is ar- ranged for displaying a visualisation of the 3D-scene based on the constructed 3D model.

In this way the operator of the capturing device may monitor the capturing and the con- structing of the 3D model. The operator may then decide to capture certain parts of the 30- scene based on this monitoring. The feedback display device may be integrated in the capturing device, such as in the case of a Microsoft Hololens or a ASUS ZenFone AR.

The visualisation of the 3D-scene based on the constructed 3D model can be enriched by CAD models, pointers and / or highlights in real-time.

Such enrichments can fulfill multiple functions. On one hand, the user of the display de- vice or the operator of the movable image capturing device can thus be guided to perform spe- cific actions. On the other hand, the 3D scene may be enriched by objects that are not present in the 3D scene. The system according to the invention thus provides an environment where the user and the operator can interact in real-time, adding, for example, content to the 3D scene.

To achieve the object, a method for incrementally capturing and remote visualization of a quasi-static 3D scene is proposed that employs any one of the systems described above, in- cluding the steps of:

1. a movable image capturing device is incrementally capturing information from at least a part of said quasi-static 3D scene by moving the image capturing device between successive captured images and sending said captured information to a processing unit in real-time;

2. said processing unit is receiving said captured information from the movable image capturing device;

3. said processing unit is incrementally constructing a 3D model of the quasi-static 3D scene based on said captured information in real-time;

4. said processing unit is generating display information in real-time and sending said dis play information to a display device using said 3D model; and

5. said display device is receiving said display information from the processing unit and visualizing in real-time at least part of said 3D scene based on said display information.

Moreover, an increment of capturing information from at least part of said quasi-static 3D scene can involve a first step of moving the capturing device and a second step of capturing information from at least part of said quasi-static 3D scene after the movement of the capturing device.

The method may further include a step where said processing unit sends a capturing re- quest to said movable capturing device that indicates capturing information that is needed to visualize at least part of said 3D scene.

In the latter, said movable capturing device can capture information from at least part of said 3D scene based on said capturing request.

Said capturing request may also include information on the motion of the movable image capturing device that is needed for capturing the requested information.

According to the invention, the method includes motions of the movable image capturing device that represent translational motions. Moreover, the method may also comprise the following optional steps:

- the display device is sending a display request to said processing unit;

- said processing unit is generating display information and sending the display infor- mation to said display device in response said display request, wherein said display request indicates the display information needed to visualize said at least part of said 3D scene.

Other optional steps are:

- the display device is sending a display request to said processing unit;

- said processing unit is generating and sending a capturing request to the movable image capturing device in order to capture information to fulfill said display request, wherein said dis- play request indicates the display information needed to visualize said at least part of said 3D scene.

Another step may be that the movable image capturing device is moved with respect to one of more objects in the 3D-scene in accordance with a movement of the display device.

Thus, the capturing of the 3D scene can be aided and/or guided with reference to specific objects.

The method may also include the step of providing feedback to an operator of the captur- ing device by displaying the part of a 3D-scene that has been captured and/or a visualisation of the 3D-scene based on the constructed 3D model.

Some or all of the steps of the method according to the invention may be repeated inde- pendently from each other.

According to another aspect of the invention, a system is provided for incrementally cap- turing and remote visualization of a quasi-static 3D scene, preferably in real-time, comprising: a) a display device arranged for receiving display information from a processing unit and for visualizing at least a part of said 3D scene based on said display information,

b) said processing unit arranged for receiving capturing information from a movable cap- turing device, for incrementally constructing a 3D model based on said capturing information, for generating and sending display information to said display device using said model, and prefer- ably for sending a capturing request to said movable capturing device, wherein said capturing request indicates capturing information needed for further constructing said 3D model;

c) said movable capturing device arranged for capturing information from at least a part of said 3D scene, preferably based on said capturing request, and for sending capturing infor- mation to said processing unit.

According to the invention, information about a 3D scene is continuously captured by the capturing device and send or transmitted to the processing unit, where a 3D model of the 3D scene is incrementally constructed. Because of the separation of capturing device, processing unit and display device, the capturing, the construction and the visualization can be performed asynchronously. E.g. capturing may be done with a frame rate by one or more capturing devices with 15 fps or 30 fps and the construction may be done with 30 fps. The display information may comprise information at a rate (or sent at a rate) of 20 fps and the visualisation may be done with 90 fps, which may be needed for rendering on the display device, which may be a Virtual Reality (VR) device.

Thus, for each device, the device specific real time requirements can be met independent- ly. This allows the capturing, the constructing and the visualizing in real time or near real time. As a result immediately after the first capturing, transmission, construction and visualization have started the user can explore the already captured part of the 3D scene starting from any location and orientation she wants.

Thus with this system it is possible for the user of the display device to immediately ex plore the continuously growing reconstruction of the 3D scene, which may create a sentiment of immersion to this user from the very beginning. Already with a partially reconstructed 3D-scene, the user can explore the 3D-scene independently from the position and orientation of the cap- turing device.

Furthermore, the user of the display device and the operator of the capturing device may communicate with each other while using this system, for example sending each other voice messages or communicate via additional visual information overlaid the reconstructed scene, e.g. pointers, highlights or colours. In this way, the user and the operator may get the perception that the user is virtually present in the same scene.

Moreover, since capturing information from only a part of the 3D scene is required, the capturing may take place using a single capturing device, for example a 3D sensor on a mobile phone.

In general, the 3D scene may be an indoor situation, for example a room. In that case, the user of the display device may be an adviser (or craftsman) regarding heating, furniture, selling houses or the like, who wants to explore the room without being actually present in that room. Alternatively, the 3D scene may comprise a car or another technical installation, which needs inspection by a mechanic. The mechanic may inspect the installation using the display device without being actually present near the installation.

The term real-time, especially in the claim or with respect to the whole system, may indi- cate that the visualisation of the part of the 3D scene takes places more or less at the same time as the capturing of that scene, i.e. with a delay of less than 1-2 seconds, independent from the asynchronously running system components.

The term“remote visualization” indicates that the visualization may takes place at another location than the quasi-static 3D scene.

The capturing device is a movable capturing device, i.e. the capturing device is movable with respect to the objects in the 3D-scene. The 3D scene is said to be quasi-static, indicating that the changes or movements of objects within in the 3D scene occur slower than the move- ment of the capturing device within or around the 3D-scene. The capturing device may be an Augmented Reality (AR) device with integrated Camera / 3D sensor.

The display device may preferably be a so-called virtual reality or augmented reality de- vice, for example the HTC Vive or Oculus Rift or Microsoft Hololens. The display device may be or comprise a head mounted display (HMD) which is to be placed on the head of the user. The HMD may comprise two displays (or two parts of a single display), one for each eye. Alternative- ly the display device may also comprise a screen that displays both images for both eyes on a screen, and a pair of glasses which has a filter function, such that each eye receives only one of the two images. Alternatively, a standard 2D screen or mobile device with a 3D Viewer or a 3D CAD-System may be used.

The processing unit may preferably be a server or any other kind of computing device. The processing unit may be located at the same location as the display device, at the same location as the capturing device or at a third location, for example in a computing centre, thus in the cloud. The functions of the processing unit may be also carried out by software, which may be executed in the display device or in the capturing device.

The capturing device may preferably be a so-called 3D sensor, for example a Microsoft Kinect or a mobile phone with an integrated 3D-sensor. The capturing device may be a camera (for example as part of a mobile phone). The capturing device is movable within and/or around the 3D scene.

The position and orientation of the capturing device with respect to the 3D scene may be derived (by the capturing device or the processing unit) by estimating the transformation be- tween the current and previous image (or frame) obtained by capturing device.

The 3D model is an incremental model that describes, at least partly, the quasi-static 3D scene to be visualized. The model may only comprise geometrical forms present in the 3D sce- ne, but may also comprise additional information like colour and reflectance properties. The 3D model may be constructed and further extended, while the capturing device is capturing infor- mation from the 3D scene and/or while the display device is visualizing a part of the 3D scene based on the 3D model. The constructed 3D model may further be enriched by CAD-models inserted into the reconstructed scene or enriched by e.g. pointers and highlights. Furthermore, the quasi static scene might be visualized by a sequence of consistent static visualizations. That means, since the changes in the 3D scene are much slower than the movement of the capturing device, the 3D scene may be captured completely and consistently.

In an embodiment of the system according to the invention, the display device is arranged for sending a display request to a processing unit, wherein said display request indicates the display information needed to visualize said at least part of said 3D and said processing unit is arranged for receiving said display request and for sending display information to said display device in response to said display request.

In another embodiment of the invention, said processing unit is arranged for continuously sending display information to said display device, for example when the processing unit is not processing a display request or when the processing unit is idle.

In an embodiment of the system according to the invention, the display device comprises a head mounted display (HDM) and tracking system of said HDM, wherein the display device is arranged for generating said display request based on tracking data, relating to the position and orientation of the HMD, from said tracking system.

In another embodiment of the system according to the invention, the system further corn- prises a feedback display device, which is arranged for displaying the part of the 3D-scene that has been captured and/or a visualisation of the 3D-scene based on the constructed 3D model. In this way the operator of the capturing device may monitor the capturing and the constructing of the 3D model. The operator may then decide to capture certain parts of the 3D-scene based on this monitoring. The feedback display device may be integrated in the capturing device, such as in the case of a Microsoft Hololens or a ASUS Zenfone AR.

In another embodiment of the system according to the invention, the capturing information comprises data relating to red, blue, green and depth channels (RBG-D). These data may be generated by the capturing device and send to the processing unit by the capturing devices.

In another embodiment of the system according to the invention, the system further corn- prises a further display device arranged for sending a further display request to said processing unit, for receiving further display information from the processing unit and for visualizing at least a part of said 3D scene based on said further display information, wherein said further display request indicates the further display information needed to visualize said at least part of said 3D scene; and, wherein said processing unit is arranged for generating and sending further display information to said further display device using said model and in response to said further dis- play request.

When the further display information needed to visualize said at least part of said 3D sce- ne is already available in the further display device, a further display request need not to be sent. The same may apply to the display request: when the display information needed to visu- alize said at least part of said 3D scene is already available in the display device, the display request need not to be sent

As an alternative to sending a further display request, the further display information may be periodically pushed by the processing unit to the further display device without the need for a further display request. The further display information may comprise information relating to the 3D model. When the further display information is periodically pushed it may comprise infor- mation relating to the incremental change of the 3D model.

The same may apply to the display request: the display information may be periodically pushed by the processing unit to the display device without the need for a display request. The display information may comprise information relating to the 3D model. When the display infor- mation is periodically pushed it may comprise information relating to the incremental change of the 3D model. In yet another embodiment of the system according to the invention, the display device is arranged for sending said display request over the internet; and/or wherein the processing unit is arranged for sending said display information over the internet, and/or wherein the capturing device is arranged for sending said capturing information over the internet.

According to a second aspect of the invention, a method is provided for capturing and re- mote visualizing a 3D scene, preferably in real-time, using a system described in this document, comprising the steps of:

- (a movable capturing device) capturing information from at least a part of a 3D scene and sending said capturing information to a processing unit, preferably in response to and based on a capturing request;

- (said processing unit) constructing a 3D model of at least a part of said 3D scene based on said capturing information;

- (the processing unit) generating display information using said 3D model and sending the display information to a display device; and,

- (the display device) visualizing a part of the 3D scene based on the display information that was received from the processing unit.

In an embodiment of the method according to the invention, the method further comprises the steps of:

- the display device sending a display request to the processing unit;

- the processing unit generating display information and sending the display information to a display device in response the display request, wherein said display request indicates the dis- play information needed to visualize said at least part of said 3D.

In an embodiment of the method according to the invention, the method further comprises the step: of the processing unit generating and sending said display information continuously to said display device, for example when the processing unit is not processing a display request.

- sending a display request, for example based on the movement of the display device; and/or

- generating and sending a capturing request, preferably based on said display request, wherein said capturing request indicates the capturing information needed to further construct said 3D model.

In an embodiment of the method according to the invention, the method further comprises the step of moving the capturing device in accordance with a movement of the display device.

In an embodiment of the method according to the invention, the method further comprises the step of displaying (by a feedback display device to an operator of the capturing device) a part of the 3D-scene that has been captured and/or a visualisation of the 3D-scene based on the constructed 3D model. In this way the operator of the capturing device may monitor the cap- turing and the constructing of the 3D model. The operator may then decide to capture certain parts of the 3D-scene based on this monitoring. The feedback display device may be integrated in the capturing device, such as in the case of a Microsoft Hololens or a ASUS ZenFone AR.

In an embodiment of the method according to the invention, one, some or all of these steps may be repeated independently from each other.

Embodiments of the method for capturing and visualizing a 3D scene in real-time accord- ing to the invention and its features and advantages are explained in this document with refer- ence to the system for capturing and visualizing a 3D scene in real-time.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, reference is established to the following description made in connection with accompanying drawings in which:

Fig. 1 shows a schematic overview of an embodiment of a system for capturing and visualiz- ing a 3D scene in real-time according to the invention; and

Fig. 2 shows a schematic overview of an embodiment of a method for capturing and visualiz- ing a 3D scene in real-time according to the invention.

DETAILED DESCRIPTION OF THE INVENTION

In figure 1 , a schematic overview of an embodiment of a system 100 for incrementally capturing and remote visualization of a quasi-static 3D scene in real-time according to the in- vention is shown. This system 100 comprises a display device 1 10, a processing unit 120 and a movable capturing device 130. The movable capturing device may be directed to a 3D scene 180 and may be movable with respect to objects within that scene.

The display device 1 10, the processing unit 120 and/or the capturing device 130 may send data or information towards each other. As these three devices may be located at two or three different locations, the information may be sent over the internet or a local area network or via a direct data cable between them.

The capturing device 130 is arranged for capturing information from the 3D scene 180. As the capturing device may be a camera, the capturing information 170 may comprise or consists of images or a video stream. The video stream may comprise so-called RGB-D information. The capturing information 170 may further comprise information about the position and orientation of the capturing device with respect to the 3D scene 180.

In an embodiment of the system 100, a further capturing device may be provided. The fur- ther capturing device may have the same features as is described in this document with respect to the capturing device 130. An advantage of several capturing devices is that the capturing of the 3D scene will take place faster.

The system 100 may be provided with a feedback display device 135 that is arranged dis playing the part of a 3D-scene 180 that has been captured and/or a visualisation of the 30- scene based on the constructed 3D model, preferably on the basis of feedback information 165. The feedback display device may be arranged for receiving said feedback information 165 and the processing unit 120 may be arranged for generating the feedback information 165 and sending it to the feedback display device. The feedback information 165 may comprise data relating to a part of a 3D-scene 180 that has been captured and/or a visualisation of the 30- scene based on the constructed 3D model.

The feedback display device 135 may comprise a communication unit and the display de- vice 110 may also comprise a communication unit. In that case, the operator of the capturing device 130 and the user of the display device 110 may communicate with each other using these two communication units.

The feedback display device 135 (with or without the communication unit) may be inte- grated in the capturing device 130. An example of such a device is the ASUS ZenFone AR or the Microsoft Hololens. This integration may allow an overlay of a visualization of a part of the 3D scene based on the constructed model with the 3D scene 180. In this way, the operator may see which parts of the 3D scene 180 are not (yet) included in the constructed 3D model.

The capturing device 130 may be arranged for receiving a capturing request 160 from the processing unit 120. This request indicates the capturing information needed by the processing unit 120 to further construct a 3D model of the 3D scene 180. This request may be processed by the capturing device 120 and may result in displaying arrows on the feedback display device 135 to the operator of the capturing device 130. These arrows may indicate to the operator in which direction she should move the captur- ing device. Furthermore, additional information may be given to the operator of the capturing device 130, by using the vibration mode of the capturing device (such as the ASUS ZenFone AR) in case that the capturing device movements get to fast or by overlaying additional CAD- data.

The capturing request 160 may comprise a voice message from the user of display device 110 indicating which part of the 3D scene she would like to explore further, but for which the model may not yet comprise (sufficient) data. The voice message from the user may be cap- tured by the display device 110 and may be directly sent to the capturing device 130 without further processing by the processing unit 120.

The capturing device 130 may be arranged for reproducing the voice message from the user of the display device 110. In this way, the operator of the capturing device 130 may be in- formed about the direction she should move the capturing device 130. Furthermore, the display device 1 10 may be arranged for reproducing a voice message that was recorded by the captur- ing device 130. In this way, the user and the operator may communicate with each other with voice messages. In this case, the capturing device 130 may be arranged for sending infor- mation relating to a recorded voice messages to the processing unit 120, which may forward to information to the display device 1 10. Or the capturing device 130 may be arranged for sending information relating to a recorded voice messages directly to the display device 110.

The processing unit 120 may be a server, possibly somewhere in the cloud connected by internet connection to both the capturing device 130 and the display device 110. The processing unit 120 is arranged for receiving the capturing information 170 from the capturing device 130. Based on this information, the processing unit 120 constructs an incremental 3D model of the 3D scene 180. The 3D model may therefore be expanded and/or improved as new capturing information 170 is received by the processing unit 120.

The 3D model that is constructed by the processing unit 120 may be a so-called Voxel- model, for example a Truncated Signed Distance Field (TSDF) model, or a Surfel-based model, or any other 3D point based method.

Besides constructing the 3D model, the processing unit 120 is also arranged for providing display information 150 to the display device 110, optionally in response to a display request 140 from the display device 110. The display request 140 may indicate the display information 150 needed to visualize at least part of said 3D scene. This part may be the part of the 3D sce- ne the user of the display device wants to explore. The display request 140 may comprise in- formation about the location and orientation of the display device 1 10.

The display information 150 is transferred to the display device 110 in response to a dis- play request 140 from the display device 1 10 and/or continuously streamed or pushed if no im- mediate display request has to be served.

The display request 140 may further indicate that a new 3D scene is to be captured. The capturing request 150 may, in response to that indication, also indicate that a new copy of the 3D scene is to be captured, for example if the quasi-static scene has changed.

Based on optionally this request 140 and using the 3D model, the processing unit 120 generates display information 150 and sends this information 150 to the display device 1 10. This may comprise rendering the images needed for the visualization of that part of the 3D sce- ne and sending these images (or data relating to those images) to the display device 110 as display information 150.

The processing unit 120 may also be arranged for continuously generating and sending display information 150 to the display device 1 10. This may happen when the processing unit 120 is not processing any display requests, capturing requests and/or is not constructing the 3D model, or may be called“idle”. In those moments, the processing power of the processing unit 120 may be used for continuously generating and sending display information 150 to the display device 1 10.

The processing unit 120 may also be arranged for providing feedback information 165 to the feedback display device 135, optionally in response to a feedback request from the feed- back display device 135. The feedback request may indicate the feedback information 165 needed to visualize the feedback.

The processing unit 120 may further be arranged for determining whether the whole 3D scene 180 has been captured, or in other words, that the 3D model is complete. In that case, further capturing and constructing is not required, until something changes in the 3D scene. The capturing request 150 may further indicate that further capturing is not required and the captur- ing device 130 (or the feedback display device 135) may be arranged for conveying this infor- mation to the operator of the capturing device 130. According to the invention, the 3D model resides in the processing unit 120 and the dis play information that is needed for visualizing the required part of the 3D scene is send to the display device 110 in response to a display request 140.

Furthermore, display information may also be sent to the display device 1 10 not in re- sponse to a display request 140. In that case, display information may be periodically or contin- uously sent or pushed to the display device and the display information may relate the incre- mental change of the 3D model.

Since the bandwidth of information transfer between the processing unit 120 and the dis- play device 110 may be limited, the display information that is sent in response to a display re- quest is prioritized (when sending by the processing unit 120) over the display information that is pushed by the processing unit 120.

Thus, the requirements to the data connection (for example in terms of bandwidth) be- tween the processing unit 120 and the display device 1 10 are less stringent in comparison with a system wherein the whole 3D model is sent to the display device when the 3D model is up- dated. Furthermore, the visualised 3D-scene in the display device 110 may be incrementally updated, further reducing the requirements to the data connection between the processing unit 120 and the display device 1 10.

The processing unit 120 might be realized as distributed system with multiple independent reconstruction, storage and management units.

The display request 140 may further comprise a voice message from the user of the dis play device 110. In that case, the processing unit 120 may be arranged for receiving that voice message and for including said voice message in the capturing request 160, which the pro- cessing unit 120 sends to the capturing device 130. The capturing device 130 may be arranged for rendering said voice message.

This provides an alternative or additional way to the communication units described above, for providing communication between the user of the display device 110 and the opera- tor of the capturing device.

The display device 1 10 is arranged for receiving the display information 150 and for visu- alizing at least a part of said 3D scene. For this, the display device 1 10 may comprise or con- sists of a Head Mounted Device (HMD), such as a HTC Vive or an Oculus Rift. The visualizing may comprise displaying images that were received from the processing unit 120 as display information 150.

The display device 1 10 or the HMD may comprise a tracking system for tracking the posi- tion and orientation of the HMD and for providing tracking information or tracking data relating to said position and orientation. In that case, the display device may be arranged for generating said display request based on said tracking data.

The user of the display device 1 10 may move the display device 110 in order to explore a further part of the 3D scene. In response of the movement the display device 110 may generate the display request 140 and send this request to the processing unit 110. The display device 110 may further comprise a microphone in order to capture a voice message from the users.

Besides the display device 110, a further display device (not shown in figure 1 ) may be comprised in an embodiment of the system according to the invention. The further display de- vice may be arranged for sending a further display request to said processing unit, for receiving further display information from the processing unit and for visualizing at least a part of said 3D scene based on said further display information. Said further display request indicates the fur- ther display information needed to visualize said at least part of said 3D scene. The processing unit 120 may be further arranged for generating and sending further display information to said further display device using said model and in response to said further display request.

This further display device may be identical to display device 110. In this embodiment, a second user using the further display device may explore the 3D scene independently from the first user using the display device 1 10. The part of the 3D scene that is visualized by display device 110 may therefore be different from the part of the 3D scene that is visualized by the further display device.

It is also possible that display device 110 is a Virtual Reality device (as discussed above) and that the further display device is a monitor. In that case the second user may see a visuali- zation of an arbitrary part of the 3D scene 180 (which may be different from the part of the 3D scene the first user is seeing) or a synchronized visualization of the same part of the 3D scene as the first user, although it may be not in 3D.

Figure 2 shows a schematic overview of an embodiment of a method 200 for incremental- ly capturing and remote visualization of a quasi-static 3D scene in real-time according to the invention. The method may use any embodiment of the system for incrementally capturing and remote visualization of a quasi-static 3D scene in real-time as described in this document. The method 200 comprises the following steps:

Step 210: (a movable capturing device 130) capturing information from at least a part of a 3D scene 180 and sending said capturing information (to a processing unit 120), in response to and preferably based on a capturing request 160.

Step 220: (a processing unit 120) constructing a 3D model of at least a part of said 3D scene based on said capturing information.

Step 230: (a display device 1 10) sending a display request 140 (to the processing unit

120).

Step 240: (the processing unit 120) generating display information 150 using said 3D model and sending the display information 150 (to said display device 1 10).

Step 250: (the display device 110) visualizing a part of the 3D scene based on the display information 150 (that was received from the processing unit 120).

In an embodiment of the method 200, the method may further comprise any of the follow- ing the steps of:

Step 260: (the display device 1 10) (again) sending a display request 140 (to the pro- cessing unit 120), for example based on the movement of the display device 1 10; and/or

Step 270: (the processing unit 120) generating and sending a capturing request 160 (to the capturing device 130), preferably based on said display request 140, wherein said capturing request 160 indicates the capturing information 170 needed to further construct said 3D model;

All the above mentioned steps 210 -270 may be executed in the order as presented above and in Figure 2, but may also be executed in any other order. Furthermore, some or all of these steps may be repeated independently from each other, as the processing unit is arranged for incrementally constructing said 3D model.

In an embodiment, step 210 may comprise moving the movable capturing device 130 with respect to one or more objects within the 3D-scene, optionally in accordance with a movement of the display device. In an embodiment, instead of or in addition to the display device sending a display request to the processing unit, said display information may be continuously generated and sent to the display device, for example when the processing unit is not processing a display request.

In an embodiment the capturing device 130 is arranged for capturing parts of the 3D sce- ne independently from the parts of the 3D scene that are visualized by the display device 1 10. In another embodiment, the capturing device 130 is arranged for capturing parts of the 3D sce- ne dependently from the parts of the 3D scene that are visualized by the display device 110, which may mean that the capturing device 130 is following the movement of the display device 1 10.

Embodiments of the method for capturing and visualizing a 3D scene in real-time accord- ing to the invention and its features are further explained above with reference to the system for capturing and visualizing a 3D scene in real-time. Furthermore, features of the embodiments of the method for capturing and visualizing a 3D scene in real-time may also be applied to embod- iments of the system for capturing and visualizing a 3D scene in real-time.

Furthermore, the invention may be described as follows: the invention relates to a system and a method for capturing and visualizing a 3D scene in real-time. The system comprises a display device arranged for visualizing at least a part of said 3D scene based on display infor- mation. The system further comprises a processing unit arranged for receiving capturing infor- mation, incrementally constructing a 3D model based on said capturing information, and gener- ating and sending display information to the display device using said model. The system fur- ther comprises a capturing device arranged for capturing information from at least a part of said 3D scene and sending capturing information to said processing unit. The display device may send a display request indicates the display information needed to visualize at least part of said 3D scene. The processing unit may generate a capturing request on the basis of the said dis- play request. The capturing device may capture said capturing information based on said cap- turing request.

Embodiments of the invention can be summarized as follows:

1. System for incrementally capturing and remote visualization of a quasi-static 3D sce- ne, preferably in real-time, comprising:

a) a display device arranged for receiving display information from a processing unit and for visualizing at least a part of said 3D scene based on said display information, b) said processing unit arranged for receiving capturing information from a movable cap- turing device, for incrementally constructing a 3D model based on said capturing information, and for generating and sending display information to said display device using said model and preferably for sending a capturing request to said movable capturing device, wherein said cap- turing request indicates capturing information needed for further constructing said 3D model; c) said movable capturing device arranged for capturing information from at least a part of said 3D scene, preferably based on said capturing request, and for sending capturing infor- mation to said processing unit.

2. System according to embodiment 1 , wherein the display device is arranged for send- ing a display request to a processing unit, wherein said display request indicates the display information needed to visualize said at least part of said 3D and said processing unit is ar- ranged for receiving said display request and for sending display information to said display device in response to said display request.

3. System according to embodiment 1 or 2, wherein said processing unit is arranged for continuously sending display information to said display device, for example when the pro- cessing unit is not processing a display request or when the processing unit is idle

4. System according to any one of embodiments 1-3, wherein the display device corn- prises a head mounted display (HDM) and tracking system of said HDM and wherein the display device is arranged for generating said display request based on tracking data, relating to the position and orientation of the HMD, from said tracking system.

5. System according to any one of embodiments 1-4, wherein capturing information comprises data relating to red, blue, green and depth channels (RBG-D).

6. System according to any one of embodiments 1-5, further comprising

a further display device arranged for sending a further display request to said processing unit, for receiving further display information from the processing unit and for visualizing at least a part of said 3D scene based on said further display information,

wherein said further display request indicates the further display information needed to visualize said at least part of said 3D scene; and,

wherein said processing unit is arranged for generating and sending further display in- formation to said further display device using said model and in response to said further display request. 7. System according to any one of embodiments 1-6, wherein the capturing device is provided with a feedback display device, which is arranged for displaying the part of a 3D-scene that has been captured and/or a visualisation of the 3D-scene based on the constructed 3D model.

8. Method for incrementally capturing and remote visualization of a quasi-static 3D sce- ne, preferably in real-time, using a system according to any one of claim 1-7, comprising the steps of:

- a movable capturing device capturing information from at least a part of a 3D scene and sending said capturing information to a processing unit, preferably in response to and based on a capturing request;

- said processing unit constructing a 3D model of at least a part of said 3D scene based on said capturing information;

- the processing unit generating display information using said 3D model and sending the display information to a display device; and,

- the display device visualizing a part of the 3D scene based on the display information that was received from the processing unit.

9. Method according to embodiment 8, further comprising the steps of

- the display device sending a display request to the processing unit;

- the processing unit generating display information and sending the display information to a display device in response the display request, wherein said display request indicates the display information needed to visualize said at least part of said 3D.

10. Method according to embodiment 8 or 9, further comprising the step of the pro- cessing unit generating and sending said display information continuously to said display de- vice, for example when the processing unit is not processing a display request.

11. Method according to any one of the embodiments 8-10, further comprising the steps of:

- the display device sending a display request to the processing unit, for example based on the movement of the display device; and/or

- the processing unit generating and sending a capturing request to the movable captur- ing device, preferably based on said display request, wherein said capturing request indicates the capturing information needed to further construct said 3D model.

12. Method according to any one of embodiments 8-1 1 , further comprising the step of moving the movable capturing device with respect to one of more objects in the 3D-scene, op- tionally in accordance with a movement of the display device.

13. Method according to any one of embodiments 8-12, further comprising providing feedback to an operator of the capturing device by displaying the part of a 3D-scene that has been captured and/or a visualisation of the 3D-scene based on the constructed 3D model.

14. Method according to any of embodiments 8-13, wherein one, some or all of these steps may be repeated independently from each other.

While the present inventions have been described and illustrated in conjunction with a number of specific embodiments, those skilled in the art will appreciate that variations and modi- fications may be made without departing from the principles of the inventions as herein illustrat ed, as described and claimed. The present inventions may be embodied in other specific forms without departing from their spirit or essential characteristics. The described embodiments are considered in all respects to be illustrative and not restrictive. The scope of the inventions is, therefore, indicated by the appended claims, rather than by the foregoing description.

G LOSSARY

Real-time

Computer systems are operated in real time if programs for processing data are constant- ly ready for operation and the processing results of these programs are (almost) simultaneously available, i.e. within a specified period of time and without delays exceeding system-related processing times. Depending on the application and system, the given time span can be, for example, a few microseconds or milliseconds, minutes or hours.

REFERENCE SIGNS

system for capturing and visualizing a 3D scene in real-time

display device

processing unit

movable capturing device

feedback display device

display request

display information

capturing request

feedback information

capturing information

3D scene

system for capturing and visualizing a 3D scene in real-time

capturing and sending capturing information

constructing a 3D model

sending a display request

generating and sending display information using said 3D model visualizing a part of the 3D scene

sending a display request based on the movement of the display device generating and sending a capturing request

Claims

1. System for incrementally capturing and remote visualization of a quasi-static 3D scene, comprising:

1.1 a movable image capturing device;

1.2 a processing unit; and

1.3 a display device remote from said capturing device;

1.4 said movable image capturing device being arranged for incrementally capturing information from at least part of said quasi-static 3D scene by moving the image capturing device be- tween successive captured images and for sending said captured information to said pro- cessing unit in real-time;

1.5 said processing unit being arranged for receiving said captured information from the mova- ble image capturing device;

1.6 said processing unit being arranged for incrementally constructing a 3D model of the quasi- static 3D scene based on said captured information in real-time;

1.7 said processing unit being arranged for generating display information in real-time and sending said display information to said display device using said 3D model;

1.8 said display device arranged for receiving said display information from the processing unit and for visualizing in real-time at least part of said 3D scene based on said display infor- mation.

2. System according to the preceding claim,

2.1 wherein said movable image capturing device is arranged for incrementally capturing infor- mation from at least part of said quasi-static 3D scene,

2.2 where an increment of capturing information from at least part of said quasi-static 3D scene involves

2.2.1 a first step of moving the capturing device and

2.2.2 a second step of capturing information from at least part of said quasi-static 3D scene af- ter the movement of the capturing device; and

2.3 wherein said movable image capturing device is arranged for sending said captured infor- mation to said processing unit.

3. System according to any one of the preceding claims,

3.1 wherein the processing unit is arranged for sending a capturing request to said movable capturing device,

3.2 wherein said capturing request indicates capturing information that is needed to visualize at least part of said 3D scene.

4. System according to the preceding claim,

wherein said movable capturing device is arranged for capturing information from at least part of said 3D scene based on said capturing request.

5. System according to any one of the preceding claims 3 to 4,

wherein said capturing request includes information on the motion of the movable image captur- ing device that is needed for capturing the requested information.

6. System according to any one of the preceding claims,

wherein a motion of the movable image capturing device includes a translational motion.

7. System according to any one of the preceding claims,

7.1 wherein the display device is arranged for sending a display request to said processing unit, wherein said display request indicates the display information needed to visualize said at least part of said 3D scene; and

7.2 said processing unit is arranged for receiving said display request and for sending display information to said display device in response to said display request.

8. System according to any one of the preceding claims,

8.1 wherein the display device is arranged for sending a display request to said processing unit,

8.2 wherein said display request indicates the display information needed to visualize at least part of said 3D scene; and

8.3 said processing unit is arranged for receiving said display request and for generating and sending a capturing request to the movable image capturing device in order to capture in- formation to fulfill said display request.

9. System according to any one of the preceding claims 7 to 8,

9.1 wherein the display device comprises a head mounted display (HMD) and

9.2 a tracking system of said HMD; and

9.3 wherein the display device is arranged for generating said display request based on tracking data relating to the position and orientation of the HMD from said tracking system.

10. System according to any one of the preceding claims 7 to 9,

further comprising

10.1 a further display device arranged for sending a further display request to said processing unit, for receiving further display information from the processing unit and for visualizing at least part of said 3D scene based on said further display information;

10.2 wherein said further display request indicates the further display information needed to visualize said at least part of said 3D scene; and

10.3 wherein said processing unit is arranged for generating and sending further display infor- mation to said further display device using said model and in response to said further dis- play request.

11. System according to any one of the preceding claims,

wherein captured image information comprises data relating to red, blue, green and depth channels (RBG-D).

12. System according to any one of the preceding claims,

wherein the image capturing device is provided with a feedback display device, which is ar- ranged for displaying a visualisation of the 3D-scene based on the constructed 3D model.

13. System according to the preceding claim,

wherein the visualisation of the 3D-scene based on the constructed 3D model is enriched by CAD models, pointers and / or highlights in real-time.

14. Method for incrementally capturing and remote visualization of a quasi-static 3D scene, us- ing a system according to any one of the preceding claims, comprising the steps of:

14.1 a movable image capturing device incrementally capturing information from at least a part of said quasi-static 3D scene by moving the image capturing device between successive captured images and sending said captured information to a processing unit in real-time;

14.2 said processing unit receiving said captured information from the movable image capturing device;

14.3 said processing unit incrementally constructing a 3D model of the quasi-static 3D scene based on said captured information in real-time;

14.4 said processing unit generating display information in real-time and sending said display information to a display device using said 3D model; and

14.5 said display device receiving said display information from the processing unit and visualiz- ing in real-time at least part of said 3D scene based on said display information.

15. Method according to the preceding claim,

15.1 where an increment of capturing information from at least part of said quasi-static 3D sce- ne involves

15.1.1 a first step of moving the capturing device and

15.1.2 a second step of capturing information from at least part of said quasi-static 3D scene after the movement of the capturing device.

16. Method according to any one of the preceding claims 14 to 15,

16.1 wherein said processing unit sends a capturing request to said movable capturing device,

16.2 wherein said capturing request indicates capturing information that is needed to visualize at least part of said 3D scene.

17. Method according to the preceding claim,

wherein said movable capturing device captures information from at least part of said 3D scene based on said capturing request.

18. Method according to any one of the preceding claims 16 to 17,

19. Method according to any one of the preceding claims 14 to 18,

20. Method according to any one of the preceding claims 14 to 19, further comprising the steps of:

20.1 the display device sending a display request to said processing unit;

20.2 said processing unit generating display information and sending the display information to said display device in response said display request, wherein said display request indi- cates the display information needed to visualize said at least part of said 3D scene.

21. Method according to any one of the preceding claims 14 to 20, further comprising the steps of

21.1 the display device sending a display request to said processing unit;

21.2 said processing unit generating and sending a capturing request to the movable image capturing device in order to capture information to fulfill said display request,

21.3 wherein said display request indicates the display information needed to visualize said at least part of said 3D scene.

22. Method according to any one of the preceding claims 14 to 21 , further comprising the step

Of moving the movable image capturing device with respect to one of more objects in the 3D- scene in accordance with a movement of the display device.

23. Method according to any one of the preceding claims 14 to 22, further comprising providing feedback to an operator of the capturing device by displaying the part of a 3D-scene that has been captured and/or a visualisation of the 3D-scene based on the constructed 3D model.

24. Method according to any of the preceding claims 14 to 23, wherein

one, some or all of these steps may be repeated independently from each other.