WO2023237768A1

WO2023237768A1 - Method of building a target structure, system and computer program therefor

Info

Publication number: WO2023237768A1
Application number: PCT/EP2023/065554
Authority: WO
Inventors: Fadri FURRER; Timothy Sandy; Julian Schwarz; Matthias Helmreich
Original assignee: Instructive Construction Ag
Priority date: 2022-06-10
Filing date: 2023-06-09
Publication date: 2023-12-14

Abstract

A method of building a target structure from a plurality of structural elements that are to be assembled is disclosed. The method comprises providing an electronic model of the target structure; providing at least one spatial orientation mark and/or at least one reference structure at a construction site, the at least one spatial orientation mark and/or the at least one reference structure being associated with a point of reference in the model; using an augmented reality device to recognize the spatial orientation mark and/or the reference structure to determine the spatial relationship of the augmented reality device with respect to the spatial orientation mark and/or the reference structure and/or the model; providing a first building instruction at the construction site by a displaying device; and providing a second building instruction on the augmented reality device. The first and/or the second building instruction serves to guide a builder to carry out a building step for assembling one or more of the plurality of structural elements to each other.

Description

METHOD OF BUILDI NG A TARGET STRUCTURE, SYSTEM AND COMPUTER PROGRAM THEREFOR

The present invention generally relates to a method of building a target structure from a plu rality of structural elements, and particularly to a method of providing building instructions to guide a builder to assemble the structural elements.

There is a strong trend in the construction industry to use three-dimensional (3D) designs of buildings, furniture or other structures. In some cases, these 3D models are even required by law, because structures designed in 3D tend to cause fewer errors on the construction site, as many problems are already visible in the 3D model. Nowadays, however, these structures are still built/assembled using two-dimensional (2D) plans, as that is what most of the builders are used to and is convenient to have at assembly stations or onsite. Creating 2D plans from 3D models reduces the amount of information though, which again leads to ambiguities as regards how to assemble the structures. To avoid having important details missing, planners often spend a substantial amount of their time in supplementing the 2D plans with information that was lost in the dimension reduction as well as in making sure that the relevant measurements are indicated.

To show this information, the use of a projector is known, e.g. from WO 2018/229769 Al. However, projectors need to have surfaces onto which they can project information, and projectors need to be installed in places where they have a direct line-of-sight to the projection area.

Chu et al., Integrating mobile Building Information Modelling and Augmented Reality systems: An experimental study, Automation in Construction 85 (2018) 305-316, describe a different approach that employs an augmented reality (AR) system. AR devices can display the information from the viewpoint set by the user by moving the device around in a 3D space. AR devices are also able to render information in an empty space. In addition to these benefits, however, AR devices are usually provided as hand-held devices or head-mounted-displays the use of which constrains a user's motion. On top of that, conventional AR systems can display information only at roughly the correct location but lack the ability to give a user feedback on his performed actions and to accurately display the information at the correct location.

To address the above issues, the present application provides the invention in accordance with the independent claims. The dependent claims recite preferred embodiments.

In one aspect, the invention relates to a method of building a target structure. There is no particular limitation on what the target structure may be, so long as building the same involves assembling a plurality of structural elements. In the sense of the target structure being built up by putting the structural elements together, the structural elements can also be referred to as building elements in the context of this application. Preferably, the target structure may be any of construction works, furniture, a framework, an electrical installation, a piping system, puzzles, toy block assemblies, tiles, sanitation installations, and/or workpiece assemblies.

The method of the present invention comprises a step of providing an electronic model of the target structure. Preferably, the electronic model is a three-dimensional (3D) model of the target structure. The 3D model preferably is a 3D computer-aided design (CAD) model but may otherwise be any other 3D model established by available tools. The electronic model may contain further information other than the target structure. For example, the electronic model may contain information required by other steps of the method of the present invention, such as the step of determining a spatial relationship between an augmented reality device and a spatial orientation mark (described below). For this step the electronic model preferably contains the corresponding information of the spatial orientation mark. As will be described below, it is possible that an additional spatial orientation mark is not provided, and the spatial position of the augmented reality device is indicated by, for example, a reference structure.

The method of the present invention further comprises a step of providing at least one spatial orientation mark and/or at least one reference structure at a construction site. The at least one spatial orientation mark and/or the at least one reference structure is preferably associated with a point of reference in the electronic model of the target structure. The method further comprises providing an augmented reality device which is configured to recognize the at least one spatial orientation mark and/or the at least one reference structure. Preferably, the at least one spatial orientation mark and/or the at least one reference structure is recognizable by the augmented reality device to allow the augmented reality device to determine its spatial relationship with respect to the at least one spatial orientation mark and/or the at least one reference structure, and to preferably correlate the at least one spatial orientation mark and/or the at least one reference structure with the corresponding information in the electronic model. The above-mentioned recognition may be realized by any available algorithm.

Preferably, the spatial orientation mark(s) comprise an optical label such as a QR code, a barcode or a predetermined pattern.

The term "reference structure" herein refers to any 3D structure that is recognizable by the augmented reality device to allow the augmented reality device to determine its spatial relationship with respect to the reference structure. Preferably, a reference structure may be a predetermined physical entity at the construction site, such as a table on which the target structure is to be built, a pre-built part of the target structure (e.g., a structure formed by some of the plurality of structural elements used to build the target structure), a calibration rig, etc.

Preferably, the augmented reality device is portable, and is preferably in the form of a tablet, a smartphone, or a goggle such as an AR/MR/VR goggle. The augmented reality device preferably comprises a screen on which an image captured by the augmented reality device may be displayed.

Preferably, the augmented reality device comprises a sensor for recognizing a two-dimensional image, such as a grayscale or an RGB image. The method of the present invention further comprises a step of providing a first building instruction at the construction site by a displaying device and providing a second building instruction on the augmented reality device, wherein the first and/or the second building instruction serves to guide a builder to carry out a building step for assembling one or more of the plurality of structural elements to each other.

In accordance with the present invention, the displaying device is used to provide building instructions from its position, preferably its fixed position, allowing the builder to perform his working steps with a similar perception of conventional two-dimensional building plans. Meanwhile, the augmented reality device is used to provide building instructions from dynamic localized viewpoints as desired by the builder. Therefore, the invention allows an intuitive, step- by-step guidance wherein building instructions are directly overlaid over the structural elements, in both reality and the augmented reality environment. The invention eliminates the need of - and thus avoids the above-mentioned issues about - creating multiple two-dimensional plans based on the electronic model of the target structure.

Preferably, the displaying device comprises a projector, a laser emitting device, and/or a screen. Preferably, the displaying device is configured to provide the spatial orientation mark in addition to the building instruction. Preferably, the displaying device is stationary with respect to the construction site.

The displaying device may provide the first building instruction on different surfaces depending on the target structures to be built. Preferably, the first building instruction may be provided on a wall; alternatively or additionally, the first building instructions may be provided on an assembly surface on which the target structure is to be disposed, preferably on which the target structure is to be rested and/or attached. In a situation that one or more structural elements are already disposed/rested/attached on the assembly surface, the first building instruction may be provided on at least one structural element disposed/rested/attached on the assembly surface. Preferably, the displaying device comprises a projector positioned over the construction site, more preferably at a height of at least 2 meters, preferably 5 meters and even more preferably 8 meters, to the ground; alternatively or additionally, the displaying device comprises a projector or a screen positioned under an assembly surface on which the target structure is to be disposed, more preferably under an assembly surface on which the target structure is to be rested. The assembly surface may be transparent or translucent.

Preferably, providing the first building instruction comprises projecting one or more patterns. When more than one pattern is projected, said more than one pattern may be projected from different angles and/or may be projected onto different surfaces, such as for example a first surface and a second surface that are arranged at an angle to each other, e.g. an angle of at least 20°, at least 45° or at least 70° (e.g., a horizontal surface and a vertical surface). Said surfaces preferably are provided by the above-mentioned wall, the assembly surface and/or the structural element disposed on the assembly surface.

Providing the first building instruction may comprise projecting a two-dimensional plan derived from the electronic model or a predetermined view of the electronic model on the wall, on the assembly surface and/or on the at least one structural element disposed on the assembly surface. The two-dimensional plan and/or the predetermined view of the electronic model preferably corresponds to edge(s), corner(s) and/or surface(s) of structural element(s) to be assembled, so that the builder is guided by the first building instruction to place the structural element(s) in their correction position.

Preferably, the predetermined view of the electronic model is a predetermined perspective view of the electronic model or a selected part of the electronic model. Particularly, a selected part of the electronic model comprises a plurality of edges in the electronic model, wherein the plurality of edges are not coplanar. Alternatively or additionally, the selected part of the electronic model may comprise a plurality of surfaces in the electronic model, wherein the plurality of surfaces are not coplanar. More preferably, a selected part of the electronic model may comprise a first surface and a second surface, e.g., a side surface and a top surface, of a model of a specific structural element in the electronic model.

Projecting such a selected part of the electronic model may comprise displaying the first building instruction by first rays of light and second rays of light, the first rays of light forming a first projection on and matching a corresponding first (e.g., side) surface of the specific structural element and the second rays of light forming a second projection on and matching a corresponding second (e.g., top) surface of the specific structural element, when the specific structural element is placed in its correct position. Accordingly, the first building instruction preferably provides guidance not only as to where the specific structural element should be placed but also the size (e.g., the exact size) and shape of the second (e.g., top) surface of the specific structural element.

Alternatively or additionally, the first building instruction preferably comprises a textual description and/or visible indications showing the builder how to place and/or assemble the one or more structural elements. More preferably, the first building instruction comprises visible marks such as lines, points, circles and/or crosses indicating positions at which the one or more structural elements are to be placed and/or assembled.

The step of providing the second building instruction on the augmented reality device is further described below. Preferably, providing the second building instruction comprises capturing with the augmented reality device a plurality of images, which preferably constitute a video, wherein the construction site and optionally one or more assembled structural elements are shown in the plurality of images. Preferably, the plurality of images is captured while moving the augmented reality device. The images may be depth images, RGB images and/or a scan, e.g. provided by a LIDAR (Light Detection and Ranging) system.

Preferably, providing the second building instruction further comprises displaying said plurality of images, preferably the video, on the augmented reality device. Particularly, the second building instruction is preferably provided in form of one or more augmented reality indications, more preferably one or more augmented reality visual elements, in one or more of the images (e.g., virtual symbols added to the images displayed).

Preferably, the one or more augmented reality indications comprise any one or a combination of the following: a) an outline or a rendering of the one or more structural elements to be placed and/or assembled, preferably wherein the outline or rendering shows how the one or more structural elements would appear at the construction site if positioned and/or assembled correctly; b) one or more indicators pointing to where a structural element should be positioned, such as a structural element for fixing two or more other structural elements to each other; c) a textual description, at least one visual or at least one animation showing how to place and/or assemble the one or more structural elements.

Preferably, the method of the present invention further comprises a step in which the one or more assembled structural elements in the plurality of images captured by the augmented reality device are recognized in order to determine a movement and/or a spatial relationship of the augmented reality device with respect to the assembled structural elements. Preferably, recognition of the one or more assembled structural elements in the images is performed by the augmented reality device or a control unit (e.g., a computer), preferably a control unit (e.g., computer) configured to control the displaying device. Particularly, one or more visual elements, preferably one or more edges, corners, surface textures or a combination thereof, of the assembled structural elements in the one or more images captured by the augmented reality device are recognized in order to determine the movement and/or the spatial relationship of the augmented reality device with respect to the assembled structural elements. Accordingly, the spatial relationship of the augmented reality device with respect to the assembled structural elements can be more accurately determined, which may help to precisely provide the second building instruction on the augmented reality device.

The above-mentioned first/second building instructions preferably are associated with a specific building step which, in combination with other building steps, constitutes a building procedure based on which the target structure may be built. Preferably, the building procedure is predetermined, or it is derived from the electronic model of the target structure. To put it differently, the method of the present invention may further comprise a step of deriving a building procedure from an electronic model of a target structure, either manually, semi- automatically or fully automatically. The building procedure comprises one or more sequential building steps, each of the one or more building steps being associated with one or more respective building instructions to be carried out by the builder.

Preferably, the electronic model, the building procedure and/or the building instructions are provided to and/or accessible by the displaying device and/or the augmented reality device.

Preferably, the method further comprises storing the electronic model, the building procedure and/or the one or more building instructions associated with a respective building step, preferably on the augmented reality device, in a remote server and/or in a control unit (e.g., computer) configured to control the displaying device.

Preferably, the method of the present invention further comprises a step of providing a third building instruction by the displaying device and/or providing a fourth building instruction on the augmented reality device, wherein the third and the fourth building instructions are associated with a subsequent building step which follows the previously indicated building step (that corresponds to the first/second building instructions) in the building procedure. Accordingly, the method of the present invention may guide a builder step by step through every stage of the building procedure, allowing the target structure to be built.

Preferably, the method of the present invention further comprises synchronizing the displaying device and the augmented reality device such that they respectively provide building instructions associated with the same building step in the building procedure.

In the course of building the target structure, the method preferably further comprises a step of inspecting the construction site with a progress tracking device, wherein the progress tracking device is configured to recognize one or more assembled structural elements and to correlate the recognized one or more assembled structural elements to a specific building step in the building procedure. Preferably, the progress tracking device is configured to recognize one or more individual structural elements of the assembled structural element(s) to determine a spatial relationship of the progress tracking device with respect to the assembled structural elements.

Preferably, the progress tracking device comprises a camera, e.g., a camera which is stationary or mobile with respect to the construction site. More preferably, the camera is the camera of the augmented reality device, i.e., the augmented reality device is configured to be used as the progress tracking device. The camera may be moved with the augmented reality device in this case.

Preferably, the progress tracking device is in communication with the displaying device, the augmented reality device, and/or the control unit so that one or more of these are capable of recognizing that a certain building step is carried out and, optionally, proceeding to a subsequent building step automatically.

The method may allow the builder to go through the steps of the building procedure manually (e.g., as a series of steps displayed on the augmented reality device and/or by the displaying device), e.g., by using a corresponding function on the augmented reality device or a separate remote control. Accordingly, the progress of construction of the target structure can be viewed and checked, and building statistics, e.g., how long each building step takes, can be logged. The statistics can be used for process optimization and timing of individual building steps, including steps performed by different project partners (as is often the case in larger construction projects where certain tasks can only be performed if others have already been executed).

When a projector is used as the displaying device, the method of the present invention preferably further comprises a step of calibrating the projector, preferably by using the augmented reality device. Particularly, the method of calibrating the projector may comprise step a) of projecting a pattern by the projector on a surface, preferably on an assembly surface on which the target structure is to be disposed or arranged. Preferably, the pattern for calibration in step a) forms the above- mentioned spatial orientation mark.

Calibrating the projector may further comprise step b) of capturing, by a camera, at least one image showing a projected shape of the projected pattern. Preferably, the camera is configured to recognize the projected pattern to determine the spatial relationship of the camera with respect to the projected pattern. This preferably includes the position and/or the orientation of the camera with respect to the pattern. Alternatively, the spatial relationship of the camera with respect to the projected pattern may be determined by other means (e.g., by one or more further cameras and/or sensors).

Preferably, said camera is a camera of the augmented reality device. Therefore, the recognition mentioned herein may be the same as the recognition described in the previous context of recognizing the spatial orientation mark.

Calibrating the projector may further comprise step c) of calculating an ideal shape of the projected pattern as seen from the position from where the image was taken. An ideal shape of the projected pattern may be determined based on, e.g., the determined spatial relationship of the camera for calibration with respect to the projected pattern, e.g. by the above-mentioned recognition of the projected pattern. The spatial relationship may be determined by a device carrying the camera (e.g., augmented reality device) and/or the control unit. Alternatively, the position of the camera and that of the projector may be predetermined by other means and the relevant position information may be accessible by the device carrying the camera, a control unit (e.g. computer) controlling the same and/or a server in communication therewith.

Calibrating the projector may further comprise step d) of identifying one or more differences between the projected shape and the ideal shape. Said differences preferably comprise differences in size, shape, location, sharpness, and/or focus. The calibration may further comprise step e) of adjusting one or more parameters of the projector in accordance with the one or more identified differences, preferably in an attempt to reduce or eliminate the differences between the projected shape and the ideal shape. The term "parameter" refers to both extrinsic and intrinsic parameters of the projector and preferably comprises one or more of a position, an orientation, a focal length, coordinates of the principal point, distortion parameters and pixel skew of the projector.

To enhance the accuracy of the calibration and/or allow using simpler patterns/marks for calibration, the calibration may be based on a plurality of images. In this context, step b) of the calibration preferably comprises capturing a first image of the projected pattern from a first position and capturing a second image of the projected pattern from a second position, the first position being different from the second position. Step c) preferably comprises determining a first ideal shape of the projected pattern as seen from the first position and a second ideal shape of the projected pattern as seen from the second position. Step d) preferably comprises identifying one or more differences between a first projected shape of the projected pattern shown in the first image and the first ideal shape and identifying one or more differences between a second projected shape of the projected pattern shown in the second image and the second ideal shape. In this context, the method preferably further comprises a step of determining the position of the camera, preferably the camera of the augmented reality device, at the first and second positions, as described above. The first and second images may be extracted from a plurality of images, preferably a plurality of images taken from a video, in which the projected pattern is shown. Preferably, the images are captured by the camera of the augmented reality device, more preferably the images being captured while the augmented reality device is being moved.

As the skilled reader will recognize, the calibration described above is not limited to the specific context of using projector to provide building instructions. Thus, in another aspect, the invention relates to a method of calibrating a projector with a portable electronic device. The method comprises steps corresponding to the above-described steps for calibrating a projector. In particular, the method may comprise steps corresponding to the above-described steps a) to d). Preferably, the method comprises step a') of projecting a calibration pattern on a reference surface; step b') of capturing, with the portable electronic device, at least one image showing a projected shape of the projected calibration pattern from at least one position; step c') of calculating an ideal shape of the calibration pattern as seen from the at least one position; step d') of identifying one or more differences between the projected shape(s) in the at least one image and the ideal shape(s) of the projected calibration pattern as seen from the position(s) of said at least one image; and step e') of adjusting one or more parameters of the projector in accordance with the identified one or more differences. The adjustable parameters may be the same as those described above.

Preferably, step b') comprises capturing a first image showing a first projected shape of the projected calibration pattern from a first position and a second image showing a second projected shape of the projected calibration pattern from a second position, the second position being different from the first position. Preferably, step c') comprises calculating a first ideal shape of the calibration pattern as seen from the first position and calculating a second ideal shape of the calibration pattern as seen from the second position. Preferably, step d') comprises identifying one or more differences between the first projected shape and the first ideal shape and identifying one or more differences between the second projected shape and the second ideal shape.

In yet another aspect, the present invention relates to a system for assisting in building a target structure from a plurality of structural elements that are to be assembled. Preferably, the system is suitable for carrying out the above-described method of the first aspect of the invention.

The system comprises a displaying device, an augmented reality device and an electronic model of the target structure, preferably a three-dimensional (3D) model, the electronic model being accessible by both the displaying device and the augmented reality device. The displaying device is configured to provide a first building instruction at a construction site, and the augmented reality device is configured to display a second building instruction thereon, wherein the first and/or the second building instruction serves to guide a builder to carry out a building step for assembling one or more structural elements. It will be understood that the system is preferably configured to carry out the method described above.

Preferably, the system comprises a spatial orientation mark provided at the construction site and associated with a point of reference in the electronic model, the spatial orientation mark being readable by the augmented reality device to allow the augmented reality device to determine its spatial relationship with respect to the spatial orientation mark. Preferably, the spatial orientation mark is provided by the displaying device.

In a further aspect, the invention relates to a computer program for assisting in building a target structure from a plurality of structural elements that are to be assembled. The computer program is executable by an augmented reality device and/or by a control unit (e.g., a computer) connected with such augmented reality device. The augmented reality device and/or the control unit may be the same as described above in the context of the method and/or the system in accordance with the invention.

The program comprises instructions which, when the program is executed by the augmented reality device and/or the control unit, cause the augmented reality device to determine its spatial relationship with respect to a spatial orientation mark or a reference structure by recognizing the spatial orientation mark and/or the reference structure, and to display a building instruction for guiding a builder to carry out a building step for assembling one or more structural elements. The building instruction may be derivable from an electronic model (e.g. a 3D model) of the target structure.

The computer program may be configured to carry out the method in accordance with the invention, as described above.

In view of the above, the present invention eliminates the need for creating 2D plans from the electronic model of the target structure. Building instructions can be shown at target locations, on an assembly surface or overlaid over assembled structural element(s), and it is possible to only display those instructions relevant for the current building step. The invention combines the benefits of the always present, hands-free visual guidance provided by the displaying device with the more targeted and dynamic guidance provided by an augmented reality device. Errors can be avoided with the progress tracking functionalities which allow to track the state of the construction and may notify users of mistakes as they occur.

The general aspects of the invention being outlined above, specific, non-limiting embodiments of the invention will be described in detail with reference to the respective figures, in which

Fig. 1 schematically shows a system for assisting in building a target structure according to an embodiment of the invention;

Fig. 2 schematically shows a state in which a building instruction is provided by a projector before corresponding structure elements are assembled; and

Fig. 3 schematically shows a state in which the corresponding structure elements are assembled in accordance with the building instruction shown in Fig. 2.

Referring to Fig. 1, the system according to this embodiment is used to assist in building a target structure. In this embodiment, the target structure is to be built on a top surface of a platform 1, such as a working table. The target structure may also be built on any other appropriate surface provided in any other appropriate construction site, and the invention is not limited to the provision of the platform 1. Such other surface may be, for example, a wall (e.g., a wall on which an electrical installation is to be provided).

The system in this embodiment comprises two displaying devices provided in form of projectors 5 and 6. The number of the displaying device may also be less than or more than two, depending on the need, e.g. the size of the target structure. A displaying region of a plurality of displaying devices may at least partially overlap in order to better display larger structures and/or displaying from different angles. For example, the displaying area of the projectors 5 and 6 overlaps partially in Fig. 1.

The system further comprises at least one augmented reality device provided in form of a smartphone 3 and/or an AR goggle 4. The smartphone 3 and the AR goggle 4, respectively, are configured to - by executing a corresponding application - capture a video in which the construction site is shown.

Additionally, a spatial orientation mark 50 is provided on the top surface of the platform 1. The spatial orientation mark 50 is associated with a point of reference in a 3D model of the target structure accessible by the respective displaying devices and the augmented reality devices. The spatial orientation mark 50 is readable by the respective augmented reality devices, i.e., the smartphone 3 and the goggle 4, allowing the augmented reality devices to determine their respective spatial relationship with respect to the spatial orientation mark. Alternatively or additionally, the spatial orientation of the one or more augmented reality devices 3, 4 may be determined by one or more control units (not shown) that are operatively connected to the one or more augmented reality devices 3, 4.

This allows the augmented reality devices 3, 4 to associate the construction site (and the object(s) disposed there) captured in the video with the 3D model so that building instructions may be provided on the augmented devices. The spatial orientation of the one or more augmented reality devices 3, 4 with respect to the model may thus be determined.

The spatial orientation mark 50 may be provided by one of the projectors 5 and 6. Each projector 5, 6 may provide a respective orientation mark (not shown).

Fig. 1 shows a status in which two structural members 2 (in form of studs) have already been assembled in parallel on the platform 1. These structural members 2 may be assembled with the guidance provided by the respective displaying devices and/or the augmented reality devices in a similar manner as the guidance described below. These structural members 2, however, may also be assembled with reliance on traditional 2D plans or any other approaches. In this case, one or more of such pre-assembled structural elements 2 could be used instead of or in addition to the spatial orientation mark 50 in order to determine the spatial orientation of the one or more augmented reality devices 3, 4 with respect to the 3D model.

The next step in the building procedure to build the target structure is to insert several structural elements, i.e., the structural elements 7 (cf. Fig. 3) between the pair of structural elements 2 and to connect the structural elements 2 and 7 to form an integral structure. To this end, and in accordance with the 3D model of the target structure, the projector 5 provides - i.e. projects - a first building instruction 51 on the platform 1. The first building instruction 51 is a projection formed by rays of light emitted from the projector 5 and indicates where the structural elements 7 should be placed. Further details of the building instruction 51 are discussed later with reference to Figs. 2 and 3. While the building instruction provided by the projector 6 is not expressly described here, it will be appreciated that the projector 6 may provide similar building instruction as the projector 5, possibly to different locations.

Similarly, in accordance with the 3D model of the target structure, the smartphone 3 displays second building instructions 31 and 32 thereon. The second building instruction 31, provided in form of grid lines matching the edges of the structural elements 7 to be placed, also indicates where the structural elements 7 should be placed and further provides the information concerning the actual appearance of the structural elements 7 in reality, helping a builder to more easily find the correct structural element. Meanwhile, the smartphone 3 also displays a second building instruction 32, provided in form of one or more augmented reality elements - in Fig. 1 screws are shown - and at a position exactly corresponding to where the corresponding elements should be placed in reality (i.e., in Fig. 1, where the screws should be driven into the slabs for connecting the structural elements 2 to the structural elements 7).

While Fig. 1 does not expressly show the building instructions provided by the goggle 4, it will be appreciated that the goggle 4 may provide similar augmented reality guidance as the smartphone

3. Accordingly, the first and second building instructions serve to guide a builder to carry out a building step for assembling structural elements 7 with the assembled structural elements 2.

The smartphone provides the user with an option to proceed to the next building step once the building step shown in Fig. 1 is completed. Completion of the step may be checked by the user. Alternatively or additionally, a progress tracking device capable of recognizing the assembled structural elements in order to determine whether the specific building step has been carried out correctly may be employed. Once this is determined, the user may be provided with an option to proceed to the next building step. The smartphone 3 may be configured as the progress tracking device.

Some additional details concerning providing the first building instruction 51 by the projector 5 are further described as follows with reference to Figs. 2 and 3 which are simplified illustrations of the system of Fig. 1; among devices providing building instructions, only the projector 5 is shown.

Displaying the first building instruction may comprise projecting a view from the 3D model, in particular a view adjusted to the spatial position of the displaying device (e.g., projector 5, 6) with respect to the platform 1.

Fig. 2 shows a scenario where a selected part of the 3D model is projected onto the surface of the platform 1 to form the first building instruction. The selected part in this embodiment includes side surfaces and top surfaces of respective structural elements 7 in the 3D model. Different surfaces of the structural elements may be shown differently (e.g., with a different color or texture). For example, in Fig. 2, when the selected part is projected onto the platform 1, the resulting projection is composed of hatched areas 511 and black areas 512. The hatched areas 511 are created by first rays of light that will end up on the side surfaces of the respective one or more structural elements 7 when the structural elements 7 are placed at their correct positions (see Fig. 3, hatched areas 511'). The black areas 512, on the other hand, are created by second rays of light that will end up on the top surfaces of the respective structural elements 7, when the structural elements 7 are placed at their correct positions (see Fig. 3, black areas 512').

Referring to Figs. 2 and 3, it will be noted that one side of each hatched area 511 marks the bottom alignment of each structural element 7 on the platform 1. While placing a structural element 7, the corresponding black area 512, 512' (or more accurately the corresponding rays of light) helps identify where the top surface of that structural element 7 should end up.

The first building instruction 51 therefore provides more information as compared to simply projecting a 2D plan. With a 2D plan, the planner would need to make the projection fit the dimension of the structural element either at the bottom or the top of the structural element, and thus, the projection is only matched either before or after the structural element is placed (but not both).

More generally speaking, the building instruction displayed by the displaying device (e.g., projector) may be configured (a) to indicate a position at which the one or more structural elements 7 are to be placed (e.g., a line, frame or area) and/or (b) to overlap with one or more surfaces of the structural elements 7 to be assembled once they are placed correctly. In other words, in option (b), a projection may be displayed that is congruent and/or identical with one or more surfaces (e.g., a top surface and a side surface) of the structural elements 7 to be assembled once they are placed correctly. It is believed that such projection allows the user to more easily identify a correct or incorrect placement of the one or more structural elements 7. Moreover, by extracting such projection directly from the 3D model, the necessary manual interventions for creating the building instructions may be simplified and reduced.

The possibilities (a) and (b) can also be combined. For example, initially the position at which the one or more structural elements 7 are to be placed may be displayed. Once the progress tracking device identifies that a structural element 7 has been placed at or in proximity to the intended position, a projection congruent with a top surface and a side surface of the respective structural element 7 may be displayed, in particular a view from the 3D model. Fig. 3 shows the projection congruent with surfaces of the one or more structural elements 7 after the structural elements 7 have been placed, see the hatched areas 511' displayed on the side surfaces and the black areas 512' displayed on the top surfaces of the structural elements 7. It will be appreciated from the above description that the use of multiple projectors, as shown in Fig. 1, may be advantageous in this context for providing instructions surfaces of structural elements that are arranged at an angle from each other and/or that may be overlapped by other elements of the construction from certain viewing directions, since it is only possible to project onto surfaces that are in direct line of sight of the projector.

Various aspects and embodiments of the invention have been described for purposes of illustration. The invention, however, should not be unduly limited by any of the details of the above disclosure, as a skilled person in the art will appreciate that changes and modifications that do not contradict the principle of the invention may be made and are still covered by the invention. In particular, the invention shall not be construed to be limited to the embodiments described above with reference to the drawings. Rather, the scope of protection of the invention is determined solely by the appended claims and their equivalents.

The invention may be embodied as the following preferred aspects.

1. A method of building a target structure from a plurality of structural elements that are to be assembled, the method comprising: providing an electronic model of the target structure, preferably a three-dimensional (3D) model; providing at least one spatial orientation mark and/or at least one reference structure at a construction site, the at least one spatial orientation mark and/or the at least one reference structure being associated with a point of reference in the model; using an augmented reality device to recognize the spatial orientation mark and/or the reference structure to determine the spatial relationship of the augmented reality device with respect to the spatial orientation mark and/or the reference structure and/or the model; providing a first building instruction at the construction site by a displaying device; and providing a second building instruction on the augmented reality device, wherein the first and/or the second building instruction serves to guide a builder to carry out a building step for assembling one or more of the plurality of structural elements to each other.

2. The method of the preceding aspect, wherein the first building instruction is provided on a wall or on an assembly surface on which the target structure is to be disposed, preferably on which the target structure is to be rested, and/or on at least one structural element disposed on the assembly surface, preferably wherein providing the first building instruction comprises projecting one or more patterns on the wall, on the assembly surface and/or on the at least one structural element disposed on the assembly surface, wherein when more than one patterns are projected said more than one patterns are optionally projected from different angles, preferably wherein providing the first building instruction comprises projecting a two- dimensional plan or a predetermined view, preferably a predetermined perspective view of the electronic model (e.g., a perspective view of a certain part in or a certain portion of the electronic model) on the wall, on the assembly surface and/or on the at least one structural element disposed on the assembly surface.

3. The method of any preceding aspect, wherein the first building instruction comprises a textual description and/or visible indications showing the builder how to place and/or assemble the one or more structural elements, preferably wherein the first building instruction comprises visible marks (such as lines, points, circles and/or crosses) indicating positions at which the one or more structural elements are to be placed and/or assembled. 4. The method of any preceding aspect, wherein providing the second building instruction comprises capturing with the augmented reality device a plurality of images, preferably a video, in which the construction site and optionally one or more assembled structural elements are shown, preferably while moving the augmented reality device.

5. The method of the preceding aspect, wherein providing the second building instruction further comprises displaying said plurality of images on the augmented reality device.

6. The method of the preceding aspect, wherein the second building instruction is provided in the form of one or more augmented reality indications, more preferably one or more augmented reality visual elements, in one or more of the images.

7. The method of the preceding aspect, the one or more augmented reality indications being: an outline or a rendering of the one or more structural elements to be placed and/or assembled, preferably wherein the outline or rendering shows how the one or more structural elements would appear at the construction site if positioned and/or assembled correctly; and/or one or more indicators pointing to where a structural element should be positioned, such as a structural element for fixing two or more other structural elements to each other; and/or a textual description, at least one visual or at least one animation showing how to place and/or assemble the one or more structural elements.

8. The method of any preceding aspect, wherein the electronic model is the 3D model, preferably a 3D computer-aided design (CAD) model.

9. The method of any preceding aspect, wherein the method further includes: providing a building procedure comprising one or more sequential building steps, wherein each of the one or more building steps is associated with one or more respective building instructions to be carried out by the builder, preferably wherein the building procedure is predetermined or derived from the model.

10. The method of the preceding aspect, further comprising: inspecting the construction site with a progress tracking device, wherein the progress tracking device is configured to recognize one or more assembled structural elements and to correlate the recognized one or more assembled structural elements to a specific building step in the building procedure.

11. The method of the preceding aspect, wherein the progress tracking device comprises a camera, preferably a camera of the augmented reality device or an additional camera, wherein the additional camera preferably is stationary.

12. The method of any of the two preceding aspects, wherein the augmented reality device is configured to be used as the progress tracking device.

13. The method of any of the four preceding aspects, further comprising: providing a third building instruction by the displaying device; and/or providing a fourth building instruction on the augmented reality device, wherein the third and the fourth building instructions are associated with a subsequent building step which follows the specific building step in the building procedure.

14. The method of any of the four preceding aspects, wherein the progress tracking device is configured to recognize one or more individual structural elements of the assembled structural element(s) to determine a spatial relationship of the progress tracking device with respect to the assembled structural elements.

15. The method of any of the six preceding aspects, further comprising: synchronizing the displaying device and the augmented reality device such that they respectively provide building instructions associated with the same building step in the building procedure.

16. The method of any of the seven preceding aspects, further comprising: storing the model, the building procedure and/or the one or more building instructions associated with a respective building step, preferably on the augmented reality device, in a remote server, or in control unit (e.g., a computer) configured to control the displaying device.

17. The method of the preceding aspect, wherein the model, the building procedure and/or the building instructions are provided to and/or accessed by the displaying device and/or the augmented reality device.

18. The method of any of aspects 4-17, wherein the one or more assembled structural elements in the plurality of images captured by the augmented reality device are recognized in order to determine the movement and/or the spatial relationship of the augmented reality device with respect to the assembled structural elements, preferably wherein recognition of the one or more assembled structural elements in the images is performed by the augmented reality device or a control unit (e.g., a computer) computer configured to control the displaying device.

19. The method of the preceding aspect, wherein one or more visual elements, preferably one or more edges, corners, surface textures or a combination thereof, of the assembled structural elements in the one or more images captured by the augmented reality device are recognized in order to determine the movement and/or the spatial relationship of the augmented reality device with respect to the assembled structural elements.

20. The method of any preceding aspect, wherein the displaying device is stationary with respect to the construction site. 21. The method of any preceding aspect, wherein the displaying device is a projector, a laser emitting device, or a screen, preferably: the displaying device being a projector positioned over the construction site, or the displaying device being a projector or a screen positioned under an assembly surface on which the target structure is to be disposed, preferably under an assembly surface on which the target structure is to be rested, preferably wherein the assembly surface is transparent or translucent.

22. The method of the preceding aspect, the method further comprising: calibrating the projector, wherein calibrating the projector comprises: projecting a pattern by the projector on a surface, preferably on an assembly surface on which the target structure is to be disposed; capturing, by a camera, at least one image showing a projected shape of the projected pattern, preferably by a camera of the augmented reality device; calculating an ideal shape of the projected pattern as seen from the position from where the image was taken; identifying one or more differences between the projected shape and the ideal shape; and adjusting one or more parameters of the projector in accordance with the one or more identified differences, preferably the parameters comprising one or more of a position, an orientation, a focal length, coordinates of the principal point, distortion parameters and pixel skew of the projector.

23. The method of the preceding aspect, wherein capturing at least one image showing the shape of the projected pattern comprises capturing a first image of the projected pattern from a first position and capturing a second image of the projected pattern from a second position, the first position being different from the second position, wherein calculating an ideal shape of the projected pattern comprises determining a first ideal shape of the projected pattern as seen from the first position and a second ideal shape of the projected pattern as seen from the second position, and wherein identifying one or more differences comprises identifying one or more differences between a first projected shape of the projected pattern shown in the first image and the first ideal shape, and identifying one or more differences between a second projected shape of the projected pattern shown in the second image and the second ideal shape.

24. The method of the preceding aspect, wherein the method further comprises determining the position of the camera, preferably of the augmented reality device, at the first and second positions.

25. The method of any of the two preceding aspects, wherein the first and second images are extracted from a plurality of images, preferably a plurality of images taken from a video, in which the projected pattern is shown, preferably the images being captured by the augmented reality device, more preferably the images being captured while the augmented reality device is being moved.

26. The method of any of the five preceding aspects, wherein the pattern forms the spatial orientation mark.

27. The method of any preceding aspect, wherein the spatial orientation mark comprises an optical label such as a QR code, a barcode, or a predetermined optical pattern.

28. The method of any preceding aspect, wherein the spatial orientation mark is provided by the displaying device.

29. The method of any preceding aspect, wherein the augmented reality device is portable, and is preferably a tablet, a smartphone, or a goggle, such as an AR/MR/VR goggle. 30. A system for assisting in building a target structure from a plurality of structural elements that are to be assembled, the system comprising: a displaying device configured to provide a first building instruction at a construction site; and an augmented reality device configured to display a second building instruction, an electronic model of the target structure, preferably a three-dimensional (3D) model; a spatial orientation mark and/or at least one reference structure provided at the construction site and associated with a point of reference in the electronic model, the spatial orientation mark being recognizable via the augmented reality device to allow the augmented reality device to determine its spatial relationship with respect to the spatial orientation mark, preferably wherein the displaying device is configured to display said spatial orientation mark, wherein the first and/or the second building instruction serves to guide a builder to carry out a building step for assembling one or more structural elements.

31. A computer program for assisting in building a target structure from a plurality of structural elements that are to be assembled, the computer program being executable by an augmented reality device and/or by a computer connected with such augmented reality device, the program comprising instructions which, when the program is executed by the augmented reality device, cause: determining a spatial relationship of the augmented reality device with respect to a spatial orientation mark or a reference structure by recognizing the spatial orientation mark or the reference structure; and displaying a building instruction for guiding a builder to carry out a building step for assembling one or more structural elements, preferably displaying a first building instruction by means of a displaying device and a second building instruction on the augmented reality device.

32. A method of calibrating a projector with a portable electronic device, the method comprising: projecting a calibration pattern on a reference surface; capturing, with the portable electronic device, at least one image showing a projected shape of the projected calibration pattern from at least one position, preferably capturing a first image showing a first projected shape of the projected calibration pattern from a first position and a second image showing a second projected shape of the projected calibration pattern from a second position, the second position being different from the first position; calculating an ideal shape of the calibration pattern as seen from the at least one position, preferably calculating a first ideal shape of the calibration pattern as seen from the first position and calculating a second ideal shape of the calibration pattern as seen from the second position; identifying one or more differences between the projected shape in the at least one image and the ideal shape of the projected calibration pattern as seen from the position of said at least one image, preferably identifying one or more differences between the first projected shape and the first ideal shape and identifying one or more differences between the second projected shape and the second ideal shape; and adjusting one or more parameters of the projector in accordance with the identified one or more differences, preferably the parameters comprising one or more of a position, an orientation, a focal length, coordinates of the principal point, distortion parameters and pixel skew of the projector.

33. The method of any of aspects 1-29, wherein the augmented reality device comprises a sensor for recognizing a two-dimensional image, such as a grayscale or an RGB image.

Claims

1. A method of building a target structure from a plurality of structural elements that are to be assembled, the method comprising: providing at least one electronic model of the target structure, preferably at least one three-dimensional (3D) model; providing at least one spatial orientation mark and/or at least one reference structure at a construction site, the at least one spatial orientation mark and/or the at least one reference structure being associated with a point of reference in the model; using at least one augmented reality device to recognize the spatial orientation mark and/or the reference structure to determine the spatial relationship of the augmented reality device with respect to the spatial orientation mark and/or the reference structure; providing at least one first building instruction at the construction site by at least one displaying device; and providing at least one second building instruction on the augmented reality device, the second building instruction being provided in the form of one or more augmented reality indications, wherein the first and the second building instructions serve to guide a builder to carry out building steps for assembling one or more of the plurality of structural elements to each other, wherein the first building instruction is provided on a wall, on an assembly surface on which the target structure is to be disposed and/or on at least one structural element disposed on the assembly surface.

2. The method of the preceding claim, wherein providing the first building instruction comprises projecting one or more patterns on the wall, on the assembly surface and/or on the at least one structural element disposed on the assembly surface, wherein when more than one patterns are projected, said more than one patterns are optionally projected from different angles, preferably wherein providing the first building instruction comprises projecting a two- dimensional plan or a predetermined view, preferably a predetermined perspective view or a selected part, of the electronic model on the wall, on the assembly surface and/or on the at least one structural element disposed on the assembly surface.

3. The method of any preceding claim, wherein the first building instruction comprises a textual description and/or visible indications showing the builder how to place and/or assemble the one or more structural elements, preferably wherein the first building instruction comprises visible marks, such as lines, points, circles and/or crosses, indicating positions at which the one or more structural elements are to be placed and/or assembled.

4. The method of any preceding claim, wherein providing the second building instruction comprises capturing with the augmented reality device a plurality of images, preferably a video, in which the construction site and optionally one or more assembled structural elements are shown, while moving the augmented reality device.

5. The method of the preceding claim, wherein the second building instruction is provided in the form of one or more augmented reality visual elements in one or more of the images, preferably, the one or more augmented reality indications being: an outline or a rendering of the one or more structural elements to be placed and/or assembled, preferably wherein the outline or rendering shows how the one or more structural elements would appear at the construction site if positioned and/or assembled correctly; and/or one or more indicators pointing to where a structural element should be positioned, such as a structural element for fixing two or more other structural elements to each other; and/or a textual description, at least one visual or at least one animation showing how to place and/or assemble the one or more structural elements.

6. The method of any preceding claim, wherein the method further includes: providing a building procedure comprising one or more sequential building steps, wherein each of the one or more building steps is associated with one or more respective building instructions to be carried out by the builder, preferably wherein the building procedure is predetermined or derived from the model.

7. The method of the preceding claim, further comprising: inspecting the construction site with a progress tracking device, wherein the progress tracking device is configured to recognize one or more assembled structural elements and to correlate the recognized one or more assembled structural elements to a specific building step in the building procedure.

8. The method of the preceding claim, further comprising: synchronizing the displaying device and the augmented reality device such that they respectively provide building instructions associated with the same building step in the building procedure.

9. The method of any of claims 4-8, wherein the one or more assembled structural elements in the plurality of images captured by the augmented reality device are recognized in order to determine the movement and/or the spatial relationship of the augmented reality device with respect to the assembled structural elements, preferably wherein recognition of the one or more assembled structural elements in the images is performed by the augmented reality device or a computer configured to control the displaying device.

10. The method of the preceding claim, wherein one or more visual elements, preferably one or more edges, corners, surface textures or a combination thereof, of the assembled structural elements in the one or more images captured by the augmented reality device are recognized in order to determine the movement and/or the spatial relationship of the augmented reality device with respect to the assembled structural elements.

11. The method of any preceding claim, wherein the displaying device comprises a projector, the method further comprising: calibrating the projector, wherein calibrating the projector comprises: projecting a pattern by the projector on a surface, preferably on an assembly surface on which the target structure is to be disposed; capturing, by a camera, at least one image showing a projected shape of the projected pattern, preferably by a camera of the augmented reality device; calculating an ideal shape of the projected pattern as seen from the position from where the image was taken; identifying one or more differences between the projected shape and the ideal shape; and adjusting one or more parameters of the projector in accordance with the one or more identified differences, preferably the parameters comprising one or more of a position, an orientation, a focal length, coordinates of the principal point, distortion parameters and pixel skew of the projector.

12. The method of the preceding claim, wherein capturing at least one image showing the shape of the projected pattern comprises capturing a first image of the projected pattern from a first position and capturing a second image of the projected pattern from a second position, the first position being different from the second position, wherein calculating an ideal shape of the projected pattern comprises determining a first ideal shape of the projected pattern as seen from the first position and a second ideal shape of the projected pattern as seen from the second position, and wherein identifying one or more differences comprises identifying one or more differences between a first projected shape of the projected pattern shown in the first image and the first ideal shape, and identifying one or more differences between a second projected shape of the projected pattern shown in the second image and the second ideal shape.

13. A system for assisting in building a target structure from a plurality of structural elements that are to be assembled, the system comprising: a displaying device configured to provide a first building instruction at a construction site; and an augmented reality device configured to display a second building instruction, an electronic model of the target structure, preferably a three-dimensional (3D) model; a spatial orientation mark provided at the construction site and associated with a point of reference in the electronic model, the spatial orientation mark being readable by the augmented reality device to allow the augmented reality device to determine its spatial relationship with respect to the spatial orientation mark, preferably wherein the displaying device is configured to display said spatial orientation mark, wherein the first and/or the second building instruction serves to guide a builder to carry out a building step for assembling one or more structural elements.

14. A computer program for assisting in building a target structure from a plurality of structural elements that are to be assembled, the computer program being executable by an augmented reality device and/or by a computer connected with such augmented reality device, the program comprising instructions which, when the program is executed by the augmented reality device, cause the augmented reality device to determine its spatial relationship with respect to a spatial orientation mark or a reference structure by recognizing the spatial orientation mark or the reference structure; and displaying a building instruction for guiding a builder to carry out a building step for assembling one or more structural elements.

15. A method of calibrating a projector with a portable electronic device, the method comprising: projecting a calibration pattern on a reference surface; capturing, with the portable electronic device, at least one image showing a projected shape of the projected calibration pattern from at least one position, preferably capturing a first image showing a first projected shape of the projected calibration pattern from a first position and a second image showing a second projected shape of the projected calibration pattern from a second position, the second position being different from the first position; calculating an ideal shape of the calibration pattern as seen from the at least one position, preferably calculating a first ideal shape of the calibration pattern as seen from the first position and calculating a second ideal shape of the calibration pattern as seen from the second position; identifying one or more differences between the projected shape in the at least one image and the ideal shape of the projected calibration pattern as seen from the position of said at least one image, preferably identifying one or more differences between the first projected shape and the first ideal shape and identifying one or more differences between the second projected shape and the second ideal shape; and adjusting one or more parameters of the projector in accordance with the identified one or more differences, preferably the parameters comprising one or more of a position, an orientation, a focal length, coordinates of the principal point, distortion parameters and pixel skew of the projector.