WO2020113273A1 - Mixed reality visualisation system - Google Patents
Mixed reality visualisation system Download PDFInfo
- Publication number
- WO2020113273A1 WO2020113273A1 PCT/AU2019/051328 AU2019051328W WO2020113273A1 WO 2020113273 A1 WO2020113273 A1 WO 2020113273A1 AU 2019051328 W AU2019051328 W AU 2019051328W WO 2020113273 A1 WO2020113273 A1 WO 2020113273A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- site
- artefacts
- floorplan
- building
- plan
- Prior art date
Links
- 238000012800 visualization Methods 0.000 title description 3
- 238000000034 method Methods 0.000 claims abstract description 62
- 238000004873 anchoring Methods 0.000 claims abstract description 5
- 230000003416 augmentation Effects 0.000 claims description 9
- 238000003384 imaging method Methods 0.000 claims description 7
- 244000025254 Cannabis sativa Species 0.000 claims description 5
- 241000283070 Equus zebra Species 0.000 claims description 5
- 230000001010 compromised effect Effects 0.000 claims description 5
- 230000005611 electricity Effects 0.000 claims description 5
- 230000000007 visual effect Effects 0.000 claims description 3
- 230000000694 effects Effects 0.000 claims description 2
- 238000004091 panning Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 8
- 238000010276 construction Methods 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/10—Geometric CAD
- G06F30/13—Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/08—Construction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/16—Real estate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/04—Architectural design, interior design
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/20—Indexing scheme for editing of 3D models
- G06T2219/2004—Aligning objects, relative positioning of parts
Definitions
- the present invention relates to a mixed reality visualisation system and, more particularly, in one form to such a system which permits accurate placement of constructs shown on a two-dimensional site map plan on the actual site depicted on the site map plan.
- the system permits construction of a 3D rendered construct from a 2D representation.
- the concept is to superpose a virtual depiction of the construct on a video view of reality.
- the reality may be a construction site upon which it is intended to build a construct in the form of a house.
- a problem in implementation of such systems is inability to properly reference the virtual depiction of the construct against the reality.
- a related and separate problem can be difficulty in seeking to properly reference in real time or near to real time .
- a further related problem relates to constructing a virtual depiction in three dimensions from a two dimensional representation .
- a mixed reality environment also called an augmented reality environment, is one where an electronic image or video view of reality (for example, a user's present surrounds) has superposed onto it or integrated with it electronically generated imagery.
- the electronically generated imagery may represent a 3D rendering of a building.
- Site In this specification, a site is a location where a building may be constructed.
- a floorplan is the 2D plan, taken from above, of a building compromised of walls, floors, windows, doors, stairs and other. May also, but not necessarily contain elevation information which then in totality contains everything to construct a 3D model.
- Siteplan In this specification, a siteplan shows the 2D floorplan of the building to be constructed within the site boundaries and neighbouring sites and/or artefacts.
- an artefact is a distinguishing feature found in the 2D siteplan that may or may not be also found in the live video at a site. Artefacts could be trees, walls, roads, utility poles (electricity, telephony) , fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings.
- 3D model In this specification, a 3D model is the 3D model information comes from one of the following methods:
- the floorplan contains top down views, side views and a scale therefore the 3D model can be constructed
- the 3D model is supplied separately as a file containing the geometric information, by projecting this file in isometric projection (without perspective) from the top down the 2D floorplan is projected and this can be aligned with the existing 2D floorplan thereby aligning the 3D model with the 2D floorplan.
- the missing height information is calculated as the default or defacto ceiling and window height for that region e.g.
- the 3D model accurately represents the interior of the building but may not reflect exterior elements such as an angled roof.
- a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional structure when the three-dimensional structure is located in a predetermined environment comprising selecting a predetermined environment; the environment referenced against a location defined against accordance in three dimensions;
- the two-dimensional representation having reference artefacts included in it which are located to scale relative to the two-dimensional representation;
- the three-dimensional representation is referenced against the reference artefacts of the two- dimensional representation.
- a method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan including locating reference artefacts on a 2D representation of the building and site; utilizing the live video view to locate at least some of the reference artefacts on the site; anchoring the 3D model of the building against the at least some of the reference artefacts during the live video view thereby accurately aligning the 3D model of the building in 3D space with reference to the 2D siteplan .
- the site is the location where the building will be constructed.
- the 2D plan is taken from above of a building compromised of walls, floors, windows, doors, stairs and other.
- the 2D plan contains elevation
- the elevation information in totality contains data to construct a 3D model.
- the 2D site plan comprises a 2D floorplan of the building.
- the 2D floorplan of the building is constructed within site boundaries of the site.
- the 2D floorplan of the building is constructed within neighbouring sites of the site.
- the 2D floorplan of the building is constructed within one or more artefacts of the site.
- each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.
- each of the one or more artefacts could be in the form of trees, walls, roads, utility poles
- the one or more artefacts also exist in the live video view of the site.
- the one or more artefacts do not exist in the live video view of the site.
- floorplan and/or 3D model to a live video view of a site on which the floorplan has been depicted on a site map; the method comprising producing an isolated site view; and wherein the isolated site view is produced by removing the floorplan from the site map by looking for floorplan artefacts such as the door and window geometry and then expanding to include geometry with straight lines forming concave shapes.
- the live video view of the site comprises a video feed.
- said video feed is produced by a camera in a phone, tablet or computer by the user (or electronics) .
- Preferably said video feed is panning over a site area of the site detecting a horizontal plane of the site area .
- said method detects artefacts on the horizontal plane of the site area.
- the artefacts are detected from high contrast changes in video change that exists as the camera move; and wherein the high contrast changes are recognized as detected vertical objects; and wherein approximate shape and location of the detected vertical objects are concluded and calculated; and wherein the approximate shape and location of the detected vertical objects are included into a 2D view of the site map.
- Preferably said method overlays the 2D view on the isolated plan view; wherein the method conducts a correlation by correlating the detected vertical objects of the 2D view with the artefacts of the isolated plan view.
- the correlation is above a certain threshold, isolated plan view and the live video view of the site; and wherein the 3D model is projected to live video feed .
- said method enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.
- the one or more artefacts comprise a spacial coordinate.
- the spacial coordinate is a location on the earth surface.
- the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.
- the correlation is above a certain threshold, isolated plan view and the live video view of the site; and wherein the 3D model is projected to live video feed.
- [00043] Preferably enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.
- a siteplan may be electronically
- the external image data is derived from satellite imagery referenced against GPS or other like spacial coordinates .
- 2D overhead image of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via internet 212 to an augmentation server 213.
- the augmention server 213 receives a 2D plan 214 from an external source.
- the 2D plan 214 may be a 2D plan 214 of a building structure.
- the augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211 thereby to generate a mixed reality image.
- the resulting mixed reality image 216 may then be viewed via an imaging device 217.
- the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219.
- the augmentation server is accessed via a web link on a website.
- a 2D representation of a floorplan has heat map output superposed on the floorplan.
- the heat map represents virtual person time spent at locations within the mixed reality image.
- a mixed reality headset having a display which combines reality vision of site of a pre-determined
- the site is the location where a building will be constructed.
- the 2D plan is a view taken from above of a building compromised of walls, floors, windows, doors, stairs and other.
- the 2D plan contains elevation
- the elevation information in totality contains data to construct a 3D model.
- the 2D site plan comprises a 2D floorplan of the building.
- the 2D floorplan of the building is constructed within site boundaries of the site.
- the 2D floorplan of the building is constructed within neighbouring sites of the site.
- the 2D floorplan of the building is constructed within one or more artefacts of the site.
- each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.
- each of the one or more artefacts may be in the form of trees, walls, roads, utility poles
- the one or more artefacts also exist in the live video view of the site.
- the one or more artefacts do not exist in the live video view of the site.
- the one or more artefacts comprise a spacial coordinate.
- the spacial coordinate is a location on the earth surface.
- the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.
- Figure 1 is a schematic diagram illustrating a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional
- Figure 2 is a schematic diagram showing various channels to acquire at least 2D renderings from which to produce the three-dimensional structure according to any one of the preferred embodiments of the present invention.
- Figures 3A and 3B are schematic diagrams
- Figure 4 is a flowchart applicable to the
- Figure 5 is a schematic diagram of an alternative embodiment for creating a composite mixed reality image relying on third party sourced imagery.
- Figure 6 is schematic diagram of an embedded web link arrangement for access to embodiments of the present invention by means of a digital communications device.
- Figure 7 is a 2D representation of a floorplan showing heat map output superposed on the floorplan in
- Figure 1 is a schematic diagram illustrating a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional
- Figure 2 is a schematic diagram showing various channels to produce the three-dimensional structure according to any one of the preferred embodiments of the present
- Stage 1 remove the floor plan from the site map to produce an isolated siteplan view containing plan artefacts
- Stage 2 onsite pan camera to detect onsite
- Stage 3 create a combined 2D top down view of detected onsite artefacts
- Stage 4 Overlay the artefact 2D view on the
- Stage 5 match at least selected onsite artefacts to plan artefacts thereby to reference 3D model against plan artefacts .
- a siteplan may be electronically generated from a floorplan and external image data.
- the external image data may be derived from satellite imagery obtained from satellites 221 and referenced against GPS or other like spacial coordinates.
- a 2D overhead image 210A of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via spacial imagery server 220 over internet 212 to augmentation server 213.
- Augmentation server 213 receives a 2D plan 214 from an external source.
- the 2D plan 214 may be a 2D plan 214 of a building structure.
- Augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211.
- the resulting mixed reality image 216 may then be viewed via an imaging device 217.
- the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219.
- the imaging device is a mixed reality headset.
- Figure 6 is schematic diagram of an embedded web link arrangement for access to embodiments of the present invention by means of a digital communications device.
- Figure 7 is a 2D representation of a floorplan showing heat map output superposed on the floorplan in
- the heat map represents virtual person time spent at locations within the mixed reality image.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Computer Hardware Design (AREA)
- Geometry (AREA)
- Tourism & Hospitality (AREA)
- Strategic Management (AREA)
- Human Resources & Organizations (AREA)
- General Engineering & Computer Science (AREA)
- Economics (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Structural Engineering (AREA)
- Mathematical Optimization (AREA)
- Architecture (AREA)
- Health & Medical Sciences (AREA)
- Computational Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Pure & Applied Mathematics (AREA)
- Primary Health Care (AREA)
- Civil Engineering (AREA)
- Mathematical Analysis (AREA)
- Computer Graphics (AREA)
- Software Systems (AREA)
- Entrepreneurship & Innovation (AREA)
- Data Mining & Analysis (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Processing Or Creating Images (AREA)
Abstract
A method for accurately anchoring a floorplan and/or 3D model to a live video view of a site on which the floorplan has been depicted on a site map; the method comprising producing an isolated site view; and wherein the isolated site view is produced by removing the floorplan from the site map by looking for floorplan artefacts such as the door and window geometry and then expanding to include geometry with straight lines forming concave shapes. Also disclosed is a mixed reality headset having a display which combines reality vision of site of a pre-determined environment with a three-dimensional representation of a three-dimensional structure; the representation of the three-dimensional structure referenced against an artefact in the pre-determined environment.
Description
MIXED REALITY VISUALISATION SYSTEM
TECHNICAL FIELD
[0001] The present invention relates to a mixed reality visualisation system and, more particularly, in one form to such a system which permits accurate placement of constructs shown on a two-dimensional site map plan on the actual site depicted on the site map plan. In an alternative form the system permits construction of a 3D rendered construct from a 2D representation.
BACKGROUND
[0002] Mixed reality systems have begun to be used to assist users to better visualize how planned constructs may appear in reality.
[0003] Broadly the concept is to superpose a virtual depiction of the construct on a video view of reality. In some instances the reality may be a construction site upon which it is intended to build a construct in the form of a house.
[0004] A problem in implementation of such systems is inability to properly reference the virtual depiction of the construct against the reality.
[0005] A related and separate problem can be difficulty in seeking to properly reference in real time or near to real time .
[0006] A further related problem relates to constructing a virtual depiction in three dimensions from a two dimensional representation .
[0007] It is an object of the present invention to address or at least ameliorate some of the above disadvantages or provide a useful alternative.
Notes
[0008] The term "comprising" (and grammatical variations thereof) is used in this specification in the inclusive sense of "having" or "including", and not in the exclusive sense of "consisting only of".
[0009] The above discussion of the prior art in the
Background of the invention, is not an admission that any information discussed therein is citable prior art or part of the common general knowledge of persons skilled in the art in any country.
SUMMARY OF INVENTION
Definitions
[00010] Mixed Reality: A mixed reality environment, also called an augmented reality environment, is one where an electronic image or video view of reality (for example, a user's present surrounds) has superposed onto it or integrated with it electronically generated imagery. In one form, the electronically generated imagery may represent a 3D rendering of a building.
[00011] Site: In this specification, a site is a location where a building may be constructed.
[00012] Floorplan: In this specification, a floorplan is the 2D plan, taken from above, of a building compromised of walls, floors, windows, doors, stairs and other. May also, but not
necessarily contain elevation information which then in totality contains everything to construct a 3D model.
[00013] Siteplan: In this specification, a siteplan shows the 2D floorplan of the building to be constructed within the site boundaries and neighbouring sites and/or artefacts.
[00014] Artefact: In this specification, an artefact is a distinguishing feature found in the 2D siteplan that may or may not be also found in the live video at a site. Artefacts could be trees, walls, roads, utility poles (electricity, telephony) , fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings.
[00015] 3D model: In this specification, a 3D model is the 3D model information comes from one of the following methods:
The floorplan contains top down views, side views and a scale therefore the 3D model can be constructed
The 3D model is supplied separately as a file containing the geometric information, by projecting this file in isometric projection (without perspective) from the top down the 2D floorplan is projected and this can be aligned with the existing 2D floorplan thereby aligning the 3D model with the 2D floorplan.
In one form it may be derived from the 2D floorplan where the missing height information is calculated as the default or defacto ceiling and window height for that region e.g.
220 cm for Australia. In this case the 3D model accurately represents the interior of the building but may not reflect exterior elements such as an angled roof.
Accordingly in one broad form there is disclosed a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional
structure when the three-dimensional structure is located in a predetermined environment; said method comprising selecting a predetermined environment; the environment referenced against a location defined against accordance in three dimensions;
selecting a two-dimensional representation for placement in the environment;
the two-dimensional representation having reference artefacts included in it which are located to scale relative to the two-dimensional representation;
constructing a three-dimensional representation from the two-dimensional representation;
placing the three-dimensional representation into a mixed reality environment;
in substantially real time projecting the three- dimensional representation visually to a user as the user navigates through the predetermined environment superposed with a reality vision of the predetermined environment.
Preferably the three-dimensional representation is referenced against the reference artefacts of the two- dimensional representation.
[00016] In a further broad form of the present invention there is provided a method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan; said method including locating reference artefacts on a 2D representation of the building and site; utilizing the live video view to locate at least some of the reference artefacts on the site; anchoring the 3D model of the building against the at least some of the reference artefacts during the live video view thereby accurately aligning the 3D model of the building in 3D space with reference to the 2D siteplan .
[00017] Preferably the site is the location where the building will be constructed.
[00018] Preferably the 2D site plan of the building
comprises a floorplan in the form of a 2D plan.
[00019] Preferably the 2D plan is taken from above of a building compromised of walls, floors, windows, doors, stairs and other.
[00020] Preferably the 2D plan contains elevation
information; the elevation information in totality contains data to construct a 3D model.
[00021] Preferably the 2D site plan comprises a 2D floorplan of the building.
[00022] Preferably the 2D floorplan of the building is constructed within site boundaries of the site.
[00023] Preferably the 2D floorplan of the building is constructed within neighbouring sites of the site.
[00024] Preferably the 2D floorplan of the building is constructed within one or more artefacts of the site.
[00025] Preferably each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.
[00026] Preferably each of the one or more artefacts could be in the form of trees, walls, roads, utility poles
(electricity, telephony) , fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings .
[00027] Preferably the one or more artefacts also exist in the live video view of the site.
[00028] Preferably the one or more artefacts do not exist in the live video view of the site.
[00029] In yet another broad form of the present invention there is provided a method for accurately anchoring a
floorplan and/or 3D model to a live video view of a site on which the floorplan has been depicted on a site map; the method comprising producing an isolated site view; and wherein the isolated site view is produced by removing the floorplan from the site map by looking for floorplan artefacts such as the door and window geometry and then expanding to include geometry with straight lines forming concave shapes.
[00030] Preferably the live video view of the site comprises a video feed.
[00031] Preferably said video feed is produced by a camera in a phone, tablet or computer by the user (or electronics) .
[00032] Preferably said video feed is panning over a site area of the site detecting a horizontal plane of the site area .
[00033] Preferably said method detects artefacts on the horizontal plane of the site area.
[00034] Preferably the artefacts are detected from high contrast changes in video change that exists as the camera move; and wherein the high contrast changes are recognized as detected vertical objects; and wherein approximate shape and
location of the detected vertical objects are concluded and calculated; and wherein the approximate shape and location of the detected vertical objects are included into a 2D view of the site map.
[00035] Preferably said method overlays the 2D view on the isolated plan view; wherein the method conducts a correlation by correlating the detected vertical objects of the 2D view with the artefacts of the isolated plan view.
[00036] Preferably the correlation is above a certain threshold, isolated plan view and the live video view of the site; and wherein the 3D model is projected to live video feed .
[00037] Preferably said method enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.
[00038] Preferably the one or more artefacts comprise a spacial coordinate.
[00039] Preferably the spacial coordinate is a location on the earth surface.
[00040] Preferably the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.
[00041] Preferably overlays the 2D view on the isolated plan view; wherein the method conducts a correlation by correlating the detected vertical objects of the 2D view with the
artefacts of the isolated plan view.
[00042] Preferably the correlation is above a certain threshold, isolated plan view and the live video view of the
site; and wherein the 3D model is projected to live video feed.
[00043] Preferably enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.
[00044] Preferably a siteplan may be electronically
generated from a floorplan and external image data.
[00045] Preferably the external image data is derived from satellite imagery referenced against GPS or other like spacial coordinates .
[00046] Preferably 2D overhead image of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via internet 212 to an augmentation server 213.
[00047] Preferably the augmention server 213 receives a 2D plan 214 from an external source.
[00048] Preferably the 2D plan 214 may be a 2D plan 214 of a building structure.
[00049] Preferably the augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211 thereby to generate a mixed reality image.
[00050] Preferably the resulting mixed reality image 216 may then be viewed via an imaging device 217.
[00051] Preferably the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219.
[00052] Preferably the augmentation server is accessed via a web link on a website.
[00053] Preferably a 2D representation of a floorplan has heat map output superposed on the floorplan.
[00054] Preferably the heat map represents virtual person time spent at locations within the mixed reality image.
[00055] In a further broad form of the invention there is provided a mixed reality headset having a display which combines reality vision of site of a pre-determined
environment with a three-dimensional representation of a three-dimensional structure; the representation of the three- dimensional structure referenced against an artefact in the pre-determined environment.
[00056] Preferably the site is the location where a building will be constructed.
[00057] Preferably the 2D site plan of the building
comprises a floorplan in the form of a 2D plan.
[00058] Preferably the 2D plan is a view taken from above of a building compromised of walls, floors, windows, doors, stairs and other.
[00059] Preferably the 2D plan contains elevation
information; the elevation information in totality contains data to construct a 3D model.
[00060] Preferably the 2D site plan comprises a 2D floorplan of the building.
[00061] Preferably the 2D floorplan of the building is constructed within site boundaries of the site.
[00062] Preferably the 2D floorplan of the building is constructed within neighbouring sites of the site.
[00063] Preferably the 2D floorplan of the building is constructed within one or more artefacts of the site.
[00064] Preferably each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.
[00065] Preferably each of the one or more artefacts may be in the form of trees, walls, roads, utility poles
(electricity, telephony), fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings .
[00066] Preferably the one or more artefacts also exist in the live video view of the site.
[00067] Preferably the one or more artefacts do not exist in the live video view of the site.
[00068] Preferably the one or more artefacts comprise a spacial coordinate.
[00069] Preferably the spacial coordinate is a location on the earth surface.
[00070] Preferably the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.
[00071] In yet a further broad form of the invention there is provided media incorporating code stored therein which, when executed by a processor effects the method described above .
BRIEF DESCRIPTION OF DRAWINGS
[00072] Embodiments of the present invention will now be described with reference to the accompanying drawings wherein:
[00073] Figure 1 is a schematic diagram illustrating a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional
structure when the three-dimensional structure is located in a predetermined environment according to one preferred
embodiment of the present invention.
[00074] Figure 2 is a schematic diagram showing various channels to acquire at least 2D renderings from which to produce the three-dimensional structure according to any one of the preferred embodiments of the present invention.
[00075] Figures 3A and 3B are schematic diagrams
illustrating a method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D
siteplan according to one preferred embodiment of the present invention.
[00076] Figure 4 is a flowchart applicable to the
arrangement of Figure 3A, 3B.
[00077] Figure 5 is a schematic diagram of an alternative embodiment for creating a composite mixed reality image relying on third party sourced imagery.
[00078] Figure 6 is schematic diagram of an embedded web link arrangement for access to embodiments of the present invention by means of a digital communications device.
[00079] Figure 7 is a 2D representation of a floorplan showing heat map output superposed on the floorplan in
accordance with a further embodiment of the present invention.
DESCRIPTION OF EMBODIMENTS
First Preferred Embodiment
[00080] Figure 1 is a schematic diagram illustrating a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional
structure when the three-dimensional structure is located in a predetermined environment according to one preferred
embodiment of the present invention.
[00081] Figure 2 is a schematic diagram showing various channels to produce the three-dimensional structure according to any one of the preferred embodiments of the present
invention .
[00082]
Further Preferred Embodiment
[00083] With reference to Figures 3A and 3B, there is provided a method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan according to one preferred embodiment of the present invention .
[00084] Stage 1: remove the floor plan from the site map to produce an isolated siteplan view containing plan artefacts;
[00085] Stage 2: onsite pan camera to detect onsite
artefacts on the horizontal plan;
[00086] Stage 3: create a combined 2D top down view of detected onsite artefacts;
[00087] Stage 4: Overlay the artefact 2D view on the
isolated siteplan view and virtually shift in order to
correlate plan artefacts with onsite artefacts;
[00088] Stage 5: match at least selected onsite artefacts to plan artefacts thereby to reference 3D model against plan artefacts .
ELECTRONIC CREATION OF SITEPLAN
[00089] In a further preferred embodiment a siteplan may be electronically generated from a floorplan and external image data.
In one form, the external image data, as illustrated in Figure 5 may be derived from satellite imagery obtained from satellites 221 and referenced against GPS or other like spacial coordinates. As shown in Figure 5, a 2D overhead image 210A of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via spacial imagery server 220 over internet 212 to
augmentation server 213. Augmentation server 213 receives a 2D plan 214 from an external source. In one form, the 2D plan 214 may be a 2D plan 214 of a building structure. Augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211. The resulting mixed reality image 216 may then be viewed via an imaging device 217. In one form, the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219. In a particular form the imaging device is a mixed reality headset.
Embedded Web link
[00090] Figure 6 is schematic diagram of an embedded web link arrangement for access to embodiments of the present invention by means of a digital communications device.
Superposed heat map
[00091] Figure 7 is a 2D representation of a floorplan showing heat map output superposed on the floorplan in
accordance with a further embodiment of the present invention.
[00092] In a preferred form, the heat map represents virtual person time spent at locations within the mixed reality image.
INDUSTRIAL APPLICABILITY
[00093] The above described methodology can be applied to depict three-dimensional representations of constructs such as dwellings on the actual site for which they are intended. This may be accomplished whilst the dwelling is still in the design phase .
Claims
1. A method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional structure when the three-dimensional structure is located in a predetermined environment; said method comprising
selecting a predetermined environment; the environment referenced against a location defined in three dimensions;
selecting a two-dimensional representation for placement in the environment;
the two-dimensional representation having reference artefacts included in it which are located to scale relative to the two-dimensional representation;
constructing a three-dimensional representation from the two-dimensional representation;
placing the three-dimensional representation into a mixed reality environment;
in substantially real time projecting the three- dimensional representation visually to a user as the user navigates through the predetermined environment superposed with a reality vision or live video view of the predetermined environment;
the three-dimensional representation referenced against the reference artefacts of the two-dimensional representation.
2. A method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan; said method including locating reference artefacts on a 2D representation of the building and site; utilizing the live video view to locate at least some of the reference artefacts on the site; anchoring the 3D model of the building against the at least some of the reference artefacts during the live video view thereby accurately aligning the 3D model of the building in 3D space with reference to the 2D siteplan.
3. The method of claim 2 wherein the site is the location where the building will be constructed.
4. The method of claims 2 or 3 wherein the 2D site plan of the building comprises a floorplan in the form of a 2D plan.
5. The method of claim 4 wherein the 2D plan is a view taken from above of a building compromised of walls, floors,
windows, doors, stairs and other.
6. The method of claim 4 wherein the 2D plan contains elevation information; the elevation information in totality contains data to construct a 3D model.
7. The method of any one of claims 2 to 6 wherein the 2D site plan comprises a 2D floorplan of the building.
8. The method of claim 7 wherein the 2D floorplan of the building is constructed within site boundaries of the site.
9. The method of claim 7 wherein the 2D floorplan of the building is constructed within neighbouring sites of the site.
10. The method of claim 7 wherein the 2D floorplan of the building is constructed within one or more artefacts of the site .
11. The method of claim 10 wherein each of the one or more artefacts is a distinguishing feature found in the 2D
siteplan .
12. The method of claim 10 or 11 wherein each of the one or more artefacts may be in the form of trees, walls, roads, utility poles (electricity, telephony) , fences, hedges, grass edges, planted gardens, gates, inclines, man holes,
rivers/streams, zebra crossings.
13. The method of any one of claims 10 to 12 wherein the one or more artefacts also exist in the live video view of the site .
14. The method of any one of claims 10 to 12 wherein the one or more artefacts do not exist in the live video view of the site .
15. The method of any previous claim wherein the one or more artefacts comprise a spacial coordinate.
16. The method of claim 15 wherein the spacial coordinate is a location on the earth surface.
17. The method of claim 15 or 16 wherein the spacial
coordinate lies within a bounded planar surface to which the floor plan is to be applied.
18. A method for accurately anchoring a floorplan and/or 3D model to a live video view of a site on which the floorplan has been depicted on a site map; the method comprising
producing an isolated site view; and wherein the isolated site view is produced by removing the floorplan from the site map by looking for floorplan artefacts such as the door and window geometry and then expanding to include geometry with straight lines forming concave shapes.
19. The method of claim 18 wherein the live video view of the site comprises a video feed.
20. The method of claim 19 wherein said video feed is produced by a camera in a phone, tablet or computer by the user (or electronics) .
21. The method of claim 19 or 20 wherein said video feed is panning over a site area of the site detecting a horizontal plane of the site area.
22. The method of any one of claims 18 to 21 wherein detects artefacts on the horizontal plane of the site area.
23. The method of claim 22 wherein the artefacts are detected from high contrast changes in video change that exists as the camera move; and wherein the high contrast changes are
recognized as detected vertical objects; and wherein
approximate shape and location of the detected vertical objects are concluded and calculated; and wherein the
approximate shape and location of the detected vertical objects are included into a 2D view of the site map.
24. The method of claim 23 wherein overlays the 2D view on the isolated plan view; wherein the method conducts a
correlation by correlating the detected vertical objects of the 2D view with the artefacts of the isolated plan view.
25. The method of claim 18 wherein the correlation is above a certain threshold, isolated plan view and the live video view of the site; and wherein the 3D model is projected to live video feed.
26. The method of claim 25 wherein enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.
27. The method of any of claims 1 to 23 wherein a siteplan may be electronically generated from a floorplan and external image data.
28. The method of claim 24 wherein the external image data is derived from satellite imagery referenced against GPS or other like spacial coordinates.
29. The method of claim 25 wherein 2D overhead image of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via internet 212 to an augmentation server 213.
30. The method of claim 26 wherein the augmention server 213 receives a 2D plan 214 from an external source.
31. The method of claim 27 wherein the 2D plan 214 may be a 2D plan 214 of a building structure.
32. The method of claim 28 wherein the augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211 thereby to generate a mixed reality image.
33. The method of claim 29 wherein the resulting mixed reality image 216 may then be viewed via an imaging device 217.
34. The method of claim 30 wherein the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219.
35. The method of any one of claims 1 to 31 wherein the augmentation server is accessed via a web link on a website.
36. The method of any one of claims 1 to 32 wherein a 2D representation of a floorplan has heat map output superposed on the floorplan.
37. The method of claim 33 wherein the heat map represents virtual person time spent at locations within the mixed reality image.
38. A mixed reality headset having a display which combines reality vision of site of a pre-determined environment with a three-dimensional representation of a three-dimensional structure; the representation of the three-dimensional
structure referenced against an artefact in the pre-determined environment .
39. The mixed reality headset of claim 38 wherein the site is the location where a building will be constructed.
40. The mixed reality headset of claims 38 or 39 wherein the 2D site plan of the building comprises a floorplan in the form of a 2D plan.
41. The mixed reality headset any one of claims 38 to 40 wherein the 2D plan is a view taken from above of a building compromised of walls, floors, windows, doors, stairs and other .
42. The mixed reality headset of claim 41 wherein the 2D plan contains elevation information; the elevation information in totality contains data to construct a 3D model.
43. The mixed reality headset of any one of claims 38 to 42 wherein the 2D site plan comprises a 2D floorplan of the building .
44. The mixed reality headset of any one of claims 38 to 43 wherein the 2D floorplan of the building is constructed within site boundaries of the site.
45. The mixed reality headset of any one of claims 38 to 44 wherein the 2D floorplan of the building is constructed within neighbouring sites of the site.
46. The mixed reality headset of any one of claims 38 to 45 wherein the 2D floorplan of the building is constructed within one or more artefacts of the site.
47. The mixed reality headset of any one of claims 38 to 46 wherein each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.
48. The mixed reality headset of any one of claims 38 to 47 wherein each of the one or more artefacts may be in the form of trees, walls, roads, utility poles (electricity,
telephony), fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings.
49. The mixed reality headset of any one of claims 38 to 48 wherein the one or more artefacts also exist in the live video view of the site.
50. The mixed reality headset of any one of claims 38 to 49 wherein the one or more artefacts do not exist in the live video view of the site.
51. The mixed reality headset of any one of claims 38 to 50 wherein the one or more artefacts comprise a spacial
coordinate .
52. The mixed reality headset of claim 51 wherein the spacial coordinate is a location on the earth surface.
53. The mixed reality headset of claim 51 of 52 wherein the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.
54. Media incorporating code stored therein which, when executed by a processor effects the method of any of claims 1 to 37.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2018904613A AU2018904613A0 (en) | 2018-12-04 | Mixed reality visualisation system | |
AU2018904613 | 2018-12-04 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020113273A1 true WO2020113273A1 (en) | 2020-06-11 |
Family
ID=70974044
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/AU2019/051328 WO2020113273A1 (en) | 2018-12-04 | 2019-12-04 | Mixed reality visualisation system |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2020113273A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2022255546A1 (en) * | 2021-05-31 | 2022-12-08 | 주식회사 쓰리아이 | Virtual indoor space content providing method and server therefor |
CN116502948A (en) * | 2023-04-25 | 2023-07-28 | 中国地铁工程咨询有限责任公司 | Method and device for measuring passenger flow conveying capacity of facilities in station |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110191069A1 (en) * | 2008-06-30 | 2011-08-04 | Tripod Components Pty Ltd | System and method for designing a building |
US20110218777A1 (en) * | 2010-03-03 | 2011-09-08 | Honeywell International Inc. | System and method for generating a building information model |
US20170256097A1 (en) * | 2016-03-07 | 2017-09-07 | F3 & Associates | Local positioning system for augmented reality applications |
US20180082414A1 (en) * | 2016-09-21 | 2018-03-22 | Astralink Ltd. | Methods Circuits Assemblies Devices Systems Platforms and Functionally Associated Machine Executable Code for Computer Vision Assisted Construction Site Inspection |
-
2019
- 2019-12-04 WO PCT/AU2019/051328 patent/WO2020113273A1/en active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110191069A1 (en) * | 2008-06-30 | 2011-08-04 | Tripod Components Pty Ltd | System and method for designing a building |
US20110218777A1 (en) * | 2010-03-03 | 2011-09-08 | Honeywell International Inc. | System and method for generating a building information model |
US20170256097A1 (en) * | 2016-03-07 | 2017-09-07 | F3 & Associates | Local positioning system for augmented reality applications |
US20180082414A1 (en) * | 2016-09-21 | 2018-03-22 | Astralink Ltd. | Methods Circuits Assemblies Devices Systems Platforms and Functionally Associated Machine Executable Code for Computer Vision Assisted Construction Site Inspection |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2022255546A1 (en) * | 2021-05-31 | 2022-12-08 | 주식회사 쓰리아이 | Virtual indoor space content providing method and server therefor |
CN116502948A (en) * | 2023-04-25 | 2023-07-28 | 中国地铁工程咨询有限责任公司 | Method and device for measuring passenger flow conveying capacity of facilities in station |
CN116502948B (en) * | 2023-04-25 | 2024-01-19 | 中国地铁工程咨询有限责任公司 | Method and device for measuring passenger flow conveying capacity of facilities in station |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9619944B2 (en) | Coordinate geometry augmented reality process for internal elements concealed behind an external element | |
US20190272676A1 (en) | Local positioning system for augmented reality applications | |
Behringer | Registration for outdoor augmented reality applications using computer vision techniques and hybrid sensors | |
Sankar et al. | Capturing indoor scenes with smartphones | |
Neumann et al. | Augmented virtual environments (ave): Dynamic fusion of imagery and 3d models | |
EP2401703B1 (en) | System and method of indicating transition between street level images | |
JP2023505212A (en) | Systems and methods for modeling structures using point clouds derived from pairs of stereoscopic images | |
Themistocleous | Model reconstruction for 3d vizualization of cultural heritage sites using open data from social media: The case study of Soli, Cyprus | |
Côté et al. | Live mobile panoramic high accuracy augmented reality for engineering and construction | |
Stal et al. | Integrating geomatics in archaeological research at the site of Thorikos (Greece) | |
El-Hakim | Semi-automatic 3D reconstruction of occluded and unmarked surfaces from widely separated views | |
CN102647512A (en) | All-round display method of spatial information | |
US20030225513A1 (en) | Method and apparatus for providing multi-level blended display of arbitrary shaped textures in a geo-spatial context | |
WO2020113273A1 (en) | Mixed reality visualisation system | |
Kersten et al. | Architectural historical 4D documentation of the old-segeberg town house by photogrammetry, terrestrial laser scanning and historical analysis | |
Wither et al. | Using aerial photographs for improved mobile AR annotation | |
US9305401B1 (en) | Real-time 3-D video-security | |
KR20210054408A (en) | Apparatus and method for modeling using gis | |
KR20140031129A (en) | Virtual face generating method of omni-directional image using user interaction | |
Stal et al. | Highly detailed 3D modelling of Mayan cultural heritage using an UAV | |
El-Hakim et al. | 3D reconstruction of complex architectures from multiple data | |
Gonzo et al. | Photo-realistic 3D reconstruction of castles with multiple-sources image-based techniques | |
Cheng et al. | Texture mapping 3d planar models of indoor environments with noisy camera poses | |
Morandi et al. | Interactive past: from 3D reconstruction to augmented and virtual reality applied to archaeological heritage. The medieval site of Bastia St. Michele (Cavaion Veronese, Verona, Italy) | |
KR101663482B1 (en) | Positional information construction method using omnidirectional image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19893784 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19893784 Country of ref document: EP Kind code of ref document: A1 |