WO2020113273A1 - Mixed reality visualisation system - Google Patents

Mixed reality visualisation system Download PDF

Info

Publication number
WO2020113273A1
WO2020113273A1 PCT/AU2019/051328 AU2019051328W WO2020113273A1 WO 2020113273 A1 WO2020113273 A1 WO 2020113273A1 AU 2019051328 W AU2019051328 W AU 2019051328W WO 2020113273 A1 WO2020113273 A1 WO 2020113273A1
Authority
WO
WIPO (PCT)
Prior art keywords
site
artefacts
floorplan
building
plan
Prior art date
Application number
PCT/AU2019/051328
Other languages
French (fr)
Inventor
Daniel Swan
Keith Ahern
Original Assignee
Startinno Ventures Pty Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from AU2018904613A external-priority patent/AU2018904613A0/en
Application filed by Startinno Ventures Pty Ltd filed Critical Startinno Ventures Pty Ltd
Publication of WO2020113273A1 publication Critical patent/WO2020113273A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/10—Geometric CAD
    • G06F30/13—Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/08—Construction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G06Q50/16—Real estate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/04—Architectural design, interior design
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2004—Aligning objects, relative positioning of parts

Definitions

  • the present invention relates to a mixed reality visualisation system and, more particularly, in one form to such a system which permits accurate placement of constructs shown on a two-dimensional site map plan on the actual site depicted on the site map plan.
  • the system permits construction of a 3D rendered construct from a 2D representation.
  • the concept is to superpose a virtual depiction of the construct on a video view of reality.
  • the reality may be a construction site upon which it is intended to build a construct in the form of a house.
  • a problem in implementation of such systems is inability to properly reference the virtual depiction of the construct against the reality.
  • a related and separate problem can be difficulty in seeking to properly reference in real time or near to real time .
  • a further related problem relates to constructing a virtual depiction in three dimensions from a two dimensional representation .
  • a mixed reality environment also called an augmented reality environment, is one where an electronic image or video view of reality (for example, a user's present surrounds) has superposed onto it or integrated with it electronically generated imagery.
  • the electronically generated imagery may represent a 3D rendering of a building.
  • Site In this specification, a site is a location where a building may be constructed.
  • a floorplan is the 2D plan, taken from above, of a building compromised of walls, floors, windows, doors, stairs and other. May also, but not necessarily contain elevation information which then in totality contains everything to construct a 3D model.
  • Siteplan In this specification, a siteplan shows the 2D floorplan of the building to be constructed within the site boundaries and neighbouring sites and/or artefacts.
  • an artefact is a distinguishing feature found in the 2D siteplan that may or may not be also found in the live video at a site. Artefacts could be trees, walls, roads, utility poles (electricity, telephony) , fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings.
  • 3D model In this specification, a 3D model is the 3D model information comes from one of the following methods:
  • the floorplan contains top down views, side views and a scale therefore the 3D model can be constructed
  • the 3D model is supplied separately as a file containing the geometric information, by projecting this file in isometric projection (without perspective) from the top down the 2D floorplan is projected and this can be aligned with the existing 2D floorplan thereby aligning the 3D model with the 2D floorplan.
  • the missing height information is calculated as the default or defacto ceiling and window height for that region e.g.
  • the 3D model accurately represents the interior of the building but may not reflect exterior elements such as an angled roof.
  • a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional structure when the three-dimensional structure is located in a predetermined environment comprising selecting a predetermined environment; the environment referenced against a location defined against accordance in three dimensions;
  • the two-dimensional representation having reference artefacts included in it which are located to scale relative to the two-dimensional representation;
  • the three-dimensional representation is referenced against the reference artefacts of the two- dimensional representation.
  • a method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan including locating reference artefacts on a 2D representation of the building and site; utilizing the live video view to locate at least some of the reference artefacts on the site; anchoring the 3D model of the building against the at least some of the reference artefacts during the live video view thereby accurately aligning the 3D model of the building in 3D space with reference to the 2D siteplan .
  • the site is the location where the building will be constructed.
  • the 2D plan is taken from above of a building compromised of walls, floors, windows, doors, stairs and other.
  • the 2D plan contains elevation
  • the elevation information in totality contains data to construct a 3D model.
  • the 2D site plan comprises a 2D floorplan of the building.
  • the 2D floorplan of the building is constructed within site boundaries of the site.
  • the 2D floorplan of the building is constructed within neighbouring sites of the site.
  • the 2D floorplan of the building is constructed within one or more artefacts of the site.
  • each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.
  • each of the one or more artefacts could be in the form of trees, walls, roads, utility poles
  • the one or more artefacts also exist in the live video view of the site.
  • the one or more artefacts do not exist in the live video view of the site.
  • floorplan and/or 3D model to a live video view of a site on which the floorplan has been depicted on a site map; the method comprising producing an isolated site view; and wherein the isolated site view is produced by removing the floorplan from the site map by looking for floorplan artefacts such as the door and window geometry and then expanding to include geometry with straight lines forming concave shapes.
  • the live video view of the site comprises a video feed.
  • said video feed is produced by a camera in a phone, tablet or computer by the user (or electronics) .
  • Preferably said video feed is panning over a site area of the site detecting a horizontal plane of the site area .
  • said method detects artefacts on the horizontal plane of the site area.
  • the artefacts are detected from high contrast changes in video change that exists as the camera move; and wherein the high contrast changes are recognized as detected vertical objects; and wherein approximate shape and location of the detected vertical objects are concluded and calculated; and wherein the approximate shape and location of the detected vertical objects are included into a 2D view of the site map.
  • Preferably said method overlays the 2D view on the isolated plan view; wherein the method conducts a correlation by correlating the detected vertical objects of the 2D view with the artefacts of the isolated plan view.
  • the correlation is above a certain threshold, isolated plan view and the live video view of the site; and wherein the 3D model is projected to live video feed .
  • said method enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.
  • the one or more artefacts comprise a spacial coordinate.
  • the spacial coordinate is a location on the earth surface.
  • the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.
  • the correlation is above a certain threshold, isolated plan view and the live video view of the site; and wherein the 3D model is projected to live video feed.
  • [00043] Preferably enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.
  • a siteplan may be electronically
  • the external image data is derived from satellite imagery referenced against GPS or other like spacial coordinates .
  • 2D overhead image of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via internet 212 to an augmentation server 213.
  • the augmention server 213 receives a 2D plan 214 from an external source.
  • the 2D plan 214 may be a 2D plan 214 of a building structure.
  • the augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211 thereby to generate a mixed reality image.
  • the resulting mixed reality image 216 may then be viewed via an imaging device 217.
  • the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219.
  • the augmentation server is accessed via a web link on a website.
  • a 2D representation of a floorplan has heat map output superposed on the floorplan.
  • the heat map represents virtual person time spent at locations within the mixed reality image.
  • a mixed reality headset having a display which combines reality vision of site of a pre-determined
  • the site is the location where a building will be constructed.
  • the 2D plan is a view taken from above of a building compromised of walls, floors, windows, doors, stairs and other.
  • the 2D plan contains elevation
  • the elevation information in totality contains data to construct a 3D model.
  • the 2D site plan comprises a 2D floorplan of the building.
  • the 2D floorplan of the building is constructed within site boundaries of the site.
  • the 2D floorplan of the building is constructed within neighbouring sites of the site.
  • the 2D floorplan of the building is constructed within one or more artefacts of the site.
  • each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.
  • each of the one or more artefacts may be in the form of trees, walls, roads, utility poles
  • the one or more artefacts also exist in the live video view of the site.
  • the one or more artefacts do not exist in the live video view of the site.
  • the one or more artefacts comprise a spacial coordinate.
  • the spacial coordinate is a location on the earth surface.
  • the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.
  • Figure 1 is a schematic diagram illustrating a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional
  • Figure 2 is a schematic diagram showing various channels to acquire at least 2D renderings from which to produce the three-dimensional structure according to any one of the preferred embodiments of the present invention.
  • Figures 3A and 3B are schematic diagrams
  • Figure 4 is a flowchart applicable to the
  • Figure 5 is a schematic diagram of an alternative embodiment for creating a composite mixed reality image relying on third party sourced imagery.
  • Figure 6 is schematic diagram of an embedded web link arrangement for access to embodiments of the present invention by means of a digital communications device.
  • Figure 7 is a 2D representation of a floorplan showing heat map output superposed on the floorplan in
  • Figure 1 is a schematic diagram illustrating a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional
  • Figure 2 is a schematic diagram showing various channels to produce the three-dimensional structure according to any one of the preferred embodiments of the present
  • Stage 1 remove the floor plan from the site map to produce an isolated siteplan view containing plan artefacts
  • Stage 2 onsite pan camera to detect onsite
  • Stage 3 create a combined 2D top down view of detected onsite artefacts
  • Stage 4 Overlay the artefact 2D view on the
  • Stage 5 match at least selected onsite artefacts to plan artefacts thereby to reference 3D model against plan artefacts .
  • a siteplan may be electronically generated from a floorplan and external image data.
  • the external image data may be derived from satellite imagery obtained from satellites 221 and referenced against GPS or other like spacial coordinates.
  • a 2D overhead image 210A of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via spacial imagery server 220 over internet 212 to augmentation server 213.
  • Augmentation server 213 receives a 2D plan 214 from an external source.
  • the 2D plan 214 may be a 2D plan 214 of a building structure.
  • Augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211.
  • the resulting mixed reality image 216 may then be viewed via an imaging device 217.
  • the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219.
  • the imaging device is a mixed reality headset.
  • Figure 6 is schematic diagram of an embedded web link arrangement for access to embodiments of the present invention by means of a digital communications device.
  • Figure 7 is a 2D representation of a floorplan showing heat map output superposed on the floorplan in
  • the heat map represents virtual person time spent at locations within the mixed reality image.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Computer Hardware Design (AREA)
  • Geometry (AREA)
  • Tourism & Hospitality (AREA)
  • Strategic Management (AREA)
  • Human Resources & Organizations (AREA)
  • General Engineering & Computer Science (AREA)
  • Economics (AREA)
  • General Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Structural Engineering (AREA)
  • Mathematical Optimization (AREA)
  • Architecture (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Pure & Applied Mathematics (AREA)
  • Primary Health Care (AREA)
  • Civil Engineering (AREA)
  • Mathematical Analysis (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Data Mining & Analysis (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Processing Or Creating Images (AREA)

Abstract

A method for accurately anchoring a floorplan and/or 3D model to a live video view of a site on which the floorplan has been depicted on a site map; the method comprising producing an isolated site view; and wherein the isolated site view is produced by removing the floorplan from the site map by looking for floorplan artefacts such as the door and window geometry and then expanding to include geometry with straight lines forming concave shapes. Also disclosed is a mixed reality headset having a display which combines reality vision of site of a pre-determined environment with a three-dimensional representation of a three-dimensional structure; the representation of the three-dimensional structure referenced against an artefact in the pre-determined environment.

Description

MIXED REALITY VISUALISATION SYSTEM
TECHNICAL FIELD
[0001] The present invention relates to a mixed reality visualisation system and, more particularly, in one form to such a system which permits accurate placement of constructs shown on a two-dimensional site map plan on the actual site depicted on the site map plan. In an alternative form the system permits construction of a 3D rendered construct from a 2D representation.
BACKGROUND
[0002] Mixed reality systems have begun to be used to assist users to better visualize how planned constructs may appear in reality.
[0003] Broadly the concept is to superpose a virtual depiction of the construct on a video view of reality. In some instances the reality may be a construction site upon which it is intended to build a construct in the form of a house.
[0004] A problem in implementation of such systems is inability to properly reference the virtual depiction of the construct against the reality.
[0005] A related and separate problem can be difficulty in seeking to properly reference in real time or near to real time .
[0006] A further related problem relates to constructing a virtual depiction in three dimensions from a two dimensional representation . [0007] It is an object of the present invention to address or at least ameliorate some of the above disadvantages or provide a useful alternative.
Notes
[0008] The term "comprising" (and grammatical variations thereof) is used in this specification in the inclusive sense of "having" or "including", and not in the exclusive sense of "consisting only of".
[0009] The above discussion of the prior art in the
Background of the invention, is not an admission that any information discussed therein is citable prior art or part of the common general knowledge of persons skilled in the art in any country.
SUMMARY OF INVENTION
Definitions
[00010] Mixed Reality: A mixed reality environment, also called an augmented reality environment, is one where an electronic image or video view of reality (for example, a user's present surrounds) has superposed onto it or integrated with it electronically generated imagery. In one form, the electronically generated imagery may represent a 3D rendering of a building.
[00011] Site: In this specification, a site is a location where a building may be constructed.
[00012] Floorplan: In this specification, a floorplan is the 2D plan, taken from above, of a building compromised of walls, floors, windows, doors, stairs and other. May also, but not necessarily contain elevation information which then in totality contains everything to construct a 3D model.
[00013] Siteplan: In this specification, a siteplan shows the 2D floorplan of the building to be constructed within the site boundaries and neighbouring sites and/or artefacts.
[00014] Artefact: In this specification, an artefact is a distinguishing feature found in the 2D siteplan that may or may not be also found in the live video at a site. Artefacts could be trees, walls, roads, utility poles (electricity, telephony) , fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings.
[00015] 3D model: In this specification, a 3D model is the 3D model information comes from one of the following methods:
The floorplan contains top down views, side views and a scale therefore the 3D model can be constructed
The 3D model is supplied separately as a file containing the geometric information, by projecting this file in isometric projection (without perspective) from the top down the 2D floorplan is projected and this can be aligned with the existing 2D floorplan thereby aligning the 3D model with the 2D floorplan.
In one form it may be derived from the 2D floorplan where the missing height information is calculated as the default or defacto ceiling and window height for that region e.g.
220 cm for Australia. In this case the 3D model accurately represents the interior of the building but may not reflect exterior elements such as an angled roof.
Accordingly in one broad form there is disclosed a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional structure when the three-dimensional structure is located in a predetermined environment; said method comprising selecting a predetermined environment; the environment referenced against a location defined against accordance in three dimensions;
selecting a two-dimensional representation for placement in the environment;
the two-dimensional representation having reference artefacts included in it which are located to scale relative to the two-dimensional representation;
constructing a three-dimensional representation from the two-dimensional representation;
placing the three-dimensional representation into a mixed reality environment;
in substantially real time projecting the three- dimensional representation visually to a user as the user navigates through the predetermined environment superposed with a reality vision of the predetermined environment.
Preferably the three-dimensional representation is referenced against the reference artefacts of the two- dimensional representation.
[00016] In a further broad form of the present invention there is provided a method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan; said method including locating reference artefacts on a 2D representation of the building and site; utilizing the live video view to locate at least some of the reference artefacts on the site; anchoring the 3D model of the building against the at least some of the reference artefacts during the live video view thereby accurately aligning the 3D model of the building in 3D space with reference to the 2D siteplan . [00017] Preferably the site is the location where the building will be constructed.
[00018] Preferably the 2D site plan of the building
comprises a floorplan in the form of a 2D plan.
[00019] Preferably the 2D plan is taken from above of a building compromised of walls, floors, windows, doors, stairs and other.
[00020] Preferably the 2D plan contains elevation
information; the elevation information in totality contains data to construct a 3D model.
[00021] Preferably the 2D site plan comprises a 2D floorplan of the building.
[00022] Preferably the 2D floorplan of the building is constructed within site boundaries of the site.
[00023] Preferably the 2D floorplan of the building is constructed within neighbouring sites of the site.
[00024] Preferably the 2D floorplan of the building is constructed within one or more artefacts of the site.
[00025] Preferably each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.
[00026] Preferably each of the one or more artefacts could be in the form of trees, walls, roads, utility poles
(electricity, telephony) , fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings . [00027] Preferably the one or more artefacts also exist in the live video view of the site.
[00028] Preferably the one or more artefacts do not exist in the live video view of the site.
[00029] In yet another broad form of the present invention there is provided a method for accurately anchoring a
floorplan and/or 3D model to a live video view of a site on which the floorplan has been depicted on a site map; the method comprising producing an isolated site view; and wherein the isolated site view is produced by removing the floorplan from the site map by looking for floorplan artefacts such as the door and window geometry and then expanding to include geometry with straight lines forming concave shapes.
[00030] Preferably the live video view of the site comprises a video feed.
[00031] Preferably said video feed is produced by a camera in a phone, tablet or computer by the user (or electronics) .
[00032] Preferably said video feed is panning over a site area of the site detecting a horizontal plane of the site area .
[00033] Preferably said method detects artefacts on the horizontal plane of the site area.
[00034] Preferably the artefacts are detected from high contrast changes in video change that exists as the camera move; and wherein the high contrast changes are recognized as detected vertical objects; and wherein approximate shape and location of the detected vertical objects are concluded and calculated; and wherein the approximate shape and location of the detected vertical objects are included into a 2D view of the site map.
[00035] Preferably said method overlays the 2D view on the isolated plan view; wherein the method conducts a correlation by correlating the detected vertical objects of the 2D view with the artefacts of the isolated plan view.
[00036] Preferably the correlation is above a certain threshold, isolated plan view and the live video view of the site; and wherein the 3D model is projected to live video feed .
[00037] Preferably said method enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.
[00038] Preferably the one or more artefacts comprise a spacial coordinate.
[00039] Preferably the spacial coordinate is a location on the earth surface.
[00040] Preferably the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.
[00041] Preferably overlays the 2D view on the isolated plan view; wherein the method conducts a correlation by correlating the detected vertical objects of the 2D view with the
artefacts of the isolated plan view.
[00042] Preferably the correlation is above a certain threshold, isolated plan view and the live video view of the site; and wherein the 3D model is projected to live video feed.
[00043] Preferably enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.
[00044] Preferably a siteplan may be electronically
generated from a floorplan and external image data.
[00045] Preferably the external image data is derived from satellite imagery referenced against GPS or other like spacial coordinates .
[00046] Preferably 2D overhead image of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via internet 212 to an augmentation server 213.
[00047] Preferably the augmention server 213 receives a 2D plan 214 from an external source.
[00048] Preferably the 2D plan 214 may be a 2D plan 214 of a building structure.
[00049] Preferably the augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211 thereby to generate a mixed reality image.
[00050] Preferably the resulting mixed reality image 216 may then be viewed via an imaging device 217.
[00051] Preferably the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219.
[00052] Preferably the augmentation server is accessed via a web link on a website.
[00053] Preferably a 2D representation of a floorplan has heat map output superposed on the floorplan.
[00054] Preferably the heat map represents virtual person time spent at locations within the mixed reality image. [00055] In a further broad form of the invention there is provided a mixed reality headset having a display which combines reality vision of site of a pre-determined
environment with a three-dimensional representation of a three-dimensional structure; the representation of the three- dimensional structure referenced against an artefact in the pre-determined environment.
[00056] Preferably the site is the location where a building will be constructed.
[00057] Preferably the 2D site plan of the building
comprises a floorplan in the form of a 2D plan.
[00058] Preferably the 2D plan is a view taken from above of a building compromised of walls, floors, windows, doors, stairs and other.
[00059] Preferably the 2D plan contains elevation
information; the elevation information in totality contains data to construct a 3D model.
[00060] Preferably the 2D site plan comprises a 2D floorplan of the building.
[00061] Preferably the 2D floorplan of the building is constructed within site boundaries of the site.
[00062] Preferably the 2D floorplan of the building is constructed within neighbouring sites of the site.
[00063] Preferably the 2D floorplan of the building is constructed within one or more artefacts of the site.
[00064] Preferably each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.
[00065] Preferably each of the one or more artefacts may be in the form of trees, walls, roads, utility poles
(electricity, telephony), fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings .
[00066] Preferably the one or more artefacts also exist in the live video view of the site. [00067] Preferably the one or more artefacts do not exist in the live video view of the site.
[00068] Preferably the one or more artefacts comprise a spacial coordinate.
[00069] Preferably the spacial coordinate is a location on the earth surface.
[00070] Preferably the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.
[00071] In yet a further broad form of the invention there is provided media incorporating code stored therein which, when executed by a processor effects the method described above .
BRIEF DESCRIPTION OF DRAWINGS
[00072] Embodiments of the present invention will now be described with reference to the accompanying drawings wherein:
[00073] Figure 1 is a schematic diagram illustrating a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional
structure when the three-dimensional structure is located in a predetermined environment according to one preferred
embodiment of the present invention.
[00074] Figure 2 is a schematic diagram showing various channels to acquire at least 2D renderings from which to produce the three-dimensional structure according to any one of the preferred embodiments of the present invention.
[00075] Figures 3A and 3B are schematic diagrams
illustrating a method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan according to one preferred embodiment of the present invention.
[00076] Figure 4 is a flowchart applicable to the
arrangement of Figure 3A, 3B.
[00077] Figure 5 is a schematic diagram of an alternative embodiment for creating a composite mixed reality image relying on third party sourced imagery.
[00078] Figure 6 is schematic diagram of an embedded web link arrangement for access to embodiments of the present invention by means of a digital communications device.
[00079] Figure 7 is a 2D representation of a floorplan showing heat map output superposed on the floorplan in
accordance with a further embodiment of the present invention.
DESCRIPTION OF EMBODIMENTS
First Preferred Embodiment
[00080] Figure 1 is a schematic diagram illustrating a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional
structure when the three-dimensional structure is located in a predetermined environment according to one preferred
embodiment of the present invention.
[00081] Figure 2 is a schematic diagram showing various channels to produce the three-dimensional structure according to any one of the preferred embodiments of the present
invention .
[00082] Further Preferred Embodiment
[00083] With reference to Figures 3A and 3B, there is provided a method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan according to one preferred embodiment of the present invention .
[00084] Stage 1: remove the floor plan from the site map to produce an isolated siteplan view containing plan artefacts;
[00085] Stage 2: onsite pan camera to detect onsite
artefacts on the horizontal plan;
[00086] Stage 3: create a combined 2D top down view of detected onsite artefacts;
[00087] Stage 4: Overlay the artefact 2D view on the
isolated siteplan view and virtually shift in order to
correlate plan artefacts with onsite artefacts;
[00088] Stage 5: match at least selected onsite artefacts to plan artefacts thereby to reference 3D model against plan artefacts .
ELECTRONIC CREATION OF SITEPLAN
[00089] In a further preferred embodiment a siteplan may be electronically generated from a floorplan and external image data.
In one form, the external image data, as illustrated in Figure 5 may be derived from satellite imagery obtained from satellites 221 and referenced against GPS or other like spacial coordinates. As shown in Figure 5, a 2D overhead image 210A of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via spacial imagery server 220 over internet 212 to augmentation server 213. Augmentation server 213 receives a 2D plan 214 from an external source. In one form, the 2D plan 214 may be a 2D plan 214 of a building structure. Augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211. The resulting mixed reality image 216 may then be viewed via an imaging device 217. In one form, the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219. In a particular form the imaging device is a mixed reality headset.
Embedded Web link
[00090] Figure 6 is schematic diagram of an embedded web link arrangement for access to embodiments of the present invention by means of a digital communications device.
Superposed heat map
[00091] Figure 7 is a 2D representation of a floorplan showing heat map output superposed on the floorplan in
accordance with a further embodiment of the present invention.
[00092] In a preferred form, the heat map represents virtual person time spent at locations within the mixed reality image.
INDUSTRIAL APPLICABILITY
[00093] The above described methodology can be applied to depict three-dimensional representations of constructs such as dwellings on the actual site for which they are intended. This may be accomplished whilst the dwelling is still in the design phase .

Claims

1. A method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional structure when the three-dimensional structure is located in a predetermined environment; said method comprising
selecting a predetermined environment; the environment referenced against a location defined in three dimensions;
selecting a two-dimensional representation for placement in the environment;
the two-dimensional representation having reference artefacts included in it which are located to scale relative to the two-dimensional representation;
constructing a three-dimensional representation from the two-dimensional representation;
placing the three-dimensional representation into a mixed reality environment;
in substantially real time projecting the three- dimensional representation visually to a user as the user navigates through the predetermined environment superposed with a reality vision or live video view of the predetermined environment;
the three-dimensional representation referenced against the reference artefacts of the two-dimensional representation.
2. A method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan; said method including locating reference artefacts on a 2D representation of the building and site; utilizing the live video view to locate at least some of the reference artefacts on the site; anchoring the 3D model of the building against the at least some of the reference artefacts during the live video view thereby accurately aligning the 3D model of the building in 3D space with reference to the 2D siteplan.
3. The method of claim 2 wherein the site is the location where the building will be constructed.
4. The method of claims 2 or 3 wherein the 2D site plan of the building comprises a floorplan in the form of a 2D plan.
5. The method of claim 4 wherein the 2D plan is a view taken from above of a building compromised of walls, floors,
windows, doors, stairs and other.
6. The method of claim 4 wherein the 2D plan contains elevation information; the elevation information in totality contains data to construct a 3D model.
7. The method of any one of claims 2 to 6 wherein the 2D site plan comprises a 2D floorplan of the building.
8. The method of claim 7 wherein the 2D floorplan of the building is constructed within site boundaries of the site.
9. The method of claim 7 wherein the 2D floorplan of the building is constructed within neighbouring sites of the site.
10. The method of claim 7 wherein the 2D floorplan of the building is constructed within one or more artefacts of the site .
11. The method of claim 10 wherein each of the one or more artefacts is a distinguishing feature found in the 2D
siteplan .
12. The method of claim 10 or 11 wherein each of the one or more artefacts may be in the form of trees, walls, roads, utility poles (electricity, telephony) , fences, hedges, grass edges, planted gardens, gates, inclines, man holes,
rivers/streams, zebra crossings.
13. The method of any one of claims 10 to 12 wherein the one or more artefacts also exist in the live video view of the site .
14. The method of any one of claims 10 to 12 wherein the one or more artefacts do not exist in the live video view of the site .
15. The method of any previous claim wherein the one or more artefacts comprise a spacial coordinate.
16. The method of claim 15 wherein the spacial coordinate is a location on the earth surface.
17. The method of claim 15 or 16 wherein the spacial
coordinate lies within a bounded planar surface to which the floor plan is to be applied.
18. A method for accurately anchoring a floorplan and/or 3D model to a live video view of a site on which the floorplan has been depicted on a site map; the method comprising
producing an isolated site view; and wherein the isolated site view is produced by removing the floorplan from the site map by looking for floorplan artefacts such as the door and window geometry and then expanding to include geometry with straight lines forming concave shapes.
19. The method of claim 18 wherein the live video view of the site comprises a video feed.
20. The method of claim 19 wherein said video feed is produced by a camera in a phone, tablet or computer by the user (or electronics) .
21. The method of claim 19 or 20 wherein said video feed is panning over a site area of the site detecting a horizontal plane of the site area.
22. The method of any one of claims 18 to 21 wherein detects artefacts on the horizontal plane of the site area.
23. The method of claim 22 wherein the artefacts are detected from high contrast changes in video change that exists as the camera move; and wherein the high contrast changes are
recognized as detected vertical objects; and wherein
approximate shape and location of the detected vertical objects are concluded and calculated; and wherein the
approximate shape and location of the detected vertical objects are included into a 2D view of the site map.
24. The method of claim 23 wherein overlays the 2D view on the isolated plan view; wherein the method conducts a
correlation by correlating the detected vertical objects of the 2D view with the artefacts of the isolated plan view.
25. The method of claim 18 wherein the correlation is above a certain threshold, isolated plan view and the live video view of the site; and wherein the 3D model is projected to live video feed.
26. The method of claim 25 wherein enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.
27. The method of any of claims 1 to 23 wherein a siteplan may be electronically generated from a floorplan and external image data.
28. The method of claim 24 wherein the external image data is derived from satellite imagery referenced against GPS or other like spacial coordinates.
29. The method of claim 25 wherein 2D overhead image of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via internet 212 to an augmentation server 213.
30. The method of claim 26 wherein the augmention server 213 receives a 2D plan 214 from an external source.
31. The method of claim 27 wherein the 2D plan 214 may be a 2D plan 214 of a building structure.
32. The method of claim 28 wherein the augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211 thereby to generate a mixed reality image.
33. The method of claim 29 wherein the resulting mixed reality image 216 may then be viewed via an imaging device 217.
34. The method of claim 30 wherein the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219.
35. The method of any one of claims 1 to 31 wherein the augmentation server is accessed via a web link on a website.
36. The method of any one of claims 1 to 32 wherein a 2D representation of a floorplan has heat map output superposed on the floorplan.
37. The method of claim 33 wherein the heat map represents virtual person time spent at locations within the mixed reality image.
38. A mixed reality headset having a display which combines reality vision of site of a pre-determined environment with a three-dimensional representation of a three-dimensional structure; the representation of the three-dimensional
structure referenced against an artefact in the pre-determined environment .
39. The mixed reality headset of claim 38 wherein the site is the location where a building will be constructed.
40. The mixed reality headset of claims 38 or 39 wherein the 2D site plan of the building comprises a floorplan in the form of a 2D plan.
41. The mixed reality headset any one of claims 38 to 40 wherein the 2D plan is a view taken from above of a building compromised of walls, floors, windows, doors, stairs and other .
42. The mixed reality headset of claim 41 wherein the 2D plan contains elevation information; the elevation information in totality contains data to construct a 3D model.
43. The mixed reality headset of any one of claims 38 to 42 wherein the 2D site plan comprises a 2D floorplan of the building .
44. The mixed reality headset of any one of claims 38 to 43 wherein the 2D floorplan of the building is constructed within site boundaries of the site.
45. The mixed reality headset of any one of claims 38 to 44 wherein the 2D floorplan of the building is constructed within neighbouring sites of the site.
46. The mixed reality headset of any one of claims 38 to 45 wherein the 2D floorplan of the building is constructed within one or more artefacts of the site.
47. The mixed reality headset of any one of claims 38 to 46 wherein each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.
48. The mixed reality headset of any one of claims 38 to 47 wherein each of the one or more artefacts may be in the form of trees, walls, roads, utility poles (electricity,
telephony), fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings.
49. The mixed reality headset of any one of claims 38 to 48 wherein the one or more artefacts also exist in the live video view of the site.
50. The mixed reality headset of any one of claims 38 to 49 wherein the one or more artefacts do not exist in the live video view of the site.
51. The mixed reality headset of any one of claims 38 to 50 wherein the one or more artefacts comprise a spacial
coordinate .
52. The mixed reality headset of claim 51 wherein the spacial coordinate is a location on the earth surface.
53. The mixed reality headset of claim 51 of 52 wherein the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.
54. Media incorporating code stored therein which, when executed by a processor effects the method of any of claims 1 to 37.
PCT/AU2019/051328 2018-12-04 2019-12-04 Mixed reality visualisation system WO2020113273A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
AU2018904613A AU2018904613A0 (en) 2018-12-04 Mixed reality visualisation system
AU2018904613 2018-12-04

Publications (1)

Publication Number Publication Date
WO2020113273A1 true WO2020113273A1 (en) 2020-06-11

Family

ID=70974044

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/AU2019/051328 WO2020113273A1 (en) 2018-12-04 2019-12-04 Mixed reality visualisation system

Country Status (1)

Country Link
WO (1) WO2020113273A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022255546A1 (en) * 2021-05-31 2022-12-08 주식회사 쓰리아이 Virtual indoor space content providing method and server therefor
CN116502948A (en) * 2023-04-25 2023-07-28 中国地铁工程咨询有限责任公司 Method and device for measuring passenger flow conveying capacity of facilities in station

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110191069A1 (en) * 2008-06-30 2011-08-04 Tripod Components Pty Ltd System and method for designing a building
US20110218777A1 (en) * 2010-03-03 2011-09-08 Honeywell International Inc. System and method for generating a building information model
US20170256097A1 (en) * 2016-03-07 2017-09-07 F3 & Associates Local positioning system for augmented reality applications
US20180082414A1 (en) * 2016-09-21 2018-03-22 Astralink Ltd. Methods Circuits Assemblies Devices Systems Platforms and Functionally Associated Machine Executable Code for Computer Vision Assisted Construction Site Inspection

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110191069A1 (en) * 2008-06-30 2011-08-04 Tripod Components Pty Ltd System and method for designing a building
US20110218777A1 (en) * 2010-03-03 2011-09-08 Honeywell International Inc. System and method for generating a building information model
US20170256097A1 (en) * 2016-03-07 2017-09-07 F3 & Associates Local positioning system for augmented reality applications
US20180082414A1 (en) * 2016-09-21 2018-03-22 Astralink Ltd. Methods Circuits Assemblies Devices Systems Platforms and Functionally Associated Machine Executable Code for Computer Vision Assisted Construction Site Inspection

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022255546A1 (en) * 2021-05-31 2022-12-08 주식회사 쓰리아이 Virtual indoor space content providing method and server therefor
CN116502948A (en) * 2023-04-25 2023-07-28 中国地铁工程咨询有限责任公司 Method and device for measuring passenger flow conveying capacity of facilities in station
CN116502948B (en) * 2023-04-25 2024-01-19 中国地铁工程咨询有限责任公司 Method and device for measuring passenger flow conveying capacity of facilities in station

Similar Documents

Publication Publication Date Title
US9619944B2 (en) Coordinate geometry augmented reality process for internal elements concealed behind an external element
US20190272676A1 (en) Local positioning system for augmented reality applications
Behringer Registration for outdoor augmented reality applications using computer vision techniques and hybrid sensors
Sankar et al. Capturing indoor scenes with smartphones
US8437554B2 (en) Method of extracting three-dimensional objects information from a single image without meta information
Simon et al. Pose estimation for planar structures
EP2401703B1 (en) System and method of indicating transition between street level images
Themistocleous Model reconstruction for 3d vizualization of cultural heritage sites using open data from social media: The case study of Soli, Cyprus
JP2023505212A (en) Systems and methods for modeling structures using point clouds derived from pairs of stereoscopic images
Côté et al. Live mobile panoramic high accuracy augmented reality for engineering and construction
Stal et al. Integrating geomatics in archaeological research at the site of Thorikos (Greece)
El-Hakim Semi-automatic 3D reconstruction of occluded and unmarked surfaces from widely separated views
CN102647512A (en) All-round display method of spatial information
US20030225513A1 (en) Method and apparatus for providing multi-level blended display of arbitrary shaped textures in a geo-spatial context
WO2020113273A1 (en) Mixed reality visualisation system
Kersten et al. Architectural historical 4D documentation of the old-segeberg town house by photogrammetry, terrestrial laser scanning and historical analysis
Wither et al. Using aerial photographs for improved mobile AR annotation
El-Hakim Three-dimensional modeling of complex environments
KR20140031129A (en) Virtual face generating method of omni-directional image using user interaction
US9305401B1 (en) Real-time 3-D video-security
Stal et al. Highly detailed 3D modelling of Mayan cultural heritage using an UAV
El-Hakim et al. 3D reconstruction of complex architectures from multiple data
Gonzo et al. Photo-realistic 3D reconstruction of castles with multiple-sources image-based techniques
Morandi et al. Interactive past: from 3D reconstruction to augmented and virtual reality applied to archaeological heritage. The medieval site of Bastia St. Michele (Cavaion Veronese, Verona, Italy)
Cheng et al. Texture mapping 3d planar models of indoor environments with noisy camera poses

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19893784

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19893784

Country of ref document: EP

Kind code of ref document: A1