AU2015268724A1 - Method, system and apparatus for forming an augmented reality (AR) image - Google Patents

Abstract

METHOD, SYSTEM AND APPARATUS FOR FORMING AN AUGMENTED REALITY (AR) IMAGE A method of forming an augmented reality image for display to a user of an optical see 5 through augmented reality (AR) system. An augmentable object in an image of a scene is identified, the augmentable object being visible to the user and being located at a first image plane from the user. An augmentation associated with the identified augmentable object is selected. A digital representation of the identified augmentable object is formed for display on the optical see-through augmented reality system at a second image plane, the second image 10 plane being different to the first image plane. An augmented reality image is formed on the augmented reality system by displaying the augmentation on the augmented reality system adjacent to the identified object and overlaying the identified augmentable object, visible at the first image plane, with the digital representation of the identified augmented reality object at the second image plane such that the digital representation and the augmentation are displayed at 15 the second image plane while pass through portions of the display are at real world image planes. 1AfQfArrr-7u,1 ID1 rM 1 A1r-i Ae EilAr -1/10 Fig. 1A 10806546v1 (P150112_FigsAs Filed)

Description

Η Ο (Ν Ο (D Q (Ν (X) (Ν Ο (Ν

METHOD, SYSTEM AND APPARATUS FOR FORMING AN AUGMENTED

REALITY (AR) IMAGE

TECHNICAL FIELD

The present invention relates to the seamless display of an image with associated augmented content when viewed through an optical-see-through augmented reality system such as an optical see-through Head-Mounted-Display. The present invention also relates to a method, system and apparatus for forming an augmented reality image, and to a computer program product including a computer readable medium having recorded thereon a computer program for forming an augmented reality image.

10 BACKGROUND

Augmented reality (AR) is a view of a physical world where some elements of physical reality are augmented by computer generated inputs such as sound, graphics and so on. Users are able to use hand-held and wearable devices to retrieve additional information related to a captured image of a real world object from a camera connected to the device (e.g. a camera 15 phone or a camera attached to a head-mounted display (HMD) or augmented reality (AR) glasses) and augment the additional information to the real world object. Such a real-world object may be a natural image in a document, a piece of textual information, a physical object such as a printer and so on.

Currently there exist two types of head-mounted displays that are used for mixed reality 20 applications: a video see-through head-mounted display (VSTHMD) and optical see-through head-mounted display (OSTHMD). A video see-through head-mounted display is a wearable headset that typically comprises a stereo-display, a processing unit and camera. Users of a video see-through head-mounted display are immersed in a fully virtual environment which may be a digital replication of the environment captured by a camera of the head-mounted display. Other 25 digital content may be included in the virtual environment to give the user the illusion of additional information being added to the real-world. Video see-through head-mounted displays are typically referred to as virtual reality devices. Video see-through head-mounted displays render digital content and the virtual scene within the same image plane, which may be undesirable in some circumstances. 10806567V1 (P150112_Speci_As Filed) ΙΟ ο (Ν Ο (D Q ΙΟ (Ν 00 Ό (Ν 12 10 ο (Ν 15 20

Optical see-through head-mounted displays are wearable headsets containing stereo see-through displays, a processing unit and a camera. The optical see-through head-mounted display processes information in a real-world scene captured by the camera and displays digital content on the see-through display to give a user the illusion that the digital content is part of the real-world scene.

An advantage that optical see-through head-mounted displays have over video see-through head-mounted displays is the optical see-through display allows users of the head-mounted display to see digital content alongside objects in the real-world; hence strengthening the relationship between digital content and real-world objects as well as providing a good user experience. Optical see-through head-mounted displays may be referred to as mixed reality devices.

Conventional video see-through head-mounted displays are only capable of displaying digital content at a fixed image plane with a fixed field-of-view. While this provides a fully immersive experience for users, the fixed rendering plane can make the experience feel unnatural for users in real-world simulations.

Conventional optical see-through head-mounted displays are not bound to a fixed field-of-view but are still limited to rendering at a fixed image plane. This is due to the type of mirror configuration that is used to display digital content within the optical see-through head-mounted displays. Common configuration types are waveguide or curved-mirror optics. When appearing at a fixed image plane, digital content appears disjoint from real-world content as the user can only focus at the digital image plane or the real-world object plane; but not both.

One conventional method tries to overcome the above described problem with optical see-through head-mounted displays by surrounding digital content with a variable transparent window. The inner-portions of the window are less transparent than the outer portions giving 25 the illusion that the digital content is surrounded and has the same depth as the surrounding real-world objects. While this conventional method works well in preserving the apparent depth of both the digital and real-world objects, portions of the real-world scene are lost under the non-transparent parts of the surrounding window.

Another conventional method adds a multi-planar optical element to an optical see-30 through head-mounted display to allow digital content to be spliced and distributed amongst multiple image planes and to thus appear to be part of the real-world scene. The addition of the 10806567V1 (P150112_Speci_As Filed)

ο (N Ο ω Q (N 00 Ό (N O (N 10 multi-planar optical element to an optical see-through head-mounted display solves the multiimage plane occlusion problem; however additional optics need to be installed with each optical see-through head-mounted display.

SUMMARY

It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

Disclosed are arrangements which seek to address the above problems by modifying a digital representation of a real-world object alongside additional digital content to ensure both the digital real-world content and the additional digital content appear to be in the same image plane as one another when viewed through an optical see-through head-mounted display.

According to one aspect of the present disclosure, there is provided a method of forming an augmented reality image for display to a user of an optical see-through augmented reality (AR) system, the method comprising: identifying an augmentable object in an image of a scene, the augmentable object being 15 visible to the user and being located at a first image plane from the user; selecting an augmentation associated with the identified augmentable object; forming a digital representation of the identified augmentable object for display on the optical see-through augmented reality system at a second image plane, the second image plane being different to the first image plane; 20 forming an augmented reality image on the augmented reality system by displaying the augmentation on the augmented reality system adjacent to the identified object and overlaying the identified augmentable object, visible at the first image plane, with the digital representation of the identified augmented reality object at the second image plane such that the digital representation and the augmentation are displayed at the second image plane while pass through 25 portions of the display are at real world image planes. 10806567V1 (P150112_Speci_As Filed) ΙΟ ο (Ν Ο <D Ο (Ν Ο οο (Ν Η Ο (Ν 10 15 25

According to another aspect of the present disclosure, there is provided a system for forming an augmented reality image for display to a user of an optical see-through augmented reality (AR) system, the system comprising: a memory for storing data and a computer program; a processor coupled to the memory for executing said computer program, said computer program comprising instructions for: identifying an augmentable object in an image of a scene, the augmentable object being visible to the user and being located at a first image plane from the user; selecting an augmentation associated with the identified augmentable object; forming a digital representation of the identified augmentable object for display on the optical see-through augmented reality system at a second image plane, the second image plane being different to the first image plane; forming an augmented reality image on the augmented reality system by displaying the augmentation on the augmented reality system adjacent to the identified object and overlaying the identified augmentable object, visible at the first image plane, with the digital representation of the identified augmented reality object at the second image plane such that the digital representation and the augmentation are displayed at the second image plane while pass through portions of the display are at real world image planes. 20 According to another aspect of the present disclosure, there is provided an apparatus for forming an augmented reality image for display to a user of an optical see-through augmented reality (AR) system, the apparatus comprising: module for identifying an augmentable object in an image of a scene, the augmentable object being visible to the user and being located at a first image plane from the user; module for selecting an augmentation associated with the identified augmentable object; 10806567V1 (P150112_Speci_As Filed)

ο (N Ο <υ Q

'it (N Γ ΟΟ Ό (N in O 10 (N module for forming a digital representation of the identified augmentable object for display on the optical see-through augmented reality system at a second image plane, the second image plane being different to the first image plane; module for forming an augmented reality image on the augmented reality system by displaying the augmentation on the augmented reality system adjacent to the identified object and overlaying the identified augmentable object, visible at the first image plane, with the digital representation of the identified augmented reality object at the second image plane such that the digital representation and the augmentation are displayed at the second image plane while pass through portions of the display are at real world image planes.

According to another aspect of the present disclosure, there is provided a computer readable medium having a computer program stored on the medium for forming an augmented reality image for display to a user of an optical see-through augmented reality (AR) system, the program comprising: code for identifying an augmentable object in an image of a scene, the augmentable 15 object being visible to the user and being located at a first image plane from the user; code for selecting an augmentation associated with the identified augmentable object; code for forming a digital representation of the identified augmentable object for display on the optical see-through augmented reality system at a second image plane, the second image plane being different to the first image plane; 20 code for forming an augmented reality image on the augmented reality system by displaying the augmentation on the augmented reality system adjacent to the identified object and overlaying the identified augmentable object, visible at the first image plane, with the digital representation of the identified augmented reality object at the second image plane such that the digital representation and the augmentation are displayed at the second image plane 25 while pass through portions of the display are at real world image planes.

Other aspects of the invention are also disclosed. 10806567V1 (P150112_Speci_As Filed)

Ο (N Ο (D Q

(N 00 Ό (N O (N 25

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the invention will now be described with reference to the following drawings, in which:

Fig. 1A is a schematic diagram of an augmented reality system on which the 5 arrangements described may be practiced;

Figs. IB and 1C show a computing device of the system of Fig. lA in more detail;

Fig. ID is a diagram showing software code modules of a software application program executing on the system of Fig. 1A according to one arrangement;

Fig 2 shows an example of the system of Fig. lA displaying output in a conventional 10 manner;

Fig 3 is a schematic flow diagram showing a method of forming an augmented reality image for display to a user of augmented reality system of Fig. 1 A;.

Fig 4A shows example output of the augmented reality system of Fig. lA before implementing the method of Fig. 3; 15 Fig 4B shows example output of the augmented reality system of Fig. lA after implementing the method of Fig. 3;

Fig 5 shows example output of the augmented reality system of Fig. 1A after implementing the method of Fig. 3 for a scene comprising two real-world objects; and

Fig. 6 is a schematic flow diagram showing another method of forming an augmented 20 reality image for display to a user of the augmented reality system of Fig. 1 A.

DETAILED DESCRIPTION INCLUDING BEST MODE

Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears. 10806567V1 (P150112_Speci_As Filed)

Η Ο (Ν Ο (D Q (Ν 00 Ό (Ν ^ 10 ο (Ν 30 7

When digital content is added to a real-world scene on a see-through display there is a disparity between the image-plane in which the digital content resides and the image-plane in which the real-world object resides. As described below, the digital content and a representation of the real-world object may be manipulated in order to seek to address the disparity problem.

Figs. lA, IB and 1C depict an augmented reality (AR) system 100 in the form of a wearable optical see-through head-mounted display 110, upon which various methods described below can be practiced. The described methods may also be performed on other electronic devices such as, for example, a mobile phone, a portable media player, desktop computers and server computers.

The optical see-through head-mounted display 110 comprises a camera 114 capable of capturing images in high-defmition (HD), a high-defmition (HD) micro-display pair 130 and an optical see-through head-mounted display computing device 140.

The optical see-through head-mounted display 110 allows users to view the real-world 15 through see-through glasses while viewing digital content through the micro-display pair 130.

In one arrangement, the optical see-through head-mounted display 110 contains waveguide optics to transmit light to the micro display 130. However, other types of optical configurations for the optical see-through head-mounted display 110 may also be used, such as curved mirror optics. 20 In one arrangement, when a user of the optical see-through head-mounted display 110 views an object the optical see-through head-mounted display 110 identifies the object and replicates the object onto the micro-display pair 130 along with additional augmented content.

Objects able to be identified by the optical see-through head-mounted display 110 include printed documents, posters, a natural image in a printed document, poster or the like, a 25 piece of textual information in a printed document, poster or the like, a physical object such as a printer, and other planar objects. However, other types of identifiable objects are also possible such as cars, faces and buildings.

As seen in Fig. IB, the computing device 140 comprises an embedded controller 102.

In the present example, the controller 102 has a processor 105 which is bi-directionally coupled to an internal storage module 109. The storage module 109 may be formed from non-volatile 10806567V1 (P150112_Speci_As Filed) ο (N Ο <υ Q IT)

(N r-' (X) (N O 10 (N 15 semiconductor read only memory (ROM) 160 and semiconductor random access memory (RAM) 170, as seen in Fig. 1C. The RAM 170 may be volatile, non-volatile or a combination of volatile and non-volatile memory.

The computing device 140 includes a display controller 107, which is connected to micro-display pair 130. The micro-display pair 130 may be implemented, for example, as a liquid crystal display (LCD), a liquid crystal on silicon (LCDS) display or an organic light-emitting diode (OLED) display, or any other suitable micro-display.

The display controller 107 is configured for displaying graphical images on the display pair 130 in accordance with instructions received from the embedded controller 102, to which the display controller 107 is connected.

The computing device 140 also has a communications interface 108 to permit coupling of the optical see-through head-mounted display 110 to a computer or communications network 120 via a connection 121. The connection 121 may be wired or wireless. For example, the connection 121 may be radio frequency or optical. An example of a wired connection includes Ethernet. Further, an example of wireless connection includes Bluetooth™ type local interconnection, Wi-Fi (including protocols based on the standards of the IEEE 802.11 family). Infrared Data Association (IrDa) and the like.

The embedded controller 102 is connected to the camera 114 which may comprise a lens, focus control and image sensor. 20 Methods described below may be implemented using the embedded controller 102, where the processes of Figs. 3 to 6 may be implemented as one or more software application programs 133 executable within the embedded controller 102. The computing device 140 of Fig. IB implements the described methods. In particular, with reference to Fig. 1C, the steps of the described methods are effected by instructions in the software 133 that are carried out 25 within the controller 102. The software instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the described methods and a second part and the corresponding code modules manage a user interface between the first part and the user. 10806567V1 (P150112_Speci_As Filed)

ο (N Ο <υ Q

(N 00 (N o (N

The software 133 of the embedded controller 102 is typically stored in the non-volatile ROM 160 of the internal storage module 109. The software 133 stored in the ROM 160 can be updated when required from a computer readable medium. The software 133 can be loaded into and executed by the processor 105. In some instances, the processor 105 may execute 5 software instructions that are located in RAM 170. Software instructions may be loaded into the RAM 170 by the processor 105 initiating a copy of one or more code modules from ROM 160 into RAM 170. Alternatively, the software instructions of one or more code modules may be pre-installed in a non-volatile region of RAM 170 by a manufacturer. After one or more code modules have been located in RAM 170, the processor 105 may execute software instructions 10 of the one or more code modules.

The application program 133 is typically pre-installed and stored in the ROM 160 by a manufacturer, prior to distribution of the optical see-through head-mounted display 110. However, in some instances, the application programs 133 may be supplied to the user encoded on one or more CD-ROM (not shown) and read via a portable memory interface (not shown) of 15 the optical see-through head-mounted display 110 prior to storage in the internal storage module 109. In another alternative, the software application program 133 may be read by the processor 105 from the network 120, or loaded into the controller 102 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that participates in providing instructions and/or data to the controller 102 for 20 execution and/or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM or integrated circuit, USB memory, a magnetooptical disk, flash memory, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the optical see-through head-mounted display 110. Examples of transitory or non-tangible computer readable transmission media that 25 may also participate in the provision of software, application programs, instructions and/or data to the optical see-through head-mounted display 110 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like. A computer readable medium having such software or computer program recorded on 30 it is a computer program product.

The second part of the application programs 133 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the micro-display pair 130 of Fig. lA. In some 10806567V1 (P150112_Speci_As Filed)

ο (N Ο ω Q

(N 00 (N ^ 10 O (N 15 20 25 30 10 arrangements, a user of the optical see-through head-mounted display 110 and the application programs 133 may manipulate the interface in a functionally adaptable manner to provide controlling commands and/or input to the applications associated with the GUI(s). For example, the optical see-through head-mounted display 110 may comprise functionally adaptable user interfaces, such as an audio interface utilizing speech prompts output via loudspeakers (not illustrated) and user voice commands input via a microphone (not illustrated).

Fig. 1C illustrates in detail the embedded controller 102 having the processor 105 for executing the application programs 133 and the internal storage 109. The internal storage 109 comprises read only memory (ROM) 160 and random access memory (RAM) 170. The processor 105 is able to execute the application programs 133 stored in one or both of the connected memories 160 and 170. When the optical see-through head-mounted display 110 is initially powered up, a system program resident in the ROM 160 is executed. The application program 133 permanently stored in the ROM 160 is sometimes referred to as “firmware”. Execution of the firmware by the processor 105 may fulfil various functions, including processor management, memory management, device management, storage management and user interface.

The processor 105 typically includes a number of functional modules including a control unit (CU) 151, an arithmetic logic unit (ALU) 152 and a local or internal memory comprising a set of registers 154 which typically contain atomic data elements 156, 157, along with internal buffer or cache memory 155. One or more internal buses 159 interconnect these functional modules. The processor 105 typically also has one or more interfaces 158 for communicating with external devices via system bus 181, using a connection 161.

The application program 133 includes a sequence of instructions 162 through 163 that may include conditional branch and loop instructions. The program 133 may also include data, which is used in execution of the program 133. This data may be stored as part of the instruction or in a separate location 164 within the ROM 160 or RAM 170.

In general, the processor 105 is given a set of instructions, which are executed therein. This set of instructions may be organised into blocks, which perform specific tasks or handle specific events that occur in the electronic device 101. Typically, the application program 133 waits for events and subsequently executes the block of code associated with that event. Events may be triggered in response to input from a user, via the user input devices 113 of Fig. IB, as 10806567V1 (P150112_Speci_As Filed) ΙΟ Ο (Ν Ο <D Ο ΙΟ (Ν οο (Ν ΙΟ Ο 10 (Ν 15 20 25 30 11 detected by the processor 105. Events may also be triggered in response to other sensors and interfaces in the optical see-through head-mounted display 110.

The execution of a set of the instructions may require numeric variables to be read and modified. Such numeric variables are stored in the RAM 170. The disclosed methods use input variables 171 that are stored in known locations 172, 173 in the memory 170. The input variables 171 are processed to produce output variables 177 that are stored in known locations 178, 179 in the memory 170. Intermediate variables 174 may be stored in additional memory locations in locations 175, 176 of the memory 170. Alternatively, some intermediate variables may only exist in the registers 154 of the processor 105.

The execution of a sequence of instructions is achieved in the processor 105 by repeated application of a fetch-execute cycle. The control unit 151 of the processor 105 maintains a register called the program counter, which contains the address in ROM 160 or RAM 170 of the next instruction to be executed. At the start of the fetch execute cycle, the contents of the memory address indexed by the program counter is loaded into the control unit 151. The instruction thus loaded controls the subsequent operation of the processor 105, causing for example, data to be loaded from ROM memory 160 into processor registers 154, the contents of a register to be arithmetically combined with the contents of another register, the contents of a register to be written to the location stored in another register and so on. At the end of the fetch execute cycle the program counter is updated to point to the next instruction in the system program code. Depending on the instruction just executed this may involve incrementing the address contained in the program counter or loading the program counter with a new address in order to achieve a branch operation.

Each step or sub-process in the processes of the methods described below is associated with one or more segments of the application program 133, and is performed by repeated execution of a fetch-execute cycle in the processor 105 or similar programmatic operation of other independent processor blocks in the optical see-through head-mounted display 110.

As described above, the steps of the described methods are effected by instructions in the software 133 that are carried out within the controller 102. In one arrangement, as seen in Fig. ID, the software instructions may be formed as one or more code modules 141, 143 and 144, each for performing one or more particular tasks. In the example of Fig. ID, the code modules include an object identification module 141, an augmented reality adjustment module 10806567V1 (P150112_Speci_As Filed)

Ο (N Ο (D Q

(N 00 (N ^ 10 O (N 12 143 and display module 144. In other arrangements, a scene analysis module 145 (shown in broken lines) may be included.

The object identification module 141 takes as input a raster image captured from the camera 114 and produces as output a set of points describing a polygon which represents the ‘position and orientation’ (i.e., POSE) of a real-world object within the raster image of the scene. The augmented reality adjustment module 143 takes as input a set of points representing the POSE of a real-world object and produces as output a set of digital images to be displayed. The display module 144 takes as input a set of digital images and produces as output, digital images on the micro-display pair 130. A detailed operation of each module 141, 143, 144 and 145 will be described below.

Fig. ID also shows an augmented database 142 which may be configured within the storage module 109.

Fig. 2 shows an example of the optical see-through head-mounted display 110 displaying output without using methods 300 (see Fig. 3) and 600 (see Fig. 6) described below. 15 Within a field of view 210 of the camera 114 is a real-world object 220 as seen by a user wearing the optical see-through head-mounted display 110. The real-world object 220 resides within an image plane 240 at some arbitrary distance from the optical see-through head-mounted display 110. Digital content 230 associated with the real-world object 220 is displayed on the micro-display pair 130. The digital content 230 resides in image plane 250 20 when viewed through the micro-display pair 130 on the optical see-through head-mounted display 110.

As seen in Fig. 2, distance 260 shows the disparity between the image plane of the real-world object 220 and the associated digital content 230 when the optical see-through head-mounted display 110 is not implementing the methods described below. As discussed in the 25 Background section above, due the nature of the optics that drive micro-display pair 130 of the optical see-through head-mounted display 110, the image plane 250 of the digital content 230 will appear to be displayed at infinity (i.e. approximately five (5) meters in real-world distance). Displaying digital content 230 at such a fixed focal distance will cause digital content 230 to appear disjoint from other real-world object 220. 30 Fig 3 is a schematic flow diagram showing a method 300 of forming an augmented reality image for display to a user of the optical see-through head-mounted display 110. The 10806567V1 (P150112_Speci_As Filed)

ΙΟ Η Ο (Ν Ο (D Q (Ν Ο οο (Ν ΙΟ Ο 10 (Ν 25 30 13 method 300 may be implemented as one or more of the code modules 141, 143 and 144 of the software application program 133 resident in the ROM 160 and being controlled in its execution by the processor 105.

The method 300 begins at capturing step 310. At capturing step 310, the camera 120 on the optical see-through head-mounted display 110 captures a raster image of a scene (or ‘scene raster image’) and transfers the captured raster image of the scene to the object identification module 141 under execution of the processor 105.

Following the capturing step 310, the method 300 proceeds to an identifying step 320. In step 320, the object identification module 141, under execution of the processor 105, identifies any known real-world object within the raster image of the scene. Each identified object may be visible to the user of the optical see-through head-mounted display 110 and is located at a first image plane from the user. Each identified object may be augmentable as described below.

In one arrangement, known real-world objects are two-dimensional planar objects such 15 as documents and images. However, as described above, other types of objects may be identified using the described methods. To identify real-world objects, the object identification module 141, under execution of the processor 105, firstly performs the step of detecting feature points within the scene raster image using a feature detection algorithm. The feature detection algorithm used in the described methods may be the scale-invariant feature transform (SIFT) 20 feature detection algorithm. However, any other suitable feature detection algorithm may also be used.

The object identification module 141, under execution of the processor 105, then queries the augmented database 142 using the feature points extracted from the raster image of the scene. In one arrangement, the augmented database 142 contains a KD-tree that is a searchable data structure containing scale-invariant feature transform (SIFT) feature points. However, any other suitable searchable data structures may also be used in the described methods.

To identify a real-world object, a raster image of the object is first registered with the KD-tree. Raster images are registered to the KD-tree by first assigning a single unique number to all SIFT points, and then adding all SIFT points to the KD-tree. The registration of the raster images to the KD-tree is performed prior to execution of the method 300. In addition, each group of SIFT points associated with a raster image within the KD-tree of the augmented 10806567V1 (P150112_Speci_As Filed) ΙΟ Ο (Ν Ο <D Ο ΙΟ (Ν Ο οο (Ν ^Ο Η Ο 10 (Ν 15 20 14 database 142 contains an identification number that is associated with augmented content. Augmented content and the identification number is directly added to the augmented database 142 configured within the storage module 109 (e g., RAM 170).

In one arrangement, augmented content stored within the augmented database 142 is a combination of text, images and videos that are displayed within a same digital container. Other types of augmented content may also be stored within the augmented database 142, such as, for example, audio, graphs and animations. In addition, a preferred position is stored with each portion of augmented content in the augmented database 142 to determine where, with respect to an identified real-world object, the augmentation is to be placed.

Once the object identification module 141 queries the augmented database 142, augmented content, with an identification number that has the highest number of matches, and the matched feature points are returned to the object identification module 141. The object identification module 141, under execution of the processor 105, then calculates the POSE of the real-world object within the scene raster image based on the returned matched feature points. The POSE of the real-world object is represented by a set of points which define a closed polygon that describes an outline of the real-world object. In one arrangement, the object identification module 141 calculates the POSE of the real-world object using an outlier removal method called random sample consensus (RANSAC). RANSAC returns a subset of the original matched feature points which define the POSE of the real-world object. However, other suitable outlier removal methods may also be used in the described methods. The real-world object POSE and associated augmented content may be stored in the storage module 109 (e.g., RAM 170) under execution of the processor 105.

After the identifying step 320, the method 300 proceeds to a registering step 330. In the registering step 330, the augmented reality adjustment module 143, under execution of the 25 processor 105, first determines if an object identified at step 320 contains associated augmented content. If the identified object contains associated augmented content, then the augmented reality adjustment module 143 retrieves the real-world object POSE from the storage module 109 for the identified object. The augmented reality adjustment module 143 then copies, into the storage module 109, the pixels within the raster image of the scene that are enclosed by the 30 POSE as a digital representation of the object identified at step 320. The copied pixels forming the digital representation of the object may be referred to below as the digital real-world object. 10806567V1 (P150112_Speci_As Filed) ΙΟ Ο (Ν Ο (D Q ΙΟ

(N 00 (N O (N 10 25 15

After the registering step 330, the method 300 proceeds to an applying step 340. In the applying step 340, the augmented reality adjustment module 143 determines a hand-eye calibration to apply to the digital representation of the digital real-world object, in order to determine how the digital representation of the digital real-world object should be sized and positioned within the micro-display 130. The hand-eye calibration may be generalised by homogenous Equation (1) as follows:

CX = XD (1) where C is a calibration matrix that transforms a point in the camera raster image captured at step 310 to a real-world coordinate and C is determined via camera calibration; D transforms a real-world point on the optical see-through head-mounted display 110 to the micro-display pair 130; X denotes the transform from a point in the image captured by the camera 120 at step 310 to a point on the micro-display pair 130.

Each of the matrices has the following form where Rc is a rotation matrix and Tc is a translation matrix. 15 Both C and D are determined by providing sample positions of points in various planes and solving for R and T. R and T may be performed during any initialisation step of the computing device 140 or pre-loaded onto the storage module 109 (e.g., ROM 160) of the optical see-through head-mounted display 110. Solving for X will allow for points to be transformed from the camera 114 to the micro-display pair 130. The hand eye calibration may 20 be pre-determined and loaded onto the storage module 109 (e.g., ROM 160) of the computing device 140.

Once the hand-eye calibration has been determined, the augmented reality adjustment module 143 applies the hand-eye calibration to the representation of the digital real-world object and to the associated augmented content stored in the storage module 109. In addition to applying the hand-eye calibration, the augmented reality adjustment module 143, under 10806567V1 (P150112_Speci_As Filed) Ο (N Ο <D Ο

(N 00 (N ^ 10 O (N 30 16 execution of the processor 105, applies a translation to the augmented content as per the preferred position stored within the augmented database 142. The translation is applied to the augmented content to ensure that the augmentation is positioned at an appropriate location with respect to the digital real-world object (e.g. to the left or right of the digital real-world object).

Following the applying step 340, the method 300 proceeds to displaying step 350. At displaying step 350, the display module 144, under execution of the processor 105, retrieves the augmented content and the adjusted digital representation of the digital real-world object from the storage module 109 and passes the augmented content and the adjusted digital representation of the digital real-world object to the optical drivers of the optical see-through head-mounted display 110 where the augmented content and the adjusted digital representation of the digital real-world object are displayed on the micro-display 130. After step 350 the methods 300 ends.

The method 300 will now be further described by way of example with reference to Figs. 4A and 4B which respectively show two example scenarios involving the optical see-through 15 head-mounted display 110. In the example scenario A of Fig. 4A, the optical see-through head-mounted display 110 does not use the methods 300 described above or the method 600 described below; whereas in example scenario B of Fig. 4B the optical see-through head-mounted display 110 implements the method 300.

In the example of Figs 4 A and 4B, the camera 114 is a high definition (HD) camera, and 20 the micro-display pair 130 is a high definition (HD) display.

In the example of Fig. 4A, the computing device 140 contains an object identification module 141 and an augmented database 142. The camera 114 captures a raster image of the scene field of view 470 and the computing device 140 uses the object identification module 141 to identify real-world object 450a. The identified object 450a may be visible to the user of the 25 optical see-through head-mounted display 110 and is located at a first image plane from the user.

The computing device 140 then uses the object identification module 141 to retrieve augmented content 470 from the augmented database 142 as described in accordance with the identifying step 320 described above. The augmented content 460 is displayed on the microdisplay pair 130 as seen in Fig. 4A. As the method 300 is not implemented by the optical see-through head-mounted display 110 in the example of Fig. 4A, the augmented content 460 when 10806567V1 (P150112_Speci_As Filed)

Ο (N Ο (D Q

(N Γ ΟΟ Ό (N O 10 (N 15 20 17 rendered to the micro-displays 130, resides within a different image plane to that of the real-world object 450a. Thus, the augmented content 460 cannot be focused on by the user at the same time as the real-world object 450a.

In the example of Fig. 4B, the computing device 140 is configured as described with reference to Fig ID. In the example of Fig. 4B, the computing device 140 implements the method 300.

In the example scenario of Fig 4B, the camera 114 captures a raster image of the scene field of view 470 and passes the raster image to the computing device 140 where the method 300 is executed as described above in relation to Fig. 3. The object identification module 141, under execution of the processor 105, identifies augmentable real-world object 450b as per the identifying step 320 described above. The real-world object 450b is visible to the user of the optical see-through head-mounted display 110 and is located at a first image plane from the user. The object identification module 141 also performs the step of selecting an augmentation in the form of augmented content 470 associated with the identified real-world object 450b.

In the example of Fig. 4B, the augmented reality adjustment module 143 then performs the step of forming a digital representation of the identified object 450b, as at steps 330 and 340 of the method 300, for display on the augmented reality system in the form of the optical see-through head-mounted display 110 at a second image plane, the second image plane being different to the first image plane at which the real-world object 450b is visible to the user as described above in relation to Fig. 4B. The second image plane may be referred to as the ‘augmented reality system image plane’.

An augmented reality image is then formed on the optical see-through head-mounted display 110 by displaying the digital representation of the real-world object 450b adjacent to 25 and overlaying the original identified real-world object 450b within the micro-display pair 130 as described in accordance with the method 300. A similar adjustment is made to the augmented content 460 as described in relation to the method 300. The final output on the micro-display pair 130 is the digital augmented content 460 and digital real-world object 450b within the same image plane (i.e., the second image plane described above in relation to Fig. 30 4B) while pass through portions of the display are at real world image planes. 10806567V1 (P150112_Speci_As Filed)

Ο (N Ο (D Q

(N r- 00 Ό (N 12 o (N 15 18

Fig 5 is a schematic diagram of an example showing two real-world objects being viewed through the optical see-through head-mounted display 110 which is implementing the method 300. In the example of Fig 5, the optical see-through head-mounted display 110 comprises the computing device 140 containing the modules 141, 143 and 144 described with reference to Fig. ID; and a HD high definition micro-display pair 130.

In the example of Fig 5, the camera 114 captures a raster image of the scene field of view that contains two real-world objects: object 1 550 and object 2 540. The raster image of the scene (or ‘scene raster image’) is then transmitted to the computing device 140 where the method 600 is being executed. As described in relation to the method 300 of Fig 3, the object identification module 141 identifies any known real-world objects within the scene. In the example depicted in Fig 5, a digital representation of object 1 550 and object 2 540 is registered in the augmented database 142 and object 1 550 contains associated augmented content but object 2 540 does not. The object identification module 141 identifies object 1 550 and object 2 540, stores a POSE for both object 1 550 and object 2 540 and retrieves any associated augmented content. As per the method 300 described above, the digital representation of the object 1 550 and associated augmented content is displayed on the micro-display pair 530 at an altered image plane. Object 2 540 does not require a digital reconstruction and hence does not require image plane altering as no augmented content is associated with object 2 540.

Fig. 6 is a schematic flow diagram showing another method 300 of forming an 20 augmented reality image for display to a user of the optical see-through head-mounted display 110. The method 600 may be implemented as one or more of the code modules 141, 143 and 144 of the software application program 133 resident in the ROM 160 and being controlled in its execution by the processor 105. The method 600 also uses a scene analysis module 145 as seen in Fig. ID. 25 The method 600 begins at capturing step 610. At capturing step 610, the camera 114 on the optical see-through head-mounted display 110 captures a raster image of a scene (or ‘scene raster image’) and transfers the captured raster image of the scene to the object identification module 141 under execution of the processor 105.

Following the capturing step 610, the method 600 proceeds to an identifying step 620. 30 In step 620, the object identification module 141, under execution of the processor 105, identifies any known real-world object within the raster image of the scene. Each identified 10806567V1 (P150112_Speci_As Filed)

ο (N Ο ω Q

(N 00 (N JO 10 o (N 15 19 object may be visible to the user of the optical see-through head-mounted display 110 and is located at a first image plane from the user. Each identified object may be augmentable as described below.

As seen in Fig. 6, a scene analysis step 621 is executed following identifying step 620. At step 621, the scene analysis module 145, under execution of the processor 105, takes as input the raster image of the scene captured at step 610 and then performs the step of determining a scene model as output. The scene model is a data structure containing scene modifications to be applied to digital content that is rendered within the scene: augmented content and digital real-world objects. In the alternate arrangement, the scene model contains a scene modification that applies an average luminance change to digital content that will be displayed in the micro-display pair 130.

In one arrangement, at step 621, the scene analysis module 145 performs the steps of defining a Gaussian window around the real-world object identified at step 620 and then determining the average luminance of pixels within the defined Gaussian image. The average luminance is then added to a change model and is stored in the storage module 109 (e.g., RAM 170).

After the scene analysis step 621, the method 600 proceeds to a registering step 630. In the registering step 630, the augmented reality adjustment module 143, under execution of the processor 105, first determines if an object identified at step 620 contains associated augmented 20 content. If the identified object contains associated augmented content, then the augmented reality adjustment module 143 retrieves the real-world object POSE from the storage module 109 (e.g., RAM 170) for the identified object. The augmented reality adjustment module 143 then copies, into the storage module 109, the pixels within the raster image of the scene that are enclosed by the POSE as a digital representation of the object identified at step 620. 25 In one arrangement, in addition to creating a raster representation of the real-world object as described in relation to Fig. 3, at the registering step 630, the change model is retrieved from the storage module 109 and the luminance change is applied to both the digital real-world object and the augmented content. After the registering step 630, the method 600 proceeds to an applying step 640. 30 In applying step 640, the augmented reality adjustment module 143 determines a hand- eye calibration to apply to the digital representation of the digital real-world object, as 10806567V1 (P150112_Speci_As Filed)

Ο (N Ο (D Q

(N 00 (N in O 10 (N 20 described above, in order to determine how the digital representation of the digital real-world object should be sized and positioned within the micro-display 130.

At displaying step 650, the display module 144, under execution of the processor 105, retrieves the augmented content and the adjusted digital representation of the digital real-world object from the storage module 109 and passes the augmented content and the adjusted digital representation of the digital real-world object to the optical drivers of the optical see-through head-mounted display 110 where the augmented content and the adjusted digital representation of the digital real-world object are displayed on the micro-display 130.

In one arrangement, the augmented reality adjustment module 143 associates the POSE of the real-world object with a corresponding real-world object. The association is performed at step 330 of the method 300. The image registered in the augmented database 142 that is associated with the real-world object is assigned to the POSE of the real-world object, that is determined in step 320. The method 300 then continues as described in Fig. 3.

In one arrangement, the scene analysis module 145 is used to create a scene model 15 which is then applied to the augmented and the digital-real world content, to blend both into the surrounding scene. As described above, the scene analysis module 145 takes as input a raster image of the scene and produces as output a scene model. The scene model can contain one or more image modification factors to apply separately to the augmented content and the digital real-world object. In one arrangement, a separate transparency modification factor is used for 20 both augmented and digital real-world content. Other modification factors such as colour, contrast and brightness, for example, can also be used.

In another arrangement, environment effects (e.g., lighting) may be applied to the augmentation in the form of the augmented content and to the digital real-world object to enable the augmented content and the digital real-world object to blend to each other. 25 In still another arrangement, opacity levels between the augmented content and the digital real-world object may be varied so that the augmented content and the digital real-world object have different opacity. Varying the opacity levels may enable one of the augmented content and the digital real-world object to look more realistic than the other.

In one arrangement, at the analysis step 621, the scene analysis module 145, under 30 execution of the processor 105, creates a transparency factor for the digital real-world content 10806567V1 (P150112_Speci_As Filed) ο (N Ο <D Q in

(N (X) (N in O 10 (N 21 using alpha blending or any other suitable transparency blending methods. The transparency factor of the augmented content is set to be less than or equal to that of the digital real-world content to allow users to distinguish the difference between real-world objects and the augmented content. Once created, the scene model is stored within t the storage module 109 (e g., RAM 170) of the optical see-through head-mounted display 110 and the scene analysis step 621 ends. The display module 144 may retrieve the scene model from the storage module 109 and all scene modification factors may be applied to their respective contents, before passing the content to the optical see-through head-mounted display 110 optical drivers.

Industrial Applicability

The arrangements described are applicable to the computer and data processing industries and particularly for image processing.

The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive. 15 In the context of this specification, the word “comprising” means “including principally but not necessarily solely” or “having” or “including”, and not “consisting only of’. Variations of the word "comprising", such as “comprise” and “comprises” have correspondingly varied meanings. 10806567V1 (P150112_Speci_As Filed)

Claims (16)

1. A method of forming an augmented reality image for display to a user of an optical see-through augmented reality (AR) system, the method comprising: identifying an augmentable object in an image of a scene, the augmentable object being visible to the user and being located at a first image plane from the user; selecting an augmentation associated with the identified augmentable object; forming a digital representation of the identified augmentable object for display on the optical see-through augmented reality system at a second image plane, the second image plane being different to the first image plane; forming an augmented reality image on the augmented reality system by displaying the augmentation on the augmented reality system adjacent to the identified object and overlaying the identified augmentable object, visible at the first image plane, with the digital representation of the identified augmented reality object at the second image plane such that the digital representation and the augmentation are displayed at the second image plane while pass through portions of the display are at real world image planes.

2. The method according to claim 1, further comprising detecting feature points within the image of a scene.

3. The method according to claim 2, further comprising: determining pose of the augmentable object based on the feature points; and determining the digital representation of the identified augmentable object based on the pose.

4. The method according to claim 1, further comprising determining a calibration to apply to the digital representation of the digital representation.

5. The method according to claim 4, wherein the calibration is a hand-eye calibration.

6. The method according to claim 1, further comprising applying a translation to the augmentation.

7. The method according to claim 1, further comprising determining a scene model based on the image of a scene.

8. The method according to claim 7, wherein the scene model contains scene modifications to be applied to the digital representation.

9. The method according to claim 7, wherein the scene modification is configured to apply an average luminance change to the digital representation.

10. The method according to claim 1, further comprising defining a Gaussian window around the identified augmentable object.

11. The method according to claim 10, further comprising determining average luminance of pixels within the defined Gaussian window.

12. The method according to claim 1, further comprising applying environment effects to the augmentation and the digital representation.

13. The method according to claim 1, further comprising varying opacity levels between the augmentation and the digital representation.

14. A system for forming an augmented reality image for display to a user of an optical see-through augmented reality (AR) system, the system comprising: a memory for storing data and a computer program; a processor coupled to the memory for executing said computer program, said computer program comprising instructions for: identifying an augmentable object in an image of a scene, the augmentable object being visible to the user and being located at a first image plane from the user; selecting an augmentation associated with the identified augmentable object; forming a digital representation of the identified augmentable object for display on the optical see-through augmented reality system at a second image plane, the second image plane being different to the first image plane; forming an augmented reality image on the augmented reality system by displaying the augmentation on the augmented reality system adjacent to the identified object and overlaying the identified augmentable object, visible at the first image plane, with the digital representation of the identified augmented reality object at the second image plane such that the digital representation and the augmentation are displayed at the second image plane while pass through portions of the display are at real world image planes.

15. An apparatus for forming an augmented reality image for display to a user of an optical see-through augmented reality (AR) system, the apparatus comprising: module for identifying an augmentable object in an image of a scene, the augmentable object being visible to the user and being located at a first image plane from the user; module for selecting an augmentation associated with the identified augmentable object; module for forming a digital representation of the identified augmentable object for display on the optical see-through augmented reality system at a second image plane, the second image plane being different to the first image plane; module for forming an augmented reality image on the augmented reality system by displaying the augmentation on the augmented reality system adjacent to the identified object and overlaying the identified augmentable object, visible at the first image plane, with the digital representation of the identified augmented reality object at the second image plane such that the digital representation and the augmentation are displayed at the second image plane while pass through portions of the display are at real world image planes.

16. A computer readable medium having a computer program stored on the medium for forming an augmented reality image for display to a user of an optical see-through augmented reality (AR) system, the program comprising: code for identifying an augmentable object in an image of a scene, the augmentable object being visible to the user and being located at a first image plane from the user; code for selecting an augmentation associated with the identified augmentable object; code for forming a digital representation of the identified augmentable object for display on the optical see-through augmented reality system at a second image plane, the second image plane being different to the first image plane; code for forming an augmented reality image on the augmented reality system by displaying the augmentation on the augmented reality system adjacent to the identified object and overlaying the identified augmentable object, visible at the first image plane, with the digital representation of the identified augmented reality object at the second image plane such that the digital representation and the augmentation are displayed at the second image plane while pass through portions of the display are at real world image planes.