EP3599588A1 - Rendering an object - Google Patents
Rendering an object Download PDFInfo
- Publication number
- EP3599588A1 EP3599588A1 EP18186166.7A EP18186166A EP3599588A1 EP 3599588 A1 EP3599588 A1 EP 3599588A1 EP 18186166 A EP18186166 A EP 18186166A EP 3599588 A1 EP3599588 A1 EP 3599588A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- virtual
- light field
- discrete
- camera
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000009877 rendering Methods 0.000 title claims abstract description 39
- 238000000034 method Methods 0.000 claims abstract description 71
- 239000011159 matrix material Substances 0.000 claims description 43
- 230000009466 transformation Effects 0.000 claims description 16
- 230000003190 augmentative effect Effects 0.000 claims description 10
- 230000001131 transforming effect Effects 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 9
- 238000012545 processing Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000005094 computer simulation Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- HPNSNYBUADCFDR-UHFFFAOYSA-N chromafenozide Chemical compound CC1=CC(C)=CC(C(=O)N(NC(=O)C=2C(=C3CCCOC3=CC=2)C)C(C)(C)C)=C1 HPNSNYBUADCFDR-UHFFFAOYSA-N 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 238000004020 luminiscence type Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000010344 pupil dilation Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000014616 translation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/506—Illumination models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2215/00—Indexing scheme for image rendering
- G06T2215/16—Using real world measurements to influence rendering
Definitions
- the present invention relates to a method and an apparatus for rendering an object, and more particularly to a method and apparatus for rendering an object in a virtual three-dimensional space.
- Rendering realistic images of virtual objects can depend on several factors such as lighting, shadowing and material properties of a given object.
- a known technique of lighting an object in a 3D computer model is to assume that a light source is at infinite distance from the object such that the lighting condition is independent of the position of the object in the virtual space.
- a known technique in AR is to capture an ambient colour of the real environment in which a user is situated, and to render a virtual object using the ambient colour.
- the problem still remains that in some environments the lighting condition assumptions will not be appropriate and hence the virtual object will not appear realistic.
- a method of rendering an object in a virtual three-dimensional space comprising: obtaining a first plurality of pixel values of a first image captured by a camera in real space, the first plurality of pixel values being associated with a first three-dimensional position, in the virtual three-dimensional space, of the camera when the first image was captured, each one of the first plurality of pixel values being associated with an orientation in the virtual three-dimensional space, of a portion of a light field represented by the pixel; estimating, using compressive sensing, and based on the first plurality of pixel values, a light field value for at least one second position, different from the first position, in the virtual three-dimensional space; and rendering an object, in the virtual three-dimensional space, located at the second position, using the estimated light field value.
- the method comprises: determining a position and orientation of the camera in real space; and determining, based at least in part on the determined position and orientation of the camera in real space, the first position in the virtual three-dimensional space and the orientation, in the virtual three-dimensional space, with which each pixel value is associated.
- determining the orientation with which each pixel value is associated comprises: projecting the pixel values of the first image onto a hemisphere located at the first position; and determining, based on the projection, the orientation with which each pixel value is associated.
- projecting the pixel values of the first image onto the hemisphere comprises: using equirectangular projection of the pixel values of the first image onto a cosine weighted hemisphere.
- the method comprises: capturing the first image using the camera.
- the camera is mounted to an augmented reality or virtual reality headset.
- rendering the object comprises rendering a representation of the object on a display screen of the augmented reality or virtual reality headset.
- the virtual three-dimensional space is discretised into discrete positions and discrete orientations, each discrete position being associated with a plurality of discrete orientations; wherein each of the one or more orientations with which each pixel value is associated is a discrete orientation, wherein the first position is a first discrete position, and wherein the second position is a second discrete position, different to the first discrete position.
- estimating the light field value using compressive sensing comprises reconstructing, from the first plurality of pixel values associated with the first discrete position, the light field at each of the discrete positions and discrete orientations, by estimating a solution to an underdetermined set of linear equations.
- ⁇ ⁇
- ⁇ ⁇
- ⁇ ⁇
- ⁇ ⁇
- ⁇ ⁇
- ⁇ a representation matrix for transforming e into a sparse domain
- ⁇ a sensing matrix comprising a set of predetermined random values
- the method further comprises applying, after the minimization of the l 1 norm of e, the inverse of ⁇ to e, to obtain the estimate of light field values at all the discretised positions and orientations within the virtual three-dimensional space.
- ⁇ is a Discrete Fourier Transform matrix.
- the light field value comprises a radiance value.
- the method further comprises: obtaining a second plurality of pixel values of a second image captured by the camera in real space, the second image being captured at a position or with an orientation in real space different to that of the camera when the first image was captured; wherein estimating the light field value for the at least one second position is based at least in part on the first plurality of pixel values and the second plurality of pixel values.
- an apparatus arranged to perform the method according to the first aspect.
- the apparatus is an augmented reality or a virtual reality headset.
- a computer readable medium having instructions stored thereon which, when executed on a computer causes the computer to perform the method according to the first aspect.
- a known technique of lighting an object in a 3D computer model is to assume that a light source is at infinite distance from the object such that the lighting condition is independent of the position of the object in the virtual space.
- this technique can lead to the generation of un-realistic images.
- One way to ensure that a lighting condition appropriate for a real environment is used to render a virtual object would be to measure the light field at each position and direction in the real environment.
- this would be costly and time consuming, and would not enable dynamic adaption to changes in the real environment.
- the five-dimensional data set produced (three position coordinates and two direction coordinates) would be large and hence may be time consuming and inefficient to acquire and process. This may be of particular concern, for example, in AR or VR headset devices where processing power and/or memory may be limited. It is therefore desirable to provide for a realistic yet efficient rendering of virtual objects.
- FIG. 1 there is illustrated a method of rendering an object in a virtual three-dimensional space.
- the method may provide for an efficient yet realistic rendering of virtual objects.
- the method comprises, in step 102, obtaining a first plurality of pixel values of a first image captured by a camera in real space.
- the first plurality of pixel values is associated with a first three-dimensional position, in the virtual three-dimensional space, of the camera when the first image was captured, and each one of the first plurality of pixel values is associated with an orientation, in the virtual three-dimensional space, of a portion of a light field represented by the pixel.
- the method comprises, in step 104, estimating, using compressive sensing, and based on the first plurality of pixel values, a light field value (e.g. radiance value) for at least one second position, different from the first position, in the virtual three-dimensional space.
- a light field value e.g. radiance value
- the method comprises, in step 106, rendering a virtual object located at the second position, in the virtual three-dimensional space, using the estimated light field value.
- rendering an object in virtual space using a light field value e.g. radiance value
- a light field value e.g. radiance value
- compressive sensing and based on pixel values from an image captured by a camera in real space may allow for an efficient yet realistic rendering of the object, i.e. one which accounts for the actual lighting conditions of the real environment.
- the real space 200 has a real-space cartesian coordinate system defined by x r , y r , z r .
- the 3D space 200 contains a user 202.
- the user is wearing an augmented reality (hereinafter AR) or virtual reality (hereinafter VR) headset 204 (see also the headset 204 described below in relation to Figure 9 ).
- the headset 204 comprises a camera (not shown in Figure 2 but see an example camera 250 is described below in relation to Figure 9 ).
- the focal axis 206 of the camera is parallel, and in some example may be substantially colinear, with a line of sight of the user 202.
- the camera is arranged to capture images of the real space 200 environment. For example, the camera may capture the first image referenced with respect to step 102 of Figure 1 .
- the first image 208 is a two-dimensional (hereinafter 2D) image.
- the first image 208 comprises a first plurality of pixels, each having a pixel value.
- Each pixel value is taken to represent the portion of the light field (e.g. a radiance value) of the real space 200, captured by the camera, at the pixel.
- the location of each pixel within the image is defined by 2D cartesian coordinate system p, q.
- FIG. 4 there is illustrated schematically an example virtual 3D space 200'.
- the virtual space has a virtual-space cartesian coordinate system defined by x, y, z.
- the virtual 3D space 200' is discretised. Specifically, the virtual space 200' is discretised into equal volume portions, each having a discrete position in the virtual space. Each discrete position within the virtual space may be given as an indexed coordinate as x i , y j , z k .
- Figure 4 illustrates a first three-dimensional position x 1 , y 1 , z 1 , in the virtual three-dimensional space 200', of the camera when the first image was captured. That is, the first three-dimensional position x 1 , y 1 , z 1 represents the position of the camera (that is in real space 200) in the virtual space 200'.
- the method may comprise determining the position of the camera in real space 200, and transforming the determined position to the position of the camera in virtual space 200' (i.e. the first position x 1 , y 1 , z 1 ).
- determining the position of the camera in virtual space may comprise determining an absolute position and/or a relative position of the camera in real space 200.
- the headset 204 may receive data from one or more position sensors (not shown), for example infrared position sensors, located in the real space 200, arranged to determine a position in real space 200 of the headset 204.
- the position of the headset 204 relative to the sensors may be determined, for example by trilateration or the like.
- the absolute position of the sensors in the real space 200 may be known and fixed.
- the absolute position of the headset 204 in real space 200 may then be determined based on the absolute position of the sensors and the position of the headset 204 relative to the sensors.
- the headset 204 may comprise means for determining the position of the headset 204 relative to some initial position of the headset 204.
- the headset 204 may comprise an accelerometer and a gyroscope arranged to track movement of the headset 204 in the real space 200 from a given initial position (which may be known). The position of the headset 204 at a given subsequent time may therefore be determined based on the initial position and the tracked movement of the headset since the headset 204 was in the initial position.
- the camera may be a depth camera, for example a RGBD (Red Green Blue Depth) camera. The depth information may be used to map and identify features of the real space environment 200. The position and orientation of the camera may then be determined and tracked relative to a given feature of the real space environment, thereby providing the position and orientation of the camera in real space 200.
- RGBD Red Green Blue Depth
- the real space 200 and the virtual space 200' coordinate systems may be or may be assumed to be coincident, and in this case, a transformation between them may not be performed or may not change the real space coordinates.
- a coordinate transformation may be performed to transform the position of the camera in real space 200 to the position of the camera as represented in virtual space 200'.
- the coordinate transformation may be achieved by applying suitable operations to the real space coordinate system, for example by applying suitable translations, rotations, and/or scaling.
- each discrete position within the virtual space may be given as an indexed coordinate as x i , y j , z k .
- the virtual space is discretised into discrete orientations.
- each discrete position x i , y j , z k is associated with a plurality of discretised orientations, which in total may cover all discrete orientations within the 3D virtual space associated with the discrete position.
- each discrete orientation may be defined using two angles ⁇ and ⁇ , where ⁇ is the angle that a direction associated with the orientation makes with the x axis, and ⁇ is the angle that the direction makes with the y axis.
- the orientations may be discretised such that each discrete orientation, associated with a given discrete position x i , y j , z k within the virtual space 200', may be given as indexed angles ⁇ l , ⁇ m .
- Figure 4 illustrates a first direction 206', in the virtual 3D space, of the camera when the first image is captured.
- the first direction 206' may be substantially colinear with the focal axis of the camera, as it is determined to be in virtual space 200'.
- the first direction 206' is defined by a first orientation ⁇ 1 , ⁇ 1 . That is, ⁇ 1 is the angle that the first direction 206' makes with the y axis, and ⁇ 1 is the angle that the first direction 206' makes with the x axis.
- the first orientation ⁇ 1 , ⁇ 1 may therefore be thought of as representative of the orientation (in virtual space 200') of the camera when the first image is captured.
- the method may comprise determining the orientation of the camera in real space 200, and transforming the determined orientation to the orientation of the camera in virtual space 200' (i.e. the first orientation ⁇ 1 , ⁇ 1 ). Determining the orientation of the camera in real space may be performed in a similar way as described for the position, for example by using a suitable internal gyroscope, or by suitable external infrared sensors, and/or by using depth information e.g. in cases where the camera is a depth camera. Similarly, determining the orientation of the camera in virtual space 200' may comprise applying a transformation to the orientation in real space 200, for example using rotation operations as appropriate.
- the position and orientation of the camera, when the first image was taken, in virtual space 200' may therefore be given by a 5-dimensional coordinate x 1 , y 1 , z 1 , ⁇ 1 , ⁇ 1 .
- all of the discretised positions and orientations within the virtual space 200' may be given by a 5-dimensional indexed coordinate x i , y j , z k , ⁇ l , ⁇ m .
- the first plurality of pixel values are from the first image 208 captured by a camera in real space 200.
- the first plurality of pixel values is associated with the first three-dimensional position x 1 , y 1 , z 1 , in the virtual three-dimensional space, of the camera when the first image 208 was captured.
- Each one of the first plurality of pixel values is associated with an orientation, in the virtual three-dimensional space, of a portion of a light field represented by the pixel (as described hereinafter).
- the first image 208 is associated with an orientation ⁇ 1 , ⁇ 1 in virtual space 200' of the camera when the first image 208 was captured.
- Figure 5 illustrates schematically an example way of determining an orientation, in the virtual three-dimensional space, of a portion of a light field represented by each pixel value.
- the method comprises projecting the pixel values of the first image 208 onto a hemisphere 210 located at the first three-dimensional position x 1 , y 1 , z 1 ; and determining, based on the projection, the orientation ⁇ l , ⁇ m with which each pixel value is associated.
- Each pixel value, having planar coordinates p, q, of the 2D first image 208 may be projected or mapped onto a point on the surface 208' of the 3D hemisphere 210 according to the position p,q of that pixel in the 2D image.
- the hemisphere 210 may be constructed so that its principal axis is colinear with the first direction 206' (having an orientation defined by ⁇ 1 , ⁇ 1 ).
- the first image 208 may be projected onto the hemisphere 210 for example using equirectangular projection, in which the planar coordinates p, q of each pixel of the first image 208 is mapped to spherical coordinates ⁇ ', ⁇ ' of the hemisphere.
- the pixel values mat be projected (e.g. using equirectangular projection) onto a cosine weighted hemisphere.
- the projection of planar coordinates p, q onto the spherical coordinates ⁇ ', ⁇ ' of the hemisphere 210 is weighted so that proportionally more pixel values are projected to smaller angles ⁇ ', ⁇ ' than to larger angles ⁇ ', ⁇ ' (relative to the first direction 206').
- the spherical coordinates ⁇ ', ⁇ ' associated with a pixel value following the projection may be used to define the orientation in the virtual 3D space 200' associated with that pixel value.
- the orientation associated with each pixel value may be given by the indexed orientation ⁇ l , ⁇ m . Therefore, each pixel value of the first image 208 may be associated with a 5-dimensional indexed coordinate x 1 , y 1 , z 1 , ⁇ l , ⁇ m .
- each pixel value is taken as a value representative of a portion of a light field at the position and orientation x 1 , y 1 , z 1 , ⁇ l , ⁇ m of the pixel value in virtual space 200'.
- the pixel value may be taken as a radiance, or amount of light, at the position and orientation x 1 , y 1 , z 1 , ⁇ l , ⁇ m of the pixel value in virtual space 200'.
- the light field at each of the other discretised positions and orientations x i , y j , z k , ⁇ l , ⁇ m in the virtual space 200' may be unknown.
- an estimate of one or more light field values at that second position should be determined (as described hereinafter).
- the method comprises estimating, based on the first plurality of pixel values, a light field value for at least one second position x 2 , y 2 , z 2 , different from the first position x 1 , y 1 , z 1 , in the virtual three-dimensional space 200'.
- the light field value may be the radiance, or an amount of light, from a given direction 212 having a given orientation ⁇ 2 , ⁇ 2 at the second position x 2 , y 2 , z 2 .
- the estimate is made using compressive sensing.
- the method comprises rendering an object 214, in the virtual three-dimensional space 200, located at the second position x 2 , y 2 , z 2 , using the estimated light field value.
- a virtual object 214 in this example a cuboid
- a virtual object 214 is to be rendered at a second position x 2 , y 2 , z 2 , so that virtual object 214 may appear, via the AR or VR headset 204 (not shown in Figure 7 ), as present at the second position x 2 , y 2 , z 2 , to the user 202 (not shown in Figure 7 ).
- Figure 8 illustrates schematically an example of an image 216 comprising the first image 208 augmented, for example overlaid, with an image 214' of the object 214 rendered using the estimated light field values.
- the image 216 may be a virtual image, which may for example be derived from the first image 208, and which includes the image 214' of the object 214 rendered using the estimated light field values.
- the example image 216 may, for example, be displayed on a display screen of the AR or VR headset 204 such that the virtual object 214 may appear as present at the second position x 2 , y 2 , z 2 , from the perspective of the user.
- the estimated light field value e.g. a radiance value, or other lighting condition value, at the second position x 2 , y 2 , z 2 , and having a given orientation ⁇ 2 , ⁇ 2 , may be used in the rendering of the virtual object 214 on a display screen of the VR or AR headset of the user (not shown in Figure 7 ).
- the light field value may be used as a lighting condition of the virtual object 214 during rendering of the virtual object 214.
- the rendering of the virtual object 214 located in the virtual space 200' may be based on an estimate of the lighting condition that the virtual object 214 would experience if it were in the real space 200. A realistic rendering of the virtual object 214 may therefore be generated.
- the estimation of the light field value for the second position x 2 , y 2 , z 2 from the plurality of first pixel values uses compressive sensing.
- Compressive sensing also known as compressed sensing, compressive sampling, or sparse sampling is a signal processing technique for efficiently reconstructing a signal from sample values of the signal by finding a solution to an underdetermined system of linear equations by optimisation.
- An underdetermined system of linear equations has more unknowns than equations and as such may have many, and in some cases infinite, solutions.
- the signal to be recovered is the light field at each position and orientation x i , y j , z k , ⁇ l , ⁇ m in the discretised virtual space 200'
- the sample values are the light field values represented by the first plurality of pixels at the first position x 1 , y 1 , z 1 , ⁇ l , ⁇ m
- the system of linear equations is that which transforms the latter to the former.
- the signal to be reconstructed is assumed to be sparse in some domain (which may be the same as or different to the domain in which the samples are captured).
- a condition is applied to the solutions of the system of underdetermined linear equations that the solution must be sparse, that is, that only solutions having a small number (or indeed only the solution having the smallest number) of non-zero coefficients are/is accepted as solutions (or the solution) to the system of linear equations.
- Light field images are sparse. Therefore, light field values may be considered as sparse signals, for example as they are not pure noise.
- estimating the light field value at the second position x 2 , y 2 , z 2 , ⁇ l , ⁇ m using compressive sensing comprises reconstructing, from the first plurality of pixel values associated with the first position x 1 , y 1 , z 1 , ⁇ l ⁇ m the light field value at each of the discrete positions and discrete orientations, by estimating a solution to an underdetermined set of linear equations.
- the solution is estimated by determining the solution with a small, for example, the smallest number of non-zero coefficients. This may be performed using optimisation techniques, such as l 1 norm minimisation, described in more detail hereinafter.
- a system of linear equations may be represented in matrix form.
- f is a s ⁇ 1 (column) vector of the plurality of first pixel values indexed for the first discrete position x 1 , y 1 , z 1 and for the discrete orientations ⁇ l ⁇ m with which each pixel value is associated.
- f may be a column vector whose entries correspond to the first plurality of pixel values stacked in a predetermined order (e.g. indexed for each discrete orientation with which the pixel value is associated) .
- e is an t ⁇ 1 vector, where s ⁇ t, of estimated light field values indexed for all of the discrete positions and discrete orientations x i , y j , z k , ⁇ l , ⁇ m into which the virtual three-dimensional space 200' is discretised.
- e may be a column vector whose entries correspond to (to be determined) light field values stacked in a predetermined order (e.g. indexed for each discrete position and orientation with which the light field value is associated).
- the vector e may be populated with random values.
- ⁇ is a s ⁇ t transformation matrix for reconstructing e from f.
- the transformation matrix may be a single matrix for reconstructing e from f.
- the transformation matrix may be a sensing matrix.
- the sensing matrix may be a matrix Gaussian distribution, that is, a matrix populated with real-valued random numbers taken from a Gaussian distribution (independent standard Gaussian random variables) .
- the sensing matrix may be a random permutation matrix, for example an identity matrix that is subject to a random permutation.
- the l 1 norm may be regarded as a function that counts the number of non-zero coefficients of a vector.
- an algorithm may be applied that minimises ⁇ e ⁇ 1 subject to ⁇ e - f ⁇ 2 ⁇ ⁇ , where ⁇ is a predetermined error parameter, and ⁇ e - f ⁇ 2 represents the l 2 norm of ⁇ e - f , i.e.
- the algorithm will return a solution close to eo.
- the l 1 norm may be minimised using other algorithms or techniques, for example using line search or gradient search techniques.
- the transformation matrix ⁇ may be a combination (for example include a multiplication of) two or more matrices.
- the transformation matrix ⁇ may comprise a matrix multiplication of a sensing matrix ⁇ (which may for example be the same as the sensing matric described above, for example a matrix Gaussian distribution), and a representation matrix ⁇ for transforming e into a sparse domain.
- ⁇ ⁇ .
- Using the representation matrix ⁇ in addition to the sensing matrix ⁇ may provide that the vector e may be more sparsely represented, which may provide for faster and or more accurate minimisation of the l 1 norm of e.
- the representation matrix ⁇ may be an orthogonal basis matrix.
- the representation matrix ⁇ may be a Discrete Fourier Transform (DFT) matrix.
- the Discrete Fourier Transform matrix may be a 5-dimensional DFT matrix, one dimension each for x i , y j , z k , ⁇ l , ⁇ m .
- the DFT matrix may be populated with Fourier transform coefficients of 5-dimensional (discrete) Fourier basis functions (i.e. each entry of the DFT matrix is the i,j,k,l,mth coefficient of a 5-dimensional discrete Fourier transform).
- the Fourier transform may be of the type used for periodic and discrete signals.
- other basis functions may be used for the representation matrix, for example Discrete Cosine Transform bases, Wavelet transform bases or 3D Zernike Descriptor bases.
- the resultant e will be in the spare domain into which it was transformed by the application of the transformation matrix ⁇ .
- the transformation matrix ⁇ is an orthogonal basis matrix
- the inverse of ⁇ may be applied to the solution e, in order to return the estimate of the light field values at all the discretised positions and orientations x i , y j , z k , ⁇ l , ⁇ m within the virtual three-dimensional space 200'.
- the method may comprise applying, after the minimization of the l 1 norm of e, the inverse ⁇ -1 of the transformation matrix ⁇ to e, to obtain the estimate of light field values at all the discretised positions and orientations x i , y j , z k , ⁇ l , ⁇ m within the virtual three-dimensional space 200'.
- an estimate of light field values at all the discretised positions and orientations x i , y j , z k , ⁇ l , ⁇ m within the virtual three-dimensional space 200' is obtained.
- the method may comprise determining the second position x 2 , y 2 , z 2 at which the virtual object 214 is to be rendered, obtaining the estimated light field values (e.g.
- the estimated light field values are used to render the virtual object 214, for example for display on a display screen of the VR or AR headset 204 of the user.
- a virtual object 214 may be rendered using light field values that accurately represent the lighting condition, at the position of the virtual object 214, as if that virtual object were in real space 200.
- the virtual object 214 may therefore be rendered realistically.
- the light field values used to render the virtual object 214 are estimated based on pixel values from an image captured by a camera mounted on the VR or AR headset 204. This may allow for the realistic rendering of the virtual object 214 to be performed efficiently, for example without the need to measure light field values at the second location at which the virtual object 214 is rendered. This may allow the processing of the light field values for rendering to be performed relatively quickly and/or using relatively few resources, which may be important in portable VR or AR headsets 204, in which processing resources may be limited.
- the above described process may be carried out successively for different first images captured over time.
- the rendering of the virtual object 214 may performed dynamically based on the successive first images and the estimated light field values derived therefrom, as described above.
- the method may therefore provide for accurate and efficient rendering that is capable of accounting for dynamic changes to the lighting environment in the real space 200. This may provide for a more realistic experience.
- further images captured by the camera of the headset 204 may be used in addition to the first image captured by the camera of the headset 204 to provide further sample values for the compressive sensing and hence for a more accurate estimate of the light field values at all the discretised positions and orientations x i , y j , z k , ⁇ l , ⁇ m within the virtual three-dimensional space 200'.
- the method may further comprise obtaining a second plurality of pixel values of a second image captured by the camera in real space 200, the second image being captured at a position or with an orientation in real space 200 different to that of the camera when the first image was captured.
- Estimating the light field value for the at least one second position may then be based at least in part on the first plurality of pixel values and the second plurality of pixel values.
- the vector f may be increased in length to include the pixel values indexed for the position and orientation x i , y j , z k , ⁇ l , ⁇ m associated with each of each the pixel values of the first image and of the second image, to produce vector f'.
- the solution of the equation that was found to minimise the l 1 norm of e when only the first plurality of pixel values of the first image was used may now be used as the starting values of e for the l 1 norm minimisation of e.
- further pluralities of pixel values may be added to the sample vector f. This may accordingly improve the accuracy with which the light field values at each position and orientation in the virtual space 200' may be estimated. Further, the more sample values that are used in f, the closer the solution for a given l 1 norm minimisation will be to the solution of a subsequent l 1 norm minimisation using further pixel values, which may successively reduce the time needed for l 1 norm minimisation as successive sample images as captured. This may further improve the efficiency of the light field estimation and hence rendering of virtual objects 214.
- having an estimate of the light field values at all the discretised positions and orientations x i , y j , z k , ⁇ l , ⁇ m in the virtual space 200' may allow for rendering of the virtual object 214 to include light field-based effects, such as aperture effects (i.e. simulating the focal point of a human eye or the like), or High Dynamic Range (HDR) effects (e.g. eye pupil dilation effects to enforce a range of luminescence similar to that experienced by a human eye) and the like. This may further improve the realism of the rendering of the virtual object 214.
- aperture effects i.e. simulating the focal point of a human eye or the like
- HDR High Dynamic Range
- the apparatus 204 may be arranged to perform a method described above with reference to Figures 1 to 8 .
- the apparatus 204 is the AR or VR headset 204, although it will be appreciated that in other examples the apparatus may take different forms.
- the method may be performed by a device (not shown) (e.g. a separate device) in communication with the headset 204.
- the headset 204 may be arranged to send images to the remote device (not shown) and receive from the device an image (e.g. including an image of a virtual object rendered using light field values estimated by the device using the techniques described above based on the first image received by the device) for display on the headset 204 to the user 202.
- the headset 204 comprises a body 258 for mounting to a head of a user 202 of the headset 204.
- the headset 204 comprises a position sensor element 260, camera 250, a display such as a display screen 252, a processor 254 and a memory 256.
- the processor 254 is communicatively connected to the position sensor element 260, the memory 256, the camera 250 and the display 252.
- the focal axis of the camera 250 may be parallel, for example substantially co-linear with an eyeline of the user.
- the position sensor element 260 is arranged to determine the position and orientation of the camera 250 in real space 200, for example using the methods described above.
- the memory 256 may store instructions which when executed by the processor 254 cause the processor 254 (and in some cases the camera 250 and the display 252) to perform the method described above with reference to Figures 1 to 8 .
- the processor 254 may be arranged to obtain a first image 208 captured by the camera 250, and to determine a first plurality of pixel values of the first image 208.
- the processor 254 may be arranged to associate the first plurality of pixel values with a first three-dimensional position x 1 , y 1 , z 1 , in the virtual three-dimensional space 200', of the camera 250 when the first image 208 was captured (for example as determined or informed by the position sensor element 260), for example using the techniques as described above.
- the processor 254 may be arranged to associate each one of the first plurality of pixel values with an orientation ⁇ l , ⁇ m , in the virtual three-dimensional space 200', of a portion of a light field represented by the pixel, for example as described above.
- the processor 254 may be arranged to estimate, using compressive sensing, and based on the first plurality of pixel values, a light field value for at least one second position x 2 , y 2 , z 2 , different from the first position x 1 , y 1 , z 1 , in the virtual three-dimensional space 200', for example using the techniques as described above.
- the processor 254 may be arranged to render a virtual object 214, in the virtual three-dimensional space 200', located at the second position x 2 , y 2 , z 2 , using the estimated light field value, for example using the techniques as described above.
- the processor 254 may be arranged to control the display 252 to display an image of the virtual object 214 having been rendered using the light field values estimated as described above.
- the processor may be arranged to control the display 252 to display an image captured by the camera 250 but augmented with a virtual object 214 having been rendered using the light field values estimated as described above.
- the display 252 may display a virtual image, which may for example be derived from the image captured by the camera 250, and which includes an image of an object 214 rendered using the estimated light field values.
- the headset 204 may provide that a virtual object 214 may be rendered using light field values that accurately represent the lighting condition, at the position of the virtual object 214, as if that virtual object were in real space 200.
- the virtual object 214 may therefore be rendered realistically, for example especially in closed real space environments such as operation rooms or classrooms.
- the light field values used to render the virtual object 214 are estimated based on pixel values from an image captured by the camera 250 mounted on the VR or AR headset 204. This may allow for the realistic rendering of the virtual object 214 to be performed efficiently, for example without the need to measure light field values at the second location at which the virtual object 214 is rendered.
- This may allow the processing of the light field values for rendering to be performed relatively quickly and/or using relatively few resources, which may allow the headset 204 to be lighter and/or consume less power and/or extend a battery life of the headset 204, for example.
- the headset 204 may be arranged to estimate light field values and render virtual objects using successive images captured by the camera 250, for example using the techniques as described above. As described, this may improve the accuracy and/or efficiency with which the light field values at each position and orientation in the virtual space 200' may be estimated, and hence may improve the efficiency and/or realism of the rendering.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Graphics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Geometry (AREA)
- Processing Or Creating Images (AREA)
- Image Generation (AREA)
Abstract
Description
- The present invention relates to a method and an apparatus for rendering an object, and more particularly to a method and apparatus for rendering an object in a virtual three-dimensional space.
- Rendering realistic images of virtual objects, such as from three-dimensional (3D) computer models for augmented reality (AR) or virtual reality (VR) applications, can depend on several factors such as lighting, shadowing and material properties of a given object.
- A known technique of lighting an object in a 3D computer model is to assume that a light source is at infinite distance from the object such that the lighting condition is independent of the position of the object in the virtual space.
- However, this technique can lead to the generation of un-realistic images. For example, in closed environments such as in medical operation rooms, it can be important that an object is rendered under appropriate lighting conditions in order to appear realistic.
- A known technique in AR is to capture an ambient colour of the real environment in which a user is situated, and to render a virtual object using the ambient colour. However, the problem still remains that in some environments the lighting condition assumptions will not be appropriate and hence the virtual object will not appear realistic.
- It is therefore desirable to provide for a realistic yet efficient rendering of virtual objects.
- According to a first aspect of the present invention, there is provided a method of rendering an object in a virtual three-dimensional space, the method comprising: obtaining a first plurality of pixel values of a first image captured by a camera in real space, the first plurality of pixel values being associated with a first three-dimensional position, in the virtual three-dimensional space, of the camera when the first image was captured, each one of the first plurality of pixel values being associated with an orientation in the virtual three-dimensional space, of a portion of a light field represented by the pixel; estimating, using compressive sensing, and based on the first plurality of pixel values, a light field value for at least one second position, different from the first position, in the virtual three-dimensional space; and rendering an object, in the virtual three-dimensional space, located at the second position, using the estimated light field value.
- Optionally, the method comprises: determining a position and orientation of the camera in real space; and determining, based at least in part on the determined position and orientation of the camera in real space, the first position in the virtual three-dimensional space and the orientation, in the virtual three-dimensional space, with which each pixel value is associated.
- Optionally, determining the orientation with which each pixel value is associated comprises: projecting the pixel values of the first image onto a hemisphere located at the first position; and determining, based on the projection, the orientation with which each pixel value is associated.
- Optionally, projecting the pixel values of the first image onto the hemisphere comprises: using equirectangular projection of the pixel values of the first image onto a cosine weighted hemisphere.
- Optionally, the method comprises: capturing the first image using the camera.
- Optionally, the camera is mounted to an augmented reality or virtual reality headset.
- Optionally, rendering the object comprises rendering a representation of the object on a display screen of the augmented reality or virtual reality headset.
- Optionally, the virtual three-dimensional space is discretised into discrete positions and discrete orientations, each discrete position being associated with a plurality of discrete orientations; wherein each of the one or more orientations with which each pixel value is associated is a discrete orientation, wherein the first position is a first discrete position, and wherein the second position is a second discrete position, different to the first discrete position.
- Optionally, estimating the light field value using compressive sensing comprises reconstructing, from the first plurality of pixel values associated with the first discrete position, the light field at each of the discrete positions and discrete orientations, by estimating a solution to an underdetermined set of linear equations.
- Optionally, estimating the light field value using compressive sensing comprises: minimizing the l1 norm of e in the equation f = Θe; where: f is an s×1 vector of the plurality of first pixel values indexed for the first discrete position and for the discrete orientations with which each pixel value is associated; e is a t×1 vector, where s<t, of estimated light field values indexed for all of the discrete positions and discrete orientations into which the virtual three-dimensional space is discretised; and Θ is a s×t transformation matrix for reconstructing e from f.
- Optionally, Θ = ΦΨ, where: Ψ is a representation matrix for transforming e into a sparse domain; and Φ is a sensing matrix comprising a set of predetermined random values; and wherein the method further comprises applying, after the minimization of the l1 norm of e, the inverse of Ψ to e, to obtain the estimate of light field values at all the discretised positions and orientations within the virtual three-dimensional space.
- Optionally, Ψ is a Discrete Fourier Transform matrix.
- Optionally, the light field value comprises a radiance value.
- Optionally, the method further comprises: obtaining a second plurality of pixel values of a second image captured by the camera in real space, the second image being captured at a position or with an orientation in real space different to that of the camera when the first image was captured; wherein estimating the light field value for the at least one second position is based at least in part on the first plurality of pixel values and the second plurality of pixel values.
- According to a second aspect of the present invention, there is provided an apparatus arranged to perform the method according to the first aspect.
- Optionally, the apparatus is an augmented reality or a virtual reality headset.
- According to a third aspect of the present invention, there is provided a computer readable medium having instructions stored thereon which, when executed on a computer causes the computer to perform the method according to the first aspect.
- Further features and advantages of the invention will become apparent from the following description of preferred embodiments of the invention, given by way of example only, which is made with reference to the accompanying drawings.
-
-
Figure 1 is a flow diagram that illustrates a method if rendering an object according to an example. -
Figure 2 is a schematic diagram that illustrates a real space containing a person wearing an AR/VR headset according to an example. -
Figure 3 is a schematic diagram that illustrates a virtual space corresponding to the real space ofFigure 2 , and a first position in the virtual space, according to an example. -
Figure 4 is a schematic diagram that illustrates an image captured by a camera of the AR/VR headset in the real space according to an example. -
Figure 5 is a schematic diagram that illustrates the virtual space ofFigure 3 with the image ofFigure 4 projected onto a hemisphere, according to an example. -
Figure 6 is a schematic diagram that illustrates the virtual space ofFigure 3 but including an estimated light field value for a second position, according to an example. -
Figure 7 is a schematic diagram that illustrates the virtual space ofFigure 6 with a virtual object at the second position to be rendered using the estimated light field value, according to an example. -
Figure 8 is a schematic diagram that illustrates an image for display on the AR/VR headset ofFigure 2 , including the object rendered using the estimated light field value, according to an example. -
Figure 9 is a schematic diagram that illustrates an apparatus, for example a AR/VR headset, according to an example. - A known technique of lighting an object in a 3D computer model is to assume that a light source is at infinite distance from the object such that the lighting condition is independent of the position of the object in the virtual space. However, this technique can lead to the generation of un-realistic images. One way to ensure that a lighting condition appropriate for a real environment is used to render a virtual object would be to measure the light field at each position and direction in the real environment. However, this would be costly and time consuming, and would not enable dynamic adaption to changes in the real environment. Further, the five-dimensional data set produced (three position coordinates and two direction coordinates) would be large and hence may be time consuming and inefficient to acquire and process. This may be of particular concern, for example, in AR or VR headset devices where processing power and/or memory may be limited. It is therefore desirable to provide for a realistic yet efficient rendering of virtual objects.
- Referring to
Figure 1 , there is illustrated a method of rendering an object in a virtual three-dimensional space. The method may provide for an efficient yet realistic rendering of virtual objects. - In broad overview, the method comprises, in
step 102, obtaining a first plurality of pixel values of a first image captured by a camera in real space. The first plurality of pixel values is associated with a first three-dimensional position, in the virtual three-dimensional space, of the camera when the first image was captured, and each one of the first plurality of pixel values is associated with an orientation, in the virtual three-dimensional space, of a portion of a light field represented by the pixel. - The method comprises, in
step 104, estimating, using compressive sensing, and based on the first plurality of pixel values, a light field value (e.g. radiance value) for at least one second position, different from the first position, in the virtual three-dimensional space. - The method comprises, in
step 106, rendering a virtual object located at the second position, in the virtual three-dimensional space, using the estimated light field value. - As explained in more detail hereinafter, rendering an object in virtual space using a light field value (e.g. radiance value) estimated using compressive sensing and based on pixel values from an image captured by a camera in real space, may allow for an efficient yet realistic rendering of the object, i.e. one which accounts for the actual lighting conditions of the real environment.
- Referring to
Figure 2 , there is illustrated schematically an example real three-dimensional (hereinafter 3D)space 200. The real space has a real-space cartesian coordinate system defined by xr, yr, zr. The3D space 200 contains auser 202. The user is wearing an augmented reality (hereinafter AR) or virtual reality (hereinafter VR) headset 204 (see also theheadset 204 described below in relation toFigure 9 ). Theheadset 204 comprises a camera (not shown inFigure 2 but see anexample camera 250 is described below in relation toFigure 9 ). In this example, thefocal axis 206 of the camera is parallel, and in some example may be substantially colinear, with a line of sight of theuser 202. The camera is arranged to capture images of thereal space 200 environment. For example, the camera may capture the first image referenced with respect tostep 102 ofFigure 1 . - Referring to
Figure 3 , there is illustrated schematically an examplefirst image 208, for example captured by the camera. Thefirst image 208 is a two-dimensional (hereinafter 2D) image. Thefirst image 208 comprises a first plurality of pixels, each having a pixel value. Each pixel value is taken to represent the portion of the light field (e.g. a radiance value) of thereal space 200, captured by the camera, at the pixel. The location of each pixel within the image is defined by 2D cartesian coordinate system p, q. - Referring to
Figure 4 , there is illustrated schematically an example virtual 3D space 200'. The virtual space has a virtual-space cartesian coordinate system defined by x, y, z. Although not explicitly shown inFigure 4 , the virtual 3D space 200' is discretised. Specifically, the virtual space 200' is discretised into equal volume portions, each having a discrete position in the virtual space. Each discrete position within the virtual space may be given as an indexed coordinate as xi, yj, zk. -
Figure 4 illustrates a first three-dimensional position x1, y1, z1, in the virtual three-dimensional space 200', of the camera when the first image was captured. That is, the first three-dimensional position x1, y1, z1 represents the position of the camera (that is in real space 200) in the virtual space 200'. - In some examples, the method may comprise determining the position of the camera in
real space 200, and transforming the determined position to the position of the camera in virtual space 200' (i.e. the first position x1, y1, z1). - In some examples, determining the position of the camera in virtual space may comprise determining an absolute position and/or a relative position of the camera in
real space 200. For example, theheadset 204 may receive data from one or more position sensors (not shown), for example infrared position sensors, located in thereal space 200, arranged to determine a position inreal space 200 of theheadset 204. The position of theheadset 204 relative to the sensors may be determined, for example by trilateration or the like. The absolute position of the sensors in thereal space 200 may be known and fixed. The absolute position of theheadset 204 inreal space 200 may then be determined based on the absolute position of the sensors and the position of theheadset 204 relative to the sensors. In another example, theheadset 204 may comprise means for determining the position of theheadset 204 relative to some initial position of theheadset 204. For example, theheadset 204 may comprise an accelerometer and a gyroscope arranged to track movement of theheadset 204 in thereal space 200 from a given initial position (which may be known). The position of theheadset 204 at a given subsequent time may therefore be determined based on the initial position and the tracked movement of the headset since theheadset 204 was in the initial position. As another example, the camera may be a depth camera, for example a RGBD (Red Green Blue Depth) camera. The depth information may be used to map and identify features of thereal space environment 200. The position and orientation of the camera may then be determined and tracked relative to a given feature of the real space environment, thereby providing the position and orientation of the camera inreal space 200. - In some examples, the
real space 200 and the virtual space 200' coordinate systems may be or may be assumed to be coincident, and in this case, a transformation between them may not be performed or may not change the real space coordinates. However, in some examples, a coordinate transformation may be performed to transform the position of the camera inreal space 200 to the position of the camera as represented in virtual space 200'. The coordinate transformation may be achieved by applying suitable operations to the real space coordinate system, for example by applying suitable translations, rotations, and/or scaling. - As described, the virtual space 200' is discretised so that each discrete position within the virtual space may be given as an indexed coordinate as xi, yj, zk. Further, the virtual space is discretised into discrete orientations. Specifically, each discrete position xi, yj, zk is associated with a plurality of discretised orientations, which in total may cover all discrete orientations within the 3D virtual space associated with the discrete position. For example, each discrete orientation may be defined using two angles θ and Φ, where θ is the angle that a direction associated with the orientation makes with the x axis, and Φ is the angle that the direction makes with the y axis. The orientations may be discretised such that each discrete orientation, associated with a given discrete position xi, yj, zk within the virtual space 200', may be given as indexed angles Φl, θm.
-
Figure 4 illustrates a first direction 206', in the virtual 3D space, of the camera when the first image is captured. The first direction 206' may be substantially colinear with the focal axis of the camera, as it is determined to be in virtual space 200'. The first direction 206' is defined by a first orientation Φ1, θ1. That is, Φ1 is the angle that the first direction 206' makes with the y axis, and θ1 is the angle that the first direction 206' makes with the x axis. The first orientation Φ1, θ1 may therefore be thought of as representative of the orientation (in virtual space 200') of the camera when the first image is captured. - In some examples, the method may comprise determining the orientation of the camera in
real space 200, and transforming the determined orientation to the orientation of the camera in virtual space 200' (i.e. the first orientation Φ1, θ1). Determining the orientation of the camera in real space may be performed in a similar way as described for the position, for example by using a suitable internal gyroscope, or by suitable external infrared sensors, and/or by using depth information e.g. in cases where the camera is a depth camera. Similarly, determining the orientation of the camera in virtual space 200' may comprise applying a transformation to the orientation inreal space 200, for example using rotation operations as appropriate. - The position and orientation of the camera, when the first image was taken, in virtual space 200' may therefore be given by a 5-dimensional coordinate x1, y1, z1, Φ1, θ1. Indeed, all of the discretised positions and orientations within the virtual space 200' may be given by a 5-dimensional indexed coordinate xi, yj, zk, Φl, θm.
- As described, the first plurality of pixel values are from the
first image 208 captured by a camera inreal space 200. The first plurality of pixel values is associated with the first three-dimensional position x1, y1, z1, in the virtual three-dimensional space, of the camera when thefirst image 208 was captured. Each one of the first plurality of pixel values is associated with an orientation, in the virtual three-dimensional space, of a portion of a light field represented by the pixel (as described hereinafter). - As described, the
first image 208 is associated with an orientation Φ1, θ1 in virtual space 200' of the camera when thefirst image 208 was captured.Figure 5 illustrates schematically an example way of determining an orientation, in the virtual three-dimensional space, of a portion of a light field represented by each pixel value. In overview, the method comprises projecting the pixel values of thefirst image 208 onto ahemisphere 210 located at the first three-dimensional position x1, y1, z1; and determining, based on the projection, the orientation Φl, θm with which each pixel value is associated. - Each pixel value, having planar coordinates p, q, of the 2D
first image 208 may be projected or mapped onto a point on the surface 208' of the3D hemisphere 210 according to the position p,q of that pixel in the 2D image. Thehemisphere 210 may be constructed so that its principal axis is colinear with the first direction 206' (having an orientation defined by Φ1, θ1). Thefirst image 208 may be projected onto thehemisphere 210 for example using equirectangular projection, in which the planar coordinates p, q of each pixel of thefirst image 208 is mapped to spherical coordinates Φ', θ' of the hemisphere. - In some examples, the pixel values mat be projected (e.g. using equirectangular projection) onto a cosine weighted hemisphere. In these examples, the projection of planar coordinates p, q onto the spherical coordinates Φ', θ' of the
hemisphere 210 is weighted so that proportionally more pixel values are projected to smaller angles Φ', θ' than to larger angles Φ', θ' (relative to the first direction 206'). For example, the pixel values may be sampled and projected onto the hemisphere according to the weighting - In any case, the spherical coordinates Φ', θ' associated with a pixel value following the projection may be used to define the orientation in the virtual 3D space 200' associated with that pixel value. In examples where the virtual 3D space 200' is discretised, the orientation associated with each pixel value may be given by the indexed orientation Φl, θm. Therefore, each pixel value of the
first image 208 may be associated with a 5-dimensional indexed coordinate x1, y1, z1, Φl, θm. - Following the projection onto the
hemisphere 210, each pixel value is taken as a value representative of a portion of a light field at the position and orientation x1, y1, z1, Φl, θm of the pixel value in virtual space 200'. For example, the pixel value may be taken as a radiance, or amount of light, at the position and orientation x1, y1, z1, Φl, θm of the pixel value in virtual space 200'. - At this point, the light field at each of the other discretised positions and orientations xi, yj, zk, Φl, θm in the virtual space 200' may be unknown. However, in order to realistically render a virtual object at a second position in the
virtual space 200 that is other than the first position x1, y1, z1, an estimate of one or more light field values at that second position should be determined (as described hereinafter). - As mentioned, in
step 104 of the method described with reference toFigure 1 , the method comprises estimating, based on the first plurality of pixel values, a light field value for at least one second position x2, y2, z2, different from the first position x1, y1, z1, in the virtual three-dimensional space 200'. For example, referring toFigure 6 , the light field value may be the radiance, or an amount of light, from a givendirection 212 having a given orientation Φ2, θ2 at the second position x2, y2, z2. As described in more detail hereinafter, the estimate is made using compressive sensing. - As mentioned, in
step 106 of the method described with reference toFigure 1 , the method comprises rendering anobject 214, in the virtual three-dimensional space 200, located at the second position x2, y2, z2, using the estimated light field value. For example, with reference toFigure 7 , a virtual object 214 (in this example a cuboid) is to be rendered at a second position x2, y2, z2, so thatvirtual object 214 may appear, via the AR or VR headset 204 (not shown inFigure 7 ), as present at the second position x2, y2, z2, to the user 202 (not shown inFigure 7 ).Figure 8 illustrates schematically an example of animage 216 comprising thefirst image 208 augmented, for example overlaid, with an image 214' of theobject 214 rendered using the estimated light field values. As another example, theimage 216 may be a virtual image, which may for example be derived from thefirst image 208, and which includes the image 214' of theobject 214 rendered using the estimated light field values. Theexample image 216 may, for example, be displayed on a display screen of the AR orVR headset 204 such that thevirtual object 214 may appear as present at the second position x2, y2, z2, from the perspective of the user. - The estimated light field value, e.g. a radiance value, or other lighting condition value, at the second position x2, y2, z2, and having a given orientation Φ2, θ2, may be used in the rendering of the
virtual object 214 on a display screen of the VR or AR headset of the user (not shown inFigure 7 ). For example, the light field value may be used as a lighting condition of thevirtual object 214 during rendering of thevirtual object 214. As such, the rendering of thevirtual object 214 located in the virtual space 200' may be based on an estimate of the lighting condition that thevirtual object 214 would experience if it were in thereal space 200. A realistic rendering of thevirtual object 214 may therefore be generated. - As mentioned, the estimation of the light field value for the second position x2, y2, z2 from the plurality of first pixel values uses compressive sensing.
- Compressive sensing, also known as compressed sensing, compressive sampling, or sparse sampling is a signal processing technique for efficiently reconstructing a signal from sample values of the signal by finding a solution to an underdetermined system of linear equations by optimisation. An underdetermined system of linear equations has more unknowns than equations and as such may have many, and in some cases infinite, solutions. In the present example, the signal to be recovered is the light field at each position and orientation xi, yj, zk, Φl, θm in the discretised virtual space 200', the sample values are the light field values represented by the first plurality of pixels at the first position x1, y1, z1, Φl, θm, and the system of linear equations is that which transforms the latter to the former. Since the total number of positions and orientations xi, yj, zk, Φl, θm in the discretised virtual space 200' is much larger than the number of orientations at the first position x1, y1, z1, Φl, θm, it will be appreciated that the system of linear equations is underdetermined.
- In compressive sensing, the signal to be reconstructed is assumed to be sparse in some domain (which may be the same as or different to the domain in which the samples are captured). Hence a condition is applied to the solutions of the system of underdetermined linear equations that the solution must be sparse, that is, that only solutions having a small number (or indeed only the solution having the smallest number) of non-zero coefficients are/is accepted as solutions (or the solution) to the system of linear equations. Light field images are sparse. Therefore, light field values may be considered as sparse signals, for example as they are not pure noise. Accordingly, in some examples, estimating the light field value at the second position x2, y2, z2, Φl, θm using compressive sensing comprises reconstructing, from the first plurality of pixel values associated with the first position x1, y1, z1, Φl θm the light field value at each of the discrete positions and discrete orientations, by estimating a solution to an underdetermined set of linear equations. The solution is estimated by determining the solution with a small, for example, the smallest number of non-zero coefficients. This may be performed using optimisation techniques, such as l1 norm minimisation, described in more detail hereinafter.
- Of course, a system of linear equations may be represented in matrix form. For example, the system of underdetermined linear equations to be solved may be written as f = Θe. In this example, f is a s×1 (column) vector of the plurality of first pixel values indexed for the first discrete position x1, y1, z1 and for the discrete orientations Φl θm with which each pixel value is associated. In other words, f may be a column vector whose entries correspond to the first plurality of pixel values stacked in a predetermined order (e.g. indexed for each discrete orientation with which the pixel value is associated) . In this example, e is an t×1 vector, where s<t, of estimated light field values indexed for all of the discrete positions and discrete orientations xi, yj, zk, Φl, θm into which the virtual three-dimensional space 200' is discretised. In other words, e may be a column vector whose entries correspond to (to be determined) light field values stacked in a predetermined order (e.g. indexed for each discrete position and orientation with which the light field value is associated). For example, at least initially, the vector e may be populated with random values. In this example, Θ is a s×t transformation matrix for reconstructing e from f. In some examples, the transformation matrix may be a single matrix for reconstructing e from f. For example, the transformation matrix may be a sensing matrix. For example, the sensing matrix may be a matrix Gaussian distribution, that is, a matrix populated with real-valued random numbers taken from a Gaussian distribution (independent standard Gaussian random variables) . As another example, the sensing matrix may be a random permutation matrix, for example an identity matrix that is subject to a random permutation.
- As mentioned, the equation f = Θe may be solved for e by iteration subject to the minimisation of the number of non-zero coefficients of e. This may be achieved, for example by the minimisation of the the l1 norm of e. The l1 norm may be regarded as a function that counts the number of non-zero coefficients of a vector. In mathematical notation, the l1 norm, ∥e∥1, of vector e may be written as
- In some examples, the transformation matrix Θ may be a combination (for example include a multiplication of) two or more matrices. For example, in some examples, the transformation matrix Θ may comprise a matrix multiplication of a sensing matrix Φ (which may for example be the same as the sensing matric described above, for example a matrix Gaussian distribution), and a representation matrix Ψ for transforming e into a sparse domain. In other words, in some examples, Θ = ΦΨ. Using the representation matrix Ψ in addition to the sensing matrix Φ may provide that the vector e may be more sparsely represented, which may provide for faster and or more accurate minimisation of the l1 norm of e. For example, the representation matrix Ψ may be an orthogonal basis matrix. For example, the representation matrix Ψ may be a Discrete Fourier Transform (DFT) matrix. For example, the Discrete Fourier Transform matrix may be a 5-dimensional DFT matrix, one dimension each for xi, yj, zk, Φl, θm. In other words, the DFT matrix may be populated with Fourier transform coefficients of 5-dimensional (discrete) Fourier basis functions (i.e. each entry of the DFT matrix is the i,j,k,l,mth coefficient of a 5-dimensional discrete Fourier transform). The Fourier transform may be of the type used for periodic and discrete signals. In other examples, other basis functions may be used for the representation matrix, for example Discrete Cosine Transform bases, Wavelet transform bases or 3D Zernike Descriptor bases.
- In cases where the transformation matrix Ψ is used, the compressive sensing may comprise minimising the l1 norm of e for the equation f = ΦΨe, for example as described above. However, in this case, the resultant e will be in the spare domain into which it was transformed by the application of the transformation matrix Ψ. However, since the transformation matrix Ψ is an orthogonal basis matrix, the inverse of Ψ may be applied to the solution e, in order to return the estimate of the light field values at all the discretised positions and orientations xi, yj, zk, Φl, θm within the virtual three-dimensional space 200'. In this case therefore, the method may comprise applying, after the minimization of the l1 norm of e, the inverse Ψ-1 of the transformation matrix Ψ to e, to obtain the estimate of light field values at all the discretised positions and orientations xi, yj, zk, Φl, θm within the virtual three-dimensional space 200'.
- In either case, as a result of the compressive sensing based on the first plurality of pixel values, an estimate of light field values at all the discretised positions and orientations xi, yj, zk, Φl, θm within the virtual three-dimensional space 200' is obtained. The method may comprise determining the second position x2, y2, z2 at which the
virtual object 214 is to be rendered, obtaining the estimated light field values (e.g. radiance values) associated with the second position x2, y2, z2 (for example the light field value of each orientation Φl, θm associated with the second position x2, y2, z2), and rendering thevirtual object 214 based on the obtained estimated light field values associated with the second position x2, y2, z2. - As described, the estimated light field values, e.g. radiance values, are used to render the
virtual object 214, for example for display on a display screen of the VR orAR headset 204 of the user. As such, therefore, avirtual object 214 may be rendered using light field values that accurately represent the lighting condition, at the position of thevirtual object 214, as if that virtual object were inreal space 200. Thevirtual object 214 may therefore be rendered realistically. Moreover, the light field values used to render thevirtual object 214 are estimated based on pixel values from an image captured by a camera mounted on the VR orAR headset 204. This may allow for the realistic rendering of thevirtual object 214 to be performed efficiently, for example without the need to measure light field values at the second location at which thevirtual object 214 is rendered. This may allow the processing of the light field values for rendering to be performed relatively quickly and/or using relatively few resources, which may be important in portable VR orAR headsets 204, in which processing resources may be limited. - In some examples, the above described process may be carried out successively for different first images captured over time. The rendering of the
virtual object 214 may performed dynamically based on the successive first images and the estimated light field values derived therefrom, as described above. The method may therefore provide for accurate and efficient rendering that is capable of accounting for dynamic changes to the lighting environment in thereal space 200. This may provide for a more realistic experience. - In some examples, further images captured by the camera of the
headset 204 may be used in addition to the first image captured by the camera of theheadset 204 to provide further sample values for the compressive sensing and hence for a more accurate estimate of the light field values at all the discretised positions and orientations xi, yj, zk, Φl, θm within the virtual three-dimensional space 200'. For example, the method may further comprise obtaining a second plurality of pixel values of a second image captured by the camera inreal space 200, the second image being captured at a position or with an orientation inreal space 200 different to that of the camera when the first image was captured. Estimating the light field value for the at least one second position may then be based at least in part on the first plurality of pixel values and the second plurality of pixel values. For example, the vector f may be increased in length to include the pixel values indexed for the position and orientation xi, yj, zk, Φl, θm associated with each of each the pixel values of the first image and of the second image, to produce vector f'. An accordingly enlarged transformation matrix Θ' may be generated, and the l1 norm of e for the equation f' = Θ'e may be minimised in a similar way as to described above. However, in this example, the solution of the equation that was found to minimise the l1 norm of e when only the first plurality of pixel values of the first image was used may now be used as the starting values of e for the l1 norm minimisation of e. These staring values of e may be close to the solution that minimises the l1 norm of e in the equation f' = Θ'e where f' includes both the first plurality and the second plurality of pixel values. Therefore, the l1 norm minimisation may take less time and/or use less resources as compared to the l1 norm minimisation based on only the first plurality of pixel values. This may provide for improved efficiency in the estimation of light field values at each discrete position and orientation in the virtual space 200', and hence may provide for efficient realistic rendering ofobjects 214 in that space. - As will be appreciated, as further successive images are captured by the camera of the VR or
AR headset 204, for example as the user moves or looks around in thereal space 200, further pluralities of pixel values may be added to the sample vector f. This may accordingly improve the accuracy with which the light field values at each position and orientation in the virtual space 200' may be estimated. Further, the more sample values that are used in f, the closer the solution for a given l1 norm minimisation will be to the solution of a subsequent l1 norm minimisation using further pixel values, which may successively reduce the time needed for l1 norm minimisation as successive sample images as captured. This may further improve the efficiency of the light field estimation and hence rendering ofvirtual objects 214. - In some examples, having an estimate of the light field values at all the discretised positions and orientations xi, yj, zk, Φl, θm in the virtual space 200' may allow for rendering of the
virtual object 214 to include light field-based effects, such as aperture effects (i.e. simulating the focal point of a human eye or the like), or High Dynamic Range (HDR) effects (e.g. eye pupil dilation effects to enforce a range of luminescence similar to that experienced by a human eye) and the like. This may further improve the realism of the rendering of thevirtual object 214. - Referring to
Figure 9 , there is illustrated schematically anapparatus 204 according to an example. Theapparatus 204 may be arranged to perform a method described above with reference toFigures 1 to 8 . In this example, theapparatus 204 is the AR orVR headset 204, although it will be appreciated that in other examples the apparatus may take different forms. For example, the method may be performed by a device (not shown) (e.g. a separate device) in communication with theheadset 204. For example, theheadset 204 may be arranged to send images to the remote device (not shown) and receive from the device an image (e.g. including an image of a virtual object rendered using light field values estimated by the device using the techniques described above based on the first image received by the device) for display on theheadset 204 to theuser 202. - In the example of
Figure 9 , theheadset 204 comprises abody 258 for mounting to a head of auser 202 of theheadset 204. Theheadset 204 comprises aposition sensor element 260,camera 250, a display such as adisplay screen 252, aprocessor 254 and amemory 256. Theprocessor 254 is communicatively connected to theposition sensor element 260, thememory 256, thecamera 250 and thedisplay 252. The focal axis of thecamera 250 may be parallel, for example substantially co-linear with an eyeline of the user. Theposition sensor element 260 is arranged to determine the position and orientation of thecamera 250 inreal space 200, for example using the methods described above. - The
memory 256 may store instructions which when executed by theprocessor 254 cause the processor 254 (and in some cases thecamera 250 and the display 252) to perform the method described above with reference toFigures 1 to 8 . For example, theprocessor 254 may be arranged to obtain afirst image 208 captured by thecamera 250, and to determine a first plurality of pixel values of thefirst image 208. Theprocessor 254 may be arranged to associate the first plurality of pixel values with a first three-dimensional position x1, y1, z1, in the virtual three-dimensional space 200', of thecamera 250 when thefirst image 208 was captured (for example as determined or informed by the position sensor element 260), for example using the techniques as described above. Theprocessor 254 may be arranged to associate each one of the first plurality of pixel values with an orientation Φl, θm, in the virtual three-dimensional space 200', of a portion of a light field represented by the pixel, for example as described above. Theprocessor 254 may be arranged to estimate, using compressive sensing, and based on the first plurality of pixel values, a light field value for at least one second position x2, y2, z2, different from the first position x1, y1, z1, in the virtual three-dimensional space 200', for example using the techniques as described above. Theprocessor 254 may be arranged to render avirtual object 214, in the virtual three-dimensional space 200', located at the second position x2, y2, z2, using the estimated light field value, for example using the techniques as described above. - The
processor 254 may be arranged to control thedisplay 252 to display an image of thevirtual object 214 having been rendered using the light field values estimated as described above. For example, the processor may be arranged to control thedisplay 252 to display an image captured by thecamera 250 but augmented with avirtual object 214 having been rendered using the light field values estimated as described above. As another example, thedisplay 252 may display a virtual image, which may for example be derived from the image captured by thecamera 250, and which includes an image of anobject 214 rendered using the estimated light field values. - As such the
headset 204 may provide that avirtual object 214 may be rendered using light field values that accurately represent the lighting condition, at the position of thevirtual object 214, as if that virtual object were inreal space 200. Thevirtual object 214 may therefore be rendered realistically, for example especially in closed real space environments such as operation rooms or classrooms. Moreover, the light field values used to render thevirtual object 214 are estimated based on pixel values from an image captured by thecamera 250 mounted on the VR orAR headset 204. This may allow for the realistic rendering of thevirtual object 214 to be performed efficiently, for example without the need to measure light field values at the second location at which thevirtual object 214 is rendered. This may allow the processing of the light field values for rendering to be performed relatively quickly and/or using relatively few resources, which may allow theheadset 204 to be lighter and/or consume less power and/or extend a battery life of theheadset 204, for example. - The
headset 204 may be arranged to estimate light field values and render virtual objects using successive images captured by thecamera 250, for example using the techniques as described above. As described, this may improve the accuracy and/or efficiency with which the light field values at each position and orientation in the virtual space 200' may be estimated, and hence may improve the efficiency and/or realism of the rendering. - The above examples are to be understood as illustrative examples of the invention. It is to be understood that any feature described in relation to any one example may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the examples, or any combination of any other of the examples. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.
Claims (17)
- A method of rendering an object (214) in a virtual three-dimensional space (200'), the method comprising:obtaining a first plurality of pixel values of a first image (208) captured by a camera (250) in real space (200), the first plurality of pixel values being associated with a first three-dimensional position (x1, y1, z1), in the virtual three-dimensional space (200'), of the camera (250) when the first image (208) was captured, each one of the first plurality of pixel values being associated with an orientation (Φl, θm) in the virtual three-dimensional space (200'), of a portion of a light field represented by the pixel;estimating, using compressive sensing, and based on the first plurality of pixel values, a light field value for at least one second position (x2, y2, z2), different from the first position (x1, y1, z1), in the virtual three-dimensional space (200'); andrendering an object (214), in the virtual three-dimensional space (200'), located at the second position (x2, y2, z2), using the estimated light field value.
- The method according to claim 1, wherein the method comprises:determining a position (xr, yr, zr) and orientation of the camera (250) in real space (200); anddetermining, based at least in part on the determined position and orientation of the camera (250) in real space, the first position (x1, y1, z1) in the virtual three-dimensional space (200') and the orientation (Φl, θm), in the virtual three-dimensional space (200'), with which each pixel value is associated.
- The method according to claim 2, wherein determining the orientation (Φl, θm) with which each pixel value is associated comprises:projecting the pixel values of the first image (208) onto a hemisphere (210) located at the first position (x1, y1, z1); anddetermining, based on the projection, the orientation (Φl, θm) with which each pixel value is associated.
- The method according to claim 3, wherein projecting the pixel values of the first image (208) onto the hemisphere (210) comprises:using equirectangular projection of the pixel values of the first image onto a cosine weighted hemisphere.
- The method according to any one of claim 1 to claim 4, wherein the method comprises:
capturing the first image (208) using the camera (250) . - The method according to claim 5, wherein the camera (250) is mounted to an augmented reality or virtual reality headset (204).
- The method according to claim 6, wherein rendering the object (214) comprises rendering a representation of the object (214) on a display screen (252) of the augmented reality or virtual reality headset (204).
- The method according to any one of claim 1 to claim 7, wherein the virtual three-dimensional space (200') is discretised into discrete positions (xi, yj, zk) and discrete orientations (Φl, θm), each discrete position (xi, yj, zk) being associated with a plurality of discrete orientations (Φl, θm); wherein each of the one or more orientations (Φl, θm) with which each pixel value is associated is a discrete orientation (Φl, θm), wherein the first position (x1, y1, z1) is a first discrete position, and wherein the second position (x2, y2, z2) is a second discrete position, different to the first discrete position (x1, y1, z1).
- The method according to claim 8, wherein estimating the light field value using compressive sensing comprises reconstructing, from the first plurality of pixel values associated with the first discrete position (x1, y1, z1), the light field at each of the discrete positions and discrete orientations (xi, yj, zk, Φl, θm), by estimating a solution to an underdetermined set of linear equations.
- The method according to claim 8 or claim 9, wherein estimating the light field value using compressive sensing comprises:minimizing the l1 norm of e in the equation f = Θe;where:f is an s×1 vector of the plurality of first pixel values indexed for the first discrete position (x1, y1, z1) and for the discrete orientations (Φl, θm) with which each pixel value is associated;e is a t×1 vector, where s<t, of estimated light field values indexed for all of the discrete positions and discrete orientations (xi, yj, zk, Φl, θm) into which the virtual three-dimensional space is discretised; andΘ is a s×t transformation matrix for reconstructing e from f.
- The method according to claim 10, wherein Θ = ΦΨ, where:Ψ is a representation matrix for transforming e into a sparse domain; andΦ is a sensing matrix comprising a set of predetermined random values; andwherein the method further comprises applying, after the minimization of the l1 norm of e, the inverse of Ψ to e, to obtain the estimate of light field values at all the discretised positions and orientations (xi, yj, zk, Φl, θm) within the virtual three-dimensional space (200').
- The method according to claim 11, wherein Ψ is a Discrete Fourier Transform matrix.
- The method according to any one of claim 1 to claim 12, wherein the light field value comprises a radiance value.
- The method according to any one of claims 1 to 13, wherein the method further comprises:obtaining a second plurality of pixel values of a second image captured by the camera in real space, the second image being captured at a position or with an orientation in real space different to that of the camera when the first image was captured;wherein estimating the light field value for the at least one second position is based at least in part on the first plurality of pixel values and the second plurality of pixel values.
- An apparatus (204) arranged to perform the method according to any one of claims 1 to 14.
- The apparatus (204) according to claim 15, wherein the apparatus (204) is an augmented reality or a virtual reality headset (204).
- A computer readable medium having instructions stored thereon which, when executed on a computer, causes the computer to perform the method of any one of claims 1 to 14.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP18186166.7A EP3599588B1 (en) | 2018-07-27 | 2018-07-27 | Rendering an object |
US16/434,198 US11049313B2 (en) | 2018-07-27 | 2019-06-07 | Rendering an object |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP18186166.7A EP3599588B1 (en) | 2018-07-27 | 2018-07-27 | Rendering an object |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3599588A1 true EP3599588A1 (en) | 2020-01-29 |
EP3599588B1 EP3599588B1 (en) | 2021-09-08 |
Family
ID=63079866
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP18186166.7A Active EP3599588B1 (en) | 2018-07-27 | 2018-07-27 | Rendering an object |
Country Status (2)
Country | Link |
---|---|
US (1) | US11049313B2 (en) |
EP (1) | EP3599588B1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2022105459A1 (en) * | 2020-11-23 | 2022-05-27 | 歌尔股份有限公司 | Image processing method and apparatus, electronic device and storage medium |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11176374B2 (en) * | 2019-05-01 | 2021-11-16 | Microsoft Technology Licensing, Llc | Deriving information from images |
US11238316B2 (en) * | 2019-12-04 | 2022-02-01 | Roblox Corporation | Detection of counterfeit virtual objects |
US11321856B1 (en) | 2020-12-18 | 2022-05-03 | Roblox Corporation | Detection of inauthentic virtual objects |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130011051A1 (en) * | 2011-07-07 | 2013-01-10 | Lockheed Martin Corporation | Coded aperture imaging |
US9654208B2 (en) * | 2011-08-30 | 2017-05-16 | The United States Of America, As Represented By The Secretary Of The Navy | System and method for photonic compressive sampling |
US9578226B2 (en) * | 2012-04-12 | 2017-02-21 | Qualcomm Incorporated | Photometric registration from arbitrary geometry for augmented reality |
US10262462B2 (en) * | 2014-04-18 | 2019-04-16 | Magic Leap, Inc. | Systems and methods for augmented and virtual reality |
JP6812271B2 (en) * | 2017-02-27 | 2021-01-13 | キヤノン株式会社 | Image processing equipment, image processing methods and programs |
-
2018
- 2018-07-27 EP EP18186166.7A patent/EP3599588B1/en active Active
-
2019
- 2019-06-07 US US16/434,198 patent/US11049313B2/en active Active
Non-Patent Citations (7)
Title |
---|
A QUINTANA ET AL: "AN AUGMENTED REALITY SYSTEM BASED ON LIGHT FIELDS", PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON COMPUTER GRAPHICS THEORY AND APPLICATIONS (GRAPP 2012) AND INTERNATIONAL CONFERENCE ON INFORMATION VISUALIZATION THEORY AND APPLICATIONS (IVAPP 2012), 24 February 2012 (2012-02-24), pages 453 - 459, XP055537854, ISBN: 978-989-8565-02-0 * |
ABAD F ET AL: "Integrating synthetic objects into real scenes", COMPUTERS AND GRAPH, ELSEVIER, GB, vol. 27, no. 1, February 2003 (2003-02-01), pages 5 - 17, XP004405346, ISSN: 0097-8493, DOI: 10.1016/S0097-8493(02)00221-2 * |
AMIT ASHOK ET AL: "Compressive light field imaging", PROCEEDINGS OF SPIE, vol. 7690, 23 April 2010 (2010-04-23), 1000 20th St. Bellingham WA 98225-6705 USA, pages 76900Q, XP055537855, ISSN: 0277-786X, ISBN: 978-1-5106-2099-5, DOI: 10.1117/12.852738 * |
BABACAN S D ET AL: "Compressive Light Field Sensing", IEEE TRANSACTIONS ON IMAGE PROCESSING, IEEE SERVICE CENTER, PISCATAWAY, NJ, US, vol. 21, no. 12, December 2012 (2012-12-01), pages 4746 - 4757, XP011492209, ISSN: 1057-7149, DOI: 10.1109/TIP.2012.2210237 * |
GREGER G ET AL: "THE IRRADIANCE VOLUME", IEEE COMPUTER GRAPHICS AND APPLICATIONS, IEEE SERVICE CENTER, NEW YORK, NY, US, vol. 18, no. 2, March 1998 (1998-03-01), pages 32 - 43, XP000927334, ISSN: 0272-1716, DOI: 10.1109/38.656788 * |
GUPTA MAYANK ET AL: "Compressive Light Field Reconstructions Using Deep Learning", 2017 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW), IEEE, 21 July 2017 (2017-07-21), pages 1277 - 1286, XP033145910, DOI: 10.1109/CVPRW.2017.168 * |
JOEL KRONANDER ET AL: "Photorealistic rendering of mixed reality scenes", COMPUTER GRAPHICS FORUM, vol. 34, no. 2, May 2015 (2015-05-01), GB, pages 643 - 665, XP055537858, ISSN: 0167-7055, DOI: 10.1111/cgf.12591 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2022105459A1 (en) * | 2020-11-23 | 2022-05-27 | 歌尔股份有限公司 | Image processing method and apparatus, electronic device and storage medium |
Also Published As
Publication number | Publication date |
---|---|
EP3599588B1 (en) | 2021-09-08 |
US11049313B2 (en) | 2021-06-29 |
US20200035011A1 (en) | 2020-01-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11049313B2 (en) | Rendering an object | |
US11393173B2 (en) | Mobile augmented reality system | |
CN107330439B (en) | Method for determining posture of object in image, client and server | |
CN111857329B (en) | Method, device and equipment for calculating fixation point | |
KR101791590B1 (en) | Object pose recognition apparatus and method using the same | |
US20200184726A1 (en) | Implementing three-dimensional augmented reality in smart glasses based on two-dimensional data | |
CN109684969B (en) | Gaze position estimation method, computer device, and storage medium | |
GB2580691A (en) | Depth estimation | |
CN109191554B (en) | Super-resolution image reconstruction method, device, terminal and storage medium | |
CN114140510B (en) | Incremental three-dimensional reconstruction method and device and computer equipment | |
CN108225537A (en) | A kind of contactless small items vibration measurement method based on high-speed photography | |
CN107862733B (en) | Large-scale scene real-time three-dimensional reconstruction method and system based on sight updating algorithm | |
CN112819875B (en) | Monocular depth estimation method and device and electronic equipment | |
EP3185212B1 (en) | Dynamic particle filter parameterization | |
CN108028904B (en) | Method and system for light field augmented reality/virtual reality on mobile devices | |
CN107818596B (en) | Scene parameter determination method and device and electronic equipment | |
CN114913287B (en) | Three-dimensional human body model reconstruction method and system | |
CN110135474A (en) | A kind of oblique aerial image matching method and system based on deep learning | |
US20210183082A1 (en) | Image registration method, apparatus, computer system, and mobile device | |
CN116883770A (en) | Training method and device of depth estimation model, electronic equipment and storage medium | |
CN110196638B (en) | Mobile terminal augmented reality method and system based on target detection and space projection | |
WO2022019128A1 (en) | Information processing device, information processing method, and computer-readable recording medium | |
Wegmann et al. | Real time motion tracking for augmented reality with TOF camera and vulkan rendering | |
US20240071009A1 (en) | Visually coherent lighting for mobile augmented reality | |
CN116386486A (en) | Virtual window self-adaptive adjusting method and device of intelligent glasses and intelligent glasses |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20180727 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20210518 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1429284 Country of ref document: AT Kind code of ref document: T Effective date: 20210915 Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602018023155 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG9D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20210908 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20211208 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20211208 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1429284 Country of ref document: AT Kind code of ref document: T Effective date: 20210908 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20211209 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220108 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220110 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602018023155 Country of ref document: DE |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 |
|
26N | No opposition filed |
Effective date: 20220609 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20220731 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220727 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220731 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220731 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220731 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220727 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20230807 Year of fee payment: 6 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230720 Year of fee payment: 6 Ref country code: DE Payment date: 20230918 Year of fee payment: 6 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 602018023155 Country of ref document: DE Owner name: SIEMENS HEALTHINEERS AG, DE Free format text: FORMER OWNER: SIEMENS HEALTHCARE GMBH, MUENCHEN, DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20180727 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210908 |