WO2013104440A1 - Procédé de texturation de maillage multi-vues et dispositif correspondant - Google Patents

Procédé de texturation de maillage multi-vues et dispositif correspondant Download PDF

Info

Publication number
WO2013104440A1
WO2013104440A1 PCT/EP2012/069862 EP2012069862W WO2013104440A1 WO 2013104440 A1 WO2013104440 A1 WO 2013104440A1 EP 2012069862 W EP2012069862 W EP 2012069862W WO 2013104440 A1 WO2013104440 A1 WO 2013104440A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
images
texture information
information
mesh
Prior art date
Application number
PCT/EP2012/069862
Other languages
English (en)
Inventor
Youssef Alj
Guillaume Boisson
Luce Morin
Philippe Bordes
Muriel Pressigout
Original Assignee
Thomson Licensing
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Thomson Licensing filed Critical Thomson Licensing
Priority to EP12769115.2A priority Critical patent/EP2803041B1/fr
Priority to US14/372,018 priority patent/US20140354632A1/en
Publication of WO2013104440A1 publication Critical patent/WO2013104440A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/282—Image signal generators for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems

Definitions

  • the invention relates to the domain of image or video processing and more particularly in the processing of data representative of texture associated to images of a multi-views video stream.
  • the invention also relates to the domain of modelling the texture of real scene by using texture data and depth data associated to images representative of the scene according to several points of view.
  • Multi-View Imaging provides a realistic depth perception to the user and allows a virtual navigation around the scene. It also offers the possibility to synthesize virtual views, opening therefore a broad spectrum of applications such as 3DTV and Free-viewpoint TV (FTV).
  • An end-to-end MVI chain presents many challenges due to the inherent problems related to scene capture, data representation and transmission. Indeed, during scene acquisition, multiple cameras are used; hence several photometric errors may occur due to the changing illumination across different views, which increase the signal ambiguity during depth estimation.
  • Scene representation is also a challenging task. One should strive to build a dense yet non-redundant scene model from the set of overlapping views, each view comprising information related to the texture and to the depth.
  • the purpose of the invention is to overcome at least one of these disadvantages of the prior art. More particularly, the invention has the notable purpose of reducing the amount of texture information associated to multi-views data representative of a scene.
  • the invention relates to a method for texturing a mesh, the mesh being associated with a surface representative of a scene captured according to a plurality of points of view, an image of the scene comprising texture information being associated with each point of view, wherein at least a mesh element of the mesh is at least partially visible from at least two first images of the plurality of images.
  • the method comprises the following steps:
  • the error information associated to a first texture information corresponds to a sum of the comparison results between the projected first texture information and each of the at least two first images.
  • the selected first texture information corresponds to the first texture information having the least error information.
  • the method further comprises the steps of:
  • the projection of the mesh element corresponds to a conversion of the coordinates of the mesh element from world space into the image space of the image.
  • each image comprises a plurality of pixels, video information being associated to the plurality of pixels.
  • each image is represented by using Multi-view Video plus Depth (MVD) representation.
  • MVD Multi-view Video plus Depth
  • the invention also relates to a computation unit configured to texture a mesh, the mesh being associated with a surface (1 ) representative of a scene captured according to a plurality of images, each image comprising texture information, wherein at least a mesh element of the mesh is at least partially visible from at least two first images of the plurality of images, the computation unit comprising:
  • the error information associated to a first texture information corresponds to a sum of the comparison results between the projected first texture information and each of the at least two first images.
  • the selected first texture information corresponds to the first texture information having the least error information.
  • the computation unit further comprises:
  • the means for projecting said mesh element comprise means for converting the coordinates of the mesh element from world space into the image space of the image.
  • FIG. 1 illustrates a meshed surface 1 representative of a scene viewed and captured according to several points of views, according to a particular embodiment of the invention ;
  • FIG. 2 illustrates the association of a first texture information with a mesh element of the surface 1 of figure 1 and the projection of the first texture information on at least a part of the views, according to a particular embodiment of the invention ;
  • FIG. 3 illustrates a mesh element of the meshed surface 1 of figure 1 entirely visible according to at least one of the points of view, according to a particular embodiment of the invention ;
  • figure 4 illustrates a mesh element of the meshed surface of figure 1 partially visible according to at least one of the points of view, according to a particular embodiment of the invention ;
  • FIG. 5 illustrates a device implementing a method for texturing a mesh of surface 1 of figure 1 , according to a particular implementation of the invention ;
  • figure 6 illustrates a method for texturing a mesh of a surface 1 of figure 1 , according to a particular implementation of the invention.
  • a mesh is associated with the surface, the mesh comprising a plurality of mesh elements (for example triangles or quadrilaterals).
  • Each image of the scene comprises a plurality of pixels, the number of pixels depending on the resolution of the image of the scene associated with the view.
  • a texture information is associated with each image of the scene, the texture information corresponding for example to a video information (for example RGB data representative of the grey level associated with each colour (RGB) information) associated with the pixels of the image corresponding to a view of the scene.
  • a first texture information extracted from one of the first images is associated to the mesh element and projected on all or part of the first image, i.e. on the image where the mesh element is at least partially visible.
  • the projected first texture information is compared with each texture information associated to the first images on which the first texture information has been projected. From the comparison results, an error information is estimated which is for example representative of the difference between the projected first texture information and the texture information associated with the first images on which the first texture information has been projected.
  • the step of associating a first texture information with the mesh element is reproduced for another first image and the steps of projecting and comparing this new first texture information are reproduced.
  • each first image forming a set of several candidate first texture information.
  • an error information has been estimated for all (candidate) first texture information
  • one of them is selected according to the error information assigned with each of them.
  • the selection of one single texture information (to be assigned with a mesh element) among the plurality of texture information associated with the plurality of views for which the mesh element is at least partially visible enables to reduce the amount of data to be transmitted to a display or any receiver over a network.
  • minimal input data needed to associate a texture with a mesh element is a meshed surface representative of a scene and texture information associated with a plurality of images representing the scenes according to different points of view. It also enables to select the best texture information candidate to map a given mesh element, reducing photometric errors, which occur for example when the texture information associated with a mesh element correspond to the average of all texture information available in the different views where the mesh element is visible or partially visible.
  • Figure 1 illustrates a surface 1 representative of a scene viewed and captured according to several points of views.
  • the surface 1 is representative of a scene acquired according to a plurality of points of view 1 1 , 12, 13, 14, 15.
  • An image 1 10, 120, 130, 140, 150 is associated with each point of view 1 1 to 15, corresponding to a representation of the scene according to a given point of view.
  • a mesh is associated with the surface 1 , the mesh comprising a plurality of mesh elements (for example thousands of mesh elements), for example triangles or quadrilaterals.
  • the surface 1 corresponds advantageously to a 3D representation of the scene.
  • the surface 1 and the associated mesh are obtained by combining methods well-known by the skilled person in the art, for example by combining the "shape from silhouette” method with the marching-cubes algorithm or by combining the space carving method with the marching- cubes algorithm.
  • the "shape from silhouette” method (described for example in “ Spatio-Temporal Shape from Silhouette using Four-Dimensional Delaunay Meshing” by Ehsan Aganj, Jean-Philippe Pons, Florent Segonne and Renaud Keriven) and the space carving method (described in "A Theory of Shape by Space Carving” published in International Journal of Computer Vision, 38(3), 199-219 (2000), by Kiriakos N. Kutulakos and Steven M.
  • a volumetric framework is employed in order to merge the input depth maps.
  • voxels are iteratively carved by ray-casting from each view, until the 3D model be geometrically consistent with every input depth map.
  • a surface mesh is then extracted from this volumetric representation thanks to the marching cubes algorithm.
  • At least one of the mesh elements is visible or partially visible in different images associated to the views of the scene.
  • the mesh element 10 is entirely visible from the points of view 1 1 , 12, 13, i.e. the mesh element 10 is entirely visible in the first images 1 10, 120, 130 associated with the points of view 1 1 , 12, 13.
  • the mesh element 10 is partially visible from the point of view 14, i.e. the mesh element 10 is partially visible in the first image 140. Indeed, as illustrated on Figure 1 , the viewing direction 104 having as origin the point of view 14 and as extremity the mesh element 10 is tangent to a point 1 1 (or mesh element 1 1 ) of the surface 1 on the path separating the point of view 14 and the mesh element 10. The mesh element is not visible from the point of view 15 and is not visible on the associated image 150. The image 150 is thus not considered as being a first image.
  • the viewing direction 105 having as origin the point of view 15 and as extremity the mesh element 10 has an intersection with the surface 1 , the intersection corresponding to the point or mesh element 12 of surface 1 , the intersection 12 being positioned between the point of view 15 and the mesh element 10 on the path corresponding to the viewing direction 105.
  • a visibility label is advantageously associated with the mesh element 10.
  • the visibility label takes for example two values, i.e. visible and non visible, or three value, i.e. entirely visible, partially visible and non visible.
  • the visibility information associated to the mesh element 10 is determined by comparing a depth information associated with the mesh element 10 and a depth information associated to pixels of a first image 1 10 to 150 on which the mesh element 10 is projected.
  • the mesh element visibility with respect to each image 1 10 to 150 is determined for example using OpenGL z-buffer.
  • the mesh i.e. all mesh elements
  • the z-buffer is extracted, the z-buffer comprising a depth information associated to each mesh element as seen in the current image (for example, a depth information is associated with each pixel of the current image).
  • Depth information is determined by projecting mesh elements into an image, i.e.
  • Each vertex of the mesh element 10 is projected onto the current image 1 10 and the depth component, denoted as z pro j e ct ed , is checked against the pixel depth z-buffer of the current image 1 10. If the projected vertex is behind the pixel in the z-buffer (i.e. the value Zprojected is greater than the depth value associated to the pixel in the z- buffer of the current image), then this vertex is hidden, and thus the set of mesh triangles lying on this vertex are not visible.
  • the projected vertex is ahead the pixel in the z-buffer (i.e. the value z pr0 j e ct e d is less than the depth value associated to the pixel in the z-buffer of the current image), then this vertex is visible, and thus the set of mesh triangles lying on this vertex are at least partially visible.
  • the visibility information associated with the mesh element is for example determined according to the following algorithm:
  • v is a hidden vertex. Mark all the triangles lying on v as hidden. end
  • the mesh is not projected into the current image as to determine the z-buffer associated with the current image.
  • the depth of the mesh element is compared to a depth information comprised in a depth map associated with the current image, the depth map being received with the image (for example in a stream of the type MVD).
  • the visibility information associated with the mesh element 10 is received with the data representatives of the images 1 10 to 150 and is not to be determined.
  • These data comprises for example RGB information for each pixel and visibility information associated with the mesh elements of the mesh of figure 1 and comprising information about the visibility of mesh elements in each and every images representing the scene (the visibility information being for example stored in a visibility map).
  • Images 1 10 to 150 belong to a multi-view video stream comprising a sequence of several images representative of a same scene.
  • the multi-view video stream is MVD type (Multi-view Video plus Depth).
  • depth information is associated with each image representing the scene.
  • a depth information is thus associated with each image 1 1 0 to 1 50, the depth information correspond to data representative of depth, for example a depth map (for example of the type of z-buffer) or a disparity map.
  • a depth information is a generic name and corresponds to any data structure representative of a depth or of a disparity.
  • the depth information associated with the pixels of the images 1 1 0 to 1 50 are for example captured with the use of appropriate sensors, for example by using infrared sensors associated with an infrared emitter (for example system of the type Kinect®) or by using a system comprising a laser emitter and an associated sensor configured for determining the crossing time of a laser ray emitted toward the captured scene and reflected by the scene, the determined time being representative of the path crossed and thus of the depth associated with the point of the scene having reflected the laser ray.
  • infrared sensors associated with an infrared emitter for example system of the type Kinect®
  • a system comprising a laser emitter and an associated sensor configured for determining the crossing time of a laser ray emitted toward the captured scene and reflected by the scene, the determined time being representative of the path crossed and thus of the depth associated with the point of the scene having reflected the laser ray.
  • video information (corresponding for example to colour information) and depth information associated with the pixels of images 1 1 0 to 1 50 are stored in memories under the form of RGBa (Red, Green, Blue, a channels) information, channels RGB being used for storing video information associated with each pixel (for example 8, 10 or 12 bits per channel) and the a channel being used for storing the depth information (or disparity information), for example on 8, 1 0 or 1 2 bits.
  • RGBa Red, Green, Blue, a channels
  • intrinsic and extrinsic parameters of acquisition devices used for acquiring the images 1 10 to 1 50 are known and for example stored in a memory.
  • Intrinsic parameters comprise for example focal length, enlargement factor of the image, coordinates of the projection of the optical centre of the acquisition device on image plane and/or a parameter representative of potential non-orthogonality of lines and columns of photosensitive cells forming the acquisition device.
  • Extrinsic parameters comprise for example the orientation matrix for passing from world space to image space (and inversely) and/or components of translation vector for passing from world space to image space (and inversely).
  • each image is acquired by a particular acquisition device according to a particular point of view, acquisition devices being spatially staggered.
  • acquisition devices being spatially staggered.
  • only a pair of left and right images is acquired by means of two acquisition devices, the other images of the plurality of images representative of the acquired scene being estimated from the left and right images by disparity compensated interpolation.
  • each image is also representative of the same scene according to a particular point of view and depth information is associated with each image, independently of the fact that the image be acquired or interpolated.
  • Figure 2 illustrates the association of a first texture information with a mesh element of the surface 1 and the projection of a first texture information on some views, according to a particular and non limitative embodiment of the invention.
  • the mesh element 1 0 is visible or partially visible from the points of view 1 1 , 1 2, 1 3 and 14, i.e. the mesh element 1 0 is visible or partially visible in the images 1 10, 1 20, 1 30 and 140 respectively associated with the points of view 1 1 , 1 2, 1 3 and 1 4.
  • the images 1 1 0 to 140 in which the mesh element is partially visible are called first images.
  • One first image 1 1 0 is selected among the plurality of first images 1 1 0 to 140 and a first texture information extracted from thus first image 1 1 0 is extracted and associated with the mesh elements.
  • the first texture information corresponds advantageously to the video information (or RGB colour information) associated with the pixels of the first image 1 1 0 corresponding to the projection of the mesh element 1 0 onto the first image 1 1 0.
  • the pixels of the first image on which is projected the mesh element 10 are illustrated on Figure 3.
  • the texture information associated with the images corresponds to YUV data.
  • Figure 3 illustrates the projection 31 of the mesh element 1 0 onto an image 30 (corresponding for example to the image 1 10) on which the mesh element is entirely visible.
  • the first texture information associated with the mesh element 1 0 corresponds to the video information associated with the pixels on which the mesh element 1 0 is projected, these pixels being "covered” or partially “covered” by the projection 31 of the mesh elements. These pixels 301 to 30i are illustrated in grey on Figure 3.
  • the mesh element 1 0 is first textured with a first texture information of image 1 1 0. This first texture information is then projected onto all first images 1 10 to 140, which is illustrated by the arrows 201 , 202, 203 and 204.
  • the projection of the mesh element 1 0 (textured with the first texture information of corresponding pixels of first image 1 1 0) onto first images 1 1 0, 1 20, 1 30 for which the mesh element 1 0 is entirely visible is illustrated by Figure 3 and the projection of the mesh element 1 0 (textured with the first texture information of corresponding pixels of first image 1 1 0) onto the first image 140 for which the mesh element 1 0 is only partially visible is illustrated by Figure 4.
  • the projection of the mesh element 1 0 onto one first image 1 1 0 to 140 corresponds to a conversion of the coordinates of the mesh element 10 from the world space into the image space associated to the first image onto which the mesh element is projected.
  • the image 30 corresponds to a first image (first images 1 1 0, 1 20 or 130) on which the mesh element 1 0 is projected.
  • the projection of the mesh element 1 0 corresponds to the mesh element referenced 31 .
  • Pixels in gray 301 to 30p represents the p pixels (p being an integer) of the projection image covered at least partially by the projection 31 of the mesh element 1 0.
  • the image 40 corresponds to the first image 140 of Figure 2 on which the mesh element is only partially visible, a part of the mesh element 1 0 being hidden by another mesh element 1 1 illustrated on Figure 1 .
  • the projection of the mesh element 1 0 on image 40 is referenced 41 and the projection of the mesh element 1 1 on image 40 is referenced 42.
  • the number q (q being an integer) of pixels 401 to 40q of image 40 at least partially covered by the projection 41 of the mesh element 10 are less than the number p of the pixels of image 30 (p > q) (with a same resolution for all first images 1 1 0 to 140).
  • the first texture information is compared with the texture information associated with the pixels of each first image 1 1 0 to 140 corresponding to the projection of the mesh elements.
  • the pixels of a first image 1 1 0 to 140 corresponding to the projection of the mesh element 10 also correspond to the pixels of this first image for which the mesh element is visible according to the point of view associated with this image.
  • M ⁇ v corresponds to the projection onto the image Vj (for example one of the image 1 20, 1 30 or 140) of the mesh element 10 textured with the first texture information of image I, (for example first image 1 1 0)
  • Dij corresponds to the distortion between the texture information of images i and j.
  • the first texture information of image 1 1 0 associated with the mesh element 1 0 is compared with each texture information of each first image 1 10 to 140.
  • the first texture information is compared with each texture information of each first image except with the first image 1 1 0, the distortion between the projected first texture information (of first image 1 1 0) and the texture information of image 1 10 being equal to 0 or close to 0.
  • the same process is repeated by associating a first texture information extracted from the first image 120 with the mesh element 10, by projecting this new first texture information onto the first images 1 1 0 to 140 and by comparing it with the texture information associated with the pixels corresponding to the projection of the mesh element 1 0 onto the first images 1 1 0 to 140 as to determine the distortion between the projected first texture information and each first image.
  • This process is reiterated by associating first texture information extracted from each first image 1 30 and 140 as to determine the distortion between each projected first texture information and each first image.
  • the distortion is determined between each first texture information (from each first image 1 1 0 to 140) with only a part of the first images (by excepting the first image providing the first texture information to be projected onto the first images for comparison).
  • a non limitative example of an algorithm used for determining the result of comparison between the first texture information and the first images is as follow: // Textures traversal
  • an error is determined and associated with respectively each first texture information.
  • the determination of the error associated with a first texture information us based on the comparison results between the projected first texture information and the texture information associated with the pixels of the first images onto which the textured mesh element is projected.
  • One first texture information is then selected among the plurality of first texture information according to the errors associated with each first texture information.
  • the selected first texture information corresponds advantageously to the first texture information having the least error value.
  • T corresponding to the set of first texture information (available in the first images 1 10 to 140).
  • a non limitative example of an algorithm used for selecting the first texture information which minimizes the error is as follow:
  • the selection of a first texture information among the set of candidate first texture information enables to assign one single texture information to a mesh element for every image in which the mesh element is partially visible, which reduces the amount of data to be coded and transmitted to any display device displaying the images or to any device decoding the data representative of the images to be displayed.
  • the selection of the first texture information minimizing photometric error also enables to reduce artefacts which could appear if the texture associated to a mesh element would correspond to an arbitrary selection of one candidate texture information or to a mix of the candidate texture information.
  • a first texture information is assigned to each mesh element of the mesh or to a part of the mesh elements of the mesh.
  • Figure 5 diagrammatically shows a hardware embodiment of a device 5 adapted and configured for texturing a mesh of a surface 1 and for the generation of display signals of one or several images, according to a particular and non limitative embodiment of the invention.
  • the device 5 corresponds for example to a personal computer PC, a laptop, a tablet or a mobile phone.
  • the device 5 comprises the following elements, connected to each other by a bus 55 of addresses and data that also transports a clock signal :
  • microprocessor 51 or CPU
  • a graphics card 52 comprising: • several Graphical Processor Units (or GPUs) 520,
  • GRAM Graphical Random Access Memory
  • I/O devices 54 such as for example a keyboard, a mouse, a webcam, and
  • the device 5 also comprises a display device 53 of display screen type directly connected to the graphics card 52 to display notably the displaying of synthesized images calculated and composed in the graphics card, for example live.
  • a dedicated bus to connect the display device 53 to the graphics card 52 offers the advantage of having much greater data transmission bitrates and thus reducing the latency time for the displaying of images composed by the graphics card.
  • a display device is external to the device 5 and is connected to the device 5 by a cable transmitting the display signals.
  • the device 5, for example the graphics card 52 comprises a means for transmission or connection (not shown in figure 5) adapted to transmit a display signal to an external display means such as for example an LCD or plasma screen or a video-projector.
  • register used in the description of memories 52, 56 and 57 designates in each of the memories mentioned, both a memory zone of low capacity (some binary data) as well as a memory zone of large capacity (enabling a whole program to be stored or all or part of the data representative of data calculated or to be displayed).
  • the microprocessor 51 When switched-on, the microprocessor 51 loads and executes the instructions of the program contained in the RAM 57.
  • the random access memory 57 notably comprises:
  • - data 572 representative of the texture associated with the images 1 10 to 1 50, for example RGB information associated with the pixels of the images 1 1 0 to 1 50
  • - data 573 representative of the mesh (for example the coordinates of the vertex of the mesh elements of the mesh) of a surface representative of the scene represented in the images 1 1 0 to 150;
  • the random access memory GRAM 521 notably comprises:
  • a part of the RAM 57 is assigned by the CPU 51 for storage of the values 521 1 to 5214 and the parameters 521 5 if the memory storage space available in GRAM 521 is insufficient.
  • This variant however causes greater latency time in the composition of an image comprising a representation of the environment 1 composed from microprograms contained in the GPUs as the data must be transmitted from the graphics card to the random access memory 57 passing by the bus 55 for which the transmission capacities are generally inferior to those available in the graphics card for transmission of data from the GPUs to the GRAM and vice-versa.
  • the power supply 58 is external to the device 5.
  • Figure 6 illustrates a method for texturing a mesh of a surface 1 implemented for example in a computation unit illustrated on Figure 5, according to a particular and non limitative embodiment of the invention.
  • the different parameters of the device 5 are updated.
  • the parameters representative of the images and associated depth maps are initialised in any way.
  • a first texture information is associated with the mesh element.
  • a first image corresponds to an image of a plurality of images representing a scene according to several points of view, in which image a given mesh element is visible or at least partially visible.
  • the visibility of the mesh element is determined by projecting it in the first images and comparing the depth of the mesh element (determined by converting the coordinates of the mesh element from the world space to the image space) with the depth information associated with the pixels of the images onto which the mesh element is projected.
  • the visibility information associated with the mesh element and each image is received in addition to the data representative of the images (texture information, for example RGB information associated with the pixels of the images).
  • candidate first texture information As the mesh element is visible in at least two first images, there is several first texture information (called candidate first texture information) which is associated with the mesh element.
  • the candidate first texture information is each projected onto the first images.
  • the candidate first texture information are each projected onto only a part of the first images, for example on all first images excepted the first image providing the candidate first texture information which is projected on the first images.
  • a projection of a mesh element onto a first image corresponds to a conversion of the coordinates of a mesh element from the world space to the image space of the first image.
  • en error information is estimated for each candidate first texture information.
  • the error information is computed according to the results of a comparison between the projected first texture information and the texture information associated with the first image on which is projected the candidate first texture information.
  • the comparison is performed for each first image onto which the candidate first texture information is projected.
  • the error information associated with a candidate first texture information corresponds to the sum of all comparison results between the candidate first texture information and the texture information associated with the first images onto which the candidate first texture information is projected.
  • the error information associated with a candidate first texture information corresponds to the average of all comparison results between the candidate first texture information and the texture information associated with the first images onto which the candidate first texture information is projected
  • one of the candidate first texture information is selected among the plurality of candidate first texture information according to the error information estimated for and associated with each candidate first texture information.
  • the selected first texture information corresponds to the candidate first texture information having the least value of error information.
  • the invention is not restricted to a method for texturing a mesh but extends to the computation unit implementing such a method and to the display device or mobile device comprising a computation unit implementing the texturing of a mesh or the display of the images resulting from the texturing process.
  • the invention also concerns a method for selecting a texture information to be assigned to a mesh element among a plurality of candidate texture information and also to a method for coding and transmitting the selected texture information associated with a mesh of a 2D or 3D representation of a scene in a multi-view system.
  • the implementation of calculations necessary to the texturing of the mesh and to the selection of a texture information to be assigned to a mesh element is not limited either to an implementation in shader type microprograms but also extends to an implementation in any program type, for example programs that can be executed by a CPU type microprocessor.
  • the use of the invention is not limited to a live utilisation but also extends to any other utilisation, for example for processing known as postproduction processing in a recording studio for the display of synthesis images for example.
  • the implementation of the invention in postproduction offers the advantage of providing an excellent visual display in terms of realism notably while reducing the required calculation time.
  • the implementations described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms (for example, an apparatus or program).
  • An apparatus may be implemented in, for example, appropriate hardware, software, and firmware.
  • the methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants ("PDAs”), and other devices that facilitate communication of information between end-users.
  • PDAs portable/personal digital assistants
  • Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and/or depth information.
  • equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices.
  • the equipment may be mobile and even installed in a mobile vehicle.
  • the methods may be implemented by instructions being performed by a processor, and such instructions (and/or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette (“CD"), an optical disc (such as, for example, a DVD, often referred to as a digital versatile disc or a digital video disc), a random access memory (“RAM”), or a read-only memory (“ROM”).
  • the instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination.
  • a processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.
  • implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted.
  • the information may include, for example, instructions for performing a method, or data produced by one of the described implementations.
  • a signal may be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntax-values written by a described embodiment.
  • Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal.
  • the formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream.
  • the information that the signal carries may be, for example, analog or digital information.
  • the signal may be transmitted over a variety of different wired or wireless links, as is known.
  • the signal may be stored on a processor-readable medium.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Graphics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)
  • Image Generation (AREA)

Abstract

L'invention concerne un procédé de texturation d'un maillage associé à une surface (1) représentative d'une scène capturée dans une pluralité d'images (110 à 150), au moins un élément de maillage (10) dudit maillage étant au moins partiellement visible à partir d'au moins deux premières images (110 à 140) de la pluralité d'images (110 à 150). De manière à réduire la quantité d'informations de texture associées à des données multi-vues représentatives de la scène, le procédé comprend les étapes suivantes : - pour chaque première image (110 à 140), l'association d'une première information de texture avec ledit élément de maillage (10); - la projection de la première information de texture dans lesdites au moins deux premières images (110 à 140); - pour chaque première information de texture, l'estimation d'une information d'erreur selon un résultat de comparaison entre la première information de texture projetée et l'information de texture desdites au moins deux premières images (110 à 140); - et enfin, la sélection de l'une desdites au moins deux premières informations de texture selon ladite information d'erreur. L'invention concerne également une unité de calcul correspondante.
PCT/EP2012/069862 2012-01-13 2012-10-08 Procédé de texturation de maillage multi-vues et dispositif correspondant WO2013104440A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP12769115.2A EP2803041B1 (fr) 2012-01-13 2012-10-08 Procédé de texturation de maillage multi-vues et dispositif correspondant
US14/372,018 US20140354632A1 (en) 2012-01-13 2012-10-08 Method for multi-view mesh texturing and corresponding device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP12305047.8 2012-01-13
EP12305047 2012-01-13

Publications (1)

Publication Number Publication Date
WO2013104440A1 true WO2013104440A1 (fr) 2013-07-18

Family

ID=46980975

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2012/069862 WO2013104440A1 (fr) 2012-01-13 2012-10-08 Procédé de texturation de maillage multi-vues et dispositif correspondant

Country Status (3)

Country Link
US (1) US20140354632A1 (fr)
EP (1) EP2803041B1 (fr)
WO (1) WO2013104440A1 (fr)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150228106A1 (en) * 2014-02-13 2015-08-13 Vixs Systems Inc. Low latency video texture mapping via tight integration of codec engine with 3d graphics engine
KR102223064B1 (ko) * 2014-03-18 2021-03-04 삼성전자주식회사 영상 처리 장치 및 방법
EP3349182A1 (fr) * 2017-01-13 2018-07-18 Thomson Licensing Procédé, appareil et flux de format vidéo immersif
JP6425780B1 (ja) * 2017-09-22 2018-11-21 キヤノン株式会社 画像処理システム、画像処理装置、画像処理方法及びプログラム
US11516452B2 (en) * 2020-06-18 2022-11-29 Microsoft Technology Licensing, Llc Systems and methods for temporal corrections for parallax reprojection
WO2022133569A1 (fr) * 2020-12-22 2022-06-30 Prevu3D Inc. Procédés et système de reconstruction de maillage texturé à partir de données de nuage de points

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR1161788A (fr) 1956-10-24 1958-09-04 Perfectionnements au traitement des liqueurs contenant des matières sèches à pouvoir calorifique élevé, notamment des liqueurs noires de papeteries

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR1161788A (fr) 1956-10-24 1958-09-04 Perfectionnements au traitement des liqueurs contenant des matières sèches à pouvoir calorifique élevé, notamment des liqueurs noires de papeteries

Non-Patent Citations (13)

* Cited by examiner, † Cited by third party
Title
BASTIAN GOLDLUECKE ET AL: "Superresolution texture maps for multiview reconstruction", COMPUTER VISION, 2009 IEEE 12TH INTERNATIONAL CONFERENCE ON, IEEE, PISCATAWAY, NJ, USA, 29 September 2009 (2009-09-29), pages 1677 - 1684, XP031672708, ISBN: 978-1-4244-4420-5 *
ECKERT G ET AL: "Shape refinement for reconstructing 3D-objects using an analysis-synthesis approach", PROCEEDINGS 2001 INTERNATIONAL CONFERENCE ON IMAGE PROCESSING. ICIP 2001 - THESSALONIKI, GREECE, OCT. 7 - 10, 2001; [INTERNATIONAL CONFERENCE ON IMAGE PROCESSING], INSTITUTE OF ELECTRICAL AND ELECTRONICS ENGINEERS, NEW YORK, NY, vol. 3, 7 October 2001 (2001-10-07), pages 903 - 906, XP010563497, ISBN: 978-0-7803-6725-8, DOI: 10.1109/ICIP.2001.958270 *
EHSAN AGANJ ET AL: "Spatio-Temporal Shape from Silhouette using Four-Dimensional Delaunay Meshing", COMPUTER VISION, 2007. ICCV 2007. IEEE 11TH INTERNATIONAL CONFERENCE O N, IEEE, PI, 1 October 2007 (2007-10-01), pages 1 - 8, XP031194505, ISBN: 978-1-4244-1630-1 *
EHSAN AGANJ; JEAN-PHILIPPE PONS; FLORENT SEGONNE; RENAUD KERIVEN, SPATIO-TEMPORAL SHAPE FROM SILHOUETTE USING FOUR-DIMENSIONAL DELAUNAY MESHING
IIYAMA M ET AL: "Super-Resolution Texture Mapping from Multiple View Images", PATTERN RECOGNITION (ICPR), 2010 20TH INTERNATIONAL CONFERENCE ON, IEEE, PISCATAWAY, NJ, USA, 23 August 2010 (2010-08-23), pages 1820 - 1823, XP031772151, ISBN: 978-1-4244-7542-1 *
JUNG LEE ET AL: "Polygonal space carving with geometric anti-aliasing", THE VISUAL COMPUTER, vol. 20, no. 4, 1 June 2004 (2004-06-01), pages 229 - 242, XP055054216, ISSN: 0178-2789, DOI: 10.1007/s00371-003-0229-8 *
KIRIAKOS N KUTULAKOS ET AL: "A Theory of Shape by Space Carving", INTERNATIONAL JOURNAL OF COMPUTER VISION, KLUWER ACADEMIC PUBLISHERS, BO, vol. 38, no. 3, 1 July 2000 (2000-07-01), pages 199 - 218, XP019216357, ISSN: 1573-1405, DOI: 10.1023/A:1008191222954 *
KIRIAKOS N. KUTULAKOS; STEVEN M. SEITZ: "A Theory of Shape by Space Carving", INTERNATIONAL JOURNAL OF COMPUTER VISION, vol. 38, no. 3, 2000, pages 199 - 219, XP019216357, DOI: doi:10.1023/A:1008191222954
LORENSEN W E ET AL: "MARCHING CUBES: A HIGH RESOLUTION 3D SURFACE CONSTRUCTION ALGORITHM", SIAM JOURNAL ON COMPUTING, SOCIETY FOR INDUSTRIAL AND APPLIED MATHEMATICS, US, vol. 21, no. 4, 1 July 1987 (1987-07-01), pages 163 - 169, XP000576999, ISSN: 0097-5397 *
LORENSEN; CLINE, SIGGRAPH PROCEEDINGS, 1987
MICHAEL WASCHBÜSCH ET AL: "Scalable 3D video of dynamic scenes", THE VISUAL COMPUTER ; INTERNATIONAL JOURNAL OF COMPUTER GRAPHICS, SPRINGER, BERLIN, DE, vol. 21, no. 8-10, 1 September 2005 (2005-09-01), pages 629 - 638, XP019339162, ISSN: 1432-8726, DOI: 10.1007/S00371-005-0346-7 *
NIEM W: "Automatic reconstruction of 3D objects using a mobile camera", IMAGE AND VISION COMPUTING, ELSEVIER, GUILDFORD, GB, vol. 17, no. 2, 1 February 1999 (1999-02-01), pages 125 - 134, XP002258790, ISSN: 0262-8856, DOI: 10.1016/S0262-8856(98)00116-4 *
SHU-KAM CHOW ET AL: "Removal of Specular Reflection Component Using Multi-view Images and 3D Object Model", 13 January 2009, LECTURE NOTES IN COMPUTER SCIENCE; VOL. 5414; PROCEEDINGS OF THE 3RD PACIFIC RIM SYMPOSIUM ON ADVANCES IN IMAGE AND VIDEO TECHNOLOGY, SPRINGER BERLIN HEIDELBERG, BERLIN, HEIDELBERG, PAGE(S) 999 - 1009, ISBN: 978-3-540-92956-7, pages: 999 - 1009, XP019137304 *

Also Published As

Publication number Publication date
EP2803041A1 (fr) 2014-11-19
EP2803041B1 (fr) 2019-03-13
US20140354632A1 (en) 2014-12-04

Similar Documents

Publication Publication Date Title
US11348285B2 (en) Mesh compression via point cloud representation
US11363249B2 (en) Layered scene decomposition CODEC with transparency
EP2150065B1 (fr) Procédé et système pour rendu vidéo, produit de programme informatique associé
WO2009091563A1 (fr) Rendu basé sur une image de profondeur
EP2803041B1 (fr) Procédé de texturation de maillage multi-vues et dispositif correspondant
RU2764187C2 (ru) Обработка информации 3d-изображения на основании текстурных карт и сеток
US20140198182A1 (en) Representation and Coding of Multi-View Images Using Tapestry Encoding
US10510179B2 (en) Method and device for enriching the content of a depth map
Do et al. Quality improving techniques for free-viewpoint DIBR
US10074211B2 (en) Method and device for establishing the frontier between objects of a scene in a depth map
WO2021240069A1 (fr) Couches de texture de décalage pour codage et signalisation de réflexion et réfraction pour vidéo immersive et procédés pour vidéo volumétrique multicouche associés
CN114051734A (zh) 一种解码三维场景的方法和装置
US11979546B2 (en) Method and apparatus for encoding and rendering a 3D scene with inpainting patches
US9235922B2 (en) Method for modelling a 3D scene and corresponding device
KR20220011180A (ko) 체적 비디오 인코딩 및 디코딩을 위한 방법, 장치 및 컴퓨터 프로그램
US20220345681A1 (en) Method and apparatus for encoding, transmitting and decoding volumetric video
WO2023198426A1 (fr) Décimation de bloc dynamique dans un décodeur v-pcc
Marrinan et al. Image Synthesis from a Collection of Depth Enhanced Panoramas: Creating Interactive Extended Reality Experiences from Static Images
WO2023278488A1 (fr) Systèmes et procédés de mise en volume et de codage d'images bidimensionnelles
WO2020016350A1 (fr) Procédé et appareil permettant de générer une image immersive à partir d'images capturées par une pluralité de caméras
Farooq Real-time rendering of large surface-scanned range data natively on a GPU
Penta Depth Image Representation for Image Based Rendering

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12769115

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2012769115

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 14372018

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE