WO2022236837A1 - 虚拟视点绘制、渲染、解码方法及装置、设备、存储介质 - Google Patents

虚拟视点绘制、渲染、解码方法及装置、设备、存储介质 Download PDF

Info

Publication number
WO2022236837A1
WO2022236837A1 PCT/CN2021/093952 CN2021093952W WO2022236837A1 WO 2022236837 A1 WO2022236837 A1 WO 2022236837A1 CN 2021093952 W CN2021093952 W CN 2021093952W WO 2022236837 A1 WO2022236837 A1 WO 2022236837A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixels
segmentation
region
type
segmented
Prior art date
Application number
PCT/CN2021/093952
Other languages
English (en)
French (fr)
Inventor
杨铀
苏永全
刘琼
吴科君
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Priority to PCT/CN2021/093952 priority Critical patent/WO2022236837A1/zh
Priority to CN202180097722.1A priority patent/CN117242768A/zh
Publication of WO2022236837A1 publication Critical patent/WO2022236837A1/zh
Priority to US18/508,689 priority patent/US20240087185A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/001—Texturing; Colouring; Generation of texture or colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/40—Filling a planar surface by adding surface attributes, e.g. colour or texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows

Definitions

  • Embodiments of the present application relate to computer vision technology, and relate to but are not limited to virtual viewpoint drawing, rendering, decoding methods and devices, devices, and storage media.
  • the generated depth map (Depth Map) under the target viewpoint will have some noise areas and transition zone areas, resulting in poor image quality of the final generated texture map. fulfil requirements.
  • the virtual viewpoint drawing, rendering, decoding method, device, device, and storage medium provided by the embodiment of the present application can better optimize the depth map under the target viewpoint, thereby obtaining a texture map with better image quality;
  • the virtual viewpoint rendering, rendering, decoding method, device, device, and storage medium provided by the embodiment are implemented as follows:
  • the virtual viewpoint drawing method includes: generating a first visible map (Visibility Map) and a first texture map (Texture Map) of the target viewpoint according to the atlas (Atlas) of the input viewpoint (Source Views); and Determining the number of segmented regions; performing region segmentation on the first texture map according to the number of segmented regions to obtain a region segmentation result; determining a parameter representing the segmentation performance according to the region segmentation result; when the parameter representing the segmentation performance When the first condition is not satisfied, re-segmentation is performed on the first texture map until the second condition is met or the parameter representing the segmentation performance satisfies the first condition, and the final region segmentation of the first texture map is obtained Result; wherein, the re-segmentation operation includes: adjusting the number of divided regions, and dividing the first texture map according to the adjusted number of divided regions to obtain a new region segmentation result; according to the new region segmentation result , to determine the parameter representing the
  • the rendering (Rendering) method provided by the embodiment of the present application includes: obtaining the second texture map of the target viewpoint according to the virtual viewpoint drawing method provided in the embodiment of the present application; generating the target according to the second texture map of the target viewpoint The target view (Viewport) of the viewpoint.
  • the decoding method provided by the embodiment of the present application includes: decoding the input code stream to obtain the atlas of the input viewpoint; performing the steps in the virtual viewpoint rendering method as described in the embodiment of the present application on the atlas of the input viewpoint , to obtain a second texture map of the target viewpoint; and generate a target view of the target viewpoint according to the second texture map of the target viewpoint.
  • the virtual viewpoint drawing device includes: a generation module, used to generate the first visible image and the first texture image of the target viewpoint according to the atlas of the input viewpoint; a determination module, used to determine the number of divided regions; A module, configured to perform region segmentation on the first texture map according to the number of segmented regions to obtain a region segmentation result; determine a parameter representing the segmentation performance according to the region segmentation result; when the parameter representing the segmentation performance is not When the first condition is met, perform a re-segmentation operation on the first texture map until the second condition is met or the parameter representing the segmentation performance satisfies the first condition, and the final region segmentation result of the first texture map is obtained; wherein , the re-segmentation operation includes: adjusting the number of divided regions, and dividing the first texture map according to the adjusted number of divided regions to obtain a new region segmentation result; according to the new region segmentation result, determining the The parameters representing the segmentation performance are described above; the
  • the rendering device includes: a virtual viewpoint drawing module, configured to obtain a second texture map of the target viewpoint according to the steps in the virtual viewpoint rendering method described in the embodiment of the present application; a target view generating module, configured to A target view of the target viewpoint is generated according to the second texture map of the target viewpoint.
  • the decoding device includes: a decoding module, configured to decode the input code stream to obtain an atlas of input viewpoints;
  • the step in the method for rendering a virtual viewpoint described in the embodiment is to obtain the second texture map of the target viewpoint;
  • the target view generating module is configured to generate the target view of the target viewpoint according to the second texture map of the target viewpoint .
  • the electronic device provided in the embodiment of the present application includes a memory and a processor, the memory stores a computer program that can run on the processor, and the processor implements the virtual view method provided in the embodiment of the present application when executing the program, Or, when the processor executes the program, the rendering method described in the embodiment of the present application is realized, or when the processor executes the program, the decoding method described in the embodiment of the present application is realized.
  • the computer-readable storage medium provided by the embodiment of the present application has a computer program stored thereon, and when the computer program is executed by the processor, the virtual view point rendering method described in the embodiment of the present application is implemented, or, when the computer program is executed by the processor Realize the rendering method described in the embodiment of the present application, or realize the decoding method described in the embodiment of the present application when the computer program is executed by a processor.
  • the quality of the first visible image is not directly used to improve the quality of the first visible image, but the Whether the parameter representing the segmentation performance of the region segmentation result satisfies the first condition; if the first condition is not satisfied, then adjust the number of the segmented regions, and re-region-segment the first texture map based on this, and iterate until the obtained region segmentation result Until the parameters representing the segmentation performance satisfy the first condition or the number of segmentations meets the second condition, the final region segmentation result of the first texture map is used to improve the quality of the first visible image; in this way, on the one hand, the first The noise area and the transition zone area in the visible image can be obtained to obtain a second visible image with better quality, and then a second texture image that meets the quality requirements; The region segmentation results; the user does not need to directly give an appropriate number of segmentation regions, thereby lowering the
  • FIG. 1 is a schematic diagram of a system architecture that may be applicable to an embodiment of the present application
  • FIG. 2 is a schematic diagram of the implementation flow of the virtual viewpoint rendering method in the embodiment of the present application.
  • FIG. 3 is a schematic diagram of an implementation process of a virtual viewpoint rendering method in an embodiment of the present application
  • FIG. 4 is a schematic diagram of an implementation flow of a virtual viewpoint rendering method according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram of an accurate segmented region and an inaccurately segmented segmented region according to an embodiment of the present application
  • Fig. 6 is a schematic flow diagram of optimizing the depth map in viewpoint generation by using superpixel segmentation on the texture map
  • FIG. 7 is a schematic diagram of an implementation process of a method for adaptively selecting the number of superpixels based on iterations according to an embodiment of the present application
  • FIG. 8 is a schematic structural diagram of a virtual viewpoint rendering device according to an embodiment of the present application.
  • FIG. 9 is a schematic structural diagram of a rendering device according to an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of a decoding device according to an embodiment of the present application.
  • FIG. 11 is a schematic diagram of a hardware entity of an electronic device according to an embodiment of the present application.
  • first ⁇ second ⁇ third and so on involved in the embodiment of the present application are used to distinguish similar or different objects, and do not represent a specific ordering of objects. Understandably, “first ⁇ second ⁇ Third, ", etc., where permitted, may be interchanged in a specific order or sequence such that the embodiments of the application described herein can be practiced in an order other than that illustrated or described herein.
  • Fig. 1 shows a kind of system architecture that the embodiment of the present application may be applicable to, namely Moving Picture Experts Group (Moving Picture Experts Group, MPEG) in 3 degrees of freedom+ (3 degrees of freedom+, 3DoF+) immersive video test model (Test The system architecture 10 of the decoding end of Model of Immersive Video (TMIV), as shown in Figure 1, the system architecture 10 includes: decoding access unit (Decoded access unit) 11 and rendering unit (Rendering unit) 12; wherein, decoding access Unit 11 includes various metadata and atlas information obtained after decoding. Afterwards, these information will be transmitted to the rendering unit 12 for virtual viewpoint rendering.
  • Decoded access unit Decoded access unit
  • rendering unit rendering unit
  • Subunits marked with optional (opt.) represent optional subunits, and since these subunits are not involved in the technical solutions of the embodiments of the present application, they will not be described here.
  • the patch culling subunit 121 of the rendering unit 12 filters the patches in the atlas information according to the target viewpoint parameters (viewport parameters), and removes the patches that do not overlap with the target view, thereby reducing the virtual viewpoint The amount of calculations when drawing.
  • the occupancy reconstruction (Occupancy reconstruction) subunit 122 of the rendering unit 12 finds out the position of each tile (patch) in the view according to the information transmitted by the decoding access unit 11, and then pastes the screened tiles (patches) into the The corresponding position completes the pruned view reconstruction (Pruned view reconstruction), and obtains the reconstructed depth map under the input viewpoint.
  • the View Weighting Synthesizer (VWS) subunit is divided into two modules: a Visibility generating module 1231 and a Shading module 1232 .
  • the visible image generating module 1231 is used to generate the visible image under the target viewpoint based on the reconstructed depth map under the input viewpoint, and the coloring module 1232 is used to color the visible image generated in the visible image generating module 1231 to generate the texture image under the target viewpoint .
  • the depth map exists in the form of a visible map.
  • the visible map and the depth map have the same meaning, and both represent the distance relationship between the scene and the camera position. Unlike the depth map, the closer to the camera position in the visible map, the smaller the pixel value.
  • the texture map is different from the depth map, and the texture map is an image representing the texture of the scene.
  • the technical solution of using superpixel segmentation on the texture map to optimize the depth map in viewpoint generation is implemented on VWS and introduces three new modules: Superpixel Segmentation (Superpixel Segmentation ) module 1233, k-means clustering (K-means Clustering) module 1234 and replacement (Replacement) module 1235.
  • the superpixel segmentation module 1233 uses a superpixel segmentation algorithm to segment the texture image generated by the VWS, and applies the segmentation result to the generated visible image to obtain several superpixels on the visible image.
  • the k-means clustering module 1234 uses the k-means clustering algorithm to cluster each superpixel on the obtained visible image, so that the noise area and the transition zone area that need to be processed can be separated from the area that does not need to be processed, and finally by The replacement module 1235 replaces the pixel values of these regions that need to be processed, so that the noise regions and transition zone regions in the obtained depth map will be greatly reduced.
  • the visible image obtained after replacement is input to the coloring module 1232, because the texture map under the target viewpoint generated by the coloring module 1232 has a certain hole area, so the filling (Inpainting) subunit 124 is required to fill the hole, and the filled image is obtained. texture map.
  • the view space processing (Viewing space handling) subunit 125 can smoothly fade the filled texture map to black, and finally obtain the target view of the target viewpoint.
  • the technical solution of using superpixel segmentation on the texture map to optimize the depth map in viewpoint generation can be implemented on the basis of VWS in MPEG 3DoF+TMIV6, or it can be implemented on other types of virtual viewpoint rendering tools Realized, the technical solution is applicable to various types of virtual viewpoint drawing tools.
  • the method provided by the embodiment of the present application can be applied to any electronic device with data processing capability, and the electronic device can be a TV, a projector, a mobile phone, a personal computer, a tablet computer or a virtual reality (Virtual Reality, VR) headset Devices and other devices that have video codec functions or only have decoding functions.
  • the functions implemented by the method can be realized by calling the program code by the processor in the electronic device, and of course the program code can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.
  • FIG. 2 is a schematic diagram of the implementation flow of the virtual viewpoint rendering method in the embodiment of the present application. As shown in FIG. 2, the method may include the following steps 201 to 208:
  • Step 201 Generate a first visible image and a first texture image of a target viewpoint according to an atlas of an input viewpoint.
  • the electronic device can decode the input code stream to obtain the atlas of the depth map and the atlas of the texture map of the input viewpoint; then, perform clipped view restoration on the atlas of the depth map of the input viewpoint, Obtaining the reconstructed depth map of the input viewpoint; generating the first visible map of the target viewpoint according to the reconstructed depth map; and performing clipped view recovery on the atlas of the texture map of the input viewpoint to obtain the reconstructed texture map of the input viewpoint; Coloring the first visible image of the target viewpoint according to the reconstructed texture image, so as to obtain the first texture image.
  • the number of input viewpoints to be used is not limited.
  • the electronic device may generate the first visible image and the first texture image of the target viewpoint according to the atlas of one or more input viewpoints.
  • Step 202 determine the number of divided regions.
  • the electronic device may directly obtain the preset number of divided regions, or may calculate the number of divided regions according to the first texture map.
  • the number of divided regions can be determined through step 302 and step 303 of the following embodiment.
  • Step 203 Perform region segmentation on the first texture map according to the number of segmented regions to obtain a region segmentation result.
  • the electronic device may use a superpixel segmentation algorithm to perform superpixel segmentation on the first texture map according to the number of segmented regions (that is, the number of superpixels), and the obtained multiple superpixels (that is, a type of segmented region) example), a superpixel is a segmented region.
  • a superpixel refers to an irregular pixel block with certain visual significance composed of adjacent pixels with similar texture, color and brightness.
  • the superpixel segmentation algorithm may be a simple linear iterative clustering (Simple Linear Iterative Cluster, SLIC) superpixel segmentation algorithm, a Superpixels Extracted via Energy-Driven Sampling (SEEDS) algorithm, Contour-Relaxed Superpixels (CRS) algorithm, ETPS or Entropy Rate Superpixels Segmentation (ERS) algorithm, etc.
  • SLIC Simple Linear Iterative Cluster
  • SEEDS Energy-Driven Sampling
  • CRS Contour-Relaxed Superpixels
  • ETPS Entropy Rate Superpixels Segmentation
  • Step 204 according to the region segmentation result, determine a parameter representing the segmentation performance.
  • the parameters representing the segmentation performance include segmentation accuracy and/or the average number of pixels in the segmented region.
  • the meaning of the segmentation accuracy representation is not limited. Segmentation accuracy may refer to the proportion of the number of accurately segmented regions in the region segmentation result.
  • the first condition is that the segmentation accuracy is greater than a threshold, or the first condition is that the average of the segmented regions The number of pixels is less than or equal to the fourth threshold. Segmentation accuracy may also refer to the proportion of the number of inaccurately segmented regions in the region segmentation result.
  • the first condition is that the segmentation accuracy is less than the third threshold, or the number of segmented regions The average number of pixels is less than or equal to the fourth threshold.
  • the first condition when the parameter representing the segmentation performance is represented by the average number of pixels, the first condition may be that the average number of pixels is less than or equal to the fourth threshold.
  • Step 205 determining whether the parameter representing the segmentation performance satisfies the first condition or whether the number of divisions satisfies the second condition; if the parameter representing the segmentation performance does not meet the first condition, and the number of divisions does not meet the second condition, perform step 206; If the parameter representing the segmentation performance satisfies the first condition or the number of divisions satisfies the second condition, step 207 is executed.
  • the second condition is a reserved early exit condition. For example, after a certain number of splits, if the parameter indicating the split performance does not meet the first condition, no split will be performed; the purpose of this is to control the processing complexity Spend.
  • Step 206 adjust the number of divided regions, and then return to step 203 .
  • the first texture map needs to be re-segmented until the number of divisions meets the second condition or the indicated The segmentation performance parameter satisfies the first condition, and the final region segmentation result of the first texture map is obtained; wherein, the re-segmentation operation includes: adjusting the number of segmented regions, and adjusting the number of segmented regions according to the adjusted number of segmented regions Segmenting the first texture map to obtain a new region segmentation result; determining the parameter representing the segmentation performance according to the new region segmentation result.
  • the adjustment step size of the number of divided regions there is no limitation on the adjustment step size of the number of divided regions, and may be any value.
  • the number of divided regions before adjustment is 2000
  • the number of divided regions after adjustment is 2500, that is, the step size of each adjustment is 500.
  • the number of divided regions before adjustment is 2000
  • the number of divided regions after adjustment is 1000, that is, the adjustment step size is 1000.
  • Step 207 Update pixels to be updated in the region of the first visible image corresponding to the segmented region of the first texture image according to the final region segmentation result of the first texture image to obtain a second visible image.
  • the final region segmentation result of the first texture image is used as the region segmentation result of the first visible image, and the pixels in the segmented region of the first visible image are classified to obtain pixels of the first type and pixels of the second type ; Then, according to the relationship between the number of pixels of these two types of pixels and the relationship between the pixel values of these two types of pixels (such as mean or median value, etc.), determine the pixel to be updated in the segmented area.
  • pixel values of pixels to be updated in the segmented regions of the first visible image may be updated to obtain the second visible image.
  • the electronic device may filter the pixel value of the pixel to be updated in the first visible image, so as to update the pixel value.
  • the electronic device may also use the pixel replacement value to replace the pixel value of the pixel to be updated, so as to update the pixel value.
  • each segment area corresponds to a pixel replacement value
  • the pixel replacement value may be an average value of pixel values of pixels not to be updated in the corresponding segment area.
  • each segmented area corresponds to a cluster center other than the pixel class to be updated, so the pixel value of the center can be used as the pixel replacement value.
  • Step 208 coloring the second visible image to obtain a second texture image of the target viewpoint.
  • the first visible image is subjected to quality improvement processing (step 207); in this way, on the one hand, the noise region and the transition zone area in the first visible image can be better eliminated, thereby obtaining a second visible image with better quality, and then satisfying quality requirements of the second texture map; on the other hand, since the number of segmented regions can be adaptively adjusted, a better region segmentation result can be obtained; therefore, it is not necessary for the user to directly provide an appropriate number of segmented regions, thereby reducing the cost of the method.
  • the use threshold reduces the requirements for the user's relevant knowledge level.
  • FIG. 3 is a schematic diagram of the implementation flow of the method for rendering a virtual viewpoint in the embodiment of the present application. As shown in FIG. 3 , the method may include the following steps 301 to 313:
  • Step 301 generating a first visible image and a first texture image of a target viewpoint according to an atlas of an input viewpoint
  • Step 302 determining the image resolution of the first texture map
  • Step 303 Determine the number of divided regions according to the image resolution and the preset average number of pixels of the divided regions.
  • the electronic device can calculate the number of divided regions init_numSuperpixel according to the following formula (1) or formula (2):
  • init_numSuperpixel round(width*height/M)(1);
  • init_numSuperpixel round(width*height/M/100)*100(2);
  • width represents the width of the image
  • height represents the height of the image
  • round represents rounding
  • M represents the average number of pixels in the preset segmented area (for example, the value is 2000)
  • the calculated precision of init_numSuperpixel is hundreds.
  • M is usually set to a maximum value, so that the initially set number of divided regions is a minimum value.
  • the electronic device can automatically determine the number of divided regions based on the image resolution of the first texture map, so that there is no need to preset the number of divided regions, thereby lowering the threshold for using the virtual rendering method provided by the embodiment of the present application.
  • Step 304 performing region segmentation on the first texture map according to the number of segmented regions to obtain a region segmentation result
  • Step 305 using the region segmentation result as the region segmentation result of the first visible image, and classifying the pixels in the segmented region of the first visible image to obtain pixels of the first type and pixels of the second type.
  • the electronic device can directly apply the region segmentation result of the first texture image to the first visible image, and the region segmentation result of the first texture image is the region segmentation result of the first visible image.
  • the electronic device may classify the pixels of each segmented area of the first visible image, for example, classify the pixels of each segmented area into two categories.
  • Classification algorithms can be varied.
  • the classification algorithm is K-means clustering, decision tree, Bayesian, artificial neural network, support vector machine or classification based on association rules, etc.
  • Step 306 according to the relationship between the first type of pixels and the second type of pixels, determine whether the segmented area is an inaccurately segmented area.
  • the electronic device may implement step 306 according to steps 405 to 407 of the following embodiments.
  • Step 307 Determine the segmentation accuracy of the region segmentation result according to the number of inaccurately segmented regions in the first visible image.
  • the segmentation accuracy may be the ratio of the number of inaccurately segmented regions in the region segmentation results.
  • the ratio is the number of inaccurately segmented regions divided by the number of accurately segmented regions, or the ratio is the number of inaccurately segmented regions divided by the number of segmented regions obtained in step 303, or the ratio is the number of inaccurately segmented regions.
  • the segmentation accuracy may also be the ratio of the number of accurately segmented regions in the region segmentation results. For example, the proportion is divided by the number of accurately segmented regions divided by the number of segmented regions obtained in step 303, or divided by the total number of segmented regions in the region segmentation result.
  • Step 308 determine whether the segmentation accuracy meets the first condition; if yes, execute step 311; otherwise, execute step 309;
  • Step 309 determine whether the number of divisions meets the second condition; if yes, execute step 311; otherwise, execute step 310;
  • Step 310 adjust the number of divided regions, and then return to step 304;
  • step 311 If the segmentation accuracy does not meet the first condition, and the number of divisions does not meet the second condition, then adjust the number of divided regions in the step; then, re-set the first texture map according to the adjusted number of divided regions Perform region segmentation until the segmentation accuracy of the obtained region segmentation result satisfies the first condition, or the number of divisions satisfies the second condition. Then, enter step 311;
  • Step 311 using the final region segmentation result of the first texture map as the region segmentation result of the first visible map, and identifying pixels to be updated in the segmented region of the first visible map;
  • Step 312 Update the pixel values of the pixels to be updated in the segmented area according to the pixel values of the non-updated pixels in the segmented area, so as to obtain the second visible image.
  • the so-called non-to-be-updated pixels refer to other pixels in the segmented area except the to-be-updated pixels.
  • the pixels in the segmented area are classified into a first type of pixel and a second type of pixel, wherein the first type of pixel is a pixel to be updated, and the second type of pixel is a pixel not to be updated.
  • the electronic device may filter the pixel values of the pixels to be updated, and may also use the pixel replacement value of the segmented area to replace the pixel values of the pixels to be updated in the segmented area.
  • the pixel values of the pixels to be updated in the segmented area may be replaced with representative pixel values such as the mean or median value of the pixel values of the non-updated pixels in the segmented area.
  • Step 313 coloring the second visible image to obtain a second texture image of the target viewpoint.
  • FIG. 4 is a schematic diagram of the implementation flow of the method for rendering a virtual viewpoint in the embodiment of the present application. As shown in FIG. 4 , the method may include the following steps 401 to 415:
  • Step 401 generating a first visible image and a first texture image of a target viewpoint according to an atlas of an input viewpoint
  • Step 402 determine the number of divided regions
  • Step 403 performing region segmentation on the first texture map according to the number of segmented regions, to obtain a region segmentation result
  • Step 404 using the region segmentation result as the region segmentation result of the first visible image, and classifying the pixels in the segmented region of the first visible image to obtain pixels of the first type and pixels of the second type;
  • Step 405 determining the absolute value of the difference between the pixel value representation of the first type of pixels and the pixel value representation of the second type of pixels.
  • the so-called representative pixel value refers to a representative pixel value among the first type of pixels, and the pixel value may be an average value or a median value of the pixel values of the first type of pixels in the segmented area.
  • Step 406 determining a first result of dividing the number of pixels of the first type of pixels by the number of pixels of the second type of pixels;
  • Step 407 if the following conditions are met: the absolute value of the difference is greater than a first threshold, the first result is less than a second threshold, and the first result is greater than the reciprocal of the second threshold, determine the segmented area for segmenting inaccurate regions.
  • the sizes of the first threshold and the second threshold are no limitation on the sizes of the first threshold and the second threshold. These thresholds can be set according to performance indicators of actual applications.
  • the value range of the first threshold is [25,33]
  • the value range of the second threshold is [5,10].
  • the first threshold is 30, and the second threshold is 5.
  • an ideal region segmentation result is segmented along the edge of the foreground, that is, each segmented region can better separate the foreground and the background.
  • the so-called unsatisfactory area segmentation refers to a segmented area that has both foreground content and background content. The segmented area spans the foreground and background. On the depth map, the color of the foreground is close to white, and the color of the background is close to black. It can be seen that there is a large difference between the foreground pixel value and the background pixel value, so when identifying the pixel to be updated, the foreground or background will be misjudged as the pixel to be updated, resulting in poor final optimization results.
  • the proportion of one type of pixels in the accurately segmented region is small, and the ratio of the number of pixels to the other type of pixels is less than a threshold, and the pixel values of the two types of pixels The difference is greater than a threshold. For example, as shown in FIG.
  • the segmented areas 51, 52, 53 and 54 are segmented areas with accurate segmentation, that is, there is a large difference in the number of pixels between one type of pixels and another type of pixels in the segmented area, and the difference in pixel values is also larger, then it may be four situations shown in area 51 to area 54, wherein area 51 is that the less one type of pixel is the foreground pixel (that is, the pixel to be updated), and the more one type of pixel is the background pixel; 52 is that less one class of pixels is background pixels (i.e. pixels to be updated), and more one class of pixels is foreground pixels; area 53 is that less one class of pixels is noise pixels (i.e.
  • a class of pixels is a background pixel; a class of pixels less in the region 54 is a noise pixel (i.e. a pixel to be updated), and a larger class of pixels is a foreground pixel; After updating the updated pixels, the edges of the final second texture map will be sharper and steeper; after updating the pixels to be updated such as in area 53 and area 54, the final second texture map will be less noise.
  • regions 55 and 56 may be inaccurately segmented regions, as shown in Figure 5, no matter whether it is region 55 or region 56, although the pixel values of the two types of pixels differ greatly, that is, the difference
  • the absolute value of is greater than the first threshold, but the number of pixels of the two types is similar, that is, the first result is less than the second threshold and the first result is greater than the reciprocal of the second threshold, indicating that this type of area contains both foreground and background, if for this If a certain class of pixels in the class area is updated, an error will occur, which will neither eliminate the noise nor make the edge sharper; therefore, this type of area is not conducive to obtaining sharper edges and less noise.
  • Second texture map Second texture map.
  • Step 408 Determine the ratio of the number of inaccurately segmented regions in the region segmentation result in the first visible image as the segmentation accuracy.
  • the proportion may be the number of inaccurately segmented regions divided by the number of accurately segmented regions, or divided by the number of segmented regions obtained in step 402, or divided by the total number of segmented regions in the region segmentation result.
  • Step 409 using the region segmentation result as the region segmentation result of the first visible image, and determining the average number of pixels in the segmented region;
  • Step 410 determine whether the segmentation accuracy is greater than a third threshold, and whether the average number of pixels is greater than a fourth threshold; if yes, perform step 411; otherwise, perform step 413.
  • the segmentation accuracy is greater than the third threshold and the average number of pixels is greater than the fourth threshold, it means that the parameters representing the segmentation performance of the region segmentation result do not meet the first condition; otherwise, the segmentation accuracy is less than or equal to the third threshold, Alternatively, if the average number of pixels is less than or equal to the fourth threshold, it means that the parameter representing the segmentation performance satisfies the first condition. At this point, the iteration ends and step 413 is entered.
  • the quality of the finally obtained second texture map will not be significantly improved. That is to say, when the average number of pixels reaches a certain precision, the obtained second texture map tends to converge. Therefore, when judging whether to use the current region segmentation result as the final region segmentation result, it depends on both the segmentation accuracy and the average number of pixels, so that the iteration can be ended as soon as possible while ensuring the image quality of the second texture map , thereby saving the calculation process and improving the real-time performance of virtual viewpoint rendering.
  • a certain precision for example, 500
  • condition of the average number of pixels may not be relied on, that is, if the segmentation accuracy is greater than the third threshold, it is determined that the parameter representing the segmentation performance of the segmentation result does not meet the first condition; otherwise, the determination of the region The parameter representing the segmentation performance of the segmentation result satisfies the first condition.
  • Step 411 determine whether the number of divisions meets the second condition; if yes, enter step 413; otherwise, execute step 412;
  • Step 412 adjust the number of divided regions, and then return to step 403;
  • the first texture map is re-segmented according to the adjusted number of segmented regions, and steps 404 to 411 are executed again until the number of divisions satisfies the first condition.
  • the second condition or the parameters representing the segmentation performance meet the first condition, and then enter step 413;
  • Step 413 using the final region segmentation result of the first texture image as the region segmentation result of the first visible image, according to the relationship between the number of pixels of the first type of pixels and the number of pixels of the second type of pixels, and the The relationship between the pixel value representative of the first type of pixel and the pixel value representative of the second type of pixel is used to determine the pixel to be updated in the segmented area.
  • step 404 when step 404 is executed, each segmented region of the first visible image has already been classified, so in step 413, the calculation result of step 404 can be used directly without re-classification operation.
  • the second result of subtracting the pixel value representation of the second type of pixels from the pixel value of the first type of pixels is greater than or equal to the fifth threshold, and the number of pixels of the first type of pixels
  • the first result of dividing by the number of pixels of the second type of pixels is greater than or equal to the sixth threshold, it is determined that the second type of pixels are pixels to be updated in the segmented area
  • the third result of subtracting the pixel value represented by the pixel value of the first type of pixels from the pixel value is greater than or equal to the fifth threshold, and the number of pixels of the second type of pixels divided by the number of pixels of the first type of pixels
  • the fourth result is greater than or equal to the sixth threshold, it is determined that the pixels of the first type are pixels to be updated in the segmented region.
  • the fifth threshold may be the same as or different from the aforementioned first threshold, and the fifth threshold may be greater than or less than the first threshold.
  • the sixth threshold may be set to be equal to or not equal to the aforementioned second threshold, and the sixth threshold may be greater than or less than the second threshold.
  • the value ranges of the first threshold and the fifth threshold are both [25, 33]
  • the value ranges of the second threshold and the sixth threshold are both [5, 10].
  • the pixels to be updated in the segmented area by determining the pixels to be updated in the segmented area by the above method, the pixels to be updated in areas 51, 52, 53 and 54 such as shown in FIG.
  • the edges in the second texture image are sharper and steeper; on the other hand, the noise area in the second texture image can be reduced. In a word, the image quality of the second texture map can be improved.
  • Step 414 updating the pixel values of the pixels to be updated in the segmented area of the first visible image to obtain a second visible image
  • Step 415 coloring the second visible image to obtain a second texture image of the target viewpoint.
  • the electronic device may use the final region segmentation result of the first texture image as the region segmentation result of the first visible image of adjacent N frames; wherein, N is greater than 0. That is to say, the region segmentation results of the first visible image of every consecutive N+1 frames are the same. It can be understood that the content of adjacent frame images has a relatively large similarity, so the first visible image of consecutive N+1 frames can share the final region segmentation result of the first texture image, so that there is no need for the first texture image of each frame The final region segmentation results are all calculated, so that the amount of calculation is saved under the premise of ensuring the image quality of each second texture map.
  • the final region segmentation result of the first texture map is used as the region segmentation result of the first visible image in the next frame to the first visible image in the next N frames; N is greater than 1; re-determine the first visible image in the next N+1 frame
  • the final area segmentation result of the texture map is used as the area segmentation result of the first visible image of subsequent adjacent N frames.
  • An embodiment of the present application provides a rendering method, which can be applied not only to electronic devices, but also to rendering devices.
  • the method may include: first, performing the steps in the virtual viewpoint rendering method described in the embodiments of the present application, Obtain a second texture map of the target viewpoint; then, generate a target view of the target viewpoint according to the second texture map of the target viewpoint.
  • An embodiment of the present application provides a decoding method, the method comprising: decoding the input code stream to obtain an atlas of input viewpoints; performing the steps in the above virtual viewpoint drawing method on the atlases of input viewpoints to obtain The second texture map of the target viewpoint; generating a target view of the target viewpoint according to the second texture map of the target viewpoint.
  • the encoding end can use a larger quantization parameter for the depth map on the basis of ensuring the image quality of the target view. Compression coding is performed to reduce the coding overhead of the depth map, thereby improving the overall coding efficiency.
  • numSuperpixel indicates the number of superpixels specified by the user when performing superpixel segmentation on the texture map, which will affect the effect of superpixel segmentation and further affect the effect of depth map optimization.
  • numSuperpixel is too small, it cannot be better segmented along the edge of the object, and when numSuperpixel is too large, it will increase the complexity of processing.
  • the above-mentioned depth map optimization technology does not give the setting method of numSuperpixel, and the user needs to determine its size by himself. This puts forward requirements on the relevant knowledge level of the user and raises the threshold for using the technology.
  • Step 701 calculate the initial number of superpixels (init_numSuperpixel).
  • the initial number of superpixels can be obtained according to the resolution of the image:
  • init_numSuperpixel round(width*height/2000/100)*100(3);
  • width represents the width of the image
  • height represents the height of the image
  • round represents rounding
  • precision of the calculated init_numSuperpixel is 100.
  • Step 703 perform superpixel segmentation on the first texture map with temp_numSuperpixel
  • Step 705 calculate the sum of the total number of inaccurately segmented superpixels.
  • the determination rule for inaccurately segmented superpixels is as follows: after the step of k-means clustering, each superpixel is divided into two parts. Assuming that the cluster centers of the two parts and the number of included pixels are respectively center 0 , center 1 , sum 0 , and sum 1 , when the conditions shown in the following formula (4) are met, the superpixel can be judged as inaccurate segmentation of superpixels:
  • Step 706, calculate segmentation accuracy accuracy sum/temp_numSuperpixel
  • the threshold accu can be slightly adjusted according to the accuracy requirements, and it is recommended to set it to 0.025;
  • the number of pixels contained in a pixel is less than 500, that is, width*height/temp_numSuperpixel ⁇ 500.
  • the iteration ends when one of the two is satisfied. If none are satisfied, increase the number of superpixels by step, and then return to step 703 .
  • the size of the parameter step affects the convergence speed of the method, and it is recommended to set it to 500.
  • this technology can set a fixed period period, and execute the above-mentioned superpixel number adaptive algorithm once per period frame for updating the number of superpixels .
  • the solution provided by the embodiment of the present application can automatically track a more appropriate number of superpixels, realize the self-adaptation of the number of superpixels, and reduce the problem of using superpixel segmentation on texture maps to generate viewpoints.
  • the usage threshold of depth map optimization technology can automatically track a more appropriate number of superpixels, realize the self-adaptation of the number of superpixels, and reduce the problem of using superpixel segmentation on texture maps to generate viewpoints.
  • the virtual viewpoint rendering device provided by the embodiments of the present application, including the included modules and the units included in each module, can be implemented by a processor; of course, it can also be implemented by a specific logic circuit;
  • the processor may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).
  • FIG. 8 is a schematic structural diagram of a virtual viewpoint rendering device according to an embodiment of the present application. As shown in FIG. 8 , the virtual viewpoint rendering device 80 includes:
  • a generating module 801 configured to generate a first visible image and a first texture image of a target viewpoint according to an atlas of an input viewpoint;
  • a determining module 802 configured to determine the number of divided regions
  • the segmentation module 803 is configured to perform region segmentation on the first texture map according to the number of segmented regions to obtain a region segmentation result; determine a parameter representing the segmentation performance according to the region segmentation result; when the parameter representing the segmentation performance is not When the first condition is met, re-segment the first texture map until the second condition is met or the segmentation accuracy meets the first condition, and the final region segmentation result of the first texture map is obtained; wherein, the The re-segmentation operation includes: adjusting the number of divided regions, and dividing the first texture map according to the adjusted number of divided regions to obtain a new region segmentation result; according to the new region segmentation result, determining the representation Parameters for segmentation performance;
  • An update module 804 configured to update pixels to be updated in the first visible image region corresponding to the segmented regions of the first texture image according to the final region segmentation result of the first texture image, to obtain a second visible image;
  • the coloring module 805 is configured to color the second visible image to obtain a second texture image of the target viewpoint.
  • the determining module 802 is configured to: determine the image resolution of the first texture map; determine the number of divided regions according to the image resolution and the preset average number of pixels of the divided regions.
  • the parameter representing the segmentation performance includes the segmentation accuracy of the region segmentation result;
  • the segmentation module 803 is configured to: use the region segmentation result as the region segmentation result of the first visible image, and Classify the pixels in the segmented area of the first visible image to obtain pixels of the first type and pixels of the second type; determine whether the segmented area is Segmenting inaccurate regions; determining the segmentation accuracy of the region segmentation result according to the number of inaccurately segmented regions in the first visible image.
  • the segmentation module 803 is configured to: determine the absolute value of the difference between the pixel value of the first type of pixel and the pixel value of the second type of pixel; determine the first The first result of dividing the number of pixels of the class pixel by the number of pixels of the second class of pixels; if the following conditions are met: the absolute value of the difference is greater than the first threshold, the first result is less than the second threshold, and the If the first result is greater than the reciprocal of the second threshold, it is determined that the segmented region is an inaccurately segmented region.
  • the value range of the first threshold is [25,33]
  • the value range of the second threshold is [5,10].
  • the parameter representing the segmentation performance includes the average number of pixels of the segmented region; the segmentation module 803 is configured to: determine the ratio of the number of segmented regions with inaccurate segmentation in the region segmentation result is the segmentation accuracy; use the region segmentation result as the region segmentation result of the first visible image, and determine the average number of pixels in the segmented region; if the segmentation accuracy is less than or equal to a third threshold, or, The average number of pixels is less than or equal to a fourth threshold, and it is determined that the parameter representing segmentation performance satisfies the first condition; if the segmentation accuracy is greater than the third threshold, and the average number of pixels is greater than the first threshold Four thresholds, determining that the parameter representing segmentation performance does not satisfy the first condition.
  • the segmentation module 803 is further configured to: use the final area segmentation result of the first texture map as the area segmentation result of the first visible image in adjacent N frames; where N is greater than 0.
  • the update module 804 is configured to: obtain the pixel number of the first type of pixels and the pixel value representative of the first type of pixels; obtain the pixel number of the second type of pixels and the second The pixel value of the class pixel represents; according to the relationship between the number of pixels of the first class of pixels and the number of pixels of the second class of pixels, and the pixel value of the first class of pixels represents the relationship between the number of pixels of the second class of pixels The relationship represented by the value determines the pixels to be updated in the segmented area.
  • the update module 804 is configured to: the second result of subtracting the pixel value representation of the second type of pixels from the pixel value representation of the first type of pixels is greater than or equal to the fifth threshold, And when the first result of dividing the number of pixels of the first type of pixels by the number of pixels of the second type of pixels is greater than or equal to the sixth threshold, it is determined that the second type of pixels are to be updated in the segmented region pixel; the third result obtained by subtracting the pixel value of the first type of pixels from the pixel value of the second type of pixels is greater than or equal to the fifth threshold, and the number of pixels of the second type of pixels is divided by If the fourth result of the number of pixels of the first type of pixels is greater than or equal to the sixth threshold, it is determined that the first type of pixels are pixels to be updated in the segmented region.
  • the update module 804 is configured to: update the pixel values of the pixels to be updated in the segmented area according to the pixel values of the non-updated pixels in the segmented area of the first visible image, so as to obtain the Describe the second visible diagram.
  • FIG. 9 is a schematic structural diagram of the rendering device according to the embodiment of the present application. As shown in FIG. 9 , the rendering device 90 includes:
  • the virtual viewpoint rendering module 901 is configured to obtain a second texture map of the target viewpoint according to the steps in the virtual viewpoint rendering method described in the embodiment of the present application;
  • the target view generating module 902 is configured to generate a target view of the target viewpoint according to the second texture map of the target viewpoint.
  • FIG. 10 is a schematic structural diagram of the decoding device according to the embodiment of the present application. As shown in FIG. 10 , the decoding device 100 includes:
  • Decoding module 1001 configured to decode the input code stream to obtain an atlas of input viewpoints
  • a virtual view point drawing module 1002 configured to perform the steps in the virtual view point drawing method described in the embodiment of the present application on the atlas of the input view point to obtain the second texture map of the target view point;
  • a target view generating module 1003, configured to generate a target view of the target viewpoint according to the second texture map of the target viewpoint.
  • the above method is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
  • the computer software products are stored in a storage medium and include several instructions to make
  • the electronic device executes all or part of the methods described in the various embodiments of the present application.
  • the aforementioned storage medium includes: various media that can store program codes such as U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), magnetic disk or optical disk.
  • embodiments of the present application are not limited to any specific combination of hardware and software.
  • FIG. 11 is a schematic diagram of a hardware entity of the electronic device according to the embodiment of the present application.
  • the electronic device 110 includes a memory 1101 and a processor 1102, and the memory 1101 stores a computer program that can run on the processor 1102, and the processor 1102 implements the steps in the methods provided in the above-mentioned embodiments when executing the program.
  • the memory 1101 is configured to store instructions and applications executable by the processor 1102, and may also cache data to be processed or processed by each module in the processor 1102 and the electronic device 110 (for example, image data, audio data, etc. , voice communication data and video communication data), can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM).
  • FLASH FLASH
  • RAM Random Access Memory
  • an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the method provided in the foregoing embodiments are implemented.
  • references throughout this specification to "one embodiment” or “an embodiment” or “some embodiments” or “other embodiments” mean that particular features, structures, or characteristics related to the embodiments are included in this specification. In at least one embodiment of the application. Thus, appearances of "in one embodiment” or “in an embodiment” or “in some embodiments” or “in other embodiments” throughout this specification are not necessarily referring to the same embodiments. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments.
  • sequence numbers of the above-mentioned processes do not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not be used in the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • the serial numbers of the above embodiments of the present application are for description only, and do not represent the advantages and disadvantages of the embodiments.
  • the disclosed devices and methods may be implemented in other ways.
  • the above-described embodiments of the touch screen system are only illustrative.
  • the division of the modules is only a logical function division.
  • the mutual coupling, or direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or modules may be in electrical, mechanical or other forms of.
  • modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed to multiple network units; Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can be used as a single unit, or two or more modules can be integrated into one unit; the above-mentioned integration
  • the modules can be implemented in the form of hardware, or in the form of hardware plus software functional units.
  • the above-mentioned integrated units of the present application are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.
  • the computer software products are stored in a storage medium and include several instructions to make
  • the electronic device executes all or part of the methods described in the various embodiments of the present application.
  • the aforementioned storage medium includes various media capable of storing program codes such as removable storage devices, ROMs, magnetic disks or optical disks.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Graphics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Geometry (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)
  • Image Generation (AREA)

Abstract

本申请实施例公开了虚拟视点绘制、渲染、解码方法及装置、设备、存储介质，虚拟视点绘制方法包括：根据输入视点的图集生成目标视点的第一可见图和第一纹理图；根据确定的分割区域数，对第一纹理图进行区域分割，得到区域分割结果；根据区域分割结果，确定表示分割性能的参数；当表示分割性能的参数不满足第一条件时，对第一纹理图进行重分割操作，直至满足第二条件或表示分割性能的参数满足第一条件，得到第一纹理图的最终区域分割结果；根据第一纹理图的最终区域分割结果，更新与第一纹理图的分割区域对应的第一可见图区域中的待更新像素，得到第二可见图；对第二可见图进行着色，得到目标视点的第二纹理图。

Description

虚拟视点绘制、渲染、解码方法及装置、设备、存储介质 技术领域
本申请实施例涉及计算机视觉技术，涉及但不限于虚拟视点绘制、渲染、解码方法及装置、设备、存储介质。
背景技术
虚拟视点绘制中，由于不准确的深度估计和压缩失真等因素，生成的目标视点下的深度图(Depth Map)会存在一些噪点区域和过渡带区域，导致最终生成的纹理图的图像质量不能够满足要求。
发明内容
有鉴于此，本申请实施例提供的虚拟视点绘制、渲染、解码方法及装置、设备、存储介质，能够更好地优化目标视点下的深度图，从而得到图像质量更好的纹理图；本申请实施例提供的虚拟视点绘制、渲染、解码方法及装置、设备、存储介质，是这样实现的：
本申请实施例提供的虚拟视点绘制方法，包括：根据输入视点(Source Views)的图集(Atlas)，生成目标视点的第一可见图(Visibility Map)和第一纹理图(Texture Map)；以及确定分割区域数；根据所述分割区域数，对所述第一纹理图进行区域分割，得到区域分割结果；根据所述区域分割结果，确定表示分割性能的参数；当所述表示分割性能的参数不满足第一条件时，对所述第一纹理图进行重分割操作，直至满足第二条件或所述表示分割性能的参数满足所述第一条件，得到所述第一纹理图的最终区域分割结果；其中，所述重分割操作包括：调整所述分割区域数，根据调整后的分割区域数对所述第一纹理图进行分割，得到新的区域分割结果；根据所述新的区域分割结果，确定所述表示分割性能的参数；根据所述第一纹理图的最终区域分割结果，更新与所述第一纹理图的分割区域对应的第一可见图区域中的待更新像素，得到第二可见图；对所述第二可见图进行着色(Shading)，得到所述目标视点的第二纹理图。
本申请实施例提供的渲染(Rendering)方法，包括：根据本申请实施例提供的虚拟视点绘制方法，得到目标视点的第二纹理图；根据所述目标视点的第二纹理图，生成所述目标视点的目标视图(Viewport)。
本申请实施例提供的解码方法，包括：对输入的码流进行解码，得到输入视点的图集；对所述输入视点的图集执行如本申请实施例所述的虚拟视点绘制方法中的步骤，得到所述目标视点的第二纹理图；根据所述目标视点的第二纹理图，生成所述目标视点的目标视图。
本申请实施例提供的虚拟视点绘制装置，包括：生成模块，用于根据输入视点的图集，生成目标视点的第一可见图和第一纹理图；确定模块，用于确定分割区域数；分割模块，用于根据所述分割区域数，对所述第一纹理图进行区域分割，得到区域分割结果；根据所述区域分割结果，确定表示分割性能的参数；当所述表示分割性能的参数不满足第一条件时，对所述第一纹理图进行重分割操作，直至满足第二条件或所述表示分割性能的参数满足第一条件，得到所述第一纹理图的最终区域分割结果；其中，所述重分割操作包括：调整所述分割区域数，根据调整后的分割区域数对所述第一纹理图进行分割，得到新的区域分割结果；根据所述新的区域分割结果，确定所述表示分割性能的参数；更新模块，用于根据所述第一纹理图的最终区域分割结果，更新与所述第一纹理图的分割区域对应的第一可见图区域中的待更新像素，得到第二可见图；着色模块，用于对所 述第二可见图进行着色，得到所述目标视点的第二纹理图。
本申请实施例提供的渲染装置，包括：虚拟视点绘制模块，用于根据本申请实施例所述的虚拟视点绘制方法中的步骤，得到目标视点的第二纹理图；目标视图生成模块，用于根据所述目标视点的第二纹理图，生成所述目标视点的目标视图。
本申请实施例提供的解码装置，包括：解码模块，用于对输入的码流进行解码，得到输入视点的图集；虚拟视点绘制模块，用于对所述输入视点的图集执行如本申请实施例所述的虚拟视点绘制方法中的步骤，得到所述目标视点的第二纹理图；目标视图生成模块，用于根据所述目标视点的第二纹理图，生成所述目标视点的目标视图。
本申请实施例提供的电子设备，包括存储器和处理器，所述存储器存储有可在处理器上运行的计算机程序，所述处理器执行所述程序时实现本申请实施例提供的虚拟视点方法，或者所述处理器执行所述程序时实现本申请实施例所述的渲染方法，或者所述处理器执行所述程序时实现本申请实施例所述的解码方法。
本申请实施例提供的计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现本申请实施例所述的虚拟视点绘制方法，或者，该计算机程序被处理器执行时实现本申请实施例所述的渲染方法，或者该计算机程序被处理器执行时实现本申请实施例所述的解码方法。
在本申请实施例提供的虚拟视点绘制方法中，在根据分割区域数对第一纹理图进行区域分割之后，不是直接利用得到的区域分割结果对第一可见图进行质量提升处理，而是先判断该区域分割结果的表示分割性能的参数是否满足第一条件；如果不满足第一条件，则调整该分割区域数，基于此重新对第一纹理图进行区域分割，如此迭代直至得到的区域分割结果的表示分割性能的参数满足第一条件或者分割次数满足第二条件为止，利用第一纹理图的最终区域分割结果对第一可见图进行质量提升处理；如此，一方面能够较好地消除第一可见图中的噪点区域和过渡带区域，从而得到质量更好的第二可见图，进而得到满足质量要求的第二纹理图；另一方面，由于能够自适应调整分割区域数，从而获得较好的区域分割结果；无需用户直接给出一个合适的分割区域数目，从而降低了该方法的使用门槛，降低了对用户的相关知识水平的要求。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，这些附图示出了符合本申请的实施例，并与说明书一起用于说明本申请的技术方案。
图1为本申请实施例可能适用的一种系统架构示意图；
图2为本申请实施例虚拟视点绘制方法的实现流程示意图；
图3为本申请实施例虚拟视点绘制方法的实现流程示意图；
图4为本申请实施例虚拟视点绘制方法的实现流程示意图；
图5为本申请实施例分割准确的分割区域与分割不准确的分割区域的示意图；
图6为在纹理图上采用超像素分割对视点生成中的深度图进行优化的流程示意图；
图7为本申请实施例基于迭代的超像素数目自适应选取的方法的实现流程示意图；
图8为本申请实施例虚拟视点绘制装置的结构示意图；
图9为本申请实施例渲染装置的结构示意图；
图10为本申请实施例解码装置的结构示意图；
图11为本申请实施例的电子设备的硬件实体示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请的具体技术方案做进一步详细描述。以下实施例用于说明本申请，但 不用来限制本申请的范围。
除非另有定义，本文所使用的所有的技术和科学术语与属于本申请的技术领域的技术人员通常理解的含义相同。本文中所使用的术语只是为了描述本申请实施例的目的，不是旨在限制本申请。
在以下的描述中，涉及到“一些实施例”，其描述了所有可能实施例的子集，但是可以理解，“一些实施例”可以是所有可能实施例的相同子集或不同子集，并且可以在不冲突的情况下相互结合。
需要指出，本申请实施例所涉及的术语“第一\第二\第三”等是为了区别类似或不同的对象，不代表针对对象的特定排序，可以理解地，“第一\第二\第三”等在允许的情况下可以互换特定的顺序或先后次序，以使这里描述的本申请实施例能够以除了在这里图示或描述的以外的顺序实施。
本申请实施例描述的系统架构以及业务场景是为了更加清楚的说明本申请实施例的技术方案，并不构成对于本申请实施例提供的技术方案的限定，本申请实施例的技术方案适用于各种类型的系统架构。本领域普通技术人员可知，随着系统架构的演变和新业务场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
图1示出了本申请实施例可能适用的一种系统架构，即动态图像专家组(Moving Picture Experts Group，MPEG)在3自由度+(3 degrees of freedom+，3DoF+)沉浸式视频测试模型(Test Model of Immersive Video，TMIV)的解码端的系统架构10，如图1所示，该系统架构10包括：解码存取单元(Decoded access unit)11和渲染单元(Rendering unit)12；其中，解码存取单元11中包含解码后得到的各类元数据和图集(atlas)信息。之后这些信息将会被传输到渲染单元12中进行虚拟视点绘制。标注有可选(opt.)字样的子单元表示可选子单元，由于这些子单元在本申请实施例的技术方案中暂不涉及，因此在此不做描述。渲染单元12的图块剔除(Patch culling)子单元121根据目标视点参数(viewport parameters)对图集信息中的图块(patches)进行筛选，剔除与目标视图没有重叠的图块，从而降低虚拟视点绘制时的计算量。渲染单元12的占用图恢复(Occupancy reconstruction)子单元122根据解码存取单元11传输来的信息找出各个图块(patch)在视图中的位置，然后将筛选出来的图块(patches)贴入相应的位置完成剪切视图恢复(Pruned view reconstruction)，得到输入视点下的重构深度图。视点加权合成器(View Weighting Synthesizer，VWS)子单元分为两个模块：可见图(Visibility)生成模块1231和着色(Shading)模块1232。可见图生成模块1231用于基于输入视点下的重构深度图生成目标视点下的可见图，着色模块1232用于对可见图生成模块1231中生成的可见图进行着色，生成目标视点下的纹理图。
在VWS中，深度图以可见图的形式存在。可见图和深度图的含义相同，均表示场景距离相机位置的远近关系。与深度图不同的是，可见图中距离相机位置越近，像素值越小。而纹理图不同于深度图，纹理图是表征场景纹理的图像。
在此基础上，如图1所示，在纹理图上采用超像素分割对视点生成中的深度图进行优化的技术方案在VWS上实现后引入了三个新的模块：超像素分割(Superpixel Segmentation)模块1233、k均值聚类(K-means Clustering)模块1234和替换(Replacement)模块1235。超像素分割模块1233采用超像素分割算法对VWS生成的纹理图进行分割，并将分割得到的结果应用到生成的可见图上，得到可见图上的若干个超像素。k均值聚类模块1234利用k均值聚类算法对得到的可见图上的每个超像素分别进行聚类，如此可以将需要处理的噪点区域以及过渡带区域与不需处理的区域分开，最后由替换模块1235对这些需要处理的区域的像素值进行替换，如此之后，得到的深度图中的噪点区域和过渡带区域将会大大减少。并将替换后得到的可见图输入至着色模块1232，由于着色 模块1232生成的目标视点下的纹理图存在一定的空洞区域，所以需要填充(Inpainting)子单元124对空洞进行填充，得到填充后的纹理图。最后，视图空间处理(Viewing space handling)子单元125可使填充后的纹理图平滑淡出为黑色，最终得到目标视点的目标视图。
需要说明的是，在纹理图上采用超像素分割对视点生成中的深度图进行优化的技术方案可以在MPEG 3DoF+TMIV6中的VWS的基础上实现，也可以是其他类型的虚拟视点绘制工具上实现，该技术方案适用于各种类型的虚拟视点绘制工具。
本申请实施例提供的方法，可以应用于任何具有数据处理能力的电子设备，所述电子设备可以是电视机、投影仪、手机、个人计算机、平板电脑或虚拟现实(Virtual Reality，VR)头戴设备等任何具有视频编解码功能或者仅有解码功能的设备。所述方法所实现的功能可以通过所述电子设备中的处理器调用程序代码来实现，当然程序代码可以保存在计算机存储介质中。可见，所述电子设备至少包括处理器和存储介质。
图2为本申请实施例虚拟视点绘制方法的实现流程示意图，如图2所示，所述方法可以包括以下步骤201至步骤208：
步骤201，根据输入视点的图集，生成目标视点的第一可见图和第一纹理图。
在一些实施例中，电子设备可以对输入的码流进行解码，得到输入视点的深度图的图集和纹理图的图集；然后，对输入视点的深度图的图集进行剪切视图恢复，得到输入视点的重构深度图；根据重构深度图，生成目标视点的第一可见图；以及，对输入视点的纹理图的图集进行剪切视图恢复，得到输入视点的重构纹理图；根据重构纹理图，对所述目标视点的第一可见图进行着色，从而得到第一纹理图。
在本申请实施例中，所依据的输入视点的数目不做限定。电子设备可以根据一个或多个输入视点的图集，生成目标视点的第一可见图和第一纹理图。
步骤202，确定分割区域数。
电子设备可以直接获得预设的分割区域数，也可以根据第一纹理图计算得到该分割区域数。例如，可以通过如下实施例的步骤302和步骤303确定该分割区域数。
步骤203，根据所述分割区域数，对所述第一纹理图进行区域分割，得到区域分割结果。
在一些实施例中，电子设备可以采用超像素分割算法，按照该分割区域数(即超像素数目)，对第一纹理图进行超像素分割，得到的多个超像素(即分割区域的一种示例)，一个超像素即为一个分割区域。可以理解地，超像素是指具有相似纹理、颜色和亮度等特征的相邻像素构成的有一定视觉意义的不规则像素块。
在本申请实施例中，所使用的超像素分割算法可以是多种多样的，在本申请实施例中对此不做限制。例如，所述超像素分割算法可以是简单的线性迭代聚类(Simple Linear Iterative Cluster，SLIC)超像素分割算法、通过能量驱动采样的超像素提取(Superpixels Extracted via Energy-Driven Sampling，SEEDS)算法、轮廓松弛超像素(Contour-Relaxed Superpixels，CRS)算法、ETPS或熵率超像素分割(Entropy Rate Superpixels Segmentation，ERS)算法等。
步骤204，根据所述区域分割结果，确定表示分割性能的参数。
在一些实施例中，表示分割性能的参数包括分割准确度和/或分割区域的平均像素数。其中，对于分割准确度表征的含义不做限定。分割准确度可以是指分割准确的分割区域数在区域分割结果中的占比，相应地，在一些实施例中，第一条件为分割准确度大于阈值，或者，第一条件为分割区域的平均像素数小于或等于第四阈值。分割准确度还可以是指分割不准确的分割区域数在区域分割结果中的占比，相应地，在一些实施例中中，第一条件为分割准确度小于第三阈值，或者，分割区域的平均像素数小于或等于第 四阈值。
在一些实施例中，表示分割性能的参数通过平均像素数表示时，第一条件可以为平均像素数小于或等于第四阈值。
步骤205，确定所述表示分割性能的参数是否满足第一条件或分割次数是否满足第二条件；如果表示分割性能的参数不满足第一条件，且分割次数不满足第二条件，执行步骤206；如果表示分割性能的参数满足第一条件或分割次数满足第二条件，执行步骤207。
可以理解地，第二条件为预留的提前退出条件，比如分割了一定次数以后，若表示分割性能的参数还不满足第一条件，则不再进行分割；这样做的目的是为了控制处理复杂度。
步骤206，调整所述分割区域数，然后返回执行步骤203。
也就是说，如果表示分割性能的参数不满足第一条件，且分割次数不满足第二条件，则需要对所述第一纹理图进行重分割操作，直至分割次数满足第二条件或所述表示分割性能的参数满足所述第一条件，得到所述第一纹理图的最终区域分割结果；其中，所述重分割操作包括：调整所述分割区域数，根据调整后的分割区域数对所述第一纹理图进行分割，得到新的区域分割结果；根据所述新的区域分割结果，确定所述表示分割性能的参数。
在本申请实施例中，对于分割区域数的调整步长不做限定，可以是任意值。例如，调整前的分割区域数为2000，调整后的分割区域数为2500，即每次调整的步长为500。又如，调整前的分割区域数为2000，调整后的分割区域数为1000，即调整步长为1000。
步骤207，根据所述第一纹理图的最终区域分割结果，更新与所述第一纹理图的分割区域对应的第一可见图区域中的待更新像素，得到第二可见图。
在一些实施例中，将第一纹理图的最终区域分割结果作为第一可见图的区域分割结果，对第一可见图的分割区域中的像素进行分类，得到第一类像素和第二类像素；然后，根据这两类像素的像素数的关系以及这两类像素的像素值代表(例如均值或中值等)的关系，确定该分割区域中的待更新像素。
在一些实施例中，可以对所述第一可见图的分割区域中的待更新像素的像素值进行更新，得到第二可见图。
待更新像素的像素值的更新方式可以是多种多样的。例如，电子设备可以对第一可见图中的待更新像素的像素值进行滤波，以实现对该像素值的更新。再如，电子设备还可以利用像素替换值替换待更新像素的像素值，以实现对该像素值的更新。其中，每一分割区域对应一像素替换值，该像素替换值可以为对应分割区域中的非待更新像素的像素值的均值。在使用聚类算法的情况下，每一分割区域均对应一个非待更新像素类的聚类中心，因此可以将该中心的像素值作为像素替换值。
步骤208，对所述第二可见图进行着色，得到所述目标视点的第二纹理图。
在本申请实施例中，在根据分割区域数对第一纹理图进行区域分割之后，不是直接利用得到的区域分割结果对第一可见图进行质量提升处理，而是先判断该区域分割结果的表示分割性能的参数是否满足第一条件；如果不满足第一条件，则进行重分割操作，直至满足第二条件或表示分割性能的参数满足第一条件，基于第一纹理图的最终区域分割结果对第一可见图进行质量提升处理(即步骤207)；如此，一方面能够较好地消除第一可见图中的噪点区域和过渡带区域，从而得到质量更好的第二可见图，进而得到满足质量要求的第二纹理图；另一方面，由于能够自适应调整分割区域数，从而获得较好的区域分割结果；因此，无需用户直接给出一个合适的分割区域数目，从而降低了该方法的使用门槛，降低了对用户的相关知识水平的要求。
本申请实施例再提供一种虚拟视点绘制方法，图3为本申请实施例虚拟视点绘制方法的实现流程示意图，如图3所示，所述方法可以包括以下步骤301至步骤313：
步骤301，根据输入视点的图集，生成目标视点的第一可见图和第一纹理图；
步骤302，确定所述第一纹理图的图像分辨率；
步骤303，根据所述图像分辨率和预设的分割区域的平均像素数，确定所述分割区域数。
在一些实施例中，电子设备可以根据如下公式(1)或者公式(2)计算得到该分割区域数init_numSuperpixel：
init_numSuperpixel＝round(width*height/M)(1)；
init_numSuperpixel＝round(width*height/M/100)*100(2)；
其中，width表示图像的宽，height表示图像的高，round表示四舍五入，M表示预设的分割区域的平均像素数(例如该值为2000)，计算得到的init_numSuperpixel的精度为百。在一些实施例中，M通常设为最大值，这样使得初始设置的分割区域数为最小值。
可以理解地，电子设备基于第一纹理图的图像分辨率能够自动确定分割区域数的方法，从而无需预先设置分割区域数，进而降低了本申请实施例提供的虚拟绘制方法的使用门槛。
步骤304，根据所述分割区域数，对所述第一纹理图进行区域分割，得到区域分割结果；
步骤305，将所述区域分割结果作为所述第一可见图的区域分割结果，对所述第一可见图的所述分割区域中的像素进行分类，得到第一类像素和第二类像素。
可以理解地，第一可见图与第一纹理图中的相同位置表达的场景内容是一致的。因此，电子设备可以直接将第一纹理图的区域分割结果直接应用于第一可见图上，第一纹理图的区域分割结果即为第一可见图的区域分割结果。
电子设备可以对第一可见图的每一分割区域的像素均做分类，例如将每一分割区域的像素均分为两类。分类算法可以是多种多样的。例如分类算法为K均值聚类、决策树、贝叶斯、人工神经网络、支持向量机或基于关联规则的分类等。
步骤306，根据所述第一类像素与所述第二类像素的关系，确定所述分割区域是否为分割不准确的区域。
在一些实施例中，电子设备可以根据如下实施例的步骤405至步骤407实现步骤306。
步骤307，根据所述第一可见图中分割不准确的分割区域数，确定所述区域分割结果的分割准确度。
在一些实施例中，分割准确度可以是分割不准确的分割区域数在区域分割结果中的占比。例如该占比为分割不准确的分割区域数除以分割准确的分割区域数，或者该占比为分割不准确的分割区域数除以步骤303得到的分割区域数，或者该占比为分割不准确的分割区域数除以区域分割结果中的分割区域总数。
在另一些实施例中，分割准确度还可以是分割准确的分割区域数在区域分割结果中的占比。例如，该占比为分割准确的分割区域数除以步骤303得到的分割区域数，或者除以区域分割结果中的分割区域总数。
步骤308，确定所述分割准确度是否满足第一条件；如果是，执行步骤311；否则，执行步骤309；
步骤309，确定分割次数是否满足第二条件；如果是，执行步骤311；否则，执行步骤310；
步骤310，调整所述分割区域数，然后返回执行步骤304；
也就是说，如果所述分割准确度不满足第一条件，且分割次数不满足第二条件，则调整步骤分割区域数；然后，根据调整后的分割区域数，重新对所述第一纹理图进行区域分割，直至得到的区域分割结果的分割准确度满足第一条件，或者分割次数满足第二条件。然后，进入步骤311；
步骤311，将第一纹理图的最终区域分割结果作为所述第一可见图的区域分割结果，识别所述第一可见图的所述分割区域中的待更新像素；
步骤312，根据所述分割区域中的非待更新像素的像素值，更新所述分割区域中的待更新像素的像素值，从而得到所述第二可见图。
所谓非待更新像素，是指所述分割区域中除所述待更新像素以外的其他像素。比如，将分割区域中的像素分类为第一类像素和第二类像素，其中，第一类像素为待更新像素，那么第二类像素即为非待更新像素。
待更新像素的像素值的更新方式可以是多种多样的。例如，电子设备可以对待更新像素的像素值进行滤波，还可以利用该分割区域的像素替换值替换该分割区域中的待更新像素的像素值。在一些实施例中，可以将该分割区域中的非待更新像素的像素值的均值或者中值等具有代表性的像素值，替代该分割区域中的待更新像素的像素值。
步骤313，对所述第二可见图进行着色，得到所述目标视点的第二纹理图。
本申请实施例再提供一种虚拟视点绘制方法，图4为本申请实施例虚拟视点绘制方法的实现流程示意图，如图4所示，所述方法可以包括以下步骤401至步骤415：
步骤401，根据输入视点的图集，生成目标视点的第一可见图和第一纹理图；
步骤402，确定分割区域数；
步骤403，根据所述分割区域数，对所述第一纹理图进行区域分割，得到区域分割结果；
步骤404，将所述区域分割结果作为所述第一可见图的区域分割结果，对所述第一可见图的所述分割区域中的像素进行分类，得到第一类像素和第二类像素；
步骤405，确定所述第一类像素的像素值代表与所述第二类像素的像素值代表的差值的绝对值。
所谓像素值代表，是指在第一类像素中具有代表性的像素值，该像素值可以是该分割区域中第一类像素的像素值的均值或中值等。
步骤406，确定所述第一类像素的像素数除以所述第二类像素的像素数的第一结果；
步骤407，如果满足以下条件：所述差值的绝对值大于第一阈值、所述第一结果小于第二阈值、以及所述第一结果大于所述第二阈值的倒数，确定所述分割区域为分割不准确的区域。
在本申请实施例中，对于第一阈值和第二阈值的大小不做限定。可以根据实际应用的性能指标，设置这些阈值。在一些实施例中，所述第一阈值的取值范围为[25,33]，所述第二阈值的取值范围为[5,10]。例如，所述第一阈值为30，所述第二阈值为5。
可以理解地，理想的区域分割结果是沿着前景的边缘来分割的，也就是说，每一分割区域均能够较好地把前景和背景分割开来。所谓的不理想的区域分割是指一个分割区域中既有前景的内容又有背景的内容，该分割区域横跨了前景和背景，而在深度图上，前景的颜色接近白色，背景的颜色接近黑色。可见，前景的像素值和背景的像素值的差异较大，这样在识别待更新像素时，就会把前景或背景误判为待更新像素，从而导致最终的优化结果不佳。总之，如果两类像素的像素值的平均水平相差较大，且二者的像素数相差较小，则认为是不准确的分割，该分割区域较有可能横跨了前景和背景区域。
在基于区域分割结果对第一可见图进行优化时，分割准确的分割区域中一类像素的 占比小，与另一类像素的像素数的比值小于一阈值，且这两类像素的像素值的差异大于一阈值。举例来说，如图5所示，分割区域51、52、53和54是分割准确的分割区域，即分割区域中一类像素与另一类像素的像素数差异较大，且像素值的差异也较大，则可能是区域51至区域54所示的四种情况，其中区域51为较少的一类像素为前景像素(即待更新像素)，较多的一类像素为背景像素；区域52为较少的一类像素为背景像素(即待更新像素)，较多的一类像素为前景像素；区域53为较少的一类像素为噪点像素(即待更新像素)，较多的一类像素为背景像素；区域54为较少的一类像素为噪点像素(即待更新像素)，较多的一类像素为前景像素；基于此，在对诸如区域51和区域52中的待更新像素进行更新后，使得最终得到的第二纹理图的边缘会更锐化、更陡峭；在对诸如区域53和区域54中的待更新像素进行更新后，使得最终得到的第二纹理图中的噪点更少。
在某些情况下，区域55和56可能是分割不准确的分割区域，如图5所示，无论是区域55还是区域56，其中的两类像素的像素值虽然差异大，即所述差值的绝对值大于第一阈值，但是两类的像素数量差不多，即所述第一结果小于第二阈值且第一结果大于第二阈值的倒数，说明这类区域同时包含前景和背景，如果对这类区域中的某一类像素进行更新，就会出现错误，既不会消除其中的噪点，也不会使得边缘更锐化；因此，这类区域不利于获得边缘更锐化、噪点更少的第二纹理图。
步骤408，将所述第一可见图中分割不准确的分割区域数在所述区域分割结果中的占比，确定为所述分割准确度。
该占比可以是分割不准确的分割区域数除以分割准确的分割区域数，或者除以步骤402获得的分割区域数，或者除以区域分割结果中的分割区域总数。
步骤409，将所述区域分割结果作为所述第一可见图的区域分割结果，确定所述分割区域的平均像素数；
步骤410，确定所述分割准确度是否大于第三阈值，且所述平均像素数是否大于第四阈值；如果是，执行步骤411；否则，执行步骤413。
也就是说，如果分割准确度大于第三阈值，且平均像素数大于第四阈值，说明区域分割结果的表示分割性能的参数不满足第一条件；否则，分割准确度小于或等于第三阈值，或者，平均像素数小于或等于第四阈值，说明表示分割性能的参数是满足第一条件的，此时结束迭代，进入步骤413。
如果平均像素数达到一定精度(例如500)，即使再调整分割区域数重新对第一纹理图进行分割，最终得到的第二纹理图，其质量改善不明显。也就是说，当平均像素数达到一定精度，得到的第二纹理图趋于收敛。因此，在判断是否将当前的区域分割结果作为最终区域分割结果时，既依赖于分割准确度，也依赖于平均像素数，如此能够在确保第二纹理图的图像质量的前提下，尽早结束迭代，从而节约计算过程，提高虚拟视点绘制的实时性。
当然，在一些实施例中，也可以不依赖与平均像素数这一条件，即，如果分割准确度大于第三阈值，确定分割结果的表示分割性能的参数不满足第一条件；否则，确定区域分割结果的表示分割性能的参数满足第一条件。
步骤411，确定分割次数是否满足第二条件；如果是，进入步骤413；否则，执行步骤412；
步骤412，调整所述分割区域数，然后返回执行步骤403；
也就是说，如果表示分割性能的参数不满足第一条件，根据调整后的分割区域数，重新对所述第一纹理图进行区域分割，并再次执行步骤404至步骤411，直至分割次数满足第二条件或表示分割性能的参数满足第一条件，然后，进入步骤413；
步骤413，将第一纹理图的最终区域分割结果作为所述第一可见图的区域分割结果，根据所述第一类像素的像素数与所述第二类像素的像素数的关系、以及所述第一类像素的像素值代表与所述第二类像素的像素值代表的关系，确定所述分割区域中的待更新像素。
可以理解地，在执行步骤404时，已经对第一可见图的每一分割区域进行了分类，因此在步骤413中，直接将步骤404的计算结果拿来用即可，而无需再重新进行分类运算。
在一些实施例中，在所述第一类像素的像素值代表减去所述第二类像素的像素值代表的第二结果大于或等于第五阈值，且所述第一类像素的像素数除以所述第二类像素的像素数的第一结果大于或等于第六阈值的情况下，确定所述第二类像素为所述分割区域的待更新像素；在所述第二类像素的像素值代表减去所述第一类像素的像素值代表的第三结果大于或等于所述第五阈值，且所述第二类像素的像素数除以所述第一类像素的像素数的第四结果大于或等于所述第六阈值的情况下，确定所述第一类像素为所述分割区域的待更新像素。
需要说明的是，第五阈值可以与前文提到的第一阈值相同，也可以不同，第五阈值可以大于第一阈值，也可以小于第一阈值。第六阈值可以设置为与前文提到的第二阈值相等，也可以不相等，第六阈值可以大于第二阈值，也可以小于第二阈值。例如，第一阈值和第五阈值的取值范围均为[25,33]，第二阈值和第六阈值的取值范围均为[5,10]。
可以理解地，通过上述方法确定分割区域中的待更新像素，能够将诸如图5所示的区域51、52、53和54中的待更新像素进行更新，从而，一方面能够使得最终得到的第二纹理图中的边缘更加锐化和陡峭；另一方面，能够减少第二纹理图中的噪点区域。总之，能够使得第二纹理图的图像质量更好。
步骤414，对所述第一可见图的所述分割区域中的待更新像素的像素值进行更新，得到第二可见图；
步骤415，对所述第二可见图进行着色，得到所述目标视点的第二纹理图。
在一些实施例中，电子设备可以将所述第一纹理图的最终区域分割结果作为相邻N帧的第一可见图的区域分割结果；其中，N大于0。也就是说，每连续N+1帧的第一可见图的区域分割结果是相同的。可以理解地，相邻帧图像的内容具有较大的相似性，因此连续N+1帧的第一可见图可以共用第一纹理图的最终区域分割结果，这样，无需对每帧第一纹理图均计算其最终区域分割结果，从而在确保每一第二纹理图的图像质量的前提下，节约了计算量。
在一些实施例中，将第一纹理图的最终区域分割结果作为下一帧第一可见图至下N帧第一可见图的区域分割结果；N大于1；重新确定下N+1帧第一纹理图的最终区域分割结果，并将此作为后续相邻N帧的第一可见图的区域分割结果。
本申请实施例提供一种渲染方法，该方法不仅可以应用于电子设备，还可以应用于渲染设备，所述方法可以包括：首先，执行本申请实施例所述的虚拟视点绘制方法中的步骤，得到目标视点的第二纹理图；然后，根据所述目标视点的第二纹理图，生成所述目标视点的目标视图。
以上对渲染方法实施例的描述，与上述虚拟视点绘制方法实施例的描述是类似的，具有同上述方法实施例相似的有益效果。对于渲染方法实施例中未披露的技术细节，请参照上述方法实施例的描述而理解。
本申请实施例提供一种解码方法，所述方法包括：对输入的码流进行解码，得到输入视点的图集；对所述输入视点的图集进行执行上述虚拟视点绘制方法中的步骤，得到所述目标视点的第二纹理图；根据所述目标视点的第二纹理图，生成所述目标视点的目 标视图。
在解码方法中，由于能够使得最终得到的目标视图中的噪点和/或过渡区明显减少，因此，在确保目标视图的图像质量的基础上，使得编码端可以使用较大的量化参数对深度图进行压缩编码，从而降低深度图的编码开销，进而提高整体的编码效率。
以上对解码方法实施例的描述，与上述虚拟视点绘制方法实施例的描述是类似的，具有同上述虚拟视点绘制方法实施例相似的有益效果。对于所述解码方法实施例中未披露的技术细节，请参照上述虚拟视点绘制方法实施例的描述而理解。
由于不准确的深度估计和压缩失真等因素，生成的目标视点下的深度图会存在一些噪点区域和过渡带区域，因此提供了在纹理图上采用超像素分割对视点生成中的深度图进行优化的技术方案，其流程如图6所示，首先对纹理图601进行超像素分割(Superpixel Segmentation)，然后将从纹理图601上得到的超像素区域分割结果应用到深度图602上，得到深度图602上的若干个超像素。对每个超像素通过聚类(Clustering)的方式提取出其中的噪点区域和过渡带区域，然后采用合适的值对这些区域进行替换(Replacement)。如此之后，得到的深度图603中的噪点区域和过渡带区域将会大大减少。
上述提出的在纹理图上采用超像素分割对视点生成中深度图进行优化技术的实际效果与超像素数目(numSuperpixel)密切相关。参数numSuperpixel表示由使用者指定的对纹理图进行超像素分割时采用的超像素数目，它会影响超像素分割的效果，进而影响深度图优化的效果。当numSuperpixel过小时，不能较好地沿着物体边缘分割，当numSuperpixel过大时，又会增加处理的复杂度。上述深度图优化技术并未给出numSuperpixel的设置方法，需要使用者自己确定其大小。这就对使用者的相关知识水平提出了要求，提高了该技术的使用门槛。
基于此，下面将说明本申请实施例在一个实际的应用场景中的示例性应用。
在本申请实施例中，提供一种基于迭代的超像素数目自适应选取的方法。该方法的流程图如图7所示，包括以下步骤701至步骤707：
步骤701，计算初始超像素数目(init_numSuperpixel)。
在一些实施例中，如公式(3)所示，初始超像素数目可以根据图像的分辨率获得：
init_numSuperpixel＝round(width*height/2000/100)*100(3)；
其中，width表示图像的宽，height表示图像的高，round表示四舍五入，计算得到的init_numSuperpixel的精度为百。
步骤702，令temp_numSuperpixel＝init_numSuperpixel；
步骤703，以temp_numSuperpixel对第一纹理图进行超像素分割；
步骤704，将步骤703的超像素区域分割结果应用在第一纹理图的第一可见图上，对第一可见图上的超像素做k均值聚类，k＝2；
步骤705，计算不准确分割的超像素总数sum。
在一些实施例中，不准确分割的超像素的判定规则如下：经过k均值聚类的步骤后，每个超像素被分为两部分。假定两部分的聚类中心和包含的像素点数分别为center 0、center 1、sum 0、sum 1，当满足下面公式(4)所示的条件时，该超像素即可被判定为不准确分割的超像素：
(|center 0-center 1|>threshold cen)且(sum 0/sum 1<threshold sum)且(sum 0/sum 1>1/threshold sum)  (4)；
其中，threshold cen和threshold sum是两个阈值，根据经验，一般采用threshold cen＝30，threshold sum＝5。
步骤706，计算分割准确度accuracy＝sum/temp_numSuperpixel；
步骤707，根据分割准确度accuracy，判断是否满足迭代结束条件；如果满足，得 到超像素数目fina_numSuperpixel＝temp_numSuperpixel；否则，令temp_numSuperpixel＝temp_numSuperpixel+step，然后返回执行步骤703；
在一些实施例中，迭代结束条件有两个：一是accuracy满足需要的精度，即accuracy<＝threshold accu，threshold accu根据精度要求可稍作调整，建议设置为0.025；二是当平均每个超像素包含的像素数小于500，即width*height/temp_numSuperpixel<500。满足两个中的一个即可结束迭代。若都不满足，则将超像素数目增加step后，返回执行步骤703。参数step的大小影响该方法的收敛速度，推荐设置为500。
考虑到相近帧图像内容的相似性和帧间距较大时图像内容的差异性，该技术可设置一个固定周期period，每period帧执行一次上述超像素数目自适应算法，用于超像素数目的更新。
上述方案在MPEG-I CTC测试序列上进行了测试，实验结果如表1所示。
表1实验结果
Figure PCTCN2021093952-appb-000001
根据实验结果可以看出，本申请实施例所提供的方案可以自动地追踪到较为合适的超像素数目，实现了超像素数目的自适应，降低了在纹理图上采用超像素分割对视点生成中深度图优化技术的使用门槛。
基于前述的实施例，本申请实施例提供的虚拟视点绘制装置，包括所包括的各模块、以及各模块所包括的各单元，可以通过处理器来实现；当然也可通过具体的逻辑电路实现；在实施的过程中，处理器可以为中央处理器(CPU)、微处理器(MPU)、数字信号处理器(DSP)或现场可编程门阵列(FPGA)等。
图8为本申请实施例虚拟视点绘制装置的结构示意图，如图8所示，虚拟视点绘制装置80包括：
生成模块801，用于根据输入视点的图集，生成目标视点的第一可见图和第一纹理图；
确定模块802，用于确定分割区域数；
分割模块803，用于根据所述分割区域数，对所述第一纹理图进行区域分割，得到区域分割结果；根据区域分割结果，确定表示分割性能的参数；当所述表示分割性能的参数不满足第一条件时，对所述第一纹理图进行重分割操作，直至满足第二条件或所述分割准确度满足第一条件，得到所述第一纹理图的最终区域分割结果；其中，所述重分割操作包括：调整所述分割区域数，根据调整后的分割区域数对所述第一纹理图进行分割，得到新的区域分割结果；根据所述新的区域分割结果，确定所述表示分割性能的参数；
更新模块804，用于根据所述第一纹理图的最终区域分割结果，更新与所述第一纹理图的分割区域对应的第一可见图区域中的待更新像素，得到第二可见图；
着色模块805，用于对所述第二可见图进行着色，得到所述目标视点的第二纹理图。
在一些实施例中，确定模块802，用于：确定所述第一纹理图的图像分辨率；根据所述图像分辨率和预设的分割区域的平均像素数，确定所述分割区域数。
在一些实施例中，所述表示分割性能的参数包括所述区域分割结果的分割准确度； 分割模块803，用于：将所述区域分割结果作为所述第一可见图的区域分割结果，对所述第一可见图的分割区域中的像素进行分类，得到第一类像素和第二类像素；根据所述第一类像素与所述第二类像素的关系，确定所述分割区域是否为分割不准确的区域；根据所述第一可见图中分割不准确的分割区域数，确定所述区域分割结果的分割准确度。
进一步地，在一些实施例中，分割模块803，用于：确定所述第一类像素的像素值代表与所述第二类像素的像素值代表的差值的绝对值；确定所述第一类像素的像素数除以所述第二类像素的像素数的第一结果；如果满足以下条件：所述差值的绝对值大于第一阈值、所述第一结果小于第二阈值、以及所述第一结果大于所述第二阈值的倒数，确定所述分割区域为分割不准确的区域。
在一些实施例中，所述第一阈值的取值范围为[25,33]，所述第二阈值的取值范围为[5,10]。
在一些实施例中，所述表示分割性能的参数包括分割区域的平均像素数；分割模块803，用于：将所述分割不准确的分割区域数在所述区域分割结果中的占比，确定为所述分割准确度；将所述区域分割结果作为所述第一可见图的区域分割结果，确定所述分割区域的平均像素数；如果所述分割准确度小于或等于第三阈值，或者，所述平均像素数小于或等于第四阈值，确定所述表示分割性能的参数满足所述第一条件；如果所述分割准确度大于所述第三阈值，且所述平均像素数大于所述第四阈值，确定所述表示分割性能的参数不满足所述第一条件。
在一些实施例中，分割模块803，还用于：将所述第一纹理图的最终区域分割结果作为相邻N帧的第一可见图的区域分割结果；其中，N大于0。
在一些实施例中，更新模块804，用于：获得所述第一类像素的像素数和所述第一类像素的像素值代表；获得所述第二类像素的像素数和所述第二类像素的像素值代表；根据所述第一类像素的像素数与所述第二类像素的像素数的关系、以及所述第一类像素的像素值代表与所述第二类像素的像素值代表的关系，确定所述分割区域中的待更新像素。
进一步地，在一些实施例中，更新模块804，用于：在所述第一类像素的像素值代表减去所述第二类像素的像素值代表的第二结果大于或等于第五阈值，且所述第一类像素的像素数除以所述第二类像素的像素数的第一结果大于或等于第六阈值的情况下，确定所述第二类像素为所述分割区域的待更新像素；在所述第二类像素的像素值代表减去所述第一类像素的像素值代表的第三结果大于或等于所述第五阈值，且所述第二类像素的像素数除以所述第一类像素的像素数的第四结果大于或等于所述第六阈值的情况下，确定所述第一类像素为所述分割区域的待更新像素。
在一些实施例中，更新模块804，用于：根据第一可见图的所述分割区域中的非待更新像素的像素值，更新所述分割区域中的待更新像素的像素值，从而得到所述第二可见图。
本申请实施例提供一种渲染装置，图9为本申请实施例渲染装置的结构示意图，如图9所示，渲染装置90包括：
虚拟视点绘制模块901，用于根据本申请实施例所述的虚拟视点绘制方法中的步骤，得到目标视点的第二纹理图；
目标视图生成模块902，用于根据所述目标视点的第二纹理图，生成所述目标视点的目标视图。
本申请实施例提供一种解码装置，图10为本申请实施例解码装置的结构示意图，如图10所示，解码装置100包括：
解码模块1001，用于对输入的码流进行解码，得到输入视点的图集；
虚拟视点绘制模块1002，用于对所述输入视点的图集执行如本申请实施例所述虚拟视点绘制方法中的步骤，得到所述目标视点的第二纹理图；
目标视图生成模块1003，用于根据所述目标视点的第二纹理图，生成所述目标视点的目标视图。
以上装置实施例的描述，与上述方法实施例的描述是类似的，具有同方法实施例相似的有益效果。对于本申请装置实施例中未披露的技术细节，请参照本申请方法实施例的描述而理解。
需要说明的是，本申请实施例中，如果以软件功能模块的形式实现上述的方法，并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得电子设备执行本申请各个实施例所述方法的全部或部分。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read Only Memory，ROM)、磁碟或者光盘等各种可以存储程序代码的介质。这样，本申请实施例不限制于任何特定的硬件和软件结合。
对应地，本申请实施例提供一种电子设备，图11为本申请实施例的电子设备的硬件实体示意图，如图11所示，所述电子设备110包括存储器1101和处理器1102，所述存储器1101存储有可在处理器1102上运行的计算机程序，所述处理器1102执行所述程序时实现上述实施例中提供的方法中的步骤。
需要说明的是，存储器1101配置为存储由处理器1102可执行的指令和应用，还可以缓存在处理器1102以及电子设备110中各模块待处理或已经处理的数据(例如，图像数据、音频数据、语音通信数据和视频通信数据)，可以通过闪存(FLASH)或随机访问存储器(Random Access Memory，RAM)实现。
对应地，本申请实施例提供一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述实施例中提供的方法中的步骤。
这里需要指出的是：以上存储介质和设备实施例的描述，与上述方法实施例的描述是类似的，具有同方法实施例相似的有益效果。对于本申请存储介质和设备实施例中未披露的技术细节，请参照本申请方法实施例的描述而理解。
应理解，说明书通篇中提到的“一个实施例”或“一实施例”或“一些实施例”或“另一些实施例”意味着与实施例有关的特定特征、结构或特性包括在本申请的至少一个实施例中。因此，在整个说明书各处出现的“在一个实施例中”或“在一实施例中”或“在一些实施例中”或“在另一些实施例中”未必一定指相同的实施例。此外，这些特定的特征、结构或特性可以任意适合的方式结合在一个或多个实施例中。应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者设备中还存在另外的相同要素。
在本申请所提供的几个实施例中，应该理解到，所揭露的设备和方法，可以通过其它的方式实现。以上所描述的触摸屏系统的实施例仅仅是示意性的，例如，所述模块的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，如：多个模块或组件可以结合，或可以集成到另一个系统，或一些特征可以忽略，或不执行。另外，所 显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口，设备或模块的间接耦合或通信连接，可以是电性的、机械的或其它形式的。
上述作为分离部件说明的模块可以是、或也可以不是物理上分开的，作为模块显示的部件可以是、或也可以不是物理模块；既可以位于一个地方，也可以分布到多个网络单元上；可以根据实际的需要选择其中的部分或全部模块来实现本实施例方案的目的。
另外，在本申请各实施例中的各功能模块可以全部集成在一个处理单元中，也可以是各模块分别单独作为一个单元，也可以两个或两个以上模块集成在一个单元中；上述集成的模块既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：移动存储设备、只读存储器(Read Only Memory，ROM)、磁碟或者光盘等各种可以存储程序代码的介质。
或者，本申请上述集成的单元如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得电子设备执行本申请各个实施例所述方法的全部或部分。而前述的存储介质包括：移动存储设备、ROM、磁碟或者光盘等各种可以存储程序代码的介质。
本申请所提供的几个方法实施例中所揭露的方法，在不冲突的情况下可以任意组合，得到新的方法实施例。
本申请所提供的几个产品实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的产品实施例。
本申请所提供的几个方法或设备实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的方法实施例或设备实施例。
以上所述，仅为本申请的实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (17)

  1. 一种虚拟视点绘制方法，所述方法包括：
    根据输入视点的图集，生成目标视点的第一可见图和第一纹理图；
    确定分割区域数；
    根据所述分割区域数，对所述第一纹理图进行区域分割，得到区域分割结果；
    根据所述区域分割结果，确定表示分割性能的参数；
    当所述表示分割性能的参数不满足第一条件时，对所述第一纹理图进行重分割操作，直至满足第二条件或所述表示分割性能的参数满足所述第一条件，得到所述第一纹理图的最终区域分割结果；其中，所述重分割操作包括：调整所述分割区域数，根据调整后的分割区域数对所述第一纹理图进行分割，得到新的区域分割结果；根据所述新的区域分割结果，确定所述表示分割性能的参数；
    根据所述第一纹理图的最终区域分割结果，更新与所述第一纹理图的分割区域对应的第一可见图区域中的待更新像素，得到第二可见图；
    对所述第二可见图进行着色，得到所述目标视点的第二纹理图。
  2. 根据权利要求1所述的方法，其中，所述确定分割区域数，包括：
    确定所述第一纹理图的图像分辨率；
    根据所述图像分辨率和预设的分割区域的平均像素数，确定所述分割区域数。
  3. 根据权利要求1所述的方法，其中，所述表示分割性能的参数包括所述区域分割结果的分割准确度；
    根据所述区域分割结果，确定所述分割准确度，包括：
    将所述区域分割结果作为所述第一可见图的区域分割结果，对所述第一可见图的分割区域中的像素进行分类，得到第一类像素和第二类像素；
    根据所述第一类像素与所述第二类像素的关系，确定所述分割区域是否为分割不准确的区域；
    根据所述第一可见图中分割不准确的分割区域数，确定所述分割准确度。
  4. 根据权利要求3所述的方法，其中，所述根据所述第一类像素与所述第二类像素的关系，确定所述分割区域是否为分割不准确的区域，包括：
    确定所述第一类像素的像素值代表与所述第二类像素的像素值代表的差值的绝对值；
    确定所述第一类像素的像素数除以所述第二类像素的像素数的第一结果；
    如果满足以下条件：所述差值的绝对值大于第一阈值、所述第一结果小于第二阈值、以及所述第一结果大于所述第二阈值的倒数，确定所述分割区域为分割不准确的区域。
  5. 根据权利要求4所述的方法，其中，所述第一阈值的取值范围为[25,33]，所述第二阈值的取值范围为[5,10]。
  6. 根据权利要求3所述的方法，其中，所述根据所述第一可见图中分割不准确的分割区域数，确定所述区域分割结果的分割准确度，包括：将所述分割不准确的分割区域数在所述区域分割结果中的占比，确定为所述分割准确度；
    所述表示分割性能的参数包括分割区域的平均像素数；所述方法还包括：
    将所述区域分割结果作为所述第一可见图的区域分割结果，确定所述平均像素数；
    如果所述分割准确度小于或等于第三阈值，或者，所述平均像素数小于或等于第四阈值，确定所述表示分割性能的参数满足所述第一条件；
    如果所述分割准确度大于所述第三阈值，且所述平均像素数大于所述第四阈值，确定所述表示分割性能的参数不满足所述第一条件。
  7. 根据权利要求1至6任一所述的方法，其中，所述方法还包括：
    将所述第一纹理图的最终区域分割结果作为相邻N帧的第一可见图的区域分割结果；其中，N大于0。
  8. 根据权利要求3所述的方法，其中，所述方法还包括：
    获得所述第一类像素的像素数和所述第一类像素的像素值代表；
    获得所述第二类像素的像素数和所述第二类像素的像素值代表；
    根据所述第一类像素的像素数与所述第二类像素的像素数的关系、以及所述第一类像素的像素值代表与所述第二类像素的像素值代表的关系，确定所述分割区域中的待更新像素。
  9. 根据权利要求8所述的方法，其中，所述根据所述第一类像素的像素数与所述第二类像素的像素数的关系、以及所述第一类像素的像素值代表与所述第二类像素的像素值代表的关系，确定所述分割区域中的待更新像素，包括：
    在所述第一类像素的像素值代表减去所述第二类像素的像素值代表的第二结果大于或等于第五阈值，且所述第一类像素的像素数除以所述第二类像素的像素数的第一结果大于或等于第六阈值的情况下，确定所述第二类像素为所述分割区域的待更新像素；
    在所述第二类像素的像素值代表减去所述第一类像素的像素值代表的第三结果大于或等于所述第五阈值，且所述第二类像素的像素数除以所述第一类像素的像素数的第四结果大于或等于所述第六阈值的情况下，确定所述第一类像素为所述分割区域的待更新像素。
  10. 根据权利要求1所述的方法，其中，所述更新与所述第一纹理图的分割区域对应的第一可见图区域中的待更新像素，得到第二可见图，包括：
    根据所述第一可见图区域中的非待更新像素的像素值，更新所述第一可见图区域中的待更新像素的像素值，从而得到所述第二可见图。
  11. 一种渲染方法，所述方法包括：
    根据权利要求1至10任一项所述的虚拟视点绘制方法，得到目标视点的第二纹理图；
    根据所述目标视点的第二纹理图，生成所述目标视点的目标视图。
  12. 一种解码方法，所述方法包括：
    对输入的码流进行解码，得到输入视点的图集；
    对所述输入视点的图集执行如权利要求1至10任一项所述方法中的步骤，得到所述目标视点的第二纹理图；
    根据所述目标视点的第二纹理图，生成所述目标视点的目标视图。
  13. 一种虚拟视点绘制装置，包括：
    生成模块，用于根据输入视点的图集，生成目标视点的第一可见图和第一纹理图；
    确定模块，用于确定分割区域数；
    分割模块，用于根据所述分割区域数，对所述第一纹理图进行区域分割，得到区域分割结果；根据区域分割结果，确定表示分割性能的参数；当所述表示分割性能的参数不满足第一条件时，对所述第一纹理图进行重分割操作，直至满足第二条件或所述表示分割性能的参数满足第一条件，得到所述第一纹理图的最终区域分割结果；其中，所述重分割操作包括：调整所述分割区域数，根据调整后的分割区域数对所述第一纹理图进行分割，得到新的区域分割结果；根据所述新的区域分割结果，确定所述表示分割性能的参数；
    更新模块，用于根据所述第一纹理图的最终区域分割结果，更新与所述第一纹理图的分割区域对应的第一可见图区域中的待更新像素，得到第二可见图；
    着色模块，用于对所述第二可见图进行着色，得到所述目标视点的第二纹理图。
  14. 一种渲染装置，包括：
    虚拟视点绘制模块，用于根据权利要求1至10任一项所述的虚拟视点绘制方法中的步骤，得到目标视点的第二纹理图；
    目标视图生成模块，用于根据所述目标视点的第二纹理图，生成所述目标视点的目标视图。
  15. 一种解码装置，包括：
    解码模块，用于对输入的码流进行解码，得到输入视点的图集；
    虚拟视点绘制模块，用于对所述输入视点的图集执行如权利要求1至10任一项所述方法中的步骤，得到所述目标视点的第二纹理图；
    目标视图生成模块，用于根据所述目标视点的第二纹理图，生成所述目标视点的目标视图。
  16. 一种电子设备，包括存储器和处理器，所述存储器存储有可在处理器上运行的计算机程序，所述处理器执行所述程序时实现权利要求1至10任一项所述的方法，或者所述处理器执行所述程序时实现权利要求11所述的方法，或者所述处理器执行所述程序时实现权利要求12所述的方法。
  17. 一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现权利要求1至10任一项所述的方法，或者，该计算机程序被处理器执行时实现权利要求11所述的方法，或者该计算机程序被处理器执行时实现权利要求12所述的方法。
PCT/CN2021/093952 2021-05-14 2021-05-14 虚拟视点绘制、渲染、解码方法及装置、设备、存储介质 WO2022236837A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/CN2021/093952 WO2022236837A1 (zh) 2021-05-14 2021-05-14 虚拟视点绘制、渲染、解码方法及装置、设备、存储介质
CN202180097722.1A CN117242768A (zh) 2021-05-14 2021-05-14 虚拟视点绘制、渲染、解码方法及装置、设备、存储介质
US18/508,689 US20240087185A1 (en) 2021-05-14 2023-11-14 Virtual view drawing method, rendering method, and decoding method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/093952 WO2022236837A1 (zh) 2021-05-14 2021-05-14 虚拟视点绘制、渲染、解码方法及装置、设备、存储介质

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/508,689 Continuation US20240087185A1 (en) 2021-05-14 2023-11-14 Virtual view drawing method, rendering method, and decoding method

Publications (1)

Publication Number Publication Date
WO2022236837A1 true WO2022236837A1 (zh) 2022-11-17

Family

ID=84027964

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/093952 WO2022236837A1 (zh) 2021-05-14 2021-05-14 虚拟视点绘制、渲染、解码方法及装置、设备、存储介质

Country Status (3)

Country Link
US (1) US20240087185A1 (zh)
CN (1) CN117242768A (zh)
WO (1) WO2022236837A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020061131A1 (en) * 2000-10-18 2002-05-23 Sawhney Harpreet Singh Method and apparatus for synthesizing new video and/or still imagery from a collection of real video and/or still imagery
CN106162198A (zh) * 2016-08-31 2016-11-23 重庆邮电大学 基于不规则匀质块分割的三维视频深度图编码及解码方法
US20200013168A1 (en) * 2017-02-20 2020-01-09 Sony Corporation Image processing apparatus and image processing method
CN111524075A (zh) * 2020-03-26 2020-08-11 北京迈格威科技有限公司 深度图像滤波方法、图像合成方法、装置、设备及介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020061131A1 (en) * 2000-10-18 2002-05-23 Sawhney Harpreet Singh Method and apparatus for synthesizing new video and/or still imagery from a collection of real video and/or still imagery
CN106162198A (zh) * 2016-08-31 2016-11-23 重庆邮电大学 基于不规则匀质块分割的三维视频深度图编码及解码方法
US20200013168A1 (en) * 2017-02-20 2020-01-09 Sony Corporation Image processing apparatus and image processing method
CN111524075A (zh) * 2020-03-26 2020-08-11 北京迈格威科技有限公司 深度图像滤波方法、图像合成方法、装置、设备及介质

Also Published As

Publication number Publication date
CN117242768A (zh) 2023-12-15
US20240087185A1 (en) 2024-03-14

Similar Documents

Publication Publication Date Title
CN107452010B (zh) 一种自动抠图算法和装置
Gao et al. Sand-dust image restoration based on reversing the blue channel prior
CN110662044B (zh) 一种视频编码方法、视频编码装置及计算机存储介质
CN108596923B (zh) 三维数据的获取方法、装置以及电子设备
CN110363753B (zh) 图像质量评估方法、装置及电子设备
US10354394B2 (en) Dynamic adjustment of frame rate conversion settings
CN111950723A (zh) 神经网络模型训练方法、图像处理方法、装置及终端设备
CN111402170A (zh) 图像增强方法、装置、终端及计算机可读存储介质
CA3137297C (en) Adaptive convolutions in neural networks
CN115797607B (zh) 一种增强vr真实效果的图像优化处理方法
CN113411582A (zh) 一种基于活动轮廓的视频编码方法、系统、装置及介质
CN113068034A (zh) 视频编码方法及装置、编码器、设备、存储介质
US20230343017A1 (en) Virtual viewport generation method and apparatus, rendering and decoding methods and apparatuses, device and storage medium
US9940543B2 (en) Control of computer vision pre-processing based on image matching using structural similarity
CN113706393A (zh) 视频增强方法、装置、设备及存储介质
CN109377552B (zh) 图像遮挡计算方法、装置、计算设备及存储介质
US20230316464A1 (en) Virtual view drawing method and apparatus, rendering method and apparatus, and decoding method and apparatus, and devices and storage medium
CN112907461A (zh) 一种红外雾天降质图像去雾增强方法
WO2022236837A1 (zh) 虚拟视点绘制、渲染、解码方法及装置、设备、存储介质
CN116824438A (zh) 基于边缘注意门控图卷积网络的半监督视频分割方法
Zhang et al. A compensation textures dehazing method for water alike area
CN113627342B (zh) 视频深度特征提取优化的方法、系统、设备及存储介质
CN115631108A (zh) 一种基于rgbd的图像去雾方法及相关设备
CN110490877B (zh) 基于Graph Cuts的双目立体图像对目标分割方法
CN117561715A (zh) 多平面图像的生成、数据处理、编码和解码方法、装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21941406

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202180097722.1

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21941406

Country of ref document: EP

Kind code of ref document: A1