WO2024174679A1 - 一种基于感知加权的点云率失真编码方法及相关装置 - Google Patents

一种基于感知加权的点云率失真编码方法及相关装置 Download PDF

Info

Publication number
WO2024174679A1
WO2024174679A1 PCT/CN2023/137731 CN2023137731W WO2024174679A1 WO 2024174679 A1 WO2024174679 A1 WO 2024174679A1 CN 2023137731 W CN2023137731 W CN 2023137731W WO 2024174679 A1 WO2024174679 A1 WO 2024174679A1
Authority
WO
WIPO (PCT)
Prior art keywords
distortion
perceptual
projection image
image
point cloud
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/137731
Other languages
English (en)
French (fr)
Inventor
丁克勤
张云
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Institute of Advanced Technology of CAS
Original Assignee
Shenzhen Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Institute of Advanced Technology of CAS filed Critical Shenzhen Institute of Advanced Technology of CAS
Publication of WO2024174679A1 publication Critical patent/WO2024174679A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146—Data rate or code amount at the encoder output
    • H04N19/147—Data rate or code amount at the encoder output according to rate distortion criteria
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/62—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding by frequency transforming in three dimensions [3D]

Definitions

  • the present application relates to the field of image processing technology, and in particular to a point cloud rate-distortion encoding method based on perceptual weighting and related devices.
  • VR virtual reality
  • AR augmented reality
  • MR mixed reality
  • 6DoF six-degree-of-freedom
  • DPC dynamic point cloud
  • Dynamic point cloud uses a large number of unstructured high-dimensional points to represent a three-dimensional scene.
  • DPC is a series of temporally continuous point clouds that can not only reflect motion and temporal light changes, but also each point includes geometric components for identifying positions in three-dimensional space and photometric information for reflecting light and object properties, such as RGB color, reflection, and transparency.
  • dynamic point clouds also generate a huge amount of data due to the large number of high-dimensional points, which requires huge storage space and huge network bandwidth for transmission.
  • V-PCC video point cloud compression
  • D1 point-to-point
  • D2 point-to-plane
  • P2mesh point-to-mesh
  • the technical problem to be solved by this application is to provide a Point cloud rate-distortion coding method based on perceptual weighting and related devices.
  • a first aspect of an embodiment of the present application provides a point cloud rate-distortion encoding method based on perceptual weighting, the method comprising:
  • a rate-distortion cost corresponding to the to-be-encoded point cloud is determined according to the perceived distortion of the projected image, and the projected image is encoded based on the rate-distortion cost.
  • the reference projection image is obtained by downsampling the projection image, and the image scales of the projection image and the plurality of reference projection images are different from each other.
  • the point cloud rate-distortion encoding method based on perceptual weighting, wherein the process of obtaining the gradient weight specifically includes:
  • the gradient value of each pixel in the coding block is calculated, and the gradient weight of the coding block is calculated based on the gradient value of each pixel.
  • the point cloud rate-distortion coding method based on perceptual weighting wherein the determining the rate-distortion cost corresponding to the to-be-coded point cloud according to the perceptual distortion of the projected image specifically comprises:
  • a rate-distortion cost corresponding to the to-be-encoded point cloud is determined based on the perceptual distortion and the perceptual Lagrangian multiplier.
  • the point cloud rate-distortion coding method based on perceptual weighting, wherein the structure is similar
  • the corresponding relationship between the distortion degree and the mean square error distortion degree is:
  • ⁇ k,i represents the perceptual coefficient of coding block i of the k-th target projection image
  • blk n represents the n-th sub-image block of coding block i
  • ⁇ n represents the linear model parameter of the n-th sub-image block of coding block i
  • N represents the number of sub-image blocks
  • E i represents the pixel content weight
  • the k-th target projection image is a projection image in the projection image set formed by the projection image and several reference projection images.
  • the point cloud rate-distortion coding method based on perceptual weighting wherein when the perceptual Lagrangian multiplier is used for coding, the determining the perceptual Lagrangian multiplier corresponding to the projected image based on the perceptual coefficient specifically includes:
  • the product of the first perceptual coefficient ratio and the mean square error Lagrangian multiplier of the projected image is calculated to obtain a perceptual Lagrangian multiplier corresponding to the coding block.
  • the point cloud rate-distortion coding method based on perceptual weighting, wherein when the perceptual Lagrangian multiplier is used for mode decision, the determining the perceptual Lagrangian multiplier corresponding to the projected image based on the perceptual coefficient specifically includes:
  • a second aspect of an embodiment of the present application provides a point cloud rate-distortion coding system based on perceptual weighting, the system comprising:
  • An acquisition module used to acquire a projection image corresponding to the point cloud to be encoded and a plurality of reference projection images corresponding to the projection module, wherein the projection image and the plurality of reference projection images both include a geometric projection image and a texture projection image;
  • a calculation module used to obtain the gradient weight and structural similarity distortion of each coding block of the projection image and each reference projection image, and calculate the perceptual distortion of the projection image based on the gradient weight and structural similarity distortion of each coding block;
  • the encoding module is used to determine the rate-distortion cost corresponding to the to-be-encoded point cloud according to the perceived distortion of the projected image, and encode the projected image based on the rate-distortion cost.
  • a third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the perceptually weighted point cloud rate-distortion encoding methods described above.
  • a fourth aspect of the embodiments of the present application provides a terminal device, comprising: a processor, a memory, and a communication bus; the memory stores a computer-readable program that can be executed by the processor;
  • the communication bus realizes the connection and communication between the processor and the memory
  • the present application provides a point cloud rate-distortion coding method and related devices based on perceptual weighting, the method comprising obtaining a projection image corresponding to the point cloud to be coded and a plurality of reference projection images corresponding to the projection module; obtaining the gradient weight and structural similarity distortion of each coding block of the projection image and each reference projection image, and based on the gradient of each coding block
  • the perceptual distortion of the projected image is calculated based on the weight and structural similarity distortion; the rate-distortion cost corresponding to the point cloud to be encoded is determined according to the perceptual distortion of the projected image, and the projected image is encoded based on the rate-distortion cost.
  • the present application generates a two-dimensional projected image by projection, and then calculates the perceptual distortion corresponding to each projected image in units of coding blocks, and makes coding decisions based on the perceptual loss, thereby improving the matching of point cloud coding and visual perception quality, thereby improving coding efficiency.
  • FIG1 is a flowchart of a point cloud rate-distortion coding method based on perceptual weighting provided in the present application.
  • FIG2 is a flow chart showing the principle of the point cloud rate-distortion coding method based on perceptual weighting provided in the present application.
  • FIG. 3 is a scatter plot of the correlation between the texture projection map of the approximate model and the original model.
  • FIG. 4 is a scatter plot of the correlation between the approximate model and the original model on the geometric projection diagram.
  • FIG5 is a structural principle diagram of the point cloud rate-distortion coding method based on perceptual weighting provided in the present application.
  • FIG6 is a schematic diagram of the structure of the terminal device provided in this application.
  • the present application provides a point cloud rate-distortion encoding method and related devices based on perceptual weighting.
  • the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.
  • sequence number and size of each step in this embodiment do not mean the order of execution.
  • the execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.
  • VR virtual reality
  • AR augmented reality
  • MR mixed reality
  • 6DoF six-degree-of-freedom
  • DPC dynamic point cloud
  • Dynamic point cloud uses a large number of unstructured high-dimensional points to represent a three-dimensional scene.
  • DPC is a series of temporally continuous point clouds that can not only reflect motion and temporal light changes, but also each point includes a feature for identifying the position in three-dimensional space.
  • the dynamic point cloud contains geometric components and photometric information used to reflect the properties of light and objects, such as RGB color, reflection and transparency.
  • dynamic point clouds also generate a huge amount of data due to the large number of high-dimensional points, which requires huge storage space and huge network bandwidth for transmission.
  • V-PCC video point cloud compression
  • D1 point-to-point
  • D2 point-to-plane
  • P2mesh point-to-mesh
  • Jiang et al. proposed a new perceptual coding scheme that complies with the H.265/HEVC standard by using spatiotemporal saliency.
  • Wu et al. proposed a rate-distortion (RD) model of perceptual weighted mean square error (PWMSE) and derived the Lagrange multipliers of the rate-distortion optimization (RDO) process based on the equivalent distortion.
  • RD rate-distortion
  • PWMSE perceptual weighted mean square error
  • RDO rate-distortion optimization
  • Li et al. optimized the RDO task using a 360-degree video evaluation metric in the spherical domain. Li et al.
  • Li et al. adopted an occupancy map-based RDO method, which designed an occupancy map-guided video compression framework and used occupancy maps to improve DPC compression performance. Since the current RDO and geometric distortion are inconsistent with the evaluation criteria D1 and D2, Xiong et al. proposed an EPM-based RDO method, which first describes the relationship between the existing distortion model and geometric quality measurement, and then estimates the normal vectors of D1 and D2 by estimating the normal vector of CU, thereby modifying the 3D geometric distance. However, the quality of the reconstructed point cloud in these rate-distortion coding methods is still measured by D1 and D2, which cannot truly reflect human perception. Therefore, the existing point cloud coding schemes do not fully consider the perceptual characteristics of point clouds, and certainly do not utilize the perceptual redundancy of point clouds.
  • a projection image corresponding to the point cloud to be encoded and several reference projection images corresponding to the projection module are obtained; the gradient weights and structural similarity distortions of each coding block of the projection image and each reference projection image are obtained, and the perceptual distortion of the projection image is calculated based on the gradient weights and structural similarity distortions of each coding block; the rate-distortion cost corresponding to the point cloud to be encoded is determined according to the perceptual distortion of the projection image, and the projection image is encoded based on the rate-distortion cost.
  • the present application generates a two-dimensional projection image by projection, and then calculates the perceptual distortion corresponding to each projection image in units of coding blocks, and makes encoding decisions based on the perceptual loss, thereby improving the matching of point cloud encoding and visual perception quality, thereby improving encoding efficiency.
  • This embodiment provides a point cloud rate-distortion encoding method based on perceptual weighting, as shown in FIG1 and FIG2 , and the method includes:
  • the projection image and the reference projection images both include a geometric projection map and a texture projection map, wherein the projection image is formed by patch segmentation and packaging of the point cloud to be encoded, the geometric projection map and the texture projection are both two-dimensional images, and the image scales of the geometric projection map and the texture projection map are the same.
  • Each reference projection image in the reference projection images is an image group, and the image group includes a reference geometric projection map and a reference texture projection map, wherein the image scales of each geometric projection map in each reference projection image are different from each other, and are not equal to the image scale of the geometric projection map; the image scales of each texture projection map in each reference projection image are different from each other, and are not equal to the image scale of the texture projection map.
  • the reference projection image is obtained by downsampling the projection image, wherein the reference projection image is obtained by downsampling the projection image, which means that the reference geometric projection map is obtained by downsampling the geometric projection map, and the reference texture projection map is obtained by downsampling the geometric projection map.
  • several reference projection images can be downsampled using a Gaussian pyramid, that is, the geometric projection map and the texture projection map included in the projection image are input into the Gaussian pyramid, and the output items of each layer of the Gaussian pyramid are obtained to obtain several reference projection images. It can be understood that through each layer of the Gaussian pyramid, the resolution of the output item of the layer is reduced to half of the input item.
  • k is used to represent different scales
  • the image scale corresponding to the reference projection image obtained by downsampling is half of the image scale corresponding to the projection image, and so on, to obtain several projection images.
  • a plurality of projection images may be sorted from large to small according to image scale to form a first reference projection image, a second reference projection image, ..., an Nth reference projection image, wherein the first reference projection image is obtained by downsampling the projection image by 2 times, the second reference projection image is obtained by downsampling the projection image by 4 times, and so on, to obtain the Nth reference projection image.
  • the corresponding relationship between the image scales of each Nth reference projection image may also be different, for example, the image scale corresponding to the projection image is 3 times the image scale of the first reference projection image, the image scale corresponding to the first reference projection image is 3 times the image scale of the second reference projection image, and so on.
  • the gradient weight is determined based on the gradient of each pixel in the coding block, and the structural similarity distortion is used to reflect the distortion of the distorted block corresponding to the coding block, where the distorted block is determined based on the geometric reconstruction map and texture reconstruction map formed by the reconstructed point cloud corresponding to the point cloud to be coded. That is to say, when obtaining the projection image and several reference projection images, Step 1: obtain the reconstructed projection image of the reconstructed point cloud corresponding to the point cloud to be encoded, and the reconstructed projection image includes a geometric reconstruction map and a texture reconstruction map; then downsample the reconstructed projection image to obtain a number of reference reconstructed projection images.
  • the determination process of the reconstructed projection image and the number of reference reconstructed projection images is the same as the determination process of the projection image and the number of reference projection images, which will not be described in detail here. It is only explained that the projection image corresponds to the reconstructed projection image, and the number of reference projection images correspond to the number of reference reconstructed projection images one by one, and the image scale of the reference projection image is the same as the image scale of the corresponding reference reconstructed projection image, that is, the image scale of the reference geometric projection map in the reference projection image is equal to the image scale of the reference geometric reconstruction map in the corresponding reference reconstructed projection image, and the image scale of the reference texture projection map in the reference projection image is equal to the image scale of the reference texture reconstruction map in the corresponding reference reconstructed projection image.
  • the process of obtaining the gradient weight specifically includes:
  • the gradient value of each pixel in the coding block is calculated, and the gradient weight of the coding block is calculated based on the gradient value of each pixel.
  • the gradient value of each pixel in the coding block includes a horizontal gradient and a vertical gradient
  • the calculation formulas of the horizontal gradient and the vertical gradient are respectively:
  • F H represents the gradient operator in the horizontal direction
  • I i represents the i-th coding block with a size of M ⁇ N
  • j represents the j-th pixel in the coding block.
  • the gradient value of the coding block is calculated based on the pixel gradient value of each pixel point in the coding block, wherein the calculation formula of the candidate gradient value of the coding block can be:
  • each candidate gradient value is normalized to obtain the gradient value of each coding block.
  • the gradient value of the coding block is used to reflect the importance of the coding block relative to all coding blocks.
  • Wi represents the gradient value of the coding block
  • Nomal( ⁇ ) represents the normalization operation, which is calculated as follows:
  • the C( ⁇ ) operation constrains the calculated maximum and minimum values to be between [0,1], that is, values less than 0 are set to 0, and values greater than 1 are set to 1.
  • ⁇ x represents the mean of xi
  • ⁇ x represents the standard deviation of xi .
  • the structural similarity distortion is the structural similarity distortion between the coding block and its corresponding reconstructed coding block.
  • the perceptual distortion corresponding to the projection image can be determined based on the gradient weight and structural similarity distortion of each coding block.
  • the texture image block distortion of the i-th texture distortion image block in the k-th target projection image in the projection image set formed by the projection image and several reference projection images can be expressed as:
  • Wk ,i represents the gradient weight of the i-th coding block in the k-th image group
  • It represents the SSIM distortion of the i-th coding block in the k-th group of images relative to the i-th reconstructed coding block.
  • the objective function of the rate-distortion cost is:
  • J represents the total rate-distortion (RD) cost
  • D represents the distortion difference between the point cloud to be encoded and the reconstructed point cloud corresponding to the unit to be encoded
  • R represents the coded bits
  • ⁇ represents the Lagrange multiplier used to balance the distortion and bit rate.
  • the objective function of determining the rate-distortion cost corresponding to the to-be-encoded point cloud according to the perceived distortion of the projected image can be expressed as:
  • D PPCM represents the perceived distortion
  • the objective function of the rate-distortion cost of the texture projection image and the objective function of the rate-distortion cost of the geometric projection image are expressed as:
  • J G represents the independent constraint of the texture projection image
  • J T represents the independent constraint of the geometric projection image
  • RT is a constant for the geometric video encoder
  • RG is a constant for the texture video encoder.
  • the encoding decision process of the texture projection image is the same as the encoding decision process of the geometric projection image, only the encoding decision process is explained here, and the texture projection image and the geometric projection image are not explained.
  • the texture projection image and the geometric projection image can adopt the following encoding decision process.
  • determining the rate-distortion cost corresponding to the to-be-encoded point cloud according to the perceived distortion of the projected image specifically includes:
  • a rate-distortion cost corresponding to the to-be-encoded point cloud is determined based on the perceptual distortion degree and the perceptual Lagrangian multiplier.
  • E i represents the pixel content weight
  • M represents the number of coding blocks
  • C 2 is the C 2 constant in SSIM
  • ⁇ t represents the filter coefficient of the t-th position of the current coding block
  • It represents the variance of the coded block before and after encoding at it.
  • the SSIM distortion of the coding block is set to the average value of 16 sub-blocks, where n represents the sub-image block of the coding block, that is:
  • ⁇ n is a linear model parameter related to the image content, where the result of encoding the same-position block in the previous frame is used as the reference image content (except for the first frame), and Qn is the quantization step size of the sub-block, which is usually applied to all sub-blocks with the same value.
  • D MSE For the coding block, D MSE can be expressed as:
  • ⁇ k,i represents the perceptual coefficient of coding block i of the k-th projection image
  • blk n the n-th sub-image block of coding block i
  • ⁇ n represents the linear model parameter of the n-th sub-image block of coding block i
  • N represents the number of sub-image blocks
  • E i represents the pixel content weight
  • the k-th projection image is a projection image in the projection image set formed by the projection image and several reference projection images.
  • D PPCM D PPCM
  • the perception Lagrangian multiplier corresponding to the projected image is determined, wherein the process of determining the perception Lagrangian multiplier may be:
  • ⁇ MSE is the MSE-based distortion and bit rate in HEVC and Lagrange multipliers, code rate and distortion The relationship between can be modeled as
  • the total bit rate of a video is the sum of the number of bits in all blocks, which can be calculated as:
  • M is the number of coding blocks in a frame/video.
  • the rate coding distortion objective function takes the partial derivative of D MSE , and when the partial derivative is zero, we can get:
  • the total number of bits of the coding channel ⁇ is calculated as:
  • the perceptual Lagrangian parameters can be determined based on the total number of bits of the coding channel ⁇ .
  • the Determining the perceptual Lagrangian multiplier corresponding to the projected image by the perceptual coefficient specifically includes:
  • the product of the first perceptual coefficient ratio and the mean square error Lagrangian multiplier of the projected image is calculated to obtain a perceptual Lagrangian multiplier corresponding to the coding block.
  • the attribute and geometry videos are encoded by the MSE-based encoder.
  • determining the perceptual Lagrangian multiplier corresponding to the projected image based on the perceptual coefficient specifically includes:
  • the V-PCC encoding standard maps the geometric features and texture features of the three-dimensional point cloud into two video sequences, respectively recorded as a geometric projection map sequence and a texture projection map sequence, and then uses the existing HEVC/VVC or other video encoders to compress the video sequence of the dynamic point cloud.
  • some meta-information needs to be generated, such as a placeholder map and auxiliary blocking information, which is used to illustrate the two video sequences.
  • the encoding process is the same as the existing process and is not described in detail here.
  • the point cloud projection patch in V-PCC is a set of information, including the 3D bounding box of the point cloud, related geometry and texture information, and atlas information required for 3D reconstruction.
  • the process of dividing the point cloud projection patch is to project each frame of point cloud from 3D space to a given 2D plane and divide the point cloud into as few patches with smooth boundaries as possible. To minimize the reconstruction error.
  • the point cloud projection process can be to first calculate the normal vector of each point in the input point cloud, and then cluster the entire point cloud according to the normal vector and project it onto the six faces of the cube to form an initial cluster. Then, according to the normal vector and the index of the nearest point, the clustering index of each point is updated through continuous iteration to achieve a finer patch division.
  • the connected component extraction method is applied to extract the connected patches and merge them into larger patch blocks to obtain the final patch set. Therefore, the point cloud projection patch set is packaged and reorganized on a two-dimensional plane, and the geometric and texture information in the three-dimensional space is converted into two-dimensional images respectively, thereby realizing the generation and filling process of geometric and texture images to obtain a geometric projection map sequence and a texture projection map sequence.
  • the following description is based on the premise that the geometric projection map sequence and the texture projection map sequence have been obtained.
  • the encoding process of the geometric projection map sequence and the texture projection map sequence includes:
  • step c) Encode the current geometric projection frame; if the geometric projection images corresponding to all video frames are encoded, proceed to step d) for texture encoding, otherwise, proceed to step a) for encoding the next frame;
  • step f) Encode the current texture projection map, and then go to step d) to encode the next frame texture projection map until all texture projection maps are encoded.
  • this embodiment provides a point cloud rate-distortion coding method based on perceptual weighting, the method comprising obtaining a projection image corresponding to a point cloud to be coded and a plurality of reference projection images corresponding to the projection module; obtaining gradient weights and structural similarity distortions of each coding block of the projection image and each reference projection image, and based on the gradient weights and structural similarity distortions of each coding block
  • the perceptual distortion of the projected image is calculated based on the structural similarity distortion
  • the rate-distortion cost corresponding to the point cloud to be encoded is determined based on the perceptual distortion of the projected image
  • the projected image is encoded based on the rate-distortion cost.
  • the present application generates a two-dimensional projected image by projection, and then calculates the perceptual distortion corresponding to each projected image in units of coding blocks, and makes coding decisions based on the perceptual loss, thereby improving the matching of point cloud coding with visual perception quality, thereby improving coding efficiency.
  • each point cloud is compressed with 17 quantization parameter (QP) pairs ranging from 20 to 32 and 27 to 42.
  • QP quantization parameter
  • Each point cloud is projected onto a two-dimensional image and decomposed into 400 blocks, i.e., 400 ⁇ 17 sampling blocks.
  • R 2 is used to calculate and The approximate model accuracy between the two, the results show that the average R2 is 0.9732 and 0.9556 on the attribute and geometry channels, and the approximation in perception is accurate.
  • V-PCC V-PCC reference software TMC2-10.0 and the corresponding HEVC reference software HM16.20-SCM8.8.
  • the latest V-PCC (TMC2-10.0+HM16.20-SCM8.8) is used as an anchor method for comparison.
  • two state-of-the-art RDOs for V-PCC occupancy map-based RDO, denoted as "OC-RDO”, and EPM-based RDO methods, denoted as "EPM-RDO” are also used as benchmarks for comparison.
  • OC-RDO occupancy map-based RDO
  • EPM-based RDO EPM-based RDO
  • the proposed coding method can be applied to intra- and inter-frame coding of V-PCC, V-PCC is configured as random access coding mode (RA) for geometry and texture coding.
  • RA random access coding mode
  • GraphSIM independent of the proposed PPCM
  • BDBR Bjontegaard Delta Bit Rate
  • Table 1 shows the RD comparison on the 6 tested DPCs, where the visual quality of the compressed video is measured using GraphSIM and RT represents the total bitrate (Mbps). It can be seen that OC-RDO can achieve a bitrate reduction from 0.34% to 17.05% with an average of 9.61% compared to V-PCC. EPM-RDO is able to achieve a bitrate reduction from -4.08% to 5.61% with an average of 1.17%. BDBR Savings. As for the proposed PWRDO, it achieves a BDBR reduction from 7.09% to 22.79%, an average reduction of 13.52% compared to V-PCC, and significantly exceeds the two coding optimization comparison schemes OC-RDO and EPM-RDO.
  • the proposed PWRDO To evaluate the perceptual coding efficiency of the proposed PWRDO, five different perceptual PCQA metrics, including D1, GraphSIM, MPED, SIAT_PCQA, and the proposed PPCM, are used to measure the visual quality of compressed DPC in different schemes. Then, the BDBR is calculated for each PCQA, which includes the number of total coded bits including geometry bits, attribute bits, and metadata bits. A negative BDBR means bit saving, while a positive one indicates a decrease in coding efficiency compared to the anchor point. Table 1 shows the RD comparison of the proposed PWRDO and the baseline RDO scheme under five different perceptual PCQA metrics. These comparison results show that the proposed PWRDO can stably achieve higher coding gains in all PCQAs, which proves the effectiveness of the proposed PWRDO.
  • this embodiment provides a point cloud rate-distortion coding system based on perceptual weighting, as shown in FIG5 , the system includes:
  • An acquisition module 100 is used to acquire a projection image corresponding to the point cloud to be encoded and a plurality of reference projection images corresponding to the projection module, wherein the projection image and the plurality of reference projection images both include a geometric projection image and a texture projection image;
  • a calculation module 200 configured to obtain the gradient weight and structural similarity distortion of each coding block of the projection image and each reference projection image, and calculate the perceptual distortion of the projection image based on the gradient weight and structural similarity distortion of each coding block;
  • the encoding module 300 is used to determine the to-be-encoded image according to the perceived distortion of the projected image.
  • a rate-distortion cost corresponding to the point cloud is calculated, and the projection image is encoded based on the rate-distortion cost.
  • this embodiment provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the point cloud rate-distortion encoding method based on perceptual weighting as described in the above embodiment.
  • the present application also provides a terminal device, as shown in FIG6 , which includes at least one processor (processor) 20; a display screen 21; and a memory (memory) 22, and may also include a communication interface (Communications Interface) 23 and a bus 24.
  • the processor 20, the display screen 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24.
  • the display screen 21 is configured to display a preset user guide interface in the initial setting mode.
  • the communication interface 23 can transmit information.
  • the processor 20 can call the logic instructions in the memory 22 to execute the method in the above-mentioned embodiment.
  • logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.
  • the memory 22, as a computer-readable storage medium, can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure.
  • the processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.
  • the memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc.
  • the memory 22 may include a high-speed random access memory and may also include a non-volatile memory.
  • a USB flash drive a mobile hard disk, a read-only memory (ROM), a random access memory (Random Access Memory, or ROM), or a memory card.
  • Access Memory (RAM) magnetic disks or optical disks, etc., which can store program code, or they can be temporary storage media.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本申请公开了一种基于感知加权的点云率失真编码方法及相关装置，所述方法包括获取待编码点云对应的投影图像以及所述投影模块对应的若干参考投影图像；获取投影图像及各参考投影图像的各编码块的梯度权重及结构相似性失真度，并基于各编码块的梯度权重及结构相似性失真度计算投影图像的感知失真度；根据投影图像的感知失真度确定所述待编码点云对应的率失真成本，并基于所述率失真成本编码所述投影图像。本申请通过投影生成二维的投影图像，然后计算以编码块为单位确定每个投影图像对应的感知失真度，并基于感知损失度进行编码决策，提高了点云编码与视觉感知质量的匹配性，从而提高了编码效率。

Description

一种基于感知加权的点云率失真编码方法及相关装置 技术领域
本申请涉及图像处理技术领域，特别涉及一种基于感知加权的点云率失真编码方法及相关装置。
背景技术
近年来，虚拟现实(VR)、增强现实(AR)和混合现实(MR)等三维因能够为用户提供独特的六自由度(6DoF)交互、逼真和沉浸式3D视觉体验，而在许多应用中开始流行，如3D电影观看、遗产保护、导航、沉浸式远程通话和远程手术等。其中，动态点云(DPC)因其所具有的逼真的表现能力，已经成为新兴的沉浸式VR、AR和MR媒体的主流表现形式之一。
动态点云(DPC)用大量非结构化的高维点来表示三维场景，其中，DPC是一系列时间上连续的点云，其不但可以反映运动和时间上的光线变化，而且其中的每个点都均包括用于识别三维空间中的位置的几何成分以及用于反映光线和物体属性的光度信息，如RGB颜色、反射和透明度等。然而，动态点云由于包括大量高维点也产生了庞大的数据量，这需要巨大的存储空间和庞大的网络带宽来传输。
为了解决这个问题，在对动态点云进行存储和传输之前，需要对动态点云进行压缩以减少动态点云所需的存储空间以及传输所需的带宽。目前动态点云普遍应用的压缩技术为视频点云压缩(V-PCC)模型，V-PCC模型是用原始点云和失真的点云之间的信号差来衡量失真，如，点对点(D1)、点对面(D2)和点对网(P2mesh)。然而，现有的V-PCC模型均未考虑动态点云的视觉感知质量，无法利用动态点云的感知冗余，进而影响编码效率。
因而现有技术还有待改进和提高。
技术问题
本申请要解决的技术问题在于，针对现有技术的不足，提供一种 基于感知加权的点云率失真编码方法及相关装置。
技术解决方案
为了解决上述技术问题，本申请实施例第一方面提供了一种基于感知加权的点云率失真编码方法，所述方法包括：
获取待编码点云对应的投影图像以及所述投影模块对应的若干参考投影图像，其中，所述投影图像和若干参考投影图像均包括几何投影图和纹理投影图；
获取投影图像及各参考投影图像的各编码块的梯度权重及结构相似性失真度，并基于各编码块的梯度权重及结构相似性失真度计算投影图像的感知失真度；
根据投影图像的感知失真度确定所述待编码点云对应的率失真成本，并基于所述率失真成本编码所述投影图像。
所述基于感知加权的点云率失真编码方法，其中，所述参考投影图像为通过对投影图像进行下采样得到的，并且投影图像和若干参考投影图像的图像尺度互不相同。
所述基于感知加权的点云率失真编码方法，其中，所述梯度权重的获取过程具体包括：
计算所述编码块中的各像素点的梯度值，并基于各像素点的梯度值计算所述编码块的梯度权重。
所述基于感知加权的点云率失真编码方法，其中，所述根据投影图像的感知失真度确定所述待编码点云对应的率失真成本具体包括：
将所述感知失真度中的结构相似性失真度转换为均方误差失真度，以得到所述投影图像对应的感知系数；
基于所述感知系数确定所述投影图像对应的感知拉格朗日乘数；
基于所述感知失真度和所述感知拉格朗日乘数确定所述待编码点云对应的率失真成本。
所述基于感知加权的点云率失真编码方法，其中，所述结构相似 性失真度与所述均方误差失真度的对应关系为：

其中，表示第k个目标投影图像的编码块i的结构相似性失真度，表示第k个目标投影图像的编码块i的均方误差失真度，Φk,i表示第k个目标投影图像的编码块i的感知系数，blkn表示编码块i的第n个子图像块，ρn表示编码块i的第n个子图像块的线性模型参数，N表示子图像块的数量，Ei表示像素内容权重，第k个目标投影图像为由投影图像和若干参考投影图像形成的投影图像集中的一投影图像。
所述基于感知加权的点云率失真编码方法，其中，当感知拉格朗日乘数用于编码时，所述基于所述感知系数确定所述投影图像对应的感知拉格朗日乘数具体包括：
基于所述投影图像的每个编码块的感知系数计算第一目标感知系数，并计算所述第一目标感知系数与所述编码块的感知系数的比值，以得到第一感知系数比值；
计算所述第一感知系数比值与所述投影图像的均方误差拉格朗日乘数的乘积，以得到所述编码块对应的感知拉格朗日乘数。
所述基于感知加权的点云率失真编码方法，其中，当感知拉格朗日乘数用于模式决策时，所述基于所述感知系数确定所述投影图像对应的感知拉格朗日乘数具体包括：
基于所述投影图像的每个编码块的感知系数计算第二目标感知系数，并计算第二目标感知系数与编码块的感知系数的平方根的比值，以得到第二感知系数比值；
计算所述第二感知系数比值与所述投影图像的所述均方误差拉格朗日乘数的平方根的乘积，以得到所述编码块对应的感知拉格朗日 乘数。
本申请实施例第二方面提供了一种基于感知加权的点云率失真编码系统，所述系统包括：
获取模块，用于获取待编码点云对应的投影图像以及所述投影模块对应的若干参考投影图像，其中，所述投影图像和若干参考投影图像均包括几何投影图和纹理投影图；
计算模块，用于获取投影图像及各参考投影图像的各编码块的梯度权重及结构相似性失真度，并基于各编码块的梯度权重及结构相似性失真度计算投影图像的感知失真度；
编码模块，用于根据投影图像的感知失真度确定所述待编码点云对应的率失真成本，并基于所述率失真成本编码所述投影图像。
本申请实施例第三方面提供了一种计算机可读存储介质，所述计算机可读存储介质存储有一个或者多个程序，所述一个或者多个程序可被一个或者多个处理器执行，以实现如上任一所述的基于感知加权的点云率失真编码方法中的步骤。
本申请实施例第四方面提供了一种终端设备，其包括：处理器、存储器及通信总线；所述存储器上存储有可被所述处理器执行的计算机可读程序；
所述通信总线实现处理器和存储器之间的连接通信；
所述处理器执行所述计算机可读程序时实现如上任一所述的基于感知加权的点云率失真编码方法中的步骤。
有益效果
与现有技术相比，本申请提供了一种基于感知加权的点云率失真编码方法及相关装置，方法包括获取待编码点云对应的投影图像以及所述投影模块对应的若干参考投影图像；获取投影图像及各参考投影图像的各编码块的梯度权重及结构相似性失真度，并基于各编码块的梯度 权重及结构相似性失真度计算投影图像的感知失真度；根据投影图像的感知失真度确定所述待编码点云对应的率失真成本，并基于所述率失真成本编码所述投影图像。本申请通过投影生成二维的投影图像，然后计算以编码块为单位确定每个投影图像对应的感知失真度，并基于感知损失度进行编码决策，提高了点云编码与视觉感知质量的匹配性，从而提高了编码效率。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员而言，在不符创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请提供的基于感知加权的点云率失真编码方法的流程图。
图2为本申请提供的基于感知加权的点云率失真编码方法的原理流程图。
图3为近似模型与原模型的纹理投影图上的相关性的散点图。
图4为近似模型与原模型的几何投影图上的相关性的散点图。
图5为本申请提供的基于感知加权的点云率失真编码方法的结构原理图。
图6为本申请提供的终端设备的结构原理图。
本发明的最佳实施方式
本申请提供一种基于感知加权的点云率失真编码方法及相关装置，为使本申请的目的、技术方案及效果更加清楚、明确，以下参照附图并举实施例对本申请进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本申请，并不用于限定本申请。
本技术领域技术人员可以理解，除非特意声明，这里使用的单数 形式“一”、“一个”、“所述”和“该”也可包括复数形式。应该进一步理解的是，本申请的说明书中使用的措辞“包括”是指存在所述特征、整数、步骤、操作、元件和/或组件，但是并不排除存在或添加一个或多个其他特征、整数、步骤、操作、元件、组件和/或它们的组。应该理解，当我们称元件被“连接”或“耦接”到另一元件时，它可以直接连接或耦接到其他元件，或者也可以存在中间元件。此外，这里使用的“连接”或“耦接”可以包括无线连接或无线耦接。这里使用的措辞“和/或”包括一个或更多个相关联的列出项的全部或任一单元和全部组合。
本技术领域技术人员可以理解，除非另外定义，这里使用的所有术语(包括技术术语和科学术语)，具有与本申请所属领域中的普通技术人员的一般理解相同的意义。还应该理解的是，诸如通用字典中定义的那些术语，应该被理解为具有与现有技术的上下文中的意义一致的意义，并且除非像这里一样被特定定义，否则不会用理想化或过于正式的含义来解释。
应理解，本实施例中各步骤的序号和大小并不意味着执行顺序的先后，各过程的执行顺序以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
经过研究发现，近年来，虚拟现实(VR)、增强现实(AR)和混合现实(MR)等三维因能够为用户提供独特的六自由度(6DoF)交互、逼真和沉浸式3D视觉体验，而在许多应用中开始流行，如3D电影观看、遗产保护、导航、沉浸式远程通话和远程手术等。其中，动态点云(DPC)因其所具有的逼真的表现能力，已经成为新兴的沉浸式VR、AR和MR媒体的主流表现形式之一。
动态点云(DPC)用大量非结构化的高维点来表示三维场景，其中，DPC是一系列时间上连续的点云，其不但可以反映运动和时间上的光线变化，而且其中的每个点都均包括用于识别三维空间中的位置 的几何成分以及用于反映光线和物体属性的光度信息，如RGB颜色、反射和透明度等。然而，动态点云由于包括大量高维点也产生了庞大的数据量，这需要巨大的存储空间和庞大的网络带宽来传输。
为了解决这个问题，在对动态点云进行存储和传输之前，需要对动态点云进行压缩以减少动态点云所需的存储空间以及传输所需的带宽。目前动态点云普遍应用的压缩技术为视频点云压缩(V-PCC)模型，V-PCC模型是用原始点云和失真的点云之间的信号差来衡量失真，如，点对点(D1)、点对面(D2)和点对网(P2mesh)。然而，现有的V-PCC模型均未考虑动态点云的视觉感知质量，无法利用动态点云的感知冗余，进而影响编码效率。
为此，将视觉感知与编码相结合一直是视频和图像处理领域的热门课题。在已有的研究工作当中，Jiang等人提出了一种新的符合H.265/HEVC标准的感知编码方案，通过使用时空显著性。Wu等人提出了感知加权平均平方误差(PWMSE)的码率失真(RD)模型，并根据等效失真得出了码率失真优化(RDO)过程的拉格朗日乘数。基于WS-PSNR，Li等人用球面域的360度视频评价指标优化了RDO任务。Li等人提出了一种针对球域视频的RDO方法，以解决二维图像平面畸变计算不准确的问题。上述方法的基本思路是利用HVS模型或其近似值作为编码模块中的失真，并以最小化感知失真为约束条件，选择最佳的编码模式或参数。然而，上述方法是针对传统的2D或360度视频编码提出的，无法用于动态点云。
为了利用点云中的视觉冗余，Li等人采用了一种基于占用图的RDO方法，该方法设计了一个占用图指导的视频压缩框架，利用占用图来提升DPC压缩性能。由于目前的RDO和几何失真与评估标准D1和D2不一致，Xiong等人提出了一种基于EPM的RDO方法，该方法首先描述了现有失真模型和几何质量测量之间的关系，然后通过估计CU的法向量来估计D1和D2的法向量，从而修改三维几何距离。然 而，这些率失真编码方式中重建的点云质量仍然是用D1和D2来衡量的，不能真实反映人类的感知。因此，现有的点云编码方案中，没有充分考虑到点云的感知特性，应然没有利用点云的感知冗余。
为了解决上述问题，在本申请实施例中，获取待编码点云对应的投影图像以及所述投影模块对应的若干参考投影图像；获取投影图像及各参考投影图像的各编码块的梯度权重及结构相似性失真度，并基于各编码块的梯度权重及结构相似性失真度计算投影图像的感知失真度；根据投影图像的感知失真度确定所述待编码点云对应的率失真成本，并基于所述率失真成本编码所述投影图像。本申请通过投影生成二维的投影图像，然后计算以编码块为单位确定每个投影图像对应的感知失真度，并基于感知损失度进行编码决策，提高了点云编码与视觉感知质量的匹配性，从而提高了编码效率。
本发明的实施方式
下面结合附图，通过对实施例的描述，对申请内容作进一步说明。
本实施例提供了一种基于感知加权的点云率失真编码方法，如图1和图2所示，所述方法包括：
S10、获取待编码点云对应的投影图像以及所述投影模块对应的若干参考投影图像。
具体地，所述投影图像和若干参考投影图像均包括几何投影图和纹理投影图，其中，投影图像为对待编码点云进行将待编码点云进行补片分割和打包所形成的，几何投影图和纹理投影均为二维图像，并且几何投影图和纹理投影图的图像尺度相同。若干参考投影图像中的每个参考投影图像均为一个图像组，图像组包括参考几何投影图和参考纹理投影图，其中，各参考投影图像中的各几何投影图的图像尺度互不相同，且均不同等于几何投影图的图像尺度；各参考投影图像中的各纹理投影图的图像尺度互不相同，且均不同等于纹理投影图的图像尺度。
参考投影图像为通过对投影图像进行下采样得到的，其中，参考投影图像为通过对投影图像进行下采样得到的指的是参考几何投影图为对几何投影图进行下采样得到，考纹理投影图为对几何投影图进行下采样得到。在一个实现方式中，若干参考投影图像可以采用高斯金字塔进行下采样操作，即将投影图像包括的几何投影图和纹理投影图输入高斯金字塔，获取高斯金字塔各层的输出项，以得到若干参考投影图。可以理解的是，通过高斯金字塔的每一层，将该层输出项的分辨率降低为输入项的二分之一，例如，用k来表示不同的尺度，k＝1表示投影图像对应的图像尺度，k＝2表示对投影图像进行下采样一次，下采样得到的参考投影图像对应的图像尺度为投影图像对应的图像尺度的二分之一，以此类推，以得到若干投影图像。
当然，在实际应用中，在通过投影生成投影图像后，若干参考投影图像的获取方式还可以采用其他方式，例如，将若干投影图像按照图像尺度从大到小排序形成第一参考投影图像、第二参考投影图像，、、、，第N参考投影图像，其中，第一参考投影图像通过对投影图像进行2倍下采样得到的，第二参考投影图像通过对投影图像进行4倍下采样得到的，依次类推，得到第N参考投影图像。此外，各第N参考投影图像的图像尺度的对应关系也可以不同，例如，投影图像对应的图像尺度为第一参考投影图像的图像尺度3倍，第一参考投影图像对应的图像尺度为第二参考投影图像对应的图像尺度的3倍等。
S20、获取投影图像及各参考投影图像的各编码块的梯度权重及结构相似性失真度，并基于各编码块的梯度权重及结构相似性失真度计算投影图像的感知失真度。
具体地，梯度权重为基于编码块中的各像素点的梯度确定，结构相似性失真度用于反映编码块对应的失真块的失真度，其中，失真块为基于待编码点云对应的重构点云所形成的几何重构图和纹理重构图确定的。也就是说，在获取投影图像和若干参考投影图像时，会同 步获取待编码点云对应的重构点云的重构投影图像，重构投影图像包括几何重构图和纹理重构图；然后再对重构投影图像进行下采样操作以得若干参考重构投影图像。其中，重构投影图像和若干参考重构投影图像的确定过程与投影图像和若干参考投影图像的确定过程相同，这里就不具体说明，仅是说明投影图像与重构投影图像相对应，若干参考投影图像与若干参考重构投影图像一一对应，并且参考投影图像的图像尺度与其对应的参考重构投影图像的图像尺度相同，即参考投影图像中的参考几何投影图的图像尺度等于其对应的参考重构投影图像中的参考几何重构图的图像尺度，参考投影图像中的参考纹理投影图的图像尺度等于其对应的参考重构投影图像中的参考纹理重构图的图像尺度。
在一个实现方式中，所述梯度权重的获取过程具体包括：
计算所述编码块中的各像素点的梯度值，并基于各像素点的梯度值计算所述编码块的梯度权重。
具体地，编码块中的各像素点的梯度值包括水平方向梯度和竖直方向梯度，水平方向梯度和竖直方向梯度的计算公式分别为：

其中，FH表示水平方向的梯度算子，Ii表示第i个编码块，尺寸大小为M×N，j代表编码块中的第j个像素点。
在得到各像素点的像素梯度值后，基于编码块中的各像素点的像素梯度值计算编码块的梯度值，其中，编码块的候选梯度值的计算公式可以为：
其中，表示候选梯度值。
在各编码块的候选梯度值后，对各候选梯度值进行归一化操作，以得到各编码块的梯度值，编码块的梯度值用于反映该编码块相对于所有编码块的重要程度。
其中，梯度值的计算公式为：
其中，Wi表示编码块的梯度值，Nomal(·)表示归一化操作，其具体计算方式为：
其中，C(·)操作表示将计算得到的最大值和最小值约束在[0,1]之间，即小于0的值置0，大于1的数置1，μx表示xi的均值，σx表示xi的标准差。
进一步，结构相似性失真度为编码块与其对应的重构编码块件的结构相似性失真度。在获取到各编码块的梯度权重及结构相似性失真度，可以基于各编码块的梯度权重及结构相似性失真度确定投影图像对应的感知失真度。其中，将由投影图像和若干参考投影图像形成的投影图像集，该投影图像集中的第k个目标投影图像中第i个纹理失真图像块的纹理图像块失真度可以表示为：
其中，Wk,i表示第k组图像组中第i个编码块的梯度权重，表示第k组图像组中第i个编码块相对于第i个重构编码块的SSIM失真度。
当然，值得说明的，在计算编码块的感知失真度时，投影图像中的几何投影图和纹理投影图分别计算各自对应的感知失真度，并且两者的计算过程相同，从而本实施例以投影图像对感知失真度的确定过程进行说明。
S30、根据投影图像的感知失真度确定所述待编码点云对应的率失真成本，并基于所述率失真成本编码所述投影图像。
具体地，率失真成本的目标函数为：
其中，J表示总的码率失真(RD)成本，D表示待编码点云和待编码单元对应的重构点云之间的失真差异，R表示编码比特。λ表示用于平衡失真和比特率的拉格朗日乘数。
基于此，根据投影图像的感知失真度确定所述待编码点云对应的率失真成本的目标函数可以表示为：
其中，DPPCM表示感知失真度。
由于V-PCC标准利用HEVC对几何视频和纹理视频分别进行编码，从而将几何视频和纹理视频的压缩看作两个独立的过程，并且纹理视频的失真只与纹理视频编码参数有关，几何视频的失真只与几何视频编码参数有关。由此，纹理投影图像的率失真成本的目标函数和几何投影图像的率失真成本的目标函数分别表示为：

其中，JG表示纹理投影图像的独立约束，JT表示几何投影图像的的独立约束，RT对于几何视频编码器来说是一个常数，RG对于纹理视频编码器。
此外，由于纹理投影图像的编码决策过程和几何投影图像的编码决策过程相同，这里仅对编码决策过程进行说明，不做纹理投影图像和几何投影图像说明，纹理投影图像和几何投影图像进步可以采用下述编码决策过程。
在一个实现方式中，所述根据投影图像的感知失真度确定所述待编码点云对应的码率失真成本具体包括：
将所述感知失真度中的结构相似性失真度转换为均方误差失真度，以得到所述投影图像对应的感知系数；
基于所述感知系数确定所述投影图像对应的感知拉格朗日乘数；
基于所述感知失真度和所述感知拉格朗日乘数确定所述待编码点云对应的码率失真成本。
具体地，在高分辨率量化近似下，编码过程将尽可能多地保留亮度信息。此外，对于不同的视频内容，即使在高QP设置下，和μ之间的皮尔逊相关系数(PCC)值也超过0.99。基于此，可以将it□像素的SSIM值重写为：

其中，表示像素级平方误差，Ei表示像素内容权重，M表示编码块的数量，C2是SSIM中的C2常数，εt表示当前编码块第t个位置的滤波系数，和表示编码块的编码前后在i-t处的方差。
进一步，为了减少计算量，编码块的SSIM失真度设置为16个子块的平均值，n表示编码块的子图像块，即：
已知HEVC中的编码单元存在以下关系
其中，ρn是与图像内容有关的线性模型参数，其中，前一帧的编码同位块的结果作为参考图像内容(第一帧除外)，Qn是子块的量化步长，通常以相同的数值应用于所有子块。
对于编码块来说，DMSE可以被表示为：


其中，表示第k个投影图像的编码块i的结构相似性失真度，表示第k个投影图像的编码块i的均方误差失真度，Φk,i表示第k个投影图像的编码块i的感知系数，blkn表示编码块i的第n个子图像块，ρn表示编码块i的第n个子图像块的线性模型参数，N表示子图像块的数量，Ei表示像素内容权重，第k个投影图像为由投影图像和若干参考投影图像形成的投影图像集中的一投影图像。
基于此，视频压缩导致的图像MSE数值损失经过适当的变换可以近似为领域图像的SSIM损失，即DPPCM可以表示为：
进一步，在获取到感知系数确定所述投影图像对应的感知拉格朗日乘数，其中，感知拉格朗日乘数的确定过程可以为：
当把基于PPCM的失真度量DPPCM应用于V-PCC中的RD目标函数时，需要对拉格朗日乘数和进行调整，以便在通道φ(其中，通道包括纹理通道和几何通道，这里以φ作为表示)的感知失真和比特率Rφ之间进行最佳折中。基于MSE的率失真成本在每一个编码块i的计算为：
其中，和λMSE是HEVC中的基于MSE的失真、比特率 和拉格朗日乘数，码率和失真之间的关系可以建模为
其中，是在帧间或帧内预测之后的编码残差的方差，α是比例常数。将对求偏导数，并在偏导数为零时可以得到：
求解上式可以得到最优的和为：
而一个视频的总比特率是所有块的比特数目总和，可以计算为：
其中，M是一帧/视频中的编码块数。
同样地，基于率编码失真的目标函数对DMSE取偏导数，并在偏导数为零时可以得到：
对上式进行求解可以得到
编码通道φ的总比特数计算为：
基于此，基于编码通道φ的总比特数可以确定感知拉格朗日参数。
在一个实现方式中，当感知拉格朗日乘数用于编码时，所述基于 所述感知系数确定所述投影图像对应的感知拉格朗日乘数具体包括：
基于所述投影图像的每个编码块的感知系数计算第一目标感知系数，并计算所述第一目标感知系数与所述编码块的感知系数的比值，以得到第一感知系数比值；
计算所述第一感知系数比值与所述投影图像的均方误差拉格朗日乘数的乘积，以得到所述编码块对应的感知拉格朗日乘数。
具体地，在V-PCC模型中，属性和几何视频是基于MSE的编码器编码的。，基于MSE的V-PCC的总比特率RMSE与基于感知失真度的V-PCC的总比特率Rφ相同，在不同的点云编码通道φ，即RMSE＝Rφ，由此可以得到λφ和λMSE之间的关系为：
由此，基于公式(1)和公式(2)，以得到模式决策的RD成本为：
因此，感知拉格朗日系数为：
在一个实现方式中，当感知拉格朗日乘数用于模式决策时，所述基于所述感知系数确定所述投影图像对应的感知拉格朗日乘数具体包括：
基于所述投影图像的每个编码块的感知系数计算第二目标感知系数，并计算第二目标感知系数与编码块的感知系数的平方根的比值，以得到第二感知系数比值；
计算所述第二感知系数比值与所述投影图像的所述平均绝对误 差拉格朗日乘数的平方根的乘积，以得到所述编码块对应的感知拉格朗日乘数。
在传统的基于MSE的视频编码器中，SAD/MAD被用作率失真优化中的失真项，以避免平方运算以保持低复杂度。RD成本被计算为：
其中和分别为HEVC中基于MAD的失真、比特率和拉格朗日乘数。ME的拉格朗日乘数同样，基于的关系，有
由此，RD成本为：
因此，基于感知失真度的感知拉格朗日乘数被更新为：
为了进一步说明本实施例的编码过程，V-PCC的编码标准将三维点云的几何特征和纹理特征分别映射为两路视频序列，分别记为几何投影图序列和纹理投影图序列，然后使用已有的HEVC/VVC或其他的视频编码器压缩动态点云的视频序列，其中，在编码过程中还需要产生一些元信息，如：占位图和辅助分块信息，用于说明这两个视频序列，其编码过程与现有过程相同，这里不做具体说明。
V-PCC中的点云投影面片(Patch)是一组信息的集合，包括点云的三维包围盒、相关的几何和纹理信息以及进行三维重构时所需的图集信息。点云投影面片的划分过程是将每一帧点云从三维空间投影到给定的二维平面上，并尽可能少地将点云划分为边界平滑的面片块， 以使重构误差最小。其中，点云投影过程可以为首先计算出输入点云中每一点的法向量，然后按照法向量将整块点云聚类并投影到立方体6个面上形成初始聚类。接着，根据法向量及最邻近点的索引，通过不断迭代更新每个点的聚类索引，从而实现更精细的面片划分。最后，应用连通分量提取法提取连通的面片，并将它们合并成较大的面片块，从而得到最终的面片集合。由此，通过点云投影面片集合在二维平面上打包重组，将三维空间中的几何和纹理信息分别转换为二维图像，从而实现几何和纹理图像的生成和填充过程，以得到几何投影图序列和纹理投影图序列。
下面以已经获取到几何投影图序列和纹理投影图序列为前提进行说明，几何投影图序列和纹理投影图序列的编码过程包括：
a)进行几何投影图分析，获取几何投影图中的每个编码块的感知系数其中，除了第一个帧参考帧，基于当前编码块和前一帧的编码同位块获取的中的ρn参数；
b)基于感知系数确定感知拉格朗日乘数；
c)对当前的几何投影帧进行编码；如果所有的视频帧对应的几何投影图像均被编码，进入步骤d)进行纹理编码，否则，转到步骤a)进行下一帧的编码；
d)进行纹理投影图分析，获取几何投影图中的每个编码块的感知系数其中，除了第一个帧参考帧，基于当前编码块和前一帧的编码同位块获取的中的ρn参数；
e)基于感知系数确定感知拉格朗日乘数；
f)对当前纹理投影图进行编码，然后转到步骤d)对下一个帧纹理投影图进行编码，直到所有纹理投影图均被编码。
综上所述，本实施例提供了一种基于感知加权的点云率失真编码方法，所述方法包括获取待编码点云对应的投影图像以及所述投影模块对应的若干参考投影图像；获取投影图像及各参考投影图像的各编码块的梯度权重及结构相似性失真度，并基于各编码块的梯度权重及 结构相似性失真度计算投影图像的感知失真度；根据投影图像的感知失真度确定所述待编码点云对应的率失真成本，并基于所述率失真成本编码所述投影图像。本申请通过投影生成二维的投影图像，然后计算以编码块为单位确定每个投影图像对应的感知失真度，并基于感知损失度进行编码决策，提高了点云编码与视觉感知质量的匹配性，从而提高了编码效率。
进一步，为了说明本实施例的准确性，对Longdress和Andrew两个点云进行了统计分析。每个点云都被压缩了17个量化参数(QP)对，范围从20到32和27到42不等，每个点云都被投射到二维图像上并分解成400个块，即400×17个采样块。如图3和图4所示，采用R2来计算和之间的近似模型准确度，结果表示在属性和几何通道上平均R2为0.9732和0.9556，感知中的近似值是准确的。
为了验证所提出的编码效率，编码方法在V-PCC参考软件TMC2-10.0和相应的HEVC参考软件HM16.20-SCM8.8上实现。最新的V-PCC(TMC2-10.0+HM16.20-SCM8.8)被用来作为比较的锚定方法。此外，两个最先进的V-PCC的RDO，基于占用图的RDO，表示为"OC-RDO"，和基于EPM的RDO方法，表示为"EPM-RDO"，也被用作比较的基准。由于提出的编码方法可以应用于V-PCC的帧内和帧间编码，将V-PCC配置为几何和纹理编码的随机存取编码方式(RA)。符合V-PCC通用测试条件(CTC)的低(r1)到高(r5)的五个比特率点被选取测试。为了进行公平的比较，GraphSIM(独立于提议的PPCM)被用来测量不同编码方案的压缩点云的感知质量。然后，Bjontegaard Delta Bit Rate(BDBR)被用来比较所提出的方案和基准方案的RD性能。
表1显示了在被测试的6个DPC上的RD比较，其中压缩视频的视觉质量是用GraphSIM测量的，RT表示总比特率(Mbps)。可以看出，与V-PCC相比，OC-RDO可以实现0.34％到17.05％的比特率降低，平均为9.61％。EPM-RDO能够实现从-4.08％到5.61％，平均1.17％的 BDBR节省。至于所提出的PWRDO，它实现了BDBR从7.09％到22.79％的降低，与V-PCC相比，平均降低了13.52％，明显超过了OC-RDO和EPM-RDO这两个编码优化的对比方案。为了评估所提出的PWRDO的感知编码效率，五个不同的感知PCQA指标，包括D1、GraphSIM、MPED、SIAT_PCQA和提出的PPCM，被用来衡量不同方案中压缩DPC的视觉质量。然后，为每个PCQA计算BDBR，其中包括几何位、属性位和元数据位在内的总编码位的数量。负的BDBR意味着节省比特，而正的则表示与锚点相比，编码效率下降。表1显示了提出的PWRDO和基准RDO方案在五个不同的感知PCQA指标下的RD比较。这些比较结果表明，所提出的PWRDO在所有PCQA中都能稳定地获得较高的编码收益，这证明了所提出的PWRDO的有效性。
表1.所提方法与2种点云编码率失真优化方法在5种评价指标上的增益

基于上述基于感知加权的点云率失真编码方法，本实施例提供了一种基于感知加权的点云率失真编码系统，如图5所示，所述系统包括：
获取模块100，用于获取待编码点云对应的投影图像以及所述投影模块对应的若干参考投影图像，其中，所述投影图像和若干参考投影图像均包括几何投影图和纹理投影图；
计算模块200，用于获取投影图像及各参考投影图像的各编码块的梯度权重及结构相似性失真度，并基于各编码块的梯度权重及结构相似性失真度计算投影图像的感知失真度；
编码模块300，用于根据投影图像的感知失真度确定所述待编码 点云对应的码率失真成本，并基于所述码率失真成本编码所述投影图像。
基于上述基于感知加权的点云率失真编码方法，本实施例提供了一种计算机可读存储介质，所述计算机可读存储介质存储有一个或者多个程序，所述一个或者多个程序可被一个或者多个处理器执行，以实现如上述实施例所述的基于感知加权的点云率失真编码方法中的步骤。
基于上述基于感知加权的点云率失真编码方法，本申请还提供了一种终端设备，如图6所示，其包括至少一个处理器(processor)20；显示屏21；以及存储器(memory)22，还可以包括通信接口(Communications Interface)23和总线24。其中，处理器20、显示屏21、存储器22和通信接口23可以通过总线24完成相互间的通信。显示屏21设置为显示初始设置模式中预设的用户引导界面。通信接口23可以传输信息。处理器20可以调用存储器22中的逻辑指令，以执行上述实施例中的方法。
此外，上述的存储器22中的逻辑指令可以通过软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。
存储器22作为一种计算机可读存储介质，可设置为存储软件程序、计算机可执行程序，如本公开实施例中的方法对应的程序指令或模块。处理器20通过运行存储在存储器22中的软件程序、指令或模块，从而执行功能应用以及数据处理，即实现上述实施例中的方法。
存储器22可包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序；存储数据区可存储根据终端设备的使用所创建的数据等。此外，存储器22可以包括高速随机存取存储器，还可以包括非易失性存储器。例如，U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random  Access Memory，RAM)、磁碟或者光盘等多种可以存储程序代码的介质，也可以是暂态存储介质。
此外，上述存储介质以及终端设备中的多条指令处理器加载并执行的具体过程在上述方法中已经详细说明，在这里就不再一一陈述。最后应说明的是：以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (10)

  1. 一种基于感知加权的点云率失真编码方法，其特征在于，所述方法包括：
    获取待编码点云对应的投影图像以及所述投影模块对应的若干参考投影图像，其中，所述投影图像和若干参考投影图像均包括几何投影图和纹理投影图；
    获取投影图像及各参考投影图像的各编码块的梯度权重及结构相似性失真度，并基于各编码块的梯度权重及结构相似性失真度计算投影图像的感知失真度；
    根据投影图像的感知失真度确定所述待编码点云对应的率失真成本，并基于所述率失真成本编码所述投影图像。
  2. 根据权利要求1所述基于感知加权的点云率失真编码方法，其特征在于，所述参考投影图像为通过对投影图像进行下采样得到的，并且投影图像和若干参考投影图像的图像尺度互不相同。
  3. 根据权利要求1所述基于感知加权的点云率失真编码方法，其特征在于，所述梯度权重的获取过程具体包括：
    计算所述编码块中的各像素点的梯度值，并基于各像素点的梯度值计算所述编码块的梯度权重。
  4. 根据权利要求1所述基于感知加权的点云率失真编码方法，其特征在于，所述根据投影图像的感知失真度确定所述待编码点云对应的率失真成本具体包括：
    将所述感知失真度中的结构相似性失真度转换为均方误差失真度，以得到所述投影图像对应的感知系数；
    基于所述感知系数确定所述投影图像对应的感知拉格朗日乘数；
    基于所述感知失真度和所述感知拉格朗日乘数确定所述待编码点云对应的率失真成本。
  5. 根据权利要求4所述基于感知加权的点云率失真编码方法，其 特征在于，所述结构相似性失真度与所述均方误差失真度的对应关系为：

    其中，表示第k个目标投影图像的编码块i的结构相似性失真度，表示第k个目标投影图像的编码块i的均方误差失真度，φk,i表示第k个目标投影图像的编码块i的感知系数，blkn表示编码块i的第n个子图像块，ρn表示编码块i的第n个子图像块的线性模型参数，N表示子图像块的数量，Ei表示像素内容权重，第k个目标投影图像为由投影图像和若干参考投影图像形成的投影图像集中的一投影图像。
  6. 根据权利要求4所述基于感知加权的点云率失真编码方法，其特征在于，当感知拉格朗日乘数用于编码时，所述基于所述感知系数确定所述投影图像对应的感知拉格朗日乘数具体包括：
    基于所述投影图像的每个编码块的感知系数计算第一目标感知系数，并计算所述第一目标感知系数与所述编码块的感知系数的比值，以得到第一感知系数比值；
    计算所述第一感知系数比值与所述投影图像的均方误差拉格朗日乘数的乘积，以得到所述编码块对应的感知拉格朗日乘数。
  7. 根据权利要求4所述基于感知加权的点云率失真编码方法，其特征在于，当感知拉格朗日乘数用于模式决策时，所述基于所述感知系数确定所述投影图像对应的感知拉格朗日乘数具体包括：
    基于所述投影图像的每个编码块的感知系数计算第二目标感知系数，并计算第二目标感知系数与编码块的感知系数的平方根的比值，以得到第二感知系数比值；
    计算所述第二感知系数比值与所述投影图像的所述均方误差拉 格朗日乘数的平方根的乘积，以得到所述编码块对应的感知拉格朗日乘数。
  8. 一种基于感知加权的点云率失真编码系统，其特征在于，所述系统包括：
    获取模块，用于获取待编码点云对应的投影图像以及所述投影模块对应的若干参考投影图像，其中，所述投影图像和若干参考投影图像均包括几何投影图和纹理投影图；
    计算模块，用于获取投影图像及各参考投影图像的各编码块的梯度权重及结构相似性失真度，并基于各编码块的梯度权重及结构相似性失真度计算投影图像的感知失真度；
    编码模块，用于根据投影图像的感知失真度确定所述待编码点云对应的率失真成本，并基于所述率失真成本编码所述投影图像。
  9. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有一个或者多个程序，所述一个或者多个程序可被一个或者多个处理器执行，以实现如权利要求1-7任意一项所述的基于感知加权的点云率失真编码方法中的步骤。
  10. 一种终端设备，其特征在于，包括：处理器、存储器及通信总线；所述存储器上存储有可被所述处理器执行的计算机可读程序；
    所述通信总线实现处理器和存储器之间的连接通信；
    所述处理器执行所述计算机可读程序时实现如权利要求1-7任意一项所述的基于感知加权的点云率失真编码方法中的步骤。
PCT/CN2023/137731 2023-02-22 2023-12-09 一种基于感知加权的点云率失真编码方法及相关装置 Ceased WO2024174679A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310183247.0 2023-02-22
CN202310183247.0A CN117376568A (zh) 2023-02-22 2023-02-22 一种基于感知加权的点云率失真编码方法及相关装置

Publications (1)

Publication Number Publication Date
WO2024174679A1 true WO2024174679A1 (zh) 2024-08-29

Family

ID=89395281

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/137731 Ceased WO2024174679A1 (zh) 2023-02-22 2023-12-09 一种基于感知加权的点云率失真编码方法及相关装置

Country Status (2)

Country Link
CN (1) CN117376568A (zh)
WO (1) WO2024174679A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119743620A (zh) * 2024-12-27 2025-04-01 中国科学院深圳先进技术研究院 一种彩色点云jnd阈值预测方法、装置、设备以及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012171113A1 (en) * 2011-06-14 2012-12-20 Zhou Wang Method and system for structural similarity based rate-distortion optimization for perceptual video coding
CN103607590A (zh) * 2013-11-28 2014-02-26 北京邮电大学 基于结构相似度的高效视频编码感知率失真优化方法
CN103918271A (zh) * 2011-06-01 2014-07-09 王舟 基于结构相似度的感知视频编码方法和系统
CN112601082A (zh) * 2020-11-30 2021-04-02 南京邮电大学 一种基于视频的快速动态点云编码方法及系统
CN115018753A (zh) * 2021-03-05 2022-09-06 中国科学院深圳先进技术研究院 基于投影的点云质量评价方法、装置、设备和存储介质
CN115631126A (zh) * 2022-01-25 2023-01-20 青岛大学 一种基于信息量加权投影的点云质量评估方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11113845B2 (en) * 2017-09-18 2021-09-07 Apple Inc. Point cloud compression using non-cubic projections and masks
CN112637608B (zh) * 2020-12-18 2022-11-29 南京邮电大学 一种动态点云几何信息压缩率失真优化方法及装置
CN115314701B (zh) * 2022-01-19 2025-03-14 青岛大学 一种v-pcc编码点云感知质量评估方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103918271A (zh) * 2011-06-01 2014-07-09 王舟 基于结构相似度的感知视频编码方法和系统
WO2012171113A1 (en) * 2011-06-14 2012-12-20 Zhou Wang Method and system for structural similarity based rate-distortion optimization for perceptual video coding
CN103607590A (zh) * 2013-11-28 2014-02-26 北京邮电大学 基于结构相似度的高效视频编码感知率失真优化方法
CN112601082A (zh) * 2020-11-30 2021-04-02 南京邮电大学 一种基于视频的快速动态点云编码方法及系统
CN115018753A (zh) * 2021-03-05 2022-09-06 中国科学院深圳先进技术研究院 基于投影的点云质量评价方法、装置、设备和存储介质
CN115631126A (zh) * 2022-01-25 2023-01-20 青岛大学 一种基于信息量加权投影的点云质量评估方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
M. KARCZEWICZ; X. WANG (QUALCOMM): "Intra Frame Rate Control Based on SATD", 104. MPEG MEETING; 22-4-2013 - 26-4-2013; INCHEON; (MOTION PICTURE EXPERT GROUP OR ISO/IEC JTC1/SC29/WG11), 25 April 2013 (2013-04-25), XP030057247 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119743620A (zh) * 2024-12-27 2025-04-01 中国科学院深圳先进技术研究院 一种彩色点云jnd阈值预测方法、装置、设备以及存储介质

Also Published As

Publication number Publication date
CN117376568A (zh) 2024-01-09

Similar Documents

Publication Publication Date Title
TWI864248B (zh) 點雲品質評估方法、編碼器、解碼器及儲存媒介
Boyce et al. MPEG immersive video coding standard
Dziembowski et al. IV-PSNR—The objective quality metric for immersive video applications
CN110163246B (zh) 基于卷积神经网络的单目光场图像无监督深度估计方法
Sun et al. CVIQD: Subjective quality evaluation of compressed virtual reality images
Li et al. Spherical domain rate-distortion optimization for 360-degree video coding
He et al. Towards a colored point cloud quality assessment method using colored texture and curvature projection
JP2024505796A (ja) 点群復号化方法、点群符号化方法、復号器及び符号器
WO2024169360A1 (zh) 基于投影和多尺度特征的点云质量评价方法及相关装置
US20230298216A1 (en) Predictive coding of boundary geometry information for mesh compression
CN102413353A (zh) 立体视频编码过程的多视点视频和深度图的码率分配方法
CN107040771B (zh) 一种针对全景视频的编码优化方法
Zhou et al. Reduced-reference stereoscopic image quality assessment based on view and disparity zero-watermarks
CN118823558B (zh) 一种基于图卷积神经网络的三维点云质量预测方法
CN108900838A (zh) 一种基于hdr-vdp-2失真准则的率失真优化方法
Shao et al. No-reference view synthesis quality prediction for 3-D videos based on color–depth interactions
Hu et al. An adaptive two-layer light field compression scheme using GNN-based reconstruction
WO2024174679A1 (zh) 一种基于感知加权的点云率失真编码方法及相关装置
Zerman et al. A spherical light field database for immersive telecommunication and telepresence applications
WO2023159428A1 (zh) 编码方法、编码器以及存储介质
CN112218088B (zh) 一种图像与视频压缩方法
Liu et al. Cracks-suppression perceptual geometry coding for dynamic point clouds
CN106791772A (zh) 基于绘制的最大可容忍深度失真计算方法
US20230306641A1 (en) Mesh geometry coding
Freitas et al. A comparative assessment of implicit and explicit plenoptic scene representations

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23923842

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23923842

Country of ref document: EP

Kind code of ref document: A1