WO2023123467A1 - 编解码方法、码流、编码器、解码器以及存储介质 - Google Patents

编解码方法、码流、编码器、解码器以及存储介质 Download PDF

Info

Publication number
WO2023123467A1
WO2023123467A1 PCT/CN2021/143943 CN2021143943W WO2023123467A1 WO 2023123467 A1 WO2023123467 A1 WO 2023123467A1 CN 2021143943 W CN2021143943 W CN 2021143943W WO 2023123467 A1 WO2023123467 A1 WO 2023123467A1
Authority
WO
WIPO (PCT)
Prior art keywords
point cloud
value
filter
component
identification information
Prior art date
Application number
PCT/CN2021/143943
Other languages
English (en)
French (fr)
Inventor
元辉
邢金睿
王璐
王婷婷
李明
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Priority to PCT/CN2021/143943 priority Critical patent/WO2023123467A1/zh
Publication of WO2023123467A1 publication Critical patent/WO2023123467A1/zh

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/117Filters, e.g. for pre-processing or post-processing
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding

Definitions

  • the embodiments of the present application relate to the technical field of video encoding and decoding, and in particular, relate to an encoding and decoding method, a code stream, an encoder, a decoder, and a storage medium.
  • V-PCC Video-based Point Cloud Compression
  • the quality of the geometric attributes of the reconstructed point cloud can be better improved after the Wiener filtering is performed on the video frames of the V-PCC codec framework; however, the existing solutions are not ideal for the improvement of the color attributes, which affects the entire The quality of the point cloud.
  • Embodiments of the present application provide a codec method, code stream, coder, decoder, and storage medium, which can not only improve the quality of point clouds, but also improve compression efficiency, thereby saving code rates and improving codec performance.
  • the embodiment of the present application provides a decoding method applied to a decoder, and the method includes:
  • the code stream is decoded to determine the filter coefficient corresponding to the attribute information
  • the filtered point cloud corresponding to the reconstructed point cloud is determined.
  • the embodiment of the present application provides an encoding method applied to an encoder, and the method includes:
  • the filter identification information indicates that the attribute information of the reconstructed point cloud is to be filtered
  • the filter coefficients corresponding to the filter identification information and the attribute information are encoded, and the obtained encoded bits are written into the code stream.
  • the embodiment of the present application provides a code stream, which is generated by bit coding according to the information to be encoded; wherein the information to be encoded includes at least one of the following: residual value of geometric video frame, texture The residual value of the video frame, the filter identification information and the filter coefficient.
  • an encoder which includes a first determination unit, a first filtering unit, and an encoding unit; wherein,
  • the first determination unit is configured to determine the initial point cloud and the reconstructed point cloud corresponding to the initial point cloud; and determine the filter coefficient of the attribute information in the reconstructed point cloud according to the initial point cloud and the reconstructed point cloud;
  • the first filtering unit is configured to determine the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient of the attribute information
  • the first determination unit is further configured to determine the filter identification information according to the reconstructed point cloud and the filtered point cloud;
  • the encoding unit is configured to encode the filter coefficients corresponding to the filter identification information and the attribute information if the filter identification information indicates that the attribute information of the reconstructed point cloud is to be filtered, and write the obtained encoded bits into the code stream.
  • the embodiment of the present application provides an encoder, where the encoder includes a first memory and a first processor; wherein,
  • a first memory for storing a computer program capable of running on the first processor
  • the first processor is configured to execute the method as described in the second aspect when running the computer program.
  • the embodiment of the present application provides a decoder, which includes a decoding unit and a second filtering unit; wherein,
  • the decoding unit is configured to decode the code stream and determine the filtering identification information; wherein the filtering identification information is used to determine whether to perform filtering processing on the reconstructed point cloud corresponding to the video frame to be processed; and if the filtering identification information indicates that the attribute information of the reconstructed point cloud is performed For filtering processing, the code stream is decoded, and the filter coefficient corresponding to the attribute information is determined;
  • the second filtering unit is configured to determine the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient corresponding to the attribute information.
  • the embodiment of the present application provides a decoder, where the decoder includes a second memory and a second processor; wherein,
  • a second memory for storing a computer program capable of running on the second processor
  • the second processor is configured to execute the method as described in the first aspect when running the computer program.
  • an embodiment of the present application provides a computer storage medium, the computer storage medium stores a computer program, and when the computer program is executed, the method as described in the first aspect is implemented, or the method as described in the second aspect is implemented. .
  • the embodiment of the present application provides a codec method, code stream, encoder, decoder, and storage medium.
  • the initial point cloud and the reconstruction point cloud corresponding to the initial point cloud are determined; according to the initial point cloud and the reconstruction point cloud Cloud, determine the filter coefficient of the attribute information in the reconstructed point cloud; determine the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient of the attribute information; determine the filtered identification information according to the reconstructed point cloud and the filtered point cloud; if the filtered identification information Instructing to filter the attribute information of the reconstructed point cloud, encode the filter coefficients corresponding to the filter identification information and the attribute information, and write the obtained encoded bits into the code stream.
  • the filter identification information is determined by decoding the code stream; wherein, the filter identification information is used to determine whether to filter the reconstructed point cloud corresponding to the video frame to be processed; if the filter identification information indicates that the attribute information of the reconstructed point cloud is For filtering processing, the code stream is decoded, and the filter coefficient corresponding to the attribute information is determined; according to the filter coefficient corresponding to the attribute information, the filtered point cloud corresponding to the reconstructed point cloud is determined.
  • the encoder uses the initial point cloud and the reconstructed point cloud to calculate the filter coefficients for filtering processing, and after determining to filter the reconstructed point cloud, the corresponding filter coefficients are passed to the decoder; correspondingly, the decoder It can be directly decoded to obtain the filter coefficient, and then use the filter coefficient to filter the reconstructed point cloud, thereby realizing the optimization of the reconstructed point cloud and improving the quality of the point cloud; in addition, the encoder uses the initial point cloud and the reconstructed point cloud to calculate When performing filter coefficients for filtering processing, it also solves the problem that the midpoint of the initial point cloud and the midpoint of the reconstructed point cloud are difficult to match, and by filtering the attribute information of the reconstructed point cloud, not only the quality of the reconstructed point cloud is enhanced, Moreover, the quality of the video frame can also be improved, further improving the compression efficiency, thereby saving the bit rate, and improving the encoding and decoding efficiency.
  • Fig. 1 is a composition framework schematic diagram of a kind of V-PCC coder
  • Fig. 2 is a composition framework schematic diagram of a kind of V-PCC decoder
  • FIG. 3 is a first schematic flowchart of a decoding method provided by an embodiment of the present application.
  • FIG. 4 is a second schematic flow diagram of a decoding method provided by an embodiment of the present application.
  • FIG. 5 is a first schematic flow diagram of an encoding method provided by an embodiment of the present application.
  • FIG. 6 is a schematic flow diagram II of an encoding method provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of an overall flow of encoding and decoding provided by an embodiment of the present application.
  • FIG. 8 is a schematic flow chart of an encoding end filtering process provided by an embodiment of the present application.
  • FIG. 9 is a schematic flowchart of a filtering process at a decoding end provided by an embodiment of the present application.
  • Figure 10 is a schematic diagram of test results based on CTC_C2 test conditions provided by the embodiment of the present application.
  • FIG. 11 is a schematic diagram of the composition and structure of an encoder provided in an embodiment of the present application.
  • FIG. 12 is a schematic diagram of a specific hardware structure of an encoder provided in an embodiment of the present application.
  • FIG. 13 is a schematic structural diagram of a decoder provided in an embodiment of the present application.
  • FIG. 14 is a schematic diagram of a specific hardware structure of a decoder provided in an embodiment of the present application.
  • FIG. 15 is a schematic diagram of the composition and structure of an encoding and decoding system provided by an embodiment of the present application.
  • references to “some embodiments” describe a subset of all possible embodiments, but it is understood that “some embodiments” may be the same subset or a different subset of all possible embodiments, and Can be combined with each other without conflict.
  • first ⁇ second ⁇ third involved in the embodiment of the present application is only used to distinguish similar objects, and does not represent a specific ordering of objects. Understandably, “first ⁇ second ⁇ The specific order or sequence of "third” may be interchanged where permitted so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein.
  • G-PCC Geometry-based Point Cloud Compression
  • Video-based point cloud compression Video-based Point Cloud Compression, V-PCC or VPCC
  • Peak Signal to Noise Ratio Peak Signal to Noise Ratio
  • MMSE Minimum Mean Squared Error
  • Luminance component (Luminance, L or Y)
  • Red chroma component Chroma red, Cr
  • Point cloud is a three-dimensional representation of the surface of an object.
  • the point cloud (data) on the surface of an object can be collected through acquisition equipment such as photoelectric radar, laser radar, laser scanner, and multi-view camera.
  • Point cloud refers to a collection of massive three-dimensional points, and the points in the point cloud can include point location information and point attribute information.
  • the point position information may be three-dimensional coordinate information of the point.
  • the location information of a point may also be referred to as geometric information of a point.
  • the attribute information of a point may include color information and/or reflectivity and the like.
  • color information may be information on any color space.
  • color information may be RGB information. Wherein, R represents red (Red, R), G represents green (Green, G), and B represents blue (Blue, B).
  • the color information may be luminance chrominance (YCbCr, YUV) information. Among them, Y represents brightness, Cb(U) represents blue chroma, and Cr(V) represents red chroma.
  • the points in the point cloud can include the three-dimensional coordinate information of the point and the laser reflection intensity (reflectance) of the point.
  • the points in the point cloud may include the three-dimensional coordinate information of the point and the color information of the point.
  • the points in the point cloud may include the three-dimensional coordinate information of the point, the laser reflection intensity (reflectance) of the point, and the color information of the point.
  • Point clouds can be divided into the following ways:
  • the first type of static point cloud that is, the object is stationary, and the device for obtaining the point cloud is also stationary;
  • the second type of dynamic point cloud the object is moving, but the device for obtaining the point cloud is still;
  • the third type of dynamic acquisition of point clouds the equipment for acquiring point clouds is in motion.
  • point cloud For example, according to the purpose of point cloud, it is divided into two categories:
  • Category 1 Machine perception point cloud, which can be used in scenarios such as autonomous navigation systems, real-time inspection systems, geographic information systems, visual sorting robots, and emergency rescue robots;
  • Category 2 Human eyes perceive point clouds, which can be used in point cloud application scenarios such as digital cultural heritage, free viewpoint broadcasting, 3D immersive communication, and 3D immersive interaction.
  • the point cloud is a collection of massive points, storing the point cloud will not only consume a large amount of memory, but also is not conducive to transmission, and there is no such a large bandwidth to support the direct transmission of the point cloud at the network layer without compression. Therefore, it is necessary to Cloud for compression.
  • the point cloud coding framework that can compress the point cloud can be the G-PCC codec framework or the V-PCC codec framework provided by the Moving Picture Experts Group (MPEG), or it can be audio and video coding
  • MPEG Moving Picture Experts Group
  • AVS-PCC codec framework provided by the standard (Audio Video Standard, AVS).
  • the G-PCC codec framework can be used to compress the first type of static point cloud and the third type of dynamically acquired point cloud
  • the V-PCC codec framework can be used to compress the second type of dynamic point cloud.
  • the description here mainly focuses on the V-PCC codec framework.
  • FIG. 1 is a schematic diagram of a composition framework of a V-PCC encoder. As shown in Figure 1, this V-PCC encoder is applied to a point cloud encoder. In the V-PCC coding framework, for the input point cloud frame to be coded, it is first divided by 3D blocks (patch) to obtain multiple patch blocks.
  • patch encapsulation can be carried out, and then combined with the input point cloud frame to generate a geometric image, and then image filling operation is performed on the geometric image, and video compression can be performed after geometric filling to obtain a geometric sub-stream; in addition, according to these patch information And input the point cloud frame to generate the attribute image, and also need to consider the geometric smoothing operation; then perform the image filling operation on the attribute image, and perform video compression after the attribute filling to obtain the attribute sub-stream; perform patch sequence compression on the patch information , to obtain the patch sub-stream; and after obtaining the occupancy map (Occupancy map), video compression is performed on the occupancy map to obtain the occupancy map sub-stream; these four sub-streams are sent to the multiplexer for processing, and the compressed stream.
  • occupancy map occupancy map
  • FIG. 2 is a schematic diagram of a composition framework of a V-PCC decoder. As shown in Fig. 2, this V-PCC decoder is applied to point cloud decoder.
  • the high-level syntax information can be sequence parameter set (Sequence Parameter Sets, SPS) information, after the SPS information is analyzed, the obtained SPS analysis information can be sent to the subsequent steps;
  • the patch sub-code stream is passed through After the patch sequence is decompressed, the patch information can be obtained; after the occupancy image substream is decompressed by the video, the occupancy image can be obtained; after the geometry substream is decompressed by the video, the geometric image can be obtained; the attribute subcode After the stream is processed by video decompression, the attribute image can be obtained;
  • FIG. 1 and Figure 2 show a schematic diagram of the framework of V-PCC encoding and decoding. Specifically, there are two smoothing processes of the point cloud in V-PCC: the first time is to smooth the reconstruction frame after it is generated; the second time is to smooth the geometry after the reconstruction point cloud is generated.
  • V-PCC For texture video frames, V-PCC aims at the problem that non-adjacent patch blocks in three-dimensional space may be very close in the image, and the patch-based encoder may confuse adjacent pixels to cause reconstruction artifacts.
  • the color smoothing algorithm uses the occupancy map to find the pixels corresponding to the patch boundary points in the reconstructed frame, and then uses the median filter for smoothing.
  • V-PCC proposes a boundary filtering algorithm to solve the problem that the patch may be discontinuous in the reconstructed point cloud, find the pixels corresponding to the boundary points of the patch, and locally change the depth value of the patch edge.
  • the V-PCC codec framework only performs the smoothing operation of the image or point cloud, so the quality of the reconstructed point cloud still has a lot of room for improvement.
  • the method of Wiener filtering for video frames in the V-PCC codec framework can better improve the quality of the geometric attributes of the reconstructed point cloud. But this method is not ideal for the promotion of color attributes.
  • the texture video frame is not processed, the BD-TotalRate of the reconstructed point cloud color attribute can only have a gain of about -0.1% by processing the geometric video frame; if the texture video frame is also Wiener Filtering, although the PSNR performance of the video frame can be improved by nearly 0.2dB, but the BD-TotalRate of the final reconstructed point cloud color attribute only has a gain in a few point cloud sequences, and for most point cloud sequences, the index does not decrease but rises . Therefore, for the reconstruction of point cloud color attributes, a more suitable codec method is needed to improve the quality and compression performance of point cloud.
  • the embodiment of the present application provides an encoding and decoding method.
  • the encoder uses the initial point cloud and the reconstructed point cloud to calculate the filter coefficients for filtering processing, and only after it is determined to perform filtering processing on the reconstructed point cloud, the corresponding The filter coefficients are passed to the decoder; correspondingly, the decoder can directly decode to obtain the filter coefficients, and then use the filter coefficients to filter the reconstructed point cloud, thereby realizing the optimization of the reconstructed point cloud and improving the quality of the point cloud; in addition, When the encoder uses the initial point cloud and the reconstructed point cloud to calculate the filter coefficients for filtering processing, it also solves the problem that the midpoint of the initial point cloud and the reconstructed point cloud are difficult to match, and through the attribute information of the reconstructed point cloud Filtering processing can not only enhance the quality of the reconstructed point cloud, especially the quality of the reconstructed point cloud attribute information with geometric loss, so that the quality of the video frame can also be improved, which further
  • FIG. 3 shows a first schematic flowchart of a decoding method provided in an embodiment of the present application.
  • the method may include:
  • S301 Decode the code stream, and determine filtering identification information; wherein, the filtering identification information is used to determine whether to perform filtering processing on the reconstructed point cloud corresponding to the video frame to be processed.
  • the decoding method described in the embodiment of the present application specifically refers to the point cloud decoding method, which can be applied to the point cloud decoder in V-PCC (in the embodiment of the present application, it may be simply referred to as "decoder").
  • the decoding method described in the embodiment of the present application is not limited to the field of V-PCC.
  • all point clouds with attribute information distortion can be used.
  • V-PCC is taken as an example, because the implementation on V-PCC can make it take full advantage of the processing of geometry and attributes, and the overall quality of point cloud is improved, with better results.
  • the decoder may decode the code stream first, so as to determine the filter identification information.
  • the filtering identification information may be used to determine whether to perform filtering processing on the reconstructed point cloud corresponding to the video frame to be processed.
  • the filter identification information may also be used to determine which color component or color components in the reconstructed point cloud are to be filtered.
  • a point in the initial point cloud when decoding the point, it can be used as a point to be decoded in the initial point cloud, and there are Multiple decoded points.
  • a point in the initial point cloud corresponds to a geometric information and an attribute information; wherein, the geometric information represents the spatial position of the point, and the attribute information represents the attribute value of the point (such as color component value).
  • the attribute information may include color components, specifically color information of any color space.
  • the attribute information may be color information in RGB space, may also be color information in YUV space, may also be color information in YCbCr space, etc., which are not specifically limited in this embodiment of the present application.
  • the color components may include a first color component, a second color component, and a third color component.
  • the first color component, the second color component and the third color component can be in turn: R component, G component, B component; if the color component conforms to the YUV color space, then the first color component
  • the component, the second color component and the third color component can be in turn: Y component, U component, V component; if the color component conforms to the YCbCr color space, then the first color component, the second color component and the third color component can be in turn: : Y component, Cb component, Cr component, but not specifically limited.
  • the attribute information of the point may be a color component, reflectivity or other attributes, which is not specifically limited in the embodiment of the present application.
  • the video frames to be processed may include texture video frames and geometric video frames.
  • the method may also include:
  • a reconstructed point cloud is obtained.
  • the code stream may include: an auxiliary information sub-code stream, a geometry sub-code stream, a texture sub-code stream, and the like.
  • the auxiliary information sub-stream is used for decoding to obtain auxiliary information related to the occupancy map and patch
  • the geometry sub-stream is used for decoding to obtain the residual value of the geometric video frame
  • the texture sub-stream is used for decoding to obtain the residual value of the texture video frame.
  • the filter identification information and corresponding filter coefficients described in the embodiment of the present application are obtained by parsing the texture sub-stream.
  • the residual value of the texture video frame is determined; after determining the predicted value of the texture video frame, the texture can be determined according to the predicted value of the texture video frame and the residual value of the texture video frame.
  • the decoder can reconstruct the point cloud accordingly. Get the reconstructed point cloud.
  • the reconstructed point cloud here can be the reconstructed point cloud output after decoding, or it can be used as a reference for subsequent point cloud decoding; in addition, the reconstructed point cloud here can not only be used in the prediction loop, that is, as an inloop filter, It can be used as a reference for decoding subsequent point clouds; it can also be used outside the prediction loop, that is, as a post filter, not as a reference for decoding subsequent point clouds; this embodiment of the application does not specifically limit it.
  • the filtering identification information may include filtering identification information of the component to be processed of the attribute information.
  • the decoding code stream and determining the filter identification information may include:
  • determining the filter identification information may specifically include:
  • the value of the filter identification information of the component to be processed is the first value, then it is determined to perform filtering processing on the component to be processed of the attribute information of the reconstructed point cloud;
  • the value of the filtering identification information of the component to be processed is the second value, it is determined not to perform filtering processing on the component to be processed of the attribute information of the reconstructed point cloud.
  • the component to be processed is a color component
  • the component to be processed is a U component
  • the value of the filter identification information corresponding to the U component is the first value
  • the value of the filter identification information corresponding to the U component is the second value
  • it can be determined that the U component of the attribute information of the reconstructed point cloud does not need to be filtered.
  • the filter identification information corresponding to each color component may be used to indicate whether to perform filtering processing on the color component.
  • the decoding the code stream to determine the filter identification information of the component to be processed may include: decoding the code stream to determine the second color component First filtering identification information and second filtering identification information.
  • the first filter identification information is used to indicate whether to filter the second color component of the attribute information of the reconstructed point cloud
  • the second filter identification information is used to indicate whether to filter the third color component of the attribute information of the reconstructed point cloud. filter processing.
  • the Y component, the U component and the V component can all be filtered, but due to the large fluctuation of the Y component relative to the U component and the V component, resulting in The filtering effect is unstable, and it will increase the code stream overhead and affect the overall compression efficiency; therefore, this embodiment of the present application mainly performs filtering enhancement processing for the U component and the V component.
  • the second color component is a U component
  • the third color component is a V component
  • the second color component is a V component
  • the third color component is a U component.
  • the decoding code stream and determining the first filtering identification information may include:
  • the value of the first filtering identification information is the first value, it is determined that the first filtering identification information indicates that the second color component of the attribute information of the reconstructed point cloud is filtered;
  • the first filtering identification information indicates that the second color component of the attribute information of the reconstructed point cloud is not to be filtered.
  • the decoding code stream and determining the second filtering identification information may include:
  • the second filtering identification information indicates that the third color component of the attribute information of the reconstructed point cloud is filtered
  • the second filtering identification information indicates that the third color component of the attribute information of the reconstructed point cloud is not to be filtered.
  • the filter identification information corresponding to a certain color component is the first value, then it can be instructed to perform The filter processing of the color component; if the filter identification information corresponding to a certain color component is the second value, it may indicate that the filter processing of the color component is not performed on the attribute information of the reconstructed point cloud.
  • the embodiment of the present application can perform one kind of filtering process on the Y component and another kind of filtering process on the UV component, so that the geometry and attribute quality of the reconstructed point cloud have greater quality.
  • the improvement of the compression efficiency is further increased.
  • the decoding code stream to determine the filter identification information of the components to be processed may include: a decoding code A stream is used to determine the first filter identification information, the second filter identification information, and the third filter identification information.
  • the first filter identification information is used to indicate whether to perform the first filtering process on the second color component of the attribute information of the reconstructed point cloud
  • the second filter identification information is used to indicate whether to perform the third color component of the attribute information of the reconstructed point cloud
  • the second filtering process is performed on the component
  • the third filtering identification information is used to indicate whether to perform the third filtering process on the first color component of the attribute information of the reconstructed point cloud.
  • the first filtering identification information if the value of the first filtering identification information is the first value, it is determined that the first filtering identification information indicates that the attribute information of the reconstructed point cloud If the value of the first filter identification information is the second value, it is determined that the first filter identification information indicates not to perform the first filter processing on the second color component of the attribute information of the reconstructed point cloud .
  • the second filtering identification information For the second filtering identification information, if the value of the second filtering identification information is the first value, it is determined that the second filtering identification information indicates that the second filtering process is performed on the third color component of the attribute information of the reconstructed point cloud; if If the value of the second filtering identification information is the second value, it is determined that the second filtering identification information indicates that the second filtering process is not performed on the third color component of the attribute information of the reconstructed point cloud.
  • the third filtering identification information For the third filtering identification information, if the value of the third filtering identification information is the first value, it is determined that the third filtering identification information indicates that the third filtering process is performed on the first color component of the attribute information of the reconstructed point cloud; if If the value of the third filtering identification information is the second value, it is determined that the third filtering identification information indicates that the third filtering process is not performed on the first color component of the attribute information of the reconstructed point cloud.
  • the filter corresponding to the third filtering process is different from the filters corresponding to the first filtering process and the second filtering process.
  • the filter corresponding to the first filtering process and the second filtering process are the same, but the filter is different from the filter corresponding to the third filtering process; or, the filter corresponding to the first filtering process and the filter corresponding to the second filtering process
  • the filter corresponding to the filter and the third filtering process are all different; or, even the filter corresponding to the first filtering process and the third filtering process may be the same, but the filter is different from the filter corresponding to the second filtering process, etc. etc., which are not specifically limited in the embodiments of the present application.
  • the filter corresponding to the first filtering process and the second filtering process can be a Wiener filter
  • the filter corresponding to the third filtering process can be an adaptive filter (the working principle is the same as that of the Wiener filter, here The order and number of filters can be adaptively selected according to neighborhood differences); or, the filter corresponding to the third filtering process can also be a Wiener filter, but it corresponds to the first filtering process and the second filtering process The order of the Wiener filter is different, etc.; this embodiment of the present application does not make any limitation thereto.
  • decoding the code stream to determine the filter identification information of the component to be processed may be to obtain the first filter identification information, the second second filter identification information and the first filter identification information once by decoding the code stream.
  • the third filtering identification information and the like are not limited in any way.
  • the first value and the second value are different, and the first value and the second value may be in the form of parameters or numbers.
  • the identification information corresponding to these color components may be a parameter written in a profile (profile), or a value of a flag (flag), which is not specifically limited here.
  • an enable flag bit (enable_flag) and a disable flag bit (disable_flag).
  • the value of the enabled flag bit is the first value
  • the value of the non-enabled flag bit is the second value
  • the first value can be set to 1
  • the second value can be set to 0; or, the first value may also be set to true, and the second value may also be set to false; however, this embodiment of the present application does not specifically limit it.
  • the decoder determines the filter identification information by decoding the code stream, if the filter identification information indicates that the attribute information of the reconstructed point cloud should be filtered, then it can further determine the Processing filter coefficients.
  • the filter used for filtering processing may be an adaptive filter, such as a neural network-based filter, a Wiener filter, etc., but it is not specifically limited.
  • the filter coefficients described in the embodiments of the present application may be used for Wiener filtering, that is, the filter coefficients are coefficients for Wiener filtering processing.
  • the Wiener filter is a linear filter whose optimal criterion is to minimize the mean square error. Under certain constraints, the square of the difference between its output and a given function (usually called the expected output) reaches the minimum, and through mathematical operations, it can finally become a problem of solving the Toblitz equation.
  • the Wiener filter is also known as the least squares filter or the least squares filter.
  • the filter coefficient may refer to a filter coefficient corresponding to the component to be processed.
  • decoding the code stream and determining the filter coefficient corresponding to the attribute information may include: If the value of the filter identification information of the processing component is the first value, the code stream is decoded to determine the filter coefficient corresponding to the component to be processed of the attribute information.
  • the color component when the component to be processed is a color component, the color component here may include at least one of the first color component, the second color component, and the third color component.
  • the following takes the YUV color space as an example. Due to the large volatility of the Y component relative to the U and V components, the filtering effect is unstable, and it will increase the code stream overhead and affect the overall compression efficiency; assume that if the quality of the Y component is not enhanced. , only perform quality enhancement processing on the U component and the V component.
  • the first filter identification information is used to indicate whether to filter the second color component (U component) of the attribute information of the reconstructed point cloud
  • the second filter identification information is used to indicate whether to filter the third color component (U component) of the attribute information of the reconstructed point cloud.
  • the color component (V component) is filtered.
  • decoding the code stream and determining the filter coefficient corresponding to the attribute information may include:
  • the first filter identification information indicates that the second color component of the attribute information of the reconstructed point cloud is to be filtered, then decode the code stream and determine the filter coefficient corresponding to the second color component;
  • the code stream is decoded to determine a filter coefficient corresponding to the third color component.
  • the first filter identification information if the first filter identification information indicates that the second color component of the attribute information of the reconstructed point cloud needs to be filtered, then it is necessary to continue decoding the code stream in order to obtain the second color The filter coefficient corresponding to the component; if the first filter identification information indicates that the second color component of the attribute information of the reconstructed point cloud is not to be filtered, it is no longer necessary to execute the decoding code stream and determine the filter coefficient corresponding to the first color component. .
  • the second filter identification information For the second filter identification information, if the second filter identification information indicates that the third color component of the attribute information of the reconstructed point cloud is to be filtered, then it is necessary to continue decoding the code stream in order to obtain the filter corresponding to the third color component Coefficient; if the second filter identification information indicates that the third color component of the attribute information of the reconstructed point cloud is not to be filtered, it is no longer necessary to execute the step of decoding the code stream and determining the filter coefficient corresponding to the third color component.
  • the first filter identification information and the second filter identification information may be in no sequence or in a sequence during the decoding process.
  • the first filtering identification information may be obtained by decoding first, and then the second filtering identification information may be obtained by decoding, but this embodiment of the present application does not specifically limit it.
  • the method may include:
  • the code stream is decoded to determine the filter coefficient corresponding to the second color component; and the code stream is decoded to determine the second filter identification information;
  • the code stream is decoded to determine the filter coefficient corresponding to the third color component.
  • the method may include:
  • the code stream is decoded to determine the filter coefficient corresponding to the second color component; and the code stream is decoded to determine the second filter identification information;
  • the step of decoding the code stream and determining the filter coefficient corresponding to the third color component is not performed.
  • the method may include:
  • the decoding code stream is not executed, and the step of determining the filter coefficient corresponding to the second color component is not performed; the code stream is directly decoded, and the second color component is determined. filter identification information;
  • the code stream is decoded to determine a filter coefficient corresponding to the third color component.
  • the method may include :
  • the decoding code stream is not executed, and the step of determining the filter coefficient corresponding to the second color component is not performed; the code stream is directly decoded, and the second color component is determined. filter identification information;
  • the step of decoding the code stream and determining the filter coefficient corresponding to the third color component is not performed.
  • the method may further include: if the filter identification information indicates that the attribute information of the reconstructed point cloud is not to be filtered, then the step of determining the filter coefficient corresponding to the attribute information is not performed by decoding the code stream.
  • the decoder after the decoder decodes the code stream and determines the first filter identification information, if the first filter identification information indicates that the second color component of the attribute information of the reconstructed point cloud is to be filtered, then The decoder can continue to decode the code stream to obtain the filter coefficient corresponding to the second color component; but if the first filter identification information indicates that the second color component of the attribute information of the reconstructed point cloud is not to be filtered, then the decoder no longer needs to decode The filter coefficient corresponding to the second color component is obtained, but the next color component is decoded, that is, the code stream is decoded, and the second filter identification information is determined.
  • the decoder can continue to decode the code stream to obtain the filter coefficient corresponding to the third color component; but if the second filter identification If the information indicates that the third color component of the attribute information of the reconstructed point cloud is not to be filtered, then the decoder no longer needs to decode to obtain the filter coefficient corresponding to the third color component.
  • the filter coefficient can be further used to filter the reconstructed point cloud Processing, so that the filtered point cloud corresponding to the reconstructed point cloud can be obtained.
  • each color component of the reconstructed point cloud can be filtered separately.
  • the filter coefficient corresponding to the Y component and the filter coefficient corresponding to the U component can be respectively determined.
  • the filter coefficient corresponding to the V component the filter coefficient corresponding to the embodiment of the present application.
  • the embodiment of the present application considers that the Y component is more fluctuant than the U component and the V component, the filtering effect is unstable, and the code stream overhead will be increased to affect the overall compression efficiency, and the quality enhancement process for the Y component may not be involved. That is to say, the embodiment of the present application may only be used to determine the filter coefficient corresponding to the U component and/or the V component.
  • the component to be processed is a color component, and the component to be processed includes the second color component and the third color component as an example.
  • the reconstruction point is determined according to the filter coefficient corresponding to the attribute information
  • the filtered point cloud corresponding to the cloud can include:
  • a filtered point cloud corresponding to the reconstructed point cloud is determined according to the filter coefficient corresponding to the second color component and/or the filter coefficient corresponding to the third color component.
  • the filter coefficient corresponding to the second color component and the filter coefficient corresponding to the third color component can be obtained by decoding the code stream, and then use the filter coefficient corresponding to the second color component to correspond to the third color component The filter coefficients determine the filtered point cloud.
  • the first filter identification information indicates that the second color component of the attribute information of the reconstructed point cloud is filtered
  • the second filter identification information indicates that the third color component of the attribute information of the reconstructed point cloud is not processed. If the color component is filtered, then only the filter coefficient corresponding to the second color component is obtained by decoding the code stream, and then the filtered point cloud is determined using the filter coefficient corresponding to the second color component.
  • the first filter identification information indicates that the second color component of the attribute information of the reconstructed point cloud is not to be filtered
  • the second filter identification information indicates that the third color component of the attribute information of the reconstructed point cloud is If the color component is filtered, then only the filter coefficient corresponding to the third color component is obtained by decoding the code stream, and then the filtered point cloud is determined using the filter coefficient corresponding to the third color component.
  • determining the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient corresponding to the attribute information may include:
  • K target points corresponding to the first point in the reconstructed point cloud; wherein, the K target points include the first point and (K-1) neighbor points adjacent to the first point, K is an integer greater than 1, and the first point represents any point in the reconstructed point cloud;
  • the filtered point cloud is determined according to the filter value of the attribute information of the at least one point.
  • the filter coefficient corresponding to the attribute information can be further used to filter the reconstructed point cloud.
  • the attribute information is filtered, so that the filtered point cloud corresponding to the reconstructed point cloud can be obtained.
  • the filter coefficient corresponding to the attribute information determine the filtered point cloud corresponding to the reconstructed point cloud.
  • the filter coefficient can be used to filter the K target points corresponding to the first point in the reconstructed point cloud, and then determine the reconstruction point
  • the cloud corresponds to the filtered point cloud.
  • the K target points include the first point and (K-1) neighbor points adjacent to the first point, K is an integer greater than 1, and the first point represents any point in the reconstructed point cloud.
  • the attribute information takes the color component as an example, and each color component of the attribute information of the reconstructed point cloud can be filtered separately.
  • the Y The filter coefficient corresponding to the component, the filter coefficient corresponding to the U component, and at least one item of the filter coefficient corresponding to the V component. These filter coefficients together constitute the filter coefficients for reconstructing the point cloud.
  • the filter coefficient can be determined by a K-order filter; in this way, when the decoder performs filtering, it can perform a KNN search for each point in the reconstructed point cloud to determine the K target points corresponding to the point , and then use the filter coefficient corresponding to the point to perform filtering to obtain the filter value of the attribute information of the point.
  • the determination of K target points corresponding to the first point in the reconstructed point cloud may include:
  • the K nearest neighbor search method is used to search for preset candidate points in the reconstructed point cloud
  • the K-nearest neighbor search method can be used to search for a preset value of candidate points in the reconstructed point cloud, calculate the distance between the first point and these candidate points, and then from these candidate points Select (K-1) neighbor points closest to the first point; determine the first point itself and (K-1) neighbor points as the final K target points.
  • the filter coefficient when using the filter coefficient to filter the K target points corresponding to the first point in the reconstructed point cloud, it may include: using the filter coefficient corresponding to the color component The vector filters the K target points corresponding to the first point in the reconstructed point cloud to obtain the filtered value of the color component of the first point in the reconstructed point cloud; according to the filtered value of the color component of the point in the reconstructed point cloud, the filtered point cloud.
  • the filter value of the color component of each point in the reconstructed point cloud can be determined according to the filter coefficient vector corresponding to the color component and the second attribute parameter; then based on the filter value of the color component of each point in the reconstructed point cloud, Get the filtered point cloud.
  • the order of the Wiener filter and the attribute information of the K target points corresponding to the midpoint of the reconstructed point cloud can be firstly Reconstruct the value to determine the second attribute parameter corresponding to the attribute information; then determine the filter value of the attribute information according to the filter coefficient and the second attribute parameter; finally, the filtered point cloud can be obtained based on the filter value of the attribute information.
  • FIG. 4 shows a second schematic flowchart of a decoding method provided by an embodiment of the present application. As shown in Figure 4, the method may include:
  • S401 Decode a code stream, and determine first filtering identification information.
  • S403 Decode the code stream, and determine the second filtering identification information.
  • S405 Determine a filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient corresponding to the second color component.
  • S406 Determine a filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient corresponding to the third color component.
  • S407 Determine a filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient corresponding to the second color component and the filter coefficient corresponding to the third color component.
  • step S401 if the first filter identification information indicates that the second color component of the attribute information of the reconstructed point cloud is to be filtered, then step S402 is executed first, and then step S403 is executed; if the first filter identification information Indicate not to filter the second color component of the attribute information of the reconstructed point cloud, then directly execute step S403. Then after step S403, if the second filter identification information indicates that the third color component of the attribute information of the reconstructed point cloud is filtered, then first perform step S404, and then perform step S406 or S407; if the second filter identification information indicates that it is not correct The third color component of the attribute information of the reconstructed point cloud is filtered, and then step S405 is directly executed.
  • steps S405, S406, and S407 are executed in one of the three steps, specifically according to whether it is necessary to filter the first color component and/or the second color component on the reconstructed point cloud definite.
  • step S405 may be selected to be performed.
  • the first filter identification information indicates that the second color component of the attribute information of the reconstructed point cloud is to be filtered
  • the second filter identification information indicates that the third color component of the attribute information of the reconstructed point cloud is not to be filtered.
  • the code stream can only obtain the filter coefficient corresponding to the second color component.
  • the determining the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient corresponding to the second color component may include: according to the filter coefficient corresponding to the second color component, Determine the filter value of the second color component corresponding to the midpoint of the reconstructed point cloud; use the filter value of the second color component to replace the reconstruction value of the second color component corresponding to the midpoint of the reconstructed point cloud to obtain a filtered point cloud corresponding to the reconstructed point cloud .
  • the reconstruction value of the second color component of the K target points corresponding to the middle point of the reconstructed point cloud can be based on the order of the filter , determine the second attribute parameter corresponding to the second color component; then the filter value of the second color component can be determined according to the filter coefficient corresponding to the second color component and the second attribute parameter; then it can be based on the filter value of the second color component value overwrites the reconstructed value of the second color component in the reconstructed point cloud, resulting in a filtered point cloud.
  • step S406 may be selected to be performed.
  • the first filter identification information indicates that the third color component of the attribute information of the reconstructed point cloud is not to be filtered
  • the second filter identification information indicates that the third color component of the attribute information of the reconstructed point cloud is filtered.
  • the code stream can only obtain the filter coefficient corresponding to the third color component.
  • the determining the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient corresponding to the third color component may include: according to the filter coefficient corresponding to the third color component, Determine the filter value of the third color component corresponding to the midpoint of the reconstructed point cloud; use the filter value of the third color component to replace the reconstruction value of the third color component corresponding to the midpoint of the reconstructed point cloud, and obtain the filtered point cloud corresponding to the reconstructed point cloud .
  • the reconstruction value of the third color component of the K target points corresponding to the center point of the reconstructed point cloud can be based on the order of the filter first , determine the second attribute parameter corresponding to the third color component; then, the filter value of the third color component can be determined according to the filter coefficient corresponding to the third color component and the second attribute parameter; then, according to the filter value of the third color component The value overwrites the reconstructed value of the third color component in the reconstructed point cloud to obtain the filtered point cloud.
  • step S407 may be selected to be performed.
  • the first filter identification information indicates that the second color component of the attribute information of the reconstructed point cloud is filtered
  • the second filter identification information also indicates that the third color component of the attribute information of the reconstructed point cloud is filtered. Only the filter coefficient corresponding to the second color component and the filter coefficient corresponding to the third color component can be obtained by decoding the code stream.
  • the determining the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient corresponding to the second color component and the filter coefficient corresponding to the third color component may include: according to The filter coefficient corresponding to the second color component and the filter coefficient corresponding to the third color component determine the filter value of the second color component corresponding to the reconstruction point cloud midpoint and the filter value of the third color component; use the filter value of the second color component The reconstructed value of the second color component and the reconstructed value of the third color component corresponding to the middle point of the reconstructed point cloud are replaced with the filtered value of the third color component to obtain a filtered point cloud corresponding to the reconstructed point cloud.
  • the reconstruction value of a certain color component of the K target points corresponding to the center point of the reconstructed point cloud can be based on the order of the filter , determine the second attribute parameter corresponding to the color component; then, the filter value of the color component can be determined according to the filter coefficient corresponding to the color component and the second attribute parameter; thus, after obtaining the filter value of the second color component and the third After the filtered values of the color components, the reconstructed values of the two color components in the reconstructed point cloud can be overlaid according to the filtered values of the second color component and the filtered values of the third color component, so as to obtain the filtered point cloud.
  • the attribute information is the color component of the YUV space
  • it can be based on the reconstruction value of the second color component or the third color component (such as U component or V component) in the attribute information , combined with the order of the filter, determine the second attribute parameter (denoted by P(n,k)) of the attribute information, that is, the second attribute parameter represents the attribute information of the K target points corresponding to the reconstruction point cloud midpoint Rebuild value.
  • the filter type may be used to indicate a filter order, and/or a filter shape, and/or a filter dimension.
  • the shape of the filter includes rhombus, rectangle, etc.
  • the dimensionality of the filter includes one-dimensional, two-dimensional or even more dimensions.
  • the filter may be a Wiener filter, an adaptive filter, or the like.
  • filter types can correspond to filters of different orders, for example, filters with orders of 12, 16, 20, 32, and 128; different types can also correspond to filters of different dimensions, for example, one-dimensional filtering devices, two-dimensional filters, etc., are not specifically limited here. That is to say, if a 16-order filter needs to be determined, then 16 points can be used to determine a 16-order asymmetric filter, or an 8-order one-dimensional symmetric filter, or other quantities (such as more special two-dimensional, three-dimensional filter, etc.), and the filter is not specifically limited here.
  • the value of the order may be 16, 20, 32, 64 and so on. Exemplarily, the value of the order is 20, but it is not specifically limited here.
  • the Wiener filter when using the Wiener filter to filter the reconstructed point cloud, the reconstructed point cloud and the filter coefficients can be input into the Wiener filter at the same time, that is, the Wiener filter
  • the input of the filter is the filter coefficient and the reconstructed point cloud.
  • the filter processing of the reconstructed point cloud can be completed based on the filter coefficient, and the corresponding filtered point cloud can be obtained.
  • the filter coefficients are obtained based on the original point cloud and the reconstructed point cloud. Therefore, applying the filter coefficients to the reconstructed point cloud can restore the original point cloud to the maximum extent.
  • the filter coefficients taking the color component of the YUV space in the point cloud sequence as an example, assuming that the order of the filter is K and the point cloud sequence is n, use the matrix P(n,k) to represent the reconstruction
  • the reconstruction value constitutes the second property parameter.
  • the filter coefficient corresponding to the U component (represented by H(k)) is applied to the reconstructed point cloud, that is, the second attribute parameter P( n, k), so that the color filter value R(n) under the U component can be obtained.
  • the embodiment of the present application can also perform quality enhancement processing on the Y component.
  • Wiener filtering of different orders can be performed on the Y component, and an adaptive filter can also be used on the Y component.
  • the working principle of the adaptive filter Similar to Wiener filtering, the order and number of filters can be adaptively selected according to neighborhood differences.
  • the Y component can perform Wiener filtering on the image and video frames by using a neighborhood difference-based method.
  • V-PCC will generate video frames of geometry and texture, encode the video frames, and then reconstruct the point cloud from the video frames, it is proposed to perform Wiener filtering on the video frames, so that the quality of the video frames can be improved, and the reconstruction can be realized indirectly. Quality enhancements for point clouds.
  • the Y component it is mainly for the geometric video frame, using the characteristics of the patch edge and the background in the video frame to change sharply, and the neighborhood difference between the pixels is large, to classify the pixels in it according to the neighborhood difference, each Neighborhood difference points are then combined, and different combinations use different Wiener filters to obtain different filter coefficients, pass the filter coefficients to the decoder, and use the filter coefficients to post-process the generated video frames, so that Effectively improve the geometric quality and subjective effect of point clouds.
  • the decoder may also use the filtered point cloud to cover the original reconstructed point cloud.
  • the quality of the filtered point cloud obtained after filtering processing is significantly enhanced; therefore, after obtaining the filtered point cloud, you can use
  • the filtered point cloud covers the original reconstructed point cloud, so as to realize the whole codec and quality enhancement operation.
  • this method can also Including: if the color components of the filtered point cloud midpoints do not conform to the RGB color space, performing color space conversion on the filtered point cloud so that the color components of the filtered point cloud midpoints conform to the RGB color space.
  • the decoder can skip the filter The wave processing flow is applied according to the reconstructed point cloud obtained by the original program, that is, the reconstructed point cloud is no longer updated.
  • the embodiment of the present application can set the value of the order to be equal to 20, so as to ensure the filtering effect and reduce the time complexity to a certain extent ; but not specifically limited.
  • the encoder-side Wiener filter considers the neighborhood difference value, due to the large changes in the attribute values at the point cloud contour or boundary, it may be difficult for a single filter to cover the filtering of the entire point cloud, resulting in room for improvement in the effect.
  • the embodiment of the present application can classify according to the neighborhood difference value, and apply different filters for different categories, such as setting multiple Wiener filters with different orders.
  • the encoder sends the decoder Pass multiple sets of filter coefficients; correspondingly, multiple sets of filter coefficients will be decoded on the decoder side to filter the reconstructed point cloud, so as to realize adaptive selection of filters for filtering to achieve better results.
  • the embodiment of this application proposes a V-PCC point cloud color attribute quality enhancement technology based on Wiener filtering.
  • Wiener filtering is performed on the reconstructed point cloud to achieve enhancement.
  • the purpose of point cloud geometry and color attribute quality in terms of geometry, the Wiener filtering method based on the neighborhood difference of geometric video frames can be used; and in terms of color, the embodiment of the present application proposes to directly perform color attribute Wiener filtering on the reconstructed point cloud at the codec end, instead of Process the reconstructed video frame; specifically, the decoding method proposed in steps S301-S303 can be used.
  • the embodiment of the present application is aimed at the V-PCC geometrically lossy and attribute lossy encoding manner.
  • Wiener filtering of color attributes is implemented for geometrically lossy reconstructed point clouds, which improves the quality of color attributes without any impact on geometric reconstruction and filtering processes.
  • geometric filtering and color filtering are applied to V-PCC at the same time; among them, the geometric aspect uses Wiener filtering technology based on neighborhood differences for geometrically reconstructed video frames, and the color aspect is used for reconstructing point clouds. Directly perform Wiener filtering operation; finally obtain a point cloud with improved geometric and color quality, and the compression performance is greatly improved.
  • Wiener filter proposed in the embodiment of the present application can be used in the prediction loop, that is, as an inloop filter, and can be used as a reference for decoding subsequent point clouds; it can also be outside the prediction loop, that is, as an Post filter used, not used as a reference for decoding subsequent point clouds.
  • This embodiment of the present application does not specifically limit it.
  • the Wiener filter proposed in the embodiment of the present application is an in-loop filter
  • the parameter information indicating the filtering process such as Write the filter identification information into the code stream, and also need to write the filter coefficient into the code stream; correspondingly, after determining not to perform filtering processing, you can choose not to write the parameter information indicating the filtering process, such as the filter identification information into the code stream , and do not write the filter coefficients into the code stream.
  • the Wiener filter proposed in the embodiment of the present application is a post-processing filter
  • the filter coefficient corresponding to the filter is located in a separate auxiliary information data unit (for example, supplementary enhancement information SEI)
  • a separate auxiliary information data unit for example, supplementary enhancement information SEI
  • the decoder does not obtain the supplementary enhancement information, it will not Reconstruct the point cloud for filtering.
  • the filter coefficient corresponding to the filter and other information are contained in an auxiliary information data unit.
  • the parameter information indicating the filtering process such as the filter identification information
  • the filter coefficients it is also necessary to write the filter coefficients into the code stream; correspondingly, after determining not to perform filtering processing, you can choose not to write the parameter information indicating the filtering process, such as the filter identification information, into the code stream, and at the same time do not write the filter coefficient The coefficients are written into the code stream.
  • one or more parts of the reconstructed point cloud can be selected to be filtered, that is, the control range of the filtered identification information can be the entire reconstructed point cloud, or the reconstructed point cloud A certain part of it is not specifically limited here.
  • the embodiment of the present application can also purely expect to improve the quality of the color attribute, and no longer filter the geometric video frame at this time, so that a lower BD-TotalRate and better compression performance. What's more, the embodiment of the present application can also filter the texture video frame and the color attribute of the reconstructed point cloud at the same time, and better results may also be obtained.
  • This embodiment provides a decoding method, which determines the filter identification information by decoding the code stream; wherein, the filter identification information is used to determine whether to perform filtering processing on the reconstruction point cloud corresponding to the video frame to be processed; if the filter identification information indicates that the reconstruction point cloud The attribute information of the cloud is filtered, and the code stream is decoded to determine the filter coefficient corresponding to the attribute information; according to the filter coefficient corresponding to the attribute information, the filtered point cloud corresponding to the reconstructed point cloud is determined.
  • the decoder can directly decode to obtain the filter coefficients, and then use the filter coefficients to filter the reconstructed point cloud, thereby realizing the optimization of the reconstructed point cloud and improving the quality of the point cloud; in addition, the encoder uses the initial point cloud and the reconstruction
  • the point cloud calculation is used to filter the filter coefficients, it also solves the problem that the midpoint of the initial point cloud and the midpoint of the reconstructed point cloud are difficult to match, and by filtering the attribute information of the reconstructed point cloud, it not only realizes the reconstruction of the point cloud
  • the quality of the video is enhanced, and the quality of the video frame can also be improved, which further improves the compression efficiency, thereby saving the bit rate and improving the encoding and decoding efficiency.
  • FIG. 5 shows a first schematic flowchart of an encoding method provided in an embodiment of the present application. As shown in Figure 5, the method may include:
  • S501 Determine an initial point cloud and a reconstructed point cloud corresponding to the initial point cloud.
  • the encoding method described in the embodiment of the present application specifically refers to the point cloud encoding method, which can be applied to the point cloud encoder in V-PCC (in the embodiment of the present application, it may be simply referred to as "encoder").
  • the encoding method described in the embodiment of the present application is not limited to the field of V-PCC.
  • all point clouds with attribute information distortion can be used.
  • V-PCC is taken as an example, because the implementation on V-PCC can make it take full advantage of the processing of geometry and attributes, and the overall quality of point cloud is improved, with better results.
  • the initial point cloud and the reconstructed point cloud corresponding to the initial point cloud may be determined first, and then the initial point cloud and the reconstructed point cloud are used to calculate the filter coefficient.
  • the initial point cloud and the reconstructed point cloud are used to calculate the filter coefficient.
  • a point in the initial point cloud when encoding the point, it can be used as a point to be encoded in the initial point cloud, and there are multiple encoded points around the point.
  • a point in the initial point cloud corresponds to a geometric information and an attribute information; wherein, the geometric information represents the spatial position of the point, and the attribute information represents the attribute value of the point (such as color component value).
  • the attribute information may include color components, specifically color information of any color space.
  • the attribute information may be color information in RGB space, may also be color information in YUV space, may also be color information in YCbCr space, etc., which are not specifically limited in this embodiment of the present application.
  • the color component may include at least one of the following: a first color component, a second color component, and a third color component.
  • the first color component, the second color component and the third color component can be in turn: R component, G component, B component; if the color component conforms to the YUV color space, then the first color component
  • the component, the second color component and the third color component can be in turn: Y component, U component, V component; if the color component conforms to the YCbCr color space, then the first color component, the second color component and the third color component can be in turn: : Y component, Cb component, Cr component, but not specifically limited.
  • the attribute information of the point may be a color component, reflectivity or other attributes, which is not specifically limited in the embodiment of the present application.
  • the predicted value and residual value of the attribute information of the point can be determined first, and then the predicted value and residual value can be used to further calculate and obtain the point The reconstruction value of the attribute information in order to construct the reconstruction point cloud.
  • the geometric information and attribute information of multiple target neighbor points of the point can be used, combined with the geometric information of the point for the The attribute information of the point is predicted to obtain the corresponding predicted value, and then the corresponding reconstruction value can be determined.
  • the point can be used as the nearest neighbor of the point in the subsequent LOD, and the reconstruction value of the attribute information of the point can be used to continue to perform subsequent reconstruction.
  • Points are used to predict attributes, so that a reconstructed point cloud can be constructed.
  • the encoder will reconstruct the point cloud accordingly.
  • the reconstructed point cloud here can be the reconstructed point cloud output after decoding, or it can be used as a reference for decoding the subsequent point cloud; in addition, the reconstructed point cloud here can not only be in the prediction loop, that is Used as an inloop filter, it can be used as a reference for decoding subsequent point clouds; it can also be used outside the prediction loop, that is, as a post filter, not as a reference for decoding subsequent point clouds; this embodiment of the application does not specifically limit this.
  • the embodiment of the present application deals with the case of lossy coding. That is, the case of encoding with lossy geometry and lossy attributes. Therefore, in some embodiments, for S501, the determining the initial point cloud and the reconstructed point cloud corresponding to the initial point cloud may include:
  • performing geometry recovery processing on the first reconstructed point cloud to obtain the reconstructed point cloud may include:
  • Scale processing is performed on the second reconstructed point cloud to obtain the reconstructed point cloud; wherein, the reconstructed point cloud has the same size and the same geometric position as the initial point cloud.
  • the geometric coordinate preprocessing of the reconstructed point cloud is first performed, and after geometric compensation processing, the geometric coordinates of each point are divided according to the reconstruction process by using the scaling scale in the configuration parameters.
  • the reconstructed point cloud can be restored to the same size as the bounding box of the original point cloud through this process, and the geometric position is the same (that is, the offset of the geometric coordinates is eliminated).
  • the embodiment of the present application first needs to find a point with the same coordinates of each point in the reconstructed point cloud in the initial point cloud as the corresponding point of the point, and then pass the minimum The mean square error criterion is used for filtering. Therefore, the embodiment of the present application also proposes a method for finding corresponding points in the reconstructed point cloud after geometric lossy encoding, and then performs Wiener filtering of color components on this basis.
  • the method may also include:
  • the preset search method is a K-nearest neighbor search method for searching a first constant number of points.
  • said searching the initial point cloud for neighboring points according to the preset search method, and determining the corresponding points of the points in the reconstructed point cloud in the initial point cloud may include:
  • the first constant value points are searched in the input point cloud using a preset search method
  • the determining the corresponding point of the current point according to the distance value may include:
  • the corresponding point of the current point is determined according to the multiple points corresponding to the minimum distance value.
  • determining the corresponding point of the current point according to the multiple points corresponding to the minimum distance value may include: randomly selecting a point from the multiple points corresponding to the minimum distance value, and The selected point is determined as the corresponding point of the current point; or, a plurality of points corresponding to the minimum distance value are fused, and the fused point is determined as the corresponding point of the current point; this is not specifically limited here.
  • the KNN search method in order to find the corresponding relationship between each point of the reconstructed point cloud and the point in the initial point cloud, the KNN search method is used to traverse each point in the reconstructed point cloud,
  • the search range is the entire initial point cloud, and the purpose is to find the nearest neighbor point of each point, and use this as the real point corresponding to the reconstruction point to construct the real initial point cloud (ie, "matching point cloud").
  • this method cannot find the corresponding points in the strict sense, due to the large number of points in the entire point cloud and the small difference in neighborhoods, it is completely feasible to use the nearest neighbor point as the corresponding point, and the effect of this scheme for dense point clouds better.
  • the embodiment of the present application when the embodiment of the present application performs a KNN search on each point of the reconstructed point cloud, it will find up to 5 neighbor points of the current point in the original point cloud, and calculate each point in turn The distance from the current point, save the nearest neighbor point, if there are multiple nearest neighbor points with the same distance, these nearest neighbor points will be fused into one point, that is, the average value of the color component values of these points will be taken as the corresponding point of the current point The true value of the color component of the point (nearest neighbor matching point).
  • the number and coordinates of the reconstructed point cloud points and even the bounding box of the entire point cloud obtained at this time have a large change.
  • you want to be more accurate To find out the corresponding relationship between points, then firstly carry out the preprocessing of the geometric coordinates of the reconstructed point cloud, after the geometric compensation processing, use the scaling scale in the configuration parameters, and then divide the geometric coordinates of each point according to the reconstruction process With this zoom scale, the reconstructed point cloud can be restored to a size equivalent to the original point cloud through this process and the offset of the geometric coordinates can be eliminated, so that the geometric position is the same.
  • a point cloud with the same number of points as the reconstructed point cloud and one-to-one correspondence with each point can be obtained, which can be input to the filter together with the reconstructed point cloud as the real initial point cloud.
  • S502 Determine a filter coefficient of attribute information in the reconstructed point cloud according to the initial point cloud and the reconstructed point cloud.
  • the points in the initial point cloud and the points in the reconstructed point cloud have a corresponding relationship.
  • the K target points corresponding to the points in the reconstructed point cloud may include the current point and (K-1) neighboring points adjacent to the current point, where K is an integer greater than 1.
  • the determining the filter coefficients according to the initial point cloud and the reconstructed point cloud may include: determining the filter coefficients according to K target points corresponding to the midpoints of the initial point cloud and the reconstructed point cloud.
  • determining K target points corresponding to points in the reconstructed point cloud may include:
  • the K nearest neighbor search method is used to search for preset candidate points in the reconstructed point cloud
  • the first point can be any point in the reconstructed point cloud.
  • the (K-1) neighbor points closest to the first point; the first point itself and the (K-1) neighbor points are determined as the final K target points.
  • the filter here may be an adaptive filter, such as a neural network-based filter, a Wiener filter, etc., which are not specifically limited here.
  • the function of the Wiener filter is mainly to calculate filter coefficients, and at the same time determine whether the quality of the point cloud after the Wiener filter is improved. That is to say, the filter coefficients described in the embodiments of the present application may be coefficients processed by the Wiener filter, that is, the filter coefficients are coefficients output by the Wiener filter.
  • the Wiener Filter (Wiener Filter) is a linear filter whose optimal criterion is to minimize the mean square error. Under certain constraints, the square of the difference between its output and a given function (usually called the expected output) reaches the minimum, and through mathematical operations, it can finally become a problem of solving the Toblitz equation.
  • the Wiener filter is also known as the least squares filter or the least squares filter.
  • Wiener filtering is a method of filtering a signal mixed with noise by using the correlation characteristics and spectral characteristics of a stationary random process, and it is currently one of the basic filtering methods.
  • the specific algorithm of Wiener filtering is as follows:
  • the output when the filter length or order is M is as follows,
  • M is the length or order of the filter
  • y(n) is the output signal
  • x(n) is a column of (mixed with noise) input signals.
  • the error between the known signal and the desired signal can be calculated, represented by e(n), as follows,
  • the Wiener filter takes the minimum mean square error as the objective function, so the objective function is expressed as follows,
  • the reciprocal of the objective function to the coefficient should be 0, namely:
  • Rxd and Rxx are the correlation matrix of the input signal and the expected signal and the autocorrelation matrix of the input signal respectively. Therefore, according to the optimal solution calculation according to the Wiener-Hough equation, the filter coefficient H can be obtained:
  • the noisy signal and the expected signal are required; and in the embodiment of the present application, for the point cloud encoding and decoding framework, the two correspond to the reconstructed point cloud and the initial point cloud respectively, so that it can be determined Input to the Wiener filter.
  • the determination of the filter coefficients of the attribute information in the reconstructed point cloud according to the initial point cloud and the reconstructed point cloud may include: using the Wiener filter to input the initial point cloud and the reconstructed point cloud to the Wiener filter The calculation is carried out in the filter, and the filter coefficient is output.
  • the initial point cloud and the reconstructed point cloud are input into the Wiener filter for calculation, and the filter coefficients are output, which may include:
  • filter coefficients of the attribute information are determined.
  • the determining the filter coefficient of the attribute information based on the first attribute parameter and the second attribute parameter may include:
  • Coefficients are calculated according to the cross-correlation parameters and auto-correlation parameters to obtain the filter coefficients of the attribute information.
  • the first attribute parameter and the second attribute parameter of the attribute information can be determined based on the original value and the reconstructed value of the attribute information (such as Y component, U component, V component, etc.), and then can be determined according to the first
  • the attribute parameter and the second attribute parameter further determine the filter coefficient used for filtering processing.
  • the attribute information here may specifically refer to the U component and the V component.
  • the first attribute parameter is determined according to the original value of the attribute information of at least one point in the initial point cloud; the second attribute parameter is reconstructed according to the attribute information of K target points corresponding to at least one point in the reconstructed point cloud The value is determined, where the K target points include the current point and (K-1) neighbor points adjacent to the current point.
  • the order of the Wiener filter is also involved.
  • the order of the Wiener filter may be set equal to M.
  • the values of M and K may be the same or different, which are not specifically limited here.
  • the filter type may be used to indicate a filter order, and/or a filter shape, and/or a filter dimension.
  • the shape of the filter includes rhombus, rectangle, etc.
  • the dimensionality of the filter includes one-dimensional, two-dimensional or even more dimensions.
  • the filter may be a Wiener filter, an adaptive filter, or the like.
  • filter types can correspond to filters of different orders, for example, filters with order M of 12, 16, 20, 32, 128; different types can also correspond to filters of different dimensions, for example, one-dimensional A filter, a two-dimensional filter, etc. are not specifically limited here. That is to say, if a 16-order filter needs to be determined, then 16 points can be used to determine a 16-order asymmetric filter, or an 8-order one-dimensional symmetric filter, or other quantities (such as more special two-dimensional, three-dimensional filter, etc.), and the filter is not specifically limited here.
  • the first attribute of the attribute information can be determined according to the original value of the attribute information of the points in the initial point cloud parameter; at the same time, the second attribute parameter of the attribute information can be determined according to the reconstruction value of the attribute information of the K target points corresponding to the center point of the reconstructed point cloud; finally, it can be determined based on the first attribute parameter and the second attribute parameter The filter coefficient of attribute information.
  • the corresponding first attribute parameter and second attribute parameter are calculated for the attribute information (such as U component, V component), and then the filter coefficient of the attribute information can be determined by using the first attribute parameter and the second attribute parameter.
  • the cross-correlation parameters corresponding to the attribute information can be determined first according to the first attribute parameter and the second attribute parameter corresponding to the attribute information; at the same time, according to The second attribute parameter determines the autocorrelation parameter corresponding to the attribute information; then the filter coefficient of the attribute information can be determined based on the cross-correlation parameter and the autocorrelation parameter; finally, all color components (such as U component and V component) can be traversed, and each color component can be obtained.
  • the filter coefficient of the color component in order to obtain the filtered point cloud.
  • the order of the filter is K, that is, using the K target points corresponding to each point in the reconstructed point cloud to Calculate the optimal coefficient, that is, calculate the filter coefficient.
  • the K target points may include the point itself and (K-1) neighboring points adjacent to it.
  • the point cloud sequence is n
  • the first attribute parameter composed of the original value of use the matrix P(n,k) to represent the reconstruction value of the K target points corresponding to all points in the reconstructed point cloud under the same color component (such as the U component), that is, P(n , k) is the second attribute parameter composed of the reconstructed values of the U components of K target points corresponding to all points in the reconstructed point cloud.
  • the autocorrelation parameter A(k,k) is obtained, as shown below.
  • the filter coefficient H(k) under the U component that is, the filter coefficient vector of the K-order Wiener filter under the U component.
  • the filter coefficient under the V component can also be determined according to the above method.
  • the process of determining the filter coefficients it is based on the YUV color space; if the initial point cloud or the reconstructed point cloud does not conform to the YUV color space (such as RGB color space, etc.), then color space conversion is also required to Make it conform to YUV color space.
  • YUV color space such as RGB color space, etc.
  • the method may further include: if the color components of the points in the initial point cloud do not conform to the YUV color space, performing color space conversion on the initial point cloud so that the color components of the points in the initial point cloud conform to YUV Color space; if the color components of the points in the reconstructed point cloud do not conform to the YUV color space, the color space conversion is performed on the reconstructed point cloud so that the color components of the points in the reconstructed point cloud conform to the YUV color space.
  • the U component and the V component of the points in the initial point cloud and the reconstructed point cloud can be firstly determined to determine the first U component and the V component corresponding to
  • the first attribute parameter and the second attribute parameter can further determine the filter coefficient of the U component and the filter coefficient of the V component.
  • the filtering effect is unstable, and it will increase the code stream overhead and affect the overall compression efficiency. It is not necessary to involve the quality enhancement processing of the Y component, so as to Ensure the high efficiency of U component and V component compression.
  • the embodiment of the present application can also perform quality enhancement processing on the Y component.
  • Wiener filtering of different orders can be performed on the Y component, and an adaptive filter can also be used on the Y component.
  • the working principle of the adaptive filter Same as Wiener filtering, the filter order, number, etc. can be adaptively selected according to the neighborhood difference. In this case, according to the initial point cloud and the reconstructed point cloud, the method based on neighborhood differences is used to determine the filter coefficient of the Y component, so as to subsequently determine the filtered point cloud corresponding to the reconstructed point cloud.
  • S503 Determine the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient of the attribute information.
  • the encoder determines the filter coefficients of the attribute information according to the initial point cloud and the reconstructed point cloud, it can further use these filter coefficients to determine the filtered point cloud corresponding to the reconstructed point cloud .
  • the determining the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient of the attribute information may include:
  • the K target points include the first point and (K-1) neighbor points adjacent to the first point, K is an integer greater than 1, and the first A point represents any point in the reconstructed point cloud;
  • the filtered point cloud is determined according to the filter value of the attribute information of the at least one point.
  • the determination of K target points corresponding to the first point in the reconstructed point cloud may include:
  • the K nearest neighbor search method is used to search for preset candidate points in the reconstructed point cloud
  • the K-nearest neighbor search method can be used to search for a preset value of candidate points in the reconstructed point cloud, calculate the distance between the first point and these candidate points, and then from these candidate points Select (K-1) neighbor points closest to the first point; determine the first point itself and (K-1) neighbor points as the final K target points.
  • the filter coefficient when using the filter coefficient to filter the K target points corresponding to the first point in the reconstructed point cloud, it may include: using the filter coefficient corresponding to the color component The vector filters the K target points corresponding to the first point in the reconstructed point cloud to obtain the filtered value of the color component of the first point in the reconstructed point cloud; according to the filtered value of the color component of the point in the reconstructed point cloud, the filtered point cloud.
  • the filter value of the attribute information of each point in the reconstructed point cloud can be determined according to the filter coefficient of the attribute information and the second attribute parameter; then the filtered point cloud can be constructed based on the filter value of the attribute information.
  • the Wiener filter when the Wiener filter is used to filter the reconstructed point cloud, the noisy signal and the expected signal are required.
  • the reconstructed point cloud can be used as the Noise signal
  • the initial point cloud is used as the desired signal; therefore, the initial point cloud and the reconstructed point cloud can be input into the Wiener filter at the same time, that is, the input of the Wiener filter is the initial point cloud and the reconstructed point cloud, and the Wiener filter
  • the output of the filter is the filter coefficient; after the filter coefficient is obtained, the filter processing of the reconstructed point cloud can also be completed based on the filter coefficient, and the corresponding filtered point cloud can be obtained.
  • the filter coefficients are obtained based on the initial point cloud and the reconstructed point cloud; therefore, applying the filter coefficients to the reconstructed point cloud can restore the original point cloud to the greatest extent.
  • the filter coefficients when filtering the reconstructed point cloud according to the filter coefficient, for one of the color components, based on the filter coefficient corresponding to the color component, combined with the second attribute parameter under the color component, the corresponding filter value.
  • the filter coefficient vector H(k) under the U component is applied to the reconstructed point cloud, that is, the second attribute parameter P(n,k), and the filtering under the U component can be obtained Value R(n), as shown below,
  • the V component can be traversed according to the above method, and finally the filter value under the U component and the V component can be determined, and then the filtered point cloud can be determined by using the filter value under the U component and the V component.
  • S504 Determine filtered identification information according to the reconstructed point cloud and the filtered point cloud.
  • the encoder may further determine the filter identification information according to the reconstructed point cloud and the filtered point cloud.
  • the filtering identification information may be used to determine whether to perform filtering processing on the reconstructed point cloud.
  • the filter identification information can be used to determine whether to perform filtering processing on the reconstructed point cloud; further, the filter identification information can also be used to determine which one or several of the reconstructed point cloud The color components are filtered.
  • the color component may include at least one of the following: a first color component, a second color component, and a third color component; wherein, the to-be-processed component of the attribute information may be the first Any one of the color component, the second color component, and the third color component.
  • FIG. 6 it shows a second schematic flowchart of an encoding method provided by an embodiment of the present application. As shown in Figure 6, for the determination of the filter identification information, the method may include:
  • S601 Determine the first-generation value of the component to be processed of the attribute information of the reconstructed point cloud, and determine the second-generation value of the component to be processed of the attribute information of the filtered point cloud.
  • the determination of the first-generation value of the to-be-processed component of the attribute information of the reconstructed point cloud may include: using the rate distortion cost method to process the to-be-processed component of the reconstructed point cloud's attribute information performing cost value calculation on the component, and using the obtained first rate-distortion value as the first generation value;
  • the determining the second generation value of the component to be processed of the attribute information of the filtered point cloud may include: calculating the cost value of the component to be processed of the attribute information of the filtered point cloud using a rate distortion cost method, and calculating the obtained second generation value of the attribute information of the filtered point cloud The second rate-distortion value is obtained as the second generation value.
  • the filter identification information may be determined in a rate-distortion cost manner.
  • the comparison result with the second-generation value determines the filter identification information of the component to be processed.
  • the cost value here may be a distortion value used for distortion measurement, or may be a rate-distortion cost result, etc., which is not specifically limited in this embodiment of the present application.
  • the embodiment of this application also performs a rate-distortion trade-off between the filtered point cloud and the reconstructed point cloud.
  • the rate-distortion cost method can be used to calculate the comprehensive quality improvement and bit stream Increased rate-distortion value.
  • the first rate-distortion value and the second rate-distortion value can respectively represent the respective rate-distortion cost results of the reconstructed point cloud and the filtered point cloud under the same color component, and are used to represent the compression efficiency of the filtered point cloud.
  • the specific calculation formula is as follows,
  • J is the rate-distortion value
  • D is the SSE of the initial point cloud and the reconstructed point cloud or the filtered point cloud, that is, the sum of squares of the corresponding point errors
  • is a quantity related to the quantization parameter QP
  • R i is the bit stream size of each attribute channel in the texture video frame; in order to simplify the calculation, an approximate value is used: for the reconstructed point cloud, Where R all is the total bit stream size of the texture video frame; for the filtered point cloud, take Among them, K is the order of the filter, and sizeof(int) indicates the bit length of an integer data in the computer.
  • the filter identification information of the attribute channel is set to 1, and the corresponding filter coefficient is saved; otherwise, the filter identification information of the attribute channel is set to 0, not Save the corresponding filter coefficients.
  • the first rate-distortion value of the to-be-processed component of the reconstructed point cloud and the second rate-distortion value of the to-be-processed component of the filtered point cloud can be calculated according to the above method, and the It is used as the first-generation value and the second-generation value, and then the filter identification information of the component to be processed is determined according to the comparison result of the two.
  • the determination of the first-generation value of the to-be-processed component of the attribute information of the reconstructed point cloud may include: using the rate-distortion cost method to calculate the value of the attribute information of the reconstructed point cloud Calculate the cost value of the component to be processed to obtain the first rate-distortion value; use the preset performance measurement index to calculate the performance value of the component to be processed to reconstruct the attribute information of the point cloud to obtain the first performance value; according to the first rate-distortion value and The first performance value, which determines the first generation value.
  • the determining the second generation value of the component to be processed of the attribute information of the filtered point cloud may include: calculating the cost value of the component to be processed of the attribute information of the filtered point cloud using a rate-distortion cost method to obtain the second rate-distortion value; use the preset performance measurement index to calculate the performance value of the component to be processed of the attribute information of the filtered point cloud to obtain the second performance value; determine the second generation value according to the second rate-distortion value and the second performance value.
  • the performance value eg, PSNR value
  • the first performance value and the second performance value may represent respective encoding and decoding performances of the reconstructed point cloud and the filtered point cloud under the same color component.
  • the first performance value may be the PSNR value of the color component of the points in the reconstructed point cloud
  • the second performance value may be the PSNR value of the color components of the points in the filtered point cloud.
  • the embodiment of the present application can not only consider the PSNR value to decide whether to perform filtering at the decoder, but also use the rate-distortion cost method to perform rate-distortion trade-offs before and after filtering.
  • the first performance value and the first rate-distortion value of the component to be processed of the reconstructed point cloud can be calculated according to the above method, and used as the first generation value; and Calculate the second performance value and the second rate distortion value of the component to be processed in the filtered point cloud, and use it as the second generation value, and then determine the value of the component to be processed according to the comparison result of the first generation value and the second generation value Filter identification information.
  • S602 Determine the filter identification information of the component to be processed according to the first generation value and the second generation value.
  • the determining the filter identification information of the component to be processed according to the first generation value and the second generation value may include:
  • the value of the second generation is less than the value of the first generation, it is determined that the value of the filter identification information of the component to be processed is the first value
  • the value of the second generation is greater than the value of the first generation, it is determined that the value of the filter identification information of the component to be processed is a second value.
  • the value of the filter identification information of the component to be processed is the first value; or, it can also be determined that the value of the filter identification information of the component to be processed is The value is the second value.
  • the filter identification information of the component to be processed indicates that the component to be processed of the attribute information of the reconstructed point cloud is to be filtered; or, If the value of the filtering identification information of the component to be processed is the second value, it may be determined that the filtering identification information of the component to be processed indicates that no filtering process is performed on the component to be processed of the attribute information of the reconstructed point cloud.
  • the determining the filter identification information of the component to be processed according to the first generation value and the second generation value may include:
  • the second performance value is greater than the first performance value and the second rate-distortion value is less than the first rate-distortion value, then determine that the value of the filter identification information of the component to be processed is the first value;
  • the second performance value is smaller than the first performance value, it is determined that the value of the filter identification information of the component to be processed is a second value.
  • the determining the filter identification information of the component to be processed according to the first generation value and the second generation value may include:
  • the second performance value is greater than the first performance value and the second rate-distortion value is less than the first rate-distortion value, then determine that the value of the filter identification information of the component to be processed is the first value;
  • the second rate-distortion value is greater than the first rate-distortion value, it is determined that the value of the filter identification information of the component to be processed is a second value.
  • the second performance value is equal to the first performance value or the second rate-distortion value is equal to the first rate-distortion value
  • it may be determined that the value of the filter identification information of the component to be processed is the first value; or , it may also be determined that the value of the filter identification information of the component to be processed is a second value.
  • the filter identification information of the component to be processed indicates that the component to be processed of the attribute information of the reconstructed point cloud is to be filtered; or, If the value of the filtering identification information of the component to be processed is the second value, it may be determined that the filtering identification information of the component to be processed indicates that no filtering process is performed on the component to be processed of the attribute information of the reconstructed point cloud.
  • the component to be processed is a color component
  • the first performance value, the first rate-distortion value, the second performance value and the second rate-distortion value it can be determined that the filtered point cloud is compared with the reconstructed point cloud , whether the performance value of one or several of these color components increases while the rate-distortion cost decreases.
  • the determining the filter identification information of the component to be processed according to the first generation value and the second generation value may include:
  • the second performance value is smaller than the first performance value, or the second rate-distortion value is greater than the first rate-distortion value, determine that the value of the filter identification information of the first color component is the second value.
  • the determining the filter identification information of the component to be processed according to the first generation value and the second generation value may include :
  • the second performance value is smaller than the first performance value, or the second rate-distortion value is greater than the first rate-distortion value, determine that the value of the filter identification information of the second color component is the second value.
  • the determining the filter identification information of the component to be processed according to the first generation value and the second generation value may include :
  • the second performance value is smaller than the first performance value, or the second rate-distortion value is greater than the first rate-distortion value, determine that the value of the filter identification information of the third color component is the second value.
  • the filter identification information of the U component will be set to indicate that the U component is not filtered; correspondingly, if the PSNR1 corresponding to the reconstructed point cloud is less than the PSNR2 corresponding to the filtered point cloud and the Cost1 corresponding to the reconstructed point cloud is greater than the Cost2 corresponding to the filtered point cloud, that is, the PSNR of the filtered U component is improved and the rate-distortion cost is reduced, then the filtering effect can be considered to be better.
  • the filter identification information of the U component will be set to indicate that the U component is to be filtered.
  • the embodiment of the present application can traverse the U component and V component according to the above method, or even the Y component, U component and V component, and finally can use the filter identification information of all color components to determine the final filter identification information.
  • the YUV color space is taken as an example.
  • the filtering effect is unstable, and it will increase the code stream overhead and affect the overall
  • the compression efficiency may not involve the quality enhancement processing of the Y component, so as to ensure the high efficiency of U component and V component compression.
  • the obtaining the filter identification information according to the filter identification information of the component to be processed may include: obtaining the first filter identification information and Second filtering identification information.
  • the first filter identification information is used to indicate whether to filter the second color component of the attribute information of the reconstructed point cloud
  • the second filter identification information is used to indicate whether to filter the third color component of the attribute information of the reconstructed point cloud. filter processing.
  • the second color component is a U component
  • the third color component is a V component
  • the second color component is a V component
  • the third color component is a U component.
  • the YUV color space is still taken as an example, and the Y component, the U component, and the V component may all be filtered.
  • the obtaining the filter identification information according to the filter identification information of the components to be processed may include: obtaining the first color component, the second color component, and the third color component when the components to be processed include the first color component, the second color component, and the third color component.
  • First filtering identification information, second filtering identification information and third filtering identification information may include: obtaining the first color component, the second color component, and the third color component when the components to be processed include the first color component, the second color component, and the third color component.
  • the first filter identification information is used to indicate whether to perform the first filtering process on the second color component of the attribute information of the reconstructed point cloud
  • the second filter identification information is used to indicate whether to perform the third color component of the attribute information of the reconstructed point cloud
  • the second filtering process is performed on the component
  • the third filtering identification information is used to indicate whether to perform the third filtering process on the first color component of the attribute information of the reconstructed point cloud.
  • the filter corresponding to the third filtering process is different from the filters corresponding to the first filtering process and the second filtering process.
  • the filtering effect is unstable, and it will increase the code stream overhead and affect the overall compression efficiency.
  • the embodiment of the present application can also perform quality enhancement processing on Y component.
  • the Y component uses an adaptive filter, where the working principle of the adaptive filter is the same as that of the Wiener filter, and the order and number of filters can be adaptively selected according to the neighborhood difference.
  • the filter identification information may be in the form of an array, or may be in the form of individual flag bits. Assuming that only the U component and the V component need to be filtered in the YUV color space, it can be a 1 ⁇ 2 array, which is composed of the filter identification information of the U component and the filter identification information of the V component, for example, expressed by [u v] ; Or it can also be encoded and decoded sequentially according to the UV color channels.
  • the Y component, U component, and V component need to be filtered in the YUV color space, it can be a 1 ⁇ 3 array, which consists of the filter identification information of the Y component, the filter identification information of the U component, and the filter identification information of the V component.
  • Composition for example represented by [y u v]; or it can be encoded and decoded sequentially according to the YUV color channels.
  • the filter identification information it can be combined together in the form of an array first, and then encode and transmit the corresponding filter coefficient; or it can also encode the filter identification information of one color channel and then encode and transmit the corresponding filter coefficient, and then encode
  • the filter identification information of a color channel is followed by encoding and transmission of the corresponding filter coefficients, etc.; the overheads of the two cases are the same, which is not specifically limited here.
  • the component to be processed is any one of the first color component, the second color component and the third color component.
  • the method may also include:
  • the filter identification information of the component to be processed indicates that the component to be processed of the attribute information of the reconstructed point cloud is filtered
  • the filtering identification information of the component to be processed indicates that no filtering process is performed on the component to be processed of the attribute information of the reconstructed point cloud.
  • the method may also include:
  • the value of the filter identification information of the first color component is the first value, it is determined to perform filtering processing on the first color component of the attribute information of the reconstructed point cloud; if the value of the filter identification information of the first color component is the second value, it is determined not to filter the first color component of the attribute information of the reconstructed point cloud; or,
  • the value of the filter identification information of the second color component is the first value, it is determined to perform filtering processing on the second color component of the attribute information of the reconstructed point cloud; if the value of the filter identification information of the second color component is the second value, it is determined not to filter the second color component of the attribute information of the reconstructed point cloud; or,
  • the value of the filter identification information of the third color component is the first value, it is determined to perform filtering processing on the third color component of the attribute information of the reconstructed point cloud; if the value of the filter identification information of the third color component is the second value value, it is determined not to filter the third color component of the attribute information of the reconstructed point cloud.
  • the filter identification information corresponding to a certain color component is the first value, then it can be instructed to perform filtering processing on the color component; if the filter identification information corresponding to a certain color component is the second value, Then it may indicate that the color component is not to be filtered.
  • the method may also include:
  • the filter identification information of the first color component, the filter identification information of the second color component, and the filter identification information of the third color component is the first value, then it is determined that the filter identification information indicates that the reconstruction point cloud is filtered;
  • the filter identification information of the first color component, the filter identification information of the second color component, and the filter identification information of the third color component all have the second value, it is determined that the filter identification information indicates that no filtering process is performed on the reconstructed point cloud.
  • the filter identification information corresponding to these color components is the second value, then it can be indicated that no filtering process is performed on these color components, and it can be determined that the filter identification information indicates that the reconstruction point cloud is not correct.
  • Perform filtering processing correspondingly, if at least one of the filtering identification information of these color components is the first value, then it can be indicated to perform filtering processing on at least one color component, that is, it can be determined that the filtering identification information indicates that filtering processing is performed on the reconstructed point cloud.
  • the components to be processed only include the second color component and the third color component, wherein the filter identification information corresponding to the second color component (that is, the first filter identification information in the foregoing embodiments) It is used to indicate whether to filter the second color component of the attribute information of the reconstructed point cloud, and the filter identification information corresponding to the third color component (that is, the second filter identification information in the foregoing embodiment) is used to indicate whether to filter the reconstructed point cloud Filtering is performed on the third color component of the attribute information; then the method may also include: if at least one of the first filtering identification information and the second filtering identification information is a first value, then determining that the filtering identification information indicates that the reconstructed point cloud is Filtering processing: if both the first filtering identification information and the second filtering identification information are the second value, then it is determined that the filtering identification information indicates that no filtering process is performed on the reconstructed point cloud.
  • the first value and the second value are different, and the first value and the second value may be in the form of parameters or numbers.
  • the identification information corresponding to these color components may be a parameter written in a profile (profile), or a value of a flag (flag), which is not specifically limited here.
  • the first value can be set to 1, and the second value can be set to 0; or, the first value can also be set to true, and the second value can also be set to false ; but not specifically limited here.
  • the filter identification information can be Write the code stream, and also optionally write the filter coefficients into the code stream.
  • the filter identification information corresponding to the first color component is 1, it is necessary to filter the first color component of the attribute information of the reconstructed point cloud ;
  • the filter identification information corresponding to the second color component is 0, that is, it is not necessary to filter the second color component of the attribute information of the reconstructed point cloud; Filter the third color component of the attribute information; at this time, only the filter coefficients corresponding to the first color component and the filter coefficients corresponding to the third color component need to be written into the code stream, without the need to write the filter coefficients corresponding to the second color component Write code stream.
  • encoding the filter coefficient corresponding to the filter identification information and attribute information may include: If the first filter identification information indicates to filter the second color component of the attribute information of the reconstructed point cloud, encode the filter coefficient corresponding to the first filter identification information and the second color component; and/or, if the second filter The identification information indicates that filtering is performed on the third color component of the attribute information of the reconstructed point cloud, and the second filtering identification information and the filter coefficient corresponding to the third color component are encoded.
  • the filtering effect is unstable, and it will increase the code stream overhead and affect the overall compression efficiency.
  • the quality enhancement processing of the Y component is not performed to ensure the high efficiency of the compression of the U component and the V component.
  • the filter identification information of the U component can be encoded first, and then the filter coefficient corresponding to the U component can be encoded and transmitted; then the filter identification information of the V component can be encoded, and then the filter coefficient corresponding to the V component can be encoded and transmitted.
  • the filter identification information of the U component is 0, then after encoding the filter identification information of the U component, there is no need to encode and transmit the filter coefficient corresponding to the U component, and the filter identification information of the V component and the filter coefficient corresponding to the V component can be encoded and transmitted immediately, No specific limitation is made here.
  • the geometric distortion if the geometric distortion is too large, it can be directly determined not to perform filtering processing on the reconstructed point cloud at this time.
  • the geometric distortion when the geometric distortion is large (that is, the quantization parameter is greater than the preset threshold value), the number of points to reconstruct the point cloud at this time is relatively small, and if the filter coefficient of the same size is written, it will be The rate-distortion cost is too high; therefore, the filtering process can no longer be performed.
  • the method may further include: encoding only the filtering identification information, and writing the obtained encoded bits into the code stream.
  • the encoder may further determine the filter identification information according to the reconstructed point cloud and the filtered point cloud. If the filter identification information indicates that the attribute information of the reconstructed point cloud is filtered, then the filter identification information and the corresponding filter coefficients can be written into the code stream; if the filter identification information indicates that the attribute information of the reconstructed point cloud is not filtered processing, then it is not necessary to write the filter coefficients into the code stream at this time, but only need to write the filter identification information into the code stream, so that it can be transmitted to the decoder through the code stream later.
  • the encoder will reconstruct the point cloud accordingly.
  • the method may further include: determining the geometric video frame and the texture video frame according to the initial point cloud; determining the predicted value of the geometric video frame and the predicted value of the texture video frame; The predicted value of the video frame determines the residual value of the geometric video frame; and encodes the residual value of the geometric video frame, and writes the obtained coded bits into the code stream; according to the original value of the texture video frame and the texture video frame The prediction value is to determine the residual value of the texture video frame; and the residual value of the texture video frame is encoded, and the obtained encoded bits are written into the code stream.
  • the encoder determines the residual value of the geometric video frame and the residual value of the texture video frame, it also needs to write the residual value of the geometric video frame and the residual value of the texture video frame into the code stream, so that Subsequently, it is transmitted to the decoder through the code stream.
  • the embodiment of the present application can set the value of the order to be equal to 20, which can not only ensure the filtering effect, but also reduce the time complexity to a certain extent;
  • the embodiment of the present application can set the value of the order to be equal to 20, which can not only ensure the filtering effect, but also reduce the time complexity to a certain extent;
  • no specific limitation is made here.
  • the method may also include:
  • the value of K is determined according to the neighborhood difference value of points in the reconstructed point cloud.
  • the neighborhood difference value may be calculated based on a component difference between a color component to be filtered of a point and a color component to be filtered of at least one neighboring point.
  • determining the value of K according to the neighborhood difference value of the reconstructed point cloud midpoint may also include: if the neighborhood difference value of the reconstructed point cloud midpoint satisfies the first preset interval, then selecting the first category The filter performs filtering processing; if the neighborhood difference value of the points in the reconstructed point cloud satisfies the second preset interval, the second type of filter is selected for filtering processing; wherein, the order of the first type of filter is the same as that of the second type of filtering The order of the device is different.
  • a possible implementation is to select different K when filtering point clouds of different sizes, specifically, the threshold can be set according to the number of points for judgment, so as to realize the adaptive selection of K; when point cloud When the number of points in the point cloud is small, K can be appropriately increased to focus on improving the quality; when the number of points in the point cloud is large, K can be appropriately reduced. At this time, first ensure that the memory is sufficient and the running speed is not too slow.
  • the embodiment of this application proposes a V-PCC point cloud color attribute quality enhancement technology based on Wiener filtering.
  • Wiener filtering is performed on the reconstructed point cloud to achieve enhancement.
  • the purpose of point cloud geometry and color attribute quality in terms of geometry, the Wiener filtering method based on the neighborhood difference of geometric video frames can be used; and in terms of color, the embodiment of the present application proposes to directly perform color attribute Wiener filtering on the reconstructed point cloud at the codec end, instead of Process the reconstructed video frame; specifically, the encoding method proposed in steps S501-S505 can be used.
  • the embodiment of the present application is aimed at the V-PCC geometrically lossy and attribute lossy encoding manner.
  • Wiener filtering of color attributes is implemented for geometrically lossy reconstructed point clouds, which improves the quality of color attributes without any impact on geometric reconstruction and filtering processes.
  • geometric filtering and color filtering are applied to V-PCC at the same time; among them, the geometric aspect uses Wiener filtering technology based on neighborhood differences for geometrically reconstructed video frames, and the color aspect is used for reconstructing point clouds. Directly perform Wiener filtering operation; finally obtain a point cloud with improved geometric and color quality, and the compression performance is greatly improved.
  • Wiener filter proposed in the embodiment of the present application can be used in the prediction loop, that is, as an inloop filter, and can be used as a reference for decoding subsequent point clouds; it can also be outside the prediction loop, that is, as an Post filter used, not used as a reference for decoding subsequent point clouds.
  • This embodiment of the present application does not specifically limit it.
  • the Wiener filter proposed in the embodiment of the present application is an in-loop filter
  • the parameter information indicating the filtering process such as Write the filter identification information into the code stream, and also need to write the filter coefficient into the code stream; correspondingly, after determining not to perform filtering processing, you can choose not to write the parameter information indicating the filtering process, such as the filter identification information into the code stream , and do not write the filter coefficients into the code stream.
  • the Wiener filter proposed in the embodiment of the present application is a post-processing filter
  • the filter coefficient corresponding to the filter is located in a separate auxiliary information data unit (for example, supplementary enhancement information SEI)
  • a separate auxiliary information data unit for example, supplementary enhancement information SEI
  • the decoder does not obtain the supplementary enhancement information, it will not Reconstruct the point cloud for filtering.
  • the filter coefficient corresponding to the filter and other information are contained in an auxiliary information data unit.
  • the parameter information indicating the filtering process such as the filter identification information
  • the filter coefficients it is also necessary to write the filter coefficients into the code stream; correspondingly, after determining not to perform filtering processing, you can choose not to write the parameter information indicating the filtering process, such as the filter identification information, into the code stream, and at the same time do not write the filter coefficient The coefficients are written into the code stream.
  • one or more parts of the reconstructed point cloud can be selected to be filtered, that is, the control range of the filter identification information can be the entire reconstructed point cloud, or the reconstructed point cloud A certain part of it is not specifically limited here.
  • the embodiment of the present application can also purely expect to improve the quality of the color attribute, and no longer filter the geometric video frame at this time, so that a lower BD-TotalRate and better compression performance. What's more, the embodiment of the present application can also filter the texture video frame and the color attribute of the reconstructed point cloud at the same time, and better results may also be obtained.
  • This embodiment provides an encoding method, which is applied to an encoder. Determine the initial point cloud and the reconstructed point cloud corresponding to the initial point cloud; determine the filter coefficient of the attribute information in the reconstructed point cloud according to the initial point cloud and the reconstructed point cloud; determine the filter coefficient corresponding to the reconstructed point cloud according to the filter coefficient of the attribute information Point cloud; according to the reconstructed point cloud and the filtered point cloud, determine the filter identification information; if the filter identification information indicates that the attribute information of the reconstructed point cloud is filtered, then the filter coefficient corresponding to the filter identification information and attribute information is encoded, and the The resulting coded bits are written into the codestream.
  • the encoder uses the initial point cloud and the reconstructed point cloud to calculate the filter coefficients for filtering processing, and after determining to filter the reconstructed point cloud, the corresponding filter coefficients are passed to the decoder; correspondingly, the decoder It can be directly decoded to obtain the filter coefficient, and then use the filter coefficient to filter the reconstructed point cloud, thereby realizing the optimization of the reconstructed point cloud and improving the quality of the point cloud; in addition, the encoder uses the initial point cloud and the reconstructed point cloud to calculate When performing filter coefficients for filtering processing, it also solves the problem that the midpoint of the initial point cloud and the midpoint of the reconstructed point cloud are difficult to match, and by filtering the attribute information of the reconstructed point cloud, not only the quality of the reconstructed point cloud is enhanced, Moreover, the quality of the video frame can also be improved, further improving the compression efficiency, thereby saving the bit rate, and improving the encoding and decoding efficiency.
  • the embodiment of the present application also provides a code stream, which is generated by performing bit coding according to the information to be encoded; wherein the information to be encoded includes at least one of the following: geometric video The residual value of the frame, the residual value of the texture video frame, the filter identification information and the filter coefficient.
  • the decoder obtains the residual value of the geometric video frame and the texture video frame by decoding
  • the residual value of the reconstructed point cloud is then constructed; and the decoder obtains the filter identification information by decoding, and can also determine whether to filter the attribute information of the reconstructed point cloud; and then filter the attribute information of the reconstructed point cloud
  • the filter coefficient corresponding to the attribute information is directly obtained by decoding; the reconstructed point cloud is filtered according to the filter coefficient, thereby realizing the optimization of the reconstructed point cloud and improving the quality of the point cloud.
  • the embodiment of the present application provides a more suitable method for processing to improve the quality and compression performance of the point cloud.
  • a Wiener filter-based V-PCC point cloud color attribute quality enhancement technology is proposed, based on geometric video frame filtering , Wiener filtering is performed on the reconstructed point cloud to achieve the purpose of enhancing the quality of point cloud geometry and color attributes.
  • the embodiment of the present application can adopt the Wiener filtering method based on the neighborhood difference of geometric video frames; in terms of color, the embodiment of the present application proposes to directly perform color attribute Wiener filtering on the reconstructed point cloud at the codec end, and No processing for reconstructed video frames.
  • the encoder obtains the corresponding filter coefficients and filter identification information through calculation and transmits them to the decoder. After decoding the filter coefficients, the decoder can perform Wiener filtering on the reconstructed geometric video frame and the reconstructed point cloud.
  • FIG. 7 is a schematic flowchart of an overall coding and decoding process provided by an embodiment of the present application.
  • the reconstructed point cloud can be obtained; then the initial point cloud and the reconstructed point cloud can be used as the input of the Wiener filter, and the output of the Wiener filter It is the filter coefficient of the attribute information; these filter coefficients will determine whether the reconstruction point cloud needs to be filtered according to the judgment effect; only when the reconstruction point cloud needs to be filtered, the corresponding filter coefficient will be written into the code stream; In addition, the geometric residual value and texture residual value of the initial point cloud after encoding will also be written into the code stream.
  • the decoder can obtain the geometric residual value and the texture residual value by decoding the code stream, and then after decoding and reconstruction processing, the reconstructed point cloud can be obtained; in addition, if there are filter coefficients in the code stream, it is also required The corresponding filter coefficients are obtained by decoding, and then these filter coefficients and the reconstructed point cloud are input to the Wiener filter, and finally the filtered point cloud (that is, the output point cloud) is output, and it has a quality enhancement effect.
  • FIG. 8 is a schematic flowchart of a filtering process at an encoding end provided in an embodiment of the present application.
  • the initial point cloud and the reconstructed point cloud will first perform a K-nearest neighbor search to determine the matching point cloud; where the points in the matching point cloud have a corresponding relationship with the points in the reconstructed point cloud, and the matching point cloud is used as Both the initial point cloud and the reconstructed point cloud are input to the Wiener filter. Then through the Wiener filter, the filter identification information and the corresponding filter coefficients are output, and written into the code stream.
  • the embodiment of the present application is aimed at the V-PCC geometrically lossy and attribute lossy encoding manner.
  • the specific operations are as follows:
  • V-PCC After the geometric video frame and texture video frame generated from the initial point cloud are compressed, the encoder will reconstruct the point cloud accordingly. Since V-PCC has only two encoding methods, that is, lossless encoding for both geometry and attributes, and lossy encoding for both geometry and attributes, the technical solutions of the embodiments of the present application deal with lossy encoding. Therefore, when the reconstructed point cloud is obtained, the geometric properties of the point cloud are also distorted, as shown in the following: the number of reconstructed point cloud points is generally different from the number of initial point cloud points, so it is difficult to find every The exact corresponding points of each point in the initial point cloud, that is, the points with the same geometric coordinates.
  • this scheme proposes a method to find the matching point cloud for the reconstructed point cloud after geometric lossy coding, and on this basis, Wiener filtering of color attributes is performed.
  • the KNN search method is used to traverse each point in the reconstructed point cloud, and the search range is the entire initial point cloud. It is to find the nearest neighbor point of each point and use it as the real point corresponding to the reconstruction point. Although this method cannot find the corresponding points in the strict sense, it is completely feasible to use the nearest neighbor point as the corresponding point due to the large number of points in the entire point cloud and the small difference in the neighborhood. method works better.
  • the technical solution performs KNN search on each point of the reconstructed point cloud, it will find up to 5 neighbor points of the current point in the initial point cloud, and calculate each neighbor point in turn The distance from the current point, save the nearest neighbor point, if there are multiple nearest neighbor points with the same distance, these nearest neighbor points will be fused into one point, that is, the average value of the color attribute values of these points will be taken as the nearest point of the current point The true value of the color attribute of the adjacent matching point.
  • the technical scheme After having the corresponding point, the technical scheme will use the index order of each point in the reconstructed point cloud to combine the attribute information of the corresponding point in the initial point cloud into a matrix with the same size as the reconstructed point cloud, and view it as Make the "real point cloud" attribute matrix, that is, match the point cloud, to facilitate the next operation.
  • the scheme uses the principle of point cloud Wiener filtering and uses a Wiener filter with an order of 20. For each frame of the point cloud sequence, the color attributes of the reconstructed point cloud and the real point cloud are converted from the RGB color space to the YUV color space. , the quality enhancement of the U and V color channels in the reconstructed point cloud is performed sequentially.
  • D is the SSE of the initial point cloud and the reconstructed point cloud or the filtered point cloud, that is, the sum of squares of the corresponding point errors
  • is a quantity related to the quantization parameter QP, which is taken as R i is the bit stream size of each attribute channel in the texture video frame; in order to simplify the calculation, an approximate value is used: for the reconstructed point cloud, Where R all is the total bit stream size of the texture video frame; for the filtered point cloud, take Among them, K is the order of the filter, and sizeof(int) indicates the bit length of an integer data in the computer.
  • the filter identification information of the attribute channel is set to 1, and the corresponding filter coefficient is saved; otherwise, the filter identification information of the attribute channel is set to 0, not Save the corresponding filter coefficients.
  • the code stream of V-PCC is mainly composed of three parts: the code stream of occupancy map and auxiliary information related to Patch, the code stream of geometric video frame, and the code stream of texture video frame. Therefore, in order to facilitate the test of the color attribute gain effect and compression performance of the filtered point cloud, the scheme writes wave identification information, filter coefficient and other related information into the texture video frame stream.
  • the code stream of the texture video frame is read, and the size of the bit stream is obtained.
  • the rate-distortion optimization is performed by using the size of the code stream, and the filter identification information of the attribute channel can be obtained by combining PSNR. If the value of the filter identification information is 1, it means that the attribute channel filter improves the rate-distortion performance, increases the compression ratio, and enhances the quality of the reconstructed point cloud. Therefore, the filter identification information and filter coefficients will be written into the code stream. If the value of the filter identification information is 0, it proves that the filtering effect is not good. At this time, the transmission of filter coefficients and the operation of the decoding end will not be performed, and only the filter identification information will be written into the code stream.
  • the specific writing method is: in the existing encoding framework, 0x00 00 00 01 is commonly used as a separator to start a new encoding, followed by NALU (Network Abstraction Layer Unit), that is, frame data information.
  • NALU Network Abstraction Layer Unit
  • This technical solution writes the relevant data that needs to be transmitted directly after the delimiter; after all frame information is written, the original data of the texture video frame after the delimiter is written, which can ensure no information loss and smoothly Write related data such as filter identification information and filter coefficients.
  • FIG. 9 is a schematic flow chart of a filtering process at a decoding end provided by an embodiment of the present application.
  • the decoder first decodes the code stream to obtain the filter identification information; then judges whether filtering of attribute information is required; if the judgment result is no, it returns directly to the V-PCC framework; if filtering of attribute information is required, then decode Code stream to obtain the filter coefficients corresponding to the attribute information; when performing Wiener filtering processing, the reconstructed point cloud and filter coefficients can be used as the input of the Wiener filter at the decoding end, and then through the Wiener filter, the filtered and high-quality The enhanced filtered point cloud is then returned to the V-PCC framework.
  • FIG. 10 is a schematic diagram of a test result based on the CTC_C2 test condition provided in the embodiment of the present application.
  • the first 32 frames of the MPEG test sequence are tested under the CTC-C2 test condition (cat2-A & cat2- B), the test results are shown in Figure 10. It can be seen from Figure 10 that, compared with the original program, after the reconstructed point cloud color attribute filtering process, and even the geometric video frame adaptive filtering process, the geometry and attribute quality of the point cloud have been greatly improved, and the compression efficiency has been further increased.
  • BD-GeomRate in terms of geometry and BD-Attr Rate in terms of attributes have both decreased, especially BD-AttrRate has decreased significantly.
  • the color code stream itself is relatively large, the transmission of filter coefficients will hardly affect it, and its own PSNR Also increased more.
  • both geometry and color attributes have obvious gains.
  • the color attribute gain of the Cat2-A sequence is more considerable, indicating that the method of the embodiment of the present application has a more obvious effect on improving the quality of the point cloud with higher reconstruction quality.
  • the embodiment of the present application provides a V-PCC point cloud color attribute quality enhancement technology based on Wiener filtering.
  • the encoder uses the initial point cloud and the reconstructed point cloud to calculate the filter coefficients for filtering processing, and after determining to filter the reconstructed point cloud, the corresponding filter coefficients are passed to the decoder; correspondingly, the decoder It can be directly decoded to obtain the filter coefficient, and then use the filter coefficient to filter the reconstructed point cloud, thereby realizing the optimization of the reconstructed point cloud and improving the quality of the point cloud; in addition, the encoder uses the initial point cloud and the reconstructed point cloud to calculate When performing filter coefficients for filtering processing, it also solves the problem that the midpoint of the initial point cloud and the midpoint of the reconstructed point cloud are difficult to match, and by filtering the attribute information of the reconstructed point cloud, not only the quality of the reconstructed point cloud is enhanced, Moreover, the quality of the video frame can also be improved, further improving the compression efficiency, thereby saving
  • FIG. 11 shows a schematic diagram of the composition and structure of an encoder 110 provided in the embodiment of the present application.
  • the encoder 110 may include: a first determining unit 1101, a first filtering unit 1102, and an encoding unit 1103; wherein,
  • the first determination unit 1101 is configured to determine the initial point cloud and the reconstructed point cloud corresponding to the initial point cloud; and determine the filter coefficient of the attribute information in the reconstructed point cloud according to the initial point cloud and the reconstructed point cloud;
  • the first filtering unit 1102 is configured to determine the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient of the attribute information
  • the first determining unit 1101 is further configured to determine the filter identification information according to the reconstructed point cloud and the filtered point cloud;
  • the encoding unit 1103 is configured to encode filter coefficients corresponding to the filter identification information and attribute information if the filter identification information indicates that the attribute information of the reconstructed point cloud is to be filtered, and write the obtained encoded bits into the code stream.
  • the first determining unit 1101 is further configured to encode and reconstruct the initial point cloud using a preset encoding method to obtain the first reconstructed point cloud; and perform geometry recovery processing on the first reconstructed point cloud to obtain Reconstruct the point cloud; wherein, the preset encoding method is used to indicate that the initial point cloud is encoded with lossy geometry and lossy attributes.
  • the first determination unit 1101 is further configured to perform geometric compensation processing on the first reconstructed point cloud to obtain a second reconstructed point cloud; and perform scaling processing on the second reconstructed point cloud to obtain a reconstructed point cloud; Among them, the reconstructed point cloud has the same size and the same geometric position as the original point cloud.
  • the encoder 110 may further include a search unit 1104 configured to perform a neighbor point search on the initial point cloud according to a preset search method, and determine the correspondence between the points in the reconstructed point cloud and the initial point cloud. points; and construct a matching point cloud according to the determined corresponding points, use the matching point cloud as the initial point cloud, and establish the corresponding relationship between the points in the reconstructed point cloud and the points in the initial point cloud.
  • a search unit 1104 configured to perform a neighbor point search on the initial point cloud according to a preset search method, and determine the correspondence between the points in the reconstructed point cloud and the initial point cloud. points; and construct a matching point cloud according to the determined corresponding points, use the matching point cloud as the initial point cloud, and establish the corresponding relationship between the points in the reconstructed point cloud and the points in the initial point cloud.
  • the preset search method is the K-nearest neighbor search method; correspondingly, the search unit 1104 is specifically configured to search for the first regular point in the initial point cloud by using the preset search method based on the current point in the reconstructed point cloud. Numerical points; and calculate the distance between the current point and the first constant value points respectively;
  • the first determining unit 1101 is further configured to determine a corresponding point of the current point according to the distance value.
  • the first determination unit 1101 is further configured to select the minimum distance value from the distance values; and if the number of the minimum distance value is 1, determine the point corresponding to the minimum distance value as the corresponding point of the current point points; and if there are multiple minimum distance values, determine the corresponding point of the current point according to the multiple points corresponding to the minimum distance value.
  • the first determination unit 1101 is further configured to determine K target points corresponding to the first point in the reconstructed point cloud; wherein, the K target points include the first point and (K -1) neighbor points, K is an integer greater than 1, and the first point represents any point in the reconstructed point cloud;
  • the first filtering unit 1102 is configured to filter the K target points corresponding to the first point in the reconstructed point cloud according to the filter coefficient corresponding to the attribute information, and determine the filter value of the attribute information of the first point in the reconstructed point cloud; After the filtered value of the attribute information of at least one point in the reconstructed point cloud is obtained, the filtered point cloud is determined according to the filtered value of the attribute information of at least one point.
  • the search unit 1104 is further configured to search for a preset value of candidate points in the reconstructed point cloud by using the K-nearest neighbor search method based on the first point in the reconstructed point cloud; and calculate the first point and the preset The distance values between candidate points are selected, and (K-1) distance values are selected from the obtained preset distance values, and the (K-1) distance values are all less than the remaining preset value distance values. the distance value;
  • the first determination unit 1101 is further configured to determine (K-1) neighboring points according to the candidate points corresponding to (K-1) distance values, and determine the first point and the (K-1) neighboring points as the first point Corresponding K target points.
  • the first determination unit 1101 is further configured to determine the first attribute parameter according to the original value of the attribute information of the point in the initial point cloud; and according to the attribute information of the K target points corresponding to the point in the reconstructed point cloud Determining a second attribute parameter of the reconstructed value of ; and determining a filter coefficient of the attribute information based on the first attribute parameter and the second attribute parameter.
  • the first determining unit 1101 is further configured to determine the cross-correlation parameter according to the first attribute parameter and the second attribute parameter; determine the autocorrelation parameter according to the second attribute parameter; and determine the Coefficient calculation is performed to obtain the filter coefficient of the attribute information.
  • the attribute information includes a color component
  • the color component includes at least one of the following: a first color component, a second color component, and a third color component; wherein, if the color component conforms to the RGB color space, then determine the first color component
  • the first color component, the second color component and the third color component are: R component, G component, B component; if the color component conforms to the YUV color space, then determine the first color component, the second color component and the third color component in order For: Y component, U component, V component.
  • the encoder 110 may further include a calculation unit 1105 configured to determine the first-generation value of the component to be processed of the attribute information of the reconstructed point cloud, and determine the value of the component to be processed of the attribute information of the filtered point cloud.
  • the second generation value of the processing component is configured to determine the first-generation value of the component to be processed of the attribute information of the reconstructed point cloud, and determine the value of the component to be processed of the attribute information of the filtered point cloud.
  • the first determining unit 1101 is further configured to determine the filter identification information of the component to be processed according to the first generation value and the second generation value; and obtain the filter identification information according to the filter identification information of the component to be processed.
  • the first determination unit 1101 is further configured to determine that the value of the filter identification information of the component to be processed is the first value if the second generation value is less than the first generation value; if the second generation value is greater than the first generation value One generation value, then determine the value of the filter identification information of the component to be processed as the second value.
  • the calculation unit 1105 is further configured to calculate the cost value of the unprocessed component of the attribute information of the reconstructed point cloud by using the rate-distortion cost method, and use the obtained first rate-distortion value as the first generation value; and
  • the rate-distortion cost method is used to calculate the cost value of the component to be processed of the attribute information of the filtered point cloud, and the obtained second rate-distortion value is used as the second-generation value.
  • the calculation unit 1105 is further configured to use the rate-distortion cost method to calculate the cost value of the unprocessed component of the attribute information of the reconstructed point cloud to obtain the first rate-distortion value; calculating the performance value of the component to be processed of the attribute information of the cloud to obtain a first performance value; and determining the first generation value according to the first rate-distortion value and the first performance value;
  • the calculation unit 1105 is further configured to use the rate-distortion cost method to calculate the cost value of the unprocessed component of the attribute information of the filtered point cloud to obtain a second rate-distortion value; performing performance value calculation on the to-be-processed component of the attribute information to obtain a second performance value; and determining a second generation value according to the second rate-distortion value and the second performance value.
  • the first determining unit 1101 is further configured to determine the value of the filter identification information of the component to be processed if the second performance value is greater than the first performance value and the second rate-distortion value is less than the first rate-distortion value is the first value; if the second performance value is smaller than the first performance value, determine that the value of the filter identification information of the component to be processed is the second value.
  • the first determining unit 1101 is further configured to determine the value of the filter identification information of the component to be processed if the second performance value is greater than the first performance value and the second rate-distortion value is less than the first rate-distortion value is the first value; if the second rate-distortion value is greater than the first rate-distortion value, determine that the value of the filter identification information of the component to be processed is the second value.
  • the first determining unit 1101 is further configured to determine that the filter identification information of the component to be processed indicates that the attribute information of the reconstructed point cloud is to be processed if the value of the filter identification information of the component to be processed is the first value.
  • the processing component performs filtering processing; if the value of the filtering identification information of the component to be processed is the second value, it is determined that the filtering identification information of the component to be processed indicates that no filtering process is performed on the component to be processed of the attribute information of the reconstructed point cloud.
  • the first determining unit 1101 is further configured to obtain the first filter identification information and the second filter identification information when the components to be processed include the second color component and the third color component; wherein, the first The filter identification information is used to indicate whether to filter the second color component of the attribute information of the reconstructed point cloud, and the second filter identification information is used to indicate whether to perform filter processing to the third color component of the attribute information of the reconstructed point cloud.
  • the first determining unit 1101 is further configured to, if at least one of the first filtering identification information and the second filtering identification information is a first value, then determine that the filtering identification information indicates to perform filtering processing on the reconstructed point cloud; if If all of the first filtering identification information and the second filtering identification information are the second value, it is determined that the filtering identification information indicates that no filtering process is performed on the reconstructed point cloud.
  • the encoding unit 1103 is further configured to encode only the filter identification information if the filter identification information indicates that the attribute information of the reconstructed point cloud is not to be filtered, and write the obtained encoded bits into the code stream.
  • the first determining unit 1101 is further configured to obtain the first filter identification information and the second filter identification information when the components to be processed include the first color component, the second color component and the third color component and third filtering identification information; wherein, the first filtering identification information is used to indicate whether to perform the first filtering process on the second color component of the attribute information of the reconstructed point cloud, and the second filtering identification information is used to indicate whether to perform the first filtering process on the reconstructed point cloud's The second filtering process is performed on the third color component of the attribute information, and the third filtering identification information is used to indicate whether to perform the third filtering process on the first color component of the attribute information of the reconstructed point cloud.
  • the filter corresponding to the third filtering process is different from the filters corresponding to the first filtering process and the second filtering process.
  • the first determination unit 1101 is further configured to determine the value of K according to the number of points of the reconstructed point cloud; and/or, determine the value of K according to the quantitative parameters of the reconstructed point cloud; and/or, The value of K is determined according to the neighborhood difference value of the points in the reconstructed point cloud; wherein, the neighborhood difference value is calculated based on the component difference between the attribute information of the point and the attribute information of at least one neighboring point.
  • the first determination unit 1101 is further configured to determine the geometric video frame and the texture video frame according to the initial point cloud; and determine the predicted value of the geometric video frame and the predicted value of the texture video frame;
  • the encoding unit 1103 is further configured to determine the residual value of the geometric video frame according to the original value of the geometric video frame and the predicted value of the geometric video frame, encode the residual value of the geometric video frame, and write the obtained coded bits into code stream; and according to the original value of the texture video frame and the predicted value of the texture video frame, determine the residual value of the texture video frame, encode the residual value of the texture video frame, and write the obtained coded bits into the code stream.
  • a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course it may also be a module, or it may be non-modular.
  • each component in this embodiment may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software function modules.
  • the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the technical solution of this embodiment is essentially or It is said that the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can It is a personal computer, a server, or a network device, etc.) or a processor (processor) that executes all or part of the steps of the method described in this embodiment.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other various media that can store program codes.
  • the embodiment of the present application provides a computer storage medium, which is applied to the encoder 110, and the computer storage medium stores a computer program, and when the computer program is executed by the first processor, it implements any one of the preceding embodiments. Methods.
  • the encoder 110 may include: a first communication interface 1201 , a first memory 1202 and a first processor 1203 ; each component is coupled together through a first bus system 1204 .
  • the first bus system 1204 is used to realize connection and communication between these components.
  • the first bus system 1204 also includes a power bus, a control bus and a status signal bus.
  • the various buses are labeled as first bus system 1204 in FIG. 12 . in,
  • the first communication interface 1201 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;
  • the first memory 1202 is used to store computer programs that can run on the first processor 1203;
  • the first processor 1203 is configured to, when running the computer program, execute:
  • the filter identification information indicates that the attribute information of the reconstructed point cloud is to be filtered
  • the filter coefficients corresponding to the filter identification information and the attribute information are encoded, and the obtained encoded bits are written into the code stream.
  • the first memory 1202 in the embodiment of the present application may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories.
  • the non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electronically programmable Erase Programmable Read-Only Memory (Electrically EPROM, EEPROM) or Flash.
  • the volatile memory can be Random Access Memory (RAM), which acts as external cache memory.
  • RAM Static Random Access Memory
  • DRAM Dynamic Random Access Memory
  • SRAM Dynamic Random Access Memory
  • Synchronous Dynamic Random Access Memory Synchronous Dynamic Random Access Memory
  • SDRAM double data rate synchronous dynamic random access memory
  • Double Data Rate SDRAM DDRSDRAM
  • enhanced SDRAM ESDRAM
  • Synchlink DRAM SLDRAM
  • Direct Memory Bus Random Access Memory Direct Rambus RAM, DRRAM
  • the first memory 1202 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
  • the first processor 1203 may be an integrated circuit chip, which has a signal processing capability. In the implementation process, each step of the above method may be implemented by an integrated logic circuit of hardware in the first processor 1203 or an instruction in the form of software.
  • the above-mentioned first processor 1203 may be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a ready-made programmable gate array (Field Programmable Gate Array, FPGA) Or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • DSP Digital Signal Processor
  • ASIC Application Specific Integrated Circuit
  • FPGA Field Programmable Gate Array
  • Various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed.
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register.
  • the storage medium is located in the first memory 1202, and the first processor 1203 reads the information in the first memory 1202, and completes the steps of the above method in combination with its hardware.
  • the embodiments described in this application may be implemented by hardware, software, firmware, middleware, microcode or a combination thereof.
  • the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing device (DSP Device, DSPD), programmable Logic device (Programmable Logic Device, PLD), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other devices used to perform the functions described in this application electronic unit or its combination.
  • the techniques described herein can be implemented through modules (eg, procedures, functions, and so on) that perform the functions described herein.
  • Software codes can be stored in memory and executed by a processor. Memory can be implemented within the processor or external to the processor.
  • the first processor 1203 is further configured to execute the method described in any one of the foregoing embodiments when running the computer program.
  • This embodiment provides an encoder, which may include a first determining unit, a first filtering unit, and an encoding unit.
  • the encoder uses the initial point cloud and the reconstructed point cloud to calculate the filter coefficients for filtering processing, and after determining to filter the reconstructed point cloud, the corresponding filter coefficients are passed to the decoder; correspondingly, the decoder It can be directly decoded to obtain the filter coefficient, and then use the filter coefficient to filter the reconstructed point cloud, thereby realizing the optimization of the reconstructed point cloud and improving the quality of the point cloud; in addition, the encoder uses the initial point cloud and the reconstructed point cloud to calculate When performing filter coefficients for filtering processing, it also solves the problem that the midpoint of the initial point cloud and the midpoint of the reconstructed point cloud are difficult to match, and by filtering the attribute information of the reconstructed point cloud, not only the quality of the reconstructed point cloud is enhanced, Moreover, the quality of the video frame can also be improved, further improving the compression efficiency
  • FIG. 13 shows a schematic diagram of the composition and structure of a decoder 130 provided in the embodiment of the present application.
  • the decoder 130 may include: a decoding unit 1301 and a second filtering unit 1302; wherein,
  • the decoding unit 1301 is configured to decode the code stream and determine the filter identification information; wherein the filter identification information is used to determine whether to filter the reconstructed point cloud corresponding to the video frame to be processed; and if the filter identification information indicates the attribute information of the reconstructed point cloud Perform filtering processing, then decode the code stream, and determine the filter coefficient corresponding to the attribute information;
  • the second filtering unit 1302 is configured to determine the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient corresponding to the attribute information.
  • the decoder 130 may further include a second determining unit 1303; wherein, the video frames to be processed include texture video frames and geometric video frames;
  • the decoding unit 1301 is further configured to decode the code stream, and determine the residual value of the texture video frame and the residual value of the geometric video frame;
  • the second determination unit 1303 is configured to, after determining the predicted value of the texture video frame, determine the reconstructed value of the texture video frame according to the predicted value of the texture video frame and the residual value of the texture video frame; and determine the prediction of the geometric video frame After the value, the reconstruction value of the geometric video frame is determined according to the predicted value of the geometric video frame and the residual value of the geometric video frame; and the reconstructed point cloud is obtained based on the reconstruction value of the texture video frame and the reconstruction value of the geometric video frame.
  • the second determining unit 1303 is further configured to determine the filter coefficient corresponding to the attribute information without performing decoding code stream if the filter identification information indicates that the attribute information of the reconstructed point cloud is not to be filtered.
  • the attribute information includes a color component
  • the color component includes at least one of the following: a first color component, a second color component, and a third color component; wherein, if the color component conforms to the RGB color space, then determine the first color component
  • the first color component, the second color component and the third color component are: R component, G component, B component; if the color component conforms to the YUV color space, then determine the first color component, the second color component and the third color component in order For: Y component, U component, V component.
  • the decoding unit 1301 is further configured to decode the code stream and determine the filter identification information of the component to be processed; wherein the component to be processed includes at least one of the following: the first color component, the second color component and the third color component The color component, and the filter identification information of the component to be processed is used to indicate whether to perform filtering processing on the component to be processed of the attribute information of the reconstructed point cloud.
  • the second determination unit 1303 is further configured to determine to perform filtering processing on the component to be processed of the attribute information of the reconstructed point cloud if the value of the filter identification information of the component to be processed is the first value; If the value of the filter identification information of the processing component is the second value, then it is determined not to filter the component to be processed of the attribute information of the reconstructed point cloud;
  • the decoding unit 1301 is further configured to decode the code stream if the value of the filter identification information of the component to be processed is the first value, and determine the filter coefficient corresponding to the component to be processed of the attribute information.
  • the decoding unit 1301 is further configured to decode the code stream when the components to be processed include the second color component and the third color component, and determine the first filter identification information and the second filter identification information; wherein, the first The filter identification information is used to indicate whether to filter the second color component of the attribute information of the reconstructed point cloud, and the second filter identification information is used to indicate whether to perform filter processing to the third color component of the attribute information of the reconstructed point cloud.
  • the decoding unit 1301 is further configured to decode the code stream and determine the filter coefficient corresponding to the second color component if the first filter identification information indicates that the second color component of the attribute information of the reconstructed point cloud is to be filtered. and/or, if the second filter identification information indicates that the third color component of the attribute information of the reconstructed point cloud is to be filtered, then decode the code stream and determine the filter coefficient corresponding to the third color component;
  • the second determining unit 1303 is further configured to determine the filtered point cloud corresponding to the reconstructed point cloud according to the filter coefficient corresponding to the second color component and/or the filter coefficient corresponding to the third color component.
  • the decoding unit 1301 is further configured to decode the code stream and determine the first filter identification information and the second filter identification information when the components to be processed include the first color component, the second color component and the third color component and third filtering identification information; wherein, the first filtering identification information is used to indicate whether to perform the first filtering process on the second color component of the attribute information of the reconstructed point cloud, and the second filtering identification information is used to indicate whether to perform the first filtering process on the reconstructed point cloud's The second filtering process is performed on the third color component of the attribute information, and the third filtering identification information is used to indicate whether to perform the third filtering process on the first color component of the attribute information of the reconstructed point cloud.
  • the filter corresponding to the third filtering process is different from the filters corresponding to the first filtering process and the second filtering process.
  • the second determination unit 1303 is further configured to determine K target points corresponding to the first point in the reconstructed point cloud; wherein, the K target points include the first point and (K -1) neighbor points, K is an integer greater than 1, the first point represents any point in the reconstructed point cloud; and filter the K target points corresponding to the first point in the reconstructed point cloud according to the filter coefficient corresponding to the attribute information Processing, determining the filter value of the attribute information of the first point in the reconstructed point cloud; and after determining the filter value of the attribute information of at least one point in the reconstructed point cloud, determining the filtered point according to the filter value of the attribute information of at least one point cloud.
  • the second determination unit 1303 is further configured to search for a preset value of candidate points in the reconstructed point cloud using the K-nearest neighbor search method based on the first point in the reconstructed point cloud; and calculate the first point and The distance value between the preset value candidate points, select (K-1) distance values from the obtained preset value distance values, and the (K-1) distance values are all less than the preset value distance values and determine (K-1) neighbor points according to the candidate points corresponding to (K-1) distance values, and determine the first point and (K-1) neighbor points as the first point corresponding K target points.
  • the second determination unit 1303 is further configured to perform color space conversion on the filtered point cloud if the color components of the points in the filtered point cloud do not conform to the RGB color space, so that the filtered point cloud in The color components of the points conform to the RGB color space.
  • a "unit” may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course it may also be a module, or it may be non-modular.
  • each component in this embodiment may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software function modules.
  • the integrated units are implemented in the form of software function modules and are not sold or used as independent products, they can be stored in a computer-readable storage medium.
  • this embodiment provides a computer storage medium, which is applied to the decoder 130, and the computer storage medium stores a computer program, and when the computer program is executed by the second processor, any one of the preceding embodiments is implemented. the method described.
  • the decoder 130 may include: a second communication interface 1401 , a second memory 1402 , and a second processor 1403 ; each component is coupled together through a second bus system 1404 .
  • the second bus system 1404 is used to realize connection and communication between these components.
  • the second bus system 1404 includes not only a data bus, but also a power bus, a control bus and a status signal bus. However, for clarity of illustration, the various buses are labeled as the second bus system 1404 in FIG. 14 . in,
  • the second communication interface 1401 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;
  • the second memory 1402 is used to store computer programs that can run on the second processor 1403;
  • the second processor 1403 is configured to, when running the computer program, execute:
  • the code stream is decoded to determine the filter coefficient corresponding to the attribute information
  • the filtered point cloud corresponding to the reconstructed point cloud is determined.
  • the second processor 1403 is further configured to execute the method described in any one of the foregoing embodiments when running the computer program.
  • the hardware function of the second memory 1402 is similar to that of the first memory 1202, and the hardware function of the second processor 1403 is similar to that of the first processor 1203; details will not be described here.
  • This embodiment provides a decoder, which may include a decoding unit and a second filtering unit.
  • the decoder can directly decode to obtain the filter coefficients, and then use the filter coefficients to filter the reconstructed point cloud, thereby realizing the optimization of the reconstructed point cloud and improving the quality of the point cloud; in addition, the encoder uses the initial point cloud and the reconstruction
  • the point cloud calculation is used to filter the filter coefficients, it also solves the problem that the midpoint of the initial point cloud and the midpoint of the reconstructed point cloud are difficult to match, and by filtering the attribute information of the reconstructed point cloud, it not only realizes the reconstruction of the point cloud
  • the quality of the video is enhanced, and the quality of the video frame can also be improved, which further improves the compression efficiency, thereby saving the bit rate and improving the encoding and decoding efficiency.
  • FIG. 15 shows a schematic diagram of the composition and structure of a codec system provided by the embodiment of the present application.
  • the codec system 150 may include an encoder 1501 and a decoder 1502 .
  • the encoder 1501 may be the encoder described in any one of the foregoing embodiments
  • the decoder 1502 may be the decoder described in any one of the foregoing embodiments.
  • the encoder 1501 uses the initial point cloud and the reconstructed point cloud to calculate the filter coefficients for filtering processing, and only after it is determined to perform filtering processing on the reconstructed point cloud, the corresponding The filter coefficients are passed to the decoder; correspondingly, the decoder 1502 can directly decode the filter coefficients, and then use the filter coefficients to filter the reconstructed point cloud, thereby realizing the optimization of the reconstructed point cloud and improving the quality of the point cloud;
  • the encoder uses the initial point cloud and the reconstructed point cloud to calculate the filter coefficients for filtering processing, it also solves the problem that the midpoint of the initial point cloud and the reconstructed point cloud are difficult to match, and through the attributes of the reconstructed point cloud
  • the information is filtered, which not only enhances the quality of the reconstructed point cloud, but also improves the quality of the video frame, which further improves the compression efficiency, thereby saving the bit rate and improving the
  • the initial point cloud and the reconstructed point cloud corresponding to the initial point cloud are determined; according to the initial point cloud and the reconstructed point cloud, the filter coefficient of the attribute information in the reconstructed point cloud is determined; according to the filtering coefficient of the attribute information coefficient to determine the filtered point cloud corresponding to the reconstructed point cloud; determine the filter identification information according to the reconstructed point cloud and the filtered point cloud;
  • the filter coefficients corresponding to the attribute information are encoded, and the obtained encoded bits are written into the code stream.
  • the filter identification information is determined by decoding the code stream; wherein, the filter identification information is used to determine whether to filter the reconstructed point cloud corresponding to the video frame to be processed; if the filter identification information indicates that the attribute information of the reconstructed point cloud is For filtering processing, the code stream is decoded, and the filter coefficient corresponding to the attribute information is determined; according to the filter coefficient corresponding to the attribute information, the filtered point cloud corresponding to the reconstructed point cloud is determined.
  • the encoder uses the initial point cloud and the reconstructed point cloud to calculate the filter coefficients for filtering processing, and after determining to filter the reconstructed point cloud, the corresponding filter coefficients are passed to the decoder; correspondingly, the decoder It can be directly decoded to obtain the filter coefficient, and then use the filter coefficient to filter the reconstructed point cloud, thereby realizing the optimization of the reconstructed point cloud and improving the quality of the point cloud; in addition, the encoder uses the initial point cloud and the reconstructed point cloud to calculate When performing filter coefficients for filtering processing, it also solves the problem that the midpoint of the initial point cloud and the midpoint of the reconstructed point cloud are difficult to match, and by filtering the attribute information of the reconstructed point cloud, not only the quality of the reconstructed point cloud is enhanced, Moreover, the quality of the video frame can also be improved, further improving the compression efficiency, thereby saving the bit rate, and improving the encoding and decoding performance.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本申请实施例公开了一种编解码方法、码流、编码器、解码器以及存储介质,该方法包括:解码码流,确定滤波标识信息;其中,滤波标识信息用于确定是否对待处理视频帧对应的重建点云进行滤波处理;若滤波标识信息指示对重建点云的属性信息进行滤波处理,则解码码流,确定属性信息对应的滤波系数;根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云。这样,不仅能够提升点云的质量,还能够提升压缩效率,进而节省码率,提升编解码性能。

Description

编解码方法、码流、编码器、解码器以及存储介质 技术领域
本申请实施例涉及视频编解码技术领域,尤其涉及一种编解码方法、码流、编码器、解码器以及存储介质。
背景技术
目前,在基于视频的点云压缩(Video-based Point Cloud Compression,V-PCC)编解码框架中,既有对有损压缩后解码得到的视频帧进行平滑的操作,也有对重建点云进行后处理的算法。在对解码后的视频帧进行基础性重建后,只进行了图像或点云的平滑操作,因此重建点云的质量依然有较大的提升空间。
然而,对于V-PCC编解码框架的视频帧进行维纳滤波处理后,可以较好地提升重建点云的几何属性质量;但是已有的解决方案对于颜色属性的提升并不理想,影响了整个点云的质量。
发明内容
本申请实施例提供一种编解码方法、码流、编码器、解码器以及存储介质,不仅能够提升点云的质量,还能够提升压缩效率,进而节省码率,提升编解码性能。
本申请实施例的技术方案可以如下实现:
第一方面,本申请实施例提供了一种解码方法,应用于解码器,该方法包括:
解码码流,确定滤波标识信息;其中,滤波标识信息用于确定是否对待处理视频帧对应的重建点云进行滤波处理;
若滤波标识信息指示对重建点云的属性信息进行滤波处理,则解码码流,确定属性信息对应的滤波系数;
根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云。
第二方面,本申请实施例提供了一种编码方法,应用于编码器,该方法包括:
确定初始点云以及初始点云对应的重建点云;
根据初始点云和重建点云,确定重建点云中属性信息的滤波系数;
根据属性信息的滤波系数,确定重建点云对应的滤波后点云;
根据重建点云和滤波后点云,确定滤波标识信息;
若滤波标识信息指示对重建点云的属性信息进行滤波处理,则对滤波标识信息和属性信息对应的滤波系数进行编码,将所得到的编码比特写入码流。
第三方面,本申请实施例提供了一种码流,该码流是根据待编码信息进行比特编码生成的;其中,待编码信息包括下述至少之一:几何视频帧的残差值、纹理视频帧的残差值、滤波标识信息和滤波系数。
第四方面,本申请实施例提供了一种编码器,该编码器包括第一确定单元、第一滤波单元和编码单元;其中,
第一确定单元,配置为确定初始点云以及初始点云对应的重建点云;以及根据初始点云和重建点云,确定重建点云中属性信息的滤波系数;
第一滤波单元,配置为根据属性信息的滤波系数,确定重建点云对应的滤波后点云;
第一确定单元,还配置为根据重建点云和滤波后点云,确定滤波标识信息;
编码单元,配置为若滤波标识信息指示对重建点云的属性信息进行滤波处理,则对滤波标识信息和属性信息对应的滤波系数进行编码,将所得到的编码比特写入码流。
第五方面,本申请实施例提供了一种编码器,该编码器包括第一存储器和第一处理器;其中,
第一存储器,用于存储能够在第一处理器上运行的计算机程序;
第一处理器,用于在运行所述计算机程序时,执行如第二方面所述的方法。
第六方面,本申请实施例提供了一种解码器,该解码器包括解码单元和第二滤波单元;其中,
解码单元,配置为解码码流,确定滤波标识信息;其中,滤波标识信息用于确定是否对待处理视频帧对应的重建点云进行滤波处理;以及若滤波标识信息指示对重建点云的属性信息进行滤波处理,则解 码码流,确定属性信息对应的滤波系数;
第二滤波单元,配置为根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云。
第七方面,本申请实施例提供了一种解码器,该解码器包括第二存储器和第二处理器;其中,
第二存储器,用于存储能够在第二处理器上运行的计算机程序;
第二处理器,用于在运行计算机程序时,执行如第一方面所述的方法。
第八方面,本申请实施例提供了一种计算机存储介质,该计算机存储介质存储有计算机程序,计算机程序被执行时实现如第一方面所述的方法、或者实现如第二方面所述的方法。
本申请实施例提供了一种编解码方法、码流、编码器、解码器以及存储介质,在编码器中,确定初始点云以及初始点云对应的重建点云;根据初始点云和重建点云,确定重建点云中属性信息的滤波系数;根据属性信息的滤波系数,确定重建点云对应的滤波后点云;根据重建点云和滤波后点云,确定滤波标识信息;若滤波标识信息指示对重建点云的属性信息进行滤波处理,则对滤波标识信息和属性信息对应的滤波系数进行编码,将所得到的编码比特写入码流。在解码器中,通过解码码流,确定滤波标识信息;其中,滤波标识信息用于确定是否对待处理视频帧对应的重建点云进行滤波处理;若滤波标识信息指示对重建点云的属性信息进行滤波处理,则解码码流,确定属性信息对应的滤波系数;根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云。这样,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数,并在确定对重建点云进行滤波处理之后,才将对应的滤波系数传递至解码器;相应地,解码器可以直接解码获得滤波系数,然后使用滤波系数对重建点云进行滤波处理,从而实现了对重建点云的优化,能够提升点云的质量;另外,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数时,还解决了初始点云中点和重建点云中点难以匹配的问题,并且通过对重建点云的属性信息进行滤波处理,不仅实现重建点云的质量增强,而且使得视频帧的质量也能够得到提升,进一步提升了压缩效率,进而节省码率,提高了编解码效率。
附图说明
图1为一种V-PCC编码器的组成框架示意图;
图2为一种V-PCC解码器的组成框架示意图;
图3为本申请实施例提供的一种解码方法的流程示意图一;
图4为本申请实施例提供的一种解码方法的流程示意图二;
图5为本申请实施例提供的一种编码方法的流程示意图一;
图6为本申请实施例提供的一种编码方法的流程示意图二;
图7为本申请实施例提供的一种编解码的总体流程示意图;
图8为本申请实施例提供的一种编码端滤波处理的流程示意图;
图9为本申请实施例提供的一种解码端滤波处理的流程示意图;
图10为本申请实施例提供的一种基于CTC_C2测试条件的测试结果示意图;
图11为本申请实施例提供的一种编码器的组成结构示意图;
图12为本申请实施例提供的一种编码器的具体硬件结构示意图;
图13为本申请实施例提供的一种解码器的组成结构示意图;
图14为本申请实施例提供的一种解码器的具体硬件结构示意图;
图15为本申请实施例提供的一种编解码系统的组成结构示意图。
具体实施方式
为了能够更加详尽地了解本申请实施例的特点与技术内容,下面结合附图对本申请实施例的实现进行详细阐述,所附附图仅供参考说明之用,并非用来限定本申请实施例。
除非另有定义,本文所使用的所有的技术和科学术语与属于本申请的技术领域的技术人员通常理解的含义相同。本文中所使用的术语只是为了描述本申请实施例的目的,不是旨在限制本申请。
在以下的描述中,涉及到“一些实施例”,其描述了所有可能实施例的子集,但是可以理解,“一些实施例”可以是所有可能实施例的相同子集或不同子集,并且可以在不冲突的情况下相互结合。还需要指出,本申请实施例所涉及的术语“第一\第二\第三”仅是用于区别类似的对象,不代表针对对象的特定排序,可以理解地,“第一\第二\第三”在允许的情况下可以互换特定的顺序或先后次序,以使这里描述的本申请实施例能够以除了在这里图示或描述的以外的顺序实施。
对本申请实施例进行进一步详细说明之前,先对本申请实施例中涉及的名词和术语进行说明,本申 请实施例中涉及的名词和术语适用于如下的解释:
基于几何的点云压缩(Geometry-based Point Cloud Compression,G-PCC或GPCC)
基于视频的点云压缩(Video-based Point Cloud Compression,V-PCC或VPCC)
维纳滤波器(Wiener Filter)
K近邻(K Nearest Neighbor,KNN)
细节层次(Level of Detail,LOD)
三维(Three Dimension,3D)
峰值信噪比(Peak Signal to Noise Ratio,PSNR)
和方差(Sum Squared Error,SSE)
最小均方误差(Minimum Mean Squared Error,MMSE)
率失真优化(Rate Distortion Optimization,RDO)
亮度分量(Luminance,L或Y)
蓝色色度分量(Chroma blue,Cb)
红色色度分量(Chroma red,Cr)
点云是物体表面的三维表现形式,通过光电雷达、激光雷达、激光扫描仪、多视角相机等采集设备,可以采集得到物体表面的点云(数据)。
点云(Point Cloud)是指海量三维点的集合,点云中的点可以包括点的位置信息和点的属性信息。例如,点的位置信息可以是点的三维坐标信息。点的位置信息也可称为点的几何信息。例如,点的属性信息可包括颜色信息和/或反射率等等。例如,颜色信息可以是任意一种色彩空间上的信息。例如,颜色信息可以是RGB信息。其中,R表示红色(Red,R),G表示绿色(Green,G),B表示蓝色(Blue,B)。再如,颜色信息可以是亮度色度(YCbCr,YUV)信息。其中,Y表示明亮度,Cb(U)表示蓝色色度,Cr(V)表示红色色度。
根据激光测量原理得到的点云,点云中的点可以包括点的三维坐标信息和点的激光反射强度(reflectance)。再如,根据摄影测量原理得到的点云,点云中的点可以可包括点的三维坐标信息和点的颜色信息。再如,结合激光测量和摄影测量原理得到点云,点云中的点可以可包括点的三维坐标信息、点的激光反射强度(reflectance)和点的颜色信息。
点云可以按获取的途径分为:
第一类静态点云:即物体是静止的,获取点云的设备也是静止的;
第二类动态点云:物体是运动的,但获取点云的设备是静止的;
第三类动态获取点云:获取点云的设备是运动的。
例如,按点云的用途分为两大类:
类别一:机器感知点云,其可以用于自主导航系统、实时巡检系统、地理信息系统、视觉分拣机器人、抢险救灾机器人等场景;
类别二:人眼感知点云,其可以用于数字文化遗产、自由视点广播、三维沉浸通信、三维沉浸交互等点云应用场景。
由于点云是海量点的集合,存储点云不仅会消耗大量的内存,而且不利于传输,也没有这么大的带宽可以支持将点云不经过压缩直接在网络层进行传输,因此,需要对点云进行压缩。
截止目前,可对点云进行压缩的点云编码框架可以是运动图像专家组(Moving Picture Experts Group,MPEG)提供的G-PCC编解码框架或V-PCC编解码框架,也可以是音视频编码标准(Audio Video Standard,AVS)提供的AVS-PCC编解码框架。其中,G-PCC编解码框架可用于针对第一类静态点云和第三类动态获取点云进行压缩,V-PCC编解码框架可用于针对第二类动态点云进行压缩。在本申请实施例中,这里主要是针对V-PCC编解码框架进行描述。
可以理解,图1为一种V-PCC编码器的组成框架示意图。如图1所示,该V-PCC编码器应用于点云编码器。在该V-PCC编码框架中,针对待编码的输入点云帧,首先通过3D块(patch)划分,得到多个patch块。根据这些patch信息,可以进行patch封装,然后结合输入点云帧生成几何图像,再对几何图像进行图像填充操作,在几何填充后可以进行视频压缩,得到几何子码流;另外,根据这些patch信息以及输入点云帧生成属性图像,同时还需要考虑几何平滑操作;然后再对属性图像进行图像填充操作,在属性填充后也可以进行视频压缩,得到属性子码流;针对patch信息进行patch序列压缩,得到patch子码流;而在获得占有图(Occupancy map)之后,针对占有图进行视频压缩,得到占有图子码流;将这四个子码流发送至整合器(Multiplexer)进行处理,得到压缩码流。
图2为一种V-PCC解码器的组成框架示意图。如图2所示,该V-PCC解码器应用于点云解码器。在该V-PCC解码框架中,针对接收到的压缩码流,在经过分离器(Demultiplexer)之后,可以得到高层 语法信息、patch子码流、占有图子码流、几何子码流和属性子码流;其中,高层语法信息可以是序列参数集(Sequence Parameter Sets,SPS)信息,对SPS信息进行解析之后,可以将所得到的SPS解析信息送入后续步骤;另外,patch子码流在经过patch序列解压缩处理后,可以得到patch信息;占有图子码流在经过视频解压缩处理后,可以得到占有图;几何子码流在经过视频解压缩处理后,可以得到几何图像;属性子码流在经过视频解压缩处理后,可以得到属性图像;然后这些SPS解析信息、patch信息、占有图、几何图像和属性图像输入到几何/属性重建、几何后处理(例如进行平滑操作)和属性迁移及平滑处理等操作之后,可以恢复出重建点云。
在V-PCC点云平滑后处理操作中,既有对有损压缩后解码得到的视频进行平滑操作,也有对重建点云进行后处理的算法。图1和图2展示了V-PCC编码和解码的框架示意图。具体地,V-PCC中有两次点云的平滑处理:第一次是重建帧生成后对其进行平滑处理;第二次是生成重建点云后,对其进行几何的平滑处理。
(1)对于纹理视频帧,V-PCC针对在三维空间中非相邻的patch块在图像内可能非常临近,基于patch的编码器可能将相邻像素混淆使得重建伪影出现的问题,提出了颜色平滑算法,在重建帧中利用占有图找到patch边界点对应的像素,然后利用中值滤波进行平滑处理。对于几何视频帧,V-PCC针对重建点云中patch可能不连续的问题,提出了边界滤波算法,找到patch边界点对应的像素,局部改变patch边缘的深度值。
(2)对于重建点云,针对patch不连续的问题,添加3D点,使边界连续。针对由于失真导致外点和噪点产生的问题,将重建点云网格化(grid),利用网格的中心对patch边缘点进行三线性滤波,提升点云的视觉效果。
然而,V-PCC编解码框架在对解码后视频帧进行基础性重建后,只进行了图像或点云的平滑操作,因而重建点云的质量依然有较大提升空间。为了进一步提升压缩性能,在码流大小变化不大的基础上得到质量增强的点云是很有必要的。
同时,对于针对V-PCC编解码框架中视频帧进行维纳滤波的方法,可以较好地提升重建点云的几何属性的质量。但是该方法对于颜色属性的提升并不理想。实验证明,如果未对纹理视频帧帧进行处理,通过对几何视频帧的处理,只能使重建点云颜色属性的BD-TotalRate具有-0.1%左右的增益;如果对纹理视频帧同样进行维纳滤波处理,尽管视频帧的PSNR性能可以提升近0.2dB,但是最终重建点云颜色属性的BD-TotalRate只在少数几个点云序列具有增益,对于大多数的点云序列该指标不降反升。因此,对于重建点云颜色属性,需要一种更合适的编解码方法进行处理以提升点云质量与压缩性能。
这样,本申请实施例提供了一种编解码方法,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数,并在确定对重建点云进行滤波处理之后,才将对应的滤波系数传递至解码器;相应地,解码器可以直接解码获得滤波系数,然后使用滤波系数对重建点云进行滤波处理,从而实现了对重建点云的优化,能够提升点云的质量;另外,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数时,还解决了初始点云中点和重建点云中点难以匹配的问题,并且通过对重建点云的属性信息进行滤波处理,不仅能够实现重建点云的质量增强,尤其是对几何有损的重建点云属性信息的质量增强,使得视频帧的质量也能够得到提升,进一步提升了压缩效率,进而节省码率,提高了编解码效率。
下面将结合附图对本申请各实施例进行清楚、完整的描述。
在本申请的一实施例中,参见图3,其示出了本申请实施例提供的一种解码方法的流程示意图一。如图3所示,该方法可以包括:
S301:解码码流,确定滤波标识信息;其中,滤波标识信息用于确定是否对待处理视频帧对应的重建点云进行滤波处理。
需要说明的是,本申请实施例所述的解码方法具体是指点云解码方法,可以应用于V-PCC中的点云解码器(本申请实施例中,可简称为“解码器”)。但是需要注意的是,本申请实施例所述的解码方法并不局限于V-PCC领域,对于其他有损点云的质量增强而言,所有具有属性信息失真的点云都可使用。这里以V-PCC为例,是因为在V-PCC上执行可以使其发挥最大的优势,能够兼具对几何与属性的处理,点云整体质量得到提高,具有更好的效果。
还需要说明的是,在本申请实施例中,解码器可以先解码码流,从而确定滤波标识信息。其中,滤波标识信息可以用于确定是否对待处理视频帧对应的重建点云进行滤波处理。具体地,滤波标识信息还可以用于确定对重建点云中的哪一个或者哪几个颜色分量进行滤波处理。
还需要说明的是,在本申请实施例中,对于初始点云中的一个点,在对该一个点进行解码时,其可以作为初始点云中的待解码点,而该一个点的周围有多个已解码点。
进一步地,在本申请实施例中,对于初始点云中的一个点,其对应一个几何信息和一个属性信息; 其中,几何信息表征该点的空间位置,属性信息表征该点的属性值(如颜色分量值)。
在这里,属性信息可以包括颜色分量,具体为任意颜色空间的颜色信息。示例性地,属性信息可以为RGB空间的颜色信息,也可以为YUV空间的颜色信息,还可以为YCbCr空间的颜色信息等等,本申请实施例不作具体限定。
进一步地,在本申请实施例中,颜色分量可以包括第一颜色分量、第二颜色分量和第三颜色分量。这样,如果颜色分量符合RGB颜色空间,那么第一颜色分量、第二颜色分量和第三颜色分量可以依次为:R分量、G分量、B分量;如果颜色分量符合YUV颜色空间,那么第一颜色分量、第二颜色分量和第三颜色分量可以依次为:Y分量、U分量、V分量;如果颜色分量符合YCbCr颜色空间,那么第一颜色分量、第二颜色分量和第三颜色分量可以依次为:Y分量、Cb分量、Cr分量,但是并不作具体限定。
还需要说明的是,在本申请实施例中,对于初始点云中的一个点,该点的属性信息可以为颜色分量,也可以是反射率或者其它属性,本申请实施例也不作具体限定。
可以理解的是,在V-PCC的实现过程中,待处理视频帧可以包括纹理视频帧和几何视频帧,具体对于初始点云生成的几何视频帧和纹理视频帧压缩完成后,根据纹理视频帧的残差值和几何视频帧的残差值可以构建出重建点云。因此,在一些实施例中,该方法还可以包括:
解码码流,确定纹理视频帧的残差值和几何视频帧的残差值;
在确定纹理视频帧的预测值后,根据纹理视频帧的预测值和纹理视频帧的残差值,确定纹理视频帧的重建值;
在确定几何视频帧的预测值后,根据几何视频帧的预测值和几何视频帧的残差值,确定几何视频帧的重建值;
基于纹理视频帧的重建值和几何视频帧的重建值,得到重建点云。
需要说明的是,在本申请实施例中,码流可以包括:辅助信息子码流、几何子码流和纹理子码流等部分。其中,辅助信息子码流用于解码获得占有图及patch有关的辅助信息,几何子码流用于解码获得几何视频帧的残差值,纹理子码流用于解码获得纹理视频帧的残差值。另外,需要注意的是,本申请实施例所述的滤波标识信息以及对应的滤波系数是通过解析纹理子码流得到的。
也就是说,通过解析纹理子码流,确定纹理视频帧的残差值;在确定纹理视频帧的预测值后,可以根据纹理视频帧的预测值和纹理视频帧的残差值,确定出纹理视频帧的重建值;以及通过解析几何子码流,确定几何视频帧的残差值;在确定几何视频帧的预测值后,可以根据几何视频帧的预测值和几何视频帧的残差值,确定出几何视频帧的重建值;然后基于纹理视频帧的重建值和几何视频帧的重建值,能够得到重建点云。
还需要说明的是,在本申请实施例中,根据V-PCC的实现过程,对由初始点云生成的几何视频帧和纹理视频帧压缩完成后,解码端可以据此进行点云的重建,得到重建点云。另外,这里的重建点云可以是解码后输出的重建点云,也可以是用作解码后续点云参考;此外,这里的重建点云不仅可以在在预测环路内,即作为inloop filter使用,可用作解码后续点云的参考;也可以在预测环路外,即作为post filter使用,不用作解码后续点云的参考;本申请实施例对此不作具体限定。
另外,在本申请实施例中,滤波标识信息可以包括属性信息的待处理分量的滤波标识信息。具体地,在一些实施例中,对于S301来说,所述解码码流,确定滤波标识信息,可以包括:
解码码流,确定待处理分量的滤波标识信息;其中,待处理分量包括下述至少之一:第一颜色分量、第二颜色分量和第三颜色分量,且待处理分量的滤波标识信息用于指示是否对重建点云的属性信息的待处理分量进行滤波处理。
进一步地,对于解码码流,确定滤波标识信息来说,在一些实施例中,具体可以包括:
若待处理分量的滤波标识信息的取值为第一值,则确定对重建点云的属性信息的待处理分量进行滤波处理;
若待处理分量的滤波标识信息的取值为第二值,则确定不对重建点云的属性信息的待处理分量进行滤波处理。
也就是说,通过解码码流,在得到待处理分量的滤波标识信息之后;如果待处理分量的滤波标识信息的取值为第一值,那么可以确定需要对重建点云的属性信息的待处理分量进行滤波处理;如果待处理分量的滤波标识信息的取值为第二值,那么可以确定不需要对重建点云的属性信息的待处理分量进行滤波处理。示例性地,如果待处理分量为颜色分量,假定待处理分量为U分量,那么若U分量对应的滤波标识信息的取值为第一值,则可以确定需要对重建点云的属性信息的U分量进行滤波处理;反之,若U分量对应的滤波标识信息的取值为第二值,则可以确定不需要对重建点云的属性信息的U分量进行滤波处理。
还需要说明的是,在待处理分量为颜色分量的情况下,每一个颜色分量对应的滤波标识信息可以用于指示是否进行该颜色分量的滤波处理。具体地,在一些实施例中,当待处理分量仅包括第二颜色分量和第三颜色分量时,所述解码码流,确定待处理分量的滤波标识信息,可以包括:解码码流,确定第一滤波标识信息和第二滤波标识信息。
在这里,第一滤波标识信息用于指示是否对重建点云的属性信息的第二颜色分量进行滤波处理,第二滤波标识信息用于指示是否对重建点云的属性信息的第三颜色分量进行滤波处理。
可以理解地,在本申请实施例中,对于YUV颜色空间来讲,Y分量、U分量和V分量都可以进行滤波处理,但是由于Y分量相对于U分量和V分量的波动性较大,导致滤波效果不稳定,且会增加码流开销影响整体压缩效率;因此,本申请实施例主要是针对U分量和V分量进行了滤波增强处理。具体地,在一些实施例中,第二颜色分量为U分量,第三颜色分量为V分量;或者,第二颜色分量为V分量,第三颜色分量为U分量。
需要说明的是,在一种具体的实施例中,所述解码码流,确定第一滤波标识信息,可以包括:
若第一滤波标识信息的取值为第一值,则确定第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理;
若第一滤波标识信息的取值为第二值,则确定第一滤波标识信息指示不对重建点云的属性信息的第二颜色分量进行滤波处理。
还需要说明的是,在另一种具体的实施例中,所述解码码流,确定第二滤波标识信息,可以包括:
若第二滤波标识信息的取值为第一值,则第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行滤波处理;
若第二滤波标识信息的取值为第二值,则第二滤波标识信息指示不对重建点云的属性信息的第三颜色分量进行滤波处理。
需要说明的是,在本申请实施例中,以第二颜色分量和第三颜色分量为例,如果某颜色分量对应的滤波标识信息为第一值,那么可以指示对重建点云的属性信息进行该颜色分量的滤波处理;如果某颜色分量对应的滤波标识信息为第二值,那么可以指示不对重建点云的属性信息进行该颜色分量的滤波处理。
还可以理解地,仍以YUV空间为例,本申请实施例可以对Y分量进行一种滤波处理,对UV分量进行另一种滤波处理,从而使得重建点云的几何与属性质量都具有较大的提升,压缩效率进一步增加。具体地,在一些实施例中,当待处理分量包括第一颜色分量、第二颜色分量和第三颜色分量时,所述解码码流,确定待处理分量的滤波标识信息,可以包括:解码码流,确定第一滤波标识信息、第二滤波标识信息和第三滤波标识信息。
在这里,第一滤波标识信息用于指示是否对重建点云的属性信息的第二颜色分量进行第一滤波处理,第二滤波标识信息用于指示是否对重建点云的属性信息的第三颜色分量进行第二滤波处理,第三滤波标识信息用于指示是否对重建点云的属性信息的第一颜色分量进行第三滤波处理。
需要说明的是,在本申请实施例中,对于第一滤波标识信息而言,若第一滤波标识信息的取值为第一值,则确定第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行第一滤波处理;若第一滤波标识信息的取值为第二值,则确定第一滤波标识信息指示不对重建点云的属性信息的第二颜色分量进行第一滤波处理。对于第二滤波标识信息而言,若第二滤波标识信息的取值为第一值,则确定第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行第二滤波处理;若第二滤波标识信息的取值为第二值,则确定第二滤波标识信息指示不对重建点云的属性信息的第三颜色分量进行第二滤波处理。对于第三滤波标识信息而言,若第三滤波标识信息的取值为第一值,则确定第三滤波标识信息指示对重建点云的属性信息的第一颜色分量进行第三滤波处理;若第三滤波标识信息的取值为第二值,则确定第三滤波标识信息指示不对重建点云的属性信息的第一颜色分量进行第三滤波处理。
还需要说明的是,第三滤波处理对应的滤波器与第一滤波处理和第二滤波处理对应的滤波器存在不同。其中,第一滤波处理和第二滤波处理对应的滤波器相同,但是该滤波器与第三滤波处理对应的滤波器不同;或者、第一滤波处理对应的滤波器、第二滤波处理对应的滤波器与第三滤波处理对应的滤波器均不相同;或者,甚至还可以是第一滤波处理和第三滤波处理对应的滤波器相同,但是该滤波器与第二滤波处理对应的滤波器不同等等,本申请实施例不作具体限定。
示例性地,第一滤波处理和第二滤波处理对应的滤波器可以为维纳滤波器,而第三滤波处理对应的滤波器可以为自适应滤波器(工作原理与维纳滤波器相同,这里可根据邻域差异自适应选择滤波器的阶数、个数等);或者,第三滤波处理对应的滤波器也可以为维纳滤波器,但是其与第一滤波处理和第二滤波处理对应维纳滤波器的阶数不同等等;本申请实施例对此并不作任何限定。
还需要说明的是,在本申请实施例中,解码码流,确定待处理分量的滤波标识信息,可以是通过解 码码流,一次获得第一滤波标识信息、第二第二滤波标识信息和第三滤波标识信息等;但是也可以分开解码获得的;例如先解码码流,获得第一滤波标识信息;然后再通过解码码流,获得第二滤波标识信息;甚至还可以接着解码码流,获得第三滤波标识信息等,对此也不作任何限定。
另外,还需要说明的是,在本申请实施例中,第一值和第二值不同,而且第一值和第二值可以是参数形式,也可以是数字形式。具体地,这些颜色分量对应的标识信息可以是写入在概述(profile)中的参数,也可以是一个标志(flag)的取值,这里对此不作具体限定。
示例性地,以flag为例,flag的设置有两种方式:使能标志位(enable_flag)和非使能标志位(disable_flag)。假定使能标志位的取值为第一值,非使能标志位的取值为第二值;那么对于第一值和第二值而言,第一值可以设置为1,第二值可以设置为0;或者,第一值还可以设置为true,第二值还可以设置为false;但是本申请实施例并不作具体限定。
S302:若滤波标识信息指示对重建点云的属性信息进行滤波处理,则解码码流,确定属性信息对应的滤波系数。
需要说明的是,在本申请实施例中,解码器在通过解码码流确定出滤波标识信息之后,如果滤波标识信息指示对重建点云的属性信息进行滤波处理,那么可以进一步确定用于进行滤波处理的滤波系数。
还需要说明的是,在本申请实施例中,用于滤波处理的滤波器可以是自适应滤波器,例如可以是基于神经网络的滤波器、维纳滤波器等等,但是并不作具体限定。以维纳滤波器为例,本申请实施例所述的滤波系数可以用于维纳滤波,即滤波系数为维纳滤波处理的系数。
在这里,维纳滤波器(Wiener filter)是一种以最小化均方误差为最优准则的线性滤波器。在一定的约束条件下,其输出与一给定函数(通常称为期望输出)的差的平方达到最小,通过数学运算最终可变为一个托布利兹方程的求解问题。维纳滤波器又被称为最小二乘滤波器或最小平方滤波器。
可以理解的是,滤波系数可以是指待处理分量对应的滤波系数。相应地,在一些实施例中,对于S302来说,所述若滤波标识信息指示对重建点云的属性信息进行滤波处理,则解码码流,确定属性信息对应的滤波系数,可以包括:若待处理分量的滤波标识信息的取值为第一值,则解码码流,确定属性信息的待处理分量对应的滤波系数。
进一步地,在本申请实施例中,在待处理分量为颜色分量的情况下,这里的颜色分量可以包括第一颜色分量、第二颜色分量和第三颜色分量中的至少一项。下面以YUV颜色空间为例,由于Y分量相对于U分量和V分量的波动性较大,导致滤波效果不稳定,且会增加码流开销影响整体压缩效率;假定如果不对Y分量进行质量增强处理,只针对U分量和V分量进行质量增强处理。其中,第一滤波标识信息用于指示是否对重建点云的属性信息的第二颜色分量(U分量)进行滤波处理,第二滤波标识信息用于指示是否对重建点云的属性信息的第三颜色分量(V分量)进行滤波处理。相应地,在一些实施例中,对于S302来说,所述若滤波标识信息指示对重建点云的属性信息进行滤波处理,则解码码流,确定属性信息对应的滤波系数,可以包括:
若所述第一滤波标识信息指示对所述重建点云的属性信息的第二颜色分量进行滤波处理,则解码码流,确定第二颜色分量对应的滤波系数;
和/或
若所述第二滤波标识信息指示对所述重建点云的属性信息的第三颜色分量进行滤波处理,则解码码流,确定第三颜色分量对应的滤波系数。
也就是说,对于第一滤波标识信息而言,如果第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理,这时候才需要继续解码码流,以便获得第二颜色分量对应的滤波系数;如果第一滤波标识信息指示不对重建点云的属性信息的第二颜色分量进行滤波处理,这时候不再需要执行解码码流,确定第一颜色分量对应的滤波系数的步骤。对于第二滤波标识信息而言,如果第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行滤波处理,这时候才需要继续解码码流,以便获得第三颜色分量对应的滤波系数;如果第二滤波标识信息指示不对重建点云的属性信息的第三颜色分量进行滤波处理,这时候不再需要执行解码码流,确定第三颜色分量对应的滤波系数的步骤。
需要说明的是,在本申请实施例中,第一滤波标识信息和第二滤波标识信息在解码过程中可以无先后顺序,也可以具有先后顺序。在一种具体的实施例中,可以先解码获得第一滤波标识信息,然后再解码获得第二滤波标识信息,但是本申请实施例并不作具体限定。
在一种可能的实施方式中,如果需要对重建点云进行第二颜色分量和第三颜色分量的滤波处理,这时候,该方法可以包括:
解码码流,确定第一滤波标识信息;
若第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理,则解码码流,确定第二颜色分量对应的滤波系数;以及解码码流,确定第二滤波标识信息;
若第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行滤波处理,则解码码流,确定第三颜色分量对应的滤波系数。
在另一种可能的实施方式中,如果仅需要对重建点云进行第二颜色分量的滤波处理,这时候,该方法可以包括:
解码码流,确定第一滤波标识信息;
若第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理,则解码码流,确定第二颜色分量对应的滤波系数;以及解码码流,确定第二滤波标识信息;
若第二滤波标识信息指示不对的属性信息的第三颜色分量进行滤波处理,则不执行解码码流,确定第三颜色分量对应的滤波系数的步骤。
在又一种可能的实施方式中,如果仅需要对重建点云进行第三颜色分量的滤波处理,这时候,该方法可以包括:
解码码流,确定第一滤波标识信息;
若第一滤波标识信息指示不对重建点云的属性信息的第二颜色分量进行滤波处理,则不执行解码码流,确定第二颜色分量对应的滤波系数的步骤;直接解码码流,确定第二滤波标识信息;
若第二滤波标识信息指示对的属性信息的第三颜色分量进行滤波处理,则解码码流,确定第三颜色分量对应的滤波系数。
在再一种可能的实施方式中,如果既不需要对重建点云进行第二颜色分量的滤波处理,又不需要对重建点云进行第三颜色分量的滤波处理,这时候,该方法可以包括:
解码码流,确定第一滤波标识信息;
若第一滤波标识信息指示不对重建点云的属性信息的第二颜色分量进行滤波处理,则不执行解码码流,确定第二颜色分量对应的滤波系数的步骤;直接解码码流,确定第二滤波标识信息;
若第二滤波标识信息指示不对重建点云的属性信息的第三颜色分量进行滤波处理,则也不执行解码码流,确定第三颜色分量对应的滤波系数的步骤。
简单来说,在一些实施例中,该方法还可以包括:若滤波标识信息指示不对重建点云的属性信息进行滤波处理,则不执行解码码流,确定属性信息对应的滤波系数的步骤。
也就是说,在本申请实施例中,解码器在解码码流,确定第一滤波标识信息之后,如果第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理,那么解码器可以继续解码码流,获得第二颜色分量对应的滤波系数;但是如果第一滤波标识信息指示不对重建点云的属性信息的第二颜色分量进行滤波处理,那么解码器便不再需要解码获得第二颜色分量对应的滤波系数,而是进行下一颜色分量的解码,即解码码流,确定第二滤波标识信息。然后,如果第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行滤波处理,那么解码器可以继续解码码流,获得第三颜色分量对应的滤波系数;但是如果第二滤波标识信息指示不对重建点云的属性信息的第三颜色分量进行滤波处理,那么解码器便不再需要解码获得第三颜色分量对应的滤波系数。
S303:根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云。
需要说明的是,在本申请实施例中,如果滤波标识信息指示对重建点云进行属性信息的滤波处理,那么在确定属性信息对应的滤波系数之后,可以进一步利用滤波系数对重建点云进行滤波处理,从而可以获得重建点云对应的滤波后点云。
还需要说明的是,在本申请实施例中,可以对重建点云的各个颜色分量单独滤波,例如,对于YUV空间的颜色信息,可以分别确定Y分量对应的滤波系数,U分量对应的滤波系数,V分量对应的滤波系数。但是本申请实施例考虑到Y分量相对于U分量和V两分量的波动性较大,滤波效果不稳定,且会增加码流开销影响整体压缩效率,可以不涉及对Y分量的质量增强处理。也就是说,本申请实施例可以仅用于确定U分量和/或V分量对应的滤波系数。
下面以待处理分量为颜色分量,且待处理分量包括第二颜色分量和第三颜色分量为例,在一些实施例中,对于S303来说,所述根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云,可以包括:
根据所述第二颜色分量对应的滤波系数和/或所述第三颜色分量对应的滤波系数,确定所述重建点云对应的滤波后点云。
具体来讲,在一种可能的实施方式中,如果第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理、且第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行滤波处理,那么通过解码码流可以获得第二颜色分量对应的滤波系数和第三颜色分量对应的滤波系数,然后利用第二颜色分量对应的滤波系数和第三颜色分量对应的滤波系数确定出滤波后点云。
在另一种可能的实施方式中,如果第一滤波标识信息指示对重建点云的属性信息的第二颜色分量 进行滤波处理、且第二滤波标识信息指示不对重建点云的属性信息的第三颜色分量进行滤波处理,那么通过解码码流仅获得第二颜色分量对应的滤波系数,然后利用第二颜色分量对应的滤波系数确定出滤波后点云。
在又一种可能的实施方式中,如果第一滤波标识信息指示不对重建点云的属性信息的第二颜色分量进行滤波处理、且第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行滤波处理,那么通过解码码流仅获得第三颜色分量对应的滤波系数,然后利用第三颜色分量对应的滤波系数确定出滤波后点云。
进一步地,在一些实施例中,以属性信息为例,对于S303来说,根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云,可以包括:
确定所述重建点云中第一点对应的K个目标点;其中,所述K个目标点包括所述第一点以及与所述第一点相邻的(K-1)个近邻点,K为大于1的整数,所述第一点表示所述重建点云中的任意点;
根据所述属性信息对应的滤波系数对所述重建点云中第一点对应的K个目标点进行滤波处理,确定所述重建点云中所述第一点的属性信息的滤波值;
在确定出所述重建点云中至少一个点的属性信息的滤波值之后,根据所述至少一个点的属性信息的滤波值确定所述滤波后点云。
需要说明的是,在本申请实施例中,如果滤波标识信息指示对重建点云进行滤波处理,那么在确定属性信息对应的滤波系数之后,可以进一步利用属性信息对应的滤波系数对重建点云的属性信息进行滤波处理,从而可以获得重建点云对应的滤波后点云。在这里,根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云,具体可以利用滤波系数对重建点云中第一点对应的K个目标点进行滤波处理,进而确定出重建点云对应的滤波后点云。其中,K个目标点包括第一点以及与第一点相邻的(K-1)个近邻点,K为大于1的整数,第一点表示重建点云中的任意点。
还需要说明的是,在本申请实施例中,属性信息以颜色分量为例,可以对重建点云的属性信息的各个颜色分量进行单独滤波,例如,对于YUV空间的颜色信息,可以分别确定Y分量对应的滤波系数,U分量对应的滤波系数,V分量对应的滤波系数中的至少一项。这些滤波系数共同组成了重建点云的滤波系数。另外,由于滤波系数可以是由K阶滤波器确定的;这样,解码端器在进行滤波时,可以是针对重建点云中的每个点进行KNN搜索以确定出该点对应的K个目标点,然后利用该点对应的滤波系数进行滤波,即可得到该点的属性信息的滤波值。
以第一点为例,在一种具体的实施例中,所述确定重建点云中第一点对应的K个目标点,可以包括:
基于重建点云中的第一点,利用K近邻搜索方式在重建点云中搜索预设值个候选点;
分别计算第一点与预设值个候选点之间的距离值,从所得到的预设值个距离值中选取(K-1)个距离值,且(K-1)个距离值均小于预设值个距离值中剩余的距离值;
根据(K-1)个距离值对应的候选点确定(K-1)个近邻点,将第一点和(K-1)个近邻点确定为第一点对应的K个目标点。
需要说明的是,以第一点为例,可以利用K近邻搜索方式在重建点云中搜索预设值个候选点,计算第一点与这些候选点之间的距离值,然后从这些候选点中选取与第一点距离最近的(K-1)个近邻点;将第一点自身和(K-1)个近邻点确定为最终的K个目标点。
还需要说明的是,仍以属性信息中的颜色分量为例,在利用滤波系数对重建点云中第一点对应的K个目标点进行滤波处理时,可以包括:利用颜色分量对应的滤波系数向量对重建点云中第一点对应的K个目标点进行滤波处理,得到重建点云中第一点的颜色分量的滤波值;根据重建点云中点的颜色分量的滤波值,得到滤波后点云。具体而言,可以根据颜色分量对应的滤波系数向量和第二属性参数,确定重建点云中每一个点的颜色分量的滤波值;然后基于重建点云中每一个点的颜色分量的滤波值,得到滤波后点云。
简单来说,在本申请实施例中,在利用滤波系数对重建点云进行滤波处理时,可以先根据维纳滤波器的阶数和重建点云中点对应的K个目标点的属性信息的重建值,确定属性信息对应的第二属性参数;然后可以根据滤波系数和第二属性参数,确定属性信息的滤波值;最终便可以基于属性信息的滤波值,得到滤波后点云。
还需要说明的是,对于滤波后点云的确定,这里假定仅需要对第二颜色分量和第三颜色分量进行滤波处理,而且第二颜色分量和第三颜色分量在解码码流是顺序执行的。在一种可能的实施方式中,参见图4,其示出了本申请实施例提供的一种解码方法的流程示意图二。如图4所示,该方法可以包括:
S401:解码码流,确定第一滤波标识信息。
S402:若第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理,则解码码流, 确定第二颜色分量对应的滤波系数。
S403:解码码流,确定第二滤波标识信息。
S404:若第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行滤波处理,则解码码流,确定第三颜色分量对应的滤波系数。
S405:根据第二颜色分量对应的滤波系数,确定重建点云对应的滤波后点云。
S406:根据第三颜色分量对应的滤波系数,确定重建点云对应的滤波后点云。
S407:根据第二颜色分量对应的滤波系数和第三颜色分量对应的滤波系数,确定重建点云对应的滤波后点云。
需要说明的是,在步骤S401之后,如果第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理,那么先执行步骤S402,再执行步骤S403;如果第一滤波标识信息指示不对重建点云的属性信息的第二颜色分量进行滤波处理,那么直接执行步骤S403。然后在步骤S403之后,如果第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行滤波处理,那么先执行步骤S404,再执行步骤S406或者S407;如果第二滤波标识信息指示不对重建点云的属性信息的第三颜色分量进行滤波处理,那么直接执行步骤S405。
还需要说明的是,对于步骤S405、S406、S407而言,这三个步骤是择一执行的,具体是根据是否需要对重建点云进行第一颜色分量和/或第二颜色分量的滤波处理确定的。
在一种可能的实施方式中,如果仅需要对重建点云的属性信息的第二颜色分量进行滤波处理,那么可以选择执行步骤S405。这时候,第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理,而第二滤波标识信息指示不对重建点云的属性信息的第三颜色分量进行滤波处理,通过解码码流仅可获得第二颜色分量对应的滤波系数。
进一步地,在一些实施例中,对于S405来说,所述根据第二颜色分量对应的滤波系数,确定重建点云对应的滤波后点云,可以包括:根据第二颜色分量对应的滤波系数,确定重建点云中点对应的第二颜色分量的滤波值;利用第二颜色分量的滤波值替换重建点云中点对应的第二颜色分量的重建值,得到重建点云对应的滤波后点云。
具体来讲,这种情况下,在利用滤波系数对重建点云进行滤波处理时,可以先根据滤波器的阶数和重建点云中点对应的K个目标点的第二颜色分量的重建值,确定出第二颜色分量对应的第二属性参数;然后可以根据第二颜色分量对应的滤波系数和第二属性参数,确定第二颜色分量的滤波值;然后便可以根据第二颜色分量的滤波值覆盖重建点云中第二颜色分量的重建值,从而得到滤波后点云。
在另一种可能的实施方式中,如果仅需要对重建点云进行第三颜色分量的滤波处理,那么可以选择执行步骤S406。这时候,第一滤波标识信息指示不对重建点云的属性信息的第三颜色分量进行滤波处理,而第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行滤波处理,通过解码码流仅可获得第三颜色分量对应的滤波系数。
进一步地,在一些实施例中,对于S406来说,所述根据第三颜色分量对应的滤波系数,确定重建点云对应的滤波后点云,可以包括:根据第三颜色分量对应的滤波系数,确定重建点云中点对应的第三颜色分量的滤波值;利用第三颜色分量的滤波值替换重建点云中点对应的第三颜色分量的重建值,得到重建点云对应的滤波后点云。
具体来讲,这种情况下,在利用滤波系数对重建点云进行滤波处理时,可以先根据滤波器的阶数和重建点云中点对应的K个目标点的第三颜色分量的重建值,确定出第三颜色分量对应的第二属性参数;然后可以根据第三颜色分量对应的滤波系数和第二属性参数,确定第三颜色分量的滤波值;然后便可以根据第三颜色分量的滤波值覆盖重建点云中第三颜色分量的重建值,从而得到滤波后点云。
在又一种可能的实施方式中,如果需要对重建点云进行第二颜色分量和第三颜色分量的滤波处理,那么可以选择执行步骤S407。这时候,第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理,而第二滤波标识信息也指示对重建点云的属性信息的第三颜色分量进行滤波处理,通过解码码流仅可获得第二颜色分量对应的滤波系数和第三颜色分量对应的滤波系数。
进一步地,在一些实施例中,对于S407来说,所述根据第二颜色分量对应的滤波系数和第三颜色分量对应的滤波系数,确定重建点云对应的滤波后点云,可以包括:根据第二颜色分量对应的滤波系数和第三颜色分量对应的滤波系数,确定重建点云中点对应的第二颜色分量的滤波值和第三颜色分量的滤波值;利用第二颜色分量的滤波值和第三颜色分量的滤波值替换重建点云中点对应的第二颜色分量的重建值和第三颜色分量的重建值,得到重建点云对应的滤波后点云。
具体来讲,这种情况下,在利用滤波系数对重建点云进行滤波处理时,可以先根据滤波器的阶数和重建点云中点对应的K个目标点的某一颜色分量的重建值,确定出该颜色分量对应的第二属性参数;然后可以根据该颜色分量对应的滤波系数和第二属性参数,确定该颜色分量的滤波值;如此在得到第二 颜色分量的滤波值和第三颜色分量的滤波值之后,便可以根据第二颜色分量的滤波值和第三颜色分量的滤波值覆盖重建点云中这两个颜色分量的重建值,从而得到滤波后点云。
还可以理解地,在本申请实施例中,如果属性信息为YUV空间的颜色分量,那么可以先基于属性信息中的第二颜色分量或者第三颜色分量(例如U分量或者V分量)的重建值,结合滤波器的阶数,确定出该属性信息的第二属性参数(用P(n,k)表示),即第二属性参数表示重建点云中点对应的K个目标点的属性信息的重建值。
需要说明的是,在本申请实施例中,对于滤波器而言,滤波器类型可以用于指示滤波器阶数,和/或滤波器形状,和/或滤波器维数。其中,滤波器形状包括菱形、矩形等,滤波器维数包括一维、二维乃至更多维等。
还需要说明的是,在本申请实施例中,滤波器可以是维纳滤波器、自适应滤波器等。另外,滤波器类型可以对应不同阶数的滤波器,例如,阶数为12、16、20、32、128的滤波器;不同的类型还可以对应不同维数的滤波器,例如,一维滤波器、二维滤波器等,这里并不作具体限定。也就是说,如果需要确定一个16阶的滤波器,那么可以用16个点来确定16阶非对称滤波器、或者8阶一维对称滤波器、或者其他数量(例如更加特殊的二维、三维滤波器等)的滤波器等等,这里并不对滤波器作具体限定。
还需要说明的是,在本申请实施例中,对于滤波器的阶数而言,阶数的取值可以为16、20、32、64等等。示例性地,阶数的取值为20,但是这里并不作具体限定。
在本申请实施例中,以维纳滤波器为例,在使用维纳滤波器对重建点云进行滤波处理时,可以将重建点云和滤波系数同时输入至维纳滤波器中,即维纳滤波器的输入为滤波系数和重建点云,最终便可以基于该滤波系数完成对重建点云的滤波处理,获得对应的滤波后点云。
也就是说,在本申请实施例中,滤波系数是基于原始点云和重建点云获得的,因此,将滤波系数作用于重建点云上,可以最大限度恢复原始点云。示例性地,在本申请实施例中,以点云序列中的YUV空间的颜色分量为例,假设滤波器的阶数为K,点云序列为n,使用矩阵P(n,k)表示重建点云中所有点对应的K个目标点在相同颜色分量(如U分量)下的重建值,即P(n,k)为由重建点云中所有点对应的K个目标点的U分量的重建值所构成的第二属性参数。
这样,基于公式R(n)=P(n,k)×H(k),将U分量对应的滤波系数(用H(k)表示)作用在重建点云也就是说第二属性参数P(n,k)上,从而可以获得U分量下的颜色滤波值R(n)。然后按照上述方法遍历V分量,最终能够确定出U分量和V分量下的颜色滤波值,进而便可以利用U分量和V分量下的颜色滤波值确定出滤波后点云。
还需要说明的是,考虑到Y分量相对于U分量和V两分量的波动性较大,滤波效果不稳定,可以不涉及对Y分量的质量增强处理,以保证U分量和V分量压缩的高效性。但是本申请实施例也可以对Y分量进行质量增强处理,这时候对Y分量可以进行不同阶数的维纳滤波,也可以对Y分量使用自适应滤波器,其中,自适应滤波器的工作原理与维纳滤波相同,可以根据邻域差异自适应选择滤波器阶数、个数等。在一种具体的实施例中,Y分量可以利用基于邻域差异的方法,对图像视频帧进行维纳滤波。由于V-PCC会生成几何与纹理的视频帧,对视频帧进行编码,再由视频帧重建点云,因此提出了对视频帧进行维纳滤波,使得视频帧的质量得以提升,可以间接实现重建点云的质量增强。对于Y分量而言,具体主要是针对几何视频帧,利用视频帧中patch边缘与背景变化剧烈、像素点之间邻域差异较大的特点,对其中的像素点根据邻域差异进行分类,各邻域差异的点再进行组合,不同的组合使用不同的维纳滤波器,能够得到不同的滤波系数,将滤波系数传递至解码端,并利用滤波系数对生成的视频帧进行后处理,从而可以有效地提升点云的几何质量与主观效果。
除此之外,在本申请实施例中,解码器在利用滤波系数获得重建点云对应的滤波后点云之后,还可以使用滤波后点云覆盖原有的重建点云。
需要说明的是,在本申请实施例中,相较于重建点云,在经过滤波处理后获得的滤波后点云的质量有较明显的增强;因此,在获得滤波后点云之后,可以使用滤波后点云覆盖原有的重建点云,从而实现整个编解码及质量增强操作。
还需要说明的是,由于点云通常是采用RGB颜色空间表示的,而且YUV分量难以用现有应用进行点云可视化;因此,在确定重建点云对应的滤波后点云之后,该方法还可以包括:若滤波后点云中点的颜色分量不符合RGB颜色空间,则对滤波后点云进行颜色空间转换,使得滤波后点云中点的颜色分量符合RGB颜色空间。
这样,在将滤波后点云中点的颜色分量由符合其他颜色空间(例如YUV颜色空间、YCbCr颜色空间等)转换为符合RGB颜色空间之后,然后再利用滤波后点云覆盖原有的重建点云。
还可以理解的是,在本申请实施例中,如果滤波标识信息指示不对重建点云进行第一颜色分量和第二颜色分量(如U分量和V分量)的滤波处理,那么解码器可以跳过滤波处理流程,按照原程序得到 的重建点云进行应用,即不再对重建点云进行更新处理。
进一步地,在本申请实施例中,对于维纳滤波器的阶数,本申请实施例可以设置阶数的取值等于20,从而既能够保证滤波效果,又可以在一定程度上减少时间复杂度;但是并不作具体限定。
另外,如果编码器端维纳滤波时考虑邻域差异值,由于点云轮廓或边界处的属性值变化较大,单个滤波器可能难以覆盖全点云的滤波,导致效果仍有提升的空间,尤其对于稀疏点云序列;因此,本申请实施例可以根据邻域差异值进行分类,不同类别应用不同的滤波器,比如设置多个阶数不同的维纳滤波器,这时候编码器向解码器传递多组滤波系数;相应地,在解码器端将会解码获得多组滤波系数以对重建点云进行滤波处理,以便实现自适应选择滤波器进行滤波以达到更好的效果。
简言之,本申请实施例提出了一种基于维纳滤波的V-PCC点云颜色属性质量增强技术,在进行几何视频帧滤波的基础上,对重建点云进行维纳滤波,以达到增强点云几何与颜色属性质量的目的。其中,在几何方面可以沿用几何视频帧基于邻域差异的维纳滤波方法;而在颜色方面,本申请实施例提出了在编解码端直接对重建点云进行颜色属性维纳滤波处理,而非对重建视频帧处理;具体可以通过步骤S301~S303所提出的解码方法。
还需要说明的是,本申请实施例针对的是V-PCC几何有损、属性有损的编码方式。一方面,对几何有损的重建点云实现了颜色属性的维纳滤波,提升颜色属性质量的同时对几何重建及滤波过程不会产生任何影响。另一方面,利用维纳滤波,将几何滤波与颜色滤波同时应用在V-PCC中;其中,几何方面对于几何重建视频帧进行利用基于邻域差异的维纳滤波技术,颜色方面对于重建点云直接进行维纳滤波操作;最后得到几何与颜色质量皆提升的点云,压缩性能大幅提升。又一方面,打破了点云维纳滤波的限制,对于重建点云与初始点云难以找到匹配点的情况,提供了一种新的对应关系,利用KNN搜索方式在初始点云中搜索重建点云中每个点的最近邻点,作为该点的匹配点,并提出了真实点云的概念,以代替原来的初始点云进行后续滤波操作。再一方面,滤波系数等信息是否传递到解码端,本申请实施例是依据率失真代价与PSNR等两个指标来确定的,当质量未能提升或系数、组合点等数据的码流代价大于质量提升的效果时,这时候不会进行滤波系数的传递。
还需要说明的是,在本申请实施例提出的维纳滤波器可以在预测环路内,即作为inloop filter使用,可用作解码后续点云的参考;也可以在预测环路外,即作为post filter使用,不用作解码后续点云的参考。对此本申请实施例不进行具体限定。
还需要说明的是,在本申请实施例中,如果本申请实施例提出的维纳滤波器为环路内滤波器,那么在确定进行滤波处理之后,需要将指示进行滤波处理的参数信息,如滤波标识信息写入码流,同时也需要将滤波系数写入码流;相应地,在确定不进行滤波处理之后,可以选择不将指示进行滤波处理的参数信息,如滤波标识信息写入码流,同时也不将滤波系数写入码流。反之,如果本申请实施例提出的维纳滤波器为后处理滤波器,一种情况下,该滤波器对应的滤波系数是位于一个单独的辅助信息数据单元(例如,补充增强信息SEI),那么可以选择不将指示进行滤波处理的参数信息,如滤波标识信息写入码流,同时也不将滤波系数写入码流,相应地,解码器在未获得该补充增强信息的情况下,即不对重建点云进行滤波处理。另一种情况下,该滤波器对应的滤波系数与其他信息在一个辅助信息数据单元中,此时在确定进行滤波处理之后,需要将指示进行滤波处理的参数信息,如滤波标识信息写入码流,同时也需要将滤波系数写入码流;相应地,在确定不进行滤波处理之后,可以选择不将指示进行滤波处理的参数信息,如滤波标识信息写入码流,同时也不将滤波系数写入码流。
进一步地,对于本申请实施例提出的解码方法,可以选择对重建点云中的一个或多个部分进行滤波处理,即滤波标识信息的控制范围可以是整个重建点云,也可以是重建点云中的某个部分,这里并不作具体限定。
进一步地,对于本申请实施例提出的解码方法,本申请实施例也可以纯期望提升颜色属性质量,这时候不再进行几何视频帧的滤波,从而也可以得到更低的BD-TotalRate与更好的压缩性能。更甚者,本申请实施例还可以同时进行纹理视频帧与重建点云颜色属性的滤波,也可能会得到更好的效果。
本实施例提供了一种解码方法,通过解码码流,确定滤波标识信息;其中,滤波标识信息用于确定是否对待处理视频帧对应的重建点云进行滤波处理;若滤波标识信息指示对重建点云的属性信息进行滤波处理,则解码码流,确定属性信息对应的滤波系数;根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云。这样,解码器可以直接解码获得滤波系数,然后使用滤波系数对重建点云进行滤波处理,从而实现了对重建点云的优化,能够提升点云的质量;另外,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数时,还解决了初始点云中点和重建点云中点难以匹配的问题,并且通过对重建点云的属性信息进行滤波处理,不仅实现重建点云的质量增强,而且使得视频帧的质量也能够得到提升,进一步提升了压缩效率,进而节省码率,提高了编解码效率。
在本申请的另一实施例中,参见图5,其示出了本申请实施例提供的一种编码方法的流程示意图一。如图5所示,该方法可以包括:
S501:确定初始点云以及初始点云对应的重建点云。
需要说明的是,本申请实施例所述的编码方法具体是指点云编码方法,可以应用于V-PCC中的点云编码器(本申请实施例中,可简称为“编码器”)。但是需要注意的是,本申请实施例所述的编码方法并不局限于V-PCC领域,对于其他有损点云的质量增强而言,所有具有属性信息失真的点云都可使用。这里以V-PCC为例,是因为在V-PCC上执行可以使其发挥最大的优势,能够兼具对几何与属性的处理,点云整体质量得到提高,具有更好的效果。
还需要说明的是,在本申请实施例中,可以先确定初始点云以及初始点云对应的重建点云,然后利用初始点云和重建点云进行滤波系数的计算。另外,对于初始点云中的一个点,在对该点进行编码时,其可以作为初始点云中的待编码点,而该点的周围存在有多个已编码点。
进一步地,在本申请实施例中,对于初始点云中的一个点,其对应一个几何信息和一个属性信息;其中,几何信息表征该点的空间位置,属性信息表征该点的属性值(如颜色分量值)。
在这里,属性信息可以包括颜色分量,具体为任意颜色空间的颜色信息。示例性地,属性信息可以为RGB空间的颜色信息,也可以为YUV空间的颜色信息,还可以为YCbCr空间的颜色信息等等,本申请实施例不作具体限定。
进一步地,在本申请实施例中,颜色分量可以包括下述至少之一:第一颜色分量、第二颜色分量和第三颜色分量。这样,如果颜色分量符合RGB颜色空间,那么第一颜色分量、第二颜色分量和第三颜色分量可以依次为:R分量、G分量、B分量;如果颜色分量符合YUV颜色空间,那么第一颜色分量、第二颜色分量和第三颜色分量可以依次为:Y分量、U分量、V分量;如果颜色分量符合YCbCr颜色空间,那么第一颜色分量、第二颜色分量和第三颜色分量可以依次为:Y分量、Cb分量、Cr分量,但是并不作具体限定。
可以理解的是,在本申请实施例中,对于初始点云中的一个点,该点的属性信息可以为颜色分量,也可以是反射率或者其它属性,本申请实施例也不作具体限定。
进一步地,在本申请实施例中,对于初始点云中的一个点,可以先确定出该点的属性信息的预测值和残差值,然后再利用预测值和残差值进一步计算获得该点的属性信息的重建值,以便构建出重建点云。具体来讲,对于初始点云中的一个点,在确定该点的属性信息的预测值时,可以利用该点的多个目标邻居点的几何信息和属性信息,结合该点的几何信息对该点的属性信息进行预测,从而获得对应的预测值,进而能够确定出对应的重建值。这样,对于初始点云中的一个点,在确定出该点的属性信息的重建值之后,该点可以作为后续LOD中点的最近邻居,以利用该点的属性信息的重建值继续对后续的点进行属性预测,如此即可构建出重建点云。
也就是说,在V-PCC的实现过程中,对于初始点云生成的几何视频帧和纹理视频帧压缩完成后,编码端会据此进行点云的重建。另外,需要注意的是,这里的重建点云可以是解码后输出的重建点云,也可以是用作解码后续点云参考;此外,这里的重建点云不仅可以在在预测环路内,即作为inloop filter使用,可用作解码后续点云的参考;也可以在预测环路外,即作为post filter使用,不用作解码后续点云的参考;本申请实施例对此不作具体限定。
可以理解的是,由于V-PCC只有两种编码方式,即几何无损且属性无损的编码方式以及几何有损且属性有损的编码方式,本申请实施例所处理的是有损编码的情况,即几何有损且属性有损的编码情况。因此,在一些实施例中,对于S501来说,所述确定初始点云以及初始点云对应的重建点云,可以包括:
利用预设编码方式对初始点云进行编码及重建处理,得到第一重建点云;
对第一重建点云进行几何恢复处理,得到重建点云;其中,预设编码方式用于指示对初始点云进行几何有损且属性有损编码。
需要说明的是,在本申请实施例中,在经过属性编码、属性重建和几何偏移后获得重建点云时,该重建点云的几何和属性都是存在失真的,例如所得到的重建点云根据设置的码率不同,点的数目和坐标甚至整个重建点云的包围盒(bounding box)都会发生较大的改变,这时候需要进行几何恢复处理。
进一步地,在一些实施例中,所述对第一重建点云进行几何恢复处理,得到重建点云,可以包括:
对第一重建点云进行几何补偿处理,得到第二重建点云;
对第二重建点云进行尺度缩放处理,得到重建点云;其中,重建点云与初始点云的大小相同且几何位置相同。
也就是说,在本申请实施例中,首先进行重建点云的几何坐标预处理,经过几何补偿处理后,利用配置参数中的缩放尺度(scale),依照重建过程对每个点的几何坐标除以该缩放尺度,通过此处理就可以将重建点云恢复到与初始点云的包围盒相当大小,而且几何位置相同(即消除了几何坐标的偏移)。
这样,由于重建点云点的数目与当前初始点云的数目一般不相同,因此对于重建点云,很难找到每个点在初始点云中的准确对应点,即几何坐标相同的点。而根据维纳滤波的原理以及点云颜色属性维纳滤波的方式,本申请实施例首先需要在初始点云中找到重建点云每个点坐标相同的点作为该点的对应点,进而通过最小均方误差准则进行滤波。因此本申请实施例还提出了一种对于几何有损编码后的重建点云寻找对应点的方法,然后在此基础上进行颜色分量的维纳滤波。
具体地,在一些实施例中,该方法还可以包括:
根据预设搜索方式对初始点云进行近邻点搜索,确定重建点云中的点在初始点云中的对应点;
根据确定的对应点构建匹配点云,将匹配点云作为初始点云,建立重建点云中的点与初始点云中的点之间的对应关系。
在本申请实施例中,预设搜索方式为搜索第一常数值个点的K近邻搜索方式。相应地,所述根据预设搜索方式对初始点云进行近邻点搜索,确定重建点云中的点在初始点云中的对应点,可以包括:
基于重建点云中的当前点,利用预设搜索方式在输入点云中搜索到第一常数值个点;
分别计算当前点与第一常数值个点之间的距离值;
根据距离值,确定当前点的对应点。
在一种具体的实施例中,所述根据距离值,确定当前点的对应点,可以包括:
从距离值中选取最小距离值;
若最小距离值的个数为1个,则将最小距离值对应的点确定为当前点的对应点;
若最小距离值的个数为多个,则根据最小距离值对应的多个点,确定当前点的对应点。
需要说明的是,在本申请实施例中,第一常数值等于5,即预设搜索方式可以为k=5的KNN搜索方式,但是这里并不作具体限定。
还需要说明的是,在本申请实施例中,根据最小距离值对应的多个点,确定当前点的对应点,可以包括:从最小距离值对应的多个点中随机选取一个点,将所选取的点确定为当前点的对应点;或者,对最小距离值对应的多个点进行融合处理,将融合后的点确定为当前点的对应点;这里对此不作具体限定。
具体来讲,在几何有损且属性有损的编码方式中,为了寻找重建点云每个点与初始点云中点的对应关系,采用了KNN搜索方式遍历重建点云中的每个点,搜索范围为整个初始点云,目的是找出每个点的最近邻点,并以此作为该重建点对应的真实点,构建出真正的初始点云(即“匹配点云”)。虽然该方法不能找到严格意义上的对应点,但是由于整个点云点数较多、邻域差异较小的特点,以最近邻点作为对应点是完全可行的,而且对于稠密点云该方案的效果更好。这样,为了提升点的对应关系的准确性,本申请实施例对重建点云每个点进行KNN搜索时,会寻找当前点在原始点云中的最多5个近邻点,并依次计算每个点与当前点的距离,将距离最近的近邻点保存,如果有多个距离相同的最近邻点,则将这些最近邻点融合成一个点,即对这些点颜色分量值取均值作为当前点的对应点(最近邻匹配点)的颜色分量真实值。
也就是说,在几何有损且属性有损的编码方式中,这时候得到的重建点云点的数目和坐标甚至整个点云的bounding box都存在有较大的改变,此时如果希望更准确地找出点与点之间的对应关系,那么首先进行重建点云的几何坐标预处理,经过几何补偿处理后,利用配置参数中的缩放尺度,然后依照重建过程对每个点的几何坐标除以该缩放尺度,这样通过此处理可以将重建点云恢复到与初始点云相当的大小且消除了几何坐标的偏移,使得几何位置相同。随后,对于重建点云中的每个点在初始点云中进行k=5的KNN搜索方式,并依次计算这5个点与当前点的距离,将距离最近的近邻点保存;如果存在多个距离相同且距离最近的近邻点,那么可以从这些近邻点中随机选取一个点;或者也可以将这些近邻点融合成一个点,即对这些近邻点的属性值取均值作为当前点的最近邻匹配点的属性值。通过这种方式可以得到一个与重建点云点数相同且每个点一一对应的点云,将其作为真正的初始点云与重建点云一起输入到滤波器。
S502:根据初始点云和重建点云,确定重建点云中属性信息的滤波系数。
需要说明的是,在本申请实施例中,初始点云中的点与重建点云中的点是具有对应关系的。在输入滤波器进行滤波系数之前,首先需要确定重建点云中点对应的K个目标点。在这里,以当前点为例,重建点云中的当前点对应的K个目标点可以包括当前点以及与当前点相邻的(K-1)个近邻点,K为大于1的整数。
在一些实施例中,所述根据初始点云和重建点云,确定滤波系数,可以包括:根据初始点云中点和重建点云中点对应的K个目标点,确定滤波系数。
在一种具体的实施例中,确定重建点云中点对应的K个目标点,可以包括:
基于重建点云中的第一点,利用K近邻搜索方式在重建点云中搜索预设值个候选点;
分别计算第一点与预设值个候选点之间的距离值,从所得到的预设值个距离值中选取(K-1)个距 离值,且(K-1)个距离值均小于预设值个距离值中剩余的距离值;
根据(K-1)个距离值对应的候选点确定(K-1)个近邻点,将第一点和(K-1)个近邻点确定为第一点对应的K个目标点。
需要说明的是,第一点可以为重建点云中的任意点。在这里,以第一点为例,可以利用K近邻搜索方式在重建点云中搜索预设值个候选点,计算第一点与这些候选点之间的距离值,然后从这些候选点中选取与第一点距离最近的(K-1)个近邻点;将第一点自身和(K-1)个近邻点确定为最终的K个目标点。
还需要说明的是,这里的滤波器可以是自适应滤波器,例如可以是基于神经网络的滤波器、维纳滤波器等等,这里不作具体限定。
在本申请实施例中,以维纳滤波器为例,维纳滤波器的作用主要是计算滤波系数,同时判断维纳滤波之后的点云质量是否提升。也就是说,本申请实施例所述的滤波系数可以为维纳滤波处理的系数,即该滤波系数为维纳滤波器输出的系数。
在这里,维纳滤波器(Wiener Filter)是一种以最小化均方误差为最优准则的线性滤波器。在一定的约束条件下,其输出与一给定函数(通常称为期望输出)的差的平方达到最小,通过数学运算最终可变为一个托布利兹方程的求解问题。维纳滤波器又被称为最小二乘滤波器或最小平方滤波器。
其中,维纳滤波是利用平稳随机过程的相关特性和频谱特性对混有噪声的信号进行滤波的方法,目前是基本的滤波方法之一。维纳滤波的具体算法如下:
对于一列(混有噪声的)输入信号,滤波器长度或阶数为M时的输出如下,
Figure PCTCN2021143943-appb-000001
其中,M为滤波器的长度或阶数,y(n)为输出信号,x(n)为一列(混有噪声的)输入信号。
将式(1)转换为矩阵形式,表示如下,
y(n)=H(m)×X(n)                                     (2)
已知期望信号为d(n),则可以计算已知信号与期望信号之间的误差,用e(n)表示,具体如下,
e(n)=d(n)-y(n)=d(n)-H(m)×X(n),m=0,1,...M       (3)
维纳滤波器以最小均方误差为目标函数,故令目标函数如下表示,
Min E(e(n) 2)=E[(d(n)-H(m)×X(n)) 2]                      (4)
当滤波系数为最优时,目标函数对系数的倒数应该为0,即:
Figure PCTCN2021143943-appb-000002
也即:
2E[(d(n)-H(m)×X(n))]×X(n)=0                       (6)
E[d(n)X(n)]-H(m)E[X(n)X(n)]=0                         (7)
进一步可以表示为:
Rxd-H×Rxx=0                                    (8)
其中,Rxd与Rxx分别为输入信号与期望信号的相关矩阵与输入信号的自相关矩阵。从而根据维纳--霍夫方程进行最优解计算,可以得到滤波系数H:
H=Rxx -1×Rxd                                     (9)
进一步地,在进行维纳滤波时,需要含噪信号与期望信号;而在本申请实施例中,对于点云编解码框架,两者分别对应重建点云与初始点云,这样就可以确定出维纳滤波器的输入。这样,对于S502来说,所述根据初始点云和重建点云,确定重建点云中属性信息的滤波系数,可以包括:利用维纳滤波器将初始点云和重建点云输入到维纳滤波器中进行计算,输出滤波系数。
在本申请实施例中,将初始点云和重建点云输入到维纳滤波器中进行计算,输出滤波系数,可以包括:
根据初始点云中点的属性信息的原始值,确定第一属性参数;
根据重建点云中点对应的K个目标点的属性信息的重建值,确定第二属性参数;
基于第一属性参数和第二属性参数,确定属性信息的滤波系数。
在一种具体的实施例中,所述基于第一属性参数和第二属性参数,确定属性信息的滤波系数,可以包括:
根据第一属性参数和第二属性参数,确定互相关参数;
根据第二属性参数确定自相关参数;
根据互相关参数和自相关参数进行系数计算,得到属性信息的滤波系数。
需要说明的是,对于初始点云和重建点云而言,以属性信息的颜色分量为例,如果颜色分量符合预 设颜色空间(例如YUV颜色空间),那么在根据初始点云和重建点云确定滤波系数时,可以先基于属性信息(如Y分量、U分量、V分量等)的原始值和重建值,进行属性信息的第一属性参数和第二属性参数的确定,进而可以根据第一属性参数和第二属性参数进一步确定出用于进行滤波处理的滤波系数。但是需要注意的是,由于Y分量相对于U分量和V分量的波动性较大,滤波效果不稳定,且会增加码流开销影响整体压缩效率;因此,本申请实施例可以不对Y分量进行质量增强处理,这里的属性信息具体可以是指U分量和V分量。
具体而言,第一属性参数是根据初始点云中至少一个点的属性信息的原始值进行确定;第二属性参数是根据重建点云中至少一个点对应的K个目标点的属性信息的重建值进行确定,这里的K个目标点包括当前点以及与当前点相邻的(K-1)个近邻点。
还需要说明的是,在利用维纳滤波器进行滤波系数确定时,还涉及到维纳滤波器的阶数。在本申请实施例,可以设置维纳滤波器的阶数等于M。其中,M和K的取值可以相同,也可以不相同,这里不作具体限定。
还需要说明的是,在本申请实施例中,对于滤波器而言,滤波器类型可以用于指示滤波器阶数,和/或滤波器形状,和/或滤波器维数。其中,滤波器形状包括菱形、矩形等,滤波器维数包括一维、二维乃至更多维等。
这样,在本申请实施例中,滤波器可以是维纳滤波器、自适应滤波器等。另外,滤波器类型可以对应不同阶数的滤波器,例如,阶数M为12、16、20、32、128的滤波器;不同的类型还可以对应不同维数的滤波器,例如,一维滤波器、二维滤波器等,这里并不作具体限定。也就是说,如果需要确定一个16阶的滤波器,那么可以用16个点来确定16阶非对称滤波器、或者8阶一维对称滤波器、或者其他数量(例如更加特殊的二维、三维滤波器等)的滤波器等等,这里并不对滤波器作具体限定。
简单来说,在本申请实施例中,在根据初始点云和重建点云,确定滤波系数的过程时,可以先根据初始点云中点的属性信息的原始值,确定属性信息的第一属性参数;同时,可以根据重建点云中点对应的K个目标点的属性信息的重建值,确定属性信息的第二属性参数;最终,便可以基于第一属性参数和第二属性参数,确定出属性信息的滤波系数。
如此,对于属性信息(如U分量,V分量)进行对应的第一属性参数和第二属性参数的计算,然后可以利用第一属性参数和第二属性参数确定出该属性信息的滤波系数。
具体来讲,在根据初始点云和重建点云,确定属性信息的滤波系数时,可以先根据属性信息对应的第一属性参数和第二属性参数确定属性信息对应的互相关参数;同时可以根据第二属性参数确定属性信息对应的自相关参数;接着便可以基于互相关参数和自相关参数确定属性信息的滤波系数;最后可以遍历全部颜色分量(如U分量,V分量),可以获得每一个颜色分量的滤波系数,以便得到滤波后点云。
示例性地,在本申请实施例中,以点云序列中的YUV空间的颜色分量为例,假设滤波器的阶数为K,即利用重建点云中每一个点对应的K个目标点去计算最优系数,即计算滤波系数。在这里,K个目标点可以包括该点自身和与其相邻的(K-1)个近邻点。
假设点云序列为n,使用向量S(n)表示初始点云中所有点在某颜色分量(如U分量)下的原始值,即S(n)为由初始点云中所有点的U分量的原始值所构成的第一属性参数;使用矩阵P(n,k)表示重建点云中所有点对应的K个目标点在相同颜色分量(如U分量)下的重建值,即P(n,k)为由重建点云中所有点对应的K个目标点的U分量的重建值所构成的第二属性参数。
具体地,可以按照如下维纳滤波算法执行:
根据第一属性参数S(n)和第二属性参数P(n,k)计算获得互相关参数B(k),如下所示,
B(k)=P(n,k) T×S(n)                                (10)
根据第二属性参数P(n,k)计算获得自相关参数A(k,k),如下所示,
A(k,k)=P(n,k) T×P(n,k)                             (11)
根据维纳--霍夫方程,U分量下的滤波系数H(k),即U分量下的K阶维纳滤波器的滤波系数向量,如下所示,
H(k)=A(k,k) -1×B(k)                               (12)
接着,可以按照上述方法也能够确定出V分量下的滤波系数。
进一步地,在确定滤波系数的过程中,这里是基于YUV颜色空间进行的;如果初始点云或者重建点云不符合YUV颜色空间(例如RGB颜色空间等),那么还需要进行颜色空间转换,以使其符合YUV颜色空间。因此,在一些实施例中,该方法还可以包括:若初始点云中点的颜色分量不符合YUV颜色空间,则对初始点云进行颜色空间转换,使得初始点云中点的颜色分量符合YUV颜色空间;若重建点云中点的颜色分量不符合YUV颜色空间,则对重建点云进行颜色空间转换,使得重建点云中点的颜色分量符合YUV颜色空间。
也就是说,在利用初始点云和重建点云进行滤波系数的确定时,可以先基于初始点云和重建点云中点的U分量和V分量,分别确定出U分量和V分量对应的第一属性参数和第二属性参数,进而可以确定出U分量的滤波系数和V分量的滤波系数。
还可以理解,考虑到Y分量相对于U分量和V两分量的波动性较大,滤波效果不稳定,且会增加码流开销影响整体压缩效率,可以不涉及对Y分量的质量增强处理,以保证U分量和V分量压缩的高效性。但是本申请实施例也可以对Y分量进行质量增强处理,这时候对Y分量可以进行不同阶数的维纳滤波,也可以对Y分量使用自适应滤波器,其中,自适应滤波器的工作原理与维纳滤波相同,可以根据邻域差异自适应选择滤波器阶数、个数等等。在这种情况下,根据初始点云和重建点云,利用基于邻域差异的方法来确定出Y分量的滤波系数,以便后续确定重建点云对应的滤波后点云。
S503:根据属性信息的滤波系数,确定重建点云对应的滤波后点云。
需要说明的是,在本申请实施例中,编码器在根据初始点云和重建点云,确定出属性信息的滤波系数之后,还可以进一步利用这些滤波系数确定重建点云对应的滤波后点云。
还需要说明的是,在本申请实施例中,首先需要确定重建点云中第一点对应的K个目标点,这里的第一点表示重建点云中的任意点。然后,利用滤波系数对重建点云中第一点对应的K个目标点进行滤波处理,进而确定出重建点云对应的滤波后点云。具体地,在一些实施例中,对于S503来说,所述根据属性信息的滤波系数,确定重建点云对应的滤波后点云,可以包括:
确定重建点云中第一点对应的K个目标点;其中,K个目标点包括第一点以及与第一点相邻的(K-1)个近邻点,K为大于1的整数,第一点表示重建点云中的任意点;
根据属性信息对应的滤波系数对重建点云中第一点对应的K个目标点进行滤波处理,确定重建点云中第一点的属性信息的滤波值;
在确定出重建点云中至少一个点的属性信息的滤波值之后,根据至少一个点的属性信息的滤波值确定滤波后点云。
在一种具体的实施例中,所述确定重建点云中第一点对应的K个目标点,可以包括:
基于重建点云中的第一点,利用K近邻搜索方式在重建点云中搜索预设值个候选点;
分别计算第一点与预设值个候选点之间的距离值,从所得到的预设值个距离值中选取(K-1)个距离值,且(K-1)个距离值均小于预设值个距离值中剩余的距离值;
根据(K-1)个距离值对应的候选点确定(K-1)个近邻点,将第一点和(K-1)个近邻点确定为第一点对应的K个目标点。
需要说明的是,以第一点为例,可以利用K近邻搜索方式在重建点云中搜索预设值个候选点,计算第一点与这些候选点之间的距离值,然后从这些候选点中选取与第一点距离最近的(K-1)个近邻点;将第一点自身和(K-1)个近邻点确定为最终的K个目标点。
还需要说明的是,仍以属性信息中的颜色分量为例,在利用滤波系数对重建点云中第一点对应的K个目标点进行滤波处理时,可以包括:利用颜色分量对应的滤波系数向量对重建点云中第一点对应的K个目标点进行滤波处理,得到重建点云中第一点的颜色分量的滤波值;根据重建点云中点的颜色分量的滤波值,得到滤波后点云。
具体而言,可以根据属性信息的滤波系数和第二属性参数,确定重建点云中每一个点的属性信息的滤波值;然后可以基于该属性信息的滤波值,构建出滤波后点云。
可以理解的是,在本申请实施例中,在使用维纳滤波器对重建点云进行滤波处理时,需要含噪信号与期望信号,在点云编解码框架中,可以将重建点云作为含噪信号,将初始点云作为期望信号;因此,可以将初始点云和重建点云同时输入至维纳滤波器中,即维纳滤波器的输入为初始点云和重建点云,维纳滤波器的输出为滤波系数;在获得滤波系数之后,还可以基于该滤波系数完成对重建点云的滤波处理,获得对应的滤波后点云。
也就是说,在本申请实施例中,滤波系数是基于初始点云和重建点云获得的;因此,将滤波系数作用于重建点云上,可以最大限度恢复出初始点云。换句话说,在根据滤波系数对重建点云进行滤波时,对于其中一个颜色分量,可以基于该颜色分量对应的滤波系数,结合该颜色分量下的第二属性参数,确定出该颜色分量对应的滤波值。
示例性地,在本申请实施例中,将U分量下的滤波系数向量H(k)作用在重建点云也就是说第二属性参数P(n,k)上,可以获得U分量下的滤波值R(n),如下所示,
R(n)=P(n,k)×H(k)                                (13)
接着,可以按照上述方法遍历V分量,最终能够确定出U分量和V分量下的滤波值,进而便可以利用U分量和V分量下的滤波值确定出滤波后点云。
S504:根据重建点云和滤波后点云,确定滤波标识信息。
需要说明的是,在本申请实施例中,编码器在获得滤波后点云之后,可以进一步根据重建点云和滤波后点云确定滤波标识信息。其中,滤波标识信息可以用于确定是否对重建点云进行滤波处理。
具体来说,在本申请实施例中,滤波标识信息可以用于确定是否对重建点云进行滤波处理;进一步地,滤波标识信息还可以用于确定对重建点云中的哪一个或者哪几个颜色分量进行滤波处理。
还需要说明的是,在本申请实施例中,颜色分量可以包括下述至少之一:第一颜色分量、第二颜色分量和第三颜色分量;其中,属性信息的待处理分量可以为第一颜色分量、第二颜色分量和第三颜色分量中的任意一项。参见图6,其示出了本申请实施例提供的一种编码方法的流程示意图二。如图6所示,对于滤波标识信息的确定,该方法可以包括:
S601:确定重建点云的属性信息的待处理分量的第一代价值,以及确定滤波后点云的属性信息的待处理分量的第二代价值。
可以理解地,在一种可能的实现方式中,所述确定重建点云的属性信息的待处理分量的第一代价值,可以包括:利用率失真代价方式对重建点云的属性信息的待处理分量进行代价值计算,将所得到的第一率失真值作为所述第一代价值;
所述确定滤波后点云的属性信息的待处理分量的第二代价值,可以包括:利用率失真代价方式对滤波后点云的属性信息的待处理分量进行代价值计算,将所得到的第二率失真值作为得到所述第二代价值。
在该实现方式下,可以利用率失真代价方式来确定滤波标识信息。首先,利用率失真代价方式分别确定重建点云的属性信息的待处理分量对应的第一代价值和滤波后点云的属性信息的待处理分量对应的第二代价值;然后根据第一代价值和第二代价值的比较结果确定出待处理分量的滤波标识信息。其中,这里的代价值可以是用于失真度量的失真值,也可以是率失真代价结果等等,本申请实施例不作具体限定。
需要说明的是,为了更准确衡量滤波前后性能提升与否,本申请实施例同时对滤波后点云与重建点云进行率失真权衡,这里可以利用率失真代价方式来计算综合质量提升与码流增加后的率失真值。其中,第一率失真值和第二率失真值可以分别表征相同的颜色分量下重建点云和滤波后点云各自的率失真代价结果,用于表示滤波前后点云的压缩效率。对于第一率失真值和第二率失真值的计算,具体计算公式如下所示,
J=D+λ×R i                                      (14)
在这里,J为率失真值,D是初始点云与重建点云或滤波后点云的SSE,即对应点误差的平方和;λ是与量化参数QP有关的量,该技术方案中取
Figure PCTCN2021143943-appb-000003
R i为纹理视频帧中每属性通道的比特流大小;为了简化计算,采用了近似的取值:对于重建点云而言,
Figure PCTCN2021143943-appb-000004
其中R all为纹理视频帧的总比特流大小;对于滤波后的点云而言,取
Figure PCTCN2021143943-appb-000005
其中K是滤波器阶数,sizeof(int)表示计算机中1个整型数据的比特长度。如果滤波后点云的代价J f小于重建点云的代价J r,则该属性通道的滤波标识信息设置为1,保存对应的滤波系数;否则,该属性通道的滤波标识信息设置为0,不保存对应的滤波系数。
这样,在获得重建点云和滤波后点云之后,可以根据上述方法计算出重建点云的待处理分量的第一率失真值以及滤波后点云的待处理分量的第二率失真值,将其作为第一代价值和第二代价值,然后根据两者的比较结果确定处待处理分量的滤波标识信息。
还可以理解地,在另一种可能的实现方式中,所述确定重建点云的属性信息的待处理分量的第一代价值,可以包括:利用率失真代价方式对重建点云的属性信息的待处理分量进行代价值计算,得到第一率失真值;利用预设性能衡量指标对重建点云的属性信息的待处理分量进行性能值计算,得到第一性能值;根据第一率失真值和第一性能值,确定第一代价值。
所述确定滤波后点云的属性信息的待处理分量的第二代价值,可以包括:利用率失真代价方式对滤波后点云的属性信息的待处理分量进行代价值计算,得到第二率失真值;利用预设性能衡量指标对滤波后点云的属性信息的待处理分量进行性能值计算,得到第二性能值;根据第二率失真值和第二性能值,确定第二代价值。
在该实现方式下,不仅可以考虑利用率失真代价方式确定的率失真值,还可以考虑利用预设性能衡量指标确定的性能值(例如PSNR值)。在这里,第一性能值和第二性能值可以分别表征相同的颜色分量下重建点云和滤波后点云各自的编解码性能。示例性地,在本申请实施例中,第一性能值可以为重建点云中点的颜色分量的PSNR值,第二性能值可以为滤波后点云中点的颜色分量的PSNR值。
需要说明的是,考虑到V-PCC编解码框架最终目的是进行点云压缩,压缩率越高代表整体性能越好。本申请实施例不仅可以考虑PSNR值决定是否在解码端进行滤波,同时还可以利用率失真代价方式 进行滤波前后的率失真权衡。也就是说,在获得重建点云和滤波后点云之后,可以根据上述方法计算出重建点云的待处理分量的第一性能值和第一率失真值,将其作为第一代价值;以及计算出滤波后点云的待处理分量的第二性能值和第二率失真值,将其作为第二代价值,然后根据第一代价值和第二代价值的比较结果确定处待处理分量的滤波标识信息。
这样,不仅考虑了属性值质量的提升,同时计算了将滤波系数等信息写入码流所需的代价,综合二者表现判断滤波后压缩性能是否有了提升,从而决定编码端是否进行滤波系数的传递。
S602:根据第一代价值和第二代价值,确定待处理分量的滤波标识信息。
在一种可能的实现方式中,所述根据第一代价值和第二代价值,确定待处理分量的滤波标识信息,可以包括:
若第二代价值小于第一代价值,则确定待处理分量的滤波标识信息的取值为第一值;
若第二代价值大于第一代价值,则确定待处理分量的滤波标识信息的取值为第二值。
需要说明的是,对于第二代价值等于第一代价值的情况,这时候可以确定待处理分量的滤波标识信息的取值为第一值;或者,也可以确定待处理分量的滤波标识信息的取值为第二值。
还需要说明的是,如果待处理分量的滤波标识信息的取值为第一值,那么可以确定待处理分量的滤波标识信息指示对重建点云的属性信息的待处理分量进行滤波处理;或者,如果待处理分量的滤波标识信息的取值为第二值,那么可以确定待处理分量的滤波标识信息指示不对重建点云的属性信息的待处理分量进行滤波处理。
在另一种可能的实现方式中,所述根据第一代价值和第二代价值,确定待处理分量的滤波标识信息,可以包括:
若第二性能值大于第一性能值且第二率失真值小于第一率失真值,则确定待处理分量的滤波标识信息的取值为第一值;
若第二性能值小于第一性能值,则确定待处理分量的滤波标识信息的取值为第二值。
在又一种可能的实现方式中,所述根据第一代价值和第二代价值,确定待处理分量的滤波标识信息,可以包括:
若第二性能值大于第一性能值且第二率失真值小于第一率失真值,则确定待处理分量的滤波标识信息的取值为第一值;
若第二率失真值大于第一率失真值,则确定待处理分量的滤波标识信息的取值为第二值。
需要说明的是,对于第二性能值等于第一性能值或者第二率失真值等于第一率失真值的情况,此时可以确定待处理分量的滤波标识信息的取值为第一值;或者,也可以确定待处理分量的滤波标识信息的取值为第二值。
还需要说明的是,如果待处理分量的滤波标识信息的取值为第一值,那么可以确定待处理分量的滤波标识信息指示对重建点云的属性信息的待处理分量进行滤波处理;或者,如果待处理分量的滤波标识信息的取值为第二值,那么可以确定待处理分量的滤波标识信息指示不对重建点云的属性信息的待处理分量进行滤波处理。
还可以理解地,当待处理分量为颜色分量时,根据第一性能值、第一率失真值、第二性能值和第二率失真值,可以确定出滤波后点云相比于重建点云,这些颜色分量中某个或者某几个分量的性能值是否增加同时率失真代价是否有所下降。
在一种可能的实施方式中,当待处理分量为第一颜色分量时,对于S602来说,所述根据第一代价值和第二代价值,确定待处理分量的滤波标识信息,可以包括:
若第二性能值大于第一性能值且第二率失真值小于第一率失真值,确定第一颜色分量的滤波标识信息的取值为第一值;
若第二性能值小于第一性能值,或者第二率失真值大于第一率失真值,确定第一颜色分量的滤波标识信息的取值为第二值。
在另一种可能的实施方式中,当待处理分量为第二颜色分量时,对于S602来说,所述根据第一代价值和第二代价值,确定待处理分量的滤波标识信息,可以包括:
若第二性能值大于第一性能值且第二率失真值小于第一率失真值,确定第二颜色分量的滤波标识信息的取值为第一值;
若第二性能值小于第一性能值,或者第二率失真值大于第一率失真值,确定第二颜色分量的滤波标识信息的取值为第二值。
在又一种可能的实施方式中,当待处理分量为第三颜色分量时,对于S602来说,所述根据第一代价值和第二代价值,确定待处理分量的滤波标识信息,可以包括:
若第二性能值大于第一性能值且第二率失真值小于第一率失真值,确定第三颜色分量的滤波标识 信息的取值为第一值;
若第二性能值小于第一性能值,或者第二率失真值大于第一率失真值,确定第三颜色分量的滤波标识信息的取值为第二值。
具体来说,在本申请实施例中,以U分量的PSNR和率失真代价结果(Cost)为例,在根据第一性能值、第一率失真值、第二性能值和第二率失真值,确定U分量的滤波标识信息时,如果重建点云对应的PSNR1大于滤波后点云对应的PSNR2、或者重建点云对应的Cost1小于滤波后点云对应的Cost2,即滤波后U分量的PSNR值下降、或者滤波后U分量的率失真代价增加,那么都可以认为滤波效果不佳,此时将设置U分量的滤波标识信息指示不对U分量进行滤波处理;相应地,如果重建点云对应的PSNR1小于滤波后点云对应的PSNR2且重建点云对应的Cost1大于滤波后点云对应的Cost2,即滤波后U分量的PSNR得到提高而且率失真代价得到减小,那么可以认为滤波效果较好,此时将设置U分量的滤波标识信息指示对U分量进行滤波处理。
这样,本申请的实施例可以按照上述方法遍历U分量和V分量,甚至也可以是Y分量、U分量和V分量,最终可以利用全部颜色分量的滤波标识信息来确定出最终的滤波标识信息。
S603:根据待处理分量的滤波标识信息,得到滤波标识信息。
需要说明的是,在本申请实施例中,以YUV颜色空间为例,考虑到Y分量相对于U分量和V两分量的波动性较大,滤波效果不稳定,且会增加码流开销影响整体压缩效率,可以不涉及对Y分量的质量增强处理,以保证U分量和V分量压缩的高效性。在这种情况下,所述根据待处理分量的滤波标识信息,得到滤波标识信息,可以包括:在待处理分量包括第二颜色分量和第三颜色分量的情况下,得到第一滤波标识信息和第二滤波标识信息。
在这里,第一滤波标识信息用于指示是否对重建点云的属性信息的第二颜色分量进行滤波处理,第二滤波标识信息用于指示是否对重建点云的属性信息的第三颜色分量进行滤波处理。其中,示例性地,第二颜色分量为U分量,第三颜色分量为V分量;或者,第二颜色分量为V分量,第三颜色分量为U分量。
还需要说明的是,在本申请实施例中,仍以YUV颜色空间为例,也可以对Y分量、U分量和V分量均进行滤波处理。在这种情况下,所述根据待处理分量的滤波标识信息,得到滤波标识信息,可以包括:在待处理分量包括第一颜色分量、第二颜色分量和第三颜色分量的情况下,得到第一滤波标识信息、第二滤波标识信息和第三滤波标识信息。
在这里,第一滤波标识信息用于指示是否对重建点云的属性信息的第二颜色分量进行第一滤波处理,第二滤波标识信息用于指示是否对重建点云的属性信息的第三颜色分量进行第二滤波处理,第三滤波标识信息用于指示是否对重建点云的属性信息的第一颜色分量进行第三滤波处理。其中,第三滤波处理对应的滤波器与第一滤波处理和第二滤波处理对应的滤波器存在不同。
也就是说,考虑到Y分量相对于U分量和V两分量的波动性较大,滤波效果不稳定,且会增加码流开销影响整体压缩效率,可以不涉及对Y分量的质量增强处理,以保证U分量和V分量压缩的高效性;但是本申请实施例也可以对Y分量进行质量增强处理,这时候对Y分量可以进行与U分量和V分量不同阶数的维纳滤波,也可以对Y分量使用自适应滤波器,其中,自适应滤波器的工作原理与维纳滤波相同,可以根据邻域差异自适应选择滤波器阶数、个数等。
还需要说明的是,在本申请实施例中,滤波标识信息可以为数组形式,也可以单独标志位形式。假定YUV颜色空间中仅需要对U分量和V分量进行滤波,那么可以为1×2的数组,其是由U分量的滤波标识信息和V分量的滤波标识信息组成,例如用[u v]表示;或者也可以是按照UV颜色通道依次进行编解码。假定YUV颜色空间中需要对Y分量、U分量和V分量进行滤波,那么可以为1×3的数组,其是由Y分量的滤波标识信息、U分量的滤波标识信息和V分量的滤波标识信息组成,例如用[y u v]表示;或者也可以是按照YUV颜色通道依次进行编解码。示例性地,对于滤波标识信息,可以先采用数组形式组合在一起,然后再编码传输对应的滤波系数;或者也可以编码一个颜色通道的滤波标识信息紧接着编码传输对应的滤波系数,再编码另一个颜色通道的滤波标识信息紧接着编码传输对应的滤波系数等;这两种情况的开销相同,这里对此不作具体限定。
可以理解地,待处理分量为第一颜色分量、第二颜色分量和第三颜色分量中的任意一项。在一些实施例中,该方法还可以包括:
若待处理分量的滤波标识信息的取值为第一值,则确定待处理分量的滤波标识信息指示对重建点云的属性信息的待处理分量进行滤波处理;
若待处理分量的滤波标识信息的取值为第二值,则确定待处理分量的滤波标识信息指示不对重建点云的属性信息的待处理分量进行滤波处理。
在一种具体的实施例中,该方法还可以包括:
若第一颜色分量的滤波标识信息的取值为第一值,则确定对重建点云的属性信息的第一颜色分量进行滤波处理;若第一颜色分量的滤波标识信息的取值为第二值,则确定不对重建点云的属性信息的第一颜色分量进行滤波处理;或者,
若第二颜色分量的滤波标识信息的取值为第一值,则确定对重建点云的属性信息的第二颜色分量进行滤波处理;若第二颜色分量的滤波标识信息的取值为第二值,则确定不对重建点云的属性信息的第二颜色分量进行滤波处理;或者,
若第三颜色分量的滤波标识信息的取值为第一值,则确定对重建点云的属性信息的第三颜色分量进行滤波处理;若第三颜色分量的滤波标识信息的取值为第二值,则确定不对重建点云的属性信息的第三颜色分量进行滤波处理。
也就是说,在本申请实施例中,如果某颜色分量对应的滤波标识信息为第一值,那么可以指示对该颜色分量进行滤波处理;如果某颜色分量对应的滤波标识信息为第二值,那么可以指示不对该颜色分量进行滤波处理。
进一步地,在一些实施例中,该方法还可以包括:
若第一颜色分量的滤波标识信息、第二颜色分量的滤波标识信息和第三颜色分量的滤波标识信息中至少一个为第一值,则确定滤波标识信息指示对重建点云进行滤波处理;
若第一颜色分量的滤波标识信息、第二颜色分量的滤波标识信息和第三颜色分量的滤波标识信息中全部为第二值,则确定滤波标识信息指示不对重建点云进行滤波处理。
也就是说,在本申请实施例中,如果这些颜色分量对应的滤波标识信息均为第二值,那么可以指示对这些颜色分量均不进行滤波处理,即可确定滤波标识信息指示不对重建点云进行滤波处理;相应地,如果这些颜色分量的滤波标识信息中至少一个为第一值,那么可以指示对至少一个颜色分量进行滤波处理,即可确定滤波标识信息指示对重建点云进行滤波处理。
在另一种具体的实施例中,假定待处理分量仅包括第二颜色分量和第三颜色分量,其中,第二颜色分量对应的滤波标识信息(即前述实施例中的第一滤波标识信息)用于指示是否对重建点云的属性信息的第二颜色分量进行滤波处理,第三颜色分量对应的滤波标识信息(即前述实施例中的第二滤波标识信息)用于指示是否对重建点云的属性信息的第三颜色分量进行滤波处理;那么该方法还可以包括:若第一滤波标识信息和第二滤波标识信息中至少一个为第一值,则确定滤波标识信息指示对重建点云进行滤波处理;若第一滤波标识信息和第二滤波标识信息中全部为第二值,则确定滤波标识信息指示不对重建点云进行滤波处理。
另外,在本申请实施例中,第一值和第二值不同,而且第一值和第二值可以是参数形式,也可以是数字形式。具体地,这些颜色分量对应的标识信息可以是写入在概述(profile)中的参数,也可以是一个标志(flag)的取值,这里对此不作具体限定。
示例性地,对于第一值和第二值而言,第一值可以设置为1,第二值可以设置为0;或者,第一值还可以设置为true,第二值还可以设置为false;但是这里并不作具体限定。
S505:若滤波标识信息指示对重建点云进行属性信息的滤波处理,则对滤波标识信息和属性信息对应的滤波系数进行编码,将所得到的编码比特写入码流。
需要说明的是,在本申请实施例中,编码器在根据重建点云和滤波后点云确定出滤波标识信息之后,如果滤波标识信息指示对重建点云进行滤波处理,那么可以将滤波标识信息写入码流,同时还可以选择性地将滤波系数写入码流。
示例性地,以滤波标识信息可以用[1 0 1]为例,此时由于第一颜色分量对应的滤波标识信息为1,即需要对重建点云的属性信息的第一颜色分量进行滤波处理;第二颜色分量对应的滤波标识信息为0,即不需要对重建点云的属性信息的第二颜色分量进行滤波处理;第三颜色分量对应的滤波标识信息为1,即需要对重建点云的属性信息的第三颜色分量进行滤波处理;这时候仅需要将第一颜色分量对应的滤波系数和第三颜色分量对应的滤波系数写入码流,而无需将第二颜色分量对应的滤波系数写入码流。
在一种可能的实现方式中,对于S505来说,所述若滤波标识信息指示对重建点云的属性信息进行滤波处理,则对滤波标识信息和属性信息对应的滤波系数进行编码,可以包括:若第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理,则对第一滤波标识信息和第二颜色分量对应的滤波系数进行编码;和/或,若第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行滤波处理,则对第二滤波标识信息和第三颜色分量对应的滤波系数进行编码。
示例性地,以按照YUV颜色通道依次进行编解码为例,考虑到Y分量相对于U分量和V两分量的波动性较大,滤波效果不稳定,且会增加码流开销影响整体压缩效率,这里不对Y分量的质量增强处理,以保证U分量和V分量压缩的高效性。这时候可以先编码U分量的滤波标识信息紧接着编码传 输U分量对应的滤波系数;然后再编码V分量的滤波标识信息紧接着编码传输V分量对应的滤波系数。如果U分量的滤波标识信息为0,那么在编码U分量的滤波标识信息后无需编码传输U分量对应的滤波系数,可以紧接着编码V分量的滤波标识信息以及编码传输V分量对应的滤波系数,这里并不作具体限定。
还需要说明的是,在本申请实施例中,针对几何有损且属性有损编码方式下,如果几何失真过大,这时候可以直接确定不对重建点云进行滤波处理。具体地,在几何有损的情况下,当几何失真较大(即量化参数大于预设门限值)时,这时候重建点云的点数比较少,如果写入差不多大小的滤波系数,将会导致率失真代价过大;因此,可以不再进行滤波处理。
进一步地,在一些实施例中,当滤波标识信息指示不对重建点云进行滤波处理时,该方法还可以包括:仅对滤波标识信息进行编码,将所得到的编码比特写入码流。
需要说明的是,在本申请实施例中,编码器在利用滤波系数获得重建点云对应的滤波后点云之后,可以进一步根据重建点云和滤波后点云确定滤波标识信息。如果滤波标识信息指示对所述重建点云的属性信息进行滤波处理,那么可以将滤波标识信息以及对应的滤波系数写入码流;如果滤波标识信息指示不对所述重建点云的属性信息进行滤波处理,那么这时候不需要将滤波系数写入码流,仅需将滤波标识信息写入码流即可,以便后续通过码流传输至解码器中。
还需要说明的是,在本申请实施例中,对于初始点云生成的几何视频帧和纹理视频帧压缩完成后,编码端会据此进行点云的重建。在一些实施例中,该方法还可以包括:根据初始点云,确定几何视频帧和纹理视频帧;确定几何视频帧的预测值和纹理视频帧的预测值;根据几何视频帧的原始值和几何视频帧的预测值,确定几何视频帧的残差值;以及对几何视频帧的残差值进行编码,将所得到的编码比特写入码流;根据纹理视频帧的原始值和纹理视频帧的预测值,确定纹理视频帧的残差值;以及对纹理视频帧的残差值进行编码,将所得到的编码比特写入码流。也就是说,编码器在确定出几何视频帧的残差值和纹理视频帧的残差值之后,还需要将几何视频帧的残差值和纹理视频帧的残差值写入码流,以便后续通过码流传输至解码器中。
进一步地,在本申请实施例中,对于维纳滤波器的阶数,本申请实施例可以设置阶数的取值等于20,既能够保证滤波效果,又可以在一定程度上减少时间复杂度;但是这里并不作具体限定。
除此之外,在本申请实施例中,对于前述K值的确定,该方法还可以包括:
根据重建点云的点数,确定K的取值;和/或,
根据重建点云的量化参数,确定K的取值;和/或,
根据重建点云中点的邻域差异值,确定K的取值。
在本申请实施例中,邻域差异值可以是基于点的待滤波颜色分量与至少一个近邻点的待滤波颜色分量之间的分量差值计算得到的。
进一步地,所述根据重建点云中点的邻域差异值,确定K的取值,还可以包括:若重建点云中点的邻域差异值满足第一预设区间,则选择第一类别滤波器进行滤波处理;若重建点云中点的邻域差异值满足第二预设区间,则选择第二类别滤波器进行滤波处理;其中,第一类别滤波器的阶数与第二类别滤波器的阶数不同。
需要说明的是,一种可能的实施方式为:针对不同大小的点云滤波时选择不同的K,具体可以根据点的数目设定阈值进行判断,从而实现K的自适应选择;当点云中的点数较小时,可以适当增大K,着重提升质量;当点云中的点数较大时,可以适当减小K,这时候首先保证内存足够,而且运行速度不会太慢。
还需要说明的是,另一种可能的实施方式为:维纳滤波时考虑邻域差异值,由于点云轮廓或边界处的属性值变化较大,单个滤波器可能难以覆盖全点云的滤波,导致效果仍有提升的空间,尤其对于稀疏点云序列。因此,可以根据邻域差异值进行分类,不同类别应用不同的滤波器,从而传递多组滤波系数,可以实现自适应选择滤波器进行滤波以达到更好的效果。
还需要说明的是,又一种可能的实施方式为:在几何有损编码的情况下,当几何失真较大(即量化参数较大)时,不再进行维纳滤波;或者,根据量化参数的大小,对K进行自适应调整,以减少码流的开销。甚至还可以对需要传递的滤波系数或滤波标识信息进行熵编码,从而消除信息冗余,进一步提升编码效率。
简言之,本申请实施例提出了一种基于维纳滤波的V-PCC点云颜色属性质量增强技术,在进行几何视频帧滤波的基础上,对重建点云进行维纳滤波,以达到增强点云几何与颜色属性质量的目的。其中,在几何方面可以沿用几何视频帧基于邻域差异的维纳滤波方法;而在颜色方面,本申请实施例提出了在编解码端直接对重建点云进行颜色属性维纳滤波处理,而非对重建视频帧处理;具体可以通过步骤S501~S505所提出的编码方法。
还需要说明的是,本申请实施例针对的是V-PCC几何有损、属性有损的编码方式。一方面,对几何有损的重建点云实现了颜色属性的维纳滤波,提升颜色属性质量的同时对几何重建及滤波过程不会产生任何影响。另一方面,利用维纳滤波,将几何滤波与颜色滤波同时应用在V-PCC中;其中,几何方面对于几何重建视频帧进行利用基于邻域差异的维纳滤波技术,颜色方面对于重建点云直接进行维纳滤波操作;最后得到几何与颜色质量皆提升的点云,压缩性能大幅提升。又一方面,打破了点云维纳滤波的限制,对于重建点云与初始点云难以找到匹配点的情况,提供了一种新的对应关系,利用KNN搜索方式在初始点云中搜索重建点云中每个点的最近邻点,作为该点的匹配点,并提出了真实点云的概念,以代替原来的初始点云进行后续滤波操作。再一方面,滤波系数等信息是否传递到解码端,本申请实施例是依据率失真代价与PSNR等两个指标来确定的,当质量未能提升或系数、组合点等数据的码流代价大于质量提升的效果时,这时候不会进行滤波系数的传递。
还需要说明的是,在本申请实施例提出的维纳滤波器可以在预测环路内,即作为inloop filter使用,可用作解码后续点云的参考;也可以在预测环路外,即作为post filter使用,不用作解码后续点云的参考。对此本申请实施例不进行具体限定。
还需要说明的是,在本申请实施例中,如果本申请实施例提出的维纳滤波器为环路内滤波器,那么在确定进行滤波处理之后,需要将指示进行滤波处理的参数信息,如滤波标识信息写入码流,同时也需要将滤波系数写入码流;相应地,在确定不进行滤波处理之后,可以选择不将指示进行滤波处理的参数信息,如滤波标识信息写入码流,同时也不将滤波系数写入码流。反之,如果本申请实施例提出的维纳滤波器为后处理滤波器,一种情况下,该滤波器对应的滤波系数是位于一个单独的辅助信息数据单元(例如,补充增强信息SEI),那么可以选择不将指示进行滤波处理的参数信息,如滤波标识信息写入码流,同时也不将滤波系数写入码流,相应地,解码器在未获得该补充增强信息的情况下,即不对重建点云进行滤波处理。另一种情况下,该滤波器对应的滤波系数与其他信息在一个辅助信息数据单元中,此时在确定进行滤波处理之后,需要将指示进行滤波处理的参数信息,如滤波标识信息写入码流,同时也需要将滤波系数写入码流;相应地,在确定不进行滤波处理之后,可以选择不将指示进行滤波处理的参数信息,如滤波标识信息写入码流,同时也不将滤波系数写入码流。
进一步地,对于本申请实施例提出的编码方法,可以选择对重建点云中的一个或多个部分进行滤波处理,即滤波标识信息的控制范围可以是整个重建点云,也可以是重建点云中的某个部分,这里并不作具体限定。
进一步地,对于本申请实施例提出的编码方法,本申请实施例也可以纯期望提升颜色属性质量,这时候不再进行几何视频帧的滤波,从而也可以得到更低的BD-TotalRate与更好的压缩性能。更甚者,本申请实施例还可以同时进行纹理视频帧与重建点云颜色属性的滤波,也可能会得到更好的效果。
本实施例提供了一种编码方法,应用于编码器。通过确定初始点云以及初始点云对应的重建点云;根据初始点云和重建点云,确定重建点云中属性信息的滤波系数;根据属性信息的滤波系数,确定重建点云对应的滤波后点云;根据重建点云和滤波后点云,确定滤波标识信息;若滤波标识信息指示对重建点云的属性信息进行滤波处理,则对滤波标识信息和属性信息对应的滤波系数进行编码,将所得到的编码比特写入码流。这样,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数,并在确定对重建点云进行滤波处理之后,才将对应的滤波系数传递至解码器;相应地,解码器可以直接解码获得滤波系数,然后使用滤波系数对重建点云进行滤波处理,从而实现了对重建点云的优化,能够提升点云的质量;另外,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数时,还解决了初始点云中点和重建点云中点难以匹配的问题,并且通过对重建点云的属性信息进行滤波处理,不仅实现重建点云的质量增强,而且使得视频帧的质量也能够得到提升,进一步提升了压缩效率,进而节省码率,提高了编解码效率。
在本申请的又一实施例中,本申请实施例还提供了一种码流,该码流是根据待编码信息进行比特编码生成的;其中,待编码信息包括下述至少之一:几何视频帧的残差值、纹理视频帧的残差值、滤波标识信息和滤波系数。
这样,当几何视频帧的残差值、纹理视频帧的残差值、滤波标识信息和滤波系数由编码器传递到解码器之后,解码器通过解码获得几何视频帧的残差值和纹理视频帧的残差值,进而构建出重建点云;而且解码器通过解码获得滤波标识信息,还可以确定是否需要对重建点云的属性信息进行滤波处理;然后在需要对重建点云的属性信息进行滤波处理时,通过解码直接获得属性信息对应的滤波系数;根据滤波系数对重建点云进行滤波处理,从而实现了对重建点云的优化,能够提升点云的质量。
可以理解地,虽然相关技术也存在一种针对V-PCC中视频帧进行维纳滤波的方法,可以较好地提 升重建点云的几何属性的质量,但是该方法对于颜色属性的提升并不理想。其中,如果未对纹理视频帧帧进行处理,通过对几何视频帧的处理,只能使重建点云颜色属性的BD-Total Rate有-0.1%左右的增益;如果对纹理视频帧同样进行维纳滤波处理,尽管视频帧的PSNR能够提升近0.2dB,但是最终重建点云颜色属性的BD-Total Rate只在少数几个点云序列有增益,对于大多数的点云序列该指标不降反升。因此,对于重建点云颜色属性,本申请实施例提供了一种更合适的方法进行处理以提升点云质量与压缩性能。
基于上述实施例提供的编解码方法,在本申请的再一实施例中,这里提出了一种基于维纳滤波的V-PCC点云颜色属性质量增强技术,在进行几何视频帧滤波的基础上,对重建点云进行维纳滤波,以达到增强点云几何与颜色属性质量的目的。在几何方面,本申请实施例可以采用几何视频帧基于邻域差异的维纳滤波方法;在颜色方面,本申请实施例提出在编解码端直接对重建点云进行颜色属性维纳滤波处理,而非对重建视频帧处理。编码端通过计算得到对应的滤波系数及滤波标识信息并传递到解码端,解码端解码滤波系数后可以对重建几何视频帧与重建点云进行维纳滤波处理。
以使用维纳滤波器进行滤波处理为例,图7为本申请实施例提供的一种编解码的总体流程示意图。如图7所示,在编码端,初始点云在经过编码及重建之后,可以得到重建点云;然后初始点云和重建点云可以作为维纳滤波器的输入,而维纳滤波器的输出即为属性信息的滤波系数;这些滤波系数会根据判断效果优劣,确定是否需要对重建点云进行滤波处理;只有需要对重建点云进行滤波时才会将对应的滤波系数写入码流;另外,初始点云在经过编码之后的几何残差值和纹理残差值等也会写入码流。这样在解码端,解码器通过解码码流,可以得到几何残差值和纹理残差值,进而经过解码及重建处理,可以得到重建点云;另外,如果码流中存在有滤波系数,还需要解码获得对应的滤波系数,然后将这些滤波系数和重建点云输入到维纳滤波器,最后输出滤波后点云(即输出点云),而且具有质量增强效果。
在一种具体的实施例中,图8为本申请实施例提供的一种编码端滤波处理的流程示意图。如图8所示,初始点云和重建点云首先会进行K近邻搜索,确定出匹配点云;其中,匹配点云中的点与重建点云中的点具有对应关系,将匹配点云作为初始点云和重建点云一起输入到维纳滤波器。然后通过维纳滤波器,输出滤波标识信息和对应的滤波系数,并写入码流。
具体地,本申请实施例针对的是V-PCC几何有损、属性有损的编码方式。在编码端,具体操作如下:
(1)依据V-PCC的实现过程,对由初始点云生成的几何视频帧和纹理视频帧压缩完成后,编码端会据此进行点云的重建。由于V-PCC只有两种编码方式,即几何与属性皆无损编码以及几何与属性皆有损编码,本申请实施例的技术方案所处理的是有损编码的情况。因此,当得到重建点云时,该点云的几何属性也是失真的,具体表现在:重建点云点的数目与初始点云点的数目一般不相同,因此对于重建点云,很难找到每个点在初始点云中的准确对应点,即几何坐标相同的点。而根据维纳滤波的原理以及点云颜色属性维纳滤波的方式,需要在初始点云中找到重建点云每个点坐标相同的点作为该点的真实值,进而通过最小均方误差准则进行滤波。因此该方案提出了一种对于几何有损编码后的重建点云寻找匹配点云的方法,并在此基础上进行颜色属性的维纳滤波。
(2)该技术方案中,为寻找重建点云每个点与初始点云中点的对应关系,采用了KNN搜索方式遍历重建点云中的每个点,搜索范围为整个初始点云,目的是找出每个点的最近邻点,并以此作为该重建点对应的真实点。虽然该方法不能找到严格意义上的对应点,但是由于整个点云点数较多、邻域差异较小的特点,以最近邻点作为对应点是完全可行的,而且对于稠密点云而言,该方法的效果更好。同时为了提升点与点对应关系的准确性,该技术方案对重建点云每个点进行KNN搜索时,会寻找当前点在初始点云中的最多5个近邻点,并依次计算每个近邻点与当前点的距离,将距离最近的近邻点保存,如果有多个距离相同的最近邻点,则将这些最近邻点融合成一个点,即对这些点颜色属性值取均值作为当前点的最近邻匹配点的颜色属性真实值。
有了对应点之后,该技术方案会利用重建点云每个点的索引顺序,将该点在初始点云中的对应点的属性信息组合成一个与重建点云大小相同的矩阵,将其视作“真实点云”属性矩阵,即匹配点云,以方便接下来的操作。之后该方案利用点云维纳滤波的原理,采用阶数为20的维纳滤波器,对于点云序列每一帧将重建点云与真实点云颜色属性由RGB颜色空间转换为YUV颜色空间之后,对重建点云中的U和V两个颜色通道逐次进行质量增强。该技术方案并未对Y分量进行质量增强,这是因为Y分量相对于U和V两分量的波动性较大,滤波效果不稳定,且会增加码流开销影响整体压缩效率。滤波后会得到滤波后点云以及对应的滤波系数。其中,对于滤波后点云的属性信息的U和V分量,该技术方案会计算其与真实点云的PSNR以及率失真性能,并与滤波前的重建点云相比较。其中,率失真代价(用J表示)的具体计算如上述的式(14)所示。在这里,D是初始点云与重建点云或滤波后点云的SSE,即对应点误差的平方和;λ是与量化参数QP有关的量,该技术方案中取
Figure PCTCN2021143943-appb-000006
R i为纹理 视频帧中每属性通道的比特流大小;为了简化计算,采用了近似的取值:对于重建点云而言,
Figure PCTCN2021143943-appb-000007
其中R all为纹理视频帧的总比特流大小;对于滤波后的点云而言,取
Figure PCTCN2021143943-appb-000008
其中K是滤波器阶数,sizeof(int)表示计算机中1个整型数据的比特长度。如果滤波后点云的代价J f小于重建点云的代价J r,则该属性通道的滤波标识信息设置为1,保存对应的滤波系数;否则,该属性通道的滤波标识信息设置为0,不保存对应的滤波系数。
(3)最后,将需要传递至解码端的相关信息写入码流。需要注意的是,V-PCC的码流主要由三部分组成:占有图及Patch有关的辅助信息码流、几何视频帧编码码流、纹理视频帧编码码流。因此为了便于对滤波后点云颜色属性增益效果及压缩性能进行测试,该方案将波标识信息、滤波系数等相关信息写入纹理视频帧码流之中。在该技术方案中,首先读取纹理视频帧码流,并获取比特流大小。在得到某一属性通道的滤波系数后,利用码流大小进行率失真优化,结合PSNR即可得到该属性通道的滤波标识信息。如果滤波标识信息的取值为1,则说明该属性通道滤波使得率失真性能提升,压缩比例增加,重建点云质量得到了增强,因此会将滤波标识信息与滤波系数写入码流中。如果滤波标识信息的取值为0,证明滤波效果不佳,此时不会进行滤波系数传递与解码端操作,仅将滤波标识信息写入码流之中。具体的写入方法为:由于在已有编码框架中,常用0x00 00 00 01作为分隔符,用于开始一段新的编码,之后便是NALU(网络抽象层单元),即帧数据信息。该技术方案将需要传递的相关数据直接写入到分隔符之后;所有帧信息全部写入后,再将纹理视频帧在分隔符之后原本的数据写入,这样可以保证无信息损失的同时顺利地写入滤波标识信息与滤波系数等相关数据。
在另一种具体的实施例中,图9为本申请实施例提供的一种解码端滤波处理的流程示意图。如图9所示,解码器首先解码码流,获得滤波标识信息;然后判断是否需要属性信息的滤波;如果判断结果为否,则直接返回V-PCC框架;如果需要属性信息的滤波,那么解码码流,获得属性信息对应的滤波系数;在进行维纳滤波处理时,可以将重建点云与滤波系数作为解码端的维纳滤波器的输入,然后通过维纳滤波器,输出滤波后的且质量增强的滤波后点云,然后再返回V-PCC框架。
具体地,在解码端,具体操作如下:
(1)由于编码端滤波器所传递的滤波系数等数据在纹理视频帧数据之前,且纹理视频帧的操作先于该技术方案的重建点云滤波,因此在纹理视频帧重建之前,首先读取编码端传递的数据并保存。对于每一帧点云属性信息的U和V通道,首先读取滤波标识信息,如果滤波标识信息的取值为1,继续解码出对应的滤波系数;如果滤波标识信息的取值为0,不再解码滤波系数,而是进行下一通道的读取。全部读取完成后,依照编码端写入码流的方式,再恢复原始码流,继续V-PCC框架的处理。其中,解码端在得到重建点云之后,将其与滤波系数一起输入至维纳滤波器。
(2)根据滤波标识信息可判断出每一帧中哪些通道需要滤波,利用维纳滤波原理,对需要滤波的通道应用滤波系数,即可得到该通道的属性信息的滤波值。之后将其覆盖当前帧重建点云对应通道的重建值;以此类推,即可得到所有帧的滤波后点云。此时该点云即为质量增强后的点云,且有压缩性能的提升。
进一步地,在本申请实施例中,图10为本申请实施例提供的一种基于CTC_C2测试条件的测试结果示意图。如图10所示,本申请实施例的技术方案在V-PCC参考软件TMC2 V14.0上实现后,在CTC-C2测试条件下对MPEG测试序列的前32帧进行了测试(cat2-A&cat2-B),其测试结果如图10所示。从图10可以得到,相较于原始程序,经过重建点云颜色属性滤波处理,甚至几何视频帧自适应滤波处理后,点云的几何与属性质量都有了较大的提升,压缩效率进一步增加,几何方面的BD-GeomRate与属性方面的BD-Attr Rate都有所降低,尤其是BD-AttrRate降低明显,这是由于颜色码流本身较大,滤波系数传递几乎不会造成影响,且自身PSNR也提升较多。从几何或颜色属性质量相对于整体码流大小来看(即BD-TotalRate),几何与颜色属性都有较为明显的增益。其中,Cat2-A序列颜色属性增益更为可观,表明本申请实施例的方法对于重建质量较高的点云质量提升效果更为明显。
本申请实施例提供了一种基于维纳滤波的V-PCC点云颜色属性质量增强技术,本实施例对前述实施例的具体实现进行了详细阐述,从中可以看出,基于前述实施例的技术方案,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数,并在确定对重建点云进行滤波处理之后,才将对应的滤波系数传递至解码器;相应地,解码器可以直接解码获得滤波系数,然后使用滤波系数对重建点云进行滤波处理,从而实现了对重建点云的优化,能够提升点云的质量;另外,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数时,还解决了初始点云中点和重建点云中点难以匹配的问题,并且通过对重建点云的属性信息进行滤波处理,不仅实现重建点云的质量增强,而且使得视频帧的质量也能够得到提升,进一步提升了压缩效率,进而节省码率,提高了编解码效率。
在本申请的再一实施例中,基于前述实施例相同的发明构思,参见图11,其示出了本申请实施例提供的一种编码器110的组成结构示意图。如图11所示,该编码器110可以包括:第一确定单元1101、第一滤波单元1102和编码单元1103;其中,
第一确定单元1101,配置为确定初始点云以及初始点云对应的重建点云;以及根据初始点云和重建点云,确定重建点云中属性信息的滤波系数;
第一滤波单元1102,配置为根据属性信息的滤波系数,确定重建点云对应的滤波后点云;
第一确定单元1101,还配置为根据重建点云和滤波后点云,确定滤波标识信息;
编码单元1103,配置为若滤波标识信息指示对重建点云的属性信息进行滤波处理,则对滤波标识信息和属性信息对应的滤波系数进行编码,将所得到的编码比特写入码流。
在一些实施例中,第一确定单元1101,还配置为利用预设编码方式对初始点云进行编码及重建处理,得到第一重建点云;以及对第一重建点云进行几何恢复处理,得到重建点云;其中,预设编码方式用于指示对初始点云进行几何有损且属性有损编码。
在一些实施例中,第一确定单元1101,还配置为对第一重建点云进行几何补偿处理,得到第二重建点云;并对第二重建点云进行尺度缩放处理,得到重建点云;其中,重建点云与初始点云的大小相同且几何位置相同。
在一些实施例中,参见图11,编码器110还可以包括搜索单元1104,配置为根据预设搜索方式对初始点云进行近邻点搜索,确定重建点云中的点在初始点云中的对应点;并根据确定的对应点构建匹配点云,将匹配点云作为初始点云,建立重建点云中的点与初始点云中的点之间的对应关系。
在一些实施例中,预设搜索方式为K近邻搜索方式;相应地,搜索单元1104,具体配置为基于重建点云中的当前点,利用预设搜索方式在初始点云中搜索到第一常数值个点;以及分别计算当前点与第一常数值个点之间的距离值;
第一确定单元1101,还配置为根据距离值,确定当前点的对应点。
在一些实施例中,第一确定单元1101,还配置为从距离值中选取最小距离值;以及若最小距离值的个数为1个,则将最小距离值对应的点确定为当前点的对应点;以及若最小距离值的个数为多个,则根据最小距离值对应的多个点,确定当前点的对应点。
在一些实施例中,第一确定单元1101,还配置为确定重建点云中第一点对应的K个目标点;其中,K个目标点包括第一点以及与第一点相邻的(K-1)个近邻点,K为大于1的整数,第一点表示重建点云中的任意点;
第一滤波单元1102,配置为根据属性信息对应的滤波系数对重建点云中第一点对应的K个目标点进行滤波处理,确定重建点云中第一点的属性信息的滤波值;在确定出重建点云中至少一个点的属性信息的滤波值之后,根据至少一个点的属性信息的滤波值确定滤波后点云。
在一些实施例中,搜索单元1104,还配置为基于重建点云中的第一点,利用K近邻搜索方式在重建点云中搜索预设值个候选点;以及分别计算第一点与预设值个候选点之间的距离值,从所得到的预设值个距离值中选取(K-1)个距离值,且(K-1)个距离值均小于预设值个距离值中剩余的距离值;
第一确定单元1101,还配置为根据(K-1)个距离值对应的候选点确定(K-1)个近邻点,将第一点和(K-1)个近邻点确定为第一点对应的K个目标点。
在一些实施例中,第一确定单元1101,还配置为根据初始点云中点的属性信息的原始值,确定第一属性参数;以及根据重建点云中点对应的K个目标点的属性信息的重建值,确定第二属性参数;以及基于第一属性参数和第二属性参数,确定属性信息的滤波系数。
在一些实施例中,第一确定单元1101,还配置为根据第一属性参数和第二属性参数,确定互相关参数;根据第二属性参数确定自相关参数;以及根据互相关参数和自相关参数进行系数计算,得到属性信息的滤波系数。
在一些实施例中,属性信息包括颜色分量,且颜色分量包括下述至少之一:第一颜色分量、第二颜色分量和第三颜色分量;其中,若颜色分量符合RGB颜色空间,则确定第一颜色分量、第二颜色分量和第三颜色分量依次为:R分量、G分量、B分量;若颜色分量符合YUV颜色空间,则确定第一颜色分量、第二颜色分量和第三颜色分量依次为:Y分量、U分量、V分量。
在一些实施例中,参见图11,编码器110还可以包括计算单元1105,配置为确定重建点云的属性信息的待处理分量的第一代价值,以及确定滤波后点云的属性信息的待处理分量的第二代价值;
第一确定单元1101,还配置为根据第一代价值和第二代价值,确定待处理分量的滤波标识信息;以及根据待处理分量的滤波标识信息,得到滤波标识信息。
在一些实施例中,第一确定单元1101,还配置为若第二代价值小于第一代价值,则确定待处理分量的滤波标识信息的取值为第一值;若第二代价值大于第一代价值,则确定待处理分量的滤波标识信息 的取值为第二值。
在一些实施例中,计算单元1105,还配置为利用率失真代价方式对重建点云的属性信息的待处理分量进行代价值计算,将所得到的第一率失真值作为第一代价值;并利用率失真代价方式对滤波后点云的属性信息的待处理分量进行代价值计算,将所得到的第二率失真值作为得到第二代价值。
在一些实施例中,计算单元1105,还配置为利用率失真代价方式对重建点云的属性信息的待处理分量进行代价值计算,得到第一率失真值;利用预设性能衡量指标对重建点云的属性信息的待处理分量进行性能值计算,得到第一性能值;以及根据第一率失真值和第一性能值,确定第一代价值;
相应地,计算单元1105,还配置为利用率失真代价方式对滤波后点云的属性信息的待处理分量进行代价值计算,得到第二率失真值;利用预设性能衡量指标对滤波后点云的属性信息的待处理分量进行性能值计算,得到第二性能值;以及根据第二率失真值和第二性能值,确定第二代价值。
在一些实施例中,第一确定单元1101,还配置为若第二性能值大于第一性能值且第二率失真值小于第一率失真值,则确定待处理分量的滤波标识信息的取值为第一值;若第二性能值小于第一性能值,则确定待处理分量的滤波标识信息的取值为第二值。
在一些实施例中,第一确定单元1101,还配置为若第二性能值大于第一性能值且第二率失真值小于第一率失真值,则确定待处理分量的滤波标识信息的取值为第一值;若第二率失真值大于第一率失真值,则确定待处理分量的滤波标识信息的取值为第二值。
在一些实施例中,第一确定单元1101,还配置为若待处理分量的滤波标识信息的取值为第一值,则确定待处理分量的滤波标识信息指示对重建点云的属性信息的待处理分量进行滤波处理;若待处理分量的滤波标识信息的取值为第二值,则确定待处理分量的滤波标识信息指示不对重建点云的属性信息的待处理分量进行滤波处理。
在一些实施例中,第一确定单元1101,还配置为在待处理分量包括第二颜色分量和第三颜色分量的情况下,得到第一滤波标识信息和第二滤波标识信息;其中,第一滤波标识信息用于指示是否对重建点云的属性信息的第二颜色分量进行滤波处理,第二滤波标识信息用于指示是否对重建点云的属性信息的第三颜色分量进行滤波处理。
在一些实施例中,第一确定单元1101,还配置为若第一滤波标识信息和第二滤波标识信息中至少一个为第一值,则确定滤波标识信息指示对重建点云进行滤波处理;若第一滤波标识信息和第二滤波标识信息中全部为第二值,则确定滤波标识信息指示不对重建点云进行滤波处理。
在一些实施例中,编码单元1103,还配置为若滤波标识信息指示不对重建点云的属性信息进行滤波处理,则仅对滤波标识信息进行编码,将所得到的编码比特写入码流。
在一些实施例中,第一确定单元1101,还配置为在待处理分量包括第一颜色分量、第二颜色分量和第三颜色分量的情况下,得到第一滤波标识信息、第二滤波标识信息和第三滤波标识信息;其中,第一滤波标识信息用于指示是否对重建点云的属性信息的第二颜色分量进行第一滤波处理,第二滤波标识信息用于指示是否对重建点云的属性信息的第三颜色分量进行第二滤波处理,第三滤波标识信息用于指示是否对重建点云的属性信息的第一颜色分量进行第三滤波处理。
在一些实施例中,第三滤波处理对应的滤波器与第一滤波处理和第二滤波处理对应的滤波器存在不同。
在一些实施例中,第一确定单元1101,还配置为根据重建点云的点数,确定K的取值;和/或,根据重建点云的量化参数,确定K的取值;和/或,根据重建点云中点的邻域差异值,确定K的取值;其中,邻域差异值是基于点的属性信息与至少一个近邻点的属性信息之间的分量差值计算得到的。
在一些实施例中,第一确定单元1101,还配置为根据初始点云,确定几何视频帧和纹理视频帧;以及确定几何视频帧的预测值和纹理视频帧的预测值;
编码单元1103,还配置为根据几何视频帧的原始值和几何视频帧的预测值,确定几何视频帧的残差值,对几何视频帧的残差值进行编码,将所得到的编码比特写入码流;以及根据纹理视频帧的原始值和纹理视频帧的预测值,确定纹理视频帧的残差值,对纹理视频帧的残差值进行编码,将所得到的编码比特写入码流。
可以理解地,在本申请实施例中,“单元”可以是部分电路、部分处理器、部分程序或软件等等,当然也可以是模块,还可以是非模块化的。而且在本实施例中的各组成部分可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能模块的形式实现。
所述集成的单元如果以软件功能模块的形式实现并非作为独立的产品进行销售或使用时,可以存储在一个计算机可读取存储介质中,基于这样的理解,本实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来,该计算机软件产品存储 在一个存储介质中,包括若干指令用以使得一台计算机设备(可以是个人计算机,服务器,或者网络设备等)或processor(处理器)执行本实施例所述方法的全部或部分步骤。而前述的存储介质包括:U盘、移动硬盘、只读存储器(Read Only Memory,ROM)、随机存取存储器(Random Access Memory,RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
因此,本申请实施例提供了一种计算机存储介质,应用于编码器110,该计算机存储介质存储有计算机程序,所述计算机程序被第一处理器执行时实现前述实施例中任一项所述的方法。
基于上述编码器110的组成以及计算机存储介质,参见图12,其示出了本申请实施例提供的编码器110的具体硬件结构示意图。如图12所示,编码器110可以包括:第一通信接口1201、第一存储器1202和第一处理器1203;各个组件通过第一总线系统1204耦合在一起。可理解,第一总线系统1204用于实现这些组件之间的连接通信。第一总线系统1204除包括数据总线之外,还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见,在图12中将各种总线都标为第一总线系统1204。其中,
第一通信接口1201,用于在与其他外部网元之间进行收发信息过程中,信号的接收和发送;
第一存储器1202,用于存储能够在第一处理器1203上运行的计算机程序;
第一处理器1203,用于在运行所述计算机程序时,执行:
确定初始点云以及初始点云对应的重建点云;
根据初始点云和重建点云,确定重建点云中属性信息的滤波系数;
根据属性信息的滤波系数,确定重建点云对应的滤波后点云;
根据重建点云和滤波后点云,确定滤波标识信息;
若滤波标识信息指示对重建点云的属性信息进行滤波处理,则对滤波标识信息和属性信息对应的滤波系数进行编码,将所得到的编码比特写入码流。
可以理解,本申请实施例中的第一存储器1202可以是易失性存储器或非易失性存储器,或可包括易失性和非易失性存储器两者。其中,非易失性存储器可以是只读存储器(Read-Only Memory,ROM)、可编程只读存储器(Programmable ROM,PROM)、可擦除可编程只读存储器(Erasable PROM,EPROM)、电可擦除可编程只读存储器(Electrically EPROM,EEPROM)或闪存。易失性存储器可以是随机存取存储器(Random Access Memory,RAM),其用作外部高速缓存。通过示例性但不是限制性说明,许多形式的RAM可用,例如静态随机存取存储器(Static RAM,SRAM)、动态随机存取存储器(Dynamic RAM,DRAM)、同步动态随机存取存储器(Synchronous DRAM,SDRAM)、双倍数据速率同步动态随机存取存储器(Double Data Rate SDRAM,DDRSDRAM)、增强型同步动态随机存取存储器(Enhanced SDRAM,ESDRAM)、同步连接动态随机存取存储器(Synchlink DRAM,SLDRAM)和直接内存总线随机存取存储器(Direct Rambus RAM,DRRAM)。本申请描述的系统和方法的第一存储器1202旨在包括但不限于这些和任意其它适合类型的存储器。
而第一处理器1203可能是一种集成电路芯片,具有信号的处理能力。在实现过程中,上述方法的各步骤可以通过第一处理器1203中的硬件的集成逻辑电路或者软件形式的指令完成。上述的第一处理器1203可以是通用处理器、数字信号处理器(Digital Signal Processor,DSP)、专用集成电路(Application Specific Integrated Circuit,ASIC)、现成可编程门阵列(Field Programmable Gate Array,FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成,或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器,闪存、只读存储器,可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于第一存储器1202,第一处理器1203读取第一存储器1202中的信息,结合其硬件完成上述方法的步骤。
可以理解的是,本申请描述的这些实施例可以用硬件、软件、固件、中间件、微码或其组合来实现。对于硬件实现,处理单元可以实现在一个或多个专用集成电路(Application Specific Integrated Circuits,ASIC)、数字信号处理器(Digital Signal Processing,DSP)、数字信号处理设备(DSP Device,DSPD)、可编程逻辑设备(Programmable Logic Device,PLD)、现场可编程门阵列(Field-Programmable Gate Array,FPGA)、通用处理器、控制器、微控制器、微处理器、用于执行本申请所述功能的其它电子单元或其组合中。对于软件实现,可通过执行本申请所述功能的模块(例如过程、函数等)来实现本申请所述的技术。软件代码可存储在存储器中并通过处理器执行。存储器可以在处理器中或在处理器外部实现。
可选地,作为另一个实施例,第一处理器1203还配置为在运行所述计算机程序时,执行前述实施例中任一项所述的方法。
本实施例提供了一种编码器,该编码器可以包括第一确定单元、第一滤波单元和编码单元。这样,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数,并在确定对重建点云进行滤波处 理之后,才将对应的滤波系数传递至解码器;相应地,解码器可以直接解码获得滤波系数,然后使用滤波系数对重建点云进行滤波处理,从而实现了对重建点云的优化,能够提升点云的质量;另外,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数时,还解决了初始点云中点和重建点云中点难以匹配的问题,并且通过对重建点云的属性信息进行滤波处理,不仅实现重建点云的质量增强,而且使得视频帧的质量也能够得到提升,进一步提升了压缩效率,进而节省码率,提高了编解码效率。
在本申请的再一实施例中,基于前述实施例相同的发明构思,参见图13,其示出了本申请实施例提供的一种解码器130的组成结构示意图。如图13所示,该解码器130可以包括:解码单元1301和第二滤波单元1302;其中,
解码单元1301,配置为解码码流,确定滤波标识信息;其中,滤波标识信息用于确定是否对待处理视频帧对应的重建点云进行滤波处理;以及若滤波标识信息指示对重建点云的属性信息进行滤波处理,则解码码流,确定属性信息对应的滤波系数;
第二滤波单元1302,配置为根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云。
在一些实施例中,参见图13,解码器130还可以包括第二确定单元1303;其中,待处理视频帧包括纹理视频帧和几何视频帧;
解码单元1301,还配置为解码码流,确定纹理视频帧的残差值和几何视频帧的残差值;
第二确定单元1303,配置为在确定纹理视频帧的预测值后,根据纹理视频帧的预测值和纹理视频帧的残差值,确定纹理视频帧的重建值;以及在确定几何视频帧的预测值后,根据几何视频帧的预测值和几何视频帧的残差值,确定几何视频帧的重建值;以及基于纹理视频帧的重建值和几何视频帧的重建值,得到重建点云。
在一些实施例中,第二确定单元1303,还配置为若滤波标识信息指示不对重建点云的属性信息进行滤波处理,则不执行解码码流,确定属性信息对应的滤波系数的步骤。
在一些实施例中,属性信息包括颜色分量,且颜色分量包括下述至少之一:第一颜色分量、第二颜色分量和第三颜色分量;其中,若颜色分量符合RGB颜色空间,则确定第一颜色分量、第二颜色分量和第三颜色分量依次为:R分量、G分量、B分量;若颜色分量符合YUV颜色空间,则确定第一颜色分量、第二颜色分量和第三颜色分量依次为:Y分量、U分量、V分量。
在一些实施例中,解码单元1301,还配置为解码码流,确定待处理分量的滤波标识信息;其中,待处理分量包括下述至少之一:第一颜色分量、第二颜色分量和第三颜色分量,且待处理分量的滤波标识信息用于指示是否对重建点云的属性信息的待处理分量进行滤波处理。
在一些实施例中,第二确定单元1303,还配置为若待处理分量的滤波标识信息的取值为第一值,则确定对重建点云的属性信息的待处理分量进行滤波处理;若待处理分量的滤波标识信息的取值为第二值,则确定不对重建点云的属性信息的待处理分量进行滤波处理;
相应地,解码单元1301,还配置为若待处理分量的滤波标识信息的取值为第一值,则解码码流,确定属性信息的待处理分量对应的滤波系数。
在一些实施例中,解码单元1301,还配置为当待处理分量包括第二颜色分量和第三颜色分量时,解码码流,确定第一滤波标识信息和第二滤波标识信息;其中,第一滤波标识信息用于指示是否对重建点云的属性信息的第二颜色分量进行滤波处理,第二滤波标识信息用于指示是否对重建点云的属性信息的第三颜色分量进行滤波处理。
在一些实施例中,解码单元1301,还配置为若第一滤波标识信息指示对重建点云的属性信息的第二颜色分量进行滤波处理,则解码码流,确定第二颜色分量对应的滤波系数;和/或,若第二滤波标识信息指示对重建点云的属性信息的第三颜色分量进行滤波处理,则解码码流,确定第三颜色分量对应的滤波系数;
相应地,第二确定单元1303,还配置为根据第二颜色分量对应的滤波系数和/或第三颜色分量对应的滤波系数,确定重建点云对应的滤波后点云。
在一些实施例中,解码单元1301,还配置为当待处理分量包括第一颜色分量、第二颜色分量和第三颜色分量时,解码码流,确定第一滤波标识信息、第二滤波标识信息和第三滤波标识信息;其中,第一滤波标识信息用于指示是否对重建点云的属性信息的第二颜色分量进行第一滤波处理,第二滤波标识信息用于指示是否对重建点云的属性信息的第三颜色分量进行第二滤波处理,第三滤波标识信息用于指示是否对重建点云的属性信息的第一颜色分量进行第三滤波处理。
在一些实施例中,第三滤波处理对应的滤波器与第一滤波处理和第二滤波处理对应的滤波器存在不同。
在一些实施例中,第二确定单元1303,还配置为确定重建点云中第一点对应的K个目标点;其中, K个目标点包括第一点以及与第一点相邻的(K-1)个近邻点,K为大于1的整数,第一点表示重建点云中的任意点;以及根据属性信息对应的滤波系数对重建点云中第一点对应的K个目标点进行滤波处理,确定重建点云中第一点的属性信息的滤波值;以及在确定出重建点云中至少一个点的属性信息的滤波值之后,根据至少一个点的属性信息的滤波值确定滤波后点云。
在一些实施例中,第二确定单元1303,还配置为基于重建点云中的第一点,利用K近邻搜索方式在重建点云中搜索预设值个候选点;以及分别计算第一点与预设值个候选点之间的距离值,从所得到的预设值个距离值中选取(K-1)个距离值,且(K-1)个距离值均小于预设值个距离值中剩余的距离值;以及根据(K-1)个距离值对应的候选点确定(K-1)个近邻点,将第一点和(K-1)个近邻点确定为第一点对应的K个目标点。
在一些实施例中,第二确定单元1303,还配置为若滤波后点云中的点的颜色分量不符合RGB颜色空间,则对滤波后点云进行颜色空间转换,使得滤波后点云中的点的颜色分量符合RGB颜色空间。
可以理解地,在本实施例中,“单元”可以是部分电路、部分处理器、部分程序或软件等等,当然也可以是模块,还可以是非模块化的。而且在本实施例中的各组成部分可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能模块的形式实现。
所述集成的单元如果以软件功能模块的形式实现并非作为独立的产品进行销售或使用时,可以存储在一个计算机可读取存储介质中。基于这样的理解,本实施例提供了一种计算机存储介质,应用于解码器130,该计算机存储介质存储有计算机程序,所述计算机程序被第二处理器执行时实现前述实施例中任一项所述的方法。
基于上述解码器130的组成以及计算机存储介质,参见图14,其示出了本申请实施例提供的解码器130的具体硬件结构示意图。如图14所示,解码器130可以包括:第二通信接口1401、第二存储器1402和第二处理器1403;各个组件通过第二总线系统1404耦合在一起。可理解,第二总线系统1404用于实现这些组件之间的连接通信。第二总线系统1404除包括数据总线之外,还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见,在图14中将各种总线都标为第二总线系统1404。其中,
第二通信接口1401,用于在与其他外部网元之间进行收发信息过程中,信号的接收和发送;
第二存储器1402,用于存储能够在第二处理器1403上运行的计算机程序;
第二处理器1403,用于在运行所述计算机程序时,执行:
解码码流,确定滤波标识信息;其中,滤波标识信息用于确定是否对待处理视频帧对应的重建点云进行滤波处理;
若滤波标识信息指示对重建点云的属性信息进行滤波处理,则解码码流,确定属性信息对应的滤波系数;
根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云。
可选地,作为另一个实施例,第二处理器1403还配置为在运行所述计算机程序时,执行前述实施例中任一项所述的方法。
可以理解,第二存储器1402与第一存储器1202的硬件功能类似,第二处理器1403与第一处理器1203的硬件功能类似;这里不再详述。
本实施例提供了一种解码器,该解码器可以包括解码单元和第二滤波单元。这样,解码器可以直接解码获得滤波系数,然后使用滤波系数对重建点云进行滤波处理,从而实现了对重建点云的优化,能够提升点云的质量;另外,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数时,还解决了初始点云中点和重建点云中点难以匹配的问题,并且通过对重建点云的属性信息进行滤波处理,不仅实现重建点云的质量增强,而且使得视频帧的质量也能够得到提升,进一步提升了压缩效率,进而节省码率,提高了编解码效率。
在本申请的再一实施例中,参见图15,其示出了本申请实施例提供的一种编解码系统的组成结构示意图。如图15所示,编解码系统150可以包括编码器1501和解码器1502。其中,编码器1501可以为前述实施例中任一项所述的编码器,解码器1502可以为前述实施例中任一项所述的解码器。
在本申请实施例中,该编解码系统150中,编码器1501利用初始点云和重建点云计算用于进行滤波处理的滤波系数,并在确定对重建点云进行滤波处理之后,才将对应的滤波系数传递至解码器;相应地,解码器1502可以直接解码获得滤波系数,然后使用滤波系数对重建点云进行滤波处理,从而实现了对重建点云的优化,能够提升点云的质量;另外,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数时,还解决了初始点云中点和重建点云中点难以匹配的问题,并且通过对重建点云的属性信息进行滤波处理,不仅实现重建点云的质量增强,而且使得视频帧的质量也能够得到提升,进一步 提升了压缩效率,进而节省码率,提高了编解码效率。
需要说明的是,在本申请中,术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含,从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素,而且还包括没有明确列出的其他要素,或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下,由语句“包括一个……”限定的要素,并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
上述本申请实施例序号仅仅为了描述,不代表实施例的优劣。
本申请所提供的几个方法实施例中所揭露的方法,在不冲突的情况下可以任意组合,得到新的方法实施例。
本申请所提供的几个产品实施例中所揭露的特征,在不冲突的情况下可以任意组合,得到新的产品实施例。
本申请所提供的几个方法或设备实施例中所揭露的特征,在不冲突的情况下可以任意组合,得到新的方法实施例或设备实施例。
以上所述,仅为本申请的具体实施方式,但本申请的保护范围并不局限于此,任何熟悉本技术领域的技术人员在本申请揭露的技术范围内,可轻易想到变化或替换,都应涵盖在本申请的保护范围之内。因此,本申请的保护范围应以所述权利要求的保护范围为准。
工业实用性
本申请实施例中,在编码器中,确定初始点云以及初始点云对应的重建点云;根据初始点云和重建点云,确定重建点云中属性信息的滤波系数;根据属性信息的滤波系数,确定重建点云对应的滤波后点云;根据重建点云和滤波后点云,确定滤波标识信息;若滤波标识信息指示对重建点云的属性信息进行滤波处理,则对滤波标识信息和属性信息对应的滤波系数进行编码,将所得到的编码比特写入码流。在解码器中,通过解码码流,确定滤波标识信息;其中,滤波标识信息用于确定是否对待处理视频帧对应的重建点云进行滤波处理;若滤波标识信息指示对重建点云的属性信息进行滤波处理,则解码码流,确定属性信息对应的滤波系数;根据属性信息对应的滤波系数,确定重建点云对应的滤波后点云。这样,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数,并在确定对重建点云进行滤波处理之后,才将对应的滤波系数传递至解码器;相应地,解码器可以直接解码获得滤波系数,然后使用滤波系数对重建点云进行滤波处理,从而实现了对重建点云的优化,能够提升点云的质量;另外,编码器利用初始点云和重建点云计算用于进行滤波处理的滤波系数时,还解决了初始点云中点和重建点云中点难以匹配的问题,并且通过对重建点云的属性信息进行滤波处理,不仅实现重建点云的质量增强,而且使得视频帧的质量也能够得到提升,进一步提升了压缩效率,进而节省码率,提高了编解码性能。

Claims (44)

  1. 一种解码方法,应用于解码器,所述方法包括:
    解码码流,确定滤波标识信息;其中,所述滤波标识信息用于确定是否对待处理视频帧对应的重建点云进行滤波处理;
    若所述滤波标识信息指示对所述重建点云的属性信息进行滤波处理,则解码码流,确定所述属性信息对应的滤波系数;
    根据所述属性信息对应的滤波系数,确定所述重建点云对应的滤波后点云。
  2. 根据权利要求1所述的方法,其中,所述待处理视频帧包括纹理视频帧和几何视频帧,所述方法还包括:
    解码码流,确定所述纹理视频帧的残差值和所述几何视频帧的残差值;
    在确定所述纹理视频帧的预测值后,根据所述纹理视频帧的预测值和所述纹理视频帧的残差值,确定所述纹理视频帧的重建值;
    在确定所述几何视频帧的预测值后,根据所述几何视频帧的预测值和所述几何视频帧的残差值,确定所述几何视频帧的重建值;
    基于所述纹理视频帧的重建值和所述几何视频帧的重建值,得到所述重建点云。
  3. 根据权利要求1所述的方法,其中,所述方法还包括:
    若所述滤波标识信息指示不对所述重建点云的属性信息进行滤波处理,则不执行解码码流,确定所述属性信息对应的滤波系数的步骤。
  4. 根据权利要求1所述的方法,其中,所述属性信息包括颜色分量,且所述颜色分量包括下述至少之一:第一颜色分量、第二颜色分量和第三颜色分量;其中,
    若所述颜色分量符合RGB颜色空间,则确定所述第一颜色分量、所述第二颜色分量和所述第三颜色分量依次为:R分量、G分量、B分量;
    若所述颜色分量符合YUV颜色空间,则确定所述第一颜色分量、所述第二颜色分量和所述第三颜色分量依次为:Y分量、U分量、V分量。
  5. 根据权利要求1所述的方法,其中,所述解码码流,确定滤波标识信息,包括:
    解码码流,确定待处理分量的滤波标识信息;
    其中,所述待处理分量包括下述至少之一:第一颜色分量、第二颜色分量和第三颜色分量,且所述待处理分量的滤波标识信息用于指示是否对所述重建点云的属性信息的待处理分量进行滤波处理。
  6. 根据权利要求5所述的方法,其中,所述解码码流,确定待处理分量的滤波标识信息,包括:
    若所述待处理分量的滤波标识信息的取值为第一值,则确定对所述重建点云的属性信息的待处理分量进行滤波处理;
    若所述待处理分量的滤波标识信息的取值为第二值,则确定不对所述重建点云的属性信息的待处理分量进行滤波处理;
    相应地,所述若所述滤波标识信息指示对所述重建点云的属性信息进行滤波处理,则解码码流,确定所述属性信息对应的滤波系数,包括:
    若所述待处理分量的滤波标识信息的取值为第一值,则解码所述码流,确定所述属性信息的待处理分量对应的滤波系数。
  7. 根据权利要求5所述的方法,其中,当所述待处理分量包括第二颜色分量和第三颜色分量时,所述解码码流,确定待处理分量的滤波标识信息,包括:
    解码所述码流,确定第一滤波标识信息和第二滤波标识信息;
    其中,所述第一滤波标识信息用于指示是否对所述重建点云的属性信息的第二颜色分量进行滤波处理,所述第二滤波标识信息用于指示是否对所述重建点云的属性信息的第三颜色分量进行滤波处理。
  8. 根据权利要求7所述的方法,其中,所述若所述滤波标识信息指示对所述重建点云的属性信息进行滤波处理,则解码码流,确定所述属性信息对应的滤波系数,包括:
    若所述第一滤波标识信息指示对所述重建点云的属性信息的第二颜色分量进行滤波处理,则解码码流,确定第二颜色分量对应的滤波系数;
    和/或
    若所述第二滤波标识信息指示对所述重建点云的属性信息的第三颜色分量进行滤波处理,则解码码流,确定第三颜色分量对应的滤波系数;
    相应地,所述根据所述属性信息对应的滤波系数,确定所述重建点云对应的滤波后点云,包括:
    根据所述第二颜色分量对应的滤波系数和/或所述第三颜色分量对应的滤波系数,确定所述重建点云对应的滤波后点云。
  9. 根据权利要求5所述的方法,其中,当所述待处理分量包括第一颜色分量、第二颜色分量和第三颜色分量时,所述解码码流,确定待处理分量的滤波标识信息,包括:
    解码所述码流,确定第一滤波标识信息、第二滤波标识信息和第三滤波标识信息;
    其中,所述第一滤波标识信息用于指示是否对所述重建点云的属性信息的第二颜色分量进行第一滤波处理,所述第二滤波标识信息用于指示是否对所述重建点云的属性信息的第三颜色分量进行第二滤波处理,所述第三滤波标识信息用于指示是否对所述重建点云的属性信息的第一颜色分量进行第三滤波处理。
  10. 根据权利要求9所述的方法,其中,所述第三滤波处理对应的滤波器与所述第一滤波处理和所述第二滤波处理对应的滤波器存在不同。
  11. 根据权利要求1所述的方法,其中,所述根据所述属性信息对应的滤波系数,确定所述重建点云对应的滤波后点云,包括:
    确定所述重建点云中第一点对应的K个目标点;其中,所述K个目标点包括所述第一点以及与所述第一点相邻的(K-1)个近邻点,K为大于1的整数,所述第一点表示所述重建点云中的任意点;
    根据所述属性信息对应的滤波系数对所述重建点云中第一点对应的K个目标点进行滤波处理,确定所述重建点云中所述第一点的属性信息的滤波值;
    在确定出所述重建点云中至少一个点的属性信息的滤波值之后,根据所述至少一个点的属性信息的滤波值确定所述滤波后点云。
  12. 根据权利要求11所述的方法,其中,所述确定所述重建点云中第一点对应的K个目标点,包括:
    基于所述重建点云中的第一点,利用K近邻搜索方式在所述重建点云中搜索预设值个候选点;
    分别计算所述第一点与所述预设值个候选点之间的距离值,从所得到的预设值个距离值中选取(K-1)个距离值,且所述(K-1)个距离值均小于所述预设值个距离值中剩余的距离值;
    根据(K-1)个距离值对应的候选点确定(K-1)个近邻点,将所述第一点和所述(K-1)个近邻点确定为所述第一点对应的K个目标点。
  13. 根据权利要求1所述的方法,其中,在所述确定所述重建点云对应的滤波后点云之后,所述方法还包括:
    若所述滤波后点云中的点的颜色分量不符合RGB颜色空间,则对所述滤波后点云进行颜色空间转换,使得所述滤波后点云中的点的颜色分量符合RGB颜色空间。
  14. 一种编码方法,应用于编码器,所述方法包括:
    确定初始点云以及所述初始点云对应的重建点云;
    根据所述初始点云和所述重建点云,确定所述重建点云中属性信息的滤波系数;
    根据所述属性信息的滤波系数,确定所述重建点云对应的滤波后点云;
    根据所述重建点云和所述滤波后点云,确定滤波标识信息;
    若所述滤波标识信息指示对所述重建点云的属性信息进行滤波处理,则对所述滤波标识信息和所述属性信息对应的滤波系数进行编码,将所得到的编码比特写入码流。
  15. 根据权利要求14所述的方法,其中,所述确定所述初始点云对应的重建点云,包括:
    利用预设编码方式对所述初始点云进行编码及重建处理,得到第一重建点云;
    对所述第一重建点云进行几何恢复处理,得到所述重建点云;
    其中,所述预设编码方式用于指示对所述初始点云进行几何有损且属性有损编码。
  16. 根据权利要求15所述的方法,其中,所述对所述第一重建点云进行几何恢复处理,得到所述重建点云,包括:
    对所述第一重建点云进行几何补偿处理,得到第二重建点云;
    对所述第二重建点云进行尺度缩放处理,得到所述重建点云;其中,所述重建点云与所述初始点云的大小相同且几何位置相同。
  17. 根据权利要求15所述的方法,其中,所述方法还包括:
    根据预设搜索方式对所述初始点云进行近邻点搜索,确定所述重建点云中的点在所述初始点云中的对应点;
    根据所述确定的对应点构建匹配点云,将所述匹配点云作为所述初始点云,建立所述重建点云中的点与所述初始点云中的点之间的对应关系。
  18. 根据权利要求17所述的方法,其中,所述预设搜索方式为K近邻搜索方式;
    所述根据预设搜索方式对所述初始点云进行近邻点搜索,确定所述重建点云中的点在所述输入点云中的对应点,包括:
    基于所述重建点云中的当前点,利用所述预设搜索方式在所述初始点云中搜索到第一常数值个点;
    分别计算所述当前点与所述第一常数值个点之间的距离值;
    根据所述距离值,确定所述当前点的对应点。
  19. 根据权利要求18所述的方法,其中,所述根据所述距离值,确定所述当前点的对应点,包括:
    从所述距离值中选取最小距离值;
    若所述最小距离值的个数为1个,则将所述最小距离值对应的点确定为所述当前点的对应点;
    若所述最小距离值的个数为多个,则根据所述最小距离值对应的多个点,确定所述当前点的对应点。
  20. 根据权利要求14所述的方法,其中,所述根据所述属性信息的滤波系数,确定所述重建点云对应的滤波后点云,包括:
    确定所述重建点云中第一点对应的K个目标点;其中,所述K个目标点包括所述第一点以及与所述第一点相邻的(K-1)个近邻点,K为大于1的整数,所述第一点表示所述重建点云中的任意点;
    根据所述属性信息对应的滤波系数对所述重建点云中第一点对应的K个目标点进行滤波处理,确定所述重建点云中所述第一点的属性信息的滤波值;
    在确定出所述重建点云中至少一个点的属性信息的滤波值之后,根据所述至少一个点的属性信息的滤波值确定所述滤波后点云。
  21. 根据权利要求20所述的方法,其中,所述确定所述重建点云中第一点对应的K个目标点,包括:
    基于所述重建点云中的第一点,利用K近邻搜索方式在所述重建点云中搜索预设值个候选点;
    分别计算所述第一点与所述预设值个候选点之间的距离值,从所得到的预设值个距离值中选取(K-1)个距离值,且所述(K-1)个距离值均小于所述预设值个距离值中剩余的距离值;
    根据(K-1)个距离值对应的候选点确定(K-1)个近邻点,将所述第一点和所述(K-1)个近邻点确定为所述第一点对应的K个目标点。
  22. 根据权利要求14所述的方法,其中,所述根据所述初始点云和所述重建点云,确定所述重建点云中属性信息的滤波系数,包括:
    根据所述初始点云中点的属性信息的原始值,确定第一属性参数;
    根据所述重建点云中点对应的K个目标点的属性信息的重建值,确定第二属性参数;
    基于所述第一属性参数和所述第二属性参数,确定所述属性信息的滤波系数。
  23. 根据权利要求22所述的方法,其中,所述基于所述第一属性参数和所述第二属性参数,确定所述属性信息的滤波系数,包括:
    根据所述第一属性参数和所述第二属性参数,确定互相关参数;
    根据所述第二属性参数确定自相关参数;
    根据所述互相关参数和所述自相关参数进行系数计算,得到所述属性信息的滤波系数。
  24. 根据权利要求14至23任一项所述的方法,其中,所述属性信息包括颜色分量,且所述颜色分量包括下述至少之一:第一颜色分量、第二颜色分量和第三颜色分量;其中,
    若所述颜色分量符合RGB颜色空间,则确定所述第一颜色分量、所述第二颜色分量和所述第三颜色分量依次为:R分量、G分量、B分量;
    若所述颜色分量符合YUV颜色空间,则确定所述第一颜色分量、所述第二颜色分量和所述第三颜色分量依次为:Y分量、U分量、V分量。
  25. 根据权利要求14所述的方法,其中,所述根据所述重建点云和所述滤波后点云,确定滤波标识信息,包括:
    确定所述重建点云的属性信息的待处理分量的第一代价值,以及确定所述滤波后点云的属性信息的待处理分量的第二代价值;
    根据所述第一代价值和所述第二代价值,确定所述待处理分量的滤波标识信息;
    根据所述待处理分量的滤波标识信息,得到所述滤波标识信息。
  26. 根据权利要求25所述的方法,其中,所述根据所述第一代价值和所述第二代价值,确定所述待处理分量的滤波标识信息,包括:
    若所述第二代价值小于所述第一代价值,则确定所述待处理分量的滤波标识信息的取值为第一值;
    若所述第二代价值大于所述第一代价值,则确定所述待处理分量的滤波标识信息的取值为第二值。
  27. 根据权利要求25所述的方法,其中,所述确定所述重建点云的属性信息的待处理分量的第一代价值,包括:
    利用率失真代价方式对所述重建点云的属性信息的待处理分量进行代价值计算,将所得到的第一率失真值作为所述第一代价值;
    相应地,所述确定所述滤波后点云的属性信息的待处理分量的第二代价值,包括:
    利用率失真代价方式对所述滤波后点云的属性信息的待处理分量进行代价值计算,将所得到的第二率失真值作为得到所述第二代价值。
  28. 根据权利要求25所述的方法,其中,所述确定所述重建点云的属性信息的待处理分量的第一代价值,包括:
    利用率失真代价方式对所述重建点云的属性信息的待处理分量进行代价值计算,得到第一率失真值;
    利用预设性能衡量指标对所述重建点云的属性信息的待处理分量进行性能值计算,得到第一性能值;
    根据所述第一率失真值和所述第一性能值,确定所述第一代价值;
    相应地,所述确定所述滤波后点云的属性信息的待处理分量的第二代价值,包括:
    利用率失真代价方式对所述滤波后点云的属性信息的待处理分量进行代价值计算,得到第二率失真值;
    利用预设性能衡量指标对所述滤波后点云的属性信息的待处理分量进行性能值计算,得到第二性能值;
    根据所述第二率失真值和所述第二性能值,确定所述第二代价值。
  29. 根据权利要求28所述的方法,其中,所述根据所述第一代价值和所述第二代价值,确定所述待处理分量的滤波标识信息,包括:
    若所述第二性能值大于所述第一性能值且所述第二率失真值小于所述第一率失真值,则确定所述待处理分量的滤波标识信息的取值为第一值;
    若所述第二性能值小于所述第一性能值,则确定所述待处理分量的滤波标识信息的取值为第二值。
  30. 根据权利要求28所述的方法,其中,所述根据所述第一代价值和所述第二代价值,确定所述待处理分量的滤波标识信息,包括:
    若所述第二性能值大于所述第一性能值且所述第二率失真值小于所述第一率失真值,则确定所述待处理分量的滤波标识信息的取值为第一值;
    若所述第二率失真值大于所述第一率失真值,则确定所述待处理分量的滤波标识信息的取值为第二值。
  31. 根据权利要求26、29或30所述的方法,其中,所述方法还包括:
    若所述待处理分量的滤波标识信息的取值为第一值,则确定所述待处理分量的滤波标识信息指示对所述重建点云的属性信息的待处理分量进行滤波处理;
    若所述待处理分量的滤波标识信息的取值为第二值,则确定所述待处理分量的滤波标识信息指示不对所述重建点云的属性信息的待处理分量进行滤波处理。
  32. 根据权利要求25所述的方法,其中,所述根据所述待处理分量的滤波标识信息,得到所述滤波标识信息,包括:
    在所述待处理分量包括第二颜色分量和第三颜色分量的情况下,得到第一滤波标识信息和第二滤波标识信息;
    其中,所述第一滤波标识信息用于指示是否对所述重建点云的属性信息的第二颜色分量进行滤波处理,所述第二滤波标识信息用于指示是否对所述重建点云的属性信息的第三颜色分量进行滤波处理。
  33. 根据权利要求32所述的方法,其中,所述方法还包括:
    若所述第一滤波标识信息和所述第二滤波标识信息中至少一个为第一值,则确定所述滤波标识信息指示对所述重建点云进行滤波处理;
    若所述第一滤波标识信息和所述第二滤波标识信息中全部为第二值,则确定所述滤波标识信息指示不对所述重建点云进行滤波处理。
  34. 根据权利要求33所述的方法,其中,所述方法还包括:
    若所述滤波标识信息指示不对所述重建点云的属性信息进行滤波处理,则仅对所述滤波标识信息进行编码,将所得到的编码比特写入所述码流。
  35. 根据权利要求25所述的方法,其中,所述根据所述待处理分量的滤波标识信息,得到所述滤波标识信息,包括:
    在所述待处理分量包括第一颜色分量、第二颜色分量和第三颜色分量的情况下,得到第一滤波标识信息、第二滤波标识信息和第三滤波标识信息;
    其中,所述第一滤波标识信息用于指示是否对所述重建点云的属性信息的第二颜色分量进行第一滤波处理,所述第二滤波标识信息用于指示是否对所述重建点云的属性信息的第三颜色分量进行第二滤波处理,所述第三滤波标识信息用于指示是否对所述重建点云的属性信息的第一颜色分量进行第三滤波处理。
  36. 根据权利要求35所述的方法,其中,所述第三滤波处理对应的滤波器与所述第一滤波处理和所述第二滤波处理对应的滤波器存在不同。
  37. 根据权利要求14所述的方法,其中,所述方法还包括:
    根据所述重建点云的点数,确定所述K的取值;和/或,
    根据所述重建点云的量化参数,确定所述K的取值;和/或,
    根据所述重建点云中点的邻域差异值,确定所述K的取值;其中,所述邻域差异值是基于所述点的属性信息与至少一个近邻点的属性信息之间的分量差值计算得到的。
  38. 根据权利要求14所述的方法,其中,所述方法还包括:
    根据所述初始点云,确定几何视频帧和纹理视频帧;
    确定所述几何视频帧的预测值和所述纹理视频帧的预测值;
    根据所述几何视频帧的原始值和所述几何视频帧的预测值,确定所述几何视频帧的残差值;以及对所述几何视频帧的残差值进行编码,将所得到的编码比特写入码流;
    根据所述纹理视频帧的原始值和所述纹理视频帧的预测值,确定所述纹理视频帧的残差值;以及对所述纹理视频帧的残差值进行编码,将所得到的编码比特写入码流。
  39. 一种码流,所述码流是根据待编码信息进行比特编码生成的;其中,所述待编码信息包括下述至少之一:几何视频帧的残差值、纹理视频帧的残差值、滤波标识信息和滤波系数。
  40. 一种编码器,所述编码器包括第一确定单元、第一滤波单元和编码单元;其中,
    所述第一确定单元,配置为确定初始点云以及所述初始点云对应的重建点云;以及根据所述初始点云和所述重建点云,确定所述重建点云中属性信息的滤波系数;
    所述第一滤波单元,配置为根据所述属性信息的滤波系数,确定所述重建点云对应的滤波后点云;
    所述第一确定单元,还配置为根据所述重建点云和所述滤波后点云,确定滤波标识信息;
    所述编码单元,配置为若所述滤波标识信息指示对所述重建点云的属性信息进行滤波处理,则对所述滤波标识信息和所述属性信息对应的滤波系数进行编码,将所得到的编码比特写入码流。
  41. 一种编码器,所述编码器包括第一存储器和第一处理器;其中,
    所述第一存储器,用于存储能够在所述第一处理器上运行的计算机程序;
    所述第一处理器,用于在运行所述计算机程序时,执行如权利要求14至38任一项所述的方法。
  42. 一种解码器,所述解码器包括解码单元和第二滤波单元;其中,
    所述解码单元,配置为解码码流,确定滤波标识信息;其中,所述滤波标识信息用于确定是否对待处理视频帧对应的重建点云进行滤波处理;以及若所述滤波标识信息指示对所述重建点云的属性信息进行滤波处理,则解码码流,确定所述属性信息对应的滤波系数;
    所述第二滤波单元,配置为根据所述属性信息对应的滤波系数,确定所述重建点云对应的滤波后点云。
  43. 一种解码器,所述解码器包括第二存储器和第二处理器;其中,
    所述第二存储器,用于存储能够在所述第二处理器上运行的计算机程序;
    所述第二处理器,用于在运行所述计算机程序时,执行如权利要求1至13任一项所述的方法。
  44. 一种计算机存储介质,其中,所述计算机存储介质存储有计算机程序,所述计算机程序被执行时实现如权利要求1至13任一项所述的方法、或者实现如权利要求14至38任一项所述的方法。
PCT/CN2021/143943 2021-12-31 2021-12-31 编解码方法、码流、编码器、解码器以及存储介质 WO2023123467A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/143943 WO2023123467A1 (zh) 2021-12-31 2021-12-31 编解码方法、码流、编码器、解码器以及存储介质

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/143943 WO2023123467A1 (zh) 2021-12-31 2021-12-31 编解码方法、码流、编码器、解码器以及存储介质

Publications (1)

Publication Number Publication Date
WO2023123467A1 true WO2023123467A1 (zh) 2023-07-06

Family

ID=86997193

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/143943 WO2023123467A1 (zh) 2021-12-31 2021-12-31 编解码方法、码流、编码器、解码器以及存储介质

Country Status (1)

Country Link
WO (1) WO2023123467A1 (zh)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110971898A (zh) * 2018-09-30 2020-04-07 华为技术有限公司 点云编解码方法和编解码器
WO2020189876A1 (ko) * 2019-03-15 2020-09-24 엘지전자 주식회사 포인트 클라우드 데이터 송신 장치, 포인트 클라우드 데이터 송신 방법, 포인트 클라우드 데이터 수신 장치 및 포인트 클라우드 데이터 수신 방법
US20210217139A1 (en) * 2018-07-11 2021-07-15 Sony Corporation Image processing apparatus and image processing method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210217139A1 (en) * 2018-07-11 2021-07-15 Sony Corporation Image processing apparatus and image processing method
CN110971898A (zh) * 2018-09-30 2020-04-07 华为技术有限公司 点云编解码方法和编解码器
WO2020189876A1 (ko) * 2019-03-15 2020-09-24 엘지전자 주식회사 포인트 클라우드 데이터 송신 장치, 포인트 클라우드 데이터 송신 방법, 포인트 클라우드 데이터 수신 장치 및 포인트 클라우드 데이터 수신 방법

Similar Documents

Publication Publication Date Title
CN114424542B (zh) 具有非规范平滑的基于视频的点云压缩
CN111133476B (zh) 包括多个点的点云的压缩和解压缩的系统、设备和方法
RU2683165C1 (ru) Внутреннее предсказание блочного копирования с асимметричными разделами и схемами поиска на стороне кодера, диапазоны поиска и подходы к разделению
US9152875B2 (en) Preserving text quality in video encoding
JP2022502961A (ja) 3d補助データを用いた動き推定
KR20200044653A (ko) 딥 뉴럴 네트워크를 이용한 영상의 ai 부호화 및 ai 복호화 방법, 및 장치
US20220217370A9 (en) Spatial Varying Transform for Video Coding
CN114788264B (zh) 用于发出虚拟边界和环绕运动补偿的信号的方法
US20110091120A1 (en) Methods and apparatus for texture compression using patch-based sampling texture synthesis
WO2023130333A1 (zh) 编解码方法、编码器、解码器以及存储介质
US10812832B2 (en) Efficient still image coding with video compression techniques
WO2020001325A1 (zh) 一种图像编码方法、解码方法、编码器、解码器及存储介质
CN112399165B (zh) 解码方法及装置以及计算机设备和存储介质
CN117897952A (zh) 用于执行组合帧间和帧内预测的方法和系统
CN113243106B (zh) 一种用于视频图像的预测块的帧内预测的设备和方法
WO2022067775A1 (zh) 点云的编码、解码方法、编码器、解码器以及编解码系统
JP2015019364A (ja) 画像圧縮装置及び方法
US11310496B2 (en) Determining quality values for blocks of encoded video
WO2023123467A1 (zh) 编解码方法、码流、编码器、解码器以及存储介质
CN114793282B (zh) 带有比特分配的基于神经网络的视频压缩
WO2022067776A1 (zh) 点云的解码、编码方法、解码器、编码器和编解码系统
WO2023123471A1 (zh) 编解码方法、码流、编码器、解码器以及存储介质
JP7293263B2 (ja) イントラ予測のためのデバイスおよび方法
WO2023201450A1 (zh) 编解码方法、码流、编码器、解码器以及存储介质
WO2022170511A1 (zh) 点云解码方法、解码器及计算机存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21969812

Country of ref document: EP

Kind code of ref document: A1