WO2021043136A1 - Point cloud projection transmission method, and reception processing method and system - Google Patents

Point cloud projection transmission method, and reception processing method and system Download PDF

Info

Publication number
WO2021043136A1
WO2021043136A1 PCT/CN2020/112857 CN2020112857W WO2021043136A1 WO 2021043136 A1 WO2021043136 A1 WO 2021043136A1 CN 2020112857 W CN2020112857 W CN 2020112857W WO 2021043136 A1 WO2021043136 A1 WO 2021043136A1
Authority
WO
WIPO (PCT)
Prior art keywords
point cloud
projection
segment
user
perspective
Prior art date
Application number
PCT/CN2020/112857
Other languages
French (fr)
Chinese (zh)
Inventor
徐异凌
王丹盈
徐英展
朱文婕
Original Assignee
上海交通大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN201910827750.9A external-priority patent/CN112449171B/en
Priority claimed from CN202010863785.0A external-priority patent/CN114119346A/en
Application filed by 上海交通大学 filed Critical 上海交通大学
Publication of WO2021043136A1 publication Critical patent/WO2021043136A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/139—Format conversion, e.g. of frame-rate or size
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/194—Transmission of image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding

Definitions

  • the invention belongs to the field of point clouds, and in particular relates to a point cloud projection transmission method, reception processing method and system based on a user's perspective.
  • Three-dimensional point cloud data is a new type of image media data, which can provide users with more accurate and vivid image information, and bring higher fidelity and more interactive communication experience.
  • Point cloud data is the surface information of the object obtained by scanning, including three-dimensional coordinate data, depth information, color information, and so on.
  • the maturity of 3D scanning technology and the perfection of the system architecture have improved the accuracy of point cloud data, while also greatly increasing the amount of point cloud data.
  • point cloud data compression algorithms for static and dynamic point cloud data, and different types of point cloud data such as machine perception and human eye perception.
  • a typical point cloud compression algorithm is to convert 3D point cloud data into 2D image data, and then perform data processing, one of which is video-based point cloud compression (Video- based Point Cloud Compression, VPCC) algorithm.
  • Video- based Point Cloud Compression Video- based Point Cloud Compression
  • This compression method first projects a 3D point cloud onto a 2D plane to obtain occupancy map information, geometric information, attribute information, and auxiliary information.
  • the attribute information usually includes texture information and color information. Therefore, the compressed information is usually divided into four categories. Data is transferred. Through this method, the overall compressed transmission of the point cloud sequence is realized.
  • a typical compression scheme applied to point cloud coding and transmission is to convert three-dimensional point cloud data into two-dimensional images for processing through segmented projection.
  • This solution uses the existing video coding tools to encode and transmit the point cloud, in which it is necessary to sequentially carry out the division of the point cloud, the subdivision, segment projection, blank filling and other steps to achieve the segment projection of the point cloud. Then, the two-dimensional data obtained by projection can be encoded and transmitted with the help of video encoding technologies such as HEVC. Through this method, the overall compressed transmission of the point cloud sequence is realized.
  • Patent document CN104778691B (application number: 201510160208.4) discloses a simplified processing method for three-dimensional point cloud data.
  • the invention mainly solves the technical problems of poor retention of three-dimensional object surface features, poor reconstruction effect, and large amount of calculation existing in the existing method for streamlining three-dimensional point clouds.
  • the invention uses the information in the color image synchronized by the Kinect camera to streamline the three-dimensional point cloud when collecting the point cloud data.
  • the method combines the color map and the point cloud for streamlining, which can avoid some curvature of the object surface caused by the collection and streamlining process There is no significant change but the point cloud data of the characteristic information is lost; you can set the corresponding reduction rate according to the gray information of the object itself for reduction; you can also reserve the characteristic points that you think are more important according to subjective factors, and selectively simplify Point cloud.
  • the user cannot observe the entire content of a point cloud object.
  • the user cannot see the media content on the back of the point cloud.
  • the process of cloud processing does not differentiate users' attention to differences in various aspects.
  • the existing point cloud compression scheme implements point cloud transmission. Since the point cloud fragments that can be observed and the point cloud fragments that cannot be observed are compressed with the same accuracy and provide the same definition of presentation, this will Unnecessary demand for transmission bandwidth, etc. will be put forward. For this unnecessary amount of information caused by the occlusion of the point cloud, the current overall compression transmission scheme for the point cloud sequence does not propose a corresponding solution. How to make distinctions based on the degree of attention of the users' different perspectives, and to reduce this information redundancy without affecting the quality of observations, and adapt it to actual user application scenarios, are key issues that need to be resolved urgently.
  • the purpose of the present invention is to provide a point cloud projection transmission method, reception processing method and system, which introduces the user's perspective to improve the quality of point cloud projection processing, and optionally, projection is performed on the side with high attention that has a greater impact on the user's perception
  • the vector is modified and perfected, or combined with optimized point cloud compression algorithm to achieve.
  • the present invention provides a point cloud projection transmission method based on the user's perspective, which has such features, including: preliminary division of point cloud objects on a standard vector to obtain the point cloud segment to be processed; and determination based on the user's perspective
  • the projection transmission scheme is to process the point cloud segment to be processed.
  • the point cloud projection transmission method may also have the feature that the closest standard normal vector is taken as the classification category of the midpoint of the point cloud object; based on six standards
  • the normal vector divides the midpoint of the point cloud object into multiple first-level point cloud fragments; in the same first-level point cloud fragment, the points with the same normal vector and the distance less than the predetermined value are then divided into second-level point cloud fragments as to be processed Cloud fragment.
  • the point cloud projection transmission method may also have such a feature, wherein the projection transmission scheme includes: further dividing the to-be-processed point cloud segment to obtain points corresponding to the perspective category Cloud fragments, including the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed; the point cloud fragments that need to be observed are assigned priority resources for encoding transmission, and the point cloud fragments that do not need to be observed are compared Prioritize the allocation of secondary resources for coding transmission.
  • the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have such a feature, wherein the projection transmission scheme includes any one or more of the following: no point cloud modification Projection plan, improve the point cloud projection plan, modify the standard normal vector plan according to the user's perspective.
  • the solution of not modifying the point cloud projection method includes: placing the point cloud object in the standard vector
  • the above preliminary division is to associate the parameter list with the standard vector as the projection vector to obtain the point cloud segment to be processed.
  • the point cloud segment to be processed is divided into the point cloud segment that needs to be observed and the point cloud segment that needs to be observed.
  • the point cloud projection transmission method may also have the feature of setting corresponding points for the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed.
  • the encoding and decoding parameters of each point cloud segment are recorded as additional information for encoding and transmission.
  • the improved point cloud projection solution includes: further dividing the to-be-processed point cloud fragments to obtain corresponding
  • the point cloud fragments of the perspective category include the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed; the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed are projected onto the projection plane, according to The size of the projection plane needs to be changed to obtain the need to observe pictures and the need not to observe pictures.
  • the point cloud projection transmission method may also have such characteristics, and further include: as the first improved point cloud projection solution, the point cloud segment and the Under the premise that the point cloud fragments that do not need to be observed are misaligned, the pictures that do not need to be observed are merged into the pictures that need to be observed, and a projected picture is obtained, which is compressed and encoded.
  • the point cloud projection transmission method may also have the feature that a projected picture obtained forms a set of video sequences, and the point cloud that does not need to be observed is recorded
  • the index parameter index corresponding to the segment and the scale of the size change are used as newly added additional information for encoding transmission.
  • the point cloud projection transmission method may also have such characteristics, and further include: as a second improved point cloud projection solution, there will be a need to observe the picture and the need not to observe the picture. Compress and encode separately.
  • the point cloud projection transmission method may also have the feature that the projected images that need to be observed and the images that do not need to be observed respectively form two sets of video sequences, and the two sets are encoded.
  • Video sequence record the change ratio of the video sequence corresponding to the point cloud segment that does not need to be observed, encode and transmit as new additional information, and further record the position of each group of video sequences in the coded stream.
  • the third improved point cloud projection solution includes: the point cloud segment to be processed according to six standard methods The vectors are grouped and projected onto six projection planes, and the size of the six projection planes is changed according to the user's perspective based on the user's perspective, and compression coding is performed respectively.
  • the point cloud segment corresponding to each standard normal vector can be respectively projected to each group of video sequences, and six groups are encoded.
  • Video sequence record the change ratio of the video sequence corresponding to the point cloud segment that does not need to be observed, encode and transmit as new additional information, and further record the position of each group of video sequences in the coded stream.
  • the modification of the standard normal vector scheme according to the user's perspective includes: determining the perspective according to the user's observation perspective Vector; rotate the view vector with one of the standard normal vectors, and then perform the same rotation mapping processing on the other standard normal vectors, and determine the projection direction after the rotation mapping on the premise that the overall modification amount is less than a predetermined value.
  • the point cloud projection transmission method based on the user's perspective may also have the feature, wherein the step of reducing the overall modification amount to be less than a predetermined value includes: the perspective vector corresponds to the standard normal vector , And after the rotation, the sum of the dot products of the six standard normal vectors after the rotation mapping including the viewing angle vector and the corresponding original six standard normal vectors takes the maximum value or a larger value within a predetermined range.
  • the point cloud projection transmission method may also have the feature that the standard normal vector after the rotation mapping is adjusted will be encoded as the newly added additional information transmission.
  • the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have such a feature, wherein the projection transmission scheme includes: the point cloud segment to be processed is distinguished from the main segment and the point cloud segment according to the user's perspective. Multiple edge segments; considering the spatial connection between the edge segment and the main segment, the projection vector of the edge segment is maintained or changed based on the user's perspective; subsequent point cloud processing is performed after projection.
  • the judgment basis considering the spatial connection between the edge segment and the main segment includes: The user’s perspective determines the principal vector in the standard normal vector; the principal segment and the edge segment are respectively projected along the principal vector direction to obtain the principal two-dimensional projection map and the edge two-dimensional projection map; based on each edge two-dimensional projection map, and each The main two-dimensional projection map is calculated to obtain the intersection ratio; multiple intersection ratios are compared with the preset threshold value as the basis for judging the spatial connection.
  • the point cloud projection transmission method may also have the feature that when multiple intersection ratios are greater than a preset threshold, the edge The projection vector of the segment is changed to the main vector; if none of the multiple intersection ratios is greater than the preset threshold, the projection vector of the edge segment is maintained.
  • the preset threshold value depends on the surface characteristics of the point cloud object to distinguish the object point cloud object.
  • the determination of the threshold value of the surface characteristics of the character point cloud object and the surface characteristics of the character point cloud object can be further adjusted and determined according to the experimental results.
  • the point cloud projection transmission method may also have the feature, wherein, according to the position where the user views the point cloud object, a point cloud is used to point the user to the point cloud.
  • the vector of the observation point represents the viewing angle of the user, which is called the viewing angle vector.
  • the distance between the viewing angle vector and the six standard normal vectors is calculated, and the standard normal vector with the closest distance to the viewing angle vector is called the principal vector under the viewing angle.
  • the point cloud projection transmission method based on the user's perspective may also have the feature, wherein, in addition to distinguishing the main segment and multiple edge segments of the point cloud object according to the user's perspective, Including distinguishing irrelevant fragments, among which, the standard normal vector closest to the user's perspective distance is determined as the main vector; among multiple standard normal vectors, the point cloud fragments to be processed that are the same as the main vector are determined as the main fragment; The to-be-processed point cloud segment opposite to the main vector in the standard normal vector is determined as an irrelevant segment; the to-be-processed point cloud segment associated with the main vector in the multiple standard normal vectors is determined as an edge segment.
  • the point cloud projection transmission method may also have the feature that, in the subsequent point cloud processing after projection, the projection relationship identification field is used to identify that the point cloud The projection vector of the point in the object after processing and adjustment, including the index value of the point cloud segment and the corresponding projection vector value.
  • the present invention also provides a point cloud receiving and processing method based on the user's perspective, which is characterized in that the codec parameters determined by the user's perspective are used to obtain a video sequence from the received code stream, and the video sequence is processed to obtain a point cloud. Segment, and decode and reconstruct each point cloud segment to obtain a point cloud object.
  • the point cloud receiving and processing method may also have the feature, in which the process of encoding, decoding and subsequent reconstruction of each point cloud segment according to the encoding and decoding parameters, Contains any one or more of the following: the encoding and decoding parameters are set at the sending end for each point cloud segment, distinguishing the point cloud segment that needs to be observed and the point cloud segment that does not need to be observed, and performing subsequent point cloud object reconstruction; or According to the received parameter information and size change information, the point cloud segment is restored to its original size, and then the subsequent point cloud object reconstruction is performed; or each video sequence is restored according to the position of each video sequence in the code stream.
  • the present invention also provides a point cloud projection transmission method based on the user's perspective, which is characterized in that: a segment generation module is used to preliminarily divide the point cloud object on the standard vector to obtain the point cloud segment to be processed; projection coding processing The module determines the projection transmission scheme based on the user's perspective, and processes the point cloud segment to be processed.
  • the present invention also provides a point cloud receiving processing system based on the user's perspective, which is characterized in that: the inverse processing module uses the codec parameters determined from the user's perspective to obtain a video sequence from the received code stream, and to obtain a video sequence for the video sequence.
  • the point cloud segment is obtained by processing, and each point cloud segment is decoded and reconstructed to obtain a point cloud object.
  • the present invention also provides a point cloud projection transmission method based on the user's perspective, which is characterized in that: the point cloud object is preliminarily divided on the standard vector to obtain the point cloud segment to be processed; the point cloud segment to be processed is further divided to obtain
  • the point cloud segment corresponding to the perspective category includes the point cloud segment that needs to be observed and the point cloud segment that does not need to be observed; priority resources are assigned to the point cloud segment that needs to be observed for encoding and transmission, and for the perspective that does not need to be observed Point cloud fragments are compared with priority allocation of secondary resources for encoding transmission
  • the present invention also provides a point cloud projection transmission method based on the user's perspective, which may also have the following characteristics: including: the point cloud segment to be processed is distinguished from the main segment and multiple edge segments according to the user's perspective; The spatial connection between the fragment and the main fragment is to maintain the projection vector of the edge fragment or change it based on the user's perspective; after the projection, the subsequent point cloud processing is performed.
  • the present invention also provides a point cloud projection transmission method based on the user's perspective, which may also have such features: including: preliminary division of the point cloud objects on the standard vector to obtain the to-be-processed point cloud segment, which is distinguished according to the user's perspective Generate the main segment and multiple edge segments; consider the spatial connection between the edge segment and the main segment, maintain the projection vector of the edge segment or change it based on the user's perspective; use the adjusted and changed main segment as the point to be observed Cloud fragments, adjusted and changed edge fragments and irrelevant fragments as point cloud fragments that do not need to be observed; and assign priority resources to the point cloud fragments that need to be observed for encoding transmission, and point cloud fragments that do not need to be observed. Compared with priority allocation of secondary resources for coding transmission.
  • the point cloud object is preliminarily divided on the standard vector to obtain the point cloud segment to be processed
  • the projection transmission scheme is determined based on the user's perspective
  • the projection is adaptively adjusted according to the user's perspective
  • the transmission scheme is optimized and adjusted in the direction of the user's attention to the point cloud object to improve the transmission quality.
  • FIG. 1 is a process diagram of processing point cloud fragments from different perspectives without modifying the point cloud projection scheme in the first embodiment
  • Figure 2-1 is a process diagram of processing point cloud fragments from the first perspective under the improved point cloud projection scheme in the first embodiment
  • Fig. 2-2 is a process diagram of processing point cloud fragments from a second perspective under the improved point cloud projection scheme in the first embodiment
  • Figure 2-3 is a process diagram of processing point cloud fragments from the third perspective under the improved point cloud projection scheme in the first embodiment
  • FIG. 3 is an example diagram of different aspects that need to be mainly observed under different user perspectives in the first embodiment
  • FIG. 4 is a schematic block diagram of the system of the sending end in the first embodiment
  • FIG. 5 is a three-dimensional schematic diagram of a three-dimensional edge box of a point cloud object in the second embodiment
  • FIG. 6 is a diagram of the projection adjustment process of the point cloud segment under the projection optimization adjustment scheme in the second embodiment
  • FIG. 7 is a corresponding table of the relationship between normal vectors and principal vectors of different point cloud segment types in the second embodiment
  • 8-1 is a schematic diagram of the relative positions of the three-dimensional edge box of the main segment and the three-dimensional edge box of the edge segment used for projection adjustment according to the main vector direction in the second embodiment;
  • Figure 8-2 is a schematic diagram of a two-dimensional projection obtained after the three-dimensional edge boxes of the main segment and the edge segment in Figure 11-1 are projected onto a two-dimensional plane;
  • FIG. 9 is a schematic flowchart of a point cloud projection transmission method at the sending end in the second embodiment.
  • FIG. 10 is a schematic flowchart of a point cloud projection receiving method at the receiving end in the second embodiment.
  • Fig. 11 is a schematic block diagram of a system at the sending end in the second embodiment.
  • the present invention provides a point cloud projection transmission method based on a user's perspective, which includes: preliminarily dividing point cloud objects on a standard vector to obtain a point cloud segment to be processed; and determining a projection transmission scheme based on the user's perspective, and the point cloud segment to be processed To process.
  • the present invention specifically describes different embodiments of the projection transmission scheme in the following.
  • a point cloud projection transmission method based on a user's perspective provided according to the present invention includes:
  • Point cloud segment determination step determine the corresponding point cloud segment according to the observation perspective, and obtain a series of point cloud segments classified into the corresponding perspective category;
  • Perspective transmission coding step According to the obtained series of point cloud segments that are divided into corresponding perspective categories, the perspective transmission scheme is adopted, and high-definition coding transmission is performed for the perspective that needs to be observed, and for perspectives that do not need to be observed Perform lower-definition coding transmission.
  • Point cloud segment determination step segmentation is performed according to the normal vector of the point cloud, and the distance between the normal vector of each point in the point cloud and the 6 standard normal vectors is calculated, and the standard normal vector with the closest distance is taken as the point category.
  • the 6 standard normal vectors adopt the existing technology, which are: (1, 0, 0), (-1, 0, 0), (0, 1, 0), (0, -1, 0), (0, 0, 1), and (0, 0, -1).
  • the extraction of the connected components includes: the point cloud fragments of the six major categories, according to the spatial distance between points and the adjacency relationship between points , Divide the point cloud fragments where the spatial distance between the points is greater than the preset distance range and the points are not adjacent to each other into smaller point cloud fragments, so that the point cloud fragments after the re-division are between the points
  • the spatial distance of is smaller than the preset distance range and the points are adjacent to each other;
  • the re-divided point cloud fragments still belong to 6 categories corresponding to the 6 standard normal vectors respectively, and then the obtained re-divided point cloud fragments are projected in different directions according to the normal vector categories;
  • Projection transmission schemes that perform sub-view transmission coding include any one or more of the following: a scheme that does not modify the point cloud projection method, a scheme that improves the point cloud projection method, and a standard normal vector scheme that is modified according to the user's perspective.
  • FIG. 1 is a process diagram of processing point cloud fragments from perspectives without modifying the point cloud projection solution in the first embodiment. It can be seen from Fig. 1 that the 3D point cloud object is divided according to 6 standard vectors to obtain point cloud segment 1 to point cloud segment t.
  • the point cloud fragments that need to be mainly observed and the point cloud fragments that do not need to be mainly observed are divided according to the observation angle of view.
  • the patch index list confirm which 2D segments are projected to be mainly observed, and which do not need to be mainly observed.
  • the selection of the standard vector projection plane as 0, 1, and 2 is the one that needs to be mainly observed.
  • the corresponding point cloud segments 1, 3, 6, 7, 9, etc. are divided into the point cloud segments that need to be mainly observed, and the rest are divided into the point cloud segments that do not need to be mainly observed Point cloud fragment.
  • Figure 3 is an example diagram of different faces that need to be mainly observed under different user perspectives in the first embodiment; for a point cloud object, among the six faces of its three-dimensional bounding box, Figure 3 shows one There are three situations in which the surface corresponding to the normal vector needs to be mainly observed, the surface corresponding to the two normal vectors needs to be mainly observed, and the surface corresponding to the three normal vectors needs to be mainly observed.
  • the point cloud projection transmission method based on the user’s perspective of the present invention includes any one of the following:
  • the 1, 2, and 3 improved point cloud projection schemes include:
  • Observation steps of main observation fragments According to the obtained series of point cloud fragments that are classified into corresponding perspective categories, according to the observation perspective, they are divided into the point cloud fragments that need to be mainly observed, and the point cloud fragments that do not need to be mainly observed, which is confirmed Which standard normal vector corresponds to the point cloud segment that needs to be mainly observed.
  • Figure 2-1 is a process diagram of processing point cloud fragments from the first perspective under the improved point cloud projection scheme in the first embodiment.
  • the first improved point cloud projection scheme is the same segment division process as in Figure 1.
  • Figure 2-1 of this embodiment after confirming which standard normal vector corresponds to the point cloud segment that needs to be mainly observed, The point cloud fragments that need to be mainly observed and the point cloud fragments that do not need to be mainly observed are projected onto different planes as needed. For example, the point cloud fragments that need to be mainly observed are projected into picture A, and they do not need to be projected.
  • the main observation point cloud fragment is projected into picture B, and the size of picture A and picture B are changed, and after the enlargement or reduction process is performed, in this embodiment, picture C is obtained after picture B is reduced, and picture A remains unchanged .
  • the unobserved picture C is merged into the observable picture A to obtain a projected picture, which is then compressed and encoded.
  • the present invention it is within the scope of the present invention to change the size of the picture, such as zooming in picture A while picture B remains the same or zoomed out, or picture A remains unchanged while reducing picture B, which is within the scope of the invention, making it unnecessary It is sufficient if there is a significant size difference between the observation picture and the picture that needs to be observed, and the combination scheme of size change and the magnification and reduction coefficients do not limit the scope of the invention.
  • the actual shape of the generated point cloud segment is based on the actual situation. In the figure, only the block diagram represents the point cloud segment for illustration.
  • Fig. 2-2 is a process diagram of processing point cloud fragments from a second perspective under the improved point cloud projection scheme in the first embodiment
  • the second improved point cloud projection scheme After confirming which standard normal vectors correspond to the point cloud fragments that need to be mainly observed, it is the same as the first improved point cloud projection scheme above, and the point cloud fragments are projected to On different planes, the point cloud fragments that need to be mainly observed are projected into picture A, and the point cloud fragments that do not need to be mainly observed are projected into picture B. In this case, no stitching is performed, but the pictures that need to be observed are based on requirements. A. There is no need to observe the pictures to be resized separately, for example, after the size is enlarged or reduced, the projected pictures after the resize are respectively subjected to subsequent compression coding.
  • Figure 2-3 is a process diagram of processing point cloud fragments from the third perspective under the improved point cloud projection scheme in the first embodiment
  • the third improved point cloud projection scheme is to project point cloud segment 1 to point cloud segment t on six different planes according to their standard normal vector types, corresponding to the six standard vectors, and get the first group of point clouds. Fragments to the sixth group of point cloud fragments. After confirming which standard normal vector corresponds to the point cloud fragments that need to be observed, the size of the planes corresponding to these point cloud fragments is changed according to requirements, as shown in Figure 2-3. According to the user’s perspective, select the point cloud segment on the projection surface of the standard vectors 0, 1, 2 as the main observation surface, that is, the point cloud segments of the 1-3 groups remain unchanged, and the point cloud segments of the 4-6 groups are reduced. , And then compress and code separately after such processing.
  • Observation angle normal vector calculation steps Obtain the direction of the user's observation angle, and calculate the normal vector of the observation angle. The specific process is as follows:
  • the rest of the standard normal vector calculation steps define the normal vector as one of the standard normal vectors, that is, the normal vector and one of the standard normal vectors are rotated and mapped, and then the other standard normal vectors are subjected to the same rotation mapping processing to make the overall Under the premise that the modification amount of is small, to obtain the other corresponding five sets of standard normal vectors, the calculation process is as follows:
  • FIG. 4 is a schematic block diagram of the system of the sending end in the first embodiment
  • the present invention provides a point cloud projection transmission system based on a user's perspective, which includes a segment generation module, a point cloud segment determination module, and a perspective transmission encoding module as a projection encoding processing module.
  • the fragment generation module is used to preliminarily divide the point cloud object on the standard vector to obtain the point cloud fragment to be processed.
  • Point cloud segment determination module Determine the corresponding point cloud segment according to the observation perspective, and obtain a series of point cloud segments that are classified into the corresponding perspective category;
  • the perspective transmission coding module is used to determine the projection transmission scheme according to the user perspective, and process the to-be-processed point cloud segment. According to the obtained series of point cloud fragments divided into corresponding viewing angle categories, the perspective transmission scheme is adopted to perform high-definition encoding transmission for the viewing angle that needs to be observed, and lower-definition viewing angle for the viewing angle that does not need to be observed. Encoding transmission.
  • the point cloud segment determination module :
  • the 6 standard normal vectors are:
  • connection component extraction refers to: taking 6 broad categories of point cloud fragments, according to the spatial distance between the points and the adjacency relationship between the points, the spatial distance between the points is greater than the preset distance range And the point cloud fragments that are not adjacent to each other are divided into smaller point cloud fragments, so that the spatial distance between the points of the re-divided point cloud fragment is less than the preset distance range and the points are adjacent to each other ;
  • the re-divided point cloud fragments still belong to 6 categories corresponding to the 6 standard normal vectors respectively, and then the obtained re-divided point cloud fragments are projected in different directions according to the normal vector categories;
  • the sub-view transmission coding module :
  • the perspective transmission solution includes any one or more of the following: a solution that does not modify the point cloud projection method, a solution that improves the point cloud projection method, and a standard normal vector solution that is modified according to the user's perspective.
  • a solution that does not modify the point cloud projection method a solution that improves the point cloud projection method
  • a standard normal vector solution that is modified according to the user's perspective.
  • the present invention provides a point cloud receiving and processing system based on the user's perspective, including:
  • the inverse processing module uses the codec parameters determined from the user's perspective to obtain a video sequence from the received bitstream, process the video sequence to obtain a point cloud segment, and decode and reconstruct each point cloud segment to obtain a point cloud object.
  • the point cloud projection transmission system based on the user's perspective and the point cloud receiving and processing system based on the user's perspective provided by the present invention can pass the point cloud projection transmission system method based on the user's perspective and the point cloud receiving processing method based on the user's perspective provided by the present invention The steps of the process are realized.
  • the purpose of the present invention is to provide a point cloud projection transmission scheme based on the user's perspective, which is modified and perfected on the basis of the existing typical point cloud compression scheme, and provides different degrees of observation at different viewing angles according to the needs of users.
  • the rendering effect is to provide a point cloud projection transmission scheme based on the user's perspective, which is modified and perfected on the basis of the existing typical point cloud compression scheme, and provides different degrees of observation at different viewing angles according to the needs of users.
  • the rendering effect is to provide a point cloud projection transmission scheme based on the user's perspective, which is modified and perfected on the basis of the existing typical point cloud compression scheme, and provides different degrees of observation at different viewing angles according to the needs of users. The rendering effect.
  • the present invention is implemented based on the current classic point cloud sequence coding transmission scheme.
  • the classic point cloud sequence coding method needs to be carried out sequentially. Segments are divided according to normal vectors, segment is subdivided according to segment connections, and projections in different directions are performed according to normal vectors. Fill the projection plane with blanks, and use a two-dimensional coding tool such as HEVC to compress and encode the video sequence obtained by the projection.
  • the data corresponding to the viewing angle that needs to be observed can be preferentially allocated to ensure that the data corresponding to the viewing angle that needs to be observed has sufficient bandwidth for transmission.
  • the data corresponding to the viewing angle that needs to be observed can be allocated according to a higher ratio, and the data corresponding to the viewing angle that does not need to be observed according to a lower ratio.
  • the first solution is based on the classic point cloud sequence coding, after obtaining the video sequence after projection, without changing the structure of the video sequence, it is achieved by modifying the coding parameters of the video coding tool. Specifically, because the point cloud sequence is divided into different fragments according to the normal vector, we can find the normal vector corresponding to the corresponding perspective according to the perspective, and further find the corresponding fragments. For these fragments, locate in the two-dimensional picture The corresponding projected segments can then be assigned different compression parameters to these segments in the video encoding tool. Different compression parameters can achieve different compression coding effects. For example, for the compression parameter quantization step sequence number, QP value for short, a low QP value can achieve higher compression quality, and a high QP value can achieve lower compression quality. .
  • the projection mode needs to be modified, so the structure of the video sequence will be modified accordingly, but the two-dimensional video sequence is still obtained and compressed using a two-dimensional coding tool.
  • a two-dimensional coding tool For example, after segment division and projection are performed on the point cloud, down-sampling processing can be performed on the two-dimensional segment obtained from the point cloud segment of the unwanted perspective. After such processing, we can obtain a video sequence with composite resolution, and then use a two-dimensional coding tool to compress with the same resolution and accuracy.
  • the standard normal vector can be modified according to the user's perspective.
  • the current classic compression algorithm of dynamic point cloud is to divide the divided point cloud fragments into categories according to the relationship between the six standard normal vectors, and then select one of them as the main projection direction and project it onto the plane. Based on this projection method, the standard normal vector can be modified according to the user's perspective, so that the user's perspective corresponds to the front of the projection.
  • the three schemes are not opposed to each other and can be used in combination.
  • the standard normal vector can be modified according to the user's perspective, and then the projection mode or coding parameters can be modified.
  • the present invention adopts three solutions.
  • the three schemes implement split-view coding in terms of different strategies, and they can all be combined with each other, or the three can be used in combination.
  • the corresponding point cloud segment needs to be determined according to the observation angle of view. It can be implemented relatively simply on the basis of the current point cloud sequence coding method. After the step of determining the point cloud segment corresponding to the perspective, a series of point cloud segments classified into the corresponding perspective category can be obtained. This is the premise of the three technical solutions in the point cloud sub-perspective transmission strategy proposed in this patent. And foundation.
  • the specific implementation algorithm is as follows.
  • the current classic coding method of point cloud sequence is to divide the segment according to the normal vector of the point cloud. By calculating the distance between the normal vector of each point in the point cloud and the 6 standard normal vectors:
  • connection component extraction processing that is, points that have been divided into six categories, according to the spatial distance and the adjacency relationship between points, are divided into smaller point cloud segments to prevent points from appearing in the same segment. The distance is too large and not adjacent.
  • a parameter index can be assigned to indicate which normal vector plane they belong to.
  • Table 2 is the patch index list in the first embodiment. ):
  • the method of confirming the corresponding normal vector from different perspectives can include the following:
  • the standard normal vector (0, 1, 0) corresponds to the perspective from the front.
  • the normal vector belongs to (0, 1). , 0) Those point cloud fragments of this category. If it is not observed from a standard forward perspective, there are more types of point cloud fragments that are mainly observed.
  • the two-dimensional pictures that need to be compressed are divided into blocks, that is, the two-dimensional segments are divided evenly, and then the QP offset values and other codes are set for these blocks respectively.
  • the parameters are encoded.
  • a parameter index can be assigned to each point cloud segment, and then a parameter list patch index list can be assigned to each block of the picture that needs to be compressed and encoded to record which point cloud segments this block is associated with.
  • these point cloud segments have been divided into point cloud segments that need to be primarily observed and point cloud segments that do not need to be primarily observed according to the observation angle of view. Therefore, according to the relationship between the point cloud segment and the two-dimensional segment, that is, the patch index list, it is possible to confirm which of the projected two-dimensional segments need to be mainly observed and which do not need to be mainly observed.
  • each segment is encoded and decoded.
  • the point cloud projection method can also be modified to achieve the coding effect of the point cloud by viewing angle and resolution.
  • the method for projecting each point cloud segment to a two-dimensional picture is as follows: Projection is performed according to the standard normal vector category of each point cloud segment, for example, the standard method of a point cloud segment The vector category is (1, 0, 0), then when projecting, the (y, z) coordinates in the three-dimensional coordinates (x, y, z) of this point cloud segment are used as plane coordinates, and the (x) coordinates are used Make the projection depth. As for where the projection position is on the picture, this is mainly based on the way to save the overall occupied space, starting from the upper left corner, filling into the two-dimensional picture in turn, so that the final total occupied plane size is the most economical.
  • the present invention proposes to optimize the way of projection of the point cloud from different perspectives. And based on this idea, the following three optimization ideas are proposed.
  • the three optimization ideas can be performed in parallel, that is, you can choose one:
  • the point cloud fragments are projected onto different planes as needed, processed, and then spliced onto a picture.
  • the point cloud fragments that need to be mainly observed are projected onto picture A according to the classic method, and the size is 1280 ⁇ 1280
  • the point cloud fragments that do not need to be mainly observed are projected onto picture B according to the classic method, and the size is 1280 ⁇ 1280 .
  • the picture B is reduced to obtain the picture C, and the size is 640 ⁇ 640. Since the projection according to the classic method ensures the minimum total occupied space, the projected position on the picture A is likely to be no more than 1/4, so the picture C can be pasted in the area of the picture A that is not occupied by the projection.
  • the picture occupies a size of 1280 ⁇ 1280 before the optimization, and the picture occupies a size of 1280 ⁇ 1280 after the optimization, but the plane space occupied by the point cloud fragments not mainly observed is smaller.
  • the coding technology can achieve the purpose of dividing the definition coding presentation according to the viewing angle requirements.
  • the point cloud segments are projected onto different planes as required, and these planes are processed according to requirements before compression and encoding.
  • point cloud fragments that need to be mainly observed are projected onto picture A according to the classic method
  • point cloud fragments that do not need to be mainly observed are projected onto picture B according to the classic method
  • picture B is reduced in size. After that, both the picture A and the reduced picture B are compressed and encoded for presentation.
  • the video sequence presented before optimization is a set of video sequences
  • the video sequence presented after optimization is two groups of video sequences.
  • the point cloud fragments that are not mainly observed are reduced due to the reduction process, which reduces the difficulty and cost of compression encoding, and adopts the same video
  • the coding technology can achieve the purpose of dividing the definition coding presentation according to the viewing angle requirements.
  • each point cloud segment is projected onto planes A, B, C, D, E, and F according to their standard normal vector categories, respectively, according to the classic method. Since it is possible to determine which standard normal vector point cloud fragments need to be mainly observed according to the user's perspective, it is possible to determine which of the six planes do not need to be mainly observed, and can be reduced in size. After performing this processing on each frame of the point cloud sequence, six groups of video sequences can be obtained, and the same video coding technology is used for coding processing.
  • the video sequence presented before optimization is a group of video sequences
  • the video sequence presented after optimization is multiple groups of video sequences.
  • the point cloud fragments that are not mainly observed are reduced due to the reduction process, which reduces the difficulty and cost of compression encoding.
  • Video coding technology can achieve the purpose of dividing definition coding and presentation according to viewing angle requirements.
  • the second method the method of modifying the point cloud projection method is realized by the following.
  • Solution 1 A solution in which segments corresponding to the user's perspective and segments corresponding to the non-user's perspective are spliced on a video sequence. It is necessary to record the index of all segments corresponding to the non-user perspective and the size reduction ratio at the sending end. This information will be encoded and transmitted as new additional information; at the receiving end, it needs to be based on the received index information and the reduction ratio. Restore these fragments to the original size, and then perform the subsequent 3D point cloud reconstruction steps.
  • Solution 2 A solution of projecting segments corresponding to the user's perspective and segments corresponding to the non-user's perspective to two video sequences respectively.
  • each video sequence needs to be restored according to the position of each video sequence in the code stream, and then these video sequences need to be restored to the original size according to the received reduction ratio, and then the subsequent steps Three-dimensional point cloud reconstruction steps, and need to stitch together the point clouds recovered from the six groups of video sequences.
  • the standard normal vectors for dividing point cloud segments are the following six normal vectors:
  • the main observing projection direction may be too much.
  • the main projected surface to be observed needs to include the front, side, and top of the observed object, that is, there are at least three sides.
  • the present invention proposes to adaptively adjust the direction of the standard normal vector according to the user's observation angle of view.
  • the direction of the user's observation angle can be obtained, and the normal vector of the observation angle can be calculated.
  • the normal vector of the observation angle can be calculated. For example, when observing a point cloud object, take the observed point cloud object as the origin of the relative coordinate system, provide the user with the relative position coordinates of the observed point cloud object, and calculate the position difference between the two to obtain the relative position.
  • the normal vector of the direction of the viewing angle which represents the direction of the user's viewing angle.
  • the normal vector is defined as one of the standard normal vectors, that is, the normal vector and one of the standard normal vectors are rotated and mapped, and then the other standard normal vectors are subjected to the same rotation mapping processing, so that the overall modification amount is small Under the premise, to obtain the other corresponding five sets of standard normal vectors.
  • the calculation formula i is as follows:
  • the normal vector has been normalized, that is, the sum of the squares of the three coordinates is 1.
  • the other five standard normal vectors (0, 1, 0), (-1, 0, 0), (0, -1 , 0), (0, 0, 1), (0, 0, -1) corresponding to the normal vector general formula of the rotation map can be expressed as:
  • the method to ensure that the overall amount of modification is small is to make the new six standard normal vectors, including the user’s perspective normal vector, correspond to the corresponding six standard methods after the user’s perspective normal vector corresponds to the standard normal vector and is rotated.
  • the sum of the dot products of the vectors is as large as possible.
  • the general formula of the normal vector of the user's viewing angle Set it to the standard normal vector (1, 0, 0) as a rotation mapping relationship
  • the other five standard normal vectors (0, 1, 0), (-1, 0, 0), (0, -1 , 0), (0, 0, 1), (0, 0, -1) corresponding to the normal vector general formula of the rotation map can be obtained by calculation:
  • segment projection and video coding can adopt the other two modification schemes mentioned in the present invention.
  • method three the method of modifying the standard normal vector.
  • the new standard normal vector needs to be recorded at the sending end, and this part of the information will be encoded and transmitted as additional additional information; at the receiving end, the inverse process of point cloud projection needs to be performed according to the new standard normal vector, so as to start from the two-dimensional video
  • the sequence reconstructs a three-dimensional point cloud sequence.
  • the transmission strategy of the present invention adaptively adjusted according to the user's perspective, there is no need to make major modifications to the current existing point cloud transmission strategy, and the user's attention to various aspects is also different, distinguishing between the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed.
  • the observed point cloud fragments can be better combined with the current high compression rate encoding algorithm, without losing the experience quality of the user's main perspective, and without affecting the compression rate and the objective quality of the point cloud, saving transmission consumption The resources to achieve the purpose of sub-view transmission.
  • the processing steps of the point cloud sequence encoding method include: dividing the point cloud object into multiple point cloud segments according to the normal vector, subdividing the segments according to the connection relationship of the point cloud segments, and dividing the subdivided point cloud
  • the segments are projected according to the normal vector, the projection plane is blank filled, and the video sequence obtained by the projection is compressed and encoded using a two-dimensional coding tool such as HEVC.
  • the present invention is implemented by improving on the existing point cloud sequence coding transmission scheme, and the user’s viewing angle is introduced in the process of compressing the three-dimensional point cloud into a two-dimensional image to optimize and adjust the point cloud projection processing process. Thereby improving compression performance.
  • the projection vector of the point cloud segment is selected by comprehensively considering the two factors of the user’s viewing angle and the normal vector of the segment.
  • This embodiment provides a point cloud projection processing method based on the user’s perspective, including: distinguishing points based on the user’s perspective The main segment and multiple edge segments of the cloud object; considering the spatial connection between the edge segment and the main segment, the projection vector of the edge segment is maintained or changed based on the user's perspective; subsequent point cloud processing is performed after projection.
  • FIG. 5 is a three-dimensional schematic diagram of a three-dimensional edge box of a point cloud object in the second embodiment
  • the point cloud object is projected by using a three-dimensional bounding box of the point cloud object to include the point cloud object.
  • a three-dimensional bounding box of a point cloud object is a rectangular cube set for a specific point cloud object, and the specific size and shape are not limited by this embodiment.
  • a vector pointing from the point cloud to the user's observation point is used to represent the user's view of the user's observation, which is called the view vector. Then calculate the distance between the viewing angle vector and the six standard normal vectors, and the standard normal vector with the closest distance to the viewing angle vector is called the principal vector under the viewing angle.
  • Table 3 is a patch index list of index parameter values and projection vector values in the second embodiment, and their parameter assignments and corresponding
  • the standard projection vector surface is the same as Table 2.
  • 0 is used to refer to the direction of the first standard normal vector, which is perpendicular to the (1, 0, 0) plane
  • 1 is used to refer to the direction of the sixth standard normal vector, which is perpendicular to the (-1, 0, 0) plane
  • use 2 to refer to the direction of the second standard normal vector, perpendicular to the (0, 1, 0) plane
  • use 3 to refer to the direction of the third standard normal vector, perpendicular to the (0, -1, 0) ) Plane
  • 4 to refer to the direction of the fourth standard normal vector, perpendicular to the (0, 0, 1) plane
  • 5 refer to the direction of the fifth standard normal vector, perpendicular to the (0, 0, -1) surface.
  • the main vector is the first standard normal vector (1, 0, 0).
  • the six standard normal vectors have a one-to-one correspondence with the six faces of the three-dimensional edge box, and the standard normal vectors are perpendicular to their corresponding faces.
  • Index parameter value Projection vector surface Normal 0 (1, 0, 0) 1 (-1, 0, 0) 2 (0, 1, 0) 3 (1, -1, 0) 4 (0, 0, 1) 5 (0, 0, -1)
  • the closest standard normal vector is the The standard normal vector corresponding to the point.
  • the points in all the point cloud objects are divided into 6 first-level point cloud segments.
  • the 6 first-level point cloud segments are subdivided, and the points that belong to the same first-level point cloud segment and are relatively close are divided into a second-level point cloud segment.
  • the method of calculating the distance between points can be selected according to the specific situation. Euclidean distance is usually used. According to this method, the 6 first-level point cloud segments are subdivided, and the normal vectors of the second-level point cloud segments obtained after subdivision all correspond to one of the above-mentioned 6 standard normal vectors.
  • the secondary point cloud segment is used as the cloud segment to be processed to continue processing, and the cloud segments to be processed on the six standard normal vector planes are further distinguished according to the direction of the view angle vector. Calculate the normal vector of the secondary point cloud segment. Then the main direction of the projection is determined according to the user's viewing angle. According to the relationship between the normal vector of the secondary point cloud segment and the main direction, the secondary point cloud segment is divided into the main segment, the edge segment, and the irrelevant segment.
  • FIG. 6 is a diagram of the projection adjustment process of the point cloud segment under the projection optimization adjustment scheme in the second embodiment
  • point cloud segment 1 to point cloud segment t are projected onto six different planes according to their standard normal vector types, corresponding to the six standard vectors, and the first 1 set of point cloud fragments to the 6th group of point cloud fragments, and then determine the main direction of projection according to the user’s viewing angle.
  • the secondary point cloud fragments are these 6 groups
  • Point cloud fragments are divided into main fragments, edge fragments, and irrelevant fragments.
  • the first group is divided into main segments
  • the second group is divided into irrelevant segments
  • the 3-6 groups are divided into edge segments.
  • projection is performed according to the optimized and adjusted projection direction.
  • the main segment and the irrelevant segment keep the projection direction unchanged, adjust the edge segment according to the large or small spatial correlation, and obtain the first group of point cloud segments to the sixth group of point clouds after the projection direction is optimized and adjusted. Fragments, it is obvious that the first group of point cloud fragments towards the user's perspective are more concentrated.
  • Fig. 7 is a corresponding table of the relationship between normal vectors and principal vectors of different point cloud segment types in the embodiment of the present invention; Fig. 7 shows the distinguishing criteria for different segment types.
  • the secondary point cloud fragments with the same normal vector and the main vector are defined as the main fragments. These main fragments are also called the first standard fragments. These main fragments form the main fragment set, the first standard fragment The set (ie, the main fragment set) is gathered on the main fragment projection gathering surface of the point cloud 3D edge box.
  • the secondary point cloud fragments whose normal vector is opposite to the main vector as irrelevant fragments. These irrelevant fragments are also called second standard fragments. These irrelevant fragments form a set of irrelevant fragments, and the second standard fragment set (ie, irrelevant point cloud fragments) Set) is gathered on the projection gathering surface of the irrelevant fragments of the 3D edge box of the point cloud.
  • edge fragments The secondary point cloud fragments whose normal vector is perpendicular to the main vector are defined as edge fragments. These edge fragments are distributed on the projection gathering surface of the four edge fragments perpendicular to the main vector.
  • Figure 5 shows only four edge fragments. One of the projection surfaces is projected, and the other three surfaces are not marked.
  • these secondary point cloud segments located on the projection gathering surface of the first edge segment that is, these edge segments form a set of the first edge segment, and the first edge segment is derived from the third standard segment.
  • the second-level point cloud segments located on the gathering surface of the other three edge segments are called the second edge segment set, the third edge segment set, and the fourth edge segment set. The above forms a collection, from the 4th to 6th standard fragments.
  • the projection vector of the main segment is projected with its standard normal vector, that is, the main vector.
  • the projection vector of the irrelevant segment is projected with its standard normal vector, that is, the opposite of the principal vector.
  • the size of the spatial connection is judged by sequentially calculating the intersection ratio (IOU) of each edge segment with each main segment. If there is an IOU greater than the threshold, it means that the edge point cloud segment has a greater spatial connection with the main point cloud segment, and the main vector is selected as the projection vector of the edge point cloud segment; if there is no IOU greater than the threshold, it means the edge point The cloud segment has a small spatial connection with the main point cloud segment, and the normal vector of the edge point cloud segment is selected as its projection vector.
  • IOU intersection ratio
  • Figure 8-1 is a schematic diagram of the relative positions of the three-dimensional edge box of the main segment and the three-dimensional edge box of the edge segment used for projection adjustment according to the main vector direction in the second embodiment
  • Figure 8-2 is the main segment and the three-dimensional edge box in Figure 8-1
  • a schematic diagram of the two-dimensional projection obtained after the three-dimensional edge box of the edge segment is projected onto the two-dimensional plane; the IOU judgment process of one of the edge segments and a certain main segment of each main segment is described in detail.
  • Fig. 8-1 is a schematic diagram of the relative positions of the three-dimensional edge box of the main segment and the three-dimensional edge box of the edge segment for projection adjustment according to the main vector direction in the embodiment of the present invention.
  • the three-dimensional edge box (called edge box 1) shown by the solid line corresponds to the main segment
  • the three-dimensional edge box (called edge box 2) shown by the dashed line corresponds to the edge segment.
  • Figure 8-1 and Figure 8-2 are only a brief enumeration of the three-dimensional edge boxes of a certain main segment and a certain edge segment, showing the projection orientation between the three-dimensional boxes and the overlap of the two-dimensional projection map.
  • Figure 8-1 and Figure 8-2 are only a brief enumeration of the three-dimensional edge boxes of a certain main segment and a certain edge segment, showing the projection orientation between the three-dimensional boxes and the overlap of the two-dimensional projection map.
  • one-by-one overlapping calculation spatial associations between a certain edge segment and other main segments, or between other edge segments and other main segments it will not be listed one by one, but only an example.
  • Fig. 8-2 is a rectangular two-dimensional projection diagram obtained after the three-dimensional edge boxes of the main segment and the edge segment are projected onto a two-dimensional plane.
  • the three-dimensional edge box of the main segment is projected along the main vector direction to obtain the main two-dimensional projection image
  • the three-dimensional edge box of the edge segment is projected along the main vector direction to obtain the edge two-dimensional projection image.
  • the pattern filling part is projected along the principal vector direction of the edge box 1 of the main segment (called rectangular area 1)
  • the dot filling part is projected along the principal vector direction of the edge box 2 of the edge segment (called rectangular).
  • Area 2), the grid point filling part is the common part where the rectangular area 1 and the rectangular area 2 overlap (referred to as the rectangular area 3).
  • S 3 corresponds to the area of the rectangular area 3 in Figure 4, that is, the overlapping common part of the main two-dimensional projection image and the edge two-dimensional projection image
  • S 1 corresponds to the main two-dimensional projection map in Figure 4 excluding the area of the common part
  • S 2 corresponds to the two-dimensional projection of the edge in Fig. 4 excluding the area of the common part.
  • a i and B i are the three-dimensional edge boxes of the main segment, that is, the vertices closest to the origin O on the edge frame 1; as shown in Figure 8-1
  • a j and B j are respectively the three-dimensional edge box of the edge segment, that is, the vertices of the edge box 2 closest to the origin O and the farthest.
  • a 'i, A' j, B 'i, B' j are A i, A j, B i , B j of the resultant vector along the main direction of the projection point.
  • Their coordinates in the XYZ coordinate system with O as the origin are: A i (x i0 ,y i0 ,z i0 ),B i (x i1 ,y i1 ,z i1 ),A j (x j0 ,y j0 ,z j0 ),B j (x j1 ,y j1 ,z j1 ),A′ i (y i0 ,z i0 ),B′ i (y i1 ,z i1 ),A′ j (y j0 ,z j0 ) ,B′ j (y j1 ,z j1 ).
  • For an edge segment compare its respective IOUs with an intersection ratio threshold. If there is an IOU greater than the intersection ratio threshold, it means that the edge segment has a greater spatial connection with the main segment, in order to optimize the point of the main vector direction Cloud visual quality, change the projection direction of the edge segment, and select the main vector as the projection vector of the edge segment; if there is no IOU greater than the intersection ratio threshold, it means that the edge segment has a small spatial connection with the main segment, then keep this The normal vector of the edge segment is used as its projection vector.
  • the final projection vector of the edge segment is determined by the normal vector of the edge segment and the spatial position relationship between the edge segment and the main segment.
  • the selection of the intersection ratio threshold depends on the surface characteristics of the point cloud object to be processed. For example, if the surface characteristics of the object point cloud object and the character point cloud object are different, then the intersection ratio threshold value selected when comparing It's different.
  • the threshold value is generally selected to be 0.2-0.3, and further optionally, the threshold value is determined mainly through adjustment and selection of experimental results.
  • the adjustment, optimization or maintenance of the projection vector is completed for the first edge segment, and the same processing of the above process is used to discriminate and adjust each edge segment one by one, until the second edge segment is set and the third edge segment is set. Set, all edge fragments in the set of 4th edge fragments have been processed.
  • the projection relationship identification field is used to identify: the projection vector of the point in the point cloud object after processing and adjustment, including the optimized and adjusted projection vector coordinate value and the corresponding index parameter value.
  • Table 4 is an example table of the corresponding relationship before and after adjustment and optimization of the projection vectors of multiple edge segments.
  • the main segment is: the index parameter value is 0, the standard normal vector coordinate value is (1, 0, 0) without optimization adjustment processing, and the main vector is maintained for projection.
  • Irrelevant edge fragments are: index parameter value 1, standard normal vector coordinate value (-1, 0, 0) without optimization and adjustment processing, keeping the original standard normal vector direction for projection.
  • 2nd edge segment 2 (0, 1, 0) Not big 2 (0, 1, 0) 3rd edge segment 4 (0, 0, 1) Big 0 (1, 0, 0) 4th edge segment 5 (0, 0, -1) Not big 5 (0, 0, -1) 5th edge segment 3 (0, -1, 0) Big 0 (1, 0, 0) K-th edge segment ... ... ... ... ... ...
  • the edge segment that has undergone projection processing, and the irrelevant segment project them according to the adjusted projection vector to generate a compressible two-dimensional image, and further process to obtain a two-dimensional video sequence, and use a video codec for the two-dimensional video sequence.
  • the video sequence is compressed.
  • FIG. 9 is a schematic flowchart of a point cloud projection transmission method based on a user's perspective in an embodiment of the present invention. the steps of a point cloud projection processing method based on a user's perspective include:
  • the point cloud object is divided to generate first-level point cloud fragments.
  • the first-level point cloud fragments are preliminarily divided according to the difference of the six standard vectors;
  • the process of generating a secondary point cloud segment from the primary point cloud segment can be completed by using the usual technical means in the existing point cloud projection step, and the secondary point cloud segment is used as a cloud segment to be processed and the following steps are performed.
  • obtaining cloud fragments to be processed is not limited to the above-mentioned steps such as the first-level point cloud fragment and the second-level point cloud fragment.
  • the preprocessing steps of any point cloud fragment in the prior art are all within the scope of the present invention. .
  • the cloud fragments to be processed are distinguished. According to the relationship between the normal vector of the secondary point cloud fragment and the main vector, the secondary point cloud fragment is divided into the main fragment, the edge fragment, and the irrelevant fragment. Among them, the normal vector and the main vector are the same. Fragment, normal vector and main vector opposite are irrelevant fragments, normal vector and main vector are perpendicular to edge fragments;
  • the projection direction of the secondary point cloud segment is calculated, and the projection direction of the edge segment in the secondary point cloud segment is optimized and adjusted.
  • the specific optimization and adjustment process is as described above;
  • FIG. 10 is a schematic flowchart of a point cloud receiving and processing method based on a user's perspective in an embodiment of the present invention.
  • the point cloud receiving and processing method based on the user's perspective in this embodiment is the inverse process of the above-mentioned point cloud projection processing method based on the user's perspective, and the decompression on the receiving end is the inverse process of compression on the transmitting end.
  • the first-level point cloud fragment is combined with auxiliary information to restore and generate a point cloud object, and the above-mentioned projection relationship identification field is used for back projection processing to restore the two-dimensional point cloud fragment to a three-dimensional point cloud object;
  • FIG. 11 is a schematic block diagram of the point cloud projection processing system based on the user's perspective in an embodiment of the present invention.
  • the point cloud projection processing system based on the user's perspective includes a point cloud input module, a segment generation module, a segment division module, a projection vector adjustment module, a projection module and a compression module, and a code stream output module.
  • Point cloud input module used to input point cloud objects
  • the fragment generation module is used to generate cloud fragments to be processed from the point cloud object according to predetermined rules, and perform subsequent projection optimization adjustments;
  • Fragment division module used to divide the cloud fragment to be processed into main fragments, edge fragments and irrelevant fragments
  • the projection vector adjustment module is used to take into account the spatial connection between the edge segment and the main segment, and to maintain the projection vector of the edge segment according to the above projection vector adjustment rule or change it based on the user's perspective;
  • Projection module used for projecting the main segment, edge segment and irrelevant segment after projection adjustment
  • the compression module and the code stream output module are used to compress the two-dimensional image after projection; and the subsequent code stream output.
  • the projection module uses the optimized and adjusted projection vector for projection, and is identified by the projection relationship identification field.
  • the projection relationship identification field contains the index parameter value of the optimized and adjusted projection vector and the corresponding projection vector value .
  • the point cloud receiving and processing system based on the user's perspective includes: a code stream processing module and a back projection module, a code stream processing module, used to receive the input of the code stream; a back projection module, used to back project two-dimensional point cloud fragments, according to The indication of the projection relationship identification field restores the two-dimensional point cloud segment to a point cloud object.
  • the projection relationship identification field contains the index parameter value of the optimized and adjusted projection vector and the corresponding projection vector value.
  • each module in the point cloud projection processing system based on the user's perspective and the point cloud receiving and processing system based on the user's perspective are the same as the above-mentioned point cloud projection processing method based on the user's perspective and the point cloud receiving based on the user's perspective.
  • the optimization and adjustment process in the processing method corresponds to the corresponding, which can be deduced by analogy. Then the structure and technical elements of the device can be formed by the corresponding conversion of the generation method and the receiving method. The repeated description is omitted here, and the details are not repeated here.
  • the point cloud segment corresponding to the same standard vector may have a large spatial distance between the points contained.
  • Such a simple rough division operation has nothing to do with the user's perspective, and the point cloud segment corresponding to the same standard vector is geometrically spatially related Small and fragmented.
  • the associated points in the point cloud object that are very close in space are segmented and projected into different standard vectors because of the difference in standard vectors.
  • the factor of the user's perspective is introduced to perform the multiple point cloud fragments.
  • the user has a high degree of attention to the front content of the point cloud object, then the main segment is determined based on the user's perspective.
  • the edge segments associated with the main segment such as the left and right sides and the top and bottom sides Point cloud segments are used as edge segments.
  • the projection direction of these edge segments further considers the distance relationship between these edge segments and the main segment.
  • the projection vector of the points in the edge segment is adjusted and optimized.
  • the projection vector of the point in the edge segment is changed based on the user's perspective, so that it is projected according to the main vector direction consistent with the main segment.
  • the spatial connection is small, keep the edge segment The projection vector of the point.
  • the above optimization and adjustment of the projection direction of the edge segment takes into account the user's perspective and the spatial correlation with the main vector, so that those points in the edge segment that are largely related to the main segment are changed and projected according to the main vector of the main segment, which pays more attention to the user.
  • the direction of higher degree of concentration is concentrated, and the adjusted and concentrated points are originally highly related and close in space with the main segment of higher attention. This avoids the scattered fragmentation and overcomes the problem of point cloud objects.
  • the associated points with very close spatial distance are divided and projected into different standard vectors because of the difference in standard vectors.
  • the present invention also provides a point cloud projection transmission method based on the user's perspective, a point cloud receiving processing method based on the user's perspective, and a point cloud projection transmission system based on the user's perspective, and a point cloud projection transmission system based on the user's perspective.
  • Point cloud receiving and processing system a point cloud projection transmission method based on the user's perspective, a point cloud receiving processing method based on the user's perspective, and a point cloud projection transmission system based on the user's perspective.
  • the above-mentioned first embodiment and the second embodiment are combined as the technical solution of the third embodiment.
  • There are three types in this embodiment and the specific content of the projection adjustment optimization and the multi-view coding transmission in the first and second embodiments are repeated, and the description is omitted here.
  • the third embodiment corresponds to the content in the first embodiment and the second embodiment, and can be obtained from the above-mentioned first embodiment and second embodiment respectively.
  • the point cloud projection transmission method based on the user's perspective includes the following steps:
  • edge segment Considering the spatial connection between the edge segment and the main segment, maintain the projection vector of the edge segment or change it based on the user's perspective;
  • the point cloud segments that need to be observed are allocated with priority resources for encoding transmission, and the point cloud segments with a perspective that do not need to be observed are allocated for encoding and transmission compared to the priority allocation of secondary resources.
  • the point cloud projection transmission system based on the user's perspective as the sending end, the point cloud receiving processing method based on the user's perspective as the receiving end, and the point cloud projection and transmission system based on the user's perspective as the receiving end can be determined by the point cloud projection transmission system based on the user's perspective.
  • the cloud projection transmission method can be deduced by analogy, and also includes the content of the combination of the above-mentioned first embodiment and the second embodiment respectively, and the overlapping content of the above-mentioned first embodiment and the second embodiment is omitted here.
  • the present invention corresponds to the above-mentioned first embodiment, and also provides a point cloud projection transmission method based on the user's perspective, which is characterized in that it includes: preliminarily dividing the point cloud object on the standard vector to obtain the point cloud segment to be processed; The point cloud fragments to be processed are further divided to obtain the point cloud fragments of the corresponding perspective category, including the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed; the point cloud fragments that need to be observed are assigned priority resources for encoding transmission, and The point cloud segment of the perspective that does not need to be observed is preferentially allocated to secondary resources for encoding transmission.
  • the point cloud projection transmission system based on the user's perspective as the sending end, the point cloud receiving processing method based on the user's perspective as the receiving end, and the point cloud projection and transmission system based on the user's perspective as the receiving end can be determined by the point cloud projection transmission system based on the user's perspective.
  • the cloud projection transmission method can be deduced by analogy, and it also includes the content of the above-mentioned first embodiment, and the overlapping content of the above-mentioned first embodiment is omitted here.
  • the present invention corresponds to the above-mentioned second embodiment, and also provides a point cloud projection transmission method based on a user's perspective, which is characterized in that it includes: a point cloud segment to be processed, distinguishing a main segment and a plurality of edge segments according to the user's perspective; Considering the spatial connection between the edge segment and the main segment, the projection vector of the edge segment is maintained or changed based on the user's perspective; subsequent point cloud processing is performed after projection.
  • the point cloud projection transmission system based on the user's perspective as the sending end, the point cloud receiving processing method based on the user's perspective as the receiving end, and the point cloud projection and transmission system based on the user's perspective as the receiving end can be determined by the point cloud projection transmission system based on the user's perspective.
  • the cloud projection transmission method can be deduced by analogy, and it also includes the content of the above-mentioned second embodiment, and the overlapping content of the above-mentioned second embodiment is omitted here.
  • the program can be stored in a computer readable storage medium, and the program can be stored in a computer readable storage medium. During execution, it may include the procedures of the above-mentioned method embodiments.
  • the storage medium may be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.
  • the embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored.
  • the program When the program is executed by a processor, it realizes the point cloud projection processing method based on the user's perspective as provided by the embodiment of the present invention.
  • Point cloud receiving and processing method When the program is executed by a processor, it realizes the point cloud projection processing method based on the user's perspective as provided by the embodiment of the present invention. Point cloud receiving and processing method.
  • the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), Erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.
  • the computer-readable signal medium may include a data signal propagated in baseband or as a part of a carrier wave, and computer-readable program code is carried therein. This propagated data signal can take many forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium.
  • the computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
  • the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
  • the computer program code used to perform the operations of the present invention can be written in one or more programming languages or a combination thereof.
  • the programming languages include object-oriented programming languages—such as Java, Smalltalk, C++, and also conventional Procedural programming language-such as "C" language or similar programming language.
  • the program code can be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to pass Internet connection).
  • LAN local area network
  • WAN wide area network

Abstract

Disclosed are a point cloud projection transmission method based on a user viewing angle, and a reception processing method and system, characterized in that the method comprises: distinguishing, on the basis of a user viewing angle, point cloud fragments of viewing angles needing to be observed and of viewing angles not needing to be observed, preferentially allocating resources to the point cloud fragments of the viewing angles needing to be observed, so as to encode and transmit same, and/or maintaining, on the basis of a spatial association space between an edge fragment and a main fragment, a projection vector of the edge fragment or changing same on the basis of the user viewing angle. In a projection transmission solution where adaptive adjustment is performed according to a user viewing angle, optimization adjustment is performed in an attention characteristic direction of a user when viewing a point cloud object, thereby improving the transmission quality.

Description

点云投影传输方法、接收处理方法及系统Point cloud projection transmission method, receiving processing method and system 技术领域Technical field
本发明属于点云领域，尤其涉及一种基于用户视角的点云投影传输方法、接收处理方法及系统。The invention belongs to the field of point clouds, and in particular relates to a point cloud projection transmission method, reception processing method and system based on a user's perspective.
背景技术Background technique
近些年来，媒体传输技术快速发展和完善，图像和视频信息的传输成本不断降低的同时精确度得到提高，这进一步促进了相关应用领域的成熟。三维点云数据是一种新型的图像媒体数据，可以为用户提供更加精确化，更加生动的图像信息，带来逼真度更高，交互性更强的通信体验。In recent years, media transmission technology has been rapidly developed and improved, and the transmission cost of image and video information has been continuously reduced while accuracy has been improved, which has further promoted the maturity of related application fields. Three-dimensional point cloud data is a new type of image media data, which can provide users with more accurate and vivid image information, and bring higher fidelity and more interactive communication experience.
点云数据是通过扫描得到的物体表面信息，包括三维坐标数据、深度信息、颜色信息等。3D扫描技术的成熟和系统架构的完善，提高了点云数据的精确度，同时也使得点云的数据量大大提升。其中针对静态、动态点云数据，及机器感知、人眼感知等不同类型的点云数据，有不同的点云数据压缩算法。例如，针对动态人眼感知点云数据，典型的点云压缩算法是将3D的点云数据转换为2D的图像数据，然后再进行数据处理，其中一种就是基于视频的点云压缩(Video-based Point Cloud Compression，VPCC)算法。这种压缩方法首先将3D点云投影到2D平面，得到占用图信息、几何信息、属性信息以及辅助信息，属性信息通常包括纹理信息和色彩信息，因此，压缩后的信息通常也分为四类数据进行传输。通过这种方法实现了对于点云序列的整体压缩传输。Point cloud data is the surface information of the object obtained by scanning, including three-dimensional coordinate data, depth information, color information, and so on. The maturity of 3D scanning technology and the perfection of the system architecture have improved the accuracy of point cloud data, while also greatly increasing the amount of point cloud data. Among them, there are different point cloud data compression algorithms for static and dynamic point cloud data, and different types of point cloud data such as machine perception and human eye perception. For example, for dynamic human eye perception point cloud data, a typical point cloud compression algorithm is to convert 3D point cloud data into 2D image data, and then perform data processing, one of which is video-based point cloud compression (Video- based Point Cloud Compression, VPCC) algorithm. This compression method first projects a 3D point cloud onto a 2D plane to obtain occupancy map information, geometric information, attribute information, and auxiliary information. The attribute information usually includes texture information and color information. Therefore, the compressed information is usually divided into four categories. Data is transferred. Through this method, the overall compressed transmission of the point cloud sequence is realized.
为了解决点云数据巨大的数据量带来的计算负担，对于点云的压缩编码算法已经有了较多的研究，也有一些点云压缩的标准化的讨论。目前针对应用于点云编码传输的典型压缩方案，是通过分片段投影将三维点云数据转换为二维图片进行处理。这种方案利用现有的视频编码工具，对于点云进行编码传输，其中需要依次进行对于点云的分法向量划分，细划分，片段投影，空白填充等步骤实现对于点云的分片段投影，之后可以借助HEVC等视频编 码技术对于投影获得的二维数据进行编码传输。通过这种方法实现了对于点云序列的整体压缩传输。In order to solve the computational burden caused by the huge amount of point cloud data, there have been more researches on the point cloud compression coding algorithm, and there are some discussions on the standardization of point cloud compression. At present, a typical compression scheme applied to point cloud coding and transmission is to convert three-dimensional point cloud data into two-dimensional images for processing through segmented projection. This solution uses the existing video coding tools to encode and transmit the point cloud, in which it is necessary to sequentially carry out the division of the point cloud, the subdivision, segment projection, blank filling and other steps to achieve the segment projection of the point cloud. Then, the two-dimensional data obtained by projection can be encoded and transmitted with the help of video encoding technologies such as HEVC. Through this method, the overall compressed transmission of the point cloud sequence is realized.
专利文献CN104778691B(申请号：201510160208.4)公开了一种三维点云数据的精简处理方法。本发明主要解决了现有对三维点云进行精简的方法存在的三维物体表面特征保留不好、重建的效果不好和计算量大的技术问题。本发明利用采集点云数据时Kinect相机同步得到的彩色图像中的信息对三维点云进行精简，该方法将彩色图与点云结合起来进行精简，能够避免采集及精简过程造成的物体表面一些曲率没有显著变化但却是特征信息的点云数据的丢失；可以根据物体本身的灰度信息设定相应的精简率进行精简；也可以根据主观因素保留自己认为比较重要的特征点，选择性的精简点云。Patent document CN104778691B (application number: 201510160208.4) discloses a simplified processing method for three-dimensional point cloud data. The invention mainly solves the technical problems of poor retention of three-dimensional object surface features, poor reconstruction effect, and large amount of calculation existing in the existing method for streamlining three-dimensional point clouds. The invention uses the information in the color image synchronized by the Kinect camera to streamline the three-dimensional point cloud when collecting the point cloud data. The method combines the color map and the point cloud for streamlining, which can avoid some curvature of the object surface caused by the collection and streamlining process There is no significant change but the point cloud data of the characteristic information is lost; you can set the corresponding reduction rate according to the gray information of the object itself for reduction; you can also reserve the characteristic points that you think are more important according to subjective factors, and selectively simplify Point cloud.
但是在实际的应用场景下，用户并不能观察到一个点云对象的全部内容，例如，当用户从正面观察点云对象的时候，用户无法看到点云背面的媒体内容，现有技术中点云处理的过程，并未对用户对于各个侧面的差异关注度予以区别。此外，现有点云压缩方案来实施点云传输，由于对于能被观察到的点云片段和不能被观察到的点云片段，进行了相同精确度的压缩，提供相同清晰度的呈现，这将会对于传输的带宽等提出不必要的需求。对于这种由于点云的遮挡带来的不必要的信息量，当前的对于点云序列的整体压缩传输方案并没有提出相应的解决方案。如何基于用户视角的差异关注度予以区别，以及在不影响观测质量的前提下，减小这种信息冗余，使之适应于实际用户应用场景，均是亟待解决的关键问题。However, in actual application scenarios, the user cannot observe the entire content of a point cloud object. For example, when the user observes the point cloud object from the front, the user cannot see the media content on the back of the point cloud. The process of cloud processing does not differentiate users' attention to differences in various aspects. In addition, the existing point cloud compression scheme implements point cloud transmission. Since the point cloud fragments that can be observed and the point cloud fragments that cannot be observed are compressed with the same accuracy and provide the same definition of presentation, this will Unnecessary demand for transmission bandwidth, etc. will be put forward. For this unnecessary amount of information caused by the occlusion of the point cloud, the current overall compression transmission scheme for the point cloud sequence does not propose a corresponding solution. How to make distinctions based on the degree of attention of the users' different perspectives, and to reduce this information redundancy without affecting the quality of observations, and adapt it to actual user application scenarios, are key issues that need to be resolved urgently.
发明内容Summary of the invention
本发明的目的是提供一种点云投影传输方法、接收处理方法及系统，引入用户视角来提升点云投影处理质量，可选地，对用户的观感影响较大的高关注度的侧面进行投影向量的修改完善，或结合优化点云的压缩算法来实现。The purpose of the present invention is to provide a point cloud projection transmission method, reception processing method and system, which introduces the user's perspective to improve the quality of point cloud projection processing, and optionally, projection is performed on the side with high attention that has a greater impact on the user's perception The vector is modified and perfected, or combined with optimized point cloud compression algorithm to achieve.
为了实现上述目的，本发明提供了一种基于用户视角的点云投影传输方法，具有这样的特征，包括：将点云对象在标准向量上初步划分，得到待处理点云片段；基于用户视角确定投影传输方案，对待处理点云片段进行处理。In order to achieve the above objective, the present invention provides a point cloud projection transmission method based on the user's perspective, which has such features, including: preliminary division of point cloud objects on a standard vector to obtain the point cloud segment to be processed; and determination based on the user's perspective The projection transmission scheme is to process the point cloud segment to be processed.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，取距离最近的标准法向量为点云对象中点的划分类别；基于六个标准法向量将点云对象中点分别划分为多个一级点云片段；将同一一级点云片段中，同一法向量且距离小于预定值的点再划分为二级点云片段作为待处理云片段。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature that the closest standard normal vector is taken as the classification category of the midpoint of the point cloud object; based on six standards The normal vector divides the midpoint of the point cloud object into multiple first-level point cloud fragments; in the same first-level point cloud fragment, the points with the same normal vector and the distance less than the predetermined value are then divided into second-level point cloud fragments as to be processed Cloud fragment.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，其中，投影传输方案包括：将待处理点云片段进一步划分得到对应视角类别的点云片段，包含需要观察到的点云片段和不需要观察到的点云片段；对需要观察到的点云片段分配优先资源进行编码传输，对不需要被观察得到的视角的点云片段相较于优先分配次级资源进行编码传输。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have such a feature, wherein the projection transmission scheme includes: further dividing the to-be-processed point cloud segment to obtain points corresponding to the perspective category Cloud fragments, including the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed; the point cloud fragments that need to be observed are assigned priority resources for encoding transmission, and the point cloud fragments that do not need to be observed are compared Prioritize the allocation of secondary resources for coding transmission.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，其中，所述投影传输方案包括以下任一种或任多种：不修改点云投影方案、改进点云投影方案、根据用户视角来修改标准法向量方案。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have such a feature, wherein the projection transmission scheme includes any one or more of the following: no point cloud modification Projection plan, improve the point cloud projection plan, modify the standard normal vector plan according to the user's perspective.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，所述不修改点云投影方法的方案中，包括：将点云对象在标准向量上初步划分，通过参数列表与作为投影向量的标准向量相关联，得到待处理点云片段，考虑用户视角和投影向量的方向，将待处理点云片段分为需要观察到的点云片段和不需要观察到的点云片段。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature that the solution of not modifying the point cloud projection method includes: placing the point cloud object in the standard vector The above preliminary division is to associate the parameter list with the standard vector as the projection vector to obtain the point cloud segment to be processed. Considering the user's perspective and the direction of the projection vector, the point cloud segment to be processed is divided into the point cloud segment that needs to be observed and the point cloud segment that needs to be observed. The point cloud fragment that needs to be observed.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，对需要观察到的点云片段和不需要观察到的点云片段，设置相对应的编解码参数，记录每个点云片段的编解码参数，作为新增附加信息用于编码传输。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature of setting corresponding points for the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed. The encoding and decoding parameters of each point cloud segment are recorded as additional information for encoding and transmission.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，所述改进点云投影方案中，包括：将待处理点云片段进一步划分得到对应视角类别的点云片段，包含需要观察到的点云片段和不需要观察到的点云片段；将需要观察到的点云片段和不需要观察到的点云片段，投影到投影平面上，根据需要更改投影平面的尺寸，得到需要观察图片 和不需要观察图片。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature that the improved point cloud projection solution includes: further dividing the to-be-processed point cloud fragments to obtain corresponding The point cloud fragments of the perspective category include the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed; the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed are projected onto the projection plane, according to The size of the projection plane needs to be changed to obtain the need to observe pictures and the need not to observe pictures.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，进一步包括：作为第一改进点云投影方案，在需要观察到的点云片段和不需要观察到的点云片段之间错位前提下将不需要观察图片拼合入需要观察图片中，得到一张投影图片，进行压缩编码。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have such characteristics, and further include: as the first improved point cloud projection solution, the point cloud segment and the Under the premise that the point cloud fragments that do not need to be observed are misaligned, the pictures that do not need to be observed are merged into the pictures that need to be observed, and a projected picture is obtained, which is compressed and encoded.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，所得到的一张投影图片形成一组视频序列，记录不需要观察到的点云片段对应的索引参数index以及尺寸更改的比例，作为新增附加信息用于编码传输。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature that a projected picture obtained forms a set of video sequences, and the point cloud that does not need to be observed is recorded The index parameter index corresponding to the segment and the scale of the size change are used as newly added additional information for encoding transmission.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，进一步包括：作为第二改进点云投影方案，将需要观察图片和不需要观察图片分别进行压缩编码。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have such characteristics, and further include: as a second improved point cloud projection solution, there will be a need to observe the picture and the need not to observe the picture. Compress and encode separately.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，投影到的需要观察图片和不需要观察图片分别形成两组视频序列，编码两组视频序列，记录不需要观察到的点云片段对应的视频序列的更改比例，作为新增附加信息进行编码传输，进一步记录各组视频序列在编码后的码流中位置。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature that the projected images that need to be observed and the images that do not need to be observed respectively form two sets of video sequences, and the two sets are encoded. Video sequence, record the change ratio of the video sequence corresponding to the point cloud segment that does not need to be observed, encode and transmit as new additional information, and further record the position of each group of video sequences in the coded stream.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，第三改进点云投影方案，包括：将待处理点云片段依照六个标准法向量分组，分别投影到六个投影平面上，对六个投影平面基于用户视角分别根据需求更改尺寸，分别进行压缩编码。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have such a feature. The third improved point cloud projection solution includes: the point cloud segment to be processed according to six standard methods The vectors are grouped and projected onto six projection planes, and the size of the six projection planes is changed according to the user's perspective based on the user's perspective, and compression coding is performed respectively.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，将各个标准法向量对应的点云片段分别投影到各组视频序列，编码六组视频序列，记录不需要观察到的点云片段对应的视频序列的更改比例，作为新的附加信息进行编码传输，进一步记录各组视频序列在编码后的码流中的位置。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, the point cloud segment corresponding to each standard normal vector can be respectively projected to each group of video sequences, and six groups are encoded. Video sequence, record the change ratio of the video sequence corresponding to the point cloud segment that does not need to be observed, encode and transmit as new additional information, and further record the position of each group of video sequences in the coded stream.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，所述根据用户视角来修改标准法向量方案中，包括：根据用户观测视角确定视角向量；将该视角向量与其中一个标准法向量进行旋转映射，然后将其他的标准法向量进行同样的旋转映射处理，在总体修改 量小于预定值的前提下，确定旋转映射后的投影方向。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature that the modification of the standard normal vector scheme according to the user's perspective includes: determining the perspective according to the user's observation perspective Vector; rotate the view vector with one of the standard normal vectors, and then perform the same rotation mapping processing on the other standard normal vectors, and determine the projection direction after the rotation mapping on the premise that the overall modification amount is less than a predetermined value.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，其中，将总体修改量小于预定值的步骤包括：在视角向量与标准法向量对应，并进行旋转之后，使包括视角向量在内的旋转映射后的六个标准法向量与对应原有的六个标准法向量的点乘积的和在预定范围之内取最大值或较大值。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature, wherein the step of reducing the overall modification amount to be less than a predetermined value includes: the perspective vector corresponds to the standard normal vector , And after the rotation, the sum of the dot products of the six standard normal vectors after the rotation mapping including the viewing angle vector and the corresponding original six standard normal vectors takes the maximum value or a larger value within a predetermined range.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，其中，将旋转映射调整后的标准法向量，将作为新增的附加信息进行编码传输。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature that the standard normal vector after the rotation mapping is adjusted will be encoded as the newly added additional information transmission.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，其中，投影传输方案包括：对待处理点云片段，依据用户视角区分出主片段和多个边缘片段；考虑到边缘片段和主片段之间的空间联系，对边缘片段的投影向量进行保持或基于用户视角进行变更；投影后进行后续点云处理。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have such a feature, wherein the projection transmission scheme includes: the point cloud segment to be processed is distinguished from the main segment and the point cloud segment according to the user's perspective. Multiple edge segments; considering the spatial connection between the edge segment and the main segment, the projection vector of the edge segment is maintained or changed based on the user's perspective; subsequent point cloud processing is performed after projection.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，其中，考虑到边缘片段和主片段之间的空间联系的判断依据，包括：依据用户视角在标准法向量中确定主向量；针对主片段和边缘片段分别沿着主向量方向投影得到主二维投影图、以及边缘二维投影图；基于每个边缘二维投影图，分别与各个主二维投影图计算得到交并比；多个交并比与预设门限值进行比较作为空间联系的判断依据。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature, in which the judgment basis considering the spatial connection between the edge segment and the main segment includes: The user’s perspective determines the principal vector in the standard normal vector; the principal segment and the edge segment are respectively projected along the principal vector direction to obtain the principal two-dimensional projection map and the edge two-dimensional projection map; based on each edge two-dimensional projection map, and each The main two-dimensional projection map is calculated to obtain the intersection ratio; multiple intersection ratios are compared with the preset threshold value as the basis for judging the spatial connection.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，其中，多个交并比中存在大于预设门限值的情况下，将边缘片段的投影向量变更为主向量；多个交并比中均不存在大于预设门限值的情况下，保持该边缘片段的投影向量。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature that when multiple intersection ratios are greater than a preset threshold, the edge The projection vector of the segment is changed to the main vector; if none of the multiple intersection ratios is greater than the preset threshold, the projection vector of the edge segment is maintained.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，其中，预设门限值取决于点云对象的表面特性，区分物体点云对象的表面特性和人物点云对象的表面特性，门限值的确定可进一步依据实验结果进行调整确定。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature that the preset threshold value depends on the surface characteristics of the point cloud object to distinguish the object point cloud object The determination of the threshold value of the surface characteristics of the character point cloud object and the surface characteristics of the character point cloud object can be further adjusted and determined according to the experimental results.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，其中，根据用户观看点云对象时所处的位置，用一个 从点云指向用户观察点的向量表示用户的观察视角，称为视角向量，计算视角向量与六个标准法向量的距离，与视角向量距离最近的标准法向量称为该观察视角下的主向量。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature, wherein, according to the position where the user views the point cloud object, a point cloud is used to point the user to the point cloud. The vector of the observation point represents the viewing angle of the user, which is called the viewing angle vector. The distance between the viewing angle vector and the six standard normal vectors is calculated, and the standard normal vector with the closest distance to the viewing angle vector is called the principal vector under the viewing angle.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，其中，除了依据用户视角区分出点云对象的主片段和多个边缘片段，还包括区分出无关片段，其中，将与用户视角距离最接近的标准法向量确定为主向量；将多个标准法向量中与该主向量相同的待处理点云片段确定为主片段；将多个标准法向量中与该主向量相反的待处理点云片段确定为无关片段；将多个标准法向量中与该主向量相关联的待处理点云片段确定为边缘片段。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature, wherein, in addition to distinguishing the main segment and multiple edge segments of the point cloud object according to the user's perspective, Including distinguishing irrelevant fragments, among which, the standard normal vector closest to the user's perspective distance is determined as the main vector; among multiple standard normal vectors, the point cloud fragments to be processed that are the same as the main vector are determined as the main fragment; The to-be-processed point cloud segment opposite to the main vector in the standard normal vector is determined as an irrelevant segment; the to-be-processed point cloud segment associated with the main vector in the multiple standard normal vectors is determined as an edge segment.
进一步可选地，在本发明所提供的基于用户视角的点云投影传输方法中，还可以具有这样的特征，其中，投影后进行后续点云处理中，利用投影关系标识字段标识出，点云对象中的点经过处理调整后的投影向量，包含点云片段索引值和相对应的投影向量值。Further optionally, in the point cloud projection transmission method based on the user's perspective provided by the present invention, it may also have the feature that, in the subsequent point cloud processing after projection, the projection relationship identification field is used to identify that the point cloud The projection vector of the point in the object after processing and adjustment, including the index value of the point cloud segment and the corresponding projection vector value.
另外，本发明还提供了一种基于用户视角的点云接收处理方法，其特征在于：利用用户视角而确定的编解码参数对接收到的码流得到视频序列，和对视频序列处理得到点云片段，以及对每个点云片段进行解码重建得到点云对象。In addition, the present invention also provides a point cloud receiving and processing method based on the user's perspective, which is characterized in that the codec parameters determined by the user's perspective are used to obtain a video sequence from the received code stream, and the video sequence is processed to obtain a point cloud. Segment, and decode and reconstruct each point cloud segment to obtain a point cloud object.
进一步可选地，在本发明所提供的基于用户视角的点云接收处理方法中，还可以具有这样的特征，其中，根据编解码参数对每个点云片段进行编解码和后续重建的过程，包含以下任意一种或者几种：所述编解码参数在发送端针对每个点云片段，区分需要观察到的点云片段和不需要观察被设置，进行后续的点云对象的重建；或者需要根据所接收到的参数信息以及尺寸更改信息来对点云片段恢复为原始尺寸，再进行后续的点云对象的重建；或者根据各视频序列在码流中的位置恢复出各视频序列，根据所接收到的更改尺寸来将这些视频序列恢复为原始尺寸，并需要将多组视频序列所恢复出来的点云进行拼接，进行点云对象的重建；或者需要根据发送端旋转映射调整后的标准法向量，进行点云投影的逆过程，从而从二维视频序列重建出三维点云序列，进行点云对象的重建。Further optionally, in the point cloud receiving and processing method based on the user's perspective provided by the present invention, it may also have the feature, in which the process of encoding, decoding and subsequent reconstruction of each point cloud segment according to the encoding and decoding parameters, Contains any one or more of the following: the encoding and decoding parameters are set at the sending end for each point cloud segment, distinguishing the point cloud segment that needs to be observed and the point cloud segment that does not need to be observed, and performing subsequent point cloud object reconstruction; or According to the received parameter information and size change information, the point cloud segment is restored to its original size, and then the subsequent point cloud object reconstruction is performed; or each video sequence is restored according to the position of each video sequence in the code stream. The received size changes to restore these video sequences to the original size, and the point clouds recovered from multiple sets of video sequences need to be spliced to reconstruct the point cloud objects; or the standard method needs to be adjusted according to the rotation mapping of the sender Vector, the inverse process of point cloud projection is performed, so as to reconstruct a three-dimensional point cloud sequence from a two-dimensional video sequence and perform point cloud object reconstruction.
另外，本发明还提供了一种基于用户视角的点云投影传输方法，其特征在于：片段生成模块，用于将点云对象在标准向量上初步划分，得到待处理点云 片段；投影编码处理模块，基于用户视角确定投影传输方案，对待处理点云片段进行处理。In addition, the present invention also provides a point cloud projection transmission method based on the user's perspective, which is characterized in that: a segment generation module is used to preliminarily divide the point cloud object on the standard vector to obtain the point cloud segment to be processed; projection coding processing The module determines the projection transmission scheme based on the user's perspective, and processes the point cloud segment to be processed.
另外，本发明还提供了一种基于用户视角的点云接收处理系统，其特征在于：逆处理模块，利用用户视角而确定的编解码参数对接收到的码流得到视频序列，和对视频序列处理得到点云片段，以及对每个点云片段进行解码重建得到点云对象。In addition, the present invention also provides a point cloud receiving processing system based on the user's perspective, which is characterized in that: the inverse processing module uses the codec parameters determined from the user's perspective to obtain a video sequence from the received code stream, and to obtain a video sequence for the video sequence. The point cloud segment is obtained by processing, and each point cloud segment is decoded and reconstructed to obtain a point cloud object.
另外，本发明还提供了一种基于用户视角的点云投影传输方法，其特征在于：将点云对象在标准向量上初步划分，得到待处理点云片段；将待处理点云片段进一步划分得到对应视角类别的点云片段，包含需要观察到的点云片段和不需要观察到的点云片段；对需要观察到的点云片段分配优先资源进行编码传输，对不需要被观察得到的视角的点云片段相较于优先分配次级资源进行编码传输In addition, the present invention also provides a point cloud projection transmission method based on the user's perspective, which is characterized in that: the point cloud object is preliminarily divided on the standard vector to obtain the point cloud segment to be processed; the point cloud segment to be processed is further divided to obtain The point cloud segment corresponding to the perspective category includes the point cloud segment that needs to be observed and the point cloud segment that does not need to be observed; priority resources are assigned to the point cloud segment that needs to be observed for encoding and transmission, and for the perspective that does not need to be observed Point cloud fragments are compared with priority allocation of secondary resources for encoding transmission
另外，本发明还提供了一种基于用户视角的点云投影传输方法，还可以具有这样的特征：包括：对待处理点云片段，依据用户视角区分出主片段和多个边缘片段；考虑到边缘片段和主片段之间的空间联系，对边缘片段的投影向量进行保持或基于用户视角进行变更；投影后进行后续点云处理。In addition, the present invention also provides a point cloud projection transmission method based on the user's perspective, which may also have the following characteristics: including: the point cloud segment to be processed is distinguished from the main segment and multiple edge segments according to the user's perspective; The spatial connection between the fragment and the main fragment is to maintain the projection vector of the edge fragment or change it based on the user's perspective; after the projection, the subsequent point cloud processing is performed.
另外，本发明还提供了一种基于用户视角的点云投影传输方法，还可以具有这样的特征：包括：将点云对象在标准向量上初步划分，得到待处理点云片段，依据用户视角区分出主片段和多个边缘片段；考虑到边缘片段和主片段之间的空间联系，对边缘片段的投影向量进行保持或基于用户视角进行变更；将调整变更后的主片段作为需要观察到的点云片段，调整变更后的边缘片段和无关片段作为不需要观察到的点云片段；以及对需要观察到的点云片段分配优先资源进行编码传输，对不需要被观察得到的视角的点云片段相较于优先分配次级资源进行编码传输。In addition, the present invention also provides a point cloud projection transmission method based on the user's perspective, which may also have such features: including: preliminary division of the point cloud objects on the standard vector to obtain the to-be-processed point cloud segment, which is distinguished according to the user's perspective Generate the main segment and multiple edge segments; consider the spatial connection between the edge segment and the main segment, maintain the projection vector of the edge segment or change it based on the user's perspective; use the adjusted and changed main segment as the point to be observed Cloud fragments, adjusted and changed edge fragments and irrelevant fragments as point cloud fragments that do not need to be observed; and assign priority resources to the point cloud fragments that need to be observed for encoding transmission, and point cloud fragments that do not need to be observed. Compared with priority allocation of secondary resources for coding transmission.
本发明的作用和效果Function and effect of the present invention
根据本发明的点云投影传输方法、接收处理方法及系统，将点云对象在标准向量上初步划分，得到待处理点云片段，基于用户视角确定投影传输方案，根据用户视角自适应调整的投影传输方案，往用户观看点云对象时的关注度特点方向进行优化调整，提升传输质量。According to the point cloud projection transmission method, the receiving processing method and the system of the present invention, the point cloud object is preliminarily divided on the standard vector to obtain the point cloud segment to be processed, the projection transmission scheme is determined based on the user's perspective, and the projection is adaptively adjusted according to the user's perspective The transmission scheme is optimized and adjusted in the direction of the user's attention to the point cloud object to improve the transmission quality.
附图说明Description of the drawings
通过阅读参照以下附图对非限制性实施例所作的详细描述，本发明的其它特征、目的和优点将会变得更明显，通过阅读参照以下附图对非限制性实施例所作的详细描述：By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more apparent, by reading the detailed description of the non-limiting embodiments with reference to the following drawings:
图1为第一实施例中在不修改点云投影方案下分视角对点云片段进行处理的过程图；FIG. 1 is a process diagram of processing point cloud fragments from different perspectives without modifying the point cloud projection scheme in the first embodiment;
图2-1为第一实施例中在改进点云投影方案下第1种分视角对点云片段进行处理的过程图；Figure 2-1 is a process diagram of processing point cloud fragments from the first perspective under the improved point cloud projection scheme in the first embodiment;
图2-2为第一实施例中在改进点云投影方案下第2种分视角对点云片段进行处理的过程图；Fig. 2-2 is a process diagram of processing point cloud fragments from a second perspective under the improved point cloud projection scheme in the first embodiment;
图2-3为第一实施例中在改进点云投影方案下第3种分视角对点云片段进行处理的过程图；Figure 2-3 is a process diagram of processing point cloud fragments from the third perspective under the improved point cloud projection scheme in the first embodiment;
图3为第一实施例中不同用户视角下需要被主要观察涉及不同面的示例图；FIG. 3 is an example diagram of different aspects that need to be mainly observed under different user perspectives in the first embodiment; FIG.
图4为第一实施例中发送端的系统示意框图；4 is a schematic block diagram of the system of the sending end in the first embodiment;
图5为第二实施例中点云对象的三维边缘盒的立体示意图；5 is a three-dimensional schematic diagram of a three-dimensional edge box of a point cloud object in the second embodiment;
图6为第二实施例中在投影优化调整方案下点云片段的投影调整过程图；6 is a diagram of the projection adjustment process of the point cloud segment under the projection optimization adjustment scheme in the second embodiment;
图7为第二实施例中不同点云片段的类型的法向量与主向量的关系对应表；FIG. 7 is a corresponding table of the relationship between normal vectors and principal vectors of different point cloud segment types in the second embodiment;
图8-1为第二实施例中主片段的三维边缘盒和边缘片段的三维边缘盒用于根据主向量方向投影调整的相对位置示意图；8-1 is a schematic diagram of the relative positions of the three-dimensional edge box of the main segment and the three-dimensional edge box of the edge segment used for projection adjustment according to the main vector direction in the second embodiment;
图8-2为附图11-1中主片段和边缘片段的三维边缘盒投影到二维平面后得到的二维投影示意图；Figure 8-2 is a schematic diagram of a two-dimensional projection obtained after the three-dimensional edge boxes of the main segment and the edge segment in Figure 11-1 are projected onto a two-dimensional plane;
图9为第二实施例中发送端的点云投影传输方法的流程示意图；FIG. 9 is a schematic flowchart of a point cloud projection transmission method at the sending end in the second embodiment;
图10为第二实施例中接收端的点云投影接收方法的流程示意图；以及10 is a schematic flowchart of a point cloud projection receiving method at the receiving end in the second embodiment; and
图11为第二实施例中发送端的系统示意框图。Fig. 11 is a schematic block diagram of a system at the sending end in the second embodiment.
具体实施方式detailed description
下面结合具体实施例对本发明进行详细说明。以下实施例将有助于本领域的技术人员进一步理解本发明，但不以任何形式限制本发明。应当指出的是，对本领域的普通技术人员来说，在不脱离本发明构思的前提下，还可以做出若干变形和改进。这些都属于本发明的保护范围。The present invention will be described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be pointed out that for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
本发明提供了一种基于用户视角的点云投影传输方法，包括：将点云对象在标准向量上初步划分，得到待处理点云片段；以及基于用户视角确定投影传输方案，对待处理点云片段进行处理。The present invention provides a point cloud projection transmission method based on a user's perspective, which includes: preliminarily dividing point cloud objects on a standard vector to obtain a point cloud segment to be processed; and determining a projection transmission scheme based on the user's perspective, and the point cloud segment to be processed To process.
本发明通过以下对投影传输方案的不同实施例进行具体说明。The present invention specifically describes different embodiments of the projection transmission scheme in the following.
{第一实施例}{First Embodiment}
根据本发明提供的一种基于用户视角的点云投影传输方法，包括：A point cloud projection transmission method based on a user's perspective provided according to the present invention includes:
点云片段确定步骤：根据观测视角来确定对应的点云片段，获得一系列被划分到对应视角类别的点云片段；以及Point cloud segment determination step: determine the corresponding point cloud segment according to the observation perspective, and obtain a series of point cloud segments classified into the corresponding perspective category; and
分视角传输编码步骤：根据获得的一系列被划分到对应视角类别的点云片段，采用分视角传输方案，对于需要观测到的视角进行高清晰度的编码传输，对于不需要被观察得到的视角进行较低清晰度的编码传输。Perspective transmission coding step: According to the obtained series of point cloud segments that are divided into corresponding perspective categories, the perspective transmission scheme is adopted, and high-definition coding transmission is performed for the perspective that needs to be observed, and for perspectives that do not need to be observed Perform lower-definition coding transmission.
点云片段确定步骤：根据点云的法向量来进行片段划分，计算点云中每一个点的法向量与6个标准法向量之间的距离，取距离最近的标准法向量为点的类别，来将所有的点划分为6个大类，即6个大类的点云片段，之后再将这6个大类的点云片段进行连接组件提取，将6个大类的点云片段重新划分，获得重新划分后的点云片段。Point cloud segment determination step: segmentation is performed according to the normal vector of the point cloud, and the distance between the normal vector of each point in the point cloud and the 6 standard normal vectors is calculated, and the standard normal vector with the closest distance is taken as the point category. To divide all points into 6 major categories, that is, 6 major categories of point cloud fragments, and then extract the connection components of these 6 major categories of point cloud fragments, and re-divide the 6 major categories of point cloud fragments , To obtain the repartitioned point cloud segment.
6个标准法向量采用现有技术，分别为：(1，0，0)，(-1，0，0)，(0，1，0)，(0，-1，0)，(0，0，1)，以及(0，0，-1)。The 6 standard normal vectors adopt the existing technology, which are: (1, 0, 0), (-1, 0, 0), (0, 1, 0), (0, -1, 0), (0, 0, 1), and (0, 0, -1).
对初步划分的6个大类的点云片段进行连接组件提取，连接组件提取包含：将6个大类的点云片段，根据点与点之间的空间距离以及点与点之间的邻接关系，将点与点之间的空间距离大于预设距离范围且点与点之间不邻接的点云片段划分为更小的点云片段，使重新划分后的点云片段的点与点之间的空间距离小于预设距离范围且点与点之间邻接；Extract the connected components of the six preliminarily divided point cloud fragments. The extraction of the connected components includes: the point cloud fragments of the six major categories, according to the spatial distance between points and the adjacency relationship between points , Divide the point cloud fragments where the spatial distance between the points is greater than the preset distance range and the points are not adjacent to each other into smaller point cloud fragments, so that the point cloud fragments after the re-division are between the points The spatial distance of is smaller than the preset distance range and the points are adjacent to each other;
所述重新划分后的点云片段，仍分别属于对应于6个标准法向量的6个 类别，然后根据法向量类别对获得的重新划分后的点云片段进行分方向的投影；The re-divided point cloud fragments still belong to 6 categories corresponding to the 6 standard normal vectors respectively, and then the obtained re-divided point cloud fragments are projected in different directions according to the normal vector categories;
通过对于不同视角寻找对应的法向量，然后再确认这些法向量所对应的有哪些点云片段，获得一系列被划分到对应视角类别的点云片段。By finding the corresponding normal vectors for different viewing angles, and then confirming which point cloud fragments correspond to these normal vectors, a series of point cloud fragments classified into corresponding viewing angle categories are obtained.
进行分视角传输编码的投影传输方案，包括以下任一种或任多种：不修改点云投影方法的方案、改进点云投影方法的方案、根据用户视角来修改标准法向量方案。Projection transmission schemes that perform sub-view transmission coding include any one or more of the following: a scheme that does not modify the point cloud projection method, a scheme that improves the point cloud projection method, and a standard normal vector scheme that is modified according to the user's perspective.
关于不修改点云投影方法的方案，图1为第一实施例中在不修改点云投影方案下分视角对点云片段进行处理的过程图。由图1可见，对3D点云对象按照6个标准向量进行划分，得到点云片段1至点云片段t。Regarding the solution of not modifying the point cloud projection method, FIG. 1 is a process diagram of processing point cloud fragments from perspectives without modifying the point cloud projection solution in the first embodiment. It can be seen from Fig. 1 that the 3D point cloud object is divided according to 6 standard vectors to obtain point cloud segment 1 to point cloud segment t.
通过分配参数列表(patch index list)，对每一个点云片段分配一个参数index，对需要被压缩编码的图片的每一个块分配一个参数列表patch index list，来记录这个块与哪些点云片段相关联。By assigning a parameter list (patch index list), assigning a parameter index to each point cloud segment, and assigning a parameter list patch index list to each block of the picture that needs to be compressed and encoding, to record which point cloud segments this block is related to United.
根据获得的一系列被划分到对应视角类别的点云片段，根据观测视角划分为需要被主要观测的点云片段，以及不需要被主要观测的点云片段。According to the obtained series of point cloud fragments that are classified into corresponding viewing angle categories, the point cloud fragments that need to be mainly observed and the point cloud fragments that do not need to be mainly observed are divided according to the observation angle of view.
再根据点云片段与二维片段之间的对应关系，即patch index list，确认哪些被投影到的二维片段是需要被主要观测的，哪些是不需要被主要观测的。本实施例中，将标准向量投影面为0、1、2的选择为需要被主要观测的。继而，在点云片段1至点云片段t中，对应的点云片段1、3、6、7、9等被划分为需要被主要观测的点云片段，其余被划分为不需要被主要观测的点云片段。Then, according to the corresponding relationship between the point cloud segment and the 2D segment, that is, the patch index list, confirm which 2D segments are projected to be mainly observed, and which do not need to be mainly observed. In this embodiment, the selection of the standard vector projection plane as 0, 1, and 2 is the one that needs to be mainly observed. Then, in the point cloud segment 1 to the point cloud segment t, the corresponding point cloud segments 1, 3, 6, 7, 9, etc. are divided into the point cloud segments that need to be mainly observed, and the rest are divided into the point cloud segments that do not need to be mainly observed Point cloud fragment.
例如，图3为第一实施例中不同用户视角下需要被主要观察涉及不同面的示例图；针对点云对象，其三维边缘盒(bounding box)的六个面中，图3分别展示了一个法向量对应的面需要被主要观测、两个法向量对应的面需要被主要观测、以及三个法向量对应的面需要被主要观测的三种情形。For example, Figure 3 is an example diagram of different faces that need to be mainly observed under different user perspectives in the first embodiment; for a point cloud object, among the six faces of its three-dimensional bounding box, Figure 3 shows one There are three situations in which the surface corresponding to the normal vector needs to be mainly observed, the surface corresponding to the two normal vectors needs to be mainly observed, and the surface corresponding to the three normal vectors needs to be mainly observed.
对于需要被主要观测的二维片段所属于的二维块，使用视频压缩技术进行编码时，对其设定较高准确度和较高分辨率的压缩参数；For the two-dimensional block to which the two-dimensional segment that needs to be mainly observed belongs, when using video compression technology for encoding, set compression parameters with higher accuracy and higher resolution;
对于不需要被主要观测的二维片段所属于的二维块，使用视频压缩技术进行编码时，对其设定较低准确度和较低分辨率的压缩参数。For the two-dimensional blocks to which the two-dimensional fragments that do not need to be mainly observed belong, when video compression technology is used for encoding, compression parameters of lower accuracy and lower resolution are set.
关于改进点云投影方法的方案，本实施例结合图2-1、图2-2以及图2-3予以说明，本发明的基于用户视角的点云投影传输方法中，包括以下任一种 第1种、第2种以及第3种改进点云投影方案包括：Regarding the solution for improving the point cloud projection method, this embodiment is described with reference to Figs. 2-1, 2-2, and Fig. 2-3. The point cloud projection transmission method based on the user’s perspective of the present invention includes any one of the following: The 1, 2, and 3 improved point cloud projection schemes include:
主要观察片段获取步骤：根据获得的一系列被划分到对应视角类别的点云片段，根据观测视角划分为需要被主要观测的点云片段，以及不需要被主要观测的点云片段，即确认了哪些标准法向量所对应的点云片段是需要被主要观察。Observation steps of main observation fragments: According to the obtained series of point cloud fragments that are classified into corresponding perspective categories, according to the observation perspective, they are divided into the point cloud fragments that need to be mainly observed, and the point cloud fragments that do not need to be mainly observed, which is confirmed Which standard normal vector corresponds to the point cloud segment that needs to be mainly observed.
图2-1为第一实施例中在改进点云投影方案下第1种分视角对点云片段进行处理的过程图。Figure 2-1 is a process diagram of processing point cloud fragments from the first perspective under the improved point cloud projection scheme in the first embodiment.
第1种改进点云投影方案，与图1中同样的片段划分过程，本实施例图2-1的中，在确认了哪些标准法向量所对应的点云片段是需要被主要观察的之后，将需要被主要观测的点云片段和不需要被主要观测的点云片段，根据需要投影到不同的平面上，例如，将需要被主要观测的点云片段投影入图片A中，将不需要被主要观测的点云片段投影入图片B中，对图片A、图片B进行尺寸更改，进行放大或缩小尺寸处理之后，本实施例中，是将图片B缩小后得到图片C,图片A保持不变，在需要观察到的点云片段和不需要观察到的点云片段之间错位前提下将不需要观察图片C拼合入需要观察图片A中，得到一张投影图片上，进行后续压缩编码。The first improved point cloud projection scheme is the same segment division process as in Figure 1. In Figure 2-1 of this embodiment, after confirming which standard normal vector corresponds to the point cloud segment that needs to be mainly observed, The point cloud fragments that need to be mainly observed and the point cloud fragments that do not need to be mainly observed are projected onto different planes as needed. For example, the point cloud fragments that need to be mainly observed are projected into picture A, and they do not need to be projected. The main observation point cloud fragment is projected into picture B, and the size of picture A and picture B are changed, and after the enlargement or reduction process is performed, in this embodiment, picture C is obtained after picture B is reduced, and picture A remains unchanged , Under the premise of misalignment between the point cloud segment that needs to be observed and the point cloud segment that does not need to be observed, the unobserved picture C is merged into the observable picture A to obtain a projected picture, which is then compressed and encoded.
本发明中，对图片进行尺寸更改，例如对图片A进行放大同时图片B保持不变或缩小，或者，图片A保持不变同时对图片B缩小均是本发明的发明范围之内，使得不需要观察图片和需要观察图片之间存在显著的尺寸差别即可，尺寸更改的组合方案以及放大、缩小系数均不限制发明范围。另本案中，被生成的点云片段的实际形态以实际为准，图中仅以框图代表点云片段进行示意。In the present invention, it is within the scope of the present invention to change the size of the picture, such as zooming in picture A while picture B remains the same or zoomed out, or picture A remains unchanged while reducing picture B, which is within the scope of the invention, making it unnecessary It is sufficient if there is a significant size difference between the observation picture and the picture that needs to be observed, and the combination scheme of size change and the magnification and reduction coefficients do not limit the scope of the invention. In addition, in this case, the actual shape of the generated point cloud segment is based on the actual situation. In the figure, only the block diagram represents the point cloud segment for illustration.
图2-2为第一实施例中在改进点云投影方案下第2种分视角对点云片段进行处理的过程图；Fig. 2-2 is a process diagram of processing point cloud fragments from a second perspective under the improved point cloud projection scheme in the first embodiment;
第2种改进点云投影方案，在确认了哪些标准法向量所对应的点云片段是需要被主要观察的之后，与上述第1种改进点云投影方案相同，将点云片段根据需要投影到不同的平面上，将需要被主要观测的点云片段投影入图片A中，将不需要被主要观测的点云片段投影入图片B中，本案中不进行拼合，而是根据需求对于需要观察图片A、不需要观察图片分别予以尺寸变更，例 如进行放大或缩小尺寸处理之后，分别对尺寸变更后的投影图片进行后续压缩编码。The second improved point cloud projection scheme. After confirming which standard normal vectors correspond to the point cloud fragments that need to be mainly observed, it is the same as the first improved point cloud projection scheme above, and the point cloud fragments are projected to On different planes, the point cloud fragments that need to be mainly observed are projected into picture A, and the point cloud fragments that do not need to be mainly observed are projected into picture B. In this case, no stitching is performed, but the pictures that need to be observed are based on requirements. A. There is no need to observe the pictures to be resized separately, for example, after the size is enlarged or reduced, the projected pictures after the resize are respectively subjected to subsequent compression coding.
图2-3为第一实施例中在改进点云投影方案下第3种分视角对点云片段进行处理的过程图；Figure 2-3 is a process diagram of processing point cloud fragments from the third perspective under the improved point cloud projection scheme in the first embodiment;
第3种改进点云投影方案，将点云片段1至点云片段t根据它们的标准法向量类别投影到六个不同的平面上，与六个标准向量相对应，分别得到第1组点云片段至第6组点云片段，在确认了哪些标准法向量所对应的点云片段是需要被主要观察的之后，根据需求对于这些点云片段对应的平面进行尺寸变更，如图2-3所示，根据用户视角选择标准向量0、1、2的投影面上的点云片段为主要观测面，即、第1-3组点云片段保持不变，第4-6组点云片段被缩小，这样处理之后分别再压缩编码。The third improved point cloud projection scheme is to project point cloud segment 1 to point cloud segment t on six different planes according to their standard normal vector types, corresponding to the six standard vectors, and get the first group of point clouds. Fragments to the sixth group of point cloud fragments. After confirming which standard normal vector corresponds to the point cloud fragments that need to be observed, the size of the planes corresponding to these point cloud fragments is changed according to requirements, as shown in Figure 2-3. According to the user’s perspective, select the point cloud segment on the projection surface of the standard vectors 0, 1, 2 as the main observation surface, that is, the point cloud segments of the 1-3 groups remain unchanged, and the point cloud segments of the 4-6 groups are reduced. , And then compress and code separately after such processing.
关于根据用户视角来修改标准法向量方案，包括如下步骤：Regarding the modification of the standard normal vector scheme according to the user's perspective, the following steps are included:
观测视角法向量计算步骤：获取用户观测视角的方向，计算该观测视角的法向量，具体过程如下：Observation angle normal vector calculation steps: Obtain the direction of the user's observation angle, and calculate the normal vector of the observation angle. The specific process is as follows:
对点云对象进行观测时，将被观察点云对象作为相对坐标系的原点，提供用户关于被观察点云对象的相对位置坐标，计算两者这件的位置差值，获得相对的观测视角的方向法向量，表示用户观测视角的方向。When observing a point cloud object, take the observed point cloud object as the origin of the relative coordinate system, provide the user with the relative position coordinates of the observed point cloud object, calculate the position difference between the two, and obtain the relative observation angle of view. The direction normal vector indicates the direction of the user's viewing angle.
其余标准法向量计算步骤：将该法向量定义为标准法向量之一，即将该法向量与其中一个标准法向量进行旋转映射，然后将其他的标准法向量进行同样的旋转映射处理，在使得总体的修改量较小的前提下，来获取其他对应的五组标准法向量，计算过程如下：The rest of the standard normal vector calculation steps: define the normal vector as one of the standard normal vectors, that is, the normal vector and one of the standard normal vectors are rotated and mapped, and then the other standard normal vectors are subjected to the same rotation mapping processing to make the overall Under the premise that the modification amount of is small, to obtain the other corresponding five sets of standard normal vectors, the calculation process is as follows:
设用户观测视角法向量通式为
Figure PCTCN2020112857-appb-000001
将其与标准法向量(1，0，0)设定为旋转映射关系，则其他的五个标准法向量(0，1，0)，(-1，0，0)，(0，-1，0)，(0，0，1)，(0，0，-1)对应的旋转映射的法向量通式分别对应的可表示为： Suppose the general formula of the normal vector of the user observation angle is
Figure PCTCN2020112857-appb-000001
Set it to the standard normal vector (1, 0, 0) as a rotation mapping relationship, then the other five standard normal vectors (0, 1, 0), (-1, 0, 0), (0, -1 , 0), (0, 0, 1), (0, 0, -1) corresponding to the normal vector general formula of the rotation map can be expressed as:
Figure PCTCN2020112857-appb-000002
Figure PCTCN2020112857-appb-000002
Figure PCTCN2020112857-appb-000003
Figure PCTCN2020112857-appb-000003
图4为第一实施例中发送端的系统示意框图；4 is a schematic block diagram of the system of the sending end in the first embodiment;
本发明提供了一种基于用户视角的点云投影传输系统，包括：片段生成模块、点云片段确定模块以及作为投影编码处理模块的分视角传输编码模块。The present invention provides a point cloud projection transmission system based on a user's perspective, which includes a segment generation module, a point cloud segment determination module, and a perspective transmission encoding module as a projection encoding processing module.
其中，片段生成模块，用于将点云对象在标准向量上初步划分，得到待处理点云片段。Among them, the fragment generation module is used to preliminarily divide the point cloud object on the standard vector to obtain the point cloud fragment to be processed.
点云片段确定模块：根据观测视角来确定对应的点云片段，获得一系列被划分到对应视角类别的点云片段；Point cloud segment determination module: Determine the corresponding point cloud segment according to the observation perspective, and obtain a series of point cloud segments that are classified into the corresponding perspective category;
分视角传输编码模块用于根据用户视角确定投影传输方案，对待处理点云片段进行处理。根据获得的一系列被划分到对应视角类别的点云片段，采用分视角传输方案，对于需要观测到的视角进行高清晰度的编码传输，对于不需要被观察得到的视角进行较低清晰度的编码传输。The perspective transmission coding module is used to determine the projection transmission scheme according to the user perspective, and process the to-be-processed point cloud segment. According to the obtained series of point cloud fragments divided into corresponding viewing angle categories, the perspective transmission scheme is adopted to perform high-definition encoding transmission for the viewing angle that needs to be observed, and lower-definition viewing angle for the viewing angle that does not need to be observed. Encoding transmission.
优选地，所述点云片段确定模块：Preferably, the point cloud segment determination module:
根据点云的法向量来进行片段划分，计算点云中每一个点的法向量与6个标准法向量之间的距离，取距离最近的标准法向量为点的类别，来将所有的点划分为6个大类，即6个大类的点云片段，之后再将这6个大类的点云片段进行连接组件提取，将6个大类的点云片段重新划分，获得重新划分后的点云片段；Divide fragments according to the normal vector of the point cloud, calculate the distance between the normal vector of each point in the point cloud and the 6 standard normal vectors, and take the closest standard normal vector as the point category to divide all points For 6 major categories, that is, 6 major categories of point cloud fragments, then these 6 major categories of point cloud fragments are extracted for connection components, and the 6 major categories of point cloud fragments are re-divided to obtain the re-divided point cloud fragments. Point cloud fragment;
所述6个标准法向量为：The 6 standard normal vectors are:
(1，0，0)，(-1，0，0)，(0，1，0)，(0，-1，0)，(0，0，1)，(0，0，-1)；(1, 0, 0), (-1, 0, 0), (0, 1, 0), (0, -1, 0), (0, 0, 1), (0, 0, -1) ；
所述连接组件提取指：将6个大类的点云片段，根据点与点之间的空间距离以及点与点之间的邻接关系，将点与点之间的空间距离大于预设距离范围且点与点之间不邻接的点云片段划分为更小的点云片段，使重新划分后的点云片段的点与点之间的空间距离小于预设距离范围且点与点之间邻接；The connection component extraction refers to: taking 6 broad categories of point cloud fragments, according to the spatial distance between the points and the adjacency relationship between the points, the spatial distance between the points is greater than the preset distance range And the point cloud fragments that are not adjacent to each other are divided into smaller point cloud fragments, so that the spatial distance between the points of the re-divided point cloud fragment is less than the preset distance range and the points are adjacent to each other ；
所述重新划分后的点云片段，仍分别属于对应于6个标准法向量的6个类别，然后根据法向量类别对获得的重新划分后的点云片段进行分方向的投影；The re-divided point cloud fragments still belong to 6 categories corresponding to the 6 standard normal vectors respectively, and then the obtained re-divided point cloud fragments are projected in different directions according to the normal vector categories;
通过对于不同视角寻找对应的法向量，然后再确认这些法向量所对应的有哪些点云片段，获得一系列被划分到对应视角类别的点云片段；By searching for the corresponding normal vectors for different perspectives, and then confirming which point cloud fragments these normal vectors correspond to, a series of point cloud fragments classified into corresponding perspective categories are obtained;
所述分视角传输编码模块：The sub-view transmission coding module:
所述分视角传输方案包括以下任一种或任多种：不修改点云投影方法的方案、改进点云投影方法的方案、根据用户视角来修改标准法向量方案。不修改点云投影方法的方案、改进点云投影方法的方案、根据用户视角来修改标准法向量方案与上述相同，在此省略重复说明。The perspective transmission solution includes any one or more of the following: a solution that does not modify the point cloud projection method, a solution that improves the point cloud projection method, and a standard normal vector solution that is modified according to the user's perspective. The solution of not modifying the point cloud projection method, the solution of improving the point cloud projection method, and the solution of modifying the standard normal vector according to the user's perspective are the same as the above, and the repeated description is omitted here.
另外，本发明提供了一种基于用户视角的点云接收处理系统，包括：In addition, the present invention provides a point cloud receiving and processing system based on the user's perspective, including:
逆处理模块，利用用户视角而确定的编解码参数对接收到的码流得到视频序列，和对视频序列处理得到点云片段，以及对每个点云片段进行解码重建得到点云对象。The inverse processing module uses the codec parameters determined from the user's perspective to obtain a video sequence from the received bitstream, process the video sequence to obtain a point cloud segment, and decode and reconstruct each point cloud segment to obtain a point cloud object.
本发明提供的基于用户视角的点云投影传输系统、基于用户视角的点云接收处理系统，可以通过本发明给的基于用户视角的点云投影传输系统方法、基于用户视角的点云接收处理方法的步骤流程实现。The point cloud projection transmission system based on the user's perspective and the point cloud receiving and processing system based on the user's perspective provided by the present invention can pass the point cloud projection transmission system method based on the user's perspective and the point cloud receiving processing method based on the user's perspective provided by the present invention The steps of the process are realized.
下面通过优选例，对本发明进行更为具体地说明。Hereinafter, the present invention will be explained more specifically through preferred examples.
优选例1：Preferred example 1:
本发明的目的是提供一种基于用户视角的点云投影传输方案，基于目前已有的典型点云压缩方案的基础上，进行修改完善，根据用户的需求，在不同的观测视角，提供不同程度的呈现效果。The purpose of the present invention is to provide a point cloud projection transmission scheme based on the user's perspective, which is modified and perfected on the basis of the existing typical point cloud compression scheme, and provides different degrees of observation at different viewing angles according to the needs of users. The rendering effect.
本发明基于目前的经典点云序列编码传输方案来进行实现，经典的点云序列编码方法需要依次进行，根据法向量进行片段划分，根据片段连接进行片段细划分，根据法向量进行不同方向投影，对投影平面进行空白填充，对投影获得的视频序列使用例如HEVC的二维编码工具来进行压缩编码。The present invention is implemented based on the current classic point cloud sequence coding transmission scheme. The classic point cloud sequence coding method needs to be carried out sequentially. Segments are divided according to normal vectors, segment is subdivided according to segment connections, and projections in different directions are performed according to normal vectors. Fill the projection plane with blanks, and use a two-dimensional coding tool such as HEVC to compress and encode the video sequence obtained by the projection.
为了实现在不同视角呈现出不同的呈现效果，可以有很多种实现方法，例如：In order to achieve different rendering effects from different perspectives, there can be many implementation methods, for example:
(1)可以只编码传输需要观测到的视角对应的数据。(1) Only the data corresponding to the viewing angle that needs to be observed can be encoded and transmitted.
(2)可以分视角分清晰度的进行编码传输。即对于需要观测到的视角进行高清晰度的编码传输，对于不需要被观察得到的视角进行较低清晰度的编码传输。(2) It can be encoded and transmitted in different viewing angles and definitions. That is, the high-definition coding transmission is performed for the viewing angle that needs to be observed, and the lower-definition coding transmission is performed for the viewing angle that does not need to be observed.
(3)可以在进行传输分配带宽时，优先分配给需要观测到的视角对应的数据，保证需要被观测到的视角对应的数据拥有充足的带宽进行传输。(3) When bandwidth is allocated for transmission, the data corresponding to the viewing angle that needs to be observed can be preferentially allocated to ensure that the data corresponding to the viewing angle that needs to be observed has sufficient bandwidth for transmission.
(4)可以在进行传输分配带宽时，按照较高比例分配给需要观测到的视角对应的数据，按照较低比例分配给不需要观测到的视角对应的数据。(4) When bandwidth is allocated for transmission, the data corresponding to the viewing angle that needs to be observed can be allocated according to a higher ratio, and the data corresponding to the viewing angle that does not need to be observed according to a lower ratio.
接下来，我们将以分视角分清晰度传输为例，来说明如何实现分视角传输，除了分视角分清晰度之外，还有许多不同的实现方式。为实现上述目的，本发明提供了以下两种技术方案分别从三个角度来实现：Next, we will take the perspective and definition transmission as an example to illustrate how to realize the perspective transmission. In addition to the perspective and definition, there are many different implementations. In order to achieve the above-mentioned objectives, the present invention provides the following two technical solutions to be implemented from three perspectives:
第一种方案，在经典的点云序列编码的基础上，在获得投影之后的视频序列之后，不改变视频序列的结构，通过修改视频编码工具的编码参数来实现。具体的来说，由于点云序列根据法向量划分为不同的片段，因此我们可以根据视角寻找到对应的视角对应的法向量，进一步寻找到对应的片段，对于这些片段，定位到在二维图片上对应的投影到的片段，然后可以在视频编码工具中分别对于这些片段分配不同的压缩参数。不同的压缩参数可以获得不同的压缩编码效果，例如对于压缩参数量化步长序号quantization step，简称QP值，低的QP值可以获得较高的压缩质量，高的QP值可以获得较低的压缩质量。The first solution is based on the classic point cloud sequence coding, after obtaining the video sequence after projection, without changing the structure of the video sequence, it is achieved by modifying the coding parameters of the video coding tool. Specifically, because the point cloud sequence is divided into different fragments according to the normal vector, we can find the normal vector corresponding to the corresponding perspective according to the perspective, and further find the corresponding fragments. For these fragments, locate in the two-dimensional picture The corresponding projected segments can then be assigned different compression parameters to these segments in the video encoding tool. Different compression parameters can achieve different compression coding effects. For example, for the compression parameter quantization step sequence number, QP value for short, a low QP value can achieve higher compression quality, and a high QP value can achieve lower compression quality. .
对于我们需要的视角对应的二维图片片段，进行高分辨率高精确度压缩，对于不需要的视角对应的二维图片片段进行低分辨率低精确度压缩。Perform high-resolution and high-precision compression for the two-dimensional image fragments corresponding to the viewing angles we need, and perform low-resolution and low-precision compression for the two-dimensional image fragments corresponding to the unwanted viewing angles.
第二种方案，需要修改投影的模式，因此视频序列的结构也将进行相应的修改，但是仍然获得二维视频序列，使用二维编码工具压缩。例如，可以在对于点云进行片段划分和投影之后，对于不需要的视角的点云片段获得的二维片段，进行降采样处理。经过这样处理之后，我们可以获得具有复合分辨率的视频序列，然后使用二维编码工具进行同样分辨率和精确度的压缩。除此之外，我们还提供了其余的一些思路。In the second solution, the projection mode needs to be modified, so the structure of the video sequence will be modified accordingly, but the two-dimensional video sequence is still obtained and compressed using a two-dimensional coding tool. For example, after segment division and projection are performed on the point cloud, down-sampling processing can be performed on the two-dimensional segment obtained from the point cloud segment of the unwanted perspective. After such processing, we can obtain a video sequence with composite resolution, and then use a two-dimensional coding tool to compress with the same resolution and accuracy. In addition, we also provide some other ideas.
第三种方案，可以根据用户的视角来修改标准法向量。目前的动态点云的经典压缩算法是将划分好的点云片段，根据与六个标准法向量之间的关系，划分类别，然后选择其中一种作为主要投影方向，投影到平面上面。基于这种投影方式，可以根据用户视角来修改标准法向量，使用户的视角对应的为投影的正面。In the third scheme, the standard normal vector can be modified according to the user's perspective. The current classic compression algorithm of dynamic point cloud is to divide the divided point cloud fragments into categories according to the relationship between the six standard normal vectors, and then select one of them as the main projection direction and project it onto the plane. Based on this projection method, the standard normal vector can be modified according to the user's perspective, so that the user's perspective corresponds to the front of the projection.
需要说明的是，三种方案之间并不对立，可以互相结合使用，例如可以根据用户视角来修改标准法向量，然后再修改投影模式或者编码参数。It should be noted that the three schemes are not opposed to each other and can be used in combination. For example, the standard normal vector can be modified according to the user's perspective, and then the projection mode or coding parameters can be modified.
根据上文所述，本发明为了实现“不同视角呈现出不同的呈现效果”，共采用了三种方案。其中，三种方案在不同的策略方面实现分视角编码，彼此 之间均可以两两结合，或者三者结合使用。According to the foregoing, in order to achieve "different viewing angles presenting different rendering effects", the present invention adopts three solutions. Among them, the three schemes implement split-view coding in terms of different strategies, and they can all be combined with each other, or the three can be used in combination.
通过以下表1对三种方案在不同的策略方面实现分视角编码进行总结展示。The following table 1 summarizes the three schemes in terms of different strategies to achieve sub-view coding.
Figure PCTCN2020112857-appb-000004
Figure PCTCN2020112857-appb-000004
表1Table 1
优选例2：Preferred example 2:
本发明所提出的三种实现点云的分视角分清晰度传输的编码方案，具体的实施方案如下：The three coding schemes proposed by the present invention for realizing the transmission of point cloud by viewing angle and resolution, the specific implementation schemes are as follows:
首先，无论采取哪一种编码方案都需要根据观测视角来确定对应的点云片段。可以在目前的点云序列编码方法的基础上较为简单的实现。经过确定视角对应的点云片段这一步骤，可以获得一系列被划分到对应视角类别的点云片段，这是这也是本专利所提出的点云分视角传输策略中的三种技术方案的前提与基础。具体实现算法如下。First of all, no matter which coding scheme is adopted, the corresponding point cloud segment needs to be determined according to the observation angle of view. It can be implemented relatively simply on the basis of the current point cloud sequence coding method. After the step of determining the point cloud segment corresponding to the perspective, a series of point cloud segments classified into the corresponding perspective category can be obtained. This is the premise of the three technical solutions in the point cloud sub-perspective transmission strategy proposed in this patent. And foundation. The specific implementation algorithm is as follows.
目前的点云序列的经典编码方式，是根据点云的法向量来进行片段划分。通过计算点云中每一个点的法向量与6个标准法向量之间的距离：The current classic coding method of point cloud sequence is to divide the segment according to the normal vector of the point cloud. By calculating the distance between the normal vector of each point in the point cloud and the 6 standard normal vectors:
(1，0，0)，(-1，0，0)，(0，1，0)，(0，-1，0)，(0，0，1)，(0，0，-1)(1, 0, 0), (-1, 0, 0), (0, 1, 0), (0, -1, 0), (0, 0, 1), (0, 0, -1)
取距离最近的标准法向量，来将所有的点划分为了6个大类，即6个大的点云片段。之后再将这6个大的点云片段进行连接组件提取，划分为更加细小的片段。但是这些更小的片段，仍分别属于对于6个标准法向量的6个类别，然后根据法向量类别进行分方向的投影。连接组件提取处理，即将已 经分为六大类的点，根据空间距离远近以及点与点之间的邻接关系，划分为更小的点云片段，防止出现在同一片段中的点与点之间距离过大且不相邻的情况。Take the nearest standard normal vector to divide all points into 6 categories, namely 6 large point cloud segments. Then, these 6 large point cloud segments are extracted by connecting components and divided into smaller segments. However, these smaller fragments still belong to the 6 categories for the 6 standard normal vectors, and then the projections are performed in different directions according to the normal vector categories. Connection component extraction processing, that is, points that have been divided into six categories, according to the spatial distance and the adjacency relationship between points, are divided into smaller point cloud segments to prevent points from appearing in the same segment. The distance is too large and not adjacent.
例如，可以对于每一个划分好的片段，分配一个参数index来标志他们属于哪一个法向量平面的，对应关系如下表2所示，表1是第一实施例中的分配参数列表(patch index list)：For example, for each divided segment, a parameter index can be assigned to indicate which normal vector plane they belong to. The corresponding relationship is shown in Table 2 below. Table 1 is the patch index list in the first embodiment. ):
参数IndexParameter Index 标准投影向量PlaneStandard Projection Vector Plane
00 (1，0，0)(1, 0, 0)
11 (-1，0，0)(-1, 0, 0)
22 (0，1，0)(0, 1, 0)
33 (0，-1，0)(0, -1, 0)
44 (0，0，1)(0, 0, 1)
55 (0，0，-1)(0, 0, -1)
表2Table 2
由于进行了对于点云的分法向量投影，因此我们可以通过对于不同视角寻找对应的法向量，然后再确认这些法向量所对应的有哪些点云片段。通过对于不同的视角来确认对应的法向量的方法，可以包括如下：Since the normal vector projection of the point cloud is performed, we can find the corresponding normal vectors from different perspectives, and then confirm which point cloud fragments these normal vectors correspond to. The method of confirming the corresponding normal vector from different perspectives can include the following:
例如，标准法向量(0，1，0)对应的为从正前方看过去的视角，那么当用户从正前方观测点云对象时，可以确认被主要观测到的是法向量属于(0，1，0)这一类别的那些点云片段。如果不是从标准正向视角观测，被主要观测的点云片段类别更多。For example, the standard normal vector (0, 1, 0) corresponds to the perspective from the front. When the user observes the point cloud object from the front, it can be confirmed that the normal vector belongs to (0, 1). , 0) Those point cloud fragments of this category. If it is not observed from a standard forward perspective, there are more types of point cloud fragments that are mainly observed.
1、不修改点云投影方法的方案：1. Plan without modifying the point cloud projection method:
对于目前的视频压缩技术，例如HEVC视频压缩技术，是通过对于需要压缩的二维图片进行块划分，即进行二维片段均匀划分，然后分别对这些块设定QP偏移值以及其他的一些编码参数进行编码的。For the current video compression technology, such as HEVC video compression technology, the two-dimensional pictures that need to be compressed are divided into blocks, that is, the two-dimensional segments are divided evenly, and then the QP offset values and other codes are set for these blocks respectively. The parameters are encoded.
因此可以在获取用户视角的基础上，确认哪些点云片段需要被主要观测。首先可以对每一个点云片段分配一个参数index，然后对需要被压缩编码的图片的每一个块分配一个参数列表patch index list，来记录这个块与哪些点云片段相关联。而这些点云片段在前述划分的过程中，已经根据观测视角 被划分为需要被主要观测的点云片段，以及不需要被主要观测的点云片段。因此，可以根据点云片段与二维片段之间的关系，即patch index list，来确认哪些被投影到的二维片段是需要被主要观测的，哪些是不需要被主要观测的。Therefore, on the basis of obtaining the user's perspective, it is possible to confirm which point cloud segments need to be mainly observed. First, a parameter index can be assigned to each point cloud segment, and then a parameter list patch index list can be assigned to each block of the picture that needs to be compressed and encoded to record which point cloud segments this block is associated with. In the aforementioned division process, these point cloud segments have been divided into point cloud segments that need to be primarily observed and point cloud segments that do not need to be primarily observed according to the observation angle of view. Therefore, according to the relationship between the point cloud segment and the two-dimensional segment, that is, the patch index list, it is possible to confirm which of the projected two-dimensional segments need to be mainly observed and which do not need to be mainly observed.
(1)对于需要被主要观测的二维片段所属于的二维块，我们在使用视频压缩技术进行编码的时候，对其设定较高准确度和较高分辨率的压缩参数。例如，使用HEVC编码时，设定较低的QP偏移值。(1) For the two-dimensional block to which the two-dimensional segment that needs to be mainly observed belongs, when we use video compression technology for encoding, we set compression parameters with higher accuracy and higher resolution. For example, when using HEVC encoding, set a lower QP offset value.
(2)对于不需要被主要观测的二维片段所属于的二维块，我们在使用视频压缩技术进行编码的时候，对其设定较低准确度和较低分辨率的压缩参数。例如，使用HEVC编码时，设定较高的QP偏移值。(2) For the two-dimensional blocks to which the two-dimensional segments that do not need to be mainly observed belong, when we use video compression technology for encoding, we set compression parameters with lower accuracy and lower resolution for them. For example, when using HEVC encoding, set a higher QP offset value.
关于发送端与接收端，针对方法一：不修改投影方案的方法，需要在发送端记录每个片段的编解码参数，这部分参数将作为新增的附加信息进行编码传输；在接收端，需要根据接受到的每个片段的编解码参数，对每个片段进行编解码。Regarding the sending end and the receiving end, for method one: the method of not modifying the projection scheme, the coding and decoding parameters of each segment need to be recorded at the sending end, and this part of the parameters will be encoded and transmitted as new additional information; at the receiving end, it is required According to the received encoding and decoding parameters of each segment, each segment is encoded and decoded.
2、改进点云投影方法的方案：2. Scheme to improve the point cloud projection method:
除了对不同的二维片段设定不同的编码参数之外，也可以通过修改点云的投影方式来实现点云的分视角分清晰度的编码效果。In addition to setting different coding parameters for different two-dimensional clips, the point cloud projection method can also be modified to achieve the coding effect of the point cloud by viewing angle and resolution.
目前的经典的点云序列编码方案中，对于各个点云片段投影到二维图片的方法如下：根据各个点云片段的标准法向量类别分方向来进行投影，例如，一个点云片段的标准法向量类别为(1，0，0)，那么进行投影时，这个点云片段的三维坐标(x，y，z)中的(y，z)坐标被用作平面坐标，(x)坐标被用作投影深度。至于投影位置在图片上的哪里，这主要根据节约总体占用空间的方式来进行布局，由左上角开始，依次填充进入二维图片，使得最终的总的占用平面大小最节约。In the current classic point cloud sequence coding scheme, the method for projecting each point cloud segment to a two-dimensional picture is as follows: Projection is performed according to the standard normal vector category of each point cloud segment, for example, the standard method of a point cloud segment The vector category is (1, 0, 0), then when projecting, the (y, z) coordinates in the three-dimensional coordinates (x, y, z) of this point cloud segment are used as plane coordinates, and the (x) coordinates are used Make the projection depth. As for where the projection position is on the picture, this is mainly based on the way to save the overall occupied space, starting from the upper left corner, filling into the two-dimensional picture in turn, so that the final total occupied plane size is the most economical.
但是在这种点云投影方案中，没有考虑到视角带来的需求，无论哪一种标准法向量的点云片段都被以同样的分辨率投影到二维图片上，也就意味着无论哪一个视角观测到的点云片段都以相同的分辨率进行投影编码。However, in this point cloud projection scheme, the need for perspective is not considered. No matter which kind of standard normal vector point cloud fragments are projected onto the two-dimensional picture with the same resolution, it means that no matter which The point cloud segments observed from one angle of view are all projected and encoded with the same resolution.
针对这种问题，本发明提出了对于点云分视角投影的方式优化。并基于 这种思想提出了如下的3种优化思路，三种优化思路可并行进行，即择一即可：To solve this problem, the present invention proposes to optimize the way of projection of the point cloud from different perspectives. And based on this idea, the following three optimization ideas are proposed. The three optimization ideas can be performed in parallel, that is, you can choose one:
(1)在确认了哪些标准法向量所对应的点云片段是需要被主要观察的之后，将点云片段，根据需要投影到不同的平面上，进行处理之后，再拼接到一张图片上。(1) After confirming which point cloud fragments corresponding to standard normal vectors need to be mainly observed, the point cloud fragments are projected onto different planes as needed, processed, and then spliced onto a picture.
例如，将需要被主要观察的点云片段按照经典方法投影到图片A上，尺寸为1280×1280，将不需要被主要观察的点云片段按照经典方法投影到图片B上，尺寸为1280×1280。然后对于图片B进行缩小处理获得图片C，尺寸为640×640。由于按照经典方法投影时，确保了总占用空间的最小，图片A上被投影到的位置很有可能不超过1/4，因此图片C可以贴在图片A的没有被投影占用到的区域内。For example, the point cloud fragments that need to be mainly observed are projected onto picture A according to the classic method, and the size is 1280×1280, and the point cloud fragments that do not need to be mainly observed are projected onto picture B according to the classic method, and the size is 1280×1280 . Then, the picture B is reduced to obtain the picture C, and the size is 640×640. Since the projection according to the classic method ensures the minimum total occupied space, the projected position on the picture A is likely to be no more than 1/4, so the picture C can be pasted in the area of the picture A that is not occupied by the projection.
对比优化前后，优化前占用的为1280×1280大小的图片，优化之后占用的仍为1280×1280大小的图片，但是不被主要观察的点云片段占据的平面空间更小了，采取同样的视频编码技术，可以达到根据视角需求来分清晰度编码呈现的目的。Comparing before and after optimization, the picture occupies a size of 1280×1280 before the optimization, and the picture occupies a size of 1280×1280 after the optimization, but the plane space occupied by the point cloud fragments not mainly observed is smaller. Take the same video The coding technology can achieve the purpose of dividing the definition coding presentation according to the viewing angle requirements.
(2)在确认了哪些标准法向量所对应的点云片段是需要被主要观察的之后，将点云片段根据需要投影到不同的平面上，根据需求对于这些平面进行处理之后再压缩编码。(2) After confirming which standard normal vectors correspond to point cloud segments that need to be mainly observed, the point cloud segments are projected onto different planes as required, and these planes are processed according to requirements before compression and encoding.
例如，将需要被主要观察的点云片段按照经典方法投影到图片A上，将不需要被主要观察的点云片段按照经典方法投影到图片B上，然后对于图片B进行缩小尺寸处理。之后，对于图片A和缩小后的图片B均进行压缩编码呈现。For example, point cloud fragments that need to be mainly observed are projected onto picture A according to the classic method, point cloud fragments that do not need to be mainly observed are projected onto picture B according to the classic method, and then picture B is reduced in size. After that, both the picture A and the reduced picture B are compressed and encoded for presentation.
对比优化前后，优化前呈现的为一组视频序列，优化之后呈现的为两组视频序列，但是不被主要观察的点云片段由于缩小处理，降低了压缩编码的难度和成本，采取同样的视频编码技术，可以达到根据视角需求来分清晰度编码呈现的目的。Before and after optimization, the video sequence presented before optimization is a set of video sequences, and the video sequence presented after optimization is two groups of video sequences. However, the point cloud fragments that are not mainly observed are reduced due to the reduction process, which reduces the difficulty and cost of compression encoding, and adopts the same video The coding technology can achieve the purpose of dividing the definition coding presentation according to the viewing angle requirements.
(3)在确认了哪些标准法向量所对应的点云片段是需要被主要观察的之后，将点云片段根据他们的标准法向量类别投影到不同的平面上，根据需求对于这些平面进行处理之后再压缩编码。(3) After confirming which point cloud fragments corresponding to standard normal vectors need to be mainly observed, project the point cloud fragments onto different planes according to their standard normal vector categories, and process these planes according to requirements Re-compress and encode.
例如，将各个点云片段按照根据他们的标准法向量类别分别按照经典方法投影到平面A，B，C，D，E，F上。由于可以根据用户的视角来确定哪些标准法向量的点云片段需要被主要观察，因此可以确定这6个平面中，哪些不需要被主要观察，可以进行缩小尺寸处理。通过对点云序列的每一帧进行这种处理之后，可以获得六组视频序列，使用同样的视频编码技术进行编码处理。For example, each point cloud segment is projected onto planes A, B, C, D, E, and F according to their standard normal vector categories, respectively, according to the classic method. Since it is possible to determine which standard normal vector point cloud fragments need to be mainly observed according to the user's perspective, it is possible to determine which of the six planes do not need to be mainly observed, and can be reduced in size. After performing this processing on each frame of the point cloud sequence, six groups of video sequences can be obtained, and the same video coding technology is used for coding processing.
对比优化前后，优化前呈现的为一组的视频序列，优化之后呈现的为多组视频序列，但是不被主要观察的点云片段由于缩小处理，降低了压缩编码的难度和成本，采取同样的视频编码技术，可以达到根据视角需求来分清晰度编码呈现的目的。Before and after optimization, the video sequence presented before optimization is a group of video sequences, and the video sequence presented after optimization is multiple groups of video sequences. However, the point cloud fragments that are not mainly observed are reduced due to the reduction process, which reduces the difficulty and cost of compression encoding. Video coding technology can achieve the purpose of dividing definition coding and presentation according to viewing angle requirements.
关于发送端与接收端，针对方法二：修改点云投影方式的方法，通过以下实现。方案一：将用户视角对应的片段与非用户视角对应的片段拼接在一个视频序列上的方案。需要在发送端记录所有的非用户视角对应的片段的index，以及尺寸缩减的比例，这些信息将作为新的附加信息进行编码传输；在接收端，需要根据所接收到的index信息以及缩减比例来将这些片段恢复为原始尺寸，再进行后续的三维点云重建步骤。方案二：将用户视角对应的片段与非用户视角对应的片段分别投影到两个视频序列的方案。在发送端需要编码两组视频序列，需要在发送端记录非用户视角对应的视频序列的缩减比例，这些信息将作为新的附加信息进行编码传输，此外还需要记录各视频序列在编码后的码流中的位置；在接收端，需要先根据各视频序列在码流中的位置恢复出各视频序列，再需要根据所接收到的缩减比例来将这些视频序列恢复为原始尺寸，再进行后续的三维点云重建步骤，并需要将两组视频序列所恢复出来的点云拼接在一起。方案三：将各个标准法向量对应的片段分别投影到各个视频序列的方案。在发送端需要编码六组视频序列，需要在发送端记录非用户视角对应的视频序列的缩减比例，这些信息将作为新的附加信息进行编码传输，此外还需要记录各视频序列在编码后的码流中的位置；在接收端，需要先根据各视频序列在码流中的位置恢复出各视频序列，再需要根据所接收到的缩减比例来将这些视频序列恢复为原始尺寸，再进行后续的三维点云重建步骤，并需要将 六组视频序列所恢复出来的点云拼接在一起。Regarding the sending end and the receiving end, the second method: the method of modifying the point cloud projection method is realized by the following. Solution 1: A solution in which segments corresponding to the user's perspective and segments corresponding to the non-user's perspective are spliced on a video sequence. It is necessary to record the index of all segments corresponding to the non-user perspective and the size reduction ratio at the sending end. This information will be encoded and transmitted as new additional information; at the receiving end, it needs to be based on the received index information and the reduction ratio. Restore these fragments to the original size, and then perform the subsequent 3D point cloud reconstruction steps. Solution 2: A solution of projecting segments corresponding to the user's perspective and segments corresponding to the non-user's perspective to two video sequences respectively. Two sets of video sequences need to be encoded at the sending end, and the reduction ratio of the video sequence corresponding to the non-user perspective needs to be recorded at the sending end. This information will be encoded and transmitted as new additional information. In addition, the encoded code of each video sequence needs to be recorded. The position in the stream; at the receiving end, each video sequence needs to be restored according to the position of each video sequence in the code stream, and then these video sequences need to be restored to the original size according to the received reduction ratio, and then the subsequent steps The step of 3D point cloud reconstruction, and the point clouds recovered from the two sets of video sequences need to be stitched together. Solution 3: Project the fragments corresponding to each standard normal vector to each video sequence. Six groups of video sequences need to be encoded at the sending end, and the reduction ratio of the video sequence corresponding to the non-user perspective needs to be recorded at the sending end. This information will be encoded and transmitted as new additional information. In addition, the encoded code of each video sequence needs to be recorded. The position in the stream; at the receiving end, each video sequence needs to be restored according to the position of each video sequence in the code stream, and then these video sequences need to be restored to the original size according to the received reduction ratio, and then the subsequent steps Three-dimensional point cloud reconstruction steps, and need to stitch together the point clouds recovered from the six groups of video sequences.
3、根据用户视角来修改标准法向量：3. Modify the standard normal vector according to the user's perspective:
目前的点云序列的经典编码方式中，点云片段划分依据的标准法向量为以下的六个法向量：In the current classic coding method for point cloud sequences, the standard normal vectors for dividing point cloud segments are the following six normal vectors:
(1，0，0)，(-1，0，0)，(0，1，0)，(0，-1，0)，(0，0，1)，(0，0，-1)(1, 0, 0), (-1, 0, 0), (0, 1, 0), (0, -1, 0), (0, 0, 1), (0, 0, -1)
但是这样导致当用户修改观测视角时，主要被观测的投影方向可以能过多。例如，当用户视角为被观测物体的斜上方，则主要被观测的投影面则需要包括被观测物体的正面，侧面，以及上面，即至少有三面。而理想的分视角传输中，主要被观测的投影面最好只有一面，因此上述情况不能达到很好的分视角传输的效果。对此，本发明提出，根据用户的观测视角自适应的调整标准法向量的方向。However, when the user modifies the viewing angle, the main observing projection direction may be too much. For example, when the user's perspective is obliquely above the observed object, the main projected surface to be observed needs to include the front, side, and top of the observed object, that is, there are at least three sides. In the ideal sub-view transmission, it is better to have only one projection surface that is mainly observed. Therefore, the above-mentioned situation cannot achieve a good sub-view transmission effect. In this regard, the present invention proposes to adaptively adjust the direction of the standard normal vector according to the user's observation angle of view.
首先，可以获取用户观测视角的方向，计算该观测视角的法向量。例如，对点云对象进行观测时，将被观察点云对象作为相对坐标系的原点，提供用户关于被观察点云对象的相对位置坐标，计算两者这件的位置差值，可以获得相对的观测视角的方向法向量，由此来表示用户观测视角的方向。First, the direction of the user's observation angle can be obtained, and the normal vector of the observation angle can be calculated. For example, when observing a point cloud object, take the observed point cloud object as the origin of the relative coordinate system, provide the user with the relative position coordinates of the observed point cloud object, and calculate the position difference between the two to obtain the relative position. The normal vector of the direction of the viewing angle, which represents the direction of the user's viewing angle.
然后将该法向量定义为标准法向量之一，即将该法向量与其中一个标准法向量进行旋转映射，然后将其他的标准法向量进行同样的旋转映射处理，在使得总体的修改量较小的前提下，来获取其他对应的五组标准法向量。计算公式i如下：Then the normal vector is defined as one of the standard normal vectors, that is, the normal vector and one of the standard normal vectors are rotated and mapped, and then the other standard normal vectors are subjected to the same rotation mapping processing, so that the overall modification amount is small Under the premise, to obtain the other corresponding five sets of standard normal vectors. The calculation formula i is as follows:
若用户观测视角法向量通式为
Figure PCTCN2020112857-appb-000005
需要注意的是，该法向量是进行过归一化处理的，即三个坐标的平方和为1。将其与标准法向量(1，0，0)设定为旋转映射关系，则其他的五个标准法向量(0，1，0)，(-1，0，0)，(0，-1，0)，(0，0，1)，(0，0，-1)对应的旋转映射的法向量通式分别对应的可表示为： If the general formula of the normal vector of the user's observation angle is
Figure PCTCN2020112857-appb-000005
It should be noted that the normal vector has been normalized, that is, the sum of the squares of the three coordinates is 1. Set it to the standard normal vector (1, 0, 0) as a rotation mapping relationship, then the other five standard normal vectors (0, 1, 0), (-1, 0, 0), (0, -1 , 0), (0, 0, 1), (0, 0, -1) corresponding to the normal vector general formula of the rotation map can be expressed as:
Figure PCTCN2020112857-appb-000006
Figure PCTCN2020112857-appb-000006
Figure PCTCN2020112857-appb-000007
Figure PCTCN2020112857-appb-000007
例如，如果获得观测视角的法向量为
Figure PCTCN2020112857-appb-000008
则可以获得对应的一组正交的标准法向量为： For example, if the normal vector of the observation angle is
Figure PCTCN2020112857-appb-000008
Then the corresponding set of orthogonal standard normal vectors can be obtained as:
Figure PCTCN2020112857-appb-000009
Figure PCTCN2020112857-appb-000010
(0，0，1)，(0，0，-1)
Figure PCTCN2020112857-appb-000009
Figure PCTCN2020112857-appb-000010
(0, 0, 1), (0, 0, -1)
其中，在使得总体的修改量较小的前提下，来获取其他对应的五组标准法向量，总体的修改量的计算过程如下：Among them, under the premise that the overall modification amount is small, the other corresponding five sets of standard normal vectors are obtained. The calculation process of the overall modification amount is as follows:
保证总体的修改量较小的方法为，在用户视角法向量与标准法向量对应，并进行旋转之后，使包括用户视角法向量在内的新的六个标准法向量与对应的六个标准法向量的点乘积的和尽量大。在这种情况下，设用户观测视角法向量通式为
Figure PCTCN2020112857-appb-000011
将其与标准法向量(1，0，0)设定为旋转映射关系，则其他的五个标准法向量(0，1，0)，(-1，0，0)，(0，-1，0)，(0，0，1)，(0，0，-1)对应的旋转映射的法向量通式分别对应的可通过计算得到： The method to ensure that the overall amount of modification is small is to make the new six standard normal vectors, including the user’s perspective normal vector, correspond to the corresponding six standard methods after the user’s perspective normal vector corresponds to the standard normal vector and is rotated. The sum of the dot products of the vectors is as large as possible. In this case, suppose the general formula of the normal vector of the user's viewing angle
Figure PCTCN2020112857-appb-000011
Set it to the standard normal vector (1, 0, 0) as a rotation mapping relationship, then the other five standard normal vectors (0, 1, 0), (-1, 0, 0), (0, -1 , 0), (0, 0, 1), (0, 0, -1) corresponding to the normal vector general formula of the rotation map can be obtained by calculation:
Figure PCTCN2020112857-appb-000012
Figure PCTCN2020112857-appb-000012
Figure PCTCN2020112857-appb-000013
Figure PCTCN2020112857-appb-000013
之后在进行点云片段划分时，将根据这些新的标准法向量进行点云片段划分，片段投影，视频编码等后续步骤。其中片段投影和视频编码可以采取本发明中所提及的其余两种修改方案。Later, when the point cloud segment is divided, the subsequent steps such as point cloud segmentation, segment projection, and video encoding will be performed according to these new standard normal vectors. Among them, segment projection and video coding can adopt the other two modification schemes mentioned in the present invention.
关于发送端与接收端，针对方法三：修改标准法向量的方法。需要在发送端记录新的标准法向量，这部分信息将作为新增的附加信息进行编码传输；在接收端，需要根据新的标准法向量，进行点云投影的逆过程，从而从二维视频序列重建出三维点云序列。Regarding the sending end and the receiving end, for method three: the method of modifying the standard normal vector. The new standard normal vector needs to be recorded at the sending end, and this part of the information will be encoded and transmitted as additional additional information; at the receiving end, the inverse process of point cloud projection needs to be performed according to the new standard normal vector, so as to start from the two-dimensional video The sequence reconstructs a three-dimensional point cloud sequence.
本发明第一实施例的作用和效果Functions and effects of the first embodiment of the present invention
本发明根据用户视角自适应调整的传输策略，无需对目前现有的点云传 输策略进行较大修改，用户对于各个侧面的关注度也有差异，区别出需要观察到的点云片段和不需要被观察到的点云片段，可以较好的结合目前的高压缩率的编码算法，在不损失用户主要视角的体验质量的同时，不影响压缩率和点云客观质量的情况下，节约传输所消耗的资源，实现了分视角传输的目的。According to the transmission strategy of the present invention adaptively adjusted according to the user's perspective, there is no need to make major modifications to the current existing point cloud transmission strategy, and the user's attention to various aspects is also different, distinguishing between the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed. The observed point cloud fragments can be better combined with the current high compression rate encoding algorithm, without losing the experience quality of the user's main perspective, and without affecting the compression rate and the objective quality of the point cloud, saving transmission consumption The resources to achieve the purpose of sub-view transmission.
{第二实施例}{Second Embodiment}
现有技术方案中，点云序列编码方法的处理步骤，包含：根据法向量划分点云对象为多个点云片段，根据点云片段的连接关系进行片段细划分，将细划分后的点云片段按法向量投影，对投影平面进行空白填充，对投影获得的视频序列使用例如HEVC的二维编码工具来进行压缩编码。In the prior art solution, the processing steps of the point cloud sequence encoding method include: dividing the point cloud object into multiple point cloud segments according to the normal vector, subdividing the segments according to the connection relationship of the point cloud segments, and dividing the subdivided point cloud The segments are projected according to the normal vector, the projection plane is blank filled, and the video sequence obtained by the projection is compressed and encoded using a two-dimensional coding tool such as HEVC.
本发明的实施例中，是在现有点云序列编码传输方案上改进实现的，在三维点云压缩为二维图像的过程中引入用户的观察视角来对点云的投影处理过程进行优化调整，从而提升压缩性能。In the embodiment of the present invention, it is implemented by improving on the existing point cloud sequence coding transmission scheme, and the user’s viewing angle is introduced in the process of compressing the three-dimensional point cloud into a two-dimensional image to optimize and adjust the point cloud projection processing process. Thereby improving compression performance.
综合考量用户的观察视角和片段的法向量两个因素来选择点云片段的投影向量，本实施例提供了一种基于用户视角的点云投影处理方法，包含：包括：依据用户视角区分出点云对象的主片段和多个边缘片段；考虑到边缘片段和主片段之间的空间联系，对边缘片段的投影向量进行保持或基于用户视角进行变更；投影后进行后续点云处理。The projection vector of the point cloud segment is selected by comprehensively considering the two factors of the user’s viewing angle and the normal vector of the segment. This embodiment provides a point cloud projection processing method based on the user’s perspective, including: distinguishing points based on the user’s perspective The main segment and multiple edge segments of the cloud object; considering the spatial connection between the edge segment and the main segment, the projection vector of the edge segment is maintained or changed based on the user's perspective; subsequent point cloud processing is performed after projection.
图5为第二实施例中点云对象的三维边缘盒的立体示意图；5 is a three-dimensional schematic diagram of a three-dimensional edge box of a point cloud object in the second embodiment;
本发明所提出的从特定方向如用户视角方向观察点云对象时，图5所示，利用点云对象三维边缘盒(bounding box)对点云对象进行投影，将点云对象包含其中。通常点云对象三维边缘盒(bounding box)为具体点云对象情况而设定的矩形立方体，具体大小和造型不受本实施例的限制。When observing a point cloud object from a specific direction, such as a user's perspective, as proposed in the present invention, as shown in FIG. 5, the point cloud object is projected by using a three-dimensional bounding box of the point cloud object to include the point cloud object. Generally, a three-dimensional bounding box of a point cloud object is a rectangular cube set for a specific point cloud object, and the specific size and shape are not limited by this embodiment.
图中未显示的，以点云对象作为原点(0，0，0)，首先定义了对应点云对象的6个投影面的标准法向量(x，y，z)：(1，0，0)，(-1，0，0)，(0，1，0)，(0，-1，0)，(0，0，1)，(0，0，-1)。Not shown in the figure, with the point cloud object as the origin (0, 0, 0), first define the standard normal vectors (x, y, z) of the 6 projection surfaces corresponding to the point cloud object: (1, 0, 0) ), (-1, 0, 0), (0, 1, 0), (0, -1, 0), (0, 0, 1), (0, 0, -1).
根据用户观看点云对象时所处的位置，用一个从点云指向用户观察点的向量表示用户的观察的用户视角，称为视角向量。然后计算视角向量与6个标准法向量的距离，与视角向量距离最近的标准法向量称为该观察视角下的主向量。According to the position where the user views the point cloud object, a vector pointing from the point cloud to the user's observation point is used to represent the user's view of the user's observation, which is called the view vector. Then calculate the distance between the viewing angle vector and the six standard normal vectors, and the standard normal vector with the closest distance to the viewing angle vector is called the principal vector under the viewing angle.
通常用6个不同的Index索引值来指示6个不同的法向量，表3是第二实施例中索引参数值和投影向量值的对应关系表(patch index list)，其参数分配和所对应的标准投影向量面与表2相同。Usually 6 different Index index values are used to indicate 6 different normal vectors. Table 3 is a patch index list of index parameter values and projection vector values in the second embodiment, and their parameter assignments and corresponding The standard projection vector surface is the same as Table 2.
本实施例中，利用0指代第一标准法向量的方向，垂直于该(1，0，0)面、利用1指代第六标准法向量的方向，垂直于该(-1，0，0)面、利用2指代第二标准法向量的方向，垂直于该(0，1，0)面、利用3指代第三标准法向量的方向，垂直于该(0，-1，0)面、利用4指代第四标准法向量的方向，垂直于该(0，0，1)面、利用5指代第五标准法向量的方向，垂直于该(0，0，-1)面。In this embodiment, 0 is used to refer to the direction of the first standard normal vector, which is perpendicular to the (1, 0, 0) plane, and 1 is used to refer to the direction of the sixth standard normal vector, which is perpendicular to the (-1, 0, 0) plane, use 2 to refer to the direction of the second standard normal vector, perpendicular to the (0, 1, 0) plane, use 3 to refer to the direction of the third standard normal vector, perpendicular to the (0, -1, 0) ) Plane, using 4 to refer to the direction of the fourth standard normal vector, perpendicular to the (0, 0, 1) plane, and using 5 to refer to the direction of the fifth standard normal vector, perpendicular to the (0, 0, -1) surface.
本实施例中，主向量是第一标准法向量(1，0，0)。In this embodiment, the main vector is the first standard normal vector (1, 0, 0).
可知，六个标准法向量与三维边缘盒的六个面是一一对应的关系，标准法向量垂直于其对应的面。It can be seen that the six standard normal vectors have a one-to-one correspondence with the six faces of the three-dimensional edge box, and the standard normal vectors are perpendicular to their corresponding faces.
索引参数值(Index)Index parameter value (Index) 投影向量面NormalProjection vector surface Normal
00 (1，0，0)(1, 0, 0)
11 (-1，0，0)(-1, 0, 0)
22 (0，1，0)(0, 1, 0)
33 (1，-1，0)(1, -1, 0)
44 (0，0，1)(0, 0, 1)
55 (0，0，-1)(0, 0, -1)
表3table 3
结合点云对象的邻居点信息计算出各点的法向量，然后计算各点的法向量与上述第一标准法向量至第六标准法向量之间的距离，距离最近的标准法向量即为该点对应的标准法向量。通过这一步骤，所有点云对象中的点被划分为6个一级点云片段，通过一级划分，标准法向量相同的点属于同一个一级点云片段。对于6个一级点云片段进行细分，属于同一个一级点云片段且距离较近的点被划分到一个二级点云片段，计算点之间距离的方法可根据具体情况进行选择，通常选用欧式距离。按此方法进行6个一级点云片段的细分，细分后得到的二级点云片段的法向量均与上述6个标准法向量中的一个相对应。Combine the neighbor point information of the point cloud object to calculate the normal vector of each point, and then calculate the distance between the normal vector of each point and the above-mentioned first standard normal vector to the sixth standard normal vector. The closest standard normal vector is the The standard normal vector corresponding to the point. Through this step, the points in all the point cloud objects are divided into 6 first-level point cloud segments. Through the first-level division, the points with the same standard normal vector belong to the same first-level point cloud segment. The 6 first-level point cloud segments are subdivided, and the points that belong to the same first-level point cloud segment and are relatively close are divided into a second-level point cloud segment. The method of calculating the distance between points can be selected according to the specific situation. Euclidean distance is usually used. According to this method, the 6 first-level point cloud segments are subdivided, and the normal vectors of the second-level point cloud segments obtained after subdivision all correspond to one of the above-mentioned 6 standard normal vectors.
本实施例中，将二级点云片段作为待处理云片段继续处理，依据视角向量的方向，将位于六个标准法向量面上的待处理云片段做进一步区分。计算出二级 点云片段的法向量。然后根据用户的观察视角确定投影的主要方向，根据二级点云片段的法向量与主要方向的关系，将二级点云片段分为主片段，边缘片段，无关片段。In this embodiment, the secondary point cloud segment is used as the cloud segment to be processed to continue processing, and the cloud segments to be processed on the six standard normal vector planes are further distinguished according to the direction of the view angle vector. Calculate the normal vector of the secondary point cloud segment. Then the main direction of the projection is determined according to the user's viewing angle. According to the relationship between the normal vector of the secondary point cloud segment and the main direction, the secondary point cloud segment is divided into the main segment, the edge segment, and the irrelevant segment.
图6为第二实施例中在投影优化调整方案下点云片段的投影调整过程图；6 is a diagram of the projection adjustment process of the point cloud segment under the projection optimization adjustment scheme in the second embodiment;
如图6所示，本第二实施例种，将点云片段1至点云片段t根据它们的标准法向量类别投影到六个不同的平面上，与六个标准向量相对应，分别得到第1组点云片段至第6组点云片段，然后根据用户的观察视角确定投影的主要方向，根据二级点云片段的法向量与主要方向的关系，将二级点云片段即这6组点云片段分为主片段，边缘片段，无关片段。本实施例中，将第1组划分为主片段，将第2组划分为无关片段，将第3-6组划分为边缘片段。As shown in Figure 6, in the second embodiment, point cloud segment 1 to point cloud segment t are projected onto six different planes according to their standard normal vector types, corresponding to the six standard vectors, and the first 1 set of point cloud fragments to the 6th group of point cloud fragments, and then determine the main direction of projection according to the user’s viewing angle. According to the relationship between the normal vector of the secondary point cloud fragment and the main direction, the secondary point cloud fragments are these 6 groups Point cloud fragments are divided into main fragments, edge fragments, and irrelevant fragments. In this embodiment, the first group is divided into main segments, the second group is divided into irrelevant segments, and the 3-6 groups are divided into edge segments.
进一步判断了该些边缘片段与主片段之间的空间关联之后，根据优化调整后的投影方向进行投影。如图6所示，主片段和无关片段保持投影方向不变，对边缘片段根据空间关联的大或者不大进行调整，获得投影方向优化调整后的第1组点云片段至第6组点云片段，明显可知，往用户视角方向的第1组点云片段被更加集中。After further determining the spatial association between the edge segments and the main segment, projection is performed according to the optimized and adjusted projection direction. As shown in Figure 6, the main segment and the irrelevant segment keep the projection direction unchanged, adjust the edge segment according to the large or small spatial correlation, and obtain the first group of point cloud segments to the sixth group of point clouds after the projection direction is optimized and adjusted. Fragments, it is obvious that the first group of point cloud fragments towards the user's perspective are more concentrated.
附图7为本发明实施例中不同点云片段的类型的法向量与主向量的关系对应表；附图7给出了不同片段类型的区分标准。Fig. 7 is a corresponding table of the relationship between normal vectors and principal vectors of different point cloud segment types in the embodiment of the present invention; Fig. 7 shows the distinguishing criteria for different segment types.
其中如图7所示，将法向量与主向量相同的二级点云片段定义为主片段，该些主片段也称之为第1标准片段，这些主片段形成主片段集合，第1标准片段集合(即主片段集合)聚集在点云三维边缘盒的主片段投影聚集面上。As shown in Figure 7, the secondary point cloud fragments with the same normal vector and the main vector are defined as the main fragments. These main fragments are also called the first standard fragments. These main fragments form the main fragment set, the first standard fragment The set (ie, the main fragment set) is gathered on the main fragment projection gathering surface of the point cloud 3D edge box.
将法向量与主向量相反的二级点云片段定义为无关片段，该些无关片段也称之为第2标准片段，这些无关片段形成无关片段集合，第二标准片段集合(即无关点云片段集合)聚集在点云三维边缘盒的无关片段投影聚集面上。Define the secondary point cloud fragments whose normal vector is opposite to the main vector as irrelevant fragments. These irrelevant fragments are also called second standard fragments. These irrelevant fragments form a set of irrelevant fragments, and the second standard fragment set (ie, irrelevant point cloud fragments) Set) is gathered on the projection gathering surface of the irrelevant fragments of the 3D edge box of the point cloud.
将法向量与主向量相垂直的二级点云片段定义为边缘片段，该些边缘片段分布在与主向量相垂直的四个边缘片段投影聚集面上，图5中仅明示了四个边缘片段投影聚集面上中的一个面，其他三个面省略标记。The secondary point cloud fragments whose normal vector is perpendicular to the main vector are defined as edge fragments. These edge fragments are distributed on the projection gathering surface of the four edge fragments perpendicular to the main vector. Figure 5 shows only four edge fragments. One of the projection surfaces is projected, and the other three surfaces are not marked.
其中，位于第一边缘片段投影聚集面上的这些二级点云片段，即这些边缘片段形成第一边缘片段的集合，第一边缘片段由第3标准片段而来。以此类推，位于其他三个边缘片段聚集面的这些二级点云片段，分别称之为第2边缘片段的集合、第3边缘片段的集合、第4边缘片段的集合，分别再其聚集面上形成集合， 由第4至6标准片段而来。Among them, these secondary point cloud segments located on the projection gathering surface of the first edge segment, that is, these edge segments form a set of the first edge segment, and the first edge segment is derived from the third standard segment. By analogy, the second-level point cloud segments located on the gathering surface of the other three edge segments are called the second edge segment set, the third edge segment set, and the fourth edge segment set. The above forms a collection, from the 4th to 6th standard fragments.
接下来，主片段的投影向量是以其标准法向量、即主向量进行投影。无关片段的投影向量是以其标准法向量、即与主向量相反进行投影。Next, the projection vector of the main segment is projected with its standard normal vector, that is, the main vector. The projection vector of the irrelevant segment is projected with its standard normal vector, that is, the opposite of the principal vector.
本实施例中与现有技术的重点区别在于，考虑到边缘片段和主片段之间的空间联系，来考量是否需要对上述边缘片段的投影向量做进一步调整优化。The key difference between this embodiment and the prior art is that considering the spatial connection between the edge segment and the main segment, it is necessary to consider whether the projection vector of the above edge segment needs to be further adjusted and optimized.
简言之，空间联系的大小，通过对每个边缘片段依次计算其与各个主片段的交并比(IOU)来判断。若存在IOU大于门限值则说明该边缘点云片段与主点云片段空间联系较大，选择主向量作为该边缘点云片段的投影向量；若不存在IOU大于门限值则说明该边缘点云片段与主点云片段空间联系较小，选择边缘点云片段的法向量作为其投影向量。In short, the size of the spatial connection is judged by sequentially calculating the intersection ratio (IOU) of each edge segment with each main segment. If there is an IOU greater than the threshold, it means that the edge point cloud segment has a greater spatial connection with the main point cloud segment, and the main vector is selected as the projection vector of the edge point cloud segment; if there is no IOU greater than the threshold, it means the edge point The cloud segment has a small spatial connection with the main point cloud segment, and the normal vector of the edge point cloud segment is selected as its projection vector.
图8-1为第二实施例中主片段的三维边缘盒和边缘片段的三维边缘盒用于根据主向量方向投影调整的相对位置示意图；图8-2为附图8-1中主片段和边缘片段的三维边缘盒投影到二维平面后得到的二维投影示意图；对其中一个边缘片段与各个主片段中某一个主片段的交并比(IOU)判断过程，进行具体阐述。Figure 8-1 is a schematic diagram of the relative positions of the three-dimensional edge box of the main segment and the three-dimensional edge box of the edge segment used for projection adjustment according to the main vector direction in the second embodiment; Figure 8-2 is the main segment and the three-dimensional edge box in Figure 8-1 A schematic diagram of the two-dimensional projection obtained after the three-dimensional edge box of the edge segment is projected onto the two-dimensional plane; the IOU judgment process of one of the edge segments and a certain main segment of each main segment is described in detail.
附图8-1为本发明实施例中主片段的三维边缘盒和边缘片段的三维边缘盒用于根据主向量方向进行投影调整的相对位置示意图。实线所示的三维边缘盒(称为边缘盒1)对应主片段，虚线所示三维边缘盒(称为边缘盒2)对应边缘片段。Fig. 8-1 is a schematic diagram of the relative positions of the three-dimensional edge box of the main segment and the three-dimensional edge box of the edge segment for projection adjustment according to the main vector direction in the embodiment of the present invention. The three-dimensional edge box (called edge box 1) shown by the solid line corresponds to the main segment, and the three-dimensional edge box (called edge box 2) shown by the dashed line corresponds to the edge segment.
图8-1和图8-2仅是某一个主片段和某一个边缘片段的三维边缘盒的简要列举说明，表明三维盒之间的投影方位和二维投影图的交叠状况。针对该某一个边缘片段与其他主片段、或者其他边缘片段与其他主片段之间的一一交叠计算空间关联的情况，不再一一罗列展示，仅做示例。Figure 8-1 and Figure 8-2 are only a brief enumeration of the three-dimensional edge boxes of a certain main segment and a certain edge segment, showing the projection orientation between the three-dimensional boxes and the overlap of the two-dimensional projection map. Regarding the case of one-by-one overlapping calculation spatial associations between a certain edge segment and other main segments, or between other edge segments and other main segments, it will not be listed one by one, but only an example.
附图8-2为主片段和边缘片段的三维边缘盒投影到二维平面后得到的矩形的二维投影图。Fig. 8-2 is a rectangular two-dimensional projection diagram obtained after the three-dimensional edge boxes of the main segment and the edge segment are projected onto a two-dimensional plane.
将主片段的三维边缘盒沿着主向量方向投影得到主二维投影图，将边缘片段的三维边缘盒沿着主向量方向投影得到边缘二维投影图。图8-2中，纹格填充部分为主片段的边缘盒1沿主向量方向投影所得(称为矩形区域1)，点填充部分为边缘片段边缘盒2沿主向量方向投影所得(称为矩形区域2)，格点填充部分为矩形区域1和矩形区域2相交叠的公共部分(称为矩形区域 3)。The three-dimensional edge box of the main segment is projected along the main vector direction to obtain the main two-dimensional projection image, and the three-dimensional edge box of the edge segment is projected along the main vector direction to obtain the edge two-dimensional projection image. In Figure 8-2, the pattern filling part is projected along the principal vector direction of the edge box 1 of the main segment (called rectangular area 1), and the dot filling part is projected along the principal vector direction of the edge box 2 of the edge segment (called rectangular). Area 2), the grid point filling part is the common part where the rectangular area 1 and the rectangular area 2 overlap (referred to as the rectangular area 3).
对每个边缘片段，依次计算其与各个主片段的交并比(Intersection over Union,IOU)。具体的计算方式，结合附图1和附图2进行说明。计算公式如下：For each edge segment, calculate its intersection over Union (IOU) with each main segment in turn. The specific calculation method will be described with reference to FIG. 1 and FIG. 2. Calculated as follows:
Figure PCTCN2020112857-appb-000014
Figure PCTCN2020112857-appb-000014
S 1＝(y i1-y i0)×(z i1-z i0)     (2) S 1 =(y i1 -y i0 )×(z i1 -z i0 ) (2)
S 2＝(y j1-y j0)×(z j1-z j0)       (3) S 2 = (y j1 -y j0 )×(z j1 -z j0 ) (3)
S 3＝(y i1-y j0)×(z i1-z j0)     (4) S 3 ＝(y i1 -y j0 )×(z i1 -z j0 ) (4)
S 3对应附图4中矩形区域3的面积，即主二维投影图和边缘二维投影图的相交叠的公共部分， S 3 corresponds to the area of the rectangular area 3 in Figure 4, that is, the overlapping common part of the main two-dimensional projection image and the edge two-dimensional projection image
S 1对应附图4中主二维投影图排除掉公共部分的面积， S 1 corresponds to the main two-dimensional projection map in Figure 4 excluding the area of the common part,
S 2对应附图4中边缘二维投影图排除掉公共部分的面积。 S 2 corresponds to the two-dimensional projection of the edge in Fig. 4 excluding the area of the common part.
如附图8-1的实线三维边缘框所示，A i和B i分别是主片段的三维边缘盒即边缘框1上距离原点O最近和最远的顶点；如附图8-1的虚线三维边缘框(边缘框2)所示，A j和B j分别是边缘片段的三维边缘盒即边缘框2上距离原点O最近和最远的顶点。 As shown in the solid three-dimensional edge box in Figure 8-1, A i and B i are the three-dimensional edge boxes of the main segment, that is, the vertices closest to the origin O on the edge frame 1; as shown in Figure 8-1 As shown by the dashed three-dimensional edge box (edge box 2), A j and B j are respectively the three-dimensional edge box of the edge segment, that is, the vertices of the edge box 2 closest to the origin O and the farthest.
如附图8-2所示，A′ i，A′ j，B′ i，B′ j分别是A i，A j，B i，B j沿主向量方向投影所得的点。它们在以O为原点的XYZ坐标系下的坐标分别为：A i(x i0,y i0,z i0),B i(x i1,y i1,z i1),A j(x j0,y j0,z j0),B j(x j1,y j1,z j1),A′ i(y i0,z i0),B′ i(y i1,z i1),A′ j(y j0,z j0),B′ j(y j1,z j1)。 As shown in the figures 8-2, A 'i, A' j, B 'i, B' j are A i, A j, B i , B j of the resultant vector along the main direction of the projection point. Their coordinates in the XYZ coordinate system with O as the origin are: A i (x i0 ,y i0 ,z i0 ),B i (x i1 ,y i1 ,z i1 ),A j (x j0 ,y j0 ,z j0 ),B j (x j1 ,y j1 ,z j1 ),A′ i (y i0 ,z i0 ),B′ i (y i1 ,z i1 ),A′ j (y j0 ,z j0 ) ,B′ j (y j1 ,z j1 ).
(4)基于二维投影图，针对每个边缘片段，与多个主片段计算得到多个IOU，IOU的个数等于主片段的个数。(4) Based on the two-dimensional projection map, for each edge segment, multiple IOUs are calculated with multiple main segments, and the number of IOUs is equal to the number of main segments.
对于一个边缘片段，将它的各个IOU与一个交并比门限值比较，若存在IOU大于交并比门限值则说明该边缘片段与主片段空间联系较大，为了优化主向量方向的点云视觉质量，改变该边缘片段的投影方向，选择主向量作为该边缘片段的投影向量；若不存在IOU大于交并比门限值则说明该边缘片段与主片段空间联系较小，则保持该边缘片段的法向量作为其投影向量。For an edge segment, compare its respective IOUs with an intersection ratio threshold. If there is an IOU greater than the intersection ratio threshold, it means that the edge segment has a greater spatial connection with the main segment, in order to optimize the point of the main vector direction Cloud visual quality, change the projection direction of the edge segment, and select the main vector as the projection vector of the edge segment; if there is no IOU greater than the intersection ratio threshold, it means that the edge segment has a small spatial connection with the main segment, then keep this The normal vector of the edge segment is used as its projection vector.
可知，边缘片段的最终投影向量由边缘片段的法向量，边缘片段与主片 段的空间位置关系共同决定。It can be seen that the final projection vector of the edge segment is determined by the normal vector of the edge segment and the spatial position relationship between the edge segment and the main segment.
其中，交并比门限值的选择取决于待处理的点云对象的表面特性，例如，物体点云对象和人物点云对象的表面特性不同，那么进行比较时选择的交并比门限值就不同。对于常见的人物点云对象，一般选取门限值为0.2-0.3，进一步可选地，门限值的确定主要通过实验结果的进行调整和选择。Among them, the selection of the intersection ratio threshold depends on the surface characteristics of the point cloud object to be processed. For example, if the surface characteristics of the object point cloud object and the character point cloud object are different, then the intersection ratio threshold value selected when comparing It's different. For common character point cloud objects, the threshold value is generally selected to be 0.2-0.3, and further optionally, the threshold value is determined mainly through adjustment and selection of experimental results.
通过上述过程，对第一个边缘片段完成了投影向量的调整优化或保持，利用上述过程相同的处理，针对每一个边缘片段逐一进行判别调整，直至第2边缘片段的集合、第3边缘片段的集合、第4边缘片段的集合中所有边缘片段均已完成处理。Through the above process, the adjustment, optimization or maintenance of the projection vector is completed for the first edge segment, and the same processing of the above process is used to discriminate and adjust each edge segment one by one, until the second edge segment is set and the third edge segment is set. Set, all edge fragments in the set of 4th edge fragments have been processed.
(5)对于主片段，其投影向量为主向量；对于无关片段，其投影向量为该无关片段的法向量。(5) For the main segment, its projection vector is the main vector; for the irrelevant segment, its projection vector is the normal vector of the irrelevant segment.
经过上述过程，所有二级点云片段即待处理云片段的投影向量均已完成优化调整。在投影后进行后续点云处理中，利用投影关系标识字段标识出：点云对象中的点经过处理调整后的投影向量，包含优化调整后的投影向量坐标值和相一一对应的索引参数值。After the above process, all the secondary point cloud segments, that is, the projection vectors of the cloud segments to be processed, have been optimized and adjusted. In the subsequent point cloud processing after projection, the projection relationship identification field is used to identify: the projection vector of the point in the point cloud object after processing and adjustment, including the optimized and adjusted projection vector coordinate value and the corresponding index parameter value. .
表4是多个边缘片段的投影向量进行调整优化前后的对应关系示例表。Table 4 is an example table of the corresponding relationship before and after adjustment and optimization of the projection vectors of multiple edge segments.
例如表4中，对于第一边缘片段而言，未经过投影调整优化之前，其自身原本的标准法向量：索引参数值2、(0，1，0)，经过上述优化调整后，投影方向变更为主向量，即索引参数值0、(1，0，0)。其他边缘片段的变动与否视每个边缘片段与主片段的空间关联大小而具体决定。简言之，对于第1～k边缘片段，要么调整优化为0，要么保持。For example, in Table 4, for the first edge segment, before the projection adjustment optimization, its original standard normal vector: index parameter value 2, (0, 1, 0), after the above optimization adjustment, the projection direction is changed Main vector, namely index parameter value 0, (1, 0, 0). Whether other edge segments change or not depends on the size of the spatial correlation between each edge segment and the main segment. In short, for the 1st to kth edge segments, either adjust and optimize to 0 or keep.
主片段即：索引参数值0、标准法向量坐标值为(1，0，0)不做优化调整处理，保持主向量进行投影。无关边缘片段即：索引参数值1、标准法向量坐标值为(-1，0，0)不做优化调整处理，保持其原本的标准法向量的方向进行投影。The main segment is: the index parameter value is 0, the standard normal vector coordinate value is (1, 0, 0) without optimization adjustment processing, and the main vector is maintained for projection. Irrelevant edge fragments are: index parameter value 1, standard normal vector coordinate value (-1, 0, 0) without optimization and adjustment processing, keeping the original standard normal vector direction for projection.
类型Types of 原索引参数值Original index parameter value 原投影向量Original projection vector 与主片段空间关联Associate with the main segment space 优化调整后索引参数值Optimized index parameter value after adjustment 优化调整后投影向量Optimize the adjusted projection vector
主片段Main fragment 00 (1，0，0)(1, 0, 0) ———— 00 (1，0，0)(1, 0, 0)
无关片段Irrelevant fragments 11 (-1，0，0)(-1, 0, 0) 保持maintain 11 (-1，0，0)(-1, 0, 0)
第1边缘片段1st edge segment 22 (0，1，0)(0, 1, 0) 大Big 00 (1，0，0)(1, 0, 0)
第2边缘片段2nd edge segment 22 (0，1，0)(0, 1, 0) 不大Not big 22 (0，1，0)(0, 1, 0)
第3边缘片段3rd edge segment 44 (0，0，1)(0, 0, 1) 大Big 00 (1，0，0)(1, 0, 0)
第4边缘片段4th edge segment 55 (0，0，-1)(0, 0, -1) 不大Not big 55 (0，0，-1)(0, 0, -1)
第5边缘片段5th edge segment 33 (0，-1，0)(0, -1, 0) 大Big 00 (1，0，0)(1, 0, 0)
第k边缘片段K-th edge segment …… …… …… …… ……
表4Table 4
继续完成后续投影处理，包含：Continue to complete subsequent projection processing, including:
对主片段和经过投影处理的边缘片段、以及无关片段，分别依据其处理调整后的投影向量进行投影生成可压缩的二维图像，进一步处理得到二维视频序列，使用视频编解码器对二维视频序列进行压缩。For the main segment, the edge segment that has undergone projection processing, and the irrelevant segment, project them according to the adjusted projection vector to generate a compressible two-dimensional image, and further process to obtain a two-dimensional video sequence, and use a video codec for the two-dimensional video sequence. The video sequence is compressed.
附图9为本发明实施例中基于用户视角的点云投影传输方法的流程示意图；基于用户视角的点云投影处理方法的步骤，包含：FIG. 9 is a schematic flowchart of a point cloud projection transmission method based on a user's perspective in an embodiment of the present invention; the steps of a point cloud projection processing method based on a user's perspective include:
输入点云对象；Input point cloud object;
生成点云片段，依次计算各点的法向量；Generate point cloud fragments, and calculate the normal vector of each point in turn;
对各点初步划分，点云对象被划分生成一级点云片段，此时，一级点云片段初步按照六个标准向量的不同而被初步划分；For the preliminary division of each point, the point cloud object is divided to generate first-level point cloud fragments. At this time, the first-level point cloud fragments are preliminarily divided according to the difference of the six standard vectors;
生成二级点云片段，其中，对一级点云片段再做进一步细分，由于同一标准向量的一级点云片段之间向量是相同的，再进一步依据点之间位置远近将一级点云片段中的点再做进一步聚集划分。值得说明的是，这样由一级点云片段生成二级点云片段的处理过程采用已有点云投影步骤中通常技术手段即可完成，二级点云片段作为待处理云片段进行以下步骤。本发明中，获取待处理云片段不限于上述步骤一级点云片段、二级点云片段的这样限制，现有技术中任意点云片段的预处理步骤，均在本发明的包含范围之内。Generate a second-level point cloud segment, where the first-level point cloud segment is further subdivided. Since the vectors between the first-level point cloud segments of the same standard vector are the same, the first-level point is further based on the distance between the points The points in the cloud segment are further clustered and divided. It is worth noting that in this way, the process of generating a secondary point cloud segment from the primary point cloud segment can be completed by using the usual technical means in the existing point cloud projection step, and the secondary point cloud segment is used as a cloud segment to be processed and the following steps are performed. In the present invention, obtaining cloud fragments to be processed is not limited to the above-mentioned steps such as the first-level point cloud fragment and the second-level point cloud fragment. The preprocessing steps of any point cloud fragment in the prior art are all within the scope of the present invention. .
根据用户的观察视角计算主向量，将标准向量中与视角向量最接近的作为主向量；Calculate the main vector according to the user's viewing angle, and use the standard vector that is closest to the angle of view vector as the main vector;
对待处理云片段进行区分，根据二级点云片段的法向量与主向量的关系，将二级点云片段分为主片段、边缘片段、无关片段，其中，法向量与主向量相同的为主片段、法向量与主向量相反的为无关片段、法向量与主向量 垂直的为边缘片段；The cloud fragments to be processed are distinguished. According to the relationship between the normal vector of the secondary point cloud fragment and the main vector, the secondary point cloud fragment is divided into the main fragment, the edge fragment, and the irrelevant fragment. Among them, the normal vector and the main vector are the same. Fragment, normal vector and main vector opposite are irrelevant fragments, normal vector and main vector are perpendicular to edge fragments;
二级点云片段投影方向计算，对二级点云片段中的边缘片段的投影方向进行优化调整，具体优化调整过程如上述所述；The projection direction of the secondary point cloud segment is calculated, and the projection direction of the edge segment in the secondary point cloud segment is optimized and adjusted. The specific optimization and adjustment process is as described above;
投影二级点云片段，对优化调整后的二级点云片段，即主片段、边缘片段、无关片段按照调整后的投影方向进行投影；Project the secondary point cloud segment, and project the optimized and adjusted secondary point cloud segment, that is, the main segment, edge segment, and irrelevant segment according to the adjusted projection direction;
投影后，生成可压缩的二维图像；After projection, a compressible two-dimensional image is generated;
生成二维视频序列；Generate a two-dimensional video sequence;
对二维视频序列进行压缩；以及Compress two-dimensional video sequences; and
最终，将压缩后的二维视频进行码流输出。Finally, the compressed two-dimensional video is output in code stream.
附图10为本发明实施例中基于用户视角的点云接收处理方法的流程示意图；FIG. 10 is a schematic flowchart of a point cloud receiving and processing method based on a user's perspective in an embodiment of the present invention;
本实施例中基于用户视角的点云接收处理方法是上述的基于用户视角的点云投影处理方法的相逆过程，接收端一侧的解压缩为发送端一侧压缩的逆过程。The point cloud receiving and processing method based on the user's perspective in this embodiment is the inverse process of the above-mentioned point cloud projection processing method based on the user's perspective, and the decompression on the receiving end is the inverse process of compression on the transmitting end.
附图10所示，首先输入压缩后的码流；As shown in Figure 10, first input the compressed code stream;
解压缩后生成二维视频序列；Generate a two-dimensional video sequence after decompression;
进而获取二维图像(通常包含三类：几何图，属性图，占用图)；Then obtain a two-dimensional image (usually including three categories: geometric map, attribute map, occupancy map);
利用二维图像信息生成二级点云片段；Use two-dimensional image information to generate secondary point cloud fragments;
然后利用二级点云片段生成一级点云片段；Then use the second-level point cloud segment to generate the first-level point cloud segment;
由一级点云片段结合辅助信息恢复生成点云对象，利用上述投影关系标识字段进行逆投影处理，将二维点云片段恢复到三维的点云对象；The first-level point cloud fragment is combined with auxiliary information to restore and generate a point cloud object, and the above-mentioned projection relationship identification field is used for back projection processing to restore the two-dimensional point cloud fragment to a three-dimensional point cloud object;
最后经过处理，获得完整的点云对象进行输出。Finally, after processing, a complete point cloud object is obtained for output.
另外，本实施例中位于发射端一侧，还提供了一种基于用户视角的点云投影处理系统，附图11为本发明实施例中基于用户视角的点云投影处理系统的示意框图。In addition, in this embodiment, located on the side of the transmitting end, a point cloud projection processing system based on the user's perspective is also provided. FIG. 11 is a schematic block diagram of the point cloud projection processing system based on the user's perspective in an embodiment of the present invention.
如图11所示，基于用户视角的点云投影处理系统包含点云输入模块、片段生成模块、片段划分模块、投影向量调整模块、投影模块以及压缩模块、以及码流输出模块。As shown in FIG. 11, the point cloud projection processing system based on the user's perspective includes a point cloud input module, a segment generation module, a segment division module, a projection vector adjustment module, a projection module and a compression module, and a code stream output module.
点云输入模块，用于输入点云对象；Point cloud input module, used to input point cloud objects;
片段生成模块，用于将点云对象依照预定规则生成待处理云片段，进行后 续的投影优化调整；The fragment generation module is used to generate cloud fragments to be processed from the point cloud object according to predetermined rules, and perform subsequent projection optimization adjustments;
片段划分模块，用于将待处理云片段划分为主片段、边缘片段以及无关片段；Fragment division module, used to divide the cloud fragment to be processed into main fragments, edge fragments and irrelevant fragments;
投影向量调整模块，用于考虑到边缘片段和主片段之间的空间联系，按照上述投影向量调整规则对边缘片段的投影向量进行保持或基于用户视角进行变更；The projection vector adjustment module is used to take into account the spatial connection between the edge segment and the main segment, and to maintain the projection vector of the edge segment according to the above projection vector adjustment rule or change it based on the user's perspective;
投影模块，用于对完成投影调整的主片段、边缘片段以及无关片段进行投影；Projection module, used for projecting the main segment, edge segment and irrelevant segment after projection adjustment;
压缩模块，以及码流输出模块，用于对投影后二维图像进行压缩；以及后续码流输出。The compression module and the code stream output module are used to compress the two-dimensional image after projection; and the subsequent code stream output.
在发送端一侧，投影模块利用优化调整后的投影向量进行投影，通过投影关系标识字段进行标识，该投影关系标识字段包含优化调整后的投影向量的索引参数值，及相对应的投影向量值。On the sender side, the projection module uses the optimized and adjusted projection vector for projection, and is identified by the projection relationship identification field. The projection relationship identification field contains the index parameter value of the optimized and adjusted projection vector and the corresponding projection vector value .
另外，本实施例中位于接收端一侧，还提供了一种基于用户视角的点云接收处理系统。基于用户视角的点云接收处理系统包括：码流处理模块和逆投影模块，码流处理模块，用于码流的输入接收；逆投影模块，用于对二维点云片段进行逆投影，依照投影关系标识字段的指示将二维点云片段恢复为点云对象。投影关系标识字段包含优化调整后的投影向量的索引参数值，及相对应的投影向量值。In addition, in this embodiment, it is located on the side of the receiving end, and a point cloud receiving and processing system based on the user's perspective is also provided. The point cloud receiving and processing system based on the user's perspective includes: a code stream processing module and a back projection module, a code stream processing module, used to receive the input of the code stream; a back projection module, used to back project two-dimensional point cloud fragments, according to The indication of the projection relationship identification field restores the two-dimensional point cloud segment to a point cloud object. The projection relationship identification field contains the index parameter value of the optimized and adjusted projection vector and the corresponding projection vector value.
本发明中，基于用户视角的点云投影处理系统、基于用户视角的点云接收处理系统中各个模块所具有的功能，与上述基于用户视角的点云投影处理方法、基于用户视角的点云接收处理方法中的优化调整过程相对应，可予以类推，那么装置中所具有的结构和技术要素可由生成方法、接收方法相应转换形成，在此省略重复描述，不再赘述。In the present invention, the functions of each module in the point cloud projection processing system based on the user's perspective and the point cloud receiving and processing system based on the user's perspective are the same as the above-mentioned point cloud projection processing method based on the user's perspective and the point cloud receiving based on the user's perspective. The optimization and adjustment process in the processing method corresponds to the corresponding, which can be deduced by analogy. Then the structure and technical elements of the device can be formed by the corresponding conversion of the generation method and the receiving method. The repeated description is omitted here, and the details are not repeated here.
本发明第二实施例的作用和效果Functions and effects of the second embodiment of the present invention
在现有的投影处理方式中，由于未考虑用户视角，当然更没有考虑边缘片段与主片段之间的空间联系上的远近关系，点云对象假如仅仅只按照多个标准向量分为多个点云片段，就会出现这样的状况：In the existing projection processing method, since the user's perspective is not considered, of course, the distance relationship between the edge segment and the main segment is not considered. If the point cloud object is only divided into multiple points according to multiple standard vectors Cloud fragments, there will be such a situation:
同一标准向量对应的点云片段，其所包含的点的空间距离有可能是很大的，这样的简单粗分操作，与用户视角无关，并且同一标准向量对应的点云片段因几何空间上关联不大而显得零碎。换言之，点云对象中空间距离很近 的关联点却因为标准向量的不同，而被分割投影到不同标准向量中。The point cloud segment corresponding to the same standard vector may have a large spatial distance between the points contained. Such a simple rough division operation has nothing to do with the user's perspective, and the point cloud segment corresponding to the same standard vector is geometrically spatially related Small and fragmented. In other words, the associated points in the point cloud object that are very close in space are segmented and projected into different standard vectors because of the difference in standard vectors.
根据本发明的基于用户视角的点云投影处理方法、接收处理方法及系统，将点云对象按照多个标准向量分为多个点云片段时，引入用户视角的因素将多个点云片段进行区分，区分为一个主片段和多个边缘片段。其中，用户对点云对象的正面内容的关注度较高，那么，该主片段是基于用户视角被确定的，另外，对与主片段相关联的边缘片段，例如左右侧面及上下底面的这样的点云片段作为边缘片段，这些边缘片段的投影方向，进一步考虑了这些边缘片段与主片段之间的空间联系上的远近关系，对该边缘片段中的点的投影向量进行必要地调整优化，假如当空间联系大时，则对该边缘片段中的点的投影向量基于用户视角进行变更，使其按照主片段一致的主向量方向进行投影，另外，假如当空间联系小时，则保持该边缘片段中点的投影向量。完成了上述投影向量的优化调整，再依据优化调整后的投影向量，对点云对象的多个点云片段即包含主片段和多个边缘片段分别执行投影操作，和后续点云压缩、编码传输处理等。According to the point cloud projection processing method, receiving processing method and system based on the user's perspective of the present invention, when the point cloud object is divided into multiple point cloud fragments according to multiple standard vectors, the factor of the user's perspective is introduced to perform the multiple point cloud fragments. Distinguish, divided into a main segment and multiple edge segments. Among them, the user has a high degree of attention to the front content of the point cloud object, then the main segment is determined based on the user's perspective. In addition, the edge segments associated with the main segment, such as the left and right sides and the top and bottom sides Point cloud segments are used as edge segments. The projection direction of these edge segments further considers the distance relationship between these edge segments and the main segment. The projection vector of the points in the edge segment is adjusted and optimized. If When the spatial connection is large, the projection vector of the point in the edge segment is changed based on the user's perspective, so that it is projected according to the main vector direction consistent with the main segment. In addition, if the spatial connection is small, keep the edge segment The projection vector of the point. After completing the optimization and adjustment of the above-mentioned projection vector, according to the optimized and adjusted projection vector, perform projection operations on multiple point cloud fragments of the point cloud object, including the main fragment and multiple edge fragments, and the subsequent point cloud compression and encoding transmission Processing etc.
上述对边缘片段的优化调整投影方向由于兼顾用户视角和与主向量的空间关联性，使得边缘片段中与主片段空间关联大的那些点，按照主片段的主向量进行变更投影，更加向用户关注度较高的方向予以集中，且被调整集中的这些点在与关注度较高的主片段在空间上本来也就是高关联、距离近的，避免出现分散零碎状况，也克服了点云对象中空间距离很近的关联点却因为标准向量的不同，而被分割投影到不同标准向量中这一缺陷。The above optimization and adjustment of the projection direction of the edge segment takes into account the user's perspective and the spatial correlation with the main vector, so that those points in the edge segment that are largely related to the main segment are changed and projected according to the main vector of the main segment, which pays more attention to the user. The direction of higher degree of concentration is concentrated, and the adjusted and concentrated points are originally highly related and close in space with the main segment of higher attention. This avoids the scattered fragmentation and overcomes the problem of point cloud objects. The associated points with very close spatial distance are divided and projected into different standard vectors because of the difference in standard vectors.
{第三实施例}{Third Embodiment}
另外，作为第三实施例，本发明还提供了一种基于用户视角的点云投影传输方法、基于用户视角的点云接收处理方法，以及基于用户视角的点云投影传输系统、基于用户视角的点云接收处理系统。In addition, as a third embodiment, the present invention also provides a point cloud projection transmission method based on the user's perspective, a point cloud receiving processing method based on the user's perspective, and a point cloud projection transmission system based on the user's perspective, and a point cloud projection transmission system based on the user's perspective. Point cloud receiving and processing system.
将上述第一实施例和第二实施例进行结合作为本第三实施例的技术方案。本实施例三种，与第一实施例和第二实施例中投影调整优化、多种分视角编码传输的具体内容是重复的，在此省略说明。在该第三实施例中，其第三实施例对应于第一实施例和第二实施例中的内容，可由上述第一实施例和第二实施例分别对应得到。The above-mentioned first embodiment and the second embodiment are combined as the technical solution of the third embodiment. There are three types in this embodiment, and the specific content of the projection adjustment optimization and the multi-view coding transmission in the first and second embodiments are repeated, and the description is omitted here. In the third embodiment, the third embodiment corresponds to the content in the first embodiment and the second embodiment, and can be obtained from the above-mentioned first embodiment and second embodiment respectively.
第三实施例中，基于用户视角的点云投影传输方法，包括如下步骤：In the third embodiment, the point cloud projection transmission method based on the user's perspective includes the following steps:
将点云对象在标准向量上初步划分，得到待处理点云片段，依据用户视角区分出主片段和多个边缘片段；Preliminarily divide the point cloud object on the standard vector to obtain the point cloud segment to be processed, and distinguish the main segment and multiple edge segments according to the user's perspective;
考虑到边缘片段和主片段之间的空间联系，对边缘片段的投影向量进行保持或基于用户视角进行变更；Considering the spatial connection between the edge segment and the main segment, maintain the projection vector of the edge segment or change it based on the user's perspective;
将调整变更后的主片段作为需要观察到的点云片段，调整变更后的边缘片段和无关片段作为不需要观察到的点云片段；以及Take the adjusted and changed main segment as the point cloud segment that needs to be observed, and adjust and change the edge segment and irrelevant segments as the point cloud segment that does not need to be observed; and
对需要观察到的点云片段分配优先资源进行编码传输，对不需要被观察得到的视角的点云片段相较于优先分配次级资源进行编码传输。The point cloud segments that need to be observed are allocated with priority resources for encoding transmission, and the point cloud segments with a perspective that do not need to be observed are allocated for encoding and transmission compared to the priority allocation of secondary resources.
第三实施例中作为发送端的基于用户视角的点云投影传输系统、以及作为接收端的基于用户视角的点云接收处理方法以及基于用户视角的点云投影传输系统，可以由上述基于用户视角的点云投影传输方法类推得到，也分别包含上述第一实施例和第二实施例进行结合的内容，上述第一实施例和第二实施例相重复的内容，在此省略说明。In the third embodiment, the point cloud projection transmission system based on the user's perspective as the sending end, the point cloud receiving processing method based on the user's perspective as the receiving end, and the point cloud projection and transmission system based on the user's perspective as the receiving end can be determined by the point cloud projection transmission system based on the user's perspective. The cloud projection transmission method can be deduced by analogy, and also includes the content of the combination of the above-mentioned first embodiment and the second embodiment respectively, and the overlapping content of the above-mentioned first embodiment and the second embodiment is omitted here.
{第四实施例}{Fourth Embodiment}
本发明对应于上述第一实施例，还提供了一种基于用户视角的点云投影传输方法，其特征在于，包括：将点云对象在标准向量上初步划分，得到待处理点云片段；将待处理点云片段进一步划分得到对应视角类别的点云片段，包含需要观察到的点云片段和不需要观察到的点云片段；对需要观察到的点云片段分配优先资源进行编码传输，对不需要被观察得到的视角的点云片段相较于优先分配次级资源进行编码传输。The present invention corresponds to the above-mentioned first embodiment, and also provides a point cloud projection transmission method based on the user's perspective, which is characterized in that it includes: preliminarily dividing the point cloud object on the standard vector to obtain the point cloud segment to be processed; The point cloud fragments to be processed are further divided to obtain the point cloud fragments of the corresponding perspective category, including the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed; the point cloud fragments that need to be observed are assigned priority resources for encoding transmission, and The point cloud segment of the perspective that does not need to be observed is preferentially allocated to secondary resources for encoding transmission.
第四实施例中作为发送端的基于用户视角的点云投影传输系统、以及作为接收端的基于用户视角的点云接收处理方法以及基于用户视角的点云投影传输系统，可以由上述基于用户视角的点云投影传输方法类推得到，也分别包含上述第一实施例的内容，上述第一实施例相重复的内容，在此省略说明。In the fourth embodiment, the point cloud projection transmission system based on the user's perspective as the sending end, the point cloud receiving processing method based on the user's perspective as the receiving end, and the point cloud projection and transmission system based on the user's perspective as the receiving end can be determined by the point cloud projection transmission system based on the user's perspective. The cloud projection transmission method can be deduced by analogy, and it also includes the content of the above-mentioned first embodiment, and the overlapping content of the above-mentioned first embodiment is omitted here.
{第五实施例}{Fifth Embodiment}
本发明对应于上述第二实施例，还提供了一种基于用户视角的点云投影传输方法，其特征在于，包括：对待处理点云片段，依据用户视角区分出主片段和多个边缘片段；考虑到边缘片段和主片段之间的空间联系，对边缘片段的投影向量进行保持或基于用户视角进行变更；投影后进行后续点云处理。The present invention corresponds to the above-mentioned second embodiment, and also provides a point cloud projection transmission method based on a user's perspective, which is characterized in that it includes: a point cloud segment to be processed, distinguishing a main segment and a plurality of edge segments according to the user's perspective; Considering the spatial connection between the edge segment and the main segment, the projection vector of the edge segment is maintained or changed based on the user's perspective; subsequent point cloud processing is performed after projection.
第五实施例中作为发送端的基于用户视角的点云投影传输系统、以及作为 接收端的基于用户视角的点云接收处理方法以及基于用户视角的点云投影传输系统，可以由上述基于用户视角的点云投影传输方法类推得到，也分别包含上述第二实施例的内容，上述第二实施例相重复的内容，在此省略说明。In the fifth embodiment, the point cloud projection transmission system based on the user's perspective as the sending end, the point cloud receiving processing method based on the user's perspective as the receiving end, and the point cloud projection and transmission system based on the user's perspective as the receiving end can be determined by the point cloud projection transmission system based on the user's perspective. The cloud projection transmission method can be deduced by analogy, and it also includes the content of the above-mentioned second embodiment, and the overlapping content of the above-mentioned second embodiment is omitted here.
在本申请的描述中，需要理解的是，术语“上”、“下”、“前”、“后”、“左”、“右”、“竖直”、“水平”、“顶”、“底”、“内”、“外”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本申请和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本申请的限制。In the description of this application, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", The orientation or positional relationship indicated by "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the application and simplifying the description, rather than indicating or implying the pointed device Or the element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present application.
本领域技术人员知道，除了以纯计算机可读程序代码方式实现本发明提供的系统、装置及其各个模块以外，完全可以通过将方法步骤进行逻辑编程来使得本发明提供的系统、装置及其各个模块以逻辑门、开关、专用集成电路、可编程逻辑控制器以及嵌入式微控制器等的形式来实现相同程序。所以，本发明提供的系统、装置及其各个模块可以被认为是一种硬件部件，而对其内包括的用于实现各种程序的模块也可以视为硬件部件内的结构；也可以将用于实现各种功能的模块视为既可以是实现方法的软件程序又可以是硬件部件内的结构。Those skilled in the art know that, in addition to implementing the system, device and various modules provided by the present invention in a purely computer-readable program code manner, it is completely possible to make the system, device and various modules provided by the present invention by logically programming method steps The same program is implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system, device and various modules provided by the present invention can be regarded as a kind of hardware component, and the modules included in it for implementing various programs can also be regarded as the structure within the hardware component; Modules for realizing various functions can be regarded as both software programs for realizing methods and structures within hardware components.
以上对本发明的具体实施例进行了描述。需要理解的是，本发明并不局限于上述特定实施方式，本领域技术人员可以在权利要求的范围内做出各种变化或修改，这并不影响本发明的实质内容。在不冲突的情况下，本申请的实施例和实施例中的特征可以任意相互组合。The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the case of no conflict, the embodiments of the application and the features in the embodiments can be combined with each other arbitrarily.
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于一计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，所述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)或随机存储记忆体(Random Access Memory，RAM)等。A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The program can be stored in a computer readable storage medium, and the program can be stored in a computer readable storage medium. During execution, it may include the procedures of the above-mentioned method embodiments. Wherein, the storage medium may be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.
本发明实施例提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如本申请发明实施例提供的基于用户视角的点云投影处理方法、基于用户视角的点云接收处理方法。The embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it realizes the point cloud projection processing method based on the user's perspective as provided by the embodiment of the present invention. Point cloud receiving and processing method.
可以采用一个或多个计算机可读的介质的任意组合。计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质。计算机可读存储介质例如可以是 ——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本文件中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。Any combination of one or more computer-readable media may be used. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), Erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.
计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括——但不限于——电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。The computer-readable signal medium may include a data signal propagated in baseband or as a part of a carrier wave, and computer-readable program code is carried therein. This propagated data signal can take many forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium. The computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于——无线、电线、光缆、RF等等，或者上述的任意合适的组合。可以以一种或多种程序设计语言或其组合来编写用于执行本发明操作的计算机程序代码，所述程序设计语言包括面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code used to perform the operations of the present invention can be written in one or more programming languages or a combination thereof. The programming languages include object-oriented programming languages—such as Java, Smalltalk, C++, and also conventional Procedural programming language-such as "C" language or similar programming language. The program code can be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to pass Internet connection).
本发明虽然已以较佳实施例公开如上，但其并不是用来限定本发明，任何本领域技术人员在不脱离本发明的精神和范围内，都可以利用上述揭示的方法和技术内容对本发明技术方案做出可能的变动和修改，因此，凡是未脱离本发明技术方案的内容，依据本发明的技术实质对以上实施例所作的任何简单修改、等同变化及修饰，均属于本发明技术方案的保护范围。Although the present invention has been disclosed as above in preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can use the methods and technical content disclosed above to improve the present invention without departing from the spirit and scope of the present invention. The technical solution makes possible changes and modifications. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention belong to the technical solution of the present invention. protected range.

Claims (30)

  1. 一种基于用户视角的点云投影传输方法，其特征在于，包括：A point cloud projection transmission method based on the user's perspective, which is characterized in that it includes:
    将点云对象在标准向量上初步划分，得到待处理点云片段；Preliminarily divide the point cloud object on the standard vector to obtain the point cloud segment to be processed;
    基于用户视角确定投影传输方案，对待处理点云片段进行处理。The projection transmission scheme is determined based on the user's perspective, and the point cloud segment to be processed is processed.
  2. 根据权利要求1所述的基于用户视角的点云投影传输方法，其特征在于：The point cloud projection transmission method based on the user's perspective according to claim 1, characterized in that:
    取距离最近的标准法向量为点云对象中点的划分类别；Take the nearest standard normal vector as the division category of the midpoint of the point cloud object;
    基于六个标准法向量将点云对象中点分别划分为多个一级点云片段；Divide the midpoint of the point cloud object into multiple first-level point cloud fragments based on six standard normal vectors;
    将同一一级点云片段中，同一法向量且距离小于预定值的点再划分为二级点云片段作为待处理云片段。In the same first-level point cloud segment, points with the same normal vector and a distance less than a predetermined value are further divided into second-level point cloud segments as cloud segments to be processed.
  3. 根据权利要求1所述的基于用户视角的点云投影传输方法，其特征在于，其中，投影传输方案包括：The point cloud projection transmission method based on the user's perspective according to claim 1, wherein the projection transmission scheme comprises:
    将待处理点云片段进一步划分得到对应视角类别的点云片段，包含需要观察到的点云片段和不需要观察到的点云片段；The point cloud fragments to be processed are further divided to obtain the point cloud fragments of the corresponding viewing angle category, including the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed;
    对需要观察到的点云片段分配优先资源进行编码传输，对不需要被观察得到的视角的点云片段相较于优先分配次级资源进行编码传输。The point cloud segments that need to be observed are allocated with priority resources for encoding transmission, and the point cloud segments with a perspective that do not need to be observed are allocated for encoding and transmission compared to the priority allocation of secondary resources.
  4. 根据权利要求1所述的基于用户视角的点云投影传输方法，其特征在于，所述投影传输方案包括以下任一种或任多种：The point cloud projection transmission method based on the user's perspective according to claim 1, wherein the projection transmission scheme includes any one or more of the following:
    不修改点云投影方案、改进点云投影方案、根据用户视角来修改标准法向量方案。Do not modify the point cloud projection scheme, improve the point cloud projection scheme, and modify the standard normal vector scheme according to the user's perspective.
  5. 根据权利要求4所述的基于用户视角的点云投影传输方法，其特征在于，所述不修改点云投影方法的方案中，包括：The point cloud projection transmission method based on the user's perspective according to claim 4, wherein the solution of not modifying the point cloud projection method comprises:
    将点云对象在标准向量上初步划分，通过参数列表与作为投影向量的标准向量相关联，得到待处理点云片段，The point cloud object is preliminarily divided on the standard vector, and the parameter list is associated with the standard vector as the projection vector to obtain the point cloud fragment to be processed,
    考虑用户视角和投影向量的方向，将待处理点云片段分为需要观察到的点云片段和不需要观察到的点云片段。Considering the user's perspective and the direction of the projection vector, the point cloud segment to be processed is divided into the point cloud segment that needs to be observed and the point cloud segment that does not need to be observed.
  6. 根据权利要求5所述的基于用户视角的点云投影传输方法，其特征在于，The point cloud projection transmission method based on the user's perspective according to claim 5, characterized in that,
    对需要观察到的点云片段和不需要观察到的点云片段，设置相对应的编解码参数，Set the corresponding codec parameters for the point cloud segments that need to be observed and the point cloud segments that do not need to be observed,
    记录每个点云片段的编解码参数，作为新增附加信息用于编码传输。Record the coding and decoding parameters of each point cloud segment, as new additional information for coding transmission.
  7. 根据权利要求4所述的基于用户视角的点云投影传输方法，其特征在于，所述改进点云投影方案中，包括：The point cloud projection transmission method based on the user's perspective according to claim 4, wherein the improved point cloud projection solution includes:
    将待处理点云片段进一步划分得到对应视角类别的点云片段，包含需要观察到的点云片段和不需要观察到的点云片段；The point cloud fragments to be processed are further divided to obtain the point cloud fragments of the corresponding viewing angle category, including the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed;
    将需要观察到的点云片段和不需要观察到的点云片段，投影到投影平面上，根据需要更改投影平面的尺寸，得到需要观察图片和不需要观察图片。Project the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed on the projection plane, and change the size of the projection plane as needed to obtain the pictures that need to be observed and the pictures that do not need to be observed.
  8. 根据权利要求7所述的基于用户视角的点云投影传输方法，其特征在于，进一步包括：The point cloud projection transmission method based on the user's perspective according to claim 7, characterized in that it further comprises:
    作为第一改进点云投影方案，在需要观察到的点云片段和不需要观察到的点云片段之间错位前提下将不需要观察图片拼合入需要观察图片中，得到一张投影图片，进行压缩编码。As the first improved point cloud projection solution, under the premise of dislocation between the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed, the pictures that do not need to be observed are merged into the pictures that need to be observed, and a projected picture is obtained. Compression coding.
  9. 根据权利要求8所述的基于用户视角的点云投影传输方法，其特征在于，所得到的一张投影图片形成一组视频序列，The point cloud projection transmission method based on the user's perspective according to claim 8, wherein the obtained one projection picture forms a group of video sequences,
    记录不需要观察到的点云片段对应的索引参数index以及尺寸更改的比例，作为新增附加信息用于编码传输。Record the index parameter index corresponding to the point cloud segment that does not need to be observed and the scale of the size change, as new additional information for encoding transmission.
  10. 根据权利要求7所述的基于用户视角的点云投影传输方法，其特征在于，进一步包括：The point cloud projection transmission method based on the user's perspective according to claim 7, characterized in that it further comprises:
    作为第二改进点云投影方案，将需要观察图片和不需要观察图片分别进行压缩编码。As the second improved point cloud projection solution, the need to observe pictures and the unneeded pictures are compressed and coded separately.
  11. 根据权利要求10所述的基于用户视角的点云投影传输方法，其特征在 于，The point cloud projection transmission method based on the user's perspective according to claim 10, characterized in that:
    投影到的需要观察图片和不需要观察图片分别形成两组视频序列，编码两组视频序列，The projected pictures that need to be observed and the pictures that do not need to be observed form two sets of video sequences respectively, and the two sets of video sequences are encoded,
    记录不需要观察到的点云片段对应的视频序列的更改比例，作为新增附加信息进行编码传输，进一步记录各组视频序列在编码后的码流中位置。Record the change ratio of the video sequence corresponding to the point cloud segment that does not need to be observed, and encode it for transmission as newly added additional information, and further record the position of each group of video sequences in the coded stream.
  12. 根据权利要求4所述的基于用户视角的点云投影传输方法，其特征在于，第三改进点云投影方案，包括：The point cloud projection transmission method based on the user's perspective according to claim 4, wherein the third improved point cloud projection solution includes:
    将待处理点云片段依照六个标准法向量分组，分别投影到六个投影平面上，对六个投影平面基于用户视角分别根据需求更改尺寸，分别进行压缩编码。The point cloud fragments to be processed are grouped according to six standard normal vectors and projected onto six projection planes respectively. Based on the user's perspective, the size of the six projection planes is changed according to requirements, and compression coding is performed respectively.
  13. 根据权利要求12所述的基于用户视角的点云投影传输方法，其特征在于，The point cloud projection transmission method based on the user's perspective according to claim 12, characterized in that,
    将各个标准法向量对应的点云片段分别投影到各组视频序列，编码六组视频序列，Project the point cloud segments corresponding to each standard normal vector to each group of video sequences, and encode six groups of video sequences,
    记录不需要观察到的点云片段对应的视频序列的更改比例，作为新的附加信息进行编码传输，进一步记录各组视频序列在编码后的码流中的位置。Record the change ratio of the video sequence corresponding to the point cloud segment that does not need to be observed, and encode and transmit it as new additional information, and further record the position of each group of video sequences in the coded stream.
  14. 根据权利要求4所述的基于用户视角的点云投影传输方法，其特征在于，所述根据用户视角来修改标准法向量方案中，包括：The point cloud projection transmission method based on the user's perspective according to claim 4, wherein the scheme for modifying the standard normal vector according to the user's perspective includes:
    根据用户观测视角确定视角向量；Determine the angle of view vector according to the user's observation angle;
    将该视角向量与其中一个标准法向量进行旋转映射，然后将其他的标准法向量进行同样的旋转映射处理，在总体修改量小于预定值的前提下，确定旋转映射后的投影方向。Perform rotation mapping on the view vector and one of the standard normal vectors, and then perform the same rotation mapping processing on the other standard normal vectors, and determine the projection direction after the rotation mapping on the premise that the overall modification amount is less than a predetermined value.
  15. 根据权利要求14所述的基于用户视角的点云投影传输方法，其特征在于：The point cloud projection transmission method based on the user's perspective according to claim 14, characterized in that:
    其中，将总体修改量小于预定值的步骤包括：在视角向量与标准法向量对应，并进行旋转之后，使包括视角向量在内的旋转映射后的六个标准法向量与对应原有的六个标准法向量的点乘积的和在预定范围之内取最大值或 较大值。Among them, the step of making the overall modification amount less than a predetermined value includes: after the view angle vector corresponds to the standard normal vector and rotates, the six standard normal vectors after the rotation mapping including the view angle vector correspond to the original six The sum of the dot products of the standard normal vector takes the maximum value or a larger value within a predetermined range.
  16. 根据权利要求15所述的基于用户视角的点云投影传输方法，其特征在于，The point cloud projection transmission method based on the user's perspective according to claim 15, characterized in that,
    将旋转映射调整后的标准法向量，将作为新增的附加信息进行编码传输。The standard normal vector after the rotation mapping is adjusted will be encoded and transmitted as newly added additional information.
  17. 根据权利要求1所述的基于用户视角的点云投影传输方法，其特征在于，包括：The point cloud projection transmission method based on the user's perspective according to claim 1, characterized in that it comprises:
    其中，投影传输方案包括：Among them, the projection transmission scheme includes:
    对待处理点云片段，依据用户视角区分出主片段和多个边缘片段；For the point cloud segment to be processed, distinguish the main segment and multiple edge segments according to the user's perspective;
    考虑到边缘片段和主片段之间的空间联系，对边缘片段的投影向量进行保持或基于用户视角进行变更；Considering the spatial connection between the edge segment and the main segment, maintain the projection vector of the edge segment or change it based on the user's perspective;
    投影后进行后续点云处理。After projection, the subsequent point cloud processing is performed.
  18. 根据权利要求17所述的基于用户视角的点云投影传输方法，其特征在于：The point cloud projection transmission method based on the user's perspective according to claim 17, characterized in that:
    其中，考虑到边缘片段和主片段之间的空间联系的判断依据，包括：Among them, taking into account the judgment basis of the spatial connection between the edge segment and the main segment, including:
    依据用户视角在标准法向量中确定主向量；Determine the main vector in the standard normal vector according to the user's perspective;
    针对主片段和边缘片段分别沿着主向量方向投影得到主二维投影图、以及边缘二维投影图；The main segment and the edge segment are respectively projected along the main vector direction to obtain the main two-dimensional projection map and the edge two-dimensional projection map;
    基于每个边缘二维投影图，分别与各个主二维投影图计算得到交并比；Based on the two-dimensional projection map of each edge, calculate the intersection and ratio with each main two-dimensional projection map respectively;
    多个交并比与预设门限值进行比较作为空间联系的判断依据。The multiple intersection ratios are compared with the preset threshold value as the basis for judging the spatial connection.
  19. 根据权利要求18所述的基于用户视角的点云投影传输方法，其特征在于：The point cloud projection transmission method based on the user's perspective according to claim 18, characterized in that:
    其中，多个交并比中存在大于预设门限值的情况下，将边缘片段的投影向量变更为主向量；多个交并比中均不存在大于预设门限值的情况下，保持该边缘片段的投影向量。Among them, if there are multiple intersection ratios greater than the preset threshold, change the projection vector of the edge segment to the main vector; if none of the multiple intersection ratios is greater than the preset threshold, keep The projection vector of this edge segment.
  20. 根据权利要求19所述的基于用户视角的点云投影传输方法，其特征在于：The point cloud projection transmission method based on the user's perspective according to claim 19, characterized in that:
    其中，预设门限值取决于点云对象的表面特性，区分物体点云对象的表面特性和人物点云对象的表面特性，Among them, the preset threshold value depends on the surface characteristics of the point cloud object, distinguishing the surface characteristics of the object point cloud object and the surface characteristics of the character point cloud object.
    门限值的确定可进一步依据实验结果进行调整确定。The determination of the threshold value can be further adjusted and determined based on the experimental results.
  21. 根据权利要求17所述的基于用户视角的点云投影传输方法，其特征在于，The point cloud projection transmission method based on the user's perspective according to claim 17, characterized in that,
    根据用户观看点云对象时所处的位置，用一个从点云指向用户观察点的向量表示用户的观察视角，称为视角向量，According to the position of the user when viewing the point cloud object, a vector from the point cloud to the user’s observation point is used to represent the user’s observation perspective, which is called the perspective vector.
    计算视角向量与六个标准法向量的距离，与视角向量距离最近的标准法向量称为该观察视角下的主向量。Calculate the distance between the viewing angle vector and the six standard normal vectors, and the standard normal vector with the closest distance to the viewing angle vector is called the principal vector under the viewing angle.
  22. 根据权利要求17所述的基于用户视角的点云投影传输方法，其特征在于，The point cloud projection transmission method based on the user's perspective according to claim 17, characterized in that,
    除了依据用户视角区分出点云对象的主片段和多个边缘片段，还包括区分出无关片段，In addition to distinguishing the main segment and multiple edge segments of the point cloud object according to the user’s perspective, it also includes distinguishing irrelevant segments.
    其中，将与用户视角距离最接近的标准法向量确定为主向量；Among them, the standard normal vector closest to the user's viewing angle is determined as the main vector;
    将多个标准法向量中与该主向量相同的待处理点云片段确定为主片段；Determine the point cloud segment to be processed that is the same as the main vector among the multiple standard normal vectors as the main segment;
    将多个标准法向量中与该主向量相反的待处理点云片段确定为无关片段；Determine the to-be-processed point cloud segment opposite to the main vector among the multiple standard normal vectors as irrelevant segments;
    将多个标准法向量中与该主向量相关联的待处理点云片段确定为边缘片段。The to-be-processed point cloud segment associated with the main vector among the multiple standard normal vectors is determined as an edge segment.
  23. 根据权利要求1所述的基于用户视角的点云投影传输方法，其特征在于：The point cloud projection transmission method based on the user's perspective according to claim 1, characterized in that:
    投影后进行后续点云处理中，利用投影关系标识字段标识出，点云对象中的点经过处理调整后的投影向量，包含点云片段索引值和相对应的投影向量值。In the subsequent point cloud processing after projection, the projection relationship identification field is used to identify the projection vector of the point in the point cloud object after processing and adjustment, including the point cloud fragment index value and the corresponding projection vector value.
  24. 一种基于用户视角的点云接收处理方法，其特征在于：A point cloud receiving and processing method based on the user's perspective, which is characterized in:
    利用用户视角而确定的编解码参数对接收到的码流得到视频序列，和对视频序列处理得到点云片段，以及对每个点云片段进行解码重建得到点云 对象。Use the codec parameters determined by the user's perspective to obtain a video sequence from the received bitstream, process the video sequence to obtain a point cloud segment, and decode and reconstruct each point cloud segment to obtain a point cloud object.
  25. 根据权利要求24所述的基于用户视角的点云接收处理方法，其特征在于，The method for receiving and processing a point cloud based on a user's perspective according to claim 24, wherein:
    根据编解码参数对每个点云片段进行编解码和后续重建的过程，包含以下任意一种或者几种：The process of encoding, decoding and subsequent reconstruction of each point cloud segment according to the encoding and decoding parameters includes any one or more of the following:
    所述编解码参数在发送端针对每个点云片段，区分需要观察到的点云片段和不需要观察被设置，进行后续的点云对象的重建；或者The coding and decoding parameters are set at the sending end for each point cloud segment to distinguish between the point cloud segment that needs to be observed and the point cloud segment that does not need to be observed, and the subsequent reconstruction of the point cloud object is performed; or
    需要根据所接收到的参数信息以及尺寸更改信息来对点云片段恢复为原始尺寸，再进行后续的点云对象的重建；或者It is necessary to restore the point cloud segment to its original size according to the received parameter information and size change information, and then perform subsequent point cloud object reconstruction; or
    根据各视频序列在码流中的位置恢复出各视频序列，根据所接收到的更改尺寸来将这些视频序列恢复为原始尺寸，并需要将多组视频序列所恢复出来的点云进行拼接，进行点云对象的重建；或者According to the position of each video sequence in the code stream, each video sequence is restored, and these video sequences are restored to the original size according to the received change size, and the point clouds recovered from multiple sets of video sequences need to be spliced. Reconstruction of point cloud objects; or
    需要根据发送端旋转映射调整后的标准法向量，进行点云投影的逆过程，从而从二维视频序列重建出三维点云序列，进行点云对象的重建。It is necessary to perform the inverse process of point cloud projection according to the standard normal vector adjusted by the rotation mapping of the transmitting end, so as to reconstruct a three-dimensional point cloud sequence from the two-dimensional video sequence and perform the reconstruction of the point cloud object.
  26. 一种基于用户视角的点云投影传输系统，其特征在于：A point cloud projection transmission system based on the user's perspective is characterized by:
    片段生成模块，用于将点云对象在标准向量上初步划分，得到待处理点云片段；The fragment generation module is used to preliminarily divide the point cloud object on the standard vector to obtain the point cloud fragment to be processed;
    投影编码处理模块，基于用户视角确定投影传输方案，对待处理点云片段进行处理。The projection coding processing module determines the projection transmission scheme based on the user's perspective, and processes the point cloud segment to be processed.
  27. 一种基于用户视角的点云接收处理系统，其特征在于：A point cloud receiving and processing system based on the user's perspective is characterized by:
    逆处理模块，利用用户视角而确定的编解码参数对接收到的码流得到视频序列，和对视频序列处理得到点云片段，以及对每个点云片段进行解码重建得到点云对象。The inverse processing module uses the codec parameters determined from the user's perspective to obtain a video sequence from the received bitstream, process the video sequence to obtain a point cloud segment, and decode and reconstruct each point cloud segment to obtain a point cloud object.
  28. 一种基于用户视角的点云投影传输方法，其特征在于，包括：A point cloud projection transmission method based on the user's perspective, which is characterized in that it includes:
    将点云对象在标准向量上初步划分，得到待处理点云片段；Preliminarily divide the point cloud object on the standard vector to obtain the point cloud segment to be processed;
    将待处理点云片段进一步划分得到对应视角类别的点云片段，包含需要观 察到的点云片段和不需要观察到的点云片段；The point cloud fragments to be processed are further divided to obtain the point cloud fragments of the corresponding viewing angle category, including the point cloud fragments that need to be observed and the point cloud fragments that do not need to be observed;
    对需要观察到的点云片段分配优先资源进行编码传输，对不需要被观察得到的视角的点云片段相较于优先分配次级资源进行编码传输The point cloud fragments that need to be observed are allocated with priority resources for encoding transmission, and the point cloud fragments that do not need to be observed are compared with the priority allocation of secondary resources for encoding transmission.
  29. 一种基于用户视角的点云投影传输方法，其特征在于，包括：A point cloud projection transmission method based on the user's perspective, which is characterized in that it includes:
    对待处理点云片段，依据用户视角区分出主片段和多个边缘片段；For the point cloud segment to be processed, distinguish the main segment and multiple edge segments according to the user's perspective;
    考虑到边缘片段和主片段之间的空间联系，对边缘片段的投影向量进行保持或基于用户视角进行变更；Considering the spatial connection between the edge segment and the main segment, maintain the projection vector of the edge segment or change it based on the user's perspective;
    投影后进行后续点云处理。After projection, the subsequent point cloud processing is performed.
  30. 一种基于用户视角的点云投影传输方法，其特征在于，包括：A point cloud projection transmission method based on the user's perspective, which is characterized in that it includes:
    将点云对象在标准向量上初步划分，得到待处理点云片段，依据用户视角区分出主片段和多个边缘片段；Preliminarily divide the point cloud object on the standard vector to obtain the point cloud segment to be processed, and distinguish the main segment and multiple edge segments according to the user's perspective;
    考虑到边缘片段和主片段之间的空间联系，对边缘片段的投影向量进行保持或基于用户视角进行变更；Considering the spatial connection between the edge segment and the main segment, maintain the projection vector of the edge segment or change it based on the user's perspective;
    将调整变更后的主片段作为需要观察到的点云片段，调整变更后的边缘片段和无关片段作为不需要观察到的点云片段；以及Take the adjusted and changed main segment as the point cloud segment that needs to be observed, and adjust and change the edge segment and irrelevant segments as the point cloud segment that does not need to be observed; and
    对需要观察到的点云片段分配优先资源进行编码传输，对不需要被观察得到的视角的点云片段相较于优先分配次级资源进行编码传输。The point cloud segments that need to be observed are allocated with priority resources for encoding transmission, and the point cloud segments with a perspective that do not need to be observed are allocated for encoding and transmission compared to the priority allocation of secondary resources.
PCT/CN2020/112857 2019-09-03 2020-09-01 Point cloud projection transmission method, and reception processing method and system WO2021043136A1 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN201910827750.9A CN112449171B (en) 2019-09-03 2019-09-03 Encoding method, system and medium for point cloud view-division transmission
CN201910827750.9 2019-09-03
CN202010863785.0 2020-08-25
CN202010863785.0A CN114119346A (en) 2020-08-25 2020-08-25 Point cloud projection processing method and system based on user view angle and receiving processing method

Publications (1)

Publication Number Publication Date
WO2021043136A1 true WO2021043136A1 (en) 2021-03-11

Family

ID=74852153

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/112857 WO2021043136A1 (en) 2019-09-03 2020-09-01 Point cloud projection transmission method, and reception processing method and system

Country Status (1)

Country Link
WO (1) WO2021043136A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190087979A1 (en) * 2017-09-18 2019-03-21 Apple Inc. Point cloud compression
CN109523581A (en) * 2017-09-19 2019-03-26 华为技术有限公司 A kind of method and apparatus of three-dimensional point cloud alignment
CN110012279A (en) * 2018-01-05 2019-07-12 上海交通大学 Divide visual angle compression and transmission method and system based on 3D point cloud data
CN110070613A (en) * 2019-04-26 2019-07-30 东北大学 Large-scale three dimensional scene web page display method based on model compression and asynchronous load
CN110070608A (en) * 2019-04-11 2019-07-30 浙江工业大学 A method of being automatically deleted the three-dimensional reconstruction redundant points based on image

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190087979A1 (en) * 2017-09-18 2019-03-21 Apple Inc. Point cloud compression
CN109523581A (en) * 2017-09-19 2019-03-26 华为技术有限公司 A kind of method and apparatus of three-dimensional point cloud alignment
CN110012279A (en) * 2018-01-05 2019-07-12 上海交通大学 Divide visual angle compression and transmission method and system based on 3D point cloud data
CN110070608A (en) * 2019-04-11 2019-07-30 浙江工业大学 A method of being automatically deleted the three-dimensional reconstruction redundant points based on image
CN110070613A (en) * 2019-04-26 2019-07-30 东北大学 Large-scale three dimensional scene web page display method based on model compression and asynchronous load

Similar Documents

Publication Publication Date Title
US20210174551A1 (en) Mesh compression via point cloud representation
US11010955B2 (en) Point cloud mapping
US20220116659A1 (en) A method, an apparatus and a computer program product for volumetric video
CN117768653A (en) Method and apparatus for encoding and decoding volumetric video
CN108027983A (en) For the generation of the triangular mesh of 3-D view
US20210006806A1 (en) An apparatus, a method and a computer program for volumetric video
CN107454468A (en) The method, apparatus and stream being formatted to immersion video
CN103828359A (en) Representation and coding of multi-view images using tapestry encoding
JP2023514853A (en) projection-based mesh compression
KR101933037B1 (en) Apparatus for reproducing 360 degrees video images for virtual reality
CN112449171B (en) Encoding method, system and medium for point cloud view-division transmission
US20200236401A1 (en) Point cloud coding using homography transform
JP7344988B2 (en) Methods, apparatus, and computer program products for volumetric video encoding and decoding
Zhu et al. An improved depth image based virtual view synthesis method for interactive 3D video
KR102491674B1 (en) Method and apparatus for generating virtual viewpoint image
CN103905812A (en) Texture/depth combination up-sampling method
US20210092345A1 (en) Unified coding of 3d objects and scenes
WO2021043136A1 (en) Point cloud projection transmission method, and reception processing method and system
WO2023272510A1 (en) Methods and apparatuses for generating, performing data processing on, coding and decoding multiplane image
Lin et al. Fast multi-view image rendering method based on reverse search for matching
EP4133719A1 (en) A method, an apparatus and a computer program product for volumetric video coding
JP2023541271A (en) High density mesh compression
WO2019185983A1 (en) A method, an apparatus and a computer program product for encoding and decoding digital volumetric video
CN116723305B (en) Virtual viewpoint quality enhancement method based on generation type countermeasure network
US20230306683A1 (en) Mesh patch sub-division

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20860252

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20860252

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20860252

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 19-05-2023)

122 Ep: pct application non-entry in european phase

Ref document number: 20860252

Country of ref document: EP

Kind code of ref document: A1