CN113129393B - Point cloud data processing method and system - Google Patents
Point cloud data processing method and system Download PDFInfo
- Publication number
- CN113129393B CN113129393B CN202010040455.1A CN202010040455A CN113129393B CN 113129393 B CN113129393 B CN 113129393B CN 202010040455 A CN202010040455 A CN 202010040455A CN 113129393 B CN113129393 B CN 113129393B
- Authority
- CN
- China
- Prior art keywords
- point cloud
- priority
- information
- content
- decoding
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/06—Notations for structuring of protocol data, e.g. abstract syntax notation one [ASN.1]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a point cloud data processing method and a point cloud data processing system, wherein the point cloud data processing method comprises the following steps: step M1: performing segmentation processing on the point cloud data to obtain a preset number of point cloud contents, and performing classification processing according to different obtained preset number of point cloud contents; step M2: carrying out priority setting processing on the point cloud contents after the classification processing; step M3: setting a priority order according to a priority setting process; step M4: and decoding the point cloud data according to the set priority order. The invention meets the diversity of the decoding end equipment, the complexity of network conditions and the individual requirements of point cloud data users, and is suitable for various environments and requirements.
Description
Technical Field
The invention relates to the field of point cloud data processing, in particular to a point cloud data processing method and system. More specifically, the packaging and transmission structure of the point cloud data is designed to be suitable for consumption, and is used for processing the point cloud data according to requirements during transmission and decoding.
Background
In recent decades, as computing and sensing devices become mature, the construction cost of image acquisition devices is reduced, and image processing algorithms and systems are developed and iterated gradually, so that the capability of acquiring actual object surface data is stronger, the original two-dimensional information of object acquisition data is converted into three-dimensional coordinate information and even higher dimensionality including attribute information, and point cloud is widely applied to related fields of academic and industrial circles as a more accurate data type.
The point cloud data is a set of a series of points obtained by scanning an object or sampling an original 3D model by using a sensing device, and records information such as geometry and attributes of the surface of the object. Due to the characteristics of high dimensionality, large data size and the like, the device is heavily burdened when computing operations such as transmission, presentation and the like and storage are carried out.
Some current point cloud compression methods process the position information and attribute information of a point cloud based on a geometric structure of a midpoint of the point cloud, and also have point cloud compression schemes based on projection, and the like. The schemes can reduce the data volume needing to be transmitted to a certain extent and have better reconstruction effect. In these point cloud compression schemes, the processed point cloud data is usually converted into a code stream and divided into a sequence parameter stream, a geometric parameter stream, an attribute parameter stream, a geometric information data stream, an attribute information data stream, and some necessary indication information data streams; the parameter stream is used for indicating the relationship between data information, necessary information in decoding and the like, and the data stream contains information such as the position, the attribute and the like of the point cloud after being coded; the geometric information records spatial position information of the point cloud, and most importantly, decoding can be completed only by means of sequence parameters and geometric parameters, and in some compression schemes, for example, in point cloud compression based on geometry, the attribute information can be decoded only by means of the geometric information on the basis of a parameter set. The point cloud information such as geometry, attribute and the like can be decoded and presented by virtue of the parameter set content according to a certain sequence. However, this solution does not take into account the diversity of the decoding devices, the complexity of the network conditions and the personalized requirements of the users of the point cloud data.
In summary, in the prior art, in the processing process of the point cloud data information stream, especially in the decoding process of the decoding end, only a single decoding structure exists, and the diversity of the network condition, the application scenario, the consumer demand and the like of the decoding end is not considered, all point clouds are encoded in the same manner and then all decoded, all scenarios are in the same decoding order, the data processing performance is poor, the presentation effect is poor, the use experience of the point cloud media by the consumer end is influenced, and certain limitations exist.
How to solve the problem among the prior art, how to carry out the structural design of encapsulation and transmission to the code stream information after the point cloud compression, make it adapt to all kinds of environment and demands, be the key problem that awaits solution urgently.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a point cloud data processing method and system.
The invention provides a point cloud data processing method, which comprises the following steps:
step M1: performing segmentation processing on the point cloud data to obtain a preset number of point cloud contents, and performing classification processing according to different obtained preset number of point cloud contents;
step M2: carrying out priority setting processing on the point cloud contents after the classification processing;
step M3: setting a priority order according to a priority setting process;
step M4: and decoding the point cloud data according to the set priority order.
Preferably, the step M2 includes:
step M2.1: encoding the point cloud content obtained after classification;
step M2.2: and packaging the encoded point cloud content according to a packaging format defined by a packaging protocol, and defining a priority identifier in the packaged point cloud content.
Preferably, the step M3 includes:
step M3.1: whether the packaged point cloud contents are grouped is judged according to the segmentation and/or classification results, and when the point cloud contents are grouped, the point cloud contents are grouped according to data obtained by segmenting the point cloud data, and the type and/or index and priority order of the grouping are set; when the point cloud data are not grouped, setting and processing a priority order according to the priority of the point cloud content;
step M3.2: and setting transmission signaling of information for describing the length or priority of each point cloud content in the point cloud data.
Preferably, the transmission information includes: a content descriptor and an interactive feedback descriptor;
the content descriptor includes: point cloud content description information, quantity information, content information, priority identification value information and/or priority order list information;
the interaction feedback descriptor provides interaction feedback between the server and the client, and the feedback descriptor includes: priority decoding information, decoding end feedback information, interaction target information, interaction type information and/or interaction content information.
Preferably, the step M4 includes:
step M4.1: decoding according to a set priority order or decoding different point cloud contents under the priority order according to the information indication of the descriptor;
step M4.2: searching whether a priority order for selection is available according to priority preference fed back by a decoding end system, and if so, directly applying the priority order to decode; otherwise, the transmission and decoding are performed according to a priority order defined by the priority preference of the decoding end system.
The invention provides a point cloud data processing system, which comprises:
a module M1: performing segmentation processing on the point cloud data to obtain a preset number of point cloud contents, and performing classification processing according to different obtained preset number of point cloud contents;
a module M2: carrying out priority setting processing on the point cloud contents after classification processing;
a module M3: setting a priority order according to a priority setting process;
a module M4: and decoding the point cloud data according to the set priority order.
Preferably, said module M2 comprises:
module M2.1: encoding the point cloud content obtained after classification;
module M2.2: and packaging the encoded point cloud content according to a packaging format defined by a packaging protocol, and defining a priority identifier in the packaged point cloud content.
Preferably, said module M3 comprises:
module M3.1: whether the packaged point cloud contents are grouped is judged according to the segmentation and/or classification results, and when the point cloud contents are grouped, the point cloud contents are grouped according to data obtained by segmenting the point cloud data, and the type and/or index and priority order of the grouping are set; when the peer cloud data are not grouped, setting a priority order according to the priority setting processing of the peer cloud content;
module M3.2: and setting transmission signaling of information for describing the length or priority of each point cloud content in the point cloud data.
Preferably, the transmission information includes: a content descriptor and an interactive feedback descriptor;
the content descriptor includes: point cloud content description information, quantity information, content information, priority identification value information and/or priority order list information;
the interaction feedback descriptor provides interaction feedback between the server and the client, and the feedback descriptor includes: priority decoding information, decoding end feedback information, interaction target information, interaction type information and/or interaction content information.
Preferably, said module M4 comprises:
module M4.1: decoding according to a set priority order or decoding different point cloud contents under the priority order according to the information indication of the descriptor;
module M4.2: searching whether a priority order for selection is available according to priority preference fed back by a decoding end system, and if so, directly applying the priority order to decode; otherwise, the transmission and decoding are performed according to a priority order defined by the priority preference of the decoding end system.
Compared with the prior art, the invention has the following beneficial effects:
1. when the point cloud processing is carried out, point cloud data are divided into different independently-coded and decodable segments according to calculation and processing of certain classification, segmentation and the like, and a packaging format of point cloud media content is designed according to a certain priority order; and providing a certain content identification for the information flow;
2. the decoding end can complete decoding and point cloud data consumption based on the priority order according to the basis provided by a specific point cloud media consumption party and the set content identification of the point cloud information stream segment;
3. the invention meets the diversity of decoding end equipment, the complexity of network conditions and the individual requirements of point cloud data users, and is suitable for various environments and requirements.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a schematic view of a processing flow of point cloud data based on priority;
fig. 2 is a schematic diagram illustrating packaging of priority point cloud content based on ISOBMFF;
FIG. 3 is a schematic diagram illustrating the description and interaction of point cloud content under priority decoding;
FIG. 4 is a flow chart of the present invention for processing a point cloud with priority order decoding.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
The invention aims to provide a point cloud data processing method and a point cloud data processing system, which aim at the decoding process of a point cloud information stream, set different orders for the information stream according to the content of the point cloud data and the requirements of a consumption end, optimize the processing, presentation and other processes of the point cloud at the decoding end.
Specifically, according to the complexity of the decoding end: under a specific packaging structure, according to feedback information of a decoding end received by a content provider, point cloud data is transmitted selectively or according to a certain priority order according to the content of the point cloud data to be processed, or the content provider and the processor transmit the processed point cloud selectively or according to a certain priority order according to the content of the point cloud data to be processed according to the characteristics of the scene, the consumer and the like of the point cloud data to be processed. Meanwhile, corresponding to the decoding end, the corresponding point clouds can be decoded and presented according to a certain priority order or selectively according to user information (such as viewpoint information, position information, scene information, equipment performance information, network condition information and the like) acquired by the sensing equipment. Therefore, under various conditions, proper point cloud data content or priority order can be selected for transmission, decoding, presentation, processing, application and the like. The point cloud consumption experience which is as satisfactory as possible under certain conditions is provided for the user.
The invention provides a point cloud data content provider and a content processor which compress the processed point cloud after the point cloud data is divided, classified and the like; and encapsulating the compressed point cloud media file, setting a type identifier and a priority identifier for the content of each part in the point cloud according to the content during processing, wherein the identifier is used for distinguishing different types or segments (a certain part in the whole point cloud after operation such as classification, segmentation and the like) after processing, recording the necessary data type and priority information of each part, grouping the type information by user segments, and setting the priority order during decoding by the priority information.
The invention provides a point cloud data processing method, which comprises the following steps:
step M1: performing segmentation processing on the point cloud data to obtain a preset number of point cloud contents, and performing classification processing according to different obtained preset number of point cloud contents;
step M2: carrying out priority setting processing on the point cloud contents after the classification processing;
specifically, the step M2 includes:
step M2.1: encoding the point cloud content obtained after classification;
step M2.2: and packaging the encoded point cloud content according to a packaging format defined by a packaging protocol, and defining a priority identifier in the packaged point cloud content.
Step M3: setting a priority order according to a priority setting process;
specifically, the step M3 includes:
step M3.1: whether the packaged point cloud contents are grouped is judged according to the segmentation and/or classification results, and when the point cloud contents are grouped, the point cloud contents are grouped according to data obtained by segmenting the point cloud data, and the type and/or index and priority order of the grouping are set; when the point cloud data are not grouped, setting and processing a priority order according to the priority of the point cloud content;
step M3.2: and setting transmission signaling of information for describing the length or priority of each point cloud content in the point cloud data.
Specifically, the transmission information includes: a content descriptor and an interactive feedback descriptor;
the content descriptor includes: point cloud content description information, quantity information, content information, priority identification value information and/or priority order list information;
the interaction feedback descriptor provides interaction feedback between the server and the client, and the feedback descriptor includes: priority decoding information, decoding end feedback information, interaction target information, interaction type information and/or interaction content information.
Step M4: and decoding the point cloud data according to the set priority order.
Specifically, the step M4 includes:
step M4.1: decoding according to a set priority order or decoding different point cloud contents under the priority order according to the information indication of the descriptor;
step M4.2: searching whether a priority order for selection is available according to priority preference fed back by a decoding end system, and if so, directly applying the priority order to decode; otherwise, the transmission and decoding are performed according to a priority order defined by the priority preference of the decoding end system.
The invention also provides a point cloud data processing system for carrying out system measurement and modeling on the point cloud data in the stages of packaging and transmitting, so that the method can effectively operate in specific hardware.
The invention provides a point cloud data processing system, which comprises:
a module M1: performing segmentation processing on the point cloud data to obtain a preset number of point cloud contents, and performing classification processing according to different obtained preset number of point cloud contents;
a module M2: carrying out priority setting processing on the point cloud contents after the classification processing;
in particular, said module M2 comprises:
module M2.1: encoding the point cloud content obtained after classification;
module M2.2: and packaging the encoded point cloud content according to a packaging format defined by a packaging protocol, and defining a priority identifier in the packaged point cloud content.
A module M3: setting a priority order according to a priority setting process;
in particular, said module M3 comprises:
module M3.1: whether the packaged point cloud contents are grouped is judged according to the segmentation and/or classification results, and when the point cloud contents are grouped, the point cloud contents are grouped according to data obtained by segmenting the point cloud data, and the type and/or index and priority order of the grouping are set; when the point cloud data are not grouped, setting and processing a priority order according to the priority of the point cloud content;
module M3.2: and setting transmission signaling of information for describing the length or priority of each point cloud content in the point cloud data.
Specifically, the transmission information includes: a content descriptor and an interactive feedback descriptor;
the content descriptor includes: point cloud content description information, quantity information, content information, priority identification value information and/or priority order list information;
the interaction feedback descriptor provides interaction feedback between the server and the client, and the feedback descriptor includes: priority decoding information, decoding end feedback information, interaction target information, interaction type information and/or interaction content information.
A module M4: and decoding the point cloud data according to the set priority order.
In particular, the module M4 comprises:
module M4.1: decoding according to a set priority order or decoding different point cloud contents under the priority order according to the information indication of the descriptor;
module M4.2: searching whether a priority order for selection is available according to the priority preference fed back by the decoding end system, and if so, directly applying the priority order for decoding; otherwise, the transmission and decoding are performed according to a priority order defined by the priority preference of the decoding end system.
The present invention is further described in detail by the following preferred examples:
as shown in the attached figure 1, the point cloud data processing method comprises the following steps:
step S01: dividing and classifying the point cloud original data to obtain a plurality of point cloud contents, or called point cloud subsamples, independently encoding each divided point cloud subsample, wherein the whole point cloud original data is a sample, each divided and classified part of the point cloud original data is a subsample, the subsamples can be independently encoded and decoded, a transmitting end performs encoding processing, and a receiving end performs decoding processing;
step S02: encapsulating the independently encoded subsamples, encapsulating the point cloud content according to an encapsulation format defined by an ISOBMFF (ISOBMFF) encapsulation protocol and related to the samples and the subsamples, defining a Priority list Priority _ list in the samples, and defining a Priority identifier subsample _ Priority in the subsamples. Using the priority identifier subsampljpriority defined in the encapsulated subsamples as the priority identifier of each subsample, which can also be regarded as the ID identifier of the subsample;
step S03: grouping each packaged sub-sample subsample according to the principle of partitioning and classifying the point cloud original data in step S01, wherein the grouping basis is the same as or similar to the partitioning and classifying basis, the packaged sub-samples subsamples of the same Type of priority content are grouped into the same group Type _ group, the group Type _ group is the content defined in the sample according to the invention, and each group Type _ group contains the priority identifier subsample _ priority of the sub-sample subsample corresponding to the same Type of priority content;
step S04: a Priority list Priority _ list is defined in sample, and the Priority list includes the Priority order of each group Type _ group and the Priority order of the subsample.
According to the point cloud data processing method provided by the invention, different packaged subsamples in the same group Type _ group also have a certain priority order, and the group Type _ group is optional: grouping or non-grouping can be selected, not all point cloud application scenes need group Type _ group, and after the point cloud original data of some point cloud application scenes are segmented, the point cloud original data are not suitable for classifying different parts of point cloud contents, such as an example of capturing objects in the following embodiment two. The existence of the group Type _ group provides a classification priority basis, and also provides a function for a consumer end to specify the priority order of the classes of the point cloud content.
The content types of the point cloud after the point cloud data or the point cloud original data is classified, segmented and the like, and the calculation and processing operations are carried out can be various and common, including but not limited to the following: the method comprises the steps of point cloud data segmentation and classification in a specific scene, the point cloud data of the point cloud scene is segmented into point cloud contents corresponding to different objects according to requirements, the point cloud contents of the object are generally distinguished, the difference between the objects is large, segmented fragments are subsamples of the whole point cloud content, the subsamples have priorities and type identifications, and the subsamples of the same type can be divided into a group of point cloud subsample sets of the same type.
The method comprises the steps of point cloud data segmentation and classification of specific objects, wherein the objects of given point cloud data are segmented into different parts according to certain rules and purposes, point cloud contents of different segments belong to the same object but have differences, and subsample subsamples have types and priority marks.
In the present invention, decoding can be performed in a set priority order, or decoding in a priority order can be adjusted according to corresponding scene information requirement feedback, and the foregoing adjustment processing may be performed according to various bases, and preferably, can be implemented by any one or more of the following sets of conditions:
network conditions are as follows: the number of data bits transmitted per unit time during data transmission generally has a positive correlation with the video quality; when the network condition is not good, the contour in the point cloud content or the content concerned by the consumer can be selected to be decoded in advance or higher priority is arranged for the contour in the point cloud content or the content concerned by the user;
scene information: specific scene information to be applied and processed when the point cloud data is recovered at a decoding end is different from the attention content of a robot or a human to the point cloud content in different scenes, and the attention degree and range of the whole point cloud content in different scenes are also different; in some application scenarios, only a specific part of the entire point cloud content is concerned, and the specific part has a higher priority;
the consumption end information, specifically, the object of the decoded point cloud data, generally different consumers, the content of the point cloud data, the focus point of the point cloud data, the processing form of the point cloud data, and the like of the different objects are different, and the content which is required to be firstly acquired and processed by the consumers has higher priority.
As shown in fig. 2, the present invention will provide a preferred embodiment, based on the encapsulation steps of the ISOBMMF encapsulation protocol.
The encapsulation syntax structure based on the ISOBMMF encapsulation protocol includes two types, which are an ST1 sub-sample data box structure and an ST2 priority order data box encapsulation structure, as will be described in detail below:
structure ST1: sub-sample data box structure
The expression defines a subsample data box structure for describing subsample information, and is used for packaging each subsample after point cloud original data is segmented and classified.
Wherein:
version: is an integer that specifies the version of the data box for this subsample information (0 or 1 in the ISOBMFF encapsulation protocol);
entry _ count: is an integer for counting the number of entries;
sample _ delta: is an integer that specifies the number of samples of the sample with the subsample package structure. Encoded as the required sample, the difference between the number and the sample number indicated in the previous entry. If the current entry is the first entry, the value indicates the sample number of the first sample having the sub-sample information, i.e., the value is the difference between the sample number and zero (0);
subsample _ count: is an integer that specifies the subsample number of the current sample. If there is no subsample structure, the value of the field is 0;
subsample _ size: is an integer that specifies the size of the current sub-sample in bytes;
subsample _ priority: is an integer that specifies the degradation priority for each subsample. The higher the value of subsample _ prior, the more significant the subsamples are to the decoding quality and the effect on the decoding quality is larger;
discard (b): a value equal to 0 indicates that the sub-sample subsample needs to be decoded, and a value equal to 1 means that the sub-sample subsample does not need to be decoded but can be used for enhancement, e.g., the sub-sample subsample consists of a Supplemental Enhancement Information (SEI) message.
A new data box is defined under the sampleTableBox to classify the contents of the subsamples and implement the functions of priority order, etc.
Structure ST2: package structure of data box with newly added priority order
PriorityBox
Definition of
Data box type: 'stdp'
Comprises the following steps: sampleTableBox ('stbl')
Mandatory type: whether or not
Quantity: 0 or 1
This new priority order data box contains the priority of each subsample. Its value is stored in this encapsulated sample table. The table size, subsamplcount, may be obtained from the subsamplcount in the subsample data box, subsampleinformationbox ('subs'). This priority data box defines the exact meaning and acceptable range of sub-sample decoding priorities.
Grammar
Grammar for grammar
version: an integer specifying the version of the data box;
type _ group: the Type priority group includes content obtained by classifying each part (encapsulated subsample) of the point cloud content after being segmented and classified, and includes a Type index list Type _ list, where there are indexes of groups corresponding to different types.
Priority _ list: and the priority list comprises the priority groups and the priority orders of the subsamples.
priority: and identifying an integer of the Priority of each subsample, decoding according to a default Priority order if the Priority list Priority _ list is empty, and separately decoding each subsample according to a Priority identification order defined in the Priority list if the Priority list Priority _ list is not empty.
The transmission protocol is required to be used in the transmission process of the encapsulated point cloud data, a descriptor for describing the point cloud content is added in a transmission signaling of the transmission protocol to describe the length and the content of the point cloud content of each independently-coded segment in the transmitted point cloud information stream and realize corresponding information of a priority coding and decoding function, so that a subsequent decoding end and a consumer can conveniently realize subsequent processing of the point cloud content according to a certain priority order or after selective acquisition, decoding and decoding, and the point cloud data can adapt to different scenes, conditions, consumers and the like during use and processing.
In order to make the above objects, features and advantages of the present invention more comprehensible, a descriptor (descriptor) describing contents of a point cloud information stream and a descriptor (descriptor) of interactive feedback contents of a user or a content provider are added to a transmission signaling in the present invention, and are described in detail below with reference to fig. 3.
For example, some descriptive descriptor descriptors are defined in signaling messages defined in the MMT transport protocol, and the descriptor descriptors are descriptive information used for defining some fields or functions in the MMT transport signaling. Such as a dependency descriptor or an MPU _ timestamp _ descriptor. Similarly, the invention newly defines a descriptor for describing the content of the point cloud information stream, and the descriptor for describing the content of the point cloud information stream can describe the type information of the point cloud media segment.
In the MPtable, there is an asset _ descriptors field, which can be implemented by adding a PointCTtype _ descriptor in the asset _ descriptors as needed, as shown in the following table:
definition of PointCTtype _ descriptor:
wherein:
descriptor _ tag: for indicating the type of descriptor;
descriptor _ length: for indicating the length of descriptor;
num _ Pointpart: for indicating the number of point cloud segments, i.e. subsamples;
IDlist _ Pointpart: ID identification order lists decoded by different priorities;
typegroup _ list: indexing of corresponding groups of different types of values
Pointpart _ content (): information of different parts in the whole point cloud is contained;
pointpart _ ID [ i ]: and indicating the ID identification of the point i cloud subsample.
And when the content described in the descriptor of the interactive feedback content of the user and the content provider, namely an interactive feedback message (InteractionFeedbackMessage), provides the media consumption, the interactive feedback between the server and the client. When the interactive feedback information needs to be sent between the server and the client in media consumption, the message is used for conversation.
The interactive feedback message mainly comprises an interactive target, an interactive type and interactive content. Along with the change of the specific interaction behavior of the user, retrieving a priority list of the transmission content according to the priority selection information of the user obtained by the interaction with the user, if the priority list of the transmission content exists, directly feeding back, regenerating the priority sequence information generated according to the priority preference of the user if the priority sequence does not exist, and transmitting the point cloud content according to the corresponding priority sequence.
The syntax structure of the interactive feedback messages in the priority order of the point clouds based on the SMT example is given below.
Interactive feedback message syntax in priority order:
semantics:
message _ id: an identifier of the interactive feedback message;
version: versions of interactive feedback messages. The information carried by the new version will overwrite any previous old version;
length: the length of the interactive feedback message in bytes, i.e. from the next field until the last byte of the interactive feedback message, is included. The "0" value is invalid in this field;
message _ source: indicating a message source, 0 indicating that the interactive feedback message is sent by the client to the server, and 1 indicating that the interactive feedback message is sent by the server to the client. This value handles 0;
priority _ flag: indicating priority to decode the interactive content, 0 indicating that priority interaction is not needed, and 1 indicating that priority interaction is needed;
reserved: reserving byte bits;
asset _ id: asset _ id indicating that the client currently consumes the content;
interaction _ num: indicating the number of interactions contained in the current signaling;
timing and map: indicating the time of current interaction generation, using UTC time;
interaction _ type: interactive type, set to 0;
client _ priority: the priority order in the interactive contents indicates the type of priority order required by the user, and contains necessary information for performing priority order decoding.
Based on the above description, in order to more clearly clarify that the decoding of the point cloud media content can be performed according to a certain priority order and selectively, the following contents give two specific application examples of the present invention.
In the case of the ISOBMFF encapsulation protocol, MMT and SMT transport protocol based, the priority order codec and presentation process in two typical application scenarios is represented as follows, as shown in the flow diagrams of fig. 1 and 4:
example one
Step S11: the method comprises the steps of dividing current scene point cloud data acquired by equipment under automatic driving, dividing point cloud data of objects of different types under the scene aiming at the purpose of acquiring the point cloud data by a machine under the automatic driving scene (for reasonably planning a route, avoiding pedestrians and the like), acquiring a plurality of independently-coded and decodable point cloud contents, and setting different priorities for the independently-coded point cloud contents. Particularly, for the requirement of obstacle avoidance, after point cloud content obtained by vehicle sensing equipment is divided, scene division and classification are performed, for example, pedestrians, vehicles, roadblocks, roads, trees, street views, buildings and the like on the roads are all in different categories, a plurality of point cloud sub-samples are often present in different categories, for example, people in the category have various postures, people in various positions, a certain division segment is called as a sub-sample of the whole point cloud sample, for example, a person wearing red clothes is divided from a certain point cloud data part, a packaging structure corresponding to the sub-sample can be independently coded and decoded, the sub-sample packaging structure is provided with a default priority identification sub-sample _ priority, and the sub-sample can be indexed according to the value;
step S12: on the basis of segmentation and classification, the processed different subsamples are grouped according to categories, for example, the subsamples with the identification values of 2,3, 10 and 11 are all human categories and are divided into a group, which is denoted as a type group0, and the subsamples with the identification values of 4, 12, 20, 35 and 41 are all vehicle categories and are divided into a group, which is denoted as a type group 1type group1. Recording the types of the different groups and the corresponding subsamples in the different groups, and simultaneously making a Type index list Type _ list for the types of the groups and the corresponding groups for indexing the groups of different types according to the types, wherein the information is contained in a Type group structure Type _ group.
Step S13: setting a default priority order for application-oriented scenarios, taking the current autopilot scenario as an example, obstacle avoidance and route planning are the primary functions, then setting the highest decoding priority order for Type 'person' followed by 'car', then 'road strike', 'sign', 'tree', 'street store', etc. in the Type index list Type _ list. The corresponding Group identifiers Group _ ID are indexed according to the type, and the Group identifiers are stored in the Group Priority list Group _ Priority list in the Priority list Priority _ list according to a certain Priority order, for the decoding order of the sub-sample structures in the same Group, different Priority orders can be set for different sub-sample structures according to the same scene requirements, such as decoding by people nearby, and the different Priority orders are stored in the sub-sample structure Priority list subsample _ Priority list in the Priority list Priority _ list.
Step S14: at a decoding end, if no specific requirement exists, the point cloud information stream processed by the point cloud content provider can be directly used according to a conventional decoding scheme; if there are specific requirements, such as poor current network conditions, the computing device performance is limited and the decoding of the entire point cloud cannot be completed within an acceptable time, at which point:
1. decoding point cloud subsamples of different categories in priority order according to description in a point cloud content descriptor PointCTtype _ descriptor describing point cloud content;
2. firstly, a user feeds back the current priority preference Client _ priority (), such as 'human' -vehicle '-tree', the feedback can be real-time and recorded so as to make a timely reflection once entering an area with a poor network condition;
3. then retrieving point cloud content descriptor PointCTtype _ descriptor, wherein a segment point cloud identifier list IDlist _ Pointpart exists in the point cloud content descriptor PointCTtype _ descriptor, and if a selectable priority order exists, directly selecting an applicable priority order for decoding;
4. if no suitable priority sequence scheme exists, finding a corresponding group according to the priority preference of the user by the content index group type list typegroup _ list fed back by the user, and transmitting and subsequently decoding the content according to the priority sequence defined by the user.
Example two
Step S21: dividing captured objects, such as people who take flowers and wear fashionable people, at the moment, sub-sample sub-samples after division are difficult to distinguish from each other, at the moment, the sub-sample sub-samples are not classified any more, only the parts of the sub-sample sub-samples after division are used for packaging, the sub-sample _ priority is identified by default priority in the sub-sample structure, and the sub-sample sub-samples can be indexed according to the value;
step S22: in the application scene, the subsamples are not classified and grouped any more, at this time, the group Type _ group content is empty, and the Priority list Priority _ list only contains the Priority order of the subsamples;
step S23: setting a default Priority order facing an application scene, taking a current flower holding and fashionable people wearing as an example, an application scene is face recognition, at the moment, a plurality of sub-samples containing a face have higher Priority, while sub-samples containing other part point cloud fragments of the person have lower Priority, and for identifiable sub-sample identification (subsampljpriority), setting higher Priority for ID identification in a Priority list (Priority _ list) of the sub-samples of the face in a decoding Priority data box (decodingpriority box) during order setting, wherein one setting is face subsampljpriority > other part subsampljpriority;
step S24: at a decoding end, if no specific requirement exists, the point cloud information stream processed by a point cloud content provider can be directly used according to a conventional decoding scheme; if there are specific requirements, such as poor current network conditions, the computing device performance is limited and the decoding of the entire point cloud cannot be completed within an acceptable time, at which point:
1. decoding point cloud subsamples of different types in priority order according to description in a point cloud content descriptor (PointCTtype _ descriptor) describing point cloud content;
2. firstly, a user feeds back the current priority preference Client _ priority (), at the moment, because no sub-sample subsamples are classified and grouped, the type of the user feedback cannot accurately carry out the priority arrangement of the sub-sample subsamples, such as a face-body-carrying object of a face recognition system example, under the condition of no classification, the feedback information of the user is limited, only the existing priority order in a point cloud fragment identification list IDlist _ Point in a content descriptor Point _ descriptor can be fed back, and the content of the sub-sample structure priority list Subjective _ priority list can be fed back, and the feedback can be real-time and recorded, so that the feedback can be reflected in time once entering an area with a poor network condition;
3. then, a point cloud segment identifier list IDlist _ Pointpart exists in the point cloud content descriptor PointCTtype _ descriptor, selectable priority orders exist in the point cloud content descriptor PointCTtype _ descriptor, only a subsample _ priorList of a sub-sample structure priority list exists in the point cloud content descriptor PointCTtype _ descriptor under the condition of no classification, and the priority orders in the point cloud content descriptor, namely the subsample _ priorList, of the sub-sample structure priority list are not grouped, so that the priority orders in the point cloud content descriptor PointCTtype _ descriptor can be directly selected for decoding;
4. at this time, the content of the user feedback is limited, for example, the first selection is that, facing a face recognition application scenario when the network condition is not good and the device performance is limited, the subsample _ priority of the ' subsample of the face ' of the subsample _ priority list is selected according to the fed-back scenario information, the subsample _ priority list also provides a selection for other scenarios, and the other one may be a subsample _ priority of a behavior detection application scenario when the network condition is not good and the device performance is limited, where the other one of the subsample _ priority lists is the subsample _ priority of the ' outline ' subsample _ priority ' of the subsample ' other position ' of the subsample. And under the condition of no classification, transmitting and decoding the content in the priority order according to the feedback information of the consumption end.
The inventive concepts, described embodiments, and scope of the present application enable the selection of appropriate point cloud data content or priority order for transmission, decoding, presentation, processing, application, and the like, under various conditions. The point cloud consumption experience which is as satisfactory as possible under certain conditions is provided for the user.
It should be noted that, although the proposed method for processing point cloud data is illustrated by using the packaging protocol MMT such as ISOBMFF, and the transmission protocol such as SMT, etc. as examples, the point cloud data of the present embodiment may also be packaged and transmitted by using other files, and the expression of the core technology of the present invention is not affected.
Those skilled in the art will appreciate that, in addition to implementing the systems, apparatus, and various modules thereof provided by the present invention in purely computer readable program code, the same procedures can be implemented entirely by logically programming method steps such that the systems, apparatus, and various modules thereof are provided in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system, the device and the modules thereof provided by the present invention can be considered as a hardware component, and the modules included in the system, the device and the modules thereof for implementing various programs can also be considered as structures in the hardware component; modules for performing various functions may also be considered to be both software programs for performing the methods and structures within hardware components.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (10)
1. A point cloud data processing method is characterized by comprising the following steps:
step M1: performing segmentation processing on the point cloud data to obtain a preset number of point cloud contents, and performing classification processing according to different obtained preset number of point cloud contents;
step M2: carrying out priority setting processing on the point cloud contents after the classification processing;
step M3: setting a priority order according to a priority setting process;
step M4: and decoding the point cloud data according to the set priority order.
2. The point cloud data processing method according to claim 1, wherein the step M2 comprises:
step M2.1: encoding the point cloud content obtained after classification;
step M2.2: and packaging the encoded point cloud content according to a packaging format defined by a packaging protocol, and defining a priority identifier in the packaged point cloud content.
3. The point cloud data processing method according to claim 1, wherein the step M3 comprises:
step M3.1: whether the packaged point cloud contents are grouped is judged according to the segmentation and/or classification results, and when the point cloud contents are grouped, the point cloud contents are grouped according to data obtained by segmenting the point cloud data, and the type and/or index and priority order of the grouping are set; when the peer cloud data are not grouped, setting a priority order according to the priority setting processing of the peer cloud content;
step M3.2: and setting transmission signaling of information for describing the length or priority of each point cloud content in the point cloud data.
4. The point cloud data processing method of claim 3, wherein the transmission information comprises: a content descriptor and an interactive feedback descriptor;
the content descriptor includes: point cloud content description information, quantity information, content information, priority identification value information and/or priority order list information;
the interaction feedback descriptor provides interaction feedback between the server and the client, and the feedback descriptor includes: priority decoding information, decoding end feedback information, interaction target information, interaction type information and/or interaction content information.
5. The point cloud data processing method according to claim 1, wherein the step M4 comprises:
step M4.1: decoding according to a set priority order or decoding different point cloud contents under the priority order according to the information indication of the descriptor;
step M4.2: searching whether a priority order for selection is available according to the priority preference fed back by the decoding end system, and if so, directly applying the priority order for decoding; otherwise, the transmission and decoding are performed according to a priority order defined by the priority preference of the decoding end system.
6. A point cloud data processing system, comprising:
a module M1: performing segmentation processing on the point cloud data to obtain a preset number of point cloud contents, and performing classification processing according to different obtained preset number of point cloud contents;
a module M2: carrying out priority setting processing on the point cloud contents after the classification processing;
a module M3: setting a priority order according to a priority setting process;
a module M4: and decoding the point cloud data according to the set priority order.
7. The point cloud data processing system of claim 6, wherein the module M2 comprises:
module M2.1: encoding the point cloud content obtained after classification;
module M2.2: and packaging the encoded point cloud content according to a packaging format defined by a packaging protocol, and defining a priority identifier in the packaged point cloud content.
8. The point cloud data processing system of claim 6, wherein the module M3 comprises:
module M3.1: whether the packaged point cloud contents are grouped or not is judged according to the segmentation and/or classification results, and when the point cloud contents are grouped, the point cloud contents are grouped according to data obtained by segmenting the point cloud data, and the types and/or indexes and priority orders of the groups are set; when the point cloud data are not grouped, setting and processing a priority order according to the priority of the point cloud content;
module M3.2: and setting transmission signaling of information for describing the length or priority of each point cloud content in the point cloud data.
9. The point cloud data processing system of claim 8, wherein the transmission information comprises: a content descriptor and an interactive feedback descriptor;
the content descriptor includes: point cloud content description information, quantity information, content information, priority identification value information and/or priority order list information;
the interaction feedback descriptor provides interaction feedback between the server and the client, and the feedback descriptor includes: priority decoding information, decoding end feedback information, interaction target information, interaction type information and/or interaction content information.
10. The point cloud data processing system of claim 6, wherein the module M4 comprises:
module M4.1: decoding according to a set priority order or decoding different point cloud contents under the priority order according to the information indication of the descriptor;
module M4.2: searching whether a priority order for selection is available according to the priority preference fed back by the decoding end system, and if so, directly applying the priority order for decoding; otherwise, the transmission and decoding are performed according to a priority order defined by the priority preference of the decoding end system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010040455.1A CN113129393B (en) | 2020-01-15 | 2020-01-15 | Point cloud data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010040455.1A CN113129393B (en) | 2020-01-15 | 2020-01-15 | Point cloud data processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113129393A CN113129393A (en) | 2021-07-16 |
| CN113129393B true CN113129393B (en) | 2022-12-27 |
Family
ID=76771403
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010040455.1A Active CN113129393B (en) | 2020-01-15 | 2020-01-15 | Point cloud data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113129393B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102130886A (en) * | 2010-01-18 | 2011-07-20 | 中国电信股份有限公司 | Network video streaming media system, transmission processing method, transmitting end and receiving end |
| CN106302458A (en) * | 2016-08-16 | 2017-01-04 | 广东中星电子有限公司 | Video information transmission method and video information transmission system |
| CN106780509A (en) * | 2016-12-01 | 2017-05-31 | 山东交通学院 | Merge the building object point cloud layer time cluster segmentation method of multidimensional characteristic |
| CN109687934A (en) * | 2017-10-18 | 2019-04-26 | 上海交通大学 | Adaptable System code FEC method, apparatus and system based on media content |
| KR20190064050A (en) * | 2017-11-30 | 2019-06-10 | (주)소프트힐스 | Visualizaion system of massive point cloud data |
| WO2019213985A1 (en) * | 2018-05-09 | 2019-11-14 | 北京大学深圳研究生院 | Hierarchical division-based point cloud attribute compression method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105827361B (en) * | 2015-01-08 | 2019-02-22 | 上海交通大学 | A kind of FEC method based on media content |
| US11210815B2 (en) * | 2019-08-09 | 2021-12-28 | Intel Corporation | Point cloud playback mechanism |
-
2020
- 2020-01-15 CN CN202010040455.1A patent/CN113129393B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102130886A (en) * | 2010-01-18 | 2011-07-20 | 中国电信股份有限公司 | Network video streaming media system, transmission processing method, transmitting end and receiving end |
| CN106302458A (en) * | 2016-08-16 | 2017-01-04 | 广东中星电子有限公司 | Video information transmission method and video information transmission system |
| CN106780509A (en) * | 2016-12-01 | 2017-05-31 | 山东交通学院 | Merge the building object point cloud layer time cluster segmentation method of multidimensional characteristic |
| CN109687934A (en) * | 2017-10-18 | 2019-04-26 | 上海交通大学 | Adaptable System code FEC method, apparatus and system based on media content |
| KR20190064050A (en) * | 2017-11-30 | 2019-06-10 | (주)소프트힐스 | Visualizaion system of massive point cloud data |
| WO2019213985A1 (en) * | 2018-05-09 | 2019-11-14 | 北京大学深圳研究生院 | Hierarchical division-based point cloud attribute compression method |
Non-Patent Citations (1)
| Title |
|---|
| 《Segmentation of 3D outdoor scenes using hierarchical clustering structure and perceptual grouping laws》;Yusheng Xu Etal;《 2016 9th IAPR Workshop on Pattern Recogniton in Remote Sensing (PRRS)》;20170302;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113129393A (en) | 2021-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Duan et al. | Compact descriptors for video analysis: The emerging MPEG standard | |
| CN109257604A (en) | A kind of color attribute coding method based on TMC3 point cloud encoder | |
| CN110971906B (en) | Hierarchical point cloud code stream packaging method and system | |
| CN101420595A (en) | Method and equipment for describing and capturing video object | |
| WO2022068672A1 (en) | Point cloud data processing method and apparatus, and storage medium and electronic apparatus | |
| CN114079781B (en) | Data processing method, device and equipment of point cloud media and storage medium | |
| KR20210061399A (en) | Apparatus and method for artificial intelligence | |
| CN114697668B (en) | Encoding and decoding method of point cloud media and related products | |
| EP4090021A1 (en) | Point cloud data encapsulation method and point cloud data transmission method | |
| WO2022062860A1 (en) | Data processing method, apparatus and device for point cloud media, and storage medium | |
| CN115379189B (en) | Data processing method of point cloud media and related equipment | |
| CN112861894A (en) | Data stream classification method, device and system | |
| WO2023207119A1 (en) | Immersive media processing method and apparatus, device, and storage medium | |
| KR20230136639A (en) | Volumetric media processing methods, devices, storage media and electronic devices | |
| CN113516102A (en) | Deep learning parabolic behavior detection method based on video | |
| CN113129393B (en) | Point cloud data processing method and system | |
| US11490134B2 (en) | Method and system for codec of visual feature data | |
| US20210112111A1 (en) | Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device | |
| KR102624994B1 (en) | File format for point cloud data | |
| US11856154B2 (en) | Three-dimensional data storage method, three-dimensional data acquisition method, three-dimensional data storage device, and three-dimensional data acquisition device | |
| CN115396646B (en) | Data processing method of point cloud media and related equipment | |
| CN105989063B (en) | Video retrieval method and device | |
| CN113474820A (en) | Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device | |
| EP1569169A1 (en) | Video system | |
| CN112639884A (en) | Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |