WO2022252235A1 - 3d地图的解码装置和方法、及3d地图的编码码流 - Google Patents
3d地图的解码装置和方法、及3d地图的编码码流 Download PDFInfo
- Publication number
- WO2022252235A1 WO2022252235A1 PCT/CN2021/098483 CN2021098483W WO2022252235A1 WO 2022252235 A1 WO2022252235 A1 WO 2022252235A1 CN 2021098483 W CN2021098483 W CN 2021098483W WO 2022252235 A1 WO2022252235 A1 WO 2022252235A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- map
- module
- code stream
- reconstruction
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 101
- 230000006837 decompression Effects 0.000 claims abstract description 188
- 238000012545 processing Methods 0.000 claims abstract description 138
- 230000005540 biological transmission Effects 0.000 claims abstract description 49
- 238000013139 quantization Methods 0.000 claims description 167
- 230000015654 memory Effects 0.000 claims description 52
- 230000008569 process Effects 0.000 claims description 46
- 238000004891 communication Methods 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 12
- 238000007906 compression Methods 0.000 abstract description 23
- 230000006835 compression Effects 0.000 abstract description 23
- 230000000007 visual effect Effects 0.000 description 94
- 238000010586 diagram Methods 0.000 description 52
- 238000004422 calculation algorithm Methods 0.000 description 24
- 230000006870 function Effects 0.000 description 10
- 238000005538 encapsulation Methods 0.000 description 7
- 238000001514 detection method Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 5
- 229920001621 AMOLED Polymers 0.000 description 4
- 230000008859 change Effects 0.000 description 4
- 238000013144 data compression Methods 0.000 description 4
- 238000005259 measurement Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 239000011521 glass Substances 0.000 description 3
- 230000033001 locomotion Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 238000003672 processing method Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000002096 quantum dot Substances 0.000 description 2
- 210000001525 retina Anatomy 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/124—Quantisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2211/00—Image generation
- G06T2211/40—Computed tomography
- G06T2211/416—Exact reconstruction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
Definitions
- the present application relates to 3D map technology, and in particular to a 3D map decoding device and method, and a 3D map encoding code stream.
- Virtual reality (virtual reality, VR), augmented reality (augmented reality, AR) and mixed reality (mixed reality, MR) technologies are emerging multimedia virtual scene technologies in recent years. This type of technology can create virtual reality and superimpose virtual reality and real world to generate new visual environment and interactive experience. In such applications, electronic devices need to determine their own pose information in the current environment, so as to accurately realize the fusion of virtual objects and real scenes.
- a typical solution is: the electronic device receives a 3D map of the environment from the server or other devices, and then collects the 3D map of the environment through the local sensor. combined with the collected visual information and the downloaded 3D map to determine the current pose of the electronic device.
- the original 3D map usually contains a huge amount of data, and the transmission of the map needs to consume a lot of bandwidth and time, which seriously limits the performance of the application and affects the user experience.
- the present application provides a 3D map decoding device and method, and a 3D map code stream, so as to reduce the data volume of the 3D map, further reduce the transmission bandwidth, and improve the transmission efficiency.
- the present application provides a codec system for a 3D map, including: an encoding device and a decoding device, and the encoding device and the decoding device are connected in communication; performing compression processing to obtain the code stream of the 3D map, and sending the code stream of the 3D map to the decoding device, the 3D map includes a plurality of 3D map points, and the data of the 3D map includes the multiple data of a 3D map point; the decoding device is configured to receive the code stream of the 3D map, and decompress the code stream of the 3D map to obtain the data of the 3D map.
- the 3D map may include multiple 3D map points, and correspondingly, the data of the 3D map may include data of multiple 3D map points.
- a 3D map point is an interesting or salient feature point in the environment.
- the compression module can compress the data of the 3D map, which can reduce the data volume of the 3D map, for example, reduce the data volume of the 3D map from Terabyte (TB) level to Gigabyte (Gigabyte, GB) level, so that in the scene where the 3D map needs to be transmitted, compared with the transmission of the original 3D map data, the transmission of the compressed 3D map data can reduce the amount of transmitted data on the one hand, and on the other hand can Reduce the bandwidth occupied in transmission and improve the transmission efficiency of 3D maps.
- TB Terabyte
- GB Gigabyte
- the encoding device is a cloud server, the decoding device is an electronic device; or, the encoding device is a first electronic device, and the decoding device is a second electronic device; the decoding The device is further configured to send a 3D map download request to the decoding device, and the 3D map download request includes position indication information; the encoding device is also configured to receive the 3D map download request, and send a request to the 3D map download request according to the 3D map download request.
- the decoding device sends a code stream of a 3D map corresponding to the location indication information.
- the foregoing electronic device may be a user terminal device.
- the encoding device is an electronic device, and the decoding device is a cloud server; the encoding device is specifically configured to send the 3D map information to the decoding device after the 3D map is created. stream.
- the electronic device collects visual information through a sensor, and determines the current pose of the electronic device in combination with the visual information and the 3D map from the server.
- the 3D map is provided by the server, that is, the server creates the 3D map, then compresses the 3D map, and transmits the compressed data of the 3D map to the electronic device; the electronic device performs decompression processing after receiving the compressed data of the 3D map, and obtains The reconstructed data of the 3D map is combined with the collected visual information and the 3D map to determine the current pose of the electronic device.
- the pose is the position and orientation information of the electronic device, which can be an absolute pose in the world coordinate system or a relative pose relative to a certain point in the environment.
- the server may pre-create the 3D map, compress the 3D map and store it locally, which can save storage space.
- the server can also transmit the compressed data of the 3D map to other devices, such as cloud storage.
- the server creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
- Compressing the 3D map by the server can save local storage space.
- the electronic device sends a map download request to the server. There are two ways to trigger the map download request:
- the user opens the 3D map application program installed on the electronic device, and the application program uploads the location information obtained based on GPS positioning or WiFi positioning to its corresponding server, and the uploading operation can trigger a map download request. Since the uploaded content includes location information, the server can make a preliminary estimate based on the location information, and transmit the compressed data of the 3D map of the area to which the positioning point indicated by the location information belongs to the electronic device.
- the scope of the area to which the anchor point indicated by the location information can be preset.
- the area to which the anchor point belongs can be the administrative region of each level (including counties, cities, countries, or administrative districts, etc.) where the anchor point is located, or it can be centered on the anchor point , Set the distance as the circle area of the radius.
- the user opens the 3D map application program installed on the electronic device, and actively enters or selects an area in the application program, for example, the user actively enters "xx business center", or selects the area from a list "A street, B street , C Street” has selected "A Street”.
- the foregoing operations of the user may trigger a map download request.
- Either the user input or selection includes a geographic location, so the server transmits the compressed data of the 3D map of the geographic location to the electronic device.
- the electronic device automatically detects whether it meets the conditions for downloading a 3D map or starts downloading a 3D map, or the electronic device passes the detection A change in ambient light or a change in the environment is detected to start downloading the 3D map, so as to request the server to download a 3D map within an area, and the size of the area is not specifically limited.
- the server sends the compressed data of the 3D map to the electronic device.
- step 3 and step 4 are independent of each other, and the sequence is not limited.
- the electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- the electronic device After receiving the compressed data of the 3D map, the electronic device does not need to decompress it immediately, but only needs to decompress to obtain the reconstruction data of the 3D map before positioning based on the visual information. For example, the user can pre-download the compressed data of a 3D map in an area by downloading an "offline map", and decompress the compressed data of the 3D map when positioning is required.
- the electronic device collects visual information through a sensor, and the server determines the current pose of the electronic device in combination with the visual information from the electronic device and a 3D map.
- the 3D map is provided by the server, that is, the server creates the 3D map, then compresses the 3D map, and stores the compressed data of the 3D map locally.
- the server When receiving the visual information from the electronic device, the server performs decompression processing to obtain the reconstructed data of the 3D map, and determines the current pose of the electronic device by combining the visual information and the 3D map.
- the server creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
- the electronic device sends the visual information to the server.
- the server decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the server compresses the 3D map to save storage space.
- the server performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- the server sends the pose to the electronic device.
- the electronic device collects visual information through a sensor, and determines the current pose of the electronic device in combination with the visual information and a 3D map.
- the 3D map is provided by the electronic device, that is, the electronic device creates the 3D map, then compresses the 3D map, and stores the compressed data of the 3D map locally.
- the electronic device performs decompression processing to obtain the reconstructed data of the 3D map, and combines the collected visual information and the 3D map to determine the current pose of the electronic device.
- the electronic device creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
- Electronic equipment collects visual information through sensors.
- the electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- the second electronic device collects visual information through a sensor, and determines the current pose of the second electronic device in combination with the visual information and the 3D map from the server.
- the 3D map is created by the first electronic device, that is, the first electronic device creates the 3D map, compresses the 3D map, and then sends the compressed data of the 3D map to the server, and the server sends the compressed data of the 3D map to the second
- the second electronic device performs decompression processing to obtain the reconstruction data of the 3D map, and determines the current pose of the second electronic device in combination with the collected visual information and the 3D map.
- the first electronic device may pre-create a 3D map, compress the 3D map and then transmit it to the server, which can reduce transmission bandwidth.
- the first electronic device creates a 3D map and compresses it to obtain compressed data of the 3D map.
- the first electronic device sends the compressed data of the 3D map to the server.
- the first electronic device compresses the 3D map before transmitting it, which can reduce the transmission bandwidth and improve the transmission efficiency.
- the second electronic device sends a map download request to the server.
- the map download request sent by the second electronic device may also be based on the trigger manner shown in FIG. 4a.
- the server sends the compressed data of the 3D map to the second electronic device.
- the second electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the second electronic device collects the visual information through the sensor.
- the second electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- the second electronic device collects visual information through a sensor, and the server determines the current pose of the second electronic device in combination with the visual information from the second electronic device and the 3D map from the first electronic device.
- the 3D map is created by the first electronic device, that is, the first electronic device creates the 3D map, compresses the 3D map, and then sends the compressed data of the 3D map to the server.
- the server performs decompression processing to obtain the reconstructed data of the 3D map, and combines the visual information from the second electronic device and the 3D map to determine the current pose of the second electronic device.
- the first electronic device creates a 3D map and compresses it to obtain compressed data of the 3D map.
- the first electronic device sends the compressed data of the 3D map to the server.
- the second electronic device acquires visual information through sensor collection.
- the second electronic device sends a positioning request to the server, and the positioning request carries visual information.
- the server decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the server performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- the server sends the pose obtained by positioning to the second electronic device.
- the second electronic device collects visual information through a sensor, and determines the current pose of the second electronic device in combination with the visual information and the 3D map from the first electronic device.
- the 3D map is created by the first electronic device, that is, the first electronic device creates a 3D map, performs compression processing on the 3D map, and then sends the compressed data of the 3D map to the second electronic device, and the second electronic device performs decompression processing , obtain the reconstructed data of the 3D map, and determine the current pose of the second electronic device in combination with the collected visual information and the 3D map from the first electronic device.
- the first electronic device creates a 3D map, and compresses the compressed data of the 3D map to be stored locally.
- the second electronic device sends a map download request to the first electronic device.
- the first electronic device sends the compressed data of the 3D map to the second electronic device.
- the second electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the second electronic device acquires visual information through sensor collection.
- the second electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- the data of the 3D map further includes a plurality of region descriptors, and any region descriptor in the plurality of region descriptors is used to describe a part of the plurality of 3D map points Or features of all 3D map points.
- the region descriptor can be used to describe the characteristics of some or all of the 3D map points in the multiple 3D map points, so that the region descriptor and the 3D map point are one-to-many Relationship.
- the features of each 3D map point among the multiple 3D map points can be described by some or all of the multiple region descriptors, so that the 3D map point and the region descriptor have a one-to-many relationship. It can be seen that there is a many-to-many relationship between multiple region descriptors and multiple 3D map points.
- regional descriptors include but are not limited to traditional methods such as bag of words (BOW), vector of aggregate locally descriptor (VLAD) and other traditional methods based on NetVLAD and artificial intelligence (AI). new method.
- BOW bag of words
- VLAD vector of aggregate locally descriptor
- AI artificial intelligence
- multiple region descriptors may also be identified with a number to distinguish the multiple region descriptors, but the number is also not a limitation on the order of the multiple region descriptors.
- the data of any 3D map point among the plurality of 3D map points includes a 3D map point descriptor and a spatial position of the 3D map point.
- the 3D map point descriptor is a vector (vector), which is used to represent the local features of the 3D map point.
- the spatial position of a 3D map point can be represented by X, Y, and Z on the axis of the three-dimensional space, it can also be represented by latitude and longitude, and altitude, and it can also be represented by polar coordinates.
- the embodiment of the present application expresses the spatial position of a 3D map point No specific restrictions are made.
- the spatial position of the 3D map point can be the absolute position of the 3D map point, or the relative position of the 3D map point
- the encoding device is further configured to create the 3D map.
- the decoding device is further configured to perform positioning according to the 3D map.
- the present application provides a 3D map decoding device, including: a transmission module and a decompression module; wherein the transmission module is used to receive a code stream of a 3D map, and the 3D map includes a plurality of 3D map points;
- the decompression module is configured to decompress the code stream of the 3D map to obtain reconstruction data of the 3D map, and the reconstruction data of the 3D map includes reconstruction data of the plurality of 3D map points.
- the decompression module can support the decompression processing of the compressed data of the 3D map, that is, the codec system can support the compression/decompression processing of the data of the 3D map, which can reduce the amount of data of the 3D map,
- transmitting compressed data of 3D maps can reduce the amount of transmitted data on the one hand, and reduce the bandwidth occupied in transmission on the other hand, improving the quality of 3D maps. transmission efficiency.
- the decompression processing may include at least one of inverse quantization processing and prediction processing, wherein,
- Quantization processing refers to mapping the data to be processed into one or more quantization indexes. Each quantization index corresponds to a quantization center. The number of bits of the quantization index is usually significantly less than the number of bits of the original data, so as to save storage or transmission bandwidth. Purpose. Quantization processing methods include, but are not limited to: scalar quantization, vector quantization, product quantization, and the like. Based on this, the inverse quantization process is the reverse process of the aforementioned quantization process, that is, reconstructed data corresponding to the original data is restored according to one or more quantization indexes, and the number of bits of the reconstructed data is greater than the number of bits of the quantization index.
- Prediction processing refers to using the residual data of the processed data and the data to be processed to perform prediction to obtain the data to be processed.
- the selection of reference data can be pre-agreed, for example, fixedly use the previously processed data as reference data, at this time it is not necessary to mark the reference data in the code stream;
- the identification information of the reference data written in the stream including the serial number of the reference data, or other information from which the reference data can be deduced.
- the processing of the 3D map data includes encapsulation, the purpose of which is to encapsulate the data to be encoded into a code stream.
- the encapsulation process can use any encoding algorithm, such as entropy encoding.
- Entropy coding is a lossless data compression method.
- Entropy coding algorithms include but are not limited to: Huffman (huffman) coding, arithmetic coding, compression/decompression improved by LZ77 compression algorithm (lempel-ziv-markov chain-algorithm, LZMA ) algorithm, providing the function library (zlib) algorithm for data compression, etc.
- the decompression processing may also include decapsulation processing, the purpose of which is to decapsulate the code stream into data to be decompressed, and the decapsulation processing may use a decoding algorithm corresponding to the encoding algorithm adopted by the encoding end, for example Entropy decoding.
- the data of the 3D map further includes reconstruction data of a plurality of region descriptors, and any one region descriptor in the plurality of region descriptors is used to describe the plurality of 3D map points Features of some or all of the 3D map points in .
- the reconstruction data of any 3D map point among the plurality of 3D map points includes reconstruction data of a 3D map point descriptor and reconstruction data of a spatial position of the 3D map point.
- the device for decoding the 3D map is a cloud server or an electronic device; the transmission module is further configured to send a 3D map download request, and the 3D map download request includes position indication information; receiving and The code stream of the 3D map corresponding to the location indication information.
- the device for decoding the 3D map is a cloud server; the transmission module is specifically configured to receive a code stream of the 3D map created by an electronic device.
- the decompression module includes a decapsulation module, and a prediction module and/or an inverse quantization module; the decapsulation module is configured to process the input code stream of the 3D map to Outputting the first data; the prediction module is used to perform prediction processing on the residual data of the input second data to output the second data; the dequantization module is used to dequantize the input third data Processing to output dequantized data of the third data; wherein, the first data is residual data of the second data or the third data; the second data is reconstruction data of the 3D map Or the third data; the inverse quantization data of the third data is the reconstruction data of the 3D map.
- the decompression module only includes a decapsulation module, and an inverse quantization module or a prediction module.
- the decompression module includes a prediction module, an inverse quantization module, and an encapsulation module.
- the prediction module includes a first prediction module and a second prediction module; and/or, the inverse quantization module includes a first inverse quantization module and a second inverse quantization module; the first The prediction module is configured to perform prediction processing on residual data of the input fourth data to output the fourth data; the first inverse quantization module is configured to perform inverse quantization processing on the input fifth data to output the The dequantization data of the fifth data; wherein, the fourth data is the reconstruction data of one of the multiple region descriptors or the fifth data; the dequantization data of the fifth data is the multiple regions The reconstruction data of one of the descriptors; the second prediction module is used to perform prediction processing on the residual data of the input sixth data to output the sixth data; the second dequantization module is used to performing dequantization processing on the input seventh data to output dequantized data of the seventh data; wherein, the sixth data is reconstruction data of one of the plurality of 3D map points or the seventh data ; The inverse quantization data of the seventh data is the reconstruction
- the decompression module only includes the first prediction module and the second prediction module, or only includes the first inverse quantization module and the second inverse quantization module.
- the decompression module includes a first prediction module and a second prediction module, and a first inverse quantization module and a second inverse quantization module.
- the prediction module includes a first prediction module, a second prediction module, and a third prediction module; and/or, the inverse quantization module includes a first inverse quantization module, a second inverse quantization module and a third inverse quantization module; the first prediction module is used to perform prediction processing on the residual data of the input eighth data to output the eighth data; the first inverse quantization module is used to input the eighth data
- the ninth data is dequantized to output the dequantized data of the ninth data; wherein, the eighth data is reconstruction data of one of the plurality of region descriptors or the ninth data; the ninth data
- the inverse quantization data of the data is the reconstruction data of one of the plurality of region descriptors; the second prediction module is configured to perform prediction processing on the input residual data of the tenth data to output the tenth data ;
- the second inverse quantization module is used to perform inverse quantization processing on the input eleventh data to output the inverse quantization data of the eleventh data; wherein, the tenth data
- the decompression module only includes the first prediction module, the second prediction module and the third prediction module, or only includes the first inverse quantization module, the second inverse quantization module and the third inverse quantization module.
- the decompression module includes a first prediction module, a second prediction module, and a third prediction module, and a first inverse quantization module, a second inverse quantization module, and a third inverse quantization module.
- the decompression module includes a first decompression submodule and a second decompression submodule; the first decompression submodule is used to process the code stream of the input fourteenth data Perform decompression processing to output the fourteenth data; the second decompression submodule is configured to compress the input code stream of the fifteenth data to output the fifteenth data; wherein, the The fourteenth data is the reconstruction data of one of the plurality of region descriptors; the fifteenth data is the reconstruction data of one of the plurality of 3D map points.
- the first decompression submodule includes a first decapsulation module, and a first prediction module and/or a first inverse quantization module;
- the second decompression submodule includes a second decompression module An encapsulation module, and a second prediction module and/or a second inverse quantization module;
- the first decapsulation module is configured to process the input code stream of the 3D map to output sixteenth data;
- the first The prediction module is used to perform prediction processing on the residual data of the input seventeenth data to output the seventeenth data;
- the first inverse quantization module is used to perform inverse quantization processing on the input eighteenth data to output output the dequantized data of the eighteenth data;
- the sixteenth data is the residual data of the seventeenth data or the eighteenth data;
- the seventeenth data is the plurality of The reconstruction data of one of the region descriptors or the eighteenth data;
- the inverse quantization data of the eighteenth data is the reconstruction data of one of the
- the decompression module includes a first decompression submodule, a second decompression submodule, and a third decompression submodule; the first decompression submodule is configured to The code stream of the twenty-second data is decompressed to output the twenty-second data; the second decompression submodule is used to compress the input code stream of the twenty-third data to output the The twenty-third data; the third decompression sub-module is used to compress the code stream of the input twenty-fourth data to output the twenty-fourth data; wherein, the twenty-second The data is the reconstruction data of one of the plurality of region descriptors; the twenty-third data is the reconstruction data of the 3D map point descriptor of one of the plurality of 3D map points; the twenty-fourth The data is reconstruction data of a spatial position of a 3D map point of one of the plurality of 3D map points.
- the first decompression submodule includes a first decapsulation module, and a first prediction module and/or a first inverse quantization module;
- the second decompression submodule includes a second decompression module The encapsulation module, and the second prediction module and/or the second dequantization module;
- the third decompression sub-module includes a third decapsulation module, and the third prediction module and/or the third dequantization module;
- the first The decapsulation module is used to process the input code stream of the 3D map to output the twenty-fifth data;
- the first prediction module is used to perform prediction processing on the residual data of the input twenty-sixth data to output the twenty-sixth data;
- the first inverse quantization module is configured to perform inverse quantization processing on the input twenty-seventh data to output the inverse quantization data of the twenty-seventh data;
- the The twenty-fifth data is the residual data of the twenty-sixth data or the twenty-seventh data; the twenty
- Residual data or the thirtieth data; the twenty-ninth data is reconstruction data of a 3D map point descriptor of one of the plurality of 3D map points or the thirtieth data; the thirtieth data
- the dequantization data of the thirty data is the reconstructed data of the 3D map point descriptor of one of the plurality of 3D map points;
- the third decapsulation module is used to perform the input code stream of the 3D map Processing to output the thirty-first data;
- the third prediction module is used to perform prediction processing on the residual data of the input thirty-second data to output the thirty-second data;
- the third inverse quantization module for performing inverse quantization processing on the input thirty-third data to output the inverse quantization data of the thirty-third data;
- the thirty-first data is the residual data of the thirty-second data Or the 33rd data;
- the 32nd data is reconstruction data of the spatial position of a 3D map point of one of the plurality of 3D map points or the 33r
- the present application provides a method for decoding a 3D map, including: receiving a code stream of a 3D map, the 3D map including a plurality of 3D map points; decompressing the code stream of the 3D map to obtain the The reconstruction data of the 3D map, the reconstruction data of the 3D map includes the reconstruction data of the plurality of 3D map points.
- the decompression module can support the decompression processing of the compressed data of the 3D map, that is, the codec system can support the compression/decompression processing of the data of the 3D map, which can reduce the amount of data of the 3D map,
- transmitting compressed data of 3D maps can reduce the amount of transmitted data on the one hand, and reduce the bandwidth occupied in transmission on the other hand, improving the quality of 3D maps. transmission efficiency.
- the reconstruction data of the 3D map further includes reconstruction data of a plurality of region descriptors, and any one region descriptor in the plurality of region descriptors is used to describe the plurality of 3D maps Features of some or all 3D map points in points.
- the reconstruction data of any 3D map point among the plurality of 3D map points includes reconstruction data of a 3D map point descriptor and reconstruction data of a spatial position of the 3D map point.
- the method further includes: sending a 3D map download request, where the 3D map download request includes location indication information; and receiving the code stream of the 3D map includes: receiving the The code stream of the above 3D map.
- the receiving the code stream of the 3D map includes: receiving the code stream of the 3D map created by an electronic device.
- the decompressing the code stream of the 3D map to obtain the data of the 3D map includes: processing the code stream of the 3D map to obtain the first data; performing prediction processing on the residual data of the second data to obtain the second data, and/or performing inverse quantization processing on the third data to obtain inverse quantization data of the third data; wherein, the first data It is the residual data of the second data or the third data; the second data is the reconstruction data of the 3D map or the third data; the inverse quantization data of the third data is the 3D The reconstruction data of the map.
- the present application provides a 3D map decoding device, including: a memory for storing a compressed 3D map in the form of a bit stream, the 3D map includes a plurality of 3D map points, and the data of the 3D map Data including the plurality of 3D map points; a decoder, configured to decompress the compressed 3D map in the form of a bit stream to obtain a reconstructed 3D map, the reconstructed 3D map including the plurality of 3D maps Map points, the reconstructed 3D map data includes reconstructed data of the plurality of 3D map points.
- the reconstructed 3D map data further includes reconstructed data of a plurality of region descriptors, and any one region descriptor in the plurality of region descriptors is used to describe the plurality of 3D maps Features of some or all 3D map points in points.
- the reconstruction data of any 3D map point among the plurality of 3D map points includes reconstruction data of a 3D map point descriptor and reconstruction data of a spatial position of the 3D map point.
- the encoding device for the 3D map is a cloud server or an electronic device.
- the present application provides a computer-readable storage medium, which is characterized by comprising a computer program, and when the computer program is executed on a computer, the computer executes the method described in any one of the above-mentioned third aspects. method.
- the present application provides a computer program product, the computer program product includes computer program code, and when the computer program code is run on a computer, it causes the computer to execute the method described in any one of the above third aspects .
- the present application provides a coded code stream of a 3D map, where the coded code stream of the 3D map includes a code stream of multiple 3D map points, and the 3D map includes the multiple 3D map points.
- the encoded code stream of the 3D map further includes code streams of multiple region descriptors, and any one of the multiple region descriptors and the multiple 3D map points Corresponds to at least one 3D map point in .
- the code stream of any 3D map point among the plurality of 3D map points includes a code stream of a 3D map point descriptor and a code stream of a spatial position of a 3D map point.
- the code stream of any 3D map point among the plurality of 3D map points includes residual data of the 3D map point.
- the code stream of the 3D map point descriptor includes the residual data of the 3D map point descriptor; and/or, the code stream of the 3D map point spatial position includes the 3D The residual data of the spatial position of the map point.
- the code stream of any 3D map point among the multiple 3D map points further includes indication information of a reference 3D map point
- the multiple 3D map points include the reference 3D map point
- the reference 3D map point is a 3D map point that has been encoded before any one of the 3D map points is encoded.
- the code stream of any 3D map point among the plurality of 3D map points includes quantized data of the 3D map point.
- the code stream of the 3D map point descriptor includes quantized data of the 3D map point descriptor; and/or, the code stream of the 3D map point spatial position includes the 3D map Quantified data of point spatial location.
- the code stream of any one of the multiple region descriptors includes residual data of the region descriptor.
- the code stream of any one of the multiple region descriptors further includes indication information of a reference region descriptor, and the multiple region descriptors include the reference region descriptor .
- the code stream of any one of the multiple region descriptors includes quantized data of the region descriptor.
- FIG. 1 is a schematic diagram of an application architecture provided by an embodiment of the present application
- FIG. 2 is a schematic structural diagram of an electronic device 20 provided in an embodiment of the present application.
- FIG. 3 is a schematic structural diagram of a server 30 provided in an embodiment of the present application.
- Fig. 4a is a schematic diagram of an application scenario provided by the embodiment of the present application.
- Fig. 4b is a schematic diagram of an application scenario provided by the embodiment of the present application.
- Fig. 4c is a schematic diagram of an application scenario provided by the embodiment of the present application.
- Fig. 4d is a schematic diagram of an application scenario provided by the embodiment of the present application.
- FIG. 4e is a schematic diagram of an application scenario provided by the embodiment of the present application.
- Fig. 4f is a schematic diagram of an application scenario provided by the embodiment of the present application.
- FIG. 4g is a schematic diagram of a user interface displayed by the electronic device provided in the embodiment of the present application.
- FIG. 5 is a structural diagram of a 3D map decoding device 50 provided in an embodiment of the present application.
- FIG. 6a is a structural diagram of a 3D map decoding device 60 provided by an embodiment of the present application.
- FIG. 6b is a structural diagram of a 3D map decoding device 60 provided by an embodiment of the present application.
- FIG. 6c is a structural diagram of a 3D map decoding device 60 provided by an embodiment of the present application.
- FIG. 6d is a structural diagram of a 3D map decoding device 60 provided by the embodiment of the present application.
- FIG. 6e is a structural diagram of a 3D map decoding device 60 provided by an embodiment of the present application.
- FIG. 7a is a structural diagram of a 3D map decoding device 70 provided by an embodiment of the present application.
- FIG. 7b is a structural diagram of a 3D map decoding device 70 provided by an embodiment of the present application.
- FIG. 8a is a structural diagram of a 3D map decoding device 80 provided by an embodiment of the present application.
- FIG. 8b is a structural diagram of a 3D map decoding device 80 provided by an embodiment of the present application.
- FIG. 9 is a flowchart of a process 900 of a decoding method for a 3D map provided in an embodiment of the present application.
- FIG. 10 is a structural diagram of a 3D map decoding device 100 provided by an embodiment of the present application.
- Fig. 11a is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
- Fig. 11b is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
- Fig. 11c is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
- Fig. 11d is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
- Fig. 11e is an exemplary structural diagram of a coding code stream of a 3D map provided by an embodiment of the present application.
- At least one (item) means one or more, and “multiple” means two or more.
- “And/or” is used to describe the association relationship of associated objects, indicating that there can be three types of relationships, for example, “A and/or B” can mean: only A exists, only B exists, and A and B exist at the same time , where A and B can be singular or plural.
- the character “/” generally indicates that the contextual objects are an “or” relationship.
- At least one of the following” or similar expressions refer to any combination of these items, including any combination of single or plural items.
- At least one item (piece) of a, b or c can mean: a, b, c, "a and b", “a and c", “b and c", or "a and b and c ", where a, b, c can be single or multiple.
- Fig. 1 is a schematic diagram of an application architecture provided by an embodiment of the present application.
- the application architecture includes a plurality of electronic devices and servers, wherein the plurality of electronic devices may include a first electronic device, one or more A second electronic device (two second electronic devices are taken as an example in FIG. 1), the one or more second electronic devices are several electronic devices other than the first electronic device. Multiple electronic devices and servers, and multiple electronic devices can communicate with each other.
- wireless-fidelity Wireless-fidelity
- WiFi wireless-fidelity
- Bluetooth Bluetooth
- cellular 2/3/4/5 generation 2/3/4/5generation, 2G/3G/4G/5G
- future communication methods which are not specifically limited.
- one or more second electronic devices in this embodiment of the present application is only used to refer to other electronic devices except the first electronic device, but does not limit whether the types of multiple electronic devices are the same.
- the above-mentioned electronic equipment can be various types of equipment equipped with cameras and display components.
- the electronic equipment can be terminal equipment such as mobile phones, tablet computers, notebook computers, video recorders (in FIG. 1, the electronic equipment is a mobile phone as an example),
- Electronic devices can also be devices used for virtual scene interaction, including VR glasses, AR devices, MR interactive devices, etc.
- Electronic devices can also be wearable electronic devices such as smart watches and smart bracelets.
- Electronic devices can also be vehicles, wireless Equipment carried in vehicles such as human-driven vehicles, drones, and industrial robots. The embodiment of the present application does not specifically limit the specific form of the electronic device.
- the above-mentioned electronic devices may also be referred to as user equipment (user equipment, UE), subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station , terminal device, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.
- user equipment user equipment
- UE user equipment
- subscriber station mobile unit
- subscriber unit wireless unit
- remote unit mobile device
- wireless device wireless device
- wireless communication device remote device
- mobile subscriber station terminal device, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.
- the above-mentioned servers may be one or more physical servers (a physical server is taken as an example in FIG. 1 ), may also be a computer cluster, or may be a virtual machine or a cloud server in a cloud computing scenario, and the like.
- the electronic device can install virtual scene applications (applications, APPs) such as VR applications, AR applications, or MR applications, and can run VR applications based on user operations (such as clicking, touching, sliding, shaking, voice control, etc.).
- application, AR application or MR application can collect the visual information of any object in the environment through the sensor, and then display the virtual object on the display component according to the collected visual information.
- the virtual object can be a virtual object in a VR scene, an AR scene or an MR scene (that is, a objects).
- the electronic device can install navigation, detection, and control application programs, and run corresponding applications based on user manipulation or preset programs.
- Electronic devices can perform applications such as path planning, object detection, and vehicle control based on their own pose and other state information in the current environment.
- the visual information involved in the embodiments of the present application includes, but is not limited to, images and videos collected by cameras (without depth information), images and videos with depth information collected by depth sensors, and laser radar (LiDAR) acquisitions.
- the virtual scene application program in the electronic device may be an application program built in the electronic device itself, or an application program provided by a third-party service provider installed by the user himself, and no specific details are given here. limited.
- the electronic device can also be equipped with a simultaneous localization and mapping (SLAM) system.
- the SLAM system can create a map in a completely unknown environment, and use the map to perform positioning, pose (position and attitude) determination, navigation, etc.
- the map created by the SLAM system is called a SLAM map.
- the SLAM map can be understood as a map drawn by the SLAM system based on the environmental information collected by the collection device.
- the collection device may include a visual information collection device in an electronic device and Inertial measurement unit (inertial measurement unit, IMU), wherein, the visual information acquisition device may include, for example, a camera, a camera, a depth camera, a lidar, a millimeter-wave radar, etc., and the IMU may include, for example, sensors such as a gyroscope and an accelerometer.
- IMU Inertial measurement unit
- the SLAM map is also referred to as a 3D map. It should be noted that the 3D map includes but is not limited to a SLAM map, and may also include a 3D map created using other technologies, which is not specifically limited in this embodiment of the present application.
- the 3D map may include multiple 3D map points, and correspondingly, data of the 3D map may include data of multiple 3D map points.
- a 3D map point is an interesting or salient feature point in the environment.
- 3D map points are to use various equipment such as lidar, aerial photography (oblique photography) from the perspective of drones, high-definition panoramic cameras, high-definition industrial cameras, etc., through accelerated segmentation test-oriented features ( features from accelerated segment test, FAST) and rotation binary robust independent elementary features (binary robust independent elementary features, BRIEF) descriptor features (oriented FAST and rotated BRIEF, ORB), scale invariant feature transform (scale invariant feature transform, SIFT), accelerated robust features (speeded-up robust feature, SURF), BRIEF, binary robust invariant scalable keypoints (binary robust invariant scalable keypoints, BRISK), fast retina keypoints (fast retina keypoint , FREAK), repeatable and reliable detector and descriptor (repeatable and reliable detector and descriptor, R2D2) and other methods are extracted from the data captured by the above equipment.
- accelerated segmentation test-oriented features features from accelerated segment test, FAST
- BRIEF binary robust independent elementary
- the data of 3D map points can include:
- the 3D map point descriptor is a vector used to represent the local features of the 3D map point.
- the 3D map point descriptor is used for matching between 3D map points.
- One possible approach is to calculate the distance between two 3D map point descriptors (it can be Euclidean distance, inner product distance, Hamming distance, etc.), and when the distance is less than a threshold, it is considered that two 3D map points match .
- the spatial position of a 3D map point can be represented by X, Y, and Z on the axis of the three-dimensional space, it can also be represented by latitude and longitude, and altitude, and it can also be represented by polar coordinates.
- the embodiment of the present application expresses the spatial position of a 3D map point No specific restrictions are made.
- the spatial position of the 3D map point can be the absolute position of the 3D map point, or the relative position of the 3D map point. For example, taking the center position of the entire area as the origin, all the spatial positions of the 3D map point are relative to the spatial position of the origin offset position.
- each 3D map point can be assigned a number and written into the data of the 3D map, and the storage order of multiple 3D map points in the memory can also be used to implicitly represent the number of the 3D map point.
- the order of multiple 3D map points contained in a 3D map has no practical significance, so the aforementioned numbers can be considered as identifiers for identifying 3D map points to distinguish each 3D map point, but this Numbering is not used to limit the order of multiple 3D map points.
- a 3D map contains 3 3D map points, and their numbers are 1, 2, and 3 respectively.
- the processing of these 3 3D map points can be done according to The order of 1, 2, 3 can also be performed in the order of 3, 2, 1, and the order of 2, 1, 3 can also be performed, and so on.
- the data of the 3D map further includes a plurality of region descriptors, and any one of the region descriptors in the plurality of region descriptors is used to describe part or all of the 3D map in the plurality of 3D map points
- the feature of the point that is, for any one of the multiple area descriptors, the area descriptor can be used to describe the features of some or all of the 3D map points in the multiple 3D map points, so that the area descriptor and the 3D map A point is a one-to-many relationship.
- each 3D map point among the multiple 3D map points can be described by some or all of the multiple region descriptors, so that the 3D map point and the region descriptor have a one-to-many relationship. It can be seen that there is a many-to-many relationship between multiple region descriptors and multiple 3D map points.
- the generation methods of regional descriptors include but are not limited to traditional methods such as bag of words (BOW), vector of aggregate locally descriptor (VLAD), and methods based on NetVLAD and artificial intelligence (AI). new method.
- BOW bag of words
- VLAD vector of aggregate locally descriptor
- AI artificial intelligence
- multiple region descriptors may also be identified with a number to distinguish the multiple region descriptors, but the number is also not a limitation on the order of the multiple region descriptors.
- the data of the 3D map also includes the correspondence between the 3D map points and the region descriptors, which explicitly describe which 3D map points any one region descriptor corresponds to, and any one Which region descriptors the 3D map points correspond to.
- the above correspondence can be explicitly described by a correspondence table between the number of the region descriptor and the number of the 3D map point.
- the 3D map contains 3 region descriptors, the numbers are T1 ⁇ T3, and there are 6
- the 3D map points are numbered from P1 to P6, and the corresponding table is shown in Table 1 or Table 2.
- Table 1 is an example of the correspondence table between the number of the region descriptor and the number of the 3D map point, and the correspondence table can also be presented in other formats or ways, which is not specifically limited in this application.
- the above corresponding relationship can also be implicitly described by the storage location of the region descriptor and the 3D map point. For example, first store T1 in the memory, then store the data of P1, P2 and P3, then store T2, and then store The data of P2 and P3 are stored, and finally T3 is stored, followed by the data of P3, P4, P5 and P6.
- FIG. 2 is a schematic structural diagram of an electronic device 20 provided by an embodiment of the present application.
- the electronic device 20 may be the first electronic device and one or more second electronic devices in the embodiment shown in FIG. 1 at least one of the .
- the structure shown in FIG. 2 does not constitute a specific limitation on the electronic device 20 .
- the electronic device 20 may include more or fewer components than the structure shown in FIG. 2 , or combine certain components, or separate certain components, or arrange different components.
- the various components shown in Figure 2 may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.
- the electronic device 20 may include: a chip 21, a memory 22 (one or more computer-readable storage media), a user interface 23, a display component 24, a camera 25, a sensor 26, a positioning module 27 for positioning the device, and a communication device. Transceiver 28. Communication between these components may be via one or more buses 29 .
- the chip 21 may include: one or more processors 211 , a clock module 212 and a power management module 213 .
- the clock module 212 integrated in the chip 21 is mainly used to provide the processor 211 with a timer required for data transmission and timing control, and the timer can realize the clock function of data transmission and timing control.
- the processor 211 can perform operations according to instruction opcodes and timing signals, generate operation control signals, and complete control of fetching and executing instructions.
- the power management module 213 integrated in the chip 21 is mainly used to provide stable and high-precision voltage for the chip 21 and other components of the electronic device 20 .
- the processor 211 can also be called a central processing unit (central processing unit, CPU), and the processor 211 can specifically include one or more processing units.
- the processor 211 can include an application processor (application processor, AP). Tuning processor, graphics processing unit (graphics processing unit, GPU), image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor , and/or, a neural network processor (neural-network processing unit, NPU), etc.
- different processing units may be independent devices, or may be integrated in one or more processors.
- the processor 211 may include one or more interfaces.
- the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or, a universal serial bus (universal serial bus, USB) interface, etc.
- I2C integrated circuit
- I2S integrated circuit built-in audio
- PCM pulse code modulation
- PCM pulse code modulation
- UART universal asynchronous transmitter
- MIPI mobile industry processor interface
- GPIO general-purpose input and output
- subscriber identity module subscriber identity module
- SIM subscriber identity module
- USB universal serial bus
- the memory 22 can be connected to the processor 211 through the bus 29, or can be coupled with the processor 311, and is used for storing various software programs and/or multiple sets of instructions.
- the memory 22 may include a high-speed random access memory (such as a cache memory), or may include a non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices or other non-volatile solid-state storage devices.
- the memory 22 can store an operating system, for example, embedded operating systems such as Android (Android), Apple mobile platform (IOS), Microsoft window operating system (Windows) or UNIX-like operating system (Linux).
- the memory 22 can also store data, for example, image data, point cloud data, 3D map data, pose data, coordinate system transformation information, map update information, and the like.
- the memory 22 may also store computer-executable program codes, where the computer-executable program codes include instructions, for example, communication program instructions, related program instructions of the SLAM system, and the like.
- the memory 22 can also store one or more application programs, such as virtual scene application programs such as AR/VR/MR, 3D map application programs, image management application programs, navigation and control application programs, and the like.
- the memory 22 can also store a user interface program.
- the user interface program can display the content of the application program through a graphical operation interface, for example, virtual objects in virtual scenes such as AR/VR/MR. 24 presentation, and realize receiving the user's control operations on the application through input controls such as menus, dialog boxes, and buttons.
- the user interface 23 may be, for example, a touch panel, and the touch panel may detect a user's operation instruction thereon, and the user interface 23 may also be, for example, a keypad, a physical button, or a mouse.
- Electronic device 20 may include one or more display components 24 .
- the electronic device 20 may jointly implement a display function through a display component 24 , a graphics processing unit (GPU) and an application processor (AP) in the chip 21 .
- the GPU is a microprocessor that implements image processing and is connected to the display assembly 24 and the application processor.
- the GPU performs mathematical and geometric calculations for graphics rendering.
- the display component 24 can display the interface content output by the electronic device 20, for example, display images, videos, etc. in virtual scenes such as AR/VR/MR, and the interface content can include the interface of the running application program and the system level menu, etc.
- interface elements such as buttons (Button), text input boxes (Text), sliders (Scroll Bar), menus (Menu), etc.; output interface elements, such as windows (Window), labels (Label), image, video, animation, etc.
- the display component 24 can be a display panel, glasses (such as VR glasses), projection screen and so on.
- the display panel may also be called a display screen, for example, may be a touch screen, a flexible screen, a curved screen, etc., or may be other optical components.
- the display screen of the electronic device in the embodiment of the present application may be a touch screen, flexible screen, curved screen or other forms of screen, that is, the display screen of the electronic device has the function of displaying images, and the specific The material and shape are not specifically limited.
- the display panel can be a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (organic light-emitting diode, OLED), an active matrix organic light-emitting diode or an active matrix Organic Light Emitting Diode (active-matrix organic light emitting diode, AMOLED), flexible light emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED )Wait.
- liquid crystal display liquid crystal display
- OLED organic light-emitting diode
- AMOLED active matrix organic light-emitting diode
- FLED flexible light emitting diode
- Miniled MicroLed, Micro-oLed
- quantum dot light emitting diodes quantum dot light emitting diodes (quantum dot light emitting diodes, QLED )Wait.
- the touch panel in the user interface 23 and the display panel in the display component 24 can be coupled together, for example, the touch panel can be arranged under the display panel, and the touch panel is used for Detecting the touch pressure on the display panel when the user inputs a touch operation (such as clicking, sliding, touching, etc.) through the display panel, and the display panel is used for displaying content.
- a touch operation such as clicking, sliding, touching, etc.
- the camera 25 may be a monocular camera, a binocular camera or a depth camera, and is used for shooting/recording the environment to obtain images/video images.
- the image/video image collected by the camera 25 can be used as a kind of input data of the SLAM system, or can be displayed by the display component 24 for image/video.
- the camera 25 can also be regarded as a sensor.
- the image collected by the camera 25 may be in the IMG format or in other formats, which is not specifically limited in this embodiment of the present application.
- the sensor 26 can be used to collect data related to state changes of the electronic device 20 (for example, rotation, swing, movement, shaking, etc.), and the data collected by the sensor 26 can be used as an input data of the SLAM system.
- the sensor 26 may include one or more sensors, for example, an inertial measurement unit (inertial measurement unit, IMU), a time of flight (time of flight, TOF) sensor, and the like.
- the IMU may include sensors such as a gyroscope and an accelerometer.
- the gyroscope is used to measure the angular velocity of the electronic device when it is in motion
- the accelerometer is used to measure the acceleration of the electronic device when it is in motion.
- the TOF sensor can include a light emitter and a light receiver.
- the light emitter is used to emit light, such as laser, infrared, radar waves, etc.
- the light receiver is used to detect reflected light, such as reflected laser, infrared, and radar waves. Wait.
- the sensor 26 may also include more other sensors, such as inertial sensors, barometers, magnetometers, wheel speedometers, etc., which are not specifically limited in this embodiment of the present application.
- the positioning module 27 is used to realize the physical positioning of the electronic device 20 , for example, to acquire the initial position of the electronic device 20 .
- the positioning module 27 may include one or more of a WiFi positioning module, a Bluetooth positioning module, a base station positioning module, and a satellite positioning module.
- the satellite positioning module can be provided with a global navigation satellite system (global navigation satellite system, GNSS) to assist positioning, and GNSS is not limited to the Beidou system, the global positioning system (global positioning system, GPS) system, the GLONASS (global navigation satellite system) , GLONASS) system, Galileo satellite navigation system (Galileo) system.
- the transceiver 28 is used to enable communication between the electronic device 20 and other devices (eg, servers, other electronic devices, etc.).
- the transceiver 28 integrates a transmitter and a receiver for sending and receiving radio frequency signals, respectively.
- the transceiver 28 includes, but is not limited to: an antenna system, a radio frequency (radio frequency, RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec ( CODEC) chip, subscriber identification module (SIM) card and storage media, etc.
- the transceiver 28 can also be implemented on a separate chip.
- the transceiver 28 supports at least one data network communication in 2G/3G/4G/5G, etc., and/or supports at least one of the following short-range wireless communication methods: Bluetooth (bluetooth, BT) communication, wireless fidelity (wireless fidelity, WiFi) communication, near field communication (near field communication, NFC), infrared (infrared, IR) wireless communication, ultra wide band (ultra wide band, UWB) communication, ZigBee protocol (ZigBee) communication.
- Bluetooth bluetooth, BT
- wireless fidelity wireless fidelity, WiFi
- NFC near field communication
- infrared infrared
- UWB ultra wide band
- ZigBee protocol ZigBee protocol
- the processor 211 executes various functional applications and data processing of the electronic device 20 by running the program code stored in the memory 22 .
- FIG. 3 is a schematic structural diagram of a server 30 provided in an embodiment of the present application.
- the server 30 may be the server in the embodiment shown in FIG. 1 .
- Server 30 includes processor 301 , memory 302 (one or more computer-readable storage media), and transceiver 303 . Communication between these components may be via one or more buses 304 .
- the processor 301 may be one or more CPUs. In the case where the processor 301 is one CPU, the CPU may be a single-core CPU or a multi-core CPU.
- the memory 302 can be connected to the processor 301 through the bus 304, or can be coupled with the processor 301, and is used to store various program codes and/or multiple sets of instructions, and data (eg, map data, pose data, etc.).
- the memory 302 includes, but is not limited to, random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), or portable read-only memory (Compact Disc Read-Only Memory, CD-ROM), etc.
- the transceiver 303 mainly integrates a receiver and a transmitter, where the receiver is used to receive data sent by the electronic device (for example, requests, images, etc.), and the transmitter is used to send data to the electronic device (for example, map data, pose data, etc.) .
- server 30 shown in FIG. 3 is only an example provided by the embodiment of the present application, and the server 30 may have more components than shown in the figure, which is not specifically limited in the embodiment of the present application.
- the processor 301 executes various functional applications and data processing of the server 30 by running the program code stored in the memory 302 .
- Coupled used in the embodiments of the present application means to be directly connected or connected through one or more intermediate components or circuits.
- Fig. 4a is a schematic diagram of an application scenario provided by the embodiment of the present application.
- the application scenario is that the electronic device collects visual information through a sensor, and determines the current pose of the electronic device in combination with the visual information and the 3D map from the server.
- the 3D map is provided by the server, that is, the server creates the 3D map, then compresses the 3D map, and transmits the compressed data of the 3D map to the electronic device; the electronic device performs decompression processing after receiving the compressed data of the 3D map, and obtains The reconstructed data of the 3D map is combined with the collected visual information and the 3D map to determine the current pose of the electronic device.
- the pose is the position information of the electronic device, which can be the absolute pose in the world coordinate system, or the relative pose relative to a certain point in the environment.
- the server may pre-create the 3D map, compress the 3D map and store it locally, which can save storage space.
- the server can also transmit the compressed data of the 3D map to other devices, such as cloud storage.
- the server creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
- Compressing the 3D map by the server can save local storage space.
- the electronic device sends a map download request to the server. There are two ways to trigger the map download request:
- the user opens the 3D map application installed on the electronic device, and the application uploads the location information based on GPS positioning or WiFi positioning to its corresponding server, and the upload operation can trigger a map download request. Since the uploaded content includes location information, the server can make a preliminary estimate based on the location information, and transmit the compressed data of the 3D map of the area to which the positioning point indicated by the location information belongs to the electronic device.
- the scope of the area to which the anchor point indicated by the location information can be preset.
- the area to which the anchor point belongs can be the administrative region of each level (including counties, cities, countries, or administrative districts, etc.) where the anchor point is located, or it can be centered on the anchor point , Set the distance as the circle area of the radius.
- the user opens the 3D map application program installed on the electronic device, and actively enters or selects an area in the application program, for example, the user actively enters "xx business center", or selects the area from a list "A street, B street , C Street” has selected "A Street”.
- the foregoing operations of the user may trigger a map download request.
- Either the user input or selection includes a geographic location, so the server transmits the compressed data of the 3D map of the geographic location to the electronic device.
- the electronic device automatically detects whether it meets the conditions for downloading a 3D map or starts downloading a 3D map, or the electronic device passes the detection A change in ambient light or a change in the environment is detected to start downloading the 3D map, so as to request the server to download a 3D map within an area, and the size of the area is not specifically limited.
- the server sends the compressed data of the 3D map to the electronic device.
- step 3 and step 4 are independent of each other, and the sequence is not limited.
- the electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- the electronic device After receiving the compressed data of the 3D map, the electronic device does not need to decompress it immediately, but only needs to decompress to obtain the reconstruction data of the 3D map before positioning based on the visual information. For example, the user can pre-download the compressed data of a 3D map in an area by downloading an "offline map", and decompress the compressed data of the 3D map when positioning is required.
- Figure 4b is a schematic diagram of an application scenario provided by the embodiment of the present application.
- the application scenario is that the electronic device collects visual information through a sensor, and the server determines the current position of the electronic device in combination with the visual information from the electronic device and a 3D map. posture.
- the 3D map is provided by the server, that is, the server creates the 3D map, then compresses the 3D map, and stores the compressed data of the 3D map locally.
- the server When receiving the visual information from the electronic device, the server performs decompression processing to obtain the reconstructed data of the 3D map, and determines the current pose of the electronic device by combining the visual information and the 3D map.
- the server creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
- the electronic device sends the visual information to the server.
- the server decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the server compresses the 3D map to save storage space.
- the server performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- the server sends the pose to the electronic device.
- Fig. 4c is a schematic diagram of an application scenario provided by the embodiment of the present application.
- the application scenario is that the electronic device collects visual information through a sensor, and determines the current pose of the electronic device in combination with the visual information and a 3D map.
- the 3D map is provided by the electronic device, that is, the electronic device creates the 3D map, then compresses the 3D map, and stores the compressed data of the 3D map locally.
- the electronic device performs decompression processing to obtain the reconstructed data of the 3D map, and combines the collected visual information and the 3D map to determine the current pose of the electronic device.
- the electronic device creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
- Electronic equipment collects visual information through sensors.
- the electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- Fig. 4d is a schematic diagram of an application scenario provided by an embodiment of the present application.
- the application scenario is that the second electronic device collects visual information through a sensor, and determines the location of the second electronic device in combination with the visual information and the 3D map from the server. current pose.
- the 3D map is created by the first electronic device, that is, the first electronic device creates the 3D map, compresses the 3D map, and then sends the compressed data of the 3D map to the server, and the server sends the compressed data of the 3D map to the second
- the second electronic device performs decompression processing to obtain the reconstruction data of the 3D map, and determines the current pose of the second electronic device in combination with the collected visual information and the 3D map.
- the first electronic device may pre-create a 3D map, compress the 3D map and then transmit it to the server, which can reduce transmission bandwidth.
- the first electronic device creates a 3D map and compresses it to obtain compressed data of the 3D map.
- the first electronic device sends the compressed data of the 3D map to the server.
- the first electronic device compresses the 3D map before transmitting it, which can reduce the transmission bandwidth and improve the transmission efficiency.
- the second electronic device sends a map download request to the server.
- the map download request sent by the second electronic device may also be based on the trigger manner shown in FIG. 4a.
- the server sends the compressed data of the 3D map to the second electronic device.
- the second electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the second electronic device collects the visual information through the sensor.
- the second electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- Fig. 4e is a schematic diagram of an application scenario provided by the embodiment of the present application.
- the application scenario is that the second electronic device collects visual information through a sensor, and the server combines the visual information from the second electronic device with the visual information from the first electronic device.
- the 3D map of the device determines the current pose of the second electronic device.
- the 3D map is created by the first electronic device, that is, the first electronic device creates the 3D map, compresses the 3D map, and then sends the compressed data of the 3D map to the server.
- the server performs decompression processing to obtain the reconstructed data of the 3D map, and combines the visual information from the second electronic device and the 3D map to determine the current pose of the second electronic device.
- the first electronic device creates a 3D map and compresses it to obtain compressed data of the 3D map.
- the first electronic device sends the compressed data of the 3D map to the server.
- the second electronic device acquires visual information through sensor collection.
- the second electronic device sends a positioning request to the server, and the positioning request carries visual information.
- the server decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the server performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- the server sends the pose obtained by positioning to the second electronic device.
- Fig. 4f is a schematic diagram of an application scenario provided by the embodiment of the present application.
- the application scenario is that the second electronic device collects visual information through a sensor, and combines the visual information and the 3D map from the first electronic device to determine the second The current pose of the electronic device.
- the 3D map is created by the first electronic device, that is, the first electronic device creates a 3D map, performs compression processing on the 3D map, and then sends the compressed data of the 3D map to the second electronic device, and the second electronic device performs decompression processing , obtain the reconstructed data of the 3D map, and determine the current pose of the second electronic device in combination with the collected visual information and the 3D map from the first electronic device.
- the first electronic device creates a 3D map, and compresses the compressed data of the 3D map to be stored locally.
- the second electronic device sends a map download request to the first electronic device.
- the first electronic device sends the compressed data of the 3D map to the second electronic device.
- the second electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
- the second electronic device acquires visual information through sensor collection.
- the second electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
- the positioning algorithm used may include:
- the distance between the region descriptor to be retrieved and each region descriptor in multiple region descriptors can be calculated, the distance can include Hamming distance, Manhattan distance or Euclidean distance, etc., and then select the matching condition (For example, at least one region descriptor whose distance is smaller than the threshold value) is a candidate region descriptor.
- the pose solving algorithm calculates the pose of the electronic device.
- FIG. 4a to FIG. 4f all involve compressing the 3D map.
- the embodiment of the present application provides a plurality of device frameworks for performing the aforementioned compression processing, and the multiple device frameworks will be described below.
- FIG. 4g is a schematic diagram of a user interface displayed by the electronic device provided in the embodiment of the present application.
- the electronic device can display the user interface shown in FIG. 4g based on the pose.
- the user interface may include a navigation arrow indication for navigating to the conference room 2, and the navigation arrow indication for navigating to the conference room 2 may be a virtual object obtained from the server or locally based on the pose.
- the user interface may also include visual information collected by sensors, for example, buildings as shown in FIG. 4g. The user goes to the meeting room 2 referring to the user interface shown in FIG. 4g of the electronic device.
- the reconstructed data of the 3D map obtained through decompression involved in the embodiment of the present application may also be referred to as the reconstructed data of the 3D map.
- FIG. 5 is a structural diagram of a 3D map decoding device 50 provided in the embodiment of the present application.
- a device that compresses the needs of the 3D map for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f, or, the server in the embodiment shown in Figure 4e .
- the 3D map decoding device 50 includes: a transmission module 51 and a decompression module 52; wherein, the transmission module 51 is used to receive the code stream of the 3D map, and the 3D map includes a plurality of 3D map points; the decompression module 52 , for decompressing the code stream of the 3D map to obtain reconstruction data of the 3D map, where the reconstruction data of the 3D map includes reconstruction data of multiple 3D map points.
- the data of the transmission module 51 is the code stream of the 3D map
- the decompression module 52 is used to decompress the code stream of the 3D map to obtain the reconstructed data of the 3D map.
- the decompression module 52 can support the decompression processing of the compressed data of the 3D map, that is, the codec system can support the compression/decompression processing of the data of the 3D map, which can reduce the amount of data of the 3D map.
- transmitting the compressed data of the 3D map can reduce the amount of transmitted data on the one hand, for example, compressing the data volume from TB level to GB level, on the other hand can reduce The bandwidth occupied during transmission improves the transmission efficiency of 3D maps.
- the decompression processing may include at least one of inverse quantization processing and prediction processing, wherein,
- Quantization processing refers to mapping the data to be processed into one or more quantization indexes. Each quantization index corresponds to a quantization center. The number of bits of the quantization index is usually significantly less than the number of bits of the original data, so as to save storage or transmission bandwidth. Purpose. Quantization processing methods include, but are not limited to: scalar quantization, vector quantization, product quantization, and the like. Based on this, the inverse quantization process is the reverse process of the aforementioned quantization process, that is, reconstructed data corresponding to the original data is restored according to one or more quantization indexes, and the number of bits of the reconstructed data is greater than the number of bits of the quantization index.
- Prediction processing refers to using the residual data of the processed data and the data to be processed to perform prediction to obtain the data to be processed.
- the selection of reference data can be pre-agreed, for example, fixedly use the previously processed data as reference data, at this time it is not necessary to mark the reference data in the code stream;
- the identification information of the reference data written in the stream including the serial number of the reference data, or other information from which the reference data can be deduced.
- the processing of the 3D map data includes encapsulation, the purpose of which is to encapsulate the data to be encoded into a code stream.
- the encapsulation process can use any encoding algorithm, such as entropy encoding.
- Entropy coding is a lossless data compression method.
- Entropy coding algorithms include but are not limited to: Huffman (huffman) coding, arithmetic coding, compression/decompression improved by LZ77 compression algorithm (lempel-ziv-markov chain-algorithm, LZMA ) algorithm, providing the function library (zlib) algorithm for data compression, etc.
- the decompression processing may also include decapsulation processing, the purpose of which is to decapsulate the code stream into data to be decompressed, and the decapsulation processing may use a decoding algorithm corresponding to the encoding algorithm adopted by the encoding end, for example Entropy decoding.
- the data of the 3D map obtained after decompression at the decoding end is not completely consistent with the original data of the 3D map before compression processing at the encoding end.
- the 3D map data output by the decompression module 52 in the embodiment of the present application can be referred to as 3D map reconstruction data, and the 3D map reconstruction data can refer to the description of the above 3D map data, which will not be repeated here.
- the embodiment of the present application provides multiple implementable structural examples of the decompression module, which will be described through the following embodiments.
- FIG. 6a is a structural diagram of a 3D map decoding device 60-1 provided in the embodiment of the present application.
- the decoding device 60-1 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
- the decoding device 60-1 includes a transmission module 61-1 and a decompression module 62-1, and the decompression module 62-1 includes: a decapsulation module 621-1 and an inverse quantization module 623-1.
- the decapsulation module 621-1 is used to process the code stream of the input 3D map to output the first data; the dequantization module 623-1 is used to dequantize the input third data to output the third data Inverse quantization data;
- the above-mentioned first data is the third data; the inverse quantization data of the third data is the reconstructed data of the 3D map.
- the input data of the decapsulation module 621-1 is the code stream of the 3D map, and the output data is the quantized data of the 3D map;
- the input data of the dequantization module 623-1 is the 3D map output by the decapsulation module 621-1 Quantify the data, and the output data is the reconstruction data of the 3D map.
- FIG. 6b is a structural diagram of a 3D map decoding device 60-2 provided in the embodiment of the present application.
- the decoding device 60-2 can be applied to the above-mentioned embodiments
- the server or electronic device there is a device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f, Or, the server in the embodiment shown in Fig. 4e.
- the decoding device 60-2 includes a transmission module 61-2 and a decompression module 62-2, and the decompression module 62-2 includes: a decapsulation module 621-2 and a prediction module 622 -1. in,
- the decapsulation module 621-2 is used to process the code stream of the input 3D map to output the first data; the prediction module 622-1 is used to perform prediction processing on the residual data of the input second data to output the second data. data.
- the above-mentioned first data is the residual data of the second data; the second data is the reconstructed data of the 3D map.
- the input data of the decapsulation module 621-2 is the code stream of the 3D map, and the output data is the residual data of the 3D map; the input data of the prediction module 622-1 is the code stream of the 3D map output by the decapsulation module 621-2. Residual data, the output data is the reconstruction data of the 3D map.
- FIG. 6c is a structural diagram of a 3D map decoding device 60-3 provided in the embodiment of the present application.
- the decoding device 60-3 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
- the decoding device 60-3 includes a transmission module 61-3 and a decompression module 62-3, and the decompression module 61-3 includes: a decapsulation module 621-3, a prediction module 622 -2 and dequantization module 623-2. in,
- the decapsulation module 621-3 is used to process the code stream of the input 3D map to output the first data; the prediction module 622-2 is used to perform prediction processing on the residual data of the input second data to output the second data. Data; an inverse quantization module 623-2, configured to perform inverse quantization processing on the input third data to output inverse quantization data of the third data.
- the above-mentioned first data is the residual data of the second data or the third data; the second data is the reconstructed data of the 3D map or the third data; the inverse quantization data of the third data is the reconstructed data of the 3D map.
- the input data of the decapsulation module 621-3 is the code stream of the 3D map, and the output data is the residual data of the 3D map; the input data of the prediction module 622-2 is the code stream of the 3D map output by the decapsulation module 621-3.
- the output data is the quantized data of the 3D map
- the input data of the inverse quantization module 623-2 is the quantized data of the 3D map output by the prediction module 622-2
- the output data is the reconstructed data of the 3D map.
- the reconstruction data of the 3D map may include the reconstruction data of multiple region descriptors and the reconstruction data of multiple 3D map points. Therefore, in the embodiments shown in FIGS. Modules except the decapsulation module can be split into a first module for processing region descriptors and a second module for processing 3D map point data.
- the dequantization module can include the first dequantization module
- the quantization module and the second inverse quantization module
- the prediction module may include a first prediction module and a second prediction module. The difference between the differentiated first module and the second module is that the input data correspond to the area descriptor and the data of the 3D map points respectively.
- Figure 6d is a structural diagram of the 3D map decoding device 60-4 provided by the embodiment of the present application.
- the decoding device 60-4 can be applied to the server or electronic device in the above embodiment, especially A device that needs to receive and decompress a 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f, or, the implementation shown in Figure 4e server in the example.
- the decoding device 60-4 includes a transmission module 61-4 and a decompression module 62-4, and the decompression module 62-4 includes: a decapsulation module 621-4, a prediction module 622 -3 and an inverse quantization module 623-3, the prediction module 622-3 includes a first prediction module 622a and a second prediction module 622b, and the inverse quantization module 623 includes a first inverse quantization module 623a and a second inverse quantization module 623b.
- the transmission module 61-4 is configured to receive the code stream of the above-mentioned 3D map.
- the input data of the decapsulation module 621-4 is the code stream of the 3D map, and the decapsulation module 621-4 decapsulates the code stream of the 3D map to obtain the residual data of the region descriptor and the residual data of the 3D map points.
- the input data of the first prediction module 622a is the residual data of the region descriptor, and the output data is the quantized data of the region descriptor;
- the input data of the first inverse quantization module 623a is the quantized data of the region descriptor, and the output data is the region descriptor reconstruction data.
- the input data of the second prediction module 622b is the residual data of the 3D map point, and the output data is the quantized data of the 3D map point;
- the input data of the second inverse quantization module 623b is the quantized data of the 3D map point, and the output data is the 3D map point reconstruction data.
- the reconstruction data of the 3D map may include the reconstruction data of multiple region descriptors, the reconstruction data of the 3D map point descriptors of multiple 3D map points and the reconstruction data of the spatial positions of the 3D map points , so in the embodiments shown in Figures 6a to 6c, other modules in the decompression module except the decapsulation module can be split into the first module for processing the region descriptor, and the first module for processing the 3D map point description
- the sub-second module and the third module for processing the spatial position of the 3D map point for example, the inverse quantization module may include the first inverse quantization module, the second inverse quantization module and the third inverse quantization module, and the prediction module may include the first A prediction module, a second prediction module and a third prediction module.
- the difference between the differentiated first module, second module and third module is that the input data corresponds to the region descriptor, the 3D map point descriptor and the spatial position of the 3D map point respectively.
- Fig. 6e is a structural diagram of a 3D map decoding device 60 provided by the embodiment of the present application.
- a device that compresses the needs of the 3D map for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f, or, the server in the embodiment shown in Figure 4e .
- the decoding device 60-5 includes a transmission module 61-5 and a decompression module 62-5
- the decompression module 62-5 includes: a decapsulation module 621-5, a prediction module 622 -4 and inverse quantization module 623-4
- prediction module 622-4 includes first prediction module 622a, second prediction module 622b and third prediction module 622c
- inverse quantization module 623-4 includes first inverse quantization module 623a, second An inverse quantization module 623b and a third inverse quantization module 623c.
- the transmission module 61-5 is configured to receive the code stream of the above-mentioned 3D map.
- the input data of the decapsulation module 621-5 is the code stream of the 3D map, and the decapsulation module 621-5 decapsulates the code stream of the 3D map to obtain the residual data of the region descriptor and the residual data of the 3D map point descriptor and the residual data of the spatial position of the 3D map point.
- the input data of the first prediction module 622a is the residual data of the region descriptor, and the output data is the quantized data of the region descriptor;
- the input data of the first inverse quantization module 623a is the quantized data of the region descriptor, and the output data is the region descriptor reconstruction data.
- the input data of the second prediction module 622b is the residual data of the 3D map point descriptor, and the output data is the quantized data of the 3D map point descriptor; the input data of the second inverse quantization module 623b is the quantized data of the 3D map point descriptor, The output data is the reconstructed data of the 3D map point descriptor.
- the input data of the third prediction module 622b is the residual data of the spatial position of the 3D map point, and the output data is the quantized data of the spatial position of the 3D map point; the input data of the third inverse quantization module 623b is the quantized data of the spatial position of the 3D map point, The output data is the reconstructed data of the spatial position of the 3D map points.
- FIG. 6d and FIG. 6e respectively show an exemplary structure of a 3D map decoding device, which is a structure obtained based on the content contained in the code stream of the 3D map, but this structure does not constitute a decoding
- the decoding device may include more modules than those shown in the embodiment shown in FIG. 6d or FIG. 6e.
- other structures are obtained based on the content contained in the code stream of the 3D map.
- the prediction module and the dequantization module are in an and/or relationship, that is, the decompression module may include one of the prediction module and the dequantization module, or the prediction module and the dequantization module Both are included. Therefore, for the case of distinguishing the first module and the second module, the processing module for the region descriptor and the processing module for the data of the 3D map point can independently set the processing method, and the two do not have to be completely consistent, for example, for the region descriptor , can be processed by the first prediction module and the first inverse quantization module respectively, and the data for 3D map points can be processed by the second prediction module.
- the processing module for the region descriptor, the processing module for the 3D map point descriptor and the processing module for the spatial position of the 3D map point can be independently set and processed
- the three methods do not have to be exactly the same.
- the region descriptor it can be processed by the first prediction module and the first inverse quantization module respectively;
- the 3D map point descriptor it can be processed by the second prediction module respectively;
- the 3D map The spatial position of the point can be processed by the third inverse quantization module.
- the present application does not specifically limit the specific implementation manner of the decompression module.
- the reconstruction data of the 3D map may include the reconstruction data of multiple area descriptors and the reconstruction data of multiple 3D map points.
- FIG. 7a is a structural diagram of a 3D map decoding device 70-1 provided in the embodiment of the present application.
- the decoding device 70-1 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
- the decoding device 70-1 includes a transmission module 71-1 and a decompression module 72-1, and the decompression module 72-1 includes: a first decompression sub-module 721-1 and The second decompression sub-module 722-1. in,
- the first decompression sub-module 721-1 is used to decompress the code stream of the input fourteenth data to output the fourteenth data;
- the second decompression sub-module 722-1 is used to decompress the input tenth data
- the code stream of the fifth data is compressed to output the fifteenth data;
- the fourteenth data is reconstruction data of one of the plurality of region descriptors; the fifteenth data is reconstruction data of one of the plurality of 3D map points.
- the input data of the first decompression sub-module 721-1 is the code stream of the region descriptor
- the output data is the reconstructed data of the region descriptor
- the input data of the second decompression sub-module 722-1 is the 3D map point
- the code stream, the output data is the reconstruction data of 3D map points.
- the code stream of the 3D map includes code streams of multiple area descriptors and code streams of multiple 3D map points.
- first decompression submodule 721-1 and the second decompression submodule 722-1 reference may be made to the structure of the decompression module in the embodiment shown in FIG. 6a to FIG. 6e. It should be noted that the first decompression submodule 721-1 and the second decompression submodule 722-1 are independent of each other, and the structures adopted by the two may be the same or different, that is, the first decompression submodule 722-1 for processing the region descriptor
- the structure of the first decompression sub-module 721-1 is the same as or different from the structure of the second decompression sub-module 722-1 for processing the data of 3D map points.
- the steps of decompression processing adopted by the data of the 3D map points may also be the same or different.
- the first decompression submodule includes a first decapsulation module and a first prediction module, so that after the code stream of the region descriptor is input into the first decompression submodule, it is decapsulated by the first decapsulation module to obtain the region descriptor
- the residual data, the residual data is processed by the first prediction module to obtain the reconstruction data of the region descriptor
- the second decompression sub-module includes the second decapsulation module, the second prediction module and the second inverse quantization module, so that the 3D
- the code stream of the map point is input into the second decompression sub-module, it is first processed by the second decapsulation module to obtain the residual data of the 3D map point, and then the residual data is processed by the second prediction module to obtain the quantized data of the 3D map point , and then the quantized data is processed by the second inverse quantization module to obtain reconstruction data of 3D map points.
- the first decompression sub-module includes a first decapsulation module, a first prediction module, and a first inverse quantization module, so that after the code stream of the region descriptor is input into the first decompression sub-module, it passes through the first decapsulation module respectively processing to obtain the residual data of the region descriptor, the residual data is processed by the first prediction module to obtain the quantized data of the region descriptor, and the quantized data is processed by the first inverse quantization module to obtain the reconstructed data of the region descriptor; the second solution
- the compression submodule includes a second decapsulation module and a second prediction module, after the description of the 3D map point is input into the second decompression submodule, it is processed by the second decapsulation module to obtain the residual data of the 3D map point, the residual data The reconstructed data of the 3D map points are obtained through processing by the second prediction module.
- FIG. 7b is a structural diagram of a 3D map decoding device 70-2 provided in the embodiment of the present application.
- the decoding device 70-2 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
- the decoding device 70-2 includes a transmission module 71-2 and a decompression module 72-2, and the decompression module 72-2 includes a first decompression sub-module 721-2 and a second decompression sub-module 721-2.
- the second decompression sub-module 722-2, the first decompression sub-module 721-2 includes the first decapsulation module 7211, the first prediction module 7212 and the first inverse quantization module 7213, the second decompression sub-module 722-2 includes the first Two decapsulation module 7221 , second prediction module 7222 and second inverse quantization module 7223 .
- the first decapsulation module 7211 is used to process the code stream of the input region descriptor to obtain the residual data of the region descriptor, and the first prediction module 7212 is used to predict the residual data of the input region descriptor Processing to obtain the quantized data of the region descriptor, the first inverse quantization module 7213 is configured to perform inverse quantization on the input quantized data of the region descriptor to obtain the reconstructed data of the region descriptor.
- the second decapsulation module 7221 is used to process the code stream of the input 3D map point to obtain the residual data of the 3D map point, and the second prediction module 7222 is used to predict the residual data of the input 3D map point.
- the second inverse quantization module 7223 is used to dequantize the input quantized data of the 3D map points to obtain the reconstruction data of the 3D map points.
- FIG. 7b only exemplifies the structure of the first decompression submodule and the second decompression submodule, but does not limit the structure of the two.
- These two submodules may include For more or fewer modules in the example, for details, refer to the structure of the decompression module 62 in the embodiment shown in FIG. 6a to FIG. 6e , which is not specifically limited in this embodiment of the present application.
- the reconstruction data of the 3D map may include reconstruction data of multiple region descriptors, reconstruction data of 3D map point descriptors of multiple 3D map points, and reconstruction data of spatial positions of 3D map points.
- FIG. 8a is a structural diagram of a 3D map decoding device 80-1 provided in the embodiment of the present application.
- the decoding device 80-1 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
- the decoding device 80-1 includes a transmission module 81-1 and a decompression module 82-1, and the decompression module 82-1 includes: a first decompression sub-module 821-1, The second decompression sub-module 822-1 and the third decompression sub-module 823-1.
- the first decompression sub-module 821-1 is used to decompress the code stream of the input twenty-second data to output the twenty-second data;
- the second decompression sub-module 822-1 is used to decompress the input code stream
- the code stream of the twenty-third data is compressed to output the twenty-third data;
- the third decompression sub-module 823-1 is used to compress the input code stream of the twenty-fourth data to output the twenty-third Four data.
- the 22nd data is the reconstruction data of one of the multiple region descriptors;
- the 23rd data is the reconstruction data of the 3D map point descriptor of one of the multiple 3D map points;
- the 24th The data is reconstruction data of a spatial position of a 3D map point of one of the plurality of 3D map points.
- the input data of the first decompression sub-module 821-1 is the code stream of the region descriptor
- the output data is the reconstructed data of the region descriptor
- the input data of the second decompression sub-module 812-1 is the 3D map point
- the code stream of the descriptor, the output data is the reconstruction data of the 3D map point descriptor
- the input data of the third decompression sub-module 813-1 is the code stream of the spatial position of the 3D map point
- the output data is the reconstruction of the spatial position of the 3D map point data.
- the code stream of the 3D map includes a code stream of multiple area descriptors, a code stream of multiple 3D map point descriptors, and a code stream of multiple spatial positions of 3D map points.
- first decompression submodule 821-1 For the first decompression submodule 821-1, the second decompression submodule 822-1, and the third decompression submodule 823-1, reference may be made to the structure of the decompression module in the embodiment shown in FIG. 6a to FIG. 6e. It should be noted that the first decompression submodule 821-1, the second decompression submodule 822-1 and the third decompression submodule 823-1 are independent of each other, and the structures adopted by the three may be the same or different.
- the structure of the first decompression submodule 821-1 for processing region descriptors, the second decompression submodule 822-1 for processing 3D map point descriptors, and the first decompression submodule 822-1 for processing 3D map point spatial positions The structures of the three decompression sub-modules 823-1 are the same or different.
- the decompression processing steps for the region descriptor, the decompression processing steps for the 3D map point descriptor, and the 3D map point space The steps of the decompression process adopted by the locations may also be the same or different.
- FIG. 8b is a structural diagram of a 3D map decoding device 80-2 provided in the embodiment of the present application.
- the decoding device 80-2 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
- the decoding device 80-2 includes a transmission module 81-2 and a decompression module 82-2, and the decompression module 82-2 includes a first decompression sub-module 821-2, a second The second decompression sub-module 822-2 and the third decompression sub-module 823-2, the first decompression sub-module 821-2 includes the first decapsulation module 8211, the first prediction module 8212 and the first dequantization module 8213, the first The second decompression sub-module 822-2 includes a second decapsulation module 8221, a second prediction module 8222, and a second inverse quantization module 8223, and the third decompression sub-module 823-2 includes a third decapsulation module 8231, a third prediction module 8232 and the third inverse quantization module 8233.
- the first decapsulation module 8211 is used to process the code stream of the input region descriptor to obtain the residual data of the region descriptor, and the first prediction module 8212 is used to predict the residual data of the input region descriptor To obtain the quantized data of the region descriptor, the first inverse quantization module 8213 is used to perform inverse quantization processing on the input quantized data of the region descriptor to obtain the reconstructed data of the region descriptor.
- the second decapsulation module 8221 is used to process the code stream of the input 3D map point descriptor to obtain the residual data of the 3D map point descriptor
- the second prediction module 8222 is used to process the input 3D map point descriptor Perform prediction processing on the residual data to obtain the quantized data of the 3D map point descriptor
- the second inverse quantization module 8223 is used to perform inverse quantization processing on the input quantized data of the 3D map point descriptor to obtain the quantized data of the 3D map point descriptor Rebuild data.
- the third decapsulation module 8231 is used to process the code stream of the input 3D map point spatial position to obtain the residual data of the 3D map point spatial position
- the third prediction module 8232 is used to process the input 3D map point spatial position Prediction processing is performed on the residual data to obtain the quantized data of the spatial position of the 3D map point
- the third inverse quantization module 8233 is used to perform inverse quantization processing on the quantized data of the input spatial position of the 3D map point to obtain the spatial position of the 3D map point Rebuild data.
- FIG. 8b only exemplifies the structures of the first decompression submodule, the second decompression submodule, and the third decompression submodule, but does not limit the structures of the three.
- These three sub-modules may include more or less modules than in the example, for details, refer to the structure of the decompression module 62 in the embodiment shown in Fig. 6a to Fig. 6e , which is not specifically limited in this embodiment of the present application.
- FIG. 9 is a flow chart of a process 900 of a method for decoding a 3D map provided in an embodiment of the present application.
- the process 900 may be executed by the decoding device in the foregoing embodiment.
- the process 900 is described as a series of steps or operations. It should be understood that the process 900 may be performed in various orders and/or concurrently, and is not limited to the order of execution shown in FIG. 9 .
- the process 900 including the following steps is executed to process the code stream of the 3D map currently being processed.
- Step 901 Receive a code stream of a 3D map.
- the decoding device can use the communication link to receive the code stream of the 3D map.
- Step 902 decompressing the code stream of the 3D map to obtain reconstruction data of the 3D map.
- the decompression processing performed on the code stream of the 3D map may include decapsulation processing, prediction processing and/or inverse quantization processing.
- decapsulation processing prediction processing
- inverse quantization processing For the various processes above, reference may be made to the description of the above embodiments, and details are not repeated here.
- FIG. 10 is a structural diagram of a 3D map decoding device 100 provided in the embodiment of the present application. As shown in FIG. The device required to compress the 3D map, for example, the server in the embodiment shown in FIG. 4b, or the electronic device in the embodiment shown in FIG. 4c.
- the 3D map decoding device 100 includes: a memory 101 for storing a compressed 3D map in the form of a bit stream, the 3D map includes a plurality of 3D map points, and the data of the 3D map includes the The data of the plurality of 3D map points; the decoder 102 is configured to decompress the compressed 3D map in the form of a bit stream to obtain a reconstructed 3D map, and the reconstructed 3D map includes the plurality of 3D maps points, the reconstructed 3D map data includes reconstructed data of the plurality of 3D map points.
- the memory 101 stores the compressed 3D map in the form of bit stream
- the input data of the decoder 102 is the compressed 3D map in the form of bit stream
- the output data is the reconstructed 3D map.
- the decoder 102 can support the decompression processing of the compressed 3D map in the form of bit stream, that is, the decoding device of the 3D map can support the compression/decompression processing of the data of the 3D map, which can reduce the data of the 3D map In the scene where the 3D map needs to be stored, compared with storing the original 3D map data, storing the compressed 3D map in the form of a bit stream can save the space occupied by storing the 3D map data.
- the decoder 102 may refer to the decompression module in the embodiment shown in FIG. 5 to FIG. 8 b , the difference is that the decoder does not include a decapsulation module.
- Figure 11a is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
- the encoded code stream of the 3D map may include: a file header, an area descriptor 1, and a 3D map point 1 , area descriptor 2, 3D map point 2, ..., area descriptor m, 3D map point n and other information.
- the 3D map includes m region descriptors and n 3D map points, and the region descriptors and 3D map points in the encoded code stream may be arranged in a crossed manner.
- Figure 11b is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
- the encoded code stream of the 3D map may include: file header, area descriptor 1, and area descriptor 2 , ..., area descriptor m, 3D map point 1, 3D map point 2, ..., 3D map point n and other information.
- the 3D map contains m region descriptors and n 3D map points, and the region descriptors and 3D map points are arranged separately in the encoded code stream.
- the region descriptors and 3D map points are arranged separately in the encoded code stream.
- coded code stream of the 3D map in the embodiment of the present application can also adopt other structures, and the specific structure is associated with the data content contained in the 3D map.
- the coded code stream of the 3D map does not contain the above file header , Region Descriptor and other information in one or more.
- Figure 11c is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
- the encoded code stream of the 3D map may include: area descriptor 1, area descriptor 2, and area description child 3.
- the structure of the coded code stream corresponds to the case where the above-mentioned decompression module does not include a prediction module, so there is no need to fill in the code stream with reference information for prediction.
- Figure 11d is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
- the encoded code stream of the 3D map may include: the reference area of the region descriptor 1 and the region descriptor 2 The index of the descriptor and the residual data of the region descriptor 2.
- the structure of the encoded code stream corresponds to the case where the above-mentioned decompression module includes a prediction module, so reference information for prediction needs to be filled in the code stream, for example, the index of the reference region descriptor of region descriptor 2, and the region descriptor 2 is to fill in its residual data instead of the original data.
- Figure 11e is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
- the encoded code stream of the 3D map may include: 3D map point 1, 3D map point 2 and 3D map Point 3.
- the code stream of the 3D map point 2 includes the 3D map point descriptor of the 3D map point 2 and the 3D map point spatial position of the 3D map point 2, if the structure of the encoded code stream corresponds to the above solution If the compression module includes the prediction module, the code stream of the 3D map point descriptor includes the reference index of the 3D map point descriptor and the residual data of the 3D map point descriptor, and the code stream of the spatial position of the 3D map point includes the 3D map point space The reference index of the position and the residual data of the spatial position of the 3D map point.
- the content contained in the code stream of the 3D map is also different according to the compression/decompression process of the data of the 3D map.
- the compression/decompression process includes quantization/dequantization
- the code stream of the 3D map includes quantized data
- the compression/decompression process includes quantization/dequantization and prediction
- the code stream of the 3D map includes the reference index and Residual data
- the compression/decompression process includes prediction
- the bitstream of the 3D map includes reference indexes and residual data. Therefore, the embodiment of the present application does not specifically limit the structure of the encoded code stream of the 3D map.
- each step of the above-mentioned method embodiments may be completed by an integrated logic circuit of hardware in a processor or instructions in the form of software.
- the processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other possible Program logic devices, discrete gate or transistor logic devices, discrete hardware components.
- a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
- the steps of the methods disclosed in the embodiments of the present application may be directly implemented by a hardware coded processor, or executed by a combination of hardware and software modules in the coded processor.
- the software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register.
- the storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware.
- the memories mentioned in the above embodiments may be volatile memories or nonvolatile memories, or may include both volatile and nonvolatile memories.
- the non-volatile memory can be read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically programmable Erases programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
- Volatile memory can be random access memory (RAM), which acts as external cache memory.
- RAM random access memory
- SRAM static random access memory
- DRAM dynamic random access memory
- DRAM synchronous dynamic random access memory
- SDRAM double data rate synchronous dynamic random access memory
- ESDRAM enhanced synchronous dynamic random access memory
- SLDRAM direct memory bus random access memory
- direct rambus RAM direct rambus RAM
- the disclosed systems, devices and methods may be implemented in other ways.
- the device embodiments described above are only illustrative.
- the division of the units is only a logical function division. In actual implementation, there may be other division methods.
- multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented.
- the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
- the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
- each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
- the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium.
- the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (personal computer, server, or network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
- the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc and other media that can store program codes. .
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Remote Sensing (AREA)
- Computer Graphics (AREA)
- Signal Processing (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
Claims (43)
- 一种3D地图的编解码系统,其特征在于,包括:编码装置和解码装置,所述编码装置和所述解码装置通信连接;其中,所述编码装置,用于对3D地图的数据进行压缩处理以得到所述3D地图的码流,并将所述3D地图的码流发送给所述解码装置,所述3D地图包括多个3D地图点,所述3D地图的数据包括所述多个3D地图点的数据;所述解码装置,用于接收所述3D地图的码流,并对所述3D地图的码流进行解压缩处理以得到所述3D地图的重构数据。
- 根据权利要求1所述的系统,其特征在于,所述编码装置为云端服务器,所述解码装置为电子设备;或者,所述编码装置为第一电子设备,所述解码装置为第二电子设备;所述解码装置,还用于向所述解码装置发送3D地图下载请求,所述3D地图下载请求包括位置指示信息;所述编码装置,还用于接收3D地图下载请求,并根据所述3D地图下载请求向所述解码装置发送与所述位置指示信息对应的3D地图的码流。
- 根据权利要求1或2所述的系统,其特征在于,所述编码装置为电子设备,所述解码装置为云端服务器;所述编码设备,具体用于在创建3D地图后,向所述解码装置发送所述3D地图的码流。
- 根据权利要求1-3中任一项所述的系统,其特征在于,所述3D地图的数据还包括多个区域描述子,所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
- 根据权利要求1或4所述的系统,其特征在于,所述多个3D地图点中的任意一个3D地图点的数据包括3D地图点描述子和3D地图点空间位置。
- 根据权利要求1-5中任一项所述的系统,其特征在于,所述编码装置,还用于创建所述3D地图。
- 根据权利要求1-6中任一项所述的系统,其特征在于,所述解码装置,还用于根据所述3D地图进行定位。
- 一种3D地图的解码装置,其特征在于,包括:传输模块和解压缩模块;其中,所述传输模块,用于接收3D地图的码流,所述3D地图包括多个3D地图点;所述解压缩模块,用于对所述3D地图的码流进行解压缩处理以得到所述3D地图的重建数据,所述3D地图的重建数据包括所述多个3D地图点的重建数据。
- 根据权利要求8所述的装置,其特征在于,所述3D地图的重建数据还包括多个区域描述子的重建数据,所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
- 根据权利要求9所述的装置,其特征在于,所述多个3D地图点中的任意一个3D地图点的重建数据包括3D地图点描述子的重建数据和3D地图点空间位置的重建数据。
- 根据权利要求8-10中任一项所述的装置,其特征在于,所述3D地图的解码装置为云端服务器或者电子设备;所述传输模块,还用于发送3D地图下载请求,所述3D地图下载请求包括位置指示信息;接收与所述位置指示信息对应的所述3D地图的码流。
- 根据权利要求8-10中任一项所述的装置,其特征在于,所述3D地图的解码装置为云端服务器;所述传输模块,具体用于接收由电子设备创建的所述3D地图的码流。
- 根据权利要求8-12中任一项所述的装置,其特征在于,所述解压缩模块包括解封装模块,以及预测模块和/或反量化模块;所述解封装模块,用于对输入的所述3D地图的码流进行处理以输出第一数据;所述预测模块,用于对输入的第二数据的残差数据进行预测处理以输出所述第二数据;所述反量化模块,用于对输入的第三数据进行反量化处理以输出所述第三数据的反量化数据;其中,所述第一数据为所述第二数据的残差数据或者所述第三数据;所述第二数据为所述3D地图的重建数据或者所述第三数据;所述第三数据的反量化数据为所述3D地图的重建数据。
- 根据权利要求13所述的装置,其特征在于,所述预测模块包括第一预测模块和第二预测模块;和/或,所述反量化模块包括第一反量化模块和第二反量化模块;所述第一预测模块,用于对输入的第四数据的残差数据进行预测处理以输出所述第四数据;所述第一反量化模块,用于对输入的第五数据进行反量化处理以输出所述第五数据的反量化数据;其中,所述第四数据为多个区域描述子的其中之一的重建数据或者所述第五数据;所述第五数据的反量化数据为所述多个区域描述子的其中之一的重建数据;所述第二预测模块,用于对输入的第六数据的残差数据进行预测处理以输出所述第六数据;所述第二反量化模块,用于对输入的第七数据进行反量化处理以输出所述第七数据的反量化数据;其中,所述第六数据为所述多个3D地图点的其中之一的重建数据或者所述第七数据;所述第七数据的反量化数据为所述多个3D地图点的其中之一的重建数据。
- 根据权利要求13所述的装置,其特征在于,所述预测模块包括第一预测模块、第二预测模块和第三预测模块;和/或,所述反量化模块包括第一反量化模块、第二反量化模块和第三反量化模块;所述第一预测模块,用于对输入的第八数据的残差数据进行预测处理以输出所述第八数据;所述第一反量化模块,用于对输入的第九数据进行反量化处理以输出所述第九数据的反量化数据;其中,所述第八数据为多个区域描述子的其中之一的重建数据或者所述第九数据;所述第九数据的反量化数据为所述多个区域描述子的其中之一的重建数据;所述第二预测模块,用于对输入的第十数据的残差数据进行预测处理以输出所述第十数据;所述第二反量化模块,用于对输入的第十一数据进行反量化处理以输出所述第十一数据的反量化数据;其中,所述第十数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据或者所述第十一数据;所述第十一数据的反量化数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据。所述第三预测模块,用于对输入的第十二数据的残差数据进行预测处理以输出所述第十二数据;所述第三反量化模块,用于对输入的第十三数据进行反量化处理以输出所述第十三数据的反量化数据;其中,所述第十二数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据或者所述第十三数据;所述第十三数据的反量化数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据。
- 根据权利要求8-12中任一项所述的装置,其特征在于,所述解压缩模块包括第一解压缩子模块和第二解压缩子模块;所述第一解压缩子模块,用于对输入的第十四数据的码流进行解压缩处理以输出所述第十四数据;所述第二解压缩子模块,用于对输入的第十五数据的码流进行压缩处理以输出所述第十五数据;其中,所述第十四数据为多个区域描述子的其中之一的重建数据;所述第十五数据为所述多个3D地图点的其中之一的重建数据。
- 根据权利要求16所述的装置,其特征在于,所述第一解压缩子模块包括第一解封装模块,以及第一预测模块和/或第一反量化模块;所述第二解压缩子模块包括第二解封装模块,以及第二预测模块和/或第二反量化模块;所述第一解封装模块,用于对输入的所述3D地图的码流进行处理以输出第十六数据;所述第一预测模块,用于对输入的第十七数据的残差数据进行预测处理以输出所述第十七数据;所述第一反量化模块,用于对输入的第十八数据进行反量化处理以输出所述第十八数据的反量化数据;其中,所述第十六数据为所述第十七数据的残差数据或者所述第十八数据;所述第十七数据为所述多个区域描述子的其中之一的重建数据或者所述第十八数据;所述第十八数据的反量化数据为所述多个区域描述子的其中之一的重建数据;所述第二解封装模块,用于对输入的所述3D地图的码流进行处理以输出第十九数据;所述第二预测模块,用于对输入的第二十数据的残差数据进行预测处理以输出所述第二十数据;所述第二反量化模块,用于对输入的第二十一数据进行反量化处理以输出所述第二十一数据的反量化数据;其中,所述第十九数据为所述第二十数据的残差数据或者所述第二十一数据;所述第二十数据为所述多个3D地图点的其中之一的重建数据或者所述第二十一数据;所述第二十一数据的反量化数据为所述多个3D地图点的其中之一的重建数据。
- 根据权利要求8-12中任一项所述的装置,其特征在于,所述解压缩模块包括第一解压缩子模块、第二解压缩子模块和第三解压缩子模块;所述第一解压缩子模块,用于对输入的第二十二数据的码流进行解压缩处理以输出所述第二十二数据;所述第二解压缩子模块,用于对输入的第二十三数据的码流进行压缩处理以输出所述第二十三数据;所述第三解压缩子模块,用于对输入的第二十四数据的码流进行压缩处理以输出所述第二十四数据;其中,所述第二十二数据为多个区域描述子的其中之一的重建数据;所述第二十三数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据;所述第二十四数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据。
- 根据权利要求18所述的装置,其特征在于,所述第一解压缩子模块包括第一解封装模块,以及第一预测模块和/或第一反量化模块;所述第二解压缩子模块包括第二解封装模块,以及第二预测模块和/或第二反量化模块;所述第三解压缩子模块包括第三解封装模块,以及第三预测模块和/或第三反量化模块;所述第一解封装模块,用于对输入的所述3D地图的码流进行处理以输出第二十五数据;所述第一预测模块,用于对输入的第二十六数据的残差数据进行预测处理以输出所述第二十六数据;所述第一反量化模块,用于对输入的第二十七数据进行反量化处理以输出所述第二十七数据的反量化数据;其中,所述第二十五数据为所述第二十六数据的残差数据或者所述第二十七数据;所述第二十六数据为所述多个区域描述子的其中之一的重建数据或者所述第二十七数据;所述第二十七数据的反量化数据为所述多个区域描述子的其中之一的重建数据;所述第二解封装模块,用于对输入的所述3D地图的码流进行处理以输出第二十八数据;所述第二预测模块,用于对输入的第二十九数据的残差数据进行预测处理以输出所述第二十九数据;所述第二反量化模块,用于对输入的第三十数据进行反量化处理以输出所述第三十数据的反量化数据;其中,所述第二十八数据为所述第二十九数据的残差数据或者所述第三十数据;所述第二十九数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据或者所述第三十数据;所述第三十数据的反量化数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据;所述第三解封装模块,用于对输入的所述3D地图的码流进行处理以输出第三十一数据;所述第三预测模块,用于对输入的第三十二数据的残差数据进行预测处理以输出所述第三十二数据;所述第三反量化模块,用于对输入的第三十三数据进行反量化处理以输出所述第三十 三数据的反量化数据;其中,所述第三十一数据为所述第三十二数据的残差数据或者所述第三十三数据;所述第三十二数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据或者所述第三十三数据;所述第三十三数据的反量化数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据。
- 一种3D地图的解码方法,其特征在于,包括:接收3D地图的码流,所述3D地图包括多个3D地图点;对所述3D地图的码流进行解压缩处理以得到所述3D地图的重建数据,所述3D地图的重建数据包括所述多个3D地图点的重建数据。
- 根据权利要求20所述的方法,其特征在于,所述3D地图的重建数据还包括多个区域描述子的重建数据,所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
- 根据权利要求21所述的方法,其特征在于,所述多个3D地图点中的任意一个3D地图点的重建数据包括3D地图点描述子的重建数据和3D地图点空间位置的重建数据。
- 根据权利要求20-22中任一项所述的方法,其特征在于,还包括:发送3D地图下载请求,所述3D地图下载请求包括位置指示信息;所述接收3D地图的码流,包括:接收与所述位置指示信息对应的所述3D地图的码流。
- 根据权利要求20-22中任一项所述的方法,其特征在于,所述接收3D地图的码流,包括:接收由电子设备创建的所述3D地图的码流。
- 根据权利要求20-24中任一项所述的方法,其特征在于,所述对所述3D地图的码流进行解压缩处理以得到所述3D地图的重建数据,包括:对所述3D地图的码流进行处理以得到第一数据;对第二数据的残差数据进行预测处理以得到所述第二数据,和/或,对第三数据进行反量化处理以得到所述第三数据的反量化数据;其中,所述第一数据为所述第二数据的残差数据或者所述第三数据;所述第二数据为所述3D地图的重建数据或者所述第三数据;所述第三数据的反量化数据为所述3D地图的重建数据。
- 一种3D地图的编码装置,其特征在于,包括:一个或多个处理器;存储器,用于存储一个或多个程序;当所述一个或多个程序被所述一个或多个处理器执行,使得所述一个或多个处理器实现如权利要求20-25中任一项所述的方法。
- 一种3D地图的解码装置,其特征在于,包括:存储器,用于存储比特流形式的经压缩得到的3D地图,所述3D地图包括多个3D地图点,所述3D地图的数据包括所述多个3D地图点的数据;解码器,用于对所述比特流形式的经压缩得到的3D地图进行解压缩处理,得到重建3D地图,所述重建3D地图包括所述多个3D地图点,所述重建3D地图的数据包括所述 多个3D地图点的重建数据。
- 根据权利要求27所述的装置,其特征在于,所述重建3D地图的数据还包括多个区域描述子的重建数据,所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
- 根据权利要求27或28所述的装置,其特征在于,所述多个3D地图点中的任意一个3D地图点的重建数据包括3D地图点描述子的重建数据和3D地图点空间位置的重建数据。
- 根据权利要求27-29中任一项所述的装置,其特征在于,所述3D地图的编码装置为云端服务器或者电子设备。
- 一种计算机可读存储介质,其特征在于,包括计算机程序,所述计算机程序在计算机上被执行时,使得所述计算机执行权利要求20-25中任一项所述的方法。
- 一种计算机程序产品,其特征在于,所述计算机程序产品包括计算机程序代码,当所述计算机程序代码在计算机上运行时,使得计算机执行权利要求20-25中任一项所述的方法。
- 一种3D地图的编码码流,其特征在于,所述3D地图的编码码流包括多个3D地图点的码流,所述3D地图包括所述多个3D地图点。
- 根据权利要求33所述的码流,其特征在于,其特征在于,所述3D地图的编码码流还包括多个区域描述子的码流,所述多个区域描述子中的任意一个区域描述子和所述多个3D地图点中的至少一个3D地图点对应。
- 根据权利要求34所述的码流,其特征在于,所述多个3D地图点中的任意一个3D地图点的码流包括3D地图点描述子的码流和3D地图点空间位置的码流。
- 根据权利要求33所述的方法,其特征在于,所述多个3D地图点中的任意一个3D地图点的码流包括3D地图点的残差数据。
- 根据权利要求35所述的码流,其特征在于,所述3D地图点描述子的码流包括所述3D地图点描述子的残差数据;和/或,所述3D地图点空间位置的码流包括所述3D地图点空间位置的残差数据。
- 根据权利要求36或37所述的码流,其特征在于,所述多个3D地图点中的任意一个3D地图点的码流还包括参考3D地图点的指示信息,所述多个3D地图点包括所述参考3D地图点,且所述参考3D地图点是在编码所述任意一个3D地图点之前已编码的3D地图点。
- 根据权利要求33所述的方法,其特征在于,所述多个3D地图点中的任意一个3D地图点的码流包括3D地图点的量化数据。
- 根据权利要求35所述的码流,其特征在于,所述3D地图点描述子的码流包括所述3D地图点描述子的量化数据;和/或,所述3D地图点空间位置的码流包括所述3D地图点空间位置的量化数据。
- 根据权利要求34所述的码流,其特征在于,所述多个区域描述子中的任意一个区域描述子的码流包括区域描述子的残差数据。
- 根据权利要求41所述的码流,其特征在于,所述多个区域描述子中的任意一个区域描述子的码流还包括参考区域描述子的指示信息,所述多个区域描述子包括所述参考区 域描述子。
- 根据权利要求34所述的码流,其特征在于,所述多个区域描述子中的任意一个区域描述子的码流包括区域描述子的量化数据。
Priority Applications (26)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020247000378A KR20240017921A (ko) | 2021-06-04 | 2021-06-04 | 3d 맵을 디코딩하기 위한 장치 및 방법, 및 3d 맵의 인코딩된 비트스트림 |
JP2023574743A JP2024521937A (ja) | 2021-06-04 | 2021-06-04 | 3dマップをデコーディングするための装置及び方法、及び3dマップのエンコーディングされたビットストリーム |
CN202180099016.0A CN117461057A (zh) | 2021-06-04 | 2021-06-04 | 3d地图的解码装置和方法、及3d地图的编码码流 |
PCT/CN2021/098483 WO2022252235A1 (zh) | 2021-06-04 | 2021-06-04 | 3d地图的解码装置和方法、及3d地图的编码码流 |
EP21943596.3A EP4336456A4 (en) | 2021-06-04 | 2021-06-04 | DECODING APPARATUS AND METHOD FOR 3D MAP AND ENCODED CODE STREAM OF A 3D MAP |
PCT/CN2021/104601 WO2022252337A1 (zh) | 2021-06-04 | 2021-07-05 | 3d地图的编解码方法及装置 |
CN202180098947.9A CN117413523A (zh) | 2021-06-04 | 2021-07-05 | 3d地图的编解码方法及装置 |
CN202180099015.6A CN117461056A (zh) | 2021-06-04 | 2021-07-09 | 3d地图的检索方法和装置 |
CN202180099018.XA CN117460972A (zh) | 2021-06-04 | 2021-07-09 | 3d地图的检索方法和装置 |
KR1020237045074A KR20240013238A (ko) | 2021-06-04 | 2021-07-09 | 3d 지도를 위한 압축 및 압축해제 방법들 및 장치들 |
JP2023574730A JP2024523186A (ja) | 2021-06-04 | 2021-07-09 | 3dマップを索出する方法、装置、及びコンピュータプログラム |
KR1020247000191A KR20240017906A (ko) | 2021-06-04 | 2021-07-09 | 3d 지도 검색 방법 및 장치 |
PCT/CN2021/105624 WO2022252346A1 (zh) | 2021-06-04 | 2021-07-09 | 3d地图的检索方法和装置 |
KR1020247000354A KR20240017918A (ko) | 2021-06-04 | 2021-07-09 | 3d 지도 검색 방법 및 장치 |
PCT/CN2021/105625 WO2022252347A1 (zh) | 2021-06-04 | 2021-07-09 | 3d地图的检索方法和装置 |
CN202180098601.9A CN117397242A (zh) | 2021-06-04 | 2021-07-09 | 3d地图的压缩、解压缩方法和装置 |
JP2023574725A JP2024521375A (ja) | 2021-06-04 | 2021-07-09 | 3dマップを取り出すための方法及び装置 |
EP21943704.3A EP4336454A1 (en) | 2021-06-04 | 2021-07-09 | 3d map retrieval method and apparatus |
EP21943703.5A EP4336223A1 (en) | 2021-06-04 | 2021-07-09 | 3d map retrieval method and apparatus |
JP2023574564A JP2024523816A (ja) | 2021-06-04 | 2021-07-09 | 3dマップを圧縮及び圧縮解除するための方法及び装置 |
EP21943702.7A EP4339881A1 (en) | 2021-06-04 | 2021-07-09 | Compression and decompression methods and apparatuses for 3d map |
PCT/CN2021/105623 WO2022252345A1 (zh) | 2021-06-04 | 2021-07-09 | 3d地图的压缩、解压缩方法和装置 |
US18/526,801 US20240095265A1 (en) | 2021-06-04 | 2023-12-01 | Method and apparatus for retrieving 3d map |
US18/526,677 US20240119639A1 (en) | 2021-06-04 | 2023-12-01 | Apparatus and method for decoding 3d map, and encoded bitstream of 3d map |
US18/526,846 US20240111793A1 (en) | 2021-06-04 | 2023-12-01 | Method and apparatus for retrieving 3d map |
US18/526,734 US20240112372A1 (en) | 2021-06-04 | 2023-12-01 | Method and apparatus for compressing and decompressing 3d map |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2021/098483 WO2022252235A1 (zh) | 2021-06-04 | 2021-06-04 | 3d地图的解码装置和方法、及3d地图的编码码流 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/526,677 Continuation US20240119639A1 (en) | 2021-06-04 | 2023-12-01 | Apparatus and method for decoding 3d map, and encoded bitstream of 3d map |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022252235A1 true WO2022252235A1 (zh) | 2022-12-08 |
Family
ID=84323760
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/098483 WO2022252235A1 (zh) | 2021-06-04 | 2021-06-04 | 3d地图的解码装置和方法、及3d地图的编码码流 |
Country Status (6)
Country | Link |
---|---|
US (1) | US20240119639A1 (zh) |
EP (1) | EP4336456A4 (zh) |
JP (1) | JP2024521937A (zh) |
KR (1) | KR20240017921A (zh) |
CN (1) | CN117461057A (zh) |
WO (1) | WO2022252235A1 (zh) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101290228A (zh) * | 2007-04-17 | 2008-10-22 | 北京协进科技发展有限公司 | 一种基于移动终端的地图数据处理方法、系统及移动终端 |
CN102168983A (zh) * | 2010-02-12 | 2011-08-31 | 三星电子株式会社 | 用于发送和接收地图数据的方法和装置 |
CN110869981A (zh) * | 2016-12-30 | 2020-03-06 | 迪普迈普有限公司 | 用于自主车辆的高清晰度地图数据的向量数据编码 |
CN111639147A (zh) * | 2020-04-24 | 2020-09-08 | 视辰信息科技(上海)有限公司 | 一种地图压缩方法、系统及计算机可读存储介质 |
-
2021
- 2021-06-04 EP EP21943596.3A patent/EP4336456A4/en active Pending
- 2021-06-04 KR KR1020247000378A patent/KR20240017921A/ko active Search and Examination
- 2021-06-04 JP JP2023574743A patent/JP2024521937A/ja active Pending
- 2021-06-04 WO PCT/CN2021/098483 patent/WO2022252235A1/zh active Application Filing
- 2021-06-04 CN CN202180099016.0A patent/CN117461057A/zh active Pending
-
2023
- 2023-12-01 US US18/526,677 patent/US20240119639A1/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101290228A (zh) * | 2007-04-17 | 2008-10-22 | 北京协进科技发展有限公司 | 一种基于移动终端的地图数据处理方法、系统及移动终端 |
CN102168983A (zh) * | 2010-02-12 | 2011-08-31 | 三星电子株式会社 | 用于发送和接收地图数据的方法和装置 |
CN110869981A (zh) * | 2016-12-30 | 2020-03-06 | 迪普迈普有限公司 | 用于自主车辆的高清晰度地图数据的向量数据编码 |
CN111639147A (zh) * | 2020-04-24 | 2020-09-08 | 视辰信息科技(上海)有限公司 | 一种地图压缩方法、系统及计算机可读存储介质 |
Non-Patent Citations (1)
Title |
---|
See also references of EP4336456A4 * |
Also Published As
Publication number | Publication date |
---|---|
KR20240017921A (ko) | 2024-02-08 |
CN117461057A (zh) | 2024-01-26 |
EP4336456A1 (en) | 2024-03-13 |
EP4336456A4 (en) | 2024-07-03 |
JP2024521937A (ja) | 2024-06-04 |
US20240119639A1 (en) | 2024-04-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112785715B (zh) | 虚拟物体显示方法以及电子设备 | |
WO2021088497A1 (zh) | 虚拟物体显示方法、全局地图更新方法以及设备 | |
US20240111793A1 (en) | Method and apparatus for retrieving 3d map | |
JP7487321B2 (ja) | 測位方法及びその装置、電子機器、記憶媒体、コンピュータプログラム製品、コンピュータプログラム | |
WO2022252235A1 (zh) | 3d地图的解码装置和方法、及3d地图的编码码流 | |
WO2018054114A1 (zh) | 一种图片编码方法及终端 | |
WO2022252234A1 (zh) | 3d地图的编码装置和方法 | |
WO2022252238A1 (zh) | 3d地图的压缩、解压缩方法和装置 | |
WO2022252237A1 (zh) | 3d地图的编解码方法及装置 | |
WO2022252236A1 (zh) | 3d地图的编解码方法及装置 | |
WO2022253228A1 (zh) | 3d地图的压缩、解压缩方法和装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21943596 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 202180099016.0 Country of ref document: CN Ref document number: 2023574743 Country of ref document: JP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2021943596 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2021943596 Country of ref document: EP Effective date: 20231208 |
|
ENP | Entry into the national phase |
Ref document number: 20247000378 Country of ref document: KR Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 1020247000378 Country of ref document: KR |
|
NENP | Non-entry into the national phase |
Ref country code: DE |