WO2022252235A1 - 3d地图的解码装置和方法、及3d地图的编码码流 - Google Patents

3d地图的解码装置和方法、及3d地图的编码码流 Download PDF

Info

Publication number
WO2022252235A1
WO2022252235A1 PCT/CN2021/098483 CN2021098483W WO2022252235A1 WO 2022252235 A1 WO2022252235 A1 WO 2022252235A1 CN 2021098483 W CN2021098483 W CN 2021098483W WO 2022252235 A1 WO2022252235 A1 WO 2022252235A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
map
module
code stream
reconstruction
Prior art date
Application number
PCT/CN2021/098483
Other languages
English (en)
French (fr)
Inventor
曹潇然
蔡康颖
王培�
涂晨曦
苏琪
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN202180099016.0A priority Critical patent/CN117461057A/zh
Priority to KR1020247000378A priority patent/KR20240017921A/ko
Priority to PCT/CN2021/098483 priority patent/WO2022252235A1/zh
Priority to EP21943596.3A priority patent/EP4336456A1/en
Priority to CN202180098947.9A priority patent/CN117413523A/zh
Priority to PCT/CN2021/104601 priority patent/WO2022252337A1/zh
Priority to KR1020237045074A priority patent/KR20240013238A/ko
Priority to KR1020247000191A priority patent/KR20240017906A/ko
Priority to KR1020247000354A priority patent/KR20240017918A/ko
Priority to CN202180099015.6A priority patent/CN117461056A/zh
Priority to PCT/CN2021/105625 priority patent/WO2022252347A1/zh
Priority to PCT/CN2021/105624 priority patent/WO2022252346A1/zh
Priority to CN202180099018.XA priority patent/CN117460972A/zh
Priority to CN202180098601.9A priority patent/CN117397242A/zh
Priority to EP21943704.3A priority patent/EP4336454A1/en
Priority to PCT/CN2021/105623 priority patent/WO2022252345A1/zh
Priority to EP21943702.7A priority patent/EP4339881A1/en
Priority to EP21943703.5A priority patent/EP4336223A1/en
Publication of WO2022252235A1 publication Critical patent/WO2022252235A1/zh
Priority to US18/526,846 priority patent/US20240111793A1/en
Priority to US18/526,801 priority patent/US20240095265A1/en
Priority to US18/526,734 priority patent/US20240112372A1/en
Priority to US18/526,677 priority patent/US20240119639A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/124—Quantisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2211/00—Image generation
    • G06T2211/40—Computed tomography
    • G06T2211/416—Exact reconstruction

Definitions

  • the present application relates to 3D map technology, and in particular to a 3D map decoding device and method, and a 3D map encoding code stream.
  • Virtual reality (virtual reality, VR), augmented reality (augmented reality, AR) and mixed reality (mixed reality, MR) technologies are emerging multimedia virtual scene technologies in recent years. This type of technology can create virtual reality and superimpose virtual reality and real world to generate new visual environment and interactive experience. In such applications, electronic devices need to determine their own pose information in the current environment, so as to accurately realize the fusion of virtual objects and real scenes.
  • a typical solution is: the electronic device receives a 3D map of the environment from the server or other devices, and then collects the 3D map of the environment through the local sensor. combined with the collected visual information and the downloaded 3D map to determine the current pose of the electronic device.
  • the original 3D map usually contains a huge amount of data, and the transmission of the map needs to consume a lot of bandwidth and time, which seriously limits the performance of the application and affects the user experience.
  • the present application provides a 3D map decoding device and method, and a 3D map code stream, so as to reduce the data volume of the 3D map, further reduce the transmission bandwidth, and improve the transmission efficiency.
  • the present application provides a codec system for a 3D map, including: an encoding device and a decoding device, and the encoding device and the decoding device are connected in communication; performing compression processing to obtain the code stream of the 3D map, and sending the code stream of the 3D map to the decoding device, the 3D map includes a plurality of 3D map points, and the data of the 3D map includes the multiple data of a 3D map point; the decoding device is configured to receive the code stream of the 3D map, and decompress the code stream of the 3D map to obtain the data of the 3D map.
  • the 3D map may include multiple 3D map points, and correspondingly, the data of the 3D map may include data of multiple 3D map points.
  • a 3D map point is an interesting or salient feature point in the environment.
  • the compression module can compress the data of the 3D map, which can reduce the data volume of the 3D map, for example, reduce the data volume of the 3D map from Terabyte (TB) level to Gigabyte (Gigabyte, GB) level, so that in the scene where the 3D map needs to be transmitted, compared with the transmission of the original 3D map data, the transmission of the compressed 3D map data can reduce the amount of transmitted data on the one hand, and on the other hand can Reduce the bandwidth occupied in transmission and improve the transmission efficiency of 3D maps.
  • TB Terabyte
  • GB Gigabyte
  • the encoding device is a cloud server, the decoding device is an electronic device; or, the encoding device is a first electronic device, and the decoding device is a second electronic device; the decoding The device is further configured to send a 3D map download request to the decoding device, and the 3D map download request includes position indication information; the encoding device is also configured to receive the 3D map download request, and send a request to the 3D map download request according to the 3D map download request.
  • the decoding device sends a code stream of a 3D map corresponding to the location indication information.
  • the foregoing electronic device may be a user terminal device.
  • the encoding device is an electronic device, and the decoding device is a cloud server; the encoding device is specifically configured to send the 3D map information to the decoding device after the 3D map is created. stream.
  • the electronic device collects visual information through a sensor, and determines the current pose of the electronic device in combination with the visual information and the 3D map from the server.
  • the 3D map is provided by the server, that is, the server creates the 3D map, then compresses the 3D map, and transmits the compressed data of the 3D map to the electronic device; the electronic device performs decompression processing after receiving the compressed data of the 3D map, and obtains The reconstructed data of the 3D map is combined with the collected visual information and the 3D map to determine the current pose of the electronic device.
  • the pose is the position and orientation information of the electronic device, which can be an absolute pose in the world coordinate system or a relative pose relative to a certain point in the environment.
  • the server may pre-create the 3D map, compress the 3D map and store it locally, which can save storage space.
  • the server can also transmit the compressed data of the 3D map to other devices, such as cloud storage.
  • the server creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
  • Compressing the 3D map by the server can save local storage space.
  • the electronic device sends a map download request to the server. There are two ways to trigger the map download request:
  • the user opens the 3D map application program installed on the electronic device, and the application program uploads the location information obtained based on GPS positioning or WiFi positioning to its corresponding server, and the uploading operation can trigger a map download request. Since the uploaded content includes location information, the server can make a preliminary estimate based on the location information, and transmit the compressed data of the 3D map of the area to which the positioning point indicated by the location information belongs to the electronic device.
  • the scope of the area to which the anchor point indicated by the location information can be preset.
  • the area to which the anchor point belongs can be the administrative region of each level (including counties, cities, countries, or administrative districts, etc.) where the anchor point is located, or it can be centered on the anchor point , Set the distance as the circle area of the radius.
  • the user opens the 3D map application program installed on the electronic device, and actively enters or selects an area in the application program, for example, the user actively enters "xx business center", or selects the area from a list "A street, B street , C Street” has selected "A Street”.
  • the foregoing operations of the user may trigger a map download request.
  • Either the user input or selection includes a geographic location, so the server transmits the compressed data of the 3D map of the geographic location to the electronic device.
  • the electronic device automatically detects whether it meets the conditions for downloading a 3D map or starts downloading a 3D map, or the electronic device passes the detection A change in ambient light or a change in the environment is detected to start downloading the 3D map, so as to request the server to download a 3D map within an area, and the size of the area is not specifically limited.
  • the server sends the compressed data of the 3D map to the electronic device.
  • step 3 and step 4 are independent of each other, and the sequence is not limited.
  • the electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • the electronic device After receiving the compressed data of the 3D map, the electronic device does not need to decompress it immediately, but only needs to decompress to obtain the reconstruction data of the 3D map before positioning based on the visual information. For example, the user can pre-download the compressed data of a 3D map in an area by downloading an "offline map", and decompress the compressed data of the 3D map when positioning is required.
  • the electronic device collects visual information through a sensor, and the server determines the current pose of the electronic device in combination with the visual information from the electronic device and a 3D map.
  • the 3D map is provided by the server, that is, the server creates the 3D map, then compresses the 3D map, and stores the compressed data of the 3D map locally.
  • the server When receiving the visual information from the electronic device, the server performs decompression processing to obtain the reconstructed data of the 3D map, and determines the current pose of the electronic device by combining the visual information and the 3D map.
  • the server creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
  • the electronic device sends the visual information to the server.
  • the server decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the server compresses the 3D map to save storage space.
  • the server performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • the server sends the pose to the electronic device.
  • the electronic device collects visual information through a sensor, and determines the current pose of the electronic device in combination with the visual information and a 3D map.
  • the 3D map is provided by the electronic device, that is, the electronic device creates the 3D map, then compresses the 3D map, and stores the compressed data of the 3D map locally.
  • the electronic device performs decompression processing to obtain the reconstructed data of the 3D map, and combines the collected visual information and the 3D map to determine the current pose of the electronic device.
  • the electronic device creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
  • Electronic equipment collects visual information through sensors.
  • the electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • the second electronic device collects visual information through a sensor, and determines the current pose of the second electronic device in combination with the visual information and the 3D map from the server.
  • the 3D map is created by the first electronic device, that is, the first electronic device creates the 3D map, compresses the 3D map, and then sends the compressed data of the 3D map to the server, and the server sends the compressed data of the 3D map to the second
  • the second electronic device performs decompression processing to obtain the reconstruction data of the 3D map, and determines the current pose of the second electronic device in combination with the collected visual information and the 3D map.
  • the first electronic device may pre-create a 3D map, compress the 3D map and then transmit it to the server, which can reduce transmission bandwidth.
  • the first electronic device creates a 3D map and compresses it to obtain compressed data of the 3D map.
  • the first electronic device sends the compressed data of the 3D map to the server.
  • the first electronic device compresses the 3D map before transmitting it, which can reduce the transmission bandwidth and improve the transmission efficiency.
  • the second electronic device sends a map download request to the server.
  • the map download request sent by the second electronic device may also be based on the trigger manner shown in FIG. 4a.
  • the server sends the compressed data of the 3D map to the second electronic device.
  • the second electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the second electronic device collects the visual information through the sensor.
  • the second electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • the second electronic device collects visual information through a sensor, and the server determines the current pose of the second electronic device in combination with the visual information from the second electronic device and the 3D map from the first electronic device.
  • the 3D map is created by the first electronic device, that is, the first electronic device creates the 3D map, compresses the 3D map, and then sends the compressed data of the 3D map to the server.
  • the server performs decompression processing to obtain the reconstructed data of the 3D map, and combines the visual information from the second electronic device and the 3D map to determine the current pose of the second electronic device.
  • the first electronic device creates a 3D map and compresses it to obtain compressed data of the 3D map.
  • the first electronic device sends the compressed data of the 3D map to the server.
  • the second electronic device acquires visual information through sensor collection.
  • the second electronic device sends a positioning request to the server, and the positioning request carries visual information.
  • the server decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the server performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • the server sends the pose obtained by positioning to the second electronic device.
  • the second electronic device collects visual information through a sensor, and determines the current pose of the second electronic device in combination with the visual information and the 3D map from the first electronic device.
  • the 3D map is created by the first electronic device, that is, the first electronic device creates a 3D map, performs compression processing on the 3D map, and then sends the compressed data of the 3D map to the second electronic device, and the second electronic device performs decompression processing , obtain the reconstructed data of the 3D map, and determine the current pose of the second electronic device in combination with the collected visual information and the 3D map from the first electronic device.
  • the first electronic device creates a 3D map, and compresses the compressed data of the 3D map to be stored locally.
  • the second electronic device sends a map download request to the first electronic device.
  • the first electronic device sends the compressed data of the 3D map to the second electronic device.
  • the second electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the second electronic device acquires visual information through sensor collection.
  • the second electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • the data of the 3D map further includes a plurality of region descriptors, and any region descriptor in the plurality of region descriptors is used to describe a part of the plurality of 3D map points Or features of all 3D map points.
  • the region descriptor can be used to describe the characteristics of some or all of the 3D map points in the multiple 3D map points, so that the region descriptor and the 3D map point are one-to-many Relationship.
  • the features of each 3D map point among the multiple 3D map points can be described by some or all of the multiple region descriptors, so that the 3D map point and the region descriptor have a one-to-many relationship. It can be seen that there is a many-to-many relationship between multiple region descriptors and multiple 3D map points.
  • regional descriptors include but are not limited to traditional methods such as bag of words (BOW), vector of aggregate locally descriptor (VLAD) and other traditional methods based on NetVLAD and artificial intelligence (AI). new method.
  • BOW bag of words
  • VLAD vector of aggregate locally descriptor
  • AI artificial intelligence
  • multiple region descriptors may also be identified with a number to distinguish the multiple region descriptors, but the number is also not a limitation on the order of the multiple region descriptors.
  • the data of any 3D map point among the plurality of 3D map points includes a 3D map point descriptor and a spatial position of the 3D map point.
  • the 3D map point descriptor is a vector (vector), which is used to represent the local features of the 3D map point.
  • the spatial position of a 3D map point can be represented by X, Y, and Z on the axis of the three-dimensional space, it can also be represented by latitude and longitude, and altitude, and it can also be represented by polar coordinates.
  • the embodiment of the present application expresses the spatial position of a 3D map point No specific restrictions are made.
  • the spatial position of the 3D map point can be the absolute position of the 3D map point, or the relative position of the 3D map point
  • the encoding device is further configured to create the 3D map.
  • the decoding device is further configured to perform positioning according to the 3D map.
  • the present application provides a 3D map decoding device, including: a transmission module and a decompression module; wherein the transmission module is used to receive a code stream of a 3D map, and the 3D map includes a plurality of 3D map points;
  • the decompression module is configured to decompress the code stream of the 3D map to obtain reconstruction data of the 3D map, and the reconstruction data of the 3D map includes reconstruction data of the plurality of 3D map points.
  • the decompression module can support the decompression processing of the compressed data of the 3D map, that is, the codec system can support the compression/decompression processing of the data of the 3D map, which can reduce the amount of data of the 3D map,
  • transmitting compressed data of 3D maps can reduce the amount of transmitted data on the one hand, and reduce the bandwidth occupied in transmission on the other hand, improving the quality of 3D maps. transmission efficiency.
  • the decompression processing may include at least one of inverse quantization processing and prediction processing, wherein,
  • Quantization processing refers to mapping the data to be processed into one or more quantization indexes. Each quantization index corresponds to a quantization center. The number of bits of the quantization index is usually significantly less than the number of bits of the original data, so as to save storage or transmission bandwidth. Purpose. Quantization processing methods include, but are not limited to: scalar quantization, vector quantization, product quantization, and the like. Based on this, the inverse quantization process is the reverse process of the aforementioned quantization process, that is, reconstructed data corresponding to the original data is restored according to one or more quantization indexes, and the number of bits of the reconstructed data is greater than the number of bits of the quantization index.
  • Prediction processing refers to using the residual data of the processed data and the data to be processed to perform prediction to obtain the data to be processed.
  • the selection of reference data can be pre-agreed, for example, fixedly use the previously processed data as reference data, at this time it is not necessary to mark the reference data in the code stream;
  • the identification information of the reference data written in the stream including the serial number of the reference data, or other information from which the reference data can be deduced.
  • the processing of the 3D map data includes encapsulation, the purpose of which is to encapsulate the data to be encoded into a code stream.
  • the encapsulation process can use any encoding algorithm, such as entropy encoding.
  • Entropy coding is a lossless data compression method.
  • Entropy coding algorithms include but are not limited to: Huffman (huffman) coding, arithmetic coding, compression/decompression improved by LZ77 compression algorithm (lempel-ziv-markov chain-algorithm, LZMA ) algorithm, providing the function library (zlib) algorithm for data compression, etc.
  • the decompression processing may also include decapsulation processing, the purpose of which is to decapsulate the code stream into data to be decompressed, and the decapsulation processing may use a decoding algorithm corresponding to the encoding algorithm adopted by the encoding end, for example Entropy decoding.
  • the data of the 3D map further includes reconstruction data of a plurality of region descriptors, and any one region descriptor in the plurality of region descriptors is used to describe the plurality of 3D map points Features of some or all of the 3D map points in .
  • the reconstruction data of any 3D map point among the plurality of 3D map points includes reconstruction data of a 3D map point descriptor and reconstruction data of a spatial position of the 3D map point.
  • the device for decoding the 3D map is a cloud server or an electronic device; the transmission module is further configured to send a 3D map download request, and the 3D map download request includes position indication information; receiving and The code stream of the 3D map corresponding to the location indication information.
  • the device for decoding the 3D map is a cloud server; the transmission module is specifically configured to receive a code stream of the 3D map created by an electronic device.
  • the decompression module includes a decapsulation module, and a prediction module and/or an inverse quantization module; the decapsulation module is configured to process the input code stream of the 3D map to Outputting the first data; the prediction module is used to perform prediction processing on the residual data of the input second data to output the second data; the dequantization module is used to dequantize the input third data Processing to output dequantized data of the third data; wherein, the first data is residual data of the second data or the third data; the second data is reconstruction data of the 3D map Or the third data; the inverse quantization data of the third data is the reconstruction data of the 3D map.
  • the decompression module only includes a decapsulation module, and an inverse quantization module or a prediction module.
  • the decompression module includes a prediction module, an inverse quantization module, and an encapsulation module.
  • the prediction module includes a first prediction module and a second prediction module; and/or, the inverse quantization module includes a first inverse quantization module and a second inverse quantization module; the first The prediction module is configured to perform prediction processing on residual data of the input fourth data to output the fourth data; the first inverse quantization module is configured to perform inverse quantization processing on the input fifth data to output the The dequantization data of the fifth data; wherein, the fourth data is the reconstruction data of one of the multiple region descriptors or the fifth data; the dequantization data of the fifth data is the multiple regions The reconstruction data of one of the descriptors; the second prediction module is used to perform prediction processing on the residual data of the input sixth data to output the sixth data; the second dequantization module is used to performing dequantization processing on the input seventh data to output dequantized data of the seventh data; wherein, the sixth data is reconstruction data of one of the plurality of 3D map points or the seventh data ; The inverse quantization data of the seventh data is the reconstruction
  • the decompression module only includes the first prediction module and the second prediction module, or only includes the first inverse quantization module and the second inverse quantization module.
  • the decompression module includes a first prediction module and a second prediction module, and a first inverse quantization module and a second inverse quantization module.
  • the prediction module includes a first prediction module, a second prediction module, and a third prediction module; and/or, the inverse quantization module includes a first inverse quantization module, a second inverse quantization module and a third inverse quantization module; the first prediction module is used to perform prediction processing on the residual data of the input eighth data to output the eighth data; the first inverse quantization module is used to input the eighth data
  • the ninth data is dequantized to output the dequantized data of the ninth data; wherein, the eighth data is reconstruction data of one of the plurality of region descriptors or the ninth data; the ninth data
  • the inverse quantization data of the data is the reconstruction data of one of the plurality of region descriptors; the second prediction module is configured to perform prediction processing on the input residual data of the tenth data to output the tenth data ;
  • the second inverse quantization module is used to perform inverse quantization processing on the input eleventh data to output the inverse quantization data of the eleventh data; wherein, the tenth data
  • the decompression module only includes the first prediction module, the second prediction module and the third prediction module, or only includes the first inverse quantization module, the second inverse quantization module and the third inverse quantization module.
  • the decompression module includes a first prediction module, a second prediction module, and a third prediction module, and a first inverse quantization module, a second inverse quantization module, and a third inverse quantization module.
  • the decompression module includes a first decompression submodule and a second decompression submodule; the first decompression submodule is used to process the code stream of the input fourteenth data Perform decompression processing to output the fourteenth data; the second decompression submodule is configured to compress the input code stream of the fifteenth data to output the fifteenth data; wherein, the The fourteenth data is the reconstruction data of one of the plurality of region descriptors; the fifteenth data is the reconstruction data of one of the plurality of 3D map points.
  • the first decompression submodule includes a first decapsulation module, and a first prediction module and/or a first inverse quantization module;
  • the second decompression submodule includes a second decompression module An encapsulation module, and a second prediction module and/or a second inverse quantization module;
  • the first decapsulation module is configured to process the input code stream of the 3D map to output sixteenth data;
  • the first The prediction module is used to perform prediction processing on the residual data of the input seventeenth data to output the seventeenth data;
  • the first inverse quantization module is used to perform inverse quantization processing on the input eighteenth data to output output the dequantized data of the eighteenth data;
  • the sixteenth data is the residual data of the seventeenth data or the eighteenth data;
  • the seventeenth data is the plurality of The reconstruction data of one of the region descriptors or the eighteenth data;
  • the inverse quantization data of the eighteenth data is the reconstruction data of one of the
  • the decompression module includes a first decompression submodule, a second decompression submodule, and a third decompression submodule; the first decompression submodule is configured to The code stream of the twenty-second data is decompressed to output the twenty-second data; the second decompression submodule is used to compress the input code stream of the twenty-third data to output the The twenty-third data; the third decompression sub-module is used to compress the code stream of the input twenty-fourth data to output the twenty-fourth data; wherein, the twenty-second The data is the reconstruction data of one of the plurality of region descriptors; the twenty-third data is the reconstruction data of the 3D map point descriptor of one of the plurality of 3D map points; the twenty-fourth The data is reconstruction data of a spatial position of a 3D map point of one of the plurality of 3D map points.
  • the first decompression submodule includes a first decapsulation module, and a first prediction module and/or a first inverse quantization module;
  • the second decompression submodule includes a second decompression module The encapsulation module, and the second prediction module and/or the second dequantization module;
  • the third decompression sub-module includes a third decapsulation module, and the third prediction module and/or the third dequantization module;
  • the first The decapsulation module is used to process the input code stream of the 3D map to output the twenty-fifth data;
  • the first prediction module is used to perform prediction processing on the residual data of the input twenty-sixth data to output the twenty-sixth data;
  • the first inverse quantization module is configured to perform inverse quantization processing on the input twenty-seventh data to output the inverse quantization data of the twenty-seventh data;
  • the The twenty-fifth data is the residual data of the twenty-sixth data or the twenty-seventh data; the twenty
  • Residual data or the thirtieth data; the twenty-ninth data is reconstruction data of a 3D map point descriptor of one of the plurality of 3D map points or the thirtieth data; the thirtieth data
  • the dequantization data of the thirty data is the reconstructed data of the 3D map point descriptor of one of the plurality of 3D map points;
  • the third decapsulation module is used to perform the input code stream of the 3D map Processing to output the thirty-first data;
  • the third prediction module is used to perform prediction processing on the residual data of the input thirty-second data to output the thirty-second data;
  • the third inverse quantization module for performing inverse quantization processing on the input thirty-third data to output the inverse quantization data of the thirty-third data;
  • the thirty-first data is the residual data of the thirty-second data Or the 33rd data;
  • the 32nd data is reconstruction data of the spatial position of a 3D map point of one of the plurality of 3D map points or the 33r
  • the present application provides a method for decoding a 3D map, including: receiving a code stream of a 3D map, the 3D map including a plurality of 3D map points; decompressing the code stream of the 3D map to obtain the The reconstruction data of the 3D map, the reconstruction data of the 3D map includes the reconstruction data of the plurality of 3D map points.
  • the decompression module can support the decompression processing of the compressed data of the 3D map, that is, the codec system can support the compression/decompression processing of the data of the 3D map, which can reduce the amount of data of the 3D map,
  • transmitting compressed data of 3D maps can reduce the amount of transmitted data on the one hand, and reduce the bandwidth occupied in transmission on the other hand, improving the quality of 3D maps. transmission efficiency.
  • the reconstruction data of the 3D map further includes reconstruction data of a plurality of region descriptors, and any one region descriptor in the plurality of region descriptors is used to describe the plurality of 3D maps Features of some or all 3D map points in points.
  • the reconstruction data of any 3D map point among the plurality of 3D map points includes reconstruction data of a 3D map point descriptor and reconstruction data of a spatial position of the 3D map point.
  • the method further includes: sending a 3D map download request, where the 3D map download request includes location indication information; and receiving the code stream of the 3D map includes: receiving the The code stream of the above 3D map.
  • the receiving the code stream of the 3D map includes: receiving the code stream of the 3D map created by an electronic device.
  • the decompressing the code stream of the 3D map to obtain the data of the 3D map includes: processing the code stream of the 3D map to obtain the first data; performing prediction processing on the residual data of the second data to obtain the second data, and/or performing inverse quantization processing on the third data to obtain inverse quantization data of the third data; wherein, the first data It is the residual data of the second data or the third data; the second data is the reconstruction data of the 3D map or the third data; the inverse quantization data of the third data is the 3D The reconstruction data of the map.
  • the present application provides a 3D map decoding device, including: a memory for storing a compressed 3D map in the form of a bit stream, the 3D map includes a plurality of 3D map points, and the data of the 3D map Data including the plurality of 3D map points; a decoder, configured to decompress the compressed 3D map in the form of a bit stream to obtain a reconstructed 3D map, the reconstructed 3D map including the plurality of 3D maps Map points, the reconstructed 3D map data includes reconstructed data of the plurality of 3D map points.
  • the reconstructed 3D map data further includes reconstructed data of a plurality of region descriptors, and any one region descriptor in the plurality of region descriptors is used to describe the plurality of 3D maps Features of some or all 3D map points in points.
  • the reconstruction data of any 3D map point among the plurality of 3D map points includes reconstruction data of a 3D map point descriptor and reconstruction data of a spatial position of the 3D map point.
  • the encoding device for the 3D map is a cloud server or an electronic device.
  • the present application provides a computer-readable storage medium, which is characterized by comprising a computer program, and when the computer program is executed on a computer, the computer executes the method described in any one of the above-mentioned third aspects. method.
  • the present application provides a computer program product, the computer program product includes computer program code, and when the computer program code is run on a computer, it causes the computer to execute the method described in any one of the above third aspects .
  • the present application provides a coded code stream of a 3D map, where the coded code stream of the 3D map includes a code stream of multiple 3D map points, and the 3D map includes the multiple 3D map points.
  • the encoded code stream of the 3D map further includes code streams of multiple region descriptors, and any one of the multiple region descriptors and the multiple 3D map points Corresponds to at least one 3D map point in .
  • the code stream of any 3D map point among the plurality of 3D map points includes a code stream of a 3D map point descriptor and a code stream of a spatial position of a 3D map point.
  • the code stream of any 3D map point among the plurality of 3D map points includes residual data of the 3D map point.
  • the code stream of the 3D map point descriptor includes the residual data of the 3D map point descriptor; and/or, the code stream of the 3D map point spatial position includes the 3D The residual data of the spatial position of the map point.
  • the code stream of any 3D map point among the multiple 3D map points further includes indication information of a reference 3D map point
  • the multiple 3D map points include the reference 3D map point
  • the reference 3D map point is a 3D map point that has been encoded before any one of the 3D map points is encoded.
  • the code stream of any 3D map point among the plurality of 3D map points includes quantized data of the 3D map point.
  • the code stream of the 3D map point descriptor includes quantized data of the 3D map point descriptor; and/or, the code stream of the 3D map point spatial position includes the 3D map Quantified data of point spatial location.
  • the code stream of any one of the multiple region descriptors includes residual data of the region descriptor.
  • the code stream of any one of the multiple region descriptors further includes indication information of a reference region descriptor, and the multiple region descriptors include the reference region descriptor .
  • the code stream of any one of the multiple region descriptors includes quantized data of the region descriptor.
  • FIG. 1 is a schematic diagram of an application architecture provided by an embodiment of the present application
  • FIG. 2 is a schematic structural diagram of an electronic device 20 provided in an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of a server 30 provided in an embodiment of the present application.
  • Fig. 4a is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • Fig. 4b is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • Fig. 4c is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • Fig. 4d is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • FIG. 4e is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • Fig. 4f is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • FIG. 4g is a schematic diagram of a user interface displayed by the electronic device provided in the embodiment of the present application.
  • FIG. 5 is a structural diagram of a 3D map decoding device 50 provided in an embodiment of the present application.
  • FIG. 6a is a structural diagram of a 3D map decoding device 60 provided by an embodiment of the present application.
  • FIG. 6b is a structural diagram of a 3D map decoding device 60 provided by an embodiment of the present application.
  • FIG. 6c is a structural diagram of a 3D map decoding device 60 provided by an embodiment of the present application.
  • FIG. 6d is a structural diagram of a 3D map decoding device 60 provided by the embodiment of the present application.
  • FIG. 6e is a structural diagram of a 3D map decoding device 60 provided by an embodiment of the present application.
  • FIG. 7a is a structural diagram of a 3D map decoding device 70 provided by an embodiment of the present application.
  • FIG. 7b is a structural diagram of a 3D map decoding device 70 provided by an embodiment of the present application.
  • FIG. 8a is a structural diagram of a 3D map decoding device 80 provided by an embodiment of the present application.
  • FIG. 8b is a structural diagram of a 3D map decoding device 80 provided by an embodiment of the present application.
  • FIG. 9 is a flowchart of a process 900 of a decoding method for a 3D map provided in an embodiment of the present application.
  • FIG. 10 is a structural diagram of a 3D map decoding device 100 provided by an embodiment of the present application.
  • Fig. 11a is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
  • Fig. 11b is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
  • Fig. 11c is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
  • Fig. 11d is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
  • Fig. 11e is an exemplary structural diagram of a coding code stream of a 3D map provided by an embodiment of the present application.
  • At least one (item) means one or more, and “multiple” means two or more.
  • “And/or” is used to describe the association relationship of associated objects, indicating that there can be three types of relationships, for example, “A and/or B” can mean: only A exists, only B exists, and A and B exist at the same time , where A and B can be singular or plural.
  • the character “/” generally indicates that the contextual objects are an “or” relationship.
  • At least one of the following” or similar expressions refer to any combination of these items, including any combination of single or plural items.
  • At least one item (piece) of a, b or c can mean: a, b, c, "a and b", “a and c", “b and c", or "a and b and c ", where a, b, c can be single or multiple.
  • Fig. 1 is a schematic diagram of an application architecture provided by an embodiment of the present application.
  • the application architecture includes a plurality of electronic devices and servers, wherein the plurality of electronic devices may include a first electronic device, one or more A second electronic device (two second electronic devices are taken as an example in FIG. 1), the one or more second electronic devices are several electronic devices other than the first electronic device. Multiple electronic devices and servers, and multiple electronic devices can communicate with each other.
  • wireless-fidelity Wireless-fidelity
  • WiFi wireless-fidelity
  • Bluetooth Bluetooth
  • cellular 2/3/4/5 generation 2/3/4/5generation, 2G/3G/4G/5G
  • future communication methods which are not specifically limited.
  • one or more second electronic devices in this embodiment of the present application is only used to refer to other electronic devices except the first electronic device, but does not limit whether the types of multiple electronic devices are the same.
  • the above-mentioned electronic equipment can be various types of equipment equipped with cameras and display components.
  • the electronic equipment can be terminal equipment such as mobile phones, tablet computers, notebook computers, video recorders (in FIG. 1, the electronic equipment is a mobile phone as an example),
  • Electronic devices can also be devices used for virtual scene interaction, including VR glasses, AR devices, MR interactive devices, etc.
  • Electronic devices can also be wearable electronic devices such as smart watches and smart bracelets.
  • Electronic devices can also be vehicles, wireless Equipment carried in vehicles such as human-driven vehicles, drones, and industrial robots. The embodiment of the present application does not specifically limit the specific form of the electronic device.
  • the above-mentioned electronic devices may also be referred to as user equipment (user equipment, UE), subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station , terminal device, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.
  • user equipment user equipment
  • UE user equipment
  • subscriber station mobile unit
  • subscriber unit wireless unit
  • remote unit mobile device
  • wireless device wireless device
  • wireless communication device remote device
  • mobile subscriber station terminal device, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.
  • the above-mentioned servers may be one or more physical servers (a physical server is taken as an example in FIG. 1 ), may also be a computer cluster, or may be a virtual machine or a cloud server in a cloud computing scenario, and the like.
  • the electronic device can install virtual scene applications (applications, APPs) such as VR applications, AR applications, or MR applications, and can run VR applications based on user operations (such as clicking, touching, sliding, shaking, voice control, etc.).
  • application, AR application or MR application can collect the visual information of any object in the environment through the sensor, and then display the virtual object on the display component according to the collected visual information.
  • the virtual object can be a virtual object in a VR scene, an AR scene or an MR scene (that is, a objects).
  • the electronic device can install navigation, detection, and control application programs, and run corresponding applications based on user manipulation or preset programs.
  • Electronic devices can perform applications such as path planning, object detection, and vehicle control based on their own pose and other state information in the current environment.
  • the visual information involved in the embodiments of the present application includes, but is not limited to, images and videos collected by cameras (without depth information), images and videos with depth information collected by depth sensors, and laser radar (LiDAR) acquisitions.
  • the virtual scene application program in the electronic device may be an application program built in the electronic device itself, or an application program provided by a third-party service provider installed by the user himself, and no specific details are given here. limited.
  • the electronic device can also be equipped with a simultaneous localization and mapping (SLAM) system.
  • the SLAM system can create a map in a completely unknown environment, and use the map to perform positioning, pose (position and attitude) determination, navigation, etc.
  • the map created by the SLAM system is called a SLAM map.
  • the SLAM map can be understood as a map drawn by the SLAM system based on the environmental information collected by the collection device.
  • the collection device may include a visual information collection device in an electronic device and Inertial measurement unit (inertial measurement unit, IMU), wherein, the visual information acquisition device may include, for example, a camera, a camera, a depth camera, a lidar, a millimeter-wave radar, etc., and the IMU may include, for example, sensors such as a gyroscope and an accelerometer.
  • IMU Inertial measurement unit
  • the SLAM map is also referred to as a 3D map. It should be noted that the 3D map includes but is not limited to a SLAM map, and may also include a 3D map created using other technologies, which is not specifically limited in this embodiment of the present application.
  • the 3D map may include multiple 3D map points, and correspondingly, data of the 3D map may include data of multiple 3D map points.
  • a 3D map point is an interesting or salient feature point in the environment.
  • 3D map points are to use various equipment such as lidar, aerial photography (oblique photography) from the perspective of drones, high-definition panoramic cameras, high-definition industrial cameras, etc., through accelerated segmentation test-oriented features ( features from accelerated segment test, FAST) and rotation binary robust independent elementary features (binary robust independent elementary features, BRIEF) descriptor features (oriented FAST and rotated BRIEF, ORB), scale invariant feature transform (scale invariant feature transform, SIFT), accelerated robust features (speeded-up robust feature, SURF), BRIEF, binary robust invariant scalable keypoints (binary robust invariant scalable keypoints, BRISK), fast retina keypoints (fast retina keypoint , FREAK), repeatable and reliable detector and descriptor (repeatable and reliable detector and descriptor, R2D2) and other methods are extracted from the data captured by the above equipment.
  • accelerated segmentation test-oriented features features from accelerated segment test, FAST
  • BRIEF binary robust independent elementary
  • the data of 3D map points can include:
  • the 3D map point descriptor is a vector used to represent the local features of the 3D map point.
  • the 3D map point descriptor is used for matching between 3D map points.
  • One possible approach is to calculate the distance between two 3D map point descriptors (it can be Euclidean distance, inner product distance, Hamming distance, etc.), and when the distance is less than a threshold, it is considered that two 3D map points match .
  • the spatial position of a 3D map point can be represented by X, Y, and Z on the axis of the three-dimensional space, it can also be represented by latitude and longitude, and altitude, and it can also be represented by polar coordinates.
  • the embodiment of the present application expresses the spatial position of a 3D map point No specific restrictions are made.
  • the spatial position of the 3D map point can be the absolute position of the 3D map point, or the relative position of the 3D map point. For example, taking the center position of the entire area as the origin, all the spatial positions of the 3D map point are relative to the spatial position of the origin offset position.
  • each 3D map point can be assigned a number and written into the data of the 3D map, and the storage order of multiple 3D map points in the memory can also be used to implicitly represent the number of the 3D map point.
  • the order of multiple 3D map points contained in a 3D map has no practical significance, so the aforementioned numbers can be considered as identifiers for identifying 3D map points to distinguish each 3D map point, but this Numbering is not used to limit the order of multiple 3D map points.
  • a 3D map contains 3 3D map points, and their numbers are 1, 2, and 3 respectively.
  • the processing of these 3 3D map points can be done according to The order of 1, 2, 3 can also be performed in the order of 3, 2, 1, and the order of 2, 1, 3 can also be performed, and so on.
  • the data of the 3D map further includes a plurality of region descriptors, and any one of the region descriptors in the plurality of region descriptors is used to describe part or all of the 3D map in the plurality of 3D map points
  • the feature of the point that is, for any one of the multiple area descriptors, the area descriptor can be used to describe the features of some or all of the 3D map points in the multiple 3D map points, so that the area descriptor and the 3D map A point is a one-to-many relationship.
  • each 3D map point among the multiple 3D map points can be described by some or all of the multiple region descriptors, so that the 3D map point and the region descriptor have a one-to-many relationship. It can be seen that there is a many-to-many relationship between multiple region descriptors and multiple 3D map points.
  • the generation methods of regional descriptors include but are not limited to traditional methods such as bag of words (BOW), vector of aggregate locally descriptor (VLAD), and methods based on NetVLAD and artificial intelligence (AI). new method.
  • BOW bag of words
  • VLAD vector of aggregate locally descriptor
  • AI artificial intelligence
  • multiple region descriptors may also be identified with a number to distinguish the multiple region descriptors, but the number is also not a limitation on the order of the multiple region descriptors.
  • the data of the 3D map also includes the correspondence between the 3D map points and the region descriptors, which explicitly describe which 3D map points any one region descriptor corresponds to, and any one Which region descriptors the 3D map points correspond to.
  • the above correspondence can be explicitly described by a correspondence table between the number of the region descriptor and the number of the 3D map point.
  • the 3D map contains 3 region descriptors, the numbers are T1 ⁇ T3, and there are 6
  • the 3D map points are numbered from P1 to P6, and the corresponding table is shown in Table 1 or Table 2.
  • Table 1 is an example of the correspondence table between the number of the region descriptor and the number of the 3D map point, and the correspondence table can also be presented in other formats or ways, which is not specifically limited in this application.
  • the above corresponding relationship can also be implicitly described by the storage location of the region descriptor and the 3D map point. For example, first store T1 in the memory, then store the data of P1, P2 and P3, then store T2, and then store The data of P2 and P3 are stored, and finally T3 is stored, followed by the data of P3, P4, P5 and P6.
  • FIG. 2 is a schematic structural diagram of an electronic device 20 provided by an embodiment of the present application.
  • the electronic device 20 may be the first electronic device and one or more second electronic devices in the embodiment shown in FIG. 1 at least one of the .
  • the structure shown in FIG. 2 does not constitute a specific limitation on the electronic device 20 .
  • the electronic device 20 may include more or fewer components than the structure shown in FIG. 2 , or combine certain components, or separate certain components, or arrange different components.
  • the various components shown in Figure 2 may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.
  • the electronic device 20 may include: a chip 21, a memory 22 (one or more computer-readable storage media), a user interface 23, a display component 24, a camera 25, a sensor 26, a positioning module 27 for positioning the device, and a communication device. Transceiver 28. Communication between these components may be via one or more buses 29 .
  • the chip 21 may include: one or more processors 211 , a clock module 212 and a power management module 213 .
  • the clock module 212 integrated in the chip 21 is mainly used to provide the processor 211 with a timer required for data transmission and timing control, and the timer can realize the clock function of data transmission and timing control.
  • the processor 211 can perform operations according to instruction opcodes and timing signals, generate operation control signals, and complete control of fetching and executing instructions.
  • the power management module 213 integrated in the chip 21 is mainly used to provide stable and high-precision voltage for the chip 21 and other components of the electronic device 20 .
  • the processor 211 can also be called a central processing unit (central processing unit, CPU), and the processor 211 can specifically include one or more processing units.
  • the processor 211 can include an application processor (application processor, AP). Tuning processor, graphics processing unit (graphics processing unit, GPU), image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor , and/or, a neural network processor (neural-network processing unit, NPU), etc.
  • different processing units may be independent devices, or may be integrated in one or more processors.
  • the processor 211 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or, a universal serial bus (universal serial bus, USB) interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input and output
  • subscriber identity module subscriber identity module
  • SIM subscriber identity module
  • USB universal serial bus
  • the memory 22 can be connected to the processor 211 through the bus 29, or can be coupled with the processor 311, and is used for storing various software programs and/or multiple sets of instructions.
  • the memory 22 may include a high-speed random access memory (such as a cache memory), or may include a non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices or other non-volatile solid-state storage devices.
  • the memory 22 can store an operating system, for example, embedded operating systems such as Android (Android), Apple mobile platform (IOS), Microsoft window operating system (Windows) or UNIX-like operating system (Linux).
  • the memory 22 can also store data, for example, image data, point cloud data, 3D map data, pose data, coordinate system transformation information, map update information, and the like.
  • the memory 22 may also store computer-executable program codes, where the computer-executable program codes include instructions, for example, communication program instructions, related program instructions of the SLAM system, and the like.
  • the memory 22 can also store one or more application programs, such as virtual scene application programs such as AR/VR/MR, 3D map application programs, image management application programs, navigation and control application programs, and the like.
  • the memory 22 can also store a user interface program.
  • the user interface program can display the content of the application program through a graphical operation interface, for example, virtual objects in virtual scenes such as AR/VR/MR. 24 presentation, and realize receiving the user's control operations on the application through input controls such as menus, dialog boxes, and buttons.
  • the user interface 23 may be, for example, a touch panel, and the touch panel may detect a user's operation instruction thereon, and the user interface 23 may also be, for example, a keypad, a physical button, or a mouse.
  • Electronic device 20 may include one or more display components 24 .
  • the electronic device 20 may jointly implement a display function through a display component 24 , a graphics processing unit (GPU) and an application processor (AP) in the chip 21 .
  • the GPU is a microprocessor that implements image processing and is connected to the display assembly 24 and the application processor.
  • the GPU performs mathematical and geometric calculations for graphics rendering.
  • the display component 24 can display the interface content output by the electronic device 20, for example, display images, videos, etc. in virtual scenes such as AR/VR/MR, and the interface content can include the interface of the running application program and the system level menu, etc.
  • interface elements such as buttons (Button), text input boxes (Text), sliders (Scroll Bar), menus (Menu), etc.; output interface elements, such as windows (Window), labels (Label), image, video, animation, etc.
  • the display component 24 can be a display panel, glasses (such as VR glasses), projection screen and so on.
  • the display panel may also be called a display screen, for example, may be a touch screen, a flexible screen, a curved screen, etc., or may be other optical components.
  • the display screen of the electronic device in the embodiment of the present application may be a touch screen, flexible screen, curved screen or other forms of screen, that is, the display screen of the electronic device has the function of displaying images, and the specific The material and shape are not specifically limited.
  • the display panel can be a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (organic light-emitting diode, OLED), an active matrix organic light-emitting diode or an active matrix Organic Light Emitting Diode (active-matrix organic light emitting diode, AMOLED), flexible light emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED )Wait.
  • liquid crystal display liquid crystal display
  • OLED organic light-emitting diode
  • AMOLED active matrix organic light-emitting diode
  • FLED flexible light emitting diode
  • Miniled MicroLed, Micro-oLed
  • quantum dot light emitting diodes quantum dot light emitting diodes (quantum dot light emitting diodes, QLED )Wait.
  • the touch panel in the user interface 23 and the display panel in the display component 24 can be coupled together, for example, the touch panel can be arranged under the display panel, and the touch panel is used for Detecting the touch pressure on the display panel when the user inputs a touch operation (such as clicking, sliding, touching, etc.) through the display panel, and the display panel is used for displaying content.
  • a touch operation such as clicking, sliding, touching, etc.
  • the camera 25 may be a monocular camera, a binocular camera or a depth camera, and is used for shooting/recording the environment to obtain images/video images.
  • the image/video image collected by the camera 25 can be used as a kind of input data of the SLAM system, or can be displayed by the display component 24 for image/video.
  • the camera 25 can also be regarded as a sensor.
  • the image collected by the camera 25 may be in the IMG format or in other formats, which is not specifically limited in this embodiment of the present application.
  • the sensor 26 can be used to collect data related to state changes of the electronic device 20 (for example, rotation, swing, movement, shaking, etc.), and the data collected by the sensor 26 can be used as an input data of the SLAM system.
  • the sensor 26 may include one or more sensors, for example, an inertial measurement unit (inertial measurement unit, IMU), a time of flight (time of flight, TOF) sensor, and the like.
  • the IMU may include sensors such as a gyroscope and an accelerometer.
  • the gyroscope is used to measure the angular velocity of the electronic device when it is in motion
  • the accelerometer is used to measure the acceleration of the electronic device when it is in motion.
  • the TOF sensor can include a light emitter and a light receiver.
  • the light emitter is used to emit light, such as laser, infrared, radar waves, etc.
  • the light receiver is used to detect reflected light, such as reflected laser, infrared, and radar waves. Wait.
  • the sensor 26 may also include more other sensors, such as inertial sensors, barometers, magnetometers, wheel speedometers, etc., which are not specifically limited in this embodiment of the present application.
  • the positioning module 27 is used to realize the physical positioning of the electronic device 20 , for example, to acquire the initial position of the electronic device 20 .
  • the positioning module 27 may include one or more of a WiFi positioning module, a Bluetooth positioning module, a base station positioning module, and a satellite positioning module.
  • the satellite positioning module can be provided with a global navigation satellite system (global navigation satellite system, GNSS) to assist positioning, and GNSS is not limited to the Beidou system, the global positioning system (global positioning system, GPS) system, the GLONASS (global navigation satellite system) , GLONASS) system, Galileo satellite navigation system (Galileo) system.
  • the transceiver 28 is used to enable communication between the electronic device 20 and other devices (eg, servers, other electronic devices, etc.).
  • the transceiver 28 integrates a transmitter and a receiver for sending and receiving radio frequency signals, respectively.
  • the transceiver 28 includes, but is not limited to: an antenna system, a radio frequency (radio frequency, RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec ( CODEC) chip, subscriber identification module (SIM) card and storage media, etc.
  • the transceiver 28 can also be implemented on a separate chip.
  • the transceiver 28 supports at least one data network communication in 2G/3G/4G/5G, etc., and/or supports at least one of the following short-range wireless communication methods: Bluetooth (bluetooth, BT) communication, wireless fidelity (wireless fidelity, WiFi) communication, near field communication (near field communication, NFC), infrared (infrared, IR) wireless communication, ultra wide band (ultra wide band, UWB) communication, ZigBee protocol (ZigBee) communication.
  • Bluetooth bluetooth, BT
  • wireless fidelity wireless fidelity, WiFi
  • NFC near field communication
  • infrared infrared
  • UWB ultra wide band
  • ZigBee protocol ZigBee protocol
  • the processor 211 executes various functional applications and data processing of the electronic device 20 by running the program code stored in the memory 22 .
  • FIG. 3 is a schematic structural diagram of a server 30 provided in an embodiment of the present application.
  • the server 30 may be the server in the embodiment shown in FIG. 1 .
  • Server 30 includes processor 301 , memory 302 (one or more computer-readable storage media), and transceiver 303 . Communication between these components may be via one or more buses 304 .
  • the processor 301 may be one or more CPUs. In the case where the processor 301 is one CPU, the CPU may be a single-core CPU or a multi-core CPU.
  • the memory 302 can be connected to the processor 301 through the bus 304, or can be coupled with the processor 301, and is used to store various program codes and/or multiple sets of instructions, and data (eg, map data, pose data, etc.).
  • the memory 302 includes, but is not limited to, random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), or portable read-only memory (Compact Disc Read-Only Memory, CD-ROM), etc.
  • the transceiver 303 mainly integrates a receiver and a transmitter, where the receiver is used to receive data sent by the electronic device (for example, requests, images, etc.), and the transmitter is used to send data to the electronic device (for example, map data, pose data, etc.) .
  • server 30 shown in FIG. 3 is only an example provided by the embodiment of the present application, and the server 30 may have more components than shown in the figure, which is not specifically limited in the embodiment of the present application.
  • the processor 301 executes various functional applications and data processing of the server 30 by running the program code stored in the memory 302 .
  • Coupled used in the embodiments of the present application means to be directly connected or connected through one or more intermediate components or circuits.
  • Fig. 4a is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • the application scenario is that the electronic device collects visual information through a sensor, and determines the current pose of the electronic device in combination with the visual information and the 3D map from the server.
  • the 3D map is provided by the server, that is, the server creates the 3D map, then compresses the 3D map, and transmits the compressed data of the 3D map to the electronic device; the electronic device performs decompression processing after receiving the compressed data of the 3D map, and obtains The reconstructed data of the 3D map is combined with the collected visual information and the 3D map to determine the current pose of the electronic device.
  • the pose is the position information of the electronic device, which can be the absolute pose in the world coordinate system, or the relative pose relative to a certain point in the environment.
  • the server may pre-create the 3D map, compress the 3D map and store it locally, which can save storage space.
  • the server can also transmit the compressed data of the 3D map to other devices, such as cloud storage.
  • the server creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
  • Compressing the 3D map by the server can save local storage space.
  • the electronic device sends a map download request to the server. There are two ways to trigger the map download request:
  • the user opens the 3D map application installed on the electronic device, and the application uploads the location information based on GPS positioning or WiFi positioning to its corresponding server, and the upload operation can trigger a map download request. Since the uploaded content includes location information, the server can make a preliminary estimate based on the location information, and transmit the compressed data of the 3D map of the area to which the positioning point indicated by the location information belongs to the electronic device.
  • the scope of the area to which the anchor point indicated by the location information can be preset.
  • the area to which the anchor point belongs can be the administrative region of each level (including counties, cities, countries, or administrative districts, etc.) where the anchor point is located, or it can be centered on the anchor point , Set the distance as the circle area of the radius.
  • the user opens the 3D map application program installed on the electronic device, and actively enters or selects an area in the application program, for example, the user actively enters "xx business center", or selects the area from a list "A street, B street , C Street” has selected "A Street”.
  • the foregoing operations of the user may trigger a map download request.
  • Either the user input or selection includes a geographic location, so the server transmits the compressed data of the 3D map of the geographic location to the electronic device.
  • the electronic device automatically detects whether it meets the conditions for downloading a 3D map or starts downloading a 3D map, or the electronic device passes the detection A change in ambient light or a change in the environment is detected to start downloading the 3D map, so as to request the server to download a 3D map within an area, and the size of the area is not specifically limited.
  • the server sends the compressed data of the 3D map to the electronic device.
  • step 3 and step 4 are independent of each other, and the sequence is not limited.
  • the electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • the electronic device After receiving the compressed data of the 3D map, the electronic device does not need to decompress it immediately, but only needs to decompress to obtain the reconstruction data of the 3D map before positioning based on the visual information. For example, the user can pre-download the compressed data of a 3D map in an area by downloading an "offline map", and decompress the compressed data of the 3D map when positioning is required.
  • Figure 4b is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • the application scenario is that the electronic device collects visual information through a sensor, and the server determines the current position of the electronic device in combination with the visual information from the electronic device and a 3D map. posture.
  • the 3D map is provided by the server, that is, the server creates the 3D map, then compresses the 3D map, and stores the compressed data of the 3D map locally.
  • the server When receiving the visual information from the electronic device, the server performs decompression processing to obtain the reconstructed data of the 3D map, and determines the current pose of the electronic device by combining the visual information and the 3D map.
  • the server creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
  • the electronic device sends the visual information to the server.
  • the server decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the server compresses the 3D map to save storage space.
  • the server performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • the server sends the pose to the electronic device.
  • Fig. 4c is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • the application scenario is that the electronic device collects visual information through a sensor, and determines the current pose of the electronic device in combination with the visual information and a 3D map.
  • the 3D map is provided by the electronic device, that is, the electronic device creates the 3D map, then compresses the 3D map, and stores the compressed data of the 3D map locally.
  • the electronic device performs decompression processing to obtain the reconstructed data of the 3D map, and combines the collected visual information and the 3D map to determine the current pose of the electronic device.
  • the electronic device creates a 3D map and compresses the compressed data of the 3D map to be stored locally.
  • Electronic equipment collects visual information through sensors.
  • the electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • Fig. 4d is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • the application scenario is that the second electronic device collects visual information through a sensor, and determines the location of the second electronic device in combination with the visual information and the 3D map from the server. current pose.
  • the 3D map is created by the first electronic device, that is, the first electronic device creates the 3D map, compresses the 3D map, and then sends the compressed data of the 3D map to the server, and the server sends the compressed data of the 3D map to the second
  • the second electronic device performs decompression processing to obtain the reconstruction data of the 3D map, and determines the current pose of the second electronic device in combination with the collected visual information and the 3D map.
  • the first electronic device may pre-create a 3D map, compress the 3D map and then transmit it to the server, which can reduce transmission bandwidth.
  • the first electronic device creates a 3D map and compresses it to obtain compressed data of the 3D map.
  • the first electronic device sends the compressed data of the 3D map to the server.
  • the first electronic device compresses the 3D map before transmitting it, which can reduce the transmission bandwidth and improve the transmission efficiency.
  • the second electronic device sends a map download request to the server.
  • the map download request sent by the second electronic device may also be based on the trigger manner shown in FIG. 4a.
  • the server sends the compressed data of the 3D map to the second electronic device.
  • the second electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the second electronic device collects the visual information through the sensor.
  • the second electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • Fig. 4e is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • the application scenario is that the second electronic device collects visual information through a sensor, and the server combines the visual information from the second electronic device with the visual information from the first electronic device.
  • the 3D map of the device determines the current pose of the second electronic device.
  • the 3D map is created by the first electronic device, that is, the first electronic device creates the 3D map, compresses the 3D map, and then sends the compressed data of the 3D map to the server.
  • the server performs decompression processing to obtain the reconstructed data of the 3D map, and combines the visual information from the second electronic device and the 3D map to determine the current pose of the second electronic device.
  • the first electronic device creates a 3D map and compresses it to obtain compressed data of the 3D map.
  • the first electronic device sends the compressed data of the 3D map to the server.
  • the second electronic device acquires visual information through sensor collection.
  • the second electronic device sends a positioning request to the server, and the positioning request carries visual information.
  • the server decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the server performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • the server sends the pose obtained by positioning to the second electronic device.
  • Fig. 4f is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • the application scenario is that the second electronic device collects visual information through a sensor, and combines the visual information and the 3D map from the first electronic device to determine the second The current pose of the electronic device.
  • the 3D map is created by the first electronic device, that is, the first electronic device creates a 3D map, performs compression processing on the 3D map, and then sends the compressed data of the 3D map to the second electronic device, and the second electronic device performs decompression processing , obtain the reconstructed data of the 3D map, and determine the current pose of the second electronic device in combination with the collected visual information and the 3D map from the first electronic device.
  • the first electronic device creates a 3D map, and compresses the compressed data of the 3D map to be stored locally.
  • the second electronic device sends a map download request to the first electronic device.
  • the first electronic device sends the compressed data of the 3D map to the second electronic device.
  • the second electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.
  • the second electronic device acquires visual information through sensor collection.
  • the second electronic device performs positioning in the 3D map according to the visual information to obtain a pose corresponding to the visual information.
  • the positioning algorithm used may include:
  • the distance between the region descriptor to be retrieved and each region descriptor in multiple region descriptors can be calculated, the distance can include Hamming distance, Manhattan distance or Euclidean distance, etc., and then select the matching condition (For example, at least one region descriptor whose distance is smaller than the threshold value) is a candidate region descriptor.
  • the pose solving algorithm calculates the pose of the electronic device.
  • FIG. 4a to FIG. 4f all involve compressing the 3D map.
  • the embodiment of the present application provides a plurality of device frameworks for performing the aforementioned compression processing, and the multiple device frameworks will be described below.
  • FIG. 4g is a schematic diagram of a user interface displayed by the electronic device provided in the embodiment of the present application.
  • the electronic device can display the user interface shown in FIG. 4g based on the pose.
  • the user interface may include a navigation arrow indication for navigating to the conference room 2, and the navigation arrow indication for navigating to the conference room 2 may be a virtual object obtained from the server or locally based on the pose.
  • the user interface may also include visual information collected by sensors, for example, buildings as shown in FIG. 4g. The user goes to the meeting room 2 referring to the user interface shown in FIG. 4g of the electronic device.
  • the reconstructed data of the 3D map obtained through decompression involved in the embodiment of the present application may also be referred to as the reconstructed data of the 3D map.
  • FIG. 5 is a structural diagram of a 3D map decoding device 50 provided in the embodiment of the present application.
  • a device that compresses the needs of the 3D map for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f, or, the server in the embodiment shown in Figure 4e .
  • the 3D map decoding device 50 includes: a transmission module 51 and a decompression module 52; wherein, the transmission module 51 is used to receive the code stream of the 3D map, and the 3D map includes a plurality of 3D map points; the decompression module 52 , for decompressing the code stream of the 3D map to obtain reconstruction data of the 3D map, where the reconstruction data of the 3D map includes reconstruction data of multiple 3D map points.
  • the data of the transmission module 51 is the code stream of the 3D map
  • the decompression module 52 is used to decompress the code stream of the 3D map to obtain the reconstructed data of the 3D map.
  • the decompression module 52 can support the decompression processing of the compressed data of the 3D map, that is, the codec system can support the compression/decompression processing of the data of the 3D map, which can reduce the amount of data of the 3D map.
  • transmitting the compressed data of the 3D map can reduce the amount of transmitted data on the one hand, for example, compressing the data volume from TB level to GB level, on the other hand can reduce The bandwidth occupied during transmission improves the transmission efficiency of 3D maps.
  • the decompression processing may include at least one of inverse quantization processing and prediction processing, wherein,
  • Quantization processing refers to mapping the data to be processed into one or more quantization indexes. Each quantization index corresponds to a quantization center. The number of bits of the quantization index is usually significantly less than the number of bits of the original data, so as to save storage or transmission bandwidth. Purpose. Quantization processing methods include, but are not limited to: scalar quantization, vector quantization, product quantization, and the like. Based on this, the inverse quantization process is the reverse process of the aforementioned quantization process, that is, reconstructed data corresponding to the original data is restored according to one or more quantization indexes, and the number of bits of the reconstructed data is greater than the number of bits of the quantization index.
  • Prediction processing refers to using the residual data of the processed data and the data to be processed to perform prediction to obtain the data to be processed.
  • the selection of reference data can be pre-agreed, for example, fixedly use the previously processed data as reference data, at this time it is not necessary to mark the reference data in the code stream;
  • the identification information of the reference data written in the stream including the serial number of the reference data, or other information from which the reference data can be deduced.
  • the processing of the 3D map data includes encapsulation, the purpose of which is to encapsulate the data to be encoded into a code stream.
  • the encapsulation process can use any encoding algorithm, such as entropy encoding.
  • Entropy coding is a lossless data compression method.
  • Entropy coding algorithms include but are not limited to: Huffman (huffman) coding, arithmetic coding, compression/decompression improved by LZ77 compression algorithm (lempel-ziv-markov chain-algorithm, LZMA ) algorithm, providing the function library (zlib) algorithm for data compression, etc.
  • the decompression processing may also include decapsulation processing, the purpose of which is to decapsulate the code stream into data to be decompressed, and the decapsulation processing may use a decoding algorithm corresponding to the encoding algorithm adopted by the encoding end, for example Entropy decoding.
  • the data of the 3D map obtained after decompression at the decoding end is not completely consistent with the original data of the 3D map before compression processing at the encoding end.
  • the 3D map data output by the decompression module 52 in the embodiment of the present application can be referred to as 3D map reconstruction data, and the 3D map reconstruction data can refer to the description of the above 3D map data, which will not be repeated here.
  • the embodiment of the present application provides multiple implementable structural examples of the decompression module, which will be described through the following embodiments.
  • FIG. 6a is a structural diagram of a 3D map decoding device 60-1 provided in the embodiment of the present application.
  • the decoding device 60-1 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
  • the decoding device 60-1 includes a transmission module 61-1 and a decompression module 62-1, and the decompression module 62-1 includes: a decapsulation module 621-1 and an inverse quantization module 623-1.
  • the decapsulation module 621-1 is used to process the code stream of the input 3D map to output the first data; the dequantization module 623-1 is used to dequantize the input third data to output the third data Inverse quantization data;
  • the above-mentioned first data is the third data; the inverse quantization data of the third data is the reconstructed data of the 3D map.
  • the input data of the decapsulation module 621-1 is the code stream of the 3D map, and the output data is the quantized data of the 3D map;
  • the input data of the dequantization module 623-1 is the 3D map output by the decapsulation module 621-1 Quantify the data, and the output data is the reconstruction data of the 3D map.
  • FIG. 6b is a structural diagram of a 3D map decoding device 60-2 provided in the embodiment of the present application.
  • the decoding device 60-2 can be applied to the above-mentioned embodiments
  • the server or electronic device there is a device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f, Or, the server in the embodiment shown in Fig. 4e.
  • the decoding device 60-2 includes a transmission module 61-2 and a decompression module 62-2, and the decompression module 62-2 includes: a decapsulation module 621-2 and a prediction module 622 -1. in,
  • the decapsulation module 621-2 is used to process the code stream of the input 3D map to output the first data; the prediction module 622-1 is used to perform prediction processing on the residual data of the input second data to output the second data. data.
  • the above-mentioned first data is the residual data of the second data; the second data is the reconstructed data of the 3D map.
  • the input data of the decapsulation module 621-2 is the code stream of the 3D map, and the output data is the residual data of the 3D map; the input data of the prediction module 622-1 is the code stream of the 3D map output by the decapsulation module 621-2. Residual data, the output data is the reconstruction data of the 3D map.
  • FIG. 6c is a structural diagram of a 3D map decoding device 60-3 provided in the embodiment of the present application.
  • the decoding device 60-3 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
  • the decoding device 60-3 includes a transmission module 61-3 and a decompression module 62-3, and the decompression module 61-3 includes: a decapsulation module 621-3, a prediction module 622 -2 and dequantization module 623-2. in,
  • the decapsulation module 621-3 is used to process the code stream of the input 3D map to output the first data; the prediction module 622-2 is used to perform prediction processing on the residual data of the input second data to output the second data. Data; an inverse quantization module 623-2, configured to perform inverse quantization processing on the input third data to output inverse quantization data of the third data.
  • the above-mentioned first data is the residual data of the second data or the third data; the second data is the reconstructed data of the 3D map or the third data; the inverse quantization data of the third data is the reconstructed data of the 3D map.
  • the input data of the decapsulation module 621-3 is the code stream of the 3D map, and the output data is the residual data of the 3D map; the input data of the prediction module 622-2 is the code stream of the 3D map output by the decapsulation module 621-3.
  • the output data is the quantized data of the 3D map
  • the input data of the inverse quantization module 623-2 is the quantized data of the 3D map output by the prediction module 622-2
  • the output data is the reconstructed data of the 3D map.
  • the reconstruction data of the 3D map may include the reconstruction data of multiple region descriptors and the reconstruction data of multiple 3D map points. Therefore, in the embodiments shown in FIGS. Modules except the decapsulation module can be split into a first module for processing region descriptors and a second module for processing 3D map point data.
  • the dequantization module can include the first dequantization module
  • the quantization module and the second inverse quantization module
  • the prediction module may include a first prediction module and a second prediction module. The difference between the differentiated first module and the second module is that the input data correspond to the area descriptor and the data of the 3D map points respectively.
  • Figure 6d is a structural diagram of the 3D map decoding device 60-4 provided by the embodiment of the present application.
  • the decoding device 60-4 can be applied to the server or electronic device in the above embodiment, especially A device that needs to receive and decompress a 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f, or, the implementation shown in Figure 4e server in the example.
  • the decoding device 60-4 includes a transmission module 61-4 and a decompression module 62-4, and the decompression module 62-4 includes: a decapsulation module 621-4, a prediction module 622 -3 and an inverse quantization module 623-3, the prediction module 622-3 includes a first prediction module 622a and a second prediction module 622b, and the inverse quantization module 623 includes a first inverse quantization module 623a and a second inverse quantization module 623b.
  • the transmission module 61-4 is configured to receive the code stream of the above-mentioned 3D map.
  • the input data of the decapsulation module 621-4 is the code stream of the 3D map, and the decapsulation module 621-4 decapsulates the code stream of the 3D map to obtain the residual data of the region descriptor and the residual data of the 3D map points.
  • the input data of the first prediction module 622a is the residual data of the region descriptor, and the output data is the quantized data of the region descriptor;
  • the input data of the first inverse quantization module 623a is the quantized data of the region descriptor, and the output data is the region descriptor reconstruction data.
  • the input data of the second prediction module 622b is the residual data of the 3D map point, and the output data is the quantized data of the 3D map point;
  • the input data of the second inverse quantization module 623b is the quantized data of the 3D map point, and the output data is the 3D map point reconstruction data.
  • the reconstruction data of the 3D map may include the reconstruction data of multiple region descriptors, the reconstruction data of the 3D map point descriptors of multiple 3D map points and the reconstruction data of the spatial positions of the 3D map points , so in the embodiments shown in Figures 6a to 6c, other modules in the decompression module except the decapsulation module can be split into the first module for processing the region descriptor, and the first module for processing the 3D map point description
  • the sub-second module and the third module for processing the spatial position of the 3D map point for example, the inverse quantization module may include the first inverse quantization module, the second inverse quantization module and the third inverse quantization module, and the prediction module may include the first A prediction module, a second prediction module and a third prediction module.
  • the difference between the differentiated first module, second module and third module is that the input data corresponds to the region descriptor, the 3D map point descriptor and the spatial position of the 3D map point respectively.
  • Fig. 6e is a structural diagram of a 3D map decoding device 60 provided by the embodiment of the present application.
  • a device that compresses the needs of the 3D map for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f, or, the server in the embodiment shown in Figure 4e .
  • the decoding device 60-5 includes a transmission module 61-5 and a decompression module 62-5
  • the decompression module 62-5 includes: a decapsulation module 621-5, a prediction module 622 -4 and inverse quantization module 623-4
  • prediction module 622-4 includes first prediction module 622a, second prediction module 622b and third prediction module 622c
  • inverse quantization module 623-4 includes first inverse quantization module 623a, second An inverse quantization module 623b and a third inverse quantization module 623c.
  • the transmission module 61-5 is configured to receive the code stream of the above-mentioned 3D map.
  • the input data of the decapsulation module 621-5 is the code stream of the 3D map, and the decapsulation module 621-5 decapsulates the code stream of the 3D map to obtain the residual data of the region descriptor and the residual data of the 3D map point descriptor and the residual data of the spatial position of the 3D map point.
  • the input data of the first prediction module 622a is the residual data of the region descriptor, and the output data is the quantized data of the region descriptor;
  • the input data of the first inverse quantization module 623a is the quantized data of the region descriptor, and the output data is the region descriptor reconstruction data.
  • the input data of the second prediction module 622b is the residual data of the 3D map point descriptor, and the output data is the quantized data of the 3D map point descriptor; the input data of the second inverse quantization module 623b is the quantized data of the 3D map point descriptor, The output data is the reconstructed data of the 3D map point descriptor.
  • the input data of the third prediction module 622b is the residual data of the spatial position of the 3D map point, and the output data is the quantized data of the spatial position of the 3D map point; the input data of the third inverse quantization module 623b is the quantized data of the spatial position of the 3D map point, The output data is the reconstructed data of the spatial position of the 3D map points.
  • FIG. 6d and FIG. 6e respectively show an exemplary structure of a 3D map decoding device, which is a structure obtained based on the content contained in the code stream of the 3D map, but this structure does not constitute a decoding
  • the decoding device may include more modules than those shown in the embodiment shown in FIG. 6d or FIG. 6e.
  • other structures are obtained based on the content contained in the code stream of the 3D map.
  • the prediction module and the dequantization module are in an and/or relationship, that is, the decompression module may include one of the prediction module and the dequantization module, or the prediction module and the dequantization module Both are included. Therefore, for the case of distinguishing the first module and the second module, the processing module for the region descriptor and the processing module for the data of the 3D map point can independently set the processing method, and the two do not have to be completely consistent, for example, for the region descriptor , can be processed by the first prediction module and the first inverse quantization module respectively, and the data for 3D map points can be processed by the second prediction module.
  • the processing module for the region descriptor, the processing module for the 3D map point descriptor and the processing module for the spatial position of the 3D map point can be independently set and processed
  • the three methods do not have to be exactly the same.
  • the region descriptor it can be processed by the first prediction module and the first inverse quantization module respectively;
  • the 3D map point descriptor it can be processed by the second prediction module respectively;
  • the 3D map The spatial position of the point can be processed by the third inverse quantization module.
  • the present application does not specifically limit the specific implementation manner of the decompression module.
  • the reconstruction data of the 3D map may include the reconstruction data of multiple area descriptors and the reconstruction data of multiple 3D map points.
  • FIG. 7a is a structural diagram of a 3D map decoding device 70-1 provided in the embodiment of the present application.
  • the decoding device 70-1 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
  • the decoding device 70-1 includes a transmission module 71-1 and a decompression module 72-1, and the decompression module 72-1 includes: a first decompression sub-module 721-1 and The second decompression sub-module 722-1. in,
  • the first decompression sub-module 721-1 is used to decompress the code stream of the input fourteenth data to output the fourteenth data;
  • the second decompression sub-module 722-1 is used to decompress the input tenth data
  • the code stream of the fifth data is compressed to output the fifteenth data;
  • the fourteenth data is reconstruction data of one of the plurality of region descriptors; the fifteenth data is reconstruction data of one of the plurality of 3D map points.
  • the input data of the first decompression sub-module 721-1 is the code stream of the region descriptor
  • the output data is the reconstructed data of the region descriptor
  • the input data of the second decompression sub-module 722-1 is the 3D map point
  • the code stream, the output data is the reconstruction data of 3D map points.
  • the code stream of the 3D map includes code streams of multiple area descriptors and code streams of multiple 3D map points.
  • first decompression submodule 721-1 and the second decompression submodule 722-1 reference may be made to the structure of the decompression module in the embodiment shown in FIG. 6a to FIG. 6e. It should be noted that the first decompression submodule 721-1 and the second decompression submodule 722-1 are independent of each other, and the structures adopted by the two may be the same or different, that is, the first decompression submodule 722-1 for processing the region descriptor
  • the structure of the first decompression sub-module 721-1 is the same as or different from the structure of the second decompression sub-module 722-1 for processing the data of 3D map points.
  • the steps of decompression processing adopted by the data of the 3D map points may also be the same or different.
  • the first decompression submodule includes a first decapsulation module and a first prediction module, so that after the code stream of the region descriptor is input into the first decompression submodule, it is decapsulated by the first decapsulation module to obtain the region descriptor
  • the residual data, the residual data is processed by the first prediction module to obtain the reconstruction data of the region descriptor
  • the second decompression sub-module includes the second decapsulation module, the second prediction module and the second inverse quantization module, so that the 3D
  • the code stream of the map point is input into the second decompression sub-module, it is first processed by the second decapsulation module to obtain the residual data of the 3D map point, and then the residual data is processed by the second prediction module to obtain the quantized data of the 3D map point , and then the quantized data is processed by the second inverse quantization module to obtain reconstruction data of 3D map points.
  • the first decompression sub-module includes a first decapsulation module, a first prediction module, and a first inverse quantization module, so that after the code stream of the region descriptor is input into the first decompression sub-module, it passes through the first decapsulation module respectively processing to obtain the residual data of the region descriptor, the residual data is processed by the first prediction module to obtain the quantized data of the region descriptor, and the quantized data is processed by the first inverse quantization module to obtain the reconstructed data of the region descriptor; the second solution
  • the compression submodule includes a second decapsulation module and a second prediction module, after the description of the 3D map point is input into the second decompression submodule, it is processed by the second decapsulation module to obtain the residual data of the 3D map point, the residual data The reconstructed data of the 3D map points are obtained through processing by the second prediction module.
  • FIG. 7b is a structural diagram of a 3D map decoding device 70-2 provided in the embodiment of the present application.
  • the decoding device 70-2 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
  • the decoding device 70-2 includes a transmission module 71-2 and a decompression module 72-2, and the decompression module 72-2 includes a first decompression sub-module 721-2 and a second decompression sub-module 721-2.
  • the second decompression sub-module 722-2, the first decompression sub-module 721-2 includes the first decapsulation module 7211, the first prediction module 7212 and the first inverse quantization module 7213, the second decompression sub-module 722-2 includes the first Two decapsulation module 7221 , second prediction module 7222 and second inverse quantization module 7223 .
  • the first decapsulation module 7211 is used to process the code stream of the input region descriptor to obtain the residual data of the region descriptor, and the first prediction module 7212 is used to predict the residual data of the input region descriptor Processing to obtain the quantized data of the region descriptor, the first inverse quantization module 7213 is configured to perform inverse quantization on the input quantized data of the region descriptor to obtain the reconstructed data of the region descriptor.
  • the second decapsulation module 7221 is used to process the code stream of the input 3D map point to obtain the residual data of the 3D map point, and the second prediction module 7222 is used to predict the residual data of the input 3D map point.
  • the second inverse quantization module 7223 is used to dequantize the input quantized data of the 3D map points to obtain the reconstruction data of the 3D map points.
  • FIG. 7b only exemplifies the structure of the first decompression submodule and the second decompression submodule, but does not limit the structure of the two.
  • These two submodules may include For more or fewer modules in the example, for details, refer to the structure of the decompression module 62 in the embodiment shown in FIG. 6a to FIG. 6e , which is not specifically limited in this embodiment of the present application.
  • the reconstruction data of the 3D map may include reconstruction data of multiple region descriptors, reconstruction data of 3D map point descriptors of multiple 3D map points, and reconstruction data of spatial positions of 3D map points.
  • FIG. 8a is a structural diagram of a 3D map decoding device 80-1 provided in the embodiment of the present application.
  • the decoding device 80-1 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
  • the decoding device 80-1 includes a transmission module 81-1 and a decompression module 82-1, and the decompression module 82-1 includes: a first decompression sub-module 821-1, The second decompression sub-module 822-1 and the third decompression sub-module 823-1.
  • the first decompression sub-module 821-1 is used to decompress the code stream of the input twenty-second data to output the twenty-second data;
  • the second decompression sub-module 822-1 is used to decompress the input code stream
  • the code stream of the twenty-third data is compressed to output the twenty-third data;
  • the third decompression sub-module 823-1 is used to compress the input code stream of the twenty-fourth data to output the twenty-third Four data.
  • the 22nd data is the reconstruction data of one of the multiple region descriptors;
  • the 23rd data is the reconstruction data of the 3D map point descriptor of one of the multiple 3D map points;
  • the 24th The data is reconstruction data of a spatial position of a 3D map point of one of the plurality of 3D map points.
  • the input data of the first decompression sub-module 821-1 is the code stream of the region descriptor
  • the output data is the reconstructed data of the region descriptor
  • the input data of the second decompression sub-module 812-1 is the 3D map point
  • the code stream of the descriptor, the output data is the reconstruction data of the 3D map point descriptor
  • the input data of the third decompression sub-module 813-1 is the code stream of the spatial position of the 3D map point
  • the output data is the reconstruction of the spatial position of the 3D map point data.
  • the code stream of the 3D map includes a code stream of multiple area descriptors, a code stream of multiple 3D map point descriptors, and a code stream of multiple spatial positions of 3D map points.
  • first decompression submodule 821-1 For the first decompression submodule 821-1, the second decompression submodule 822-1, and the third decompression submodule 823-1, reference may be made to the structure of the decompression module in the embodiment shown in FIG. 6a to FIG. 6e. It should be noted that the first decompression submodule 821-1, the second decompression submodule 822-1 and the third decompression submodule 823-1 are independent of each other, and the structures adopted by the three may be the same or different.
  • the structure of the first decompression submodule 821-1 for processing region descriptors, the second decompression submodule 822-1 for processing 3D map point descriptors, and the first decompression submodule 822-1 for processing 3D map point spatial positions The structures of the three decompression sub-modules 823-1 are the same or different.
  • the decompression processing steps for the region descriptor, the decompression processing steps for the 3D map point descriptor, and the 3D map point space The steps of the decompression process adopted by the locations may also be the same or different.
  • FIG. 8b is a structural diagram of a 3D map decoding device 80-2 provided in the embodiment of the present application.
  • the decoding device 80-2 can be applied to the above-mentioned embodiments In the server or electronic device, especially the device that needs to receive and decompress the 3D map, for example, the electronic device in the embodiment shown in Figure 4a, or the second electronic device in the embodiment shown in Figure 4d and Figure 4f device, or the server in the embodiment shown in Fig. 4e.
  • the decoding device 80-2 includes a transmission module 81-2 and a decompression module 82-2, and the decompression module 82-2 includes a first decompression sub-module 821-2, a second The second decompression sub-module 822-2 and the third decompression sub-module 823-2, the first decompression sub-module 821-2 includes the first decapsulation module 8211, the first prediction module 8212 and the first dequantization module 8213, the first The second decompression sub-module 822-2 includes a second decapsulation module 8221, a second prediction module 8222, and a second inverse quantization module 8223, and the third decompression sub-module 823-2 includes a third decapsulation module 8231, a third prediction module 8232 and the third inverse quantization module 8233.
  • the first decapsulation module 8211 is used to process the code stream of the input region descriptor to obtain the residual data of the region descriptor, and the first prediction module 8212 is used to predict the residual data of the input region descriptor To obtain the quantized data of the region descriptor, the first inverse quantization module 8213 is used to perform inverse quantization processing on the input quantized data of the region descriptor to obtain the reconstructed data of the region descriptor.
  • the second decapsulation module 8221 is used to process the code stream of the input 3D map point descriptor to obtain the residual data of the 3D map point descriptor
  • the second prediction module 8222 is used to process the input 3D map point descriptor Perform prediction processing on the residual data to obtain the quantized data of the 3D map point descriptor
  • the second inverse quantization module 8223 is used to perform inverse quantization processing on the input quantized data of the 3D map point descriptor to obtain the quantized data of the 3D map point descriptor Rebuild data.
  • the third decapsulation module 8231 is used to process the code stream of the input 3D map point spatial position to obtain the residual data of the 3D map point spatial position
  • the third prediction module 8232 is used to process the input 3D map point spatial position Prediction processing is performed on the residual data to obtain the quantized data of the spatial position of the 3D map point
  • the third inverse quantization module 8233 is used to perform inverse quantization processing on the quantized data of the input spatial position of the 3D map point to obtain the spatial position of the 3D map point Rebuild data.
  • FIG. 8b only exemplifies the structures of the first decompression submodule, the second decompression submodule, and the third decompression submodule, but does not limit the structures of the three.
  • These three sub-modules may include more or less modules than in the example, for details, refer to the structure of the decompression module 62 in the embodiment shown in Fig. 6a to Fig. 6e , which is not specifically limited in this embodiment of the present application.
  • FIG. 9 is a flow chart of a process 900 of a method for decoding a 3D map provided in an embodiment of the present application.
  • the process 900 may be executed by the decoding device in the foregoing embodiment.
  • the process 900 is described as a series of steps or operations. It should be understood that the process 900 may be performed in various orders and/or concurrently, and is not limited to the order of execution shown in FIG. 9 .
  • the process 900 including the following steps is executed to process the code stream of the 3D map currently being processed.
  • Step 901 Receive a code stream of a 3D map.
  • the decoding device can use the communication link to receive the code stream of the 3D map.
  • Step 902 decompressing the code stream of the 3D map to obtain reconstruction data of the 3D map.
  • the decompression processing performed on the code stream of the 3D map may include decapsulation processing, prediction processing and/or inverse quantization processing.
  • decapsulation processing prediction processing
  • inverse quantization processing For the various processes above, reference may be made to the description of the above embodiments, and details are not repeated here.
  • FIG. 10 is a structural diagram of a 3D map decoding device 100 provided in the embodiment of the present application. As shown in FIG. The device required to compress the 3D map, for example, the server in the embodiment shown in FIG. 4b, or the electronic device in the embodiment shown in FIG. 4c.
  • the 3D map decoding device 100 includes: a memory 101 for storing a compressed 3D map in the form of a bit stream, the 3D map includes a plurality of 3D map points, and the data of the 3D map includes the The data of the plurality of 3D map points; the decoder 102 is configured to decompress the compressed 3D map in the form of a bit stream to obtain a reconstructed 3D map, and the reconstructed 3D map includes the plurality of 3D maps points, the reconstructed 3D map data includes reconstructed data of the plurality of 3D map points.
  • the memory 101 stores the compressed 3D map in the form of bit stream
  • the input data of the decoder 102 is the compressed 3D map in the form of bit stream
  • the output data is the reconstructed 3D map.
  • the decoder 102 can support the decompression processing of the compressed 3D map in the form of bit stream, that is, the decoding device of the 3D map can support the compression/decompression processing of the data of the 3D map, which can reduce the data of the 3D map In the scene where the 3D map needs to be stored, compared with storing the original 3D map data, storing the compressed 3D map in the form of a bit stream can save the space occupied by storing the 3D map data.
  • the decoder 102 may refer to the decompression module in the embodiment shown in FIG. 5 to FIG. 8 b , the difference is that the decoder does not include a decapsulation module.
  • Figure 11a is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
  • the encoded code stream of the 3D map may include: a file header, an area descriptor 1, and a 3D map point 1 , area descriptor 2, 3D map point 2, ..., area descriptor m, 3D map point n and other information.
  • the 3D map includes m region descriptors and n 3D map points, and the region descriptors and 3D map points in the encoded code stream may be arranged in a crossed manner.
  • Figure 11b is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
  • the encoded code stream of the 3D map may include: file header, area descriptor 1, and area descriptor 2 , ..., area descriptor m, 3D map point 1, 3D map point 2, ..., 3D map point n and other information.
  • the 3D map contains m region descriptors and n 3D map points, and the region descriptors and 3D map points are arranged separately in the encoded code stream.
  • the region descriptors and 3D map points are arranged separately in the encoded code stream.
  • coded code stream of the 3D map in the embodiment of the present application can also adopt other structures, and the specific structure is associated with the data content contained in the 3D map.
  • the coded code stream of the 3D map does not contain the above file header , Region Descriptor and other information in one or more.
  • Figure 11c is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
  • the encoded code stream of the 3D map may include: area descriptor 1, area descriptor 2, and area description child 3.
  • the structure of the coded code stream corresponds to the case where the above-mentioned decompression module does not include a prediction module, so there is no need to fill in the code stream with reference information for prediction.
  • Figure 11d is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
  • the encoded code stream of the 3D map may include: the reference area of the region descriptor 1 and the region descriptor 2 The index of the descriptor and the residual data of the region descriptor 2.
  • the structure of the encoded code stream corresponds to the case where the above-mentioned decompression module includes a prediction module, so reference information for prediction needs to be filled in the code stream, for example, the index of the reference region descriptor of region descriptor 2, and the region descriptor 2 is to fill in its residual data instead of the original data.
  • Figure 11e is an exemplary structural diagram of the encoded code stream of the 3D map provided by the embodiment of the present application.
  • the encoded code stream of the 3D map may include: 3D map point 1, 3D map point 2 and 3D map Point 3.
  • the code stream of the 3D map point 2 includes the 3D map point descriptor of the 3D map point 2 and the 3D map point spatial position of the 3D map point 2, if the structure of the encoded code stream corresponds to the above solution If the compression module includes the prediction module, the code stream of the 3D map point descriptor includes the reference index of the 3D map point descriptor and the residual data of the 3D map point descriptor, and the code stream of the spatial position of the 3D map point includes the 3D map point space The reference index of the position and the residual data of the spatial position of the 3D map point.
  • the content contained in the code stream of the 3D map is also different according to the compression/decompression process of the data of the 3D map.
  • the compression/decompression process includes quantization/dequantization
  • the code stream of the 3D map includes quantized data
  • the compression/decompression process includes quantization/dequantization and prediction
  • the code stream of the 3D map includes the reference index and Residual data
  • the compression/decompression process includes prediction
  • the bitstream of the 3D map includes reference indexes and residual data. Therefore, the embodiment of the present application does not specifically limit the structure of the encoded code stream of the 3D map.
  • each step of the above-mentioned method embodiments may be completed by an integrated logic circuit of hardware in a processor or instructions in the form of software.
  • the processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other possible Program logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the steps of the methods disclosed in the embodiments of the present application may be directly implemented by a hardware coded processor, or executed by a combination of hardware and software modules in the coded processor.
  • the software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register.
  • the storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware.
  • the memories mentioned in the above embodiments may be volatile memories or nonvolatile memories, or may include both volatile and nonvolatile memories.
  • the non-volatile memory can be read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically programmable Erases programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
  • Volatile memory can be random access memory (RAM), which acts as external cache memory.
  • RAM random access memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • DRAM synchronous dynamic random access memory
  • SDRAM double data rate synchronous dynamic random access memory
  • ESDRAM enhanced synchronous dynamic random access memory
  • SLDRAM direct memory bus random access memory
  • direct rambus RAM direct rambus RAM
  • the disclosed systems, devices and methods may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium.
  • the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (personal computer, server, or network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Remote Sensing (AREA)
  • Computer Graphics (AREA)
  • Signal Processing (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请提供一种3D地图的解码装置和方法、及3D地图的编码码流。本申请3D地图的解码装置，包括：传输模块和解压缩模块；其中，所述传输模块，用于接收3D地图的码流，所述3D地图包括多个3D地图点；所述解压缩模块，用于对所述3D地图的码流进行解压缩处理以得到所述3D地图的重建数据，所述3D地图的重建数据包括所述多个3D地图点的重建数据。本申请解压缩端可以支持对3D地图的压缩数据进行解压缩处理，从而支持对3D地图的数据进行压缩/解压缩处理，可以减少3D地图的数据占用的比特数，进而减少传输带宽，提高传输效率。

Description

3D地图的解码装置和方法、及3D地图的编码码流 技术领域
本申请涉及3D地图技术，尤其涉及一种3D地图的解码装置和方法、及3D地图的编码码流。
背景技术
虚拟现实(virtual reality，VR)、增强现实(augmented reality，AR)和混合现实(mixed reality，MR)技术是近年来新兴的多媒体虚拟场景技术。这一类技术可以创建虚拟现实，并将虚拟现实和真实世界叠加，产生新的可视化环境和交互体验。在这类应用中，电子设备需要确定自身在当前环境中的位姿信息，以精确的实现虚拟物体与真实场景的融合。
另一方面，在自动驾驶、自主导航、无人机自动巡检、工业机器人等应用中，车辆、无人机、机器人等载具，需要通过确定其上搭载的电子设备的位姿来确定自身在当前环境中的位姿，以进行精确的路径规划、导航、检测、操控。
针对以上应用中，需确定电子设备在当前环境中的位姿的问题，一个典型的解决方案是：电子设备从服务器或者其他设备接收所处环境的3D地图，再通过本地传感器采集所处环境中的视觉信息，结合采集的视觉信息和下载的3D地图确定电子设备的当前位姿。
然而，原始的3D地图通常包含的数据量巨大，传输地图需要消耗大量的带宽和时间，严重的限制了应用的性能、影响了用户体验。
发明内容
本申请提供一种3D地图的解码装置和方法、及3D地图的编码码流，以减少3D地图的数据量，进而减少传输带宽，提高传输效率。
第一方面，本申请提供一种3D地图的编解码系统，包括：编码装置和解码装置，所述编码装置和所述解码装置通信连接；其中，所述编码装置，用于对3D地图的数据进行压缩处理以得到所述3D地图的码流，并将所述3D地图的码流发送给所述解码装置，所述3D地图包括多个3D地图点，所述3D地图的数据包括所述多个3D地图点的数据；所述解码装置，用于接收所述3D地图的码流，并对所述3D地图的码流进行解压缩处理以得到所述3D地图的数据。
3D地图可以包括多个3D地图点，相应的，3D地图的数据可以包括多个3D地图点的数据。3D地图点是环境中令人感兴趣的或者具有显著特征的点。
本申请实施例中压缩模块可以对3D地图的数据进行压缩处理，这样可以减少3D地图的数据量，例如，使3D地图的数据量从太字节(Terabyte，TB)级减少至千兆字节(Gigabyte，GB)级，从而在需要传输3D地图的场景下，相较于传输原始的3D地图的数据，传输经压缩的3D地图的数据，一方面可以减少传输的数据量，另一方面可以减少传输中占用的带宽，提高3D地图的传输效率。
在一种可能的实现方式中，所述编码装置为云端服务器，所述解码装置为电子设备；或者，所述编码装置为第一电子设备，所述解码装置为第二电子设备；所述解码装置，还用于向所述解码装置发送3D地图下载请求，所述3D地图下载请求包括位置指示信息； 所述编码装置，还用于接收3D地图下载请求，并根据所述3D地图下载请求向所述解码装置发送与所述位置指示信息对应的3D地图的码流。
上述电子设备可以是用户终端设备。
在一种可能的实现方式中，所述编码装置为电子设备，所述解码装置为云端服务器；所述编码设备，具体用于在创建3D地图后，向所述解码装置发送所述3D地图的码流。
可选的，本申请实施例可以是电子设备通过传感器采集视觉信息，结合视觉信息以及来自服务器的3D地图确定电子设备的当前位姿。
3D地图是由服务器提供的，即服务器创建3D地图，然后对该3D地图进行压缩处理，将3D地图的压缩数据传输给电子设备；电子设备收到3D地图的压缩数据后进行解压缩处理，得到3D地图的重构数据，结合采集的视觉信息以及3D地图确定电子设备的当前位姿。位姿即电子设备的位置和朝向信息，可以是在世界坐标系中的绝对位姿，也可以是相对于环境中某个点的相对位姿。
本申请实施例中，服务器可以预先创建3D地图，将3D地图进行压缩处理后存储在本地，这样可以节省存储空间。另外，服务器也可以将3D地图的压缩数据传输给其他设备，例如，云存储器。
1、服务器创建3D地图，并压缩得到3D地图的压缩数据存储在本地。
服务器对3D地图进行压缩处理可以节省本地的存储空间。
2、电子设备向服务器发送地图下载请求，该地图下载请求的触发方式有两种：
(1)用户打开安装于电子设备上的3D地图类应用程序，该应用程序向其对应的服务器上传基于GPS定位或者WiFi定位得到的位置信息，该上传操作可以触发地图下载请求。由于上传的内容包括位置信息，因此服务器可以基于位置信息进行一个初步估计，将位置信息指示的定位点所属区域的3D地图的压缩数据传输给电子设备。位置信息指示的定位点所属区域的范围可以预先设定，例如，该所属区域可以是定位点所在的各个级别的行政区(包括县、市、国家或者行政区等)，也可以是以定位点为中心、设定距离为半径的圆周区域。
(2)用户打开安装于电子设备上的3D地图类应用程序，在该应用程序中主动输入或者选择一个区域，例如，用户主动输入“xx商业中心”，或者从一个列表“A街道、B街道、C街道”中选择了“A街道”。用户的前述操作可以触发地图下载请求。无论是用户输入或是选择包含了一个地理位置，因此服务器将该地理位置的3D地图的压缩数据传输给电子设备。
应当理解的是，本申请实施例中除了上述两种方式还可以采用其他方式触发地图下载请求，例如，电子设备自动检测是否符合下载3D地图或者启动下载3D地图的条件，或者，电子设备通过检测环境光线的变化或者检测环境的变化来启动下载3D地图，以向服务器请求下载一个区域范围内的3D地图，区域范围的大小不做具体限定。
3、服务器向电子设备发送3D地图的压缩数据。
4、电子设备采集视觉信息。
需要说明的是，步骤3和步骤4是相互独立的，不限定先后顺序。
5、电子设备对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
6、电子设备根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
电子设备在收到3D地图的压缩数据后，不需要立即解压缩，只需要在基于视觉信息进行定位之前解压缩得到3D地图的重构数据即可。例如，用户可以以下载“离线地图”的方式预先下载一个区域范围的3D地图的压缩数据，而在需要定位时，才对3D地图的压缩数据进行解压缩。
可选的，本申请实施例可以是电子设备通过传感器采集视觉信息，服务器结合来自电子设备的视觉信息以及3D地图确定电子设备的当前位姿。
3D地图是由服务器提供的，即服务器创建3D地图，然后对该3D地图进行压缩处理，将3D地图的压缩数据存储在本地。当收到来自电子设备的视觉信息时，服务器进行解压缩处理，得到3D地图的重构数据，结合视觉信息以及3D地图确定电子设备的当前位姿。
1、服务器创建3D地图，并压缩得到3D地图的压缩数据存储在本地。
2、电子设备采集视觉信息。
3、电子设备向服务器发送视觉信息。
4、服务器对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
应当理解的，服务器对3D地图进行压缩处理是为了节省存储空间。
5、服务器根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
6、服务器向电子设备发送位姿。
可选的，本申请实施例可以是电子设备通过传感器采集视觉信息，结合视觉信息以及3D地图确定电子设备的当前位姿。
3D地图是由电子设备提供的，即电子设备创建3D地图，然后对该3D地图进行压缩处理，将3D地图的压缩数据存储在本地。当采集到的视觉信息时，电子设备进行解压缩处理，得到3D地图的重构数据，结合采集的视觉信息以及3D地图确定电子设备的当前位姿。
1、电子设备创建3D地图，并压缩得到3D地图的压缩数据存储在本地。
应当理解的，电子设备对3D地图进行压缩处理是为了节省存储空间。
2、电子设备通过传感器采集得到视觉信息。
3、电子设备对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
4、电子设备根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
可选的，本申请实施例可以是第二电子设备通过传感器采集视觉信息，结合视觉信息以及来自服务器的3D地图确定第二电子设备的当前位姿。
3D地图是由第一电子设备创建的，即第一电子设备创建3D地图，对该3D地图进行压缩处理，然后将3D地图的压缩数据发送给服务器，服务器再把3D地图的压缩数据发送给第二电子设备，第二电子设备进行解压缩处理，得到3D地图的重构数据，结合采集的视觉信息以及3D地图确定第二电子设备的当前位姿。
本申请实施例中，第一电子设备可以预先创建3D地图，将3D地图进行压缩处理后传输给服务器，这样可以减少传输带宽。
1、第一电子设备创建3D地图，并压缩得到3D地图的压缩数据。
2、第一电子设备向服务器发送3D地图的压缩数据。
第一电子设备对3D地图进行压缩处理后再传输，可以减少传输带宽，提高传输效率。
3、第二电子设备向服务器发送地图下载请求。
第二电子设备发送地图下载请求也可以基于图4a所示的触发方式。
4、服务器向第二电子设备发送3D地图的压缩数据。
5、第二电子设备对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
6、第二电子设备通过传感器采集得到视觉信息。
7、第二电子设备根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
可选的，本申请实施例可以是第二电子设备通过传感器采集视觉信息，服务器结合来自第二电子设备的视觉信息以及来自第一电子设备的3D地图确定第二电子设备的当前位姿。
3D地图是由第一电子设备创建的，即第一电子设备创建3D地图，对该3D地图进行压缩处理，然后将3D地图的压缩数据发送给服务器。服务器进行解压缩处理，得到3D地图的重构数据，结合来自第二电子设备的视觉信息以及3D地图确定第二电子设备的当前位姿。
1、第一电子设备创建3D地图，并压缩得到3D地图的压缩数据。
2、第一电子设备向服务器发送3D地图的压缩数据。
3、第二电子设备通过传感器采集得到视觉信息。
4、第二电子设备向服务器发送定位请求，该定位请求中携带视觉信息。
5、服务器对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
6、服务器根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
7、服务器向第二电子设备发送定位得到的位姿。
可选的，本申请实施例可以是第二电子设备通过传感器采集视觉信息，结合视觉信息以及来自第一电子设备的3D地图确定第二电子设备的当前位姿。
3D地图是由第一电子设备创建的，即第一电子设备创建3D地图，对该3D地图进行压缩处理，然后将3D地图的压缩数据发送给第二电子设备，第二电子设备进行解压缩处理，得到3D地图的重构数据，结合采集的视觉信息以及来自第一电子设备的3D地图确定第二电子设备的当前位姿。
1、第一电子设备创建3D地图，并压缩得到3D地图的压缩数据存储在本地。
2、第二电子设备向第一电子设备发送地图下载请求。
3、第一电子设备向第二电子设备发送3D地图的压缩数据。
4、第二电子设备对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
5、第二电子设备通过传感器采集得到视觉信息。
6、第二电子设备根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
在一种可能的实现方式中，所述3D地图的数据还包括多个区域描述子，所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
针对多个区域描述子中的任意一个区域描述子，该区域描述子可以用于描述多个3D地图点中的部分或全部3D地图点的特征，这样区域描述子和3D地图点是一对多的关系。而多个3D地图点中的每个3D地图点的特征可以被多个区域描述子中的部分或全部区域描述子描述，这样3D地图点和区域描述子是一对多的关系。由此可见，多个区域描述子和多个3D地图点之间是多对多的关系。区域描述子的生成方法包括但不限于词袋(bag of  words，BOW)、局部聚集描述子向量(vector of aggragate locally descriptor，VLAD)等传统方法以及基于NetVLAD、人工智能(artificial intelligence，AI)的新型方法。同理，多个区域描述子也可以用编号进行标识，以对多个区域描述子进行区分，但是该编号同样不是对多个区域描述子之间的先后顺序的限定。
在一种可能的实现方式中，所述多个3D地图点中的任意一个3D地图点的数据包括3D地图点描述子和3D地图点空间位置。
3D地图点描述子是一条向量(vector)，用来表示3D地图点的局部特征。
3D地图点空间位置可以采用三维空间轴线上的X、Y、Z表示，也可以采用经纬度、海拔高度表示，还可以采用极坐标等方式表示，本申请实施例对3D地图点空间位置的表示方法不做具体限制。3D地图点空间位置可以是3D地图点的绝对位置，也可以是3D地图点的相对位置
在一种可能的实现方式中，所述编码装置，还用于创建所述3D地图。
在一种可能的实现方式中，所述解码装置，还用于根据所述3D地图进行定位。
第二方面，本申请提供一种3D地图的解码装置，包括：传输模块和解压缩模块；其中，所述传输模块，用于接收3D地图的码流，所述3D地图包括多个3D地图点；所述解压缩模块，用于对所述3D地图的码流进行解压缩处理以得到所述3D地图的重建数据，所述3D地图的重建数据包括所述多个3D地图点的重建数据。
本申请实施例中解压缩模块可以支持对3D地图的压缩数据进行解压缩处理，亦即，编解码系统可以支持对3D地图的数据进行压缩/解压缩处理，这样可以减少3D地图的数据量，在需要传输3D地图的场景下，相较于传输原始的3D地图的数据，传输3D地图的压缩数据，一方面可以减少传输的数据量，另一方面可以减少传输中占用的带宽，提高3D地图的传输效率。
本申请实施例中，解压缩处理可以包括反量化处理和预测处理中的至少一者，其中，
量化处理是指将待处理数据映射为一个或多个量化索引，每个量化索引对应一个量化中心，量化索引的比特数通常会明显少于原始数据的比特数，从而达到节省存储或传输带宽的目的。量化处理的方法包括但不限于：标量量化、矢量量化、乘积量化等。基于此，反量化处理是前述量化处理的反向处理，即根据一个或多个量化索引恢复出对应于原始数据的重建数据，重建数据的比特数多于量化索引的比特数。
预测处理是指使用已处理数据和待处理数据的残差数据进行预测以得到待处理数据。参考数据的选择可以是预先约定的，例如，固定使用前一个处理的数据做参考数据，此时可以不用在码流中标识参考数据；又例如，采用任意一个已处理数据，此时要在码流中写入参考数据的标识信息，包括参考数据的编号，或者其他能够推断出参考数据的信息。
在上述关于3D地图的数据的描述中，可知3D地图包含的多个3D地图点之间的先后顺序没有实际意义，因此解码端如果涉及对3D地图点的解压缩或解码处理，并不限定多个3D地图点的先后顺序，可以按照码流的先后顺序对多个3D地图点分别进行解压缩或解码处理。
另外，在编码端，对3D地图的数据的处理包括封装处理，目的是为了将待编码的数据封装成码流，封装处理可以采用任意一种编码算法，例如熵编码。熵编码是一种数据的 无损压缩方法，熵编码算法包括但不限于：哈夫曼(huffman)编码、算术编码，由LZ77压缩算法改进的压缩/解压(lempel-ziv-markov chain-algorithm，LZMA)算法，提供数据压缩用的函数库(zlib)算法等。相应的，在解码端，解压缩处理还可以包括解封装处理，目的是为了将码流解封装成待解压缩的数据，解封装处理可以采用与编码端采用的编码算法对应的解码算法，例如熵解码。
在一种可能的实现方式中，所述3D地图的数据还包括多个区域描述子的重建数据，所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
在一种可能的实现方式中，所述多个3D地图点中的任意一个3D地图点的重建数据包括3D地图点描述子的重建数据和3D地图点空间位置的重建数据。
在一种可能的实现方式中，所述3D地图的解码装置为云端服务器或者电子设备；所述传输模块，还用于发送3D地图下载请求，所述3D地图下载请求包括位置指示信息；接收与所述位置指示信息对应的所述3D地图的码流。
在一种可能的实现方式中，所述3D地图的解码装置为云端服务器；所述传输模块，具体用于接收由电子设备创建的所述3D地图的码流。
以下是压缩模块的多种实施方式：
在一种可能的实现方式中，所述解压缩模块包括解封装模块，以及预测模块和/或反量化模块；所述解封装模块，用于对输入的所述3D地图的码流进行处理以输出第一数据；所述预测模块，用于对输入的第二数据的残差数据进行预测处理以输出所述第二数据；所述反量化模块，用于对输入的第三数据进行反量化处理以输出所述第三数据的反量化数据；其中，所述第一数据为所述第二数据的残差数据或者所述第三数据；所述第二数据为所述3D地图的重建数据或者所述第三数据；所述第三数据的反量化数据为所述3D地图的重建数据。
可选的，解压缩模块只包括解封装模块，以及反量化模块或者预测模块。
可选的，解压缩模块包括预测模块和反量化模块，以及封装模块。
在一种可能的实现方式中，所述预测模块包括第一预测模块和第二预测模块；和/或，所述反量化模块包括第一反量化模块和第二反量化模块；所述第一预测模块，用于对输入的第四数据的残差数据进行预测处理以输出所述第四数据；所述第一反量化模块，用于对输入的第五数据进行反量化处理以输出所述第五数据的反量化数据；其中，所述第四数据为多个区域描述子的其中之一的重建数据或者所述第五数据；所述第五数据的反量化数据为所述多个区域描述子的其中之一的重建数据；所述第二预测模块，用于对输入的第六数据的残差数据进行预测处理以输出所述第六数据；所述第二反量化模块，用于对输入的第七数据进行反量化处理以输出所述第七数据的反量化数据；其中，所述第六数据为所述多个3D地图点的其中之一的重建数据或者所述第七数据；所述第七数据的反量化数据为所述多个3D地图点的其中之一的重建数据。
可选的，解压缩模块只包括第一预测模块和第二预测模块，或者只包括第一反量化模块和第二反量化模块。
可选的，解压缩模块包括第一预测模块和第二预测模块，以及第一反量化模块和第二反量化模块。
在一种可能的实现方式中，所述预测模块包括第一预测模块、第二预测模块和第三预测模块；和/或，所述反量化模块包括第一反量化模块、第二反量化模块和第三反量化模块；所述第一预测模块，用于对输入的第八数据的残差数据进行预测处理以输出所述第八数据；所述第一反量化模块，用于对输入的第九数据进行反量化处理以输出所述第九数据的反量化数据；其中，所述第八数据为多个区域描述子的其中之一的重建数据或者所述第九数据；所述第九数据的反量化数据为所述多个区域描述子的其中之一的重建数据；所述第二预测模块，用于对输入的第十数据的残差数据进行预测处理以输出所述第十数据；所述第二反量化模块，用于对输入的第十一数据进行反量化处理以输出所述第十一数据的反量化数据；其中，所述第十数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据或者所述第十一数据；所述第十一数据的反量化数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据；所述第三预测模块，用于对输入的第十二数据的残差数据进行预测处理以输出所述第十二数据；所述第三反量化模块，用于对输入的第十三数据进行反量化处理以输出所述第十三数据的反量化数据；其中，所述第十二数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据或者所述第十三数据；所述第十三数据的反量化数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据。
可选的，解压缩模块只包括第一预测模块、第二预测模块和第三预测模块，或者只包括第一反量化模块、第二反量化模块和第三反量化模块。
可选的，解压缩模块包括第一预测模块、第二预测模块和第三预测模块，以及第一反量化模块、第二反量化模块和第三反量化模块。
在一种可能的实现方式中，所述解压缩模块包括第一解压缩子模块和第二解压缩子模块；所述第一解压缩子模块，用于对输入的第十四数据的码流进行解压缩处理以输出所述第十四数据；所述第二解压缩子模块，用于对输入的第十五数据的码流进行压缩处理以输出所述第十五数据；其中，所述第十四数据为多个区域描述子的其中之一的重建数据；所述第十五数据为所述多个3D地图点的其中之一的重建数据。
在一种可能的实现方式中，所述第一解压缩子模块包括第一解封装模块，以及第一预测模块和/或第一反量化模块；所述第二解压缩子模块包括第二解封装模块，以及第二预测模块和/或第二反量化模块；所述第一解封装模块，用于对输入的所述3D地图的码流进行处理以输出第十六数据；所述第一预测模块，用于对输入的第十七数据的残差数据进行预测处理以输出所述第十七数据；所述第一反量化模块，用于对输入的第十八数据进行反量化处理以输出所述第十八数据的反量化数据；其中，所述第十六数据为所述第十七数据的残差数据或者所述第十八数据；所述第十七数据为所述多个区域描述子的其中之一的重建数据或者所述第十八数据；所述第十八数据的反量化数据为所述多个区域描述子的其中之一的重建数据；所述第二解封装模块，用于对输入的所述3D地图的码流进行处理以输出第十九数据；所述第二预测模块，用于对输入的第二十数据的残差数据进行预测处理以输出所述第二十数据；所述第二反量化模块，用于对输入的第二十一数据进行反量化处理以输出所述第二十一数据的反量化数据；其中，所述第十九数据为所述第二十数据的残差数据或者所述第二十一数据；所述第二十数据为所述多个3D地图点的其中之一的重建数据或者所述第二十一数据；所述第二十一数据的反量化数据为所述多个3D地图点的其中之一的重建数据。
在一种可能的实现方式中，所述解压缩模块包括第一解压缩子模块、第二解压缩子模块和第三解压缩子模块；所述第一解压缩子模块，用于对输入的第二十二数据的码流进行解压缩处理以输出所述第二十二数据；所述第二解压缩子模块，用于对输入的第二十三数据的码流进行压缩处理以输出所述第二十三数据；所述第三解压缩子模块，用于对输入的第二十四数据的码流进行压缩处理以输出所述第二十四数据；其中，所述第二十二数据为多个区域描述子的其中之一的重建数据；所述第二十三数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据；所述第二十四数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据。
在一种可能的实现方式中，所述第一解压缩子模块包括第一解封装模块，以及第一预测模块和/或第一反量化模块；所述第二解压缩子模块包括第二解封装模块，以及第二预测模块和/或第二反量化模块；所述第三解压缩子模块包括第三解封装模块，以及第三预测模块和/或第三反量化模块；所述第一解封装模块，用于对输入的所述3D地图的码流进行处理以输出第二十五数据；所述第一预测模块，用于对输入的第二十六数据的残差数据进行预测处理以输出所述第二十六数据；所述第一反量化模块，用于对输入的第二十七数据进行反量化处理以输出所述第二十七数据的反量化数据；其中，所述第二十五数据为所述第二十六数据的残差数据或者所述第二十七数据；所述第二十六数据为所述多个区域描述子的其中之一的重建数据或者所述第二十七数据；所述第二十七数据的反量化数据为所述多个区域描述子的其中之一的重建数据；所述第二解封装模块，用于对输入的所述3D地图的码流进行处理以输出第二十八数据；所述第二预测模块，用于对输入的第二十九数据的残差数据进行预测处理以输出所述第二十九数据；所述第二反量化模块，用于对输入的第三十数据进行反量化处理以输出所述第三十数据的反量化数据；其中，所述第二十八数据为所述第二十九数据的残差数据或者所述第三十数据；所述第二十九数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据或者所述第三十数据；所述第三十数据的反量化数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据；所述第三解封装模块，用于对输入的所述3D地图的码流进行处理以输出第三十一数据；所述第三预测模块，用于对输入的第三十二数据的残差数据进行预测处理以输出所述第三十二数据；所述第三反量化模块，用于对输入的第三十三数据进行反量化处理以输出所述第三十三数据的反量化数据；其中，所述第三十一数据为所述第三十二数据的残差数据或者所述第三十三数据；所述第三十二数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据或者所述第三十三数据；所述第三十三数据的反量化数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据。
第三方面，本申请提供一种3D地图的解码方法，包括：接收3D地图的码流，所述3D地图包括多个3D地图点；对所述3D地图的码流进行解压缩处理以得到所述3D地图的重建数据，所述3D地图的重建数据包括所述多个3D地图点的重建数据。
本申请实施例中解压缩模块可以支持对3D地图的压缩数据进行解压缩处理，亦即，编解码系统可以支持对3D地图的数据进行压缩/解压缩处理，这样可以减少3D地图的数据量，在需要传输3D地图的场景下，相较于传输原始的3D地图的数据，传输3D地图的压缩数据，一方面可以减少传输的数据量，另一方面可以减少传输中占用的带宽，提高3D地图的传输效率。
在一种可能的实现方式中，所述3D地图的重建数据还包括多个区域描述子的重建数据，所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
在一种可能的实现方式中，所述多个3D地图点中的任意一个3D地图点的重建数据包括3D地图点描述子的重建数据和3D地图点空间位置的重建数据。
在一种可能的实现方式中，还包括：发送3D地图下载请求，所述3D地图下载请求包括位置指示信息；所述接收3D地图的码流，包括：接收与所述位置指示信息对应的所述3D地图的码流。
在一种可能的实现方式中，所述接收3D地图的码流，包括：接收由电子设备创建的所述3D地图的码流。
在一种可能的实现方式中，所述对所述3D地图的码流进行解压缩处理以得到所述3D地图的数据，包括：对所述3D地图的码流进行处理以得到第一数据；对第二数据的残差数据进行预测处理以得到所述第二数据，和/或，对第三数据进行反量化处理以得到所述第三数据的反量化数据；其中，所述第一数据为所述第二数据的残差数据或者所述第三数据；所述第二数据为所述3D地图的重建数据或者所述第三数据；所述第三数据的反量化数据为所述3D地图的重建数据。
第四方面，本申请提供一种3D地图的解码装置，包括：存储器，用于存储比特流形式的经压缩得到的3D地图，所述3D地图包括多个3D地图点，所述3D地图的数据包括所述多个3D地图点的数据；解码器，用于对所述比特流形式的经压缩得到的3D地图进行解压缩处理，得到重建3D地图，所述重建3D地图包括所述多个3D地图点，所述重建3D地图的数据包括所述多个3D地图点的重建数据。
在一种可能的实现方式中，所述重建3D地图的数据还包括多个区域描述子的重建数据，所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
在一种可能的实现方式中，所述多个3D地图点中的任意一个3D地图点的重建数据包括3D地图点描述子的重建数据和3D地图点空间位置的重建数据。
在一种可能的实现方式中，所述3D地图的编码装置为云端服务器或者电子设备。
第五方面，本申请提供一种计算机可读存储介质，其特征在于，包括计算机程序，所述计算机程序在计算机上被执行时，使得所述计算机执行上述第三方面中任一项所述的方法。
第六方面，本申请提供一种计算机程序产品，所述计算机程序产品包括计算机程序代码，当所述计算机程序代码在计算机上运行时，使得计算机执行上述第三方面中任一项所述的方法。
第七方面，本申请提供一种3D地图的编码码流，所述3D地图的编码码流包括多个3D地图点的码流，所述3D地图包括所述多个3D地图点。
在一种可能的实现方式中，所述3D地图的编码码流还包括多个区域描述子的码流，所述多个区域描述子中的任意一个区域描述子和所述多个3D地图点中的至少一个3D地图点对应。
在一种可能的实现方式中，所述多个3D地图点中的任意一个3D地图点的码流包括 3D地图点描述子的码流和3D地图点空间位置的码流。
在一种可能的实现方式中，所述多个3D地图点中的任意一个3D地图点的码流包括3D地图点的残差数据。
在一种可能的实现方式中，所述3D地图点描述子的码流包括所述3D地图点描述子的残差数据；和/或，所述3D地图点空间位置的码流包括所述3D地图点空间位置的残差数据。
在一种可能的实现方式中，所述多个3D地图点中的任意一个3D地图点的码流还包括参考3D地图点的指示信息，所述多个3D地图点包括所述参考3D地图点，且所述参考3D地图点是在编码所述任意一个3D地图点之前已编码的3D地图点。
在一种可能的实现方式中，所述多个3D地图点中的任意一个3D地图点的码流包括3D地图点的量化数据。
在一种可能的实现方式中，所述3D地图点描述子的码流包括所述3D地图点描述子的量化数据；和/或，所述3D地图点空间位置的码流包括所述3D地图点空间位置的量化数据。
在一种可能的实现方式中，所述多个区域描述子中的任意一个区域描述子的码流包括区域描述子的残差数据。
在一种可能的实现方式中，所述多个区域描述子中的任意一个区域描述子的码流还包括参考区域描述子的指示信息，所述多个区域描述子包括所述参考区域描述子。
在一种可能的实现方式中，所述多个区域描述子中的任意一个区域描述子的码流包括区域描述子的量化数据。
附图说明
图1为本申请实施例所提供的一种应用架构的示意图；
图2为本申请实施例提供的电子设备20的一种结构示意图；
图3为本申请实施例提供的服务器30的一种结构示意图；
图4a为本申请实施例提供的一个应用场景的示意图；
图4b为本申请实施例提供的一个应用场景的示意图；
图4c为本申请实施例提供的一个应用场景的示意图；
图4d为本申请实施例提供的一个应用场景的示意图；
图4e为本申请实施例提供的一个应用场景的示意图；
图4f为本申请实施例提供的一个应用场景的示意图；
图4g为本申请实施例提供的电子设备显示的一种用户界面的示意图；
图5为本申请实施例提供的3D地图的解码装置50的一个结构图；
图6a为本申请实施例提供的3D地图的解码装置60的一个结构图；
图6b为本申请实施例提供的3D地图的解码装置60的一个结构图；
图6c为本申请实施例提供的3D地图的解码装置60的一个结构图；
图6d为本申请实施例提供的3D地图的解码装置60的一个结构图；
图6e为本申请实施例提供的3D地图的解码装置60的一个结构图；
图7a为本申请实施例提供的3D地图的解码装置70的一个结构图；
图7b为本申请实施例提供的3D地图的解码装置70的一个结构图；
图8a为本申请实施例提供的3D地图的解码装置80的一个结构图；
图8b为本申请实施例提供的3D地图的解码装置80的一个结构图；
图9为本申请实施例提供的3D地图的解码方法的过程900的一个流程图；
图10为本申请实施例提供的3D地图的解码装置100的一个结构图；
图11a为本申请实施例提供的3D地图的编码码流的一个示例性的结构图；
图11b为本申请实施例提供的3D地图的编码码流的一个示例性的结构图；
图11c为本申请实施例提供的3D地图的编码码流的一个示例性的结构图；
图11d为本申请实施例提供的3D地图的编码码流的一个示例性的结构图；
图11e为本申请实施例提供的3D地图的编码码流的一个示例性的结构图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合本申请中的附图，对本申请中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请的说明书实施例和权利要求书及附图中的术语“第一”、“第二”等仅用于区分描述的目的，而不能理解为指示或暗示相对重要性，也不能理解为指示或暗示顺序。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元。方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
应当理解，在本申请中，“至少一个(项)”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，用于描述关联对象的关联关系，表示可以存在三种关系，例如，“A和/或B”可以表示：只存在A，只存在B以及同时存在A和B三种情况，其中A，B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项(个)”或其类似表达，是指这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a，b或c中的至少一项(个)，可以表示：a，b，c，“a和b”，“a和c”，“b和c”，或“a和b和c”，其中a，b，c可以是单个，也可以是多个。
图1为本申请实施例所提供的一种应用架构的示意图，如图1所示，该应用架构包括多个电子设备和服务器，其中，多个电子设备可以包括第一电子设备、一个或多个第二电子设备(图1中以两个第二电子设备为例)，该一个或多个第二电子设备是除第一电子设备外的若干个电子设备。多个电子设备和服务器之间、多个电子设备彼此之间均可进行通信，例如，对于应用架构中的任意一个设备，可以通过无线保真(wireless-fidelity，WiFi)通信、蓝牙通信或者蜂窝2/3/4/5代(2/3/4/5generation，2G/3G/4G/5G)通信等方式与其它设备进行通信，应当理解的，服务器和电子设备之间还可以采用其他通信方式，包括未来的通信方式，对此不做具体限定。需要说明的是，本申请实施例中的“一个或多个第二电子设备”仅用于表示除第一电子设备外的其它电子设备，但并不限定多个电子设备的类型是否相同。
上述电子设备，可以是配置有摄像头和显示组件的各种类型的设备，例如，电子设备 可以是手机、平板电脑、笔记本电脑、录像机等终端设备(图1中以电子设备是手机为例)，电子设备也可以是用于虚拟场景交互的设备，包括VR眼镜、AR设备，MR互动设备等，电子设备还可以是智能手表、智能手环等可穿戴电子设备，电子设备还可以是车辆、无人驾驶车辆、无人机、工业机器人等载具中所搭载的设备。本申请实施例对电子设备的具体形式不做具体限定。
此外，上述电子设备也可以被称为用户装备(user equipment，UE)、订户站、移动单元、订户单元、无线单元、远程单元、移动设备、无线设备、无线通信设备、远程设备、移动订户站、终端设备、接入终端、移动终端、无线终端、智能终端、远程终端、手持机、用户代理、移动客户端、客户端、或其他某个合适的术语。
上述服务器可以是一台或多台物理服务器(图1中以一台物理服务器为例)，也可以是计算机集群，还可以是云计算场景的虚拟机或者云端服务器，等等。
本申请实施例中，电子设备可以安装VR应用、AR应用或者MR应用等虚拟场景应用程序(application，APP)，并可以基于用户的操作(例如点击、触摸、滑动、抖动、声控等)运行VR应用、AR应用或者MR应用。电子设备可以通过传感器采集环境中任意物体的视觉信息，再根据采集的视觉信息在显示组件上显示虚拟物体，该虚拟物体可以是VR场景、AR场景或MR场景中的虚拟物体(即虚拟环境中的物体)。
本申请实施例中，电子设备可以安装导航、检测、控制类应用程序，并基于用户的操控或是预设的程序来运行相应的应用。电子设备可以基于自身在当前环境中的位姿和其他状态信息，进行路径规划、物体检测、载具操控等应用。
本申请实施例所涉及的视觉信息，包括但不限于摄像头采集得到的图像视频(不带深度信息)、深度感应器(depth sensor)采集得到的带深度信息的图像视频、激光雷达(LiDAR)采集得到的数据、毫米波雷达(RaDAR)采集得到的数据。
需要说明的是，本申请实施例中，电子设备中的虚拟场景应用程序可以是电子设备自身内置的应用程序，也可以是用户自行安装的第三方服务商提供的应用程序，对此不做具体限定。
本申请实施例中，电子设备还可以配置有即时定位与地图构建(simultaneous localization and mapping，SLAM)系统，SLAM系统能够在完全未知的环境中创建地图，并利用该地图进行定位、位姿(位置与姿态)确定、导航等。本申请实施例中将SLAM系统创建的地图称作SLAM地图，SLAM地图可以理解为是SLAM系统根据采集设备采集到的环境信息绘制出的地图，采集设备可以包括电子设备中的视觉信息采集装置和惯性测量单元(inertial measurement unit，IMU)，其中，视觉信息采集装置例如可以包括相机、摄像头、深度相机、激光雷达、毫米波雷达等，IMU例如可以包括陀螺仪、加速度计等传感器。
本申请实施例中将SLAM地图亦称作3D地图。需要说明的是，3D地图包括但不限于SLAM地图，其还可以包括采用其它技术创建的三维地图，本申请实施例对此不做具体限定。
在一种可能的实现方式中，3D地图可以包括多个3D地图点，相应的，3D地图的数据可以包括多个3D地图点的数据。3D地图点是环境中令人感兴趣的或者具有显著特征的点。
3D地图点的一种可能的获取方式为，使用激光雷达、无人机视角的空中摄影(倾斜摄影)、高清全景相机、高清工业相机等多种设备拍摄，通过面向基于加速分割测试的特征(features from accelerated segment test，FAST)和旋转二进制鲁棒独立基本特征(binary robust independent elementary features，BRIEF)描述子的特征(oriented FAST and rotated BRIEF，ORB)、尺度不变特征转换(scale invariant feature transform，SIFT)、加速的具有鲁棒性的特征(speeded-up robust feature，SURF)、BRIEF、二进制鲁棒不变可扩展关键点(binary robust invariant scalable keypoints，BRISK)、快速视网膜关键点(fast retina keypoint，FREAK)、可重复且可靠的检测和描述符(repeatable and reliable detector and descriptor，R2D2)等方法从上述设备拍摄到的数据中提取。
3D地图点的数据可以包括：
(1)3D地图点描述子
3D地图点描述子是一条向量，用来表示3D地图点的局部特征。在视觉定位算法中，3D地图点描述子用于进行3D地图点间的匹配。一种可能的做法是，计算两个3D地图点描述子之间的距离(可以是欧式距离、内积距离、汉明距离等)，当距离小于一个阈值时，视为两个3D地图点匹配。
(2)3D地图点空间位置
3D地图点空间位置可以采用三维空间轴线上的X、Y、Z表示，也可以采用经纬度、海拔高度表示，还可以采用极坐标等方式表示，本申请实施例对3D地图点空间位置的表示方法不做具体限制。3D地图点空间位置可以是3D地图点的绝对位置，也可以是3D地图点的相对位置，例如，以整个区域的中心位置为原点，所有3D地图点空间位置都是相对于原点的空间位置的偏移位置。
本申请实施例中，可以给每个3D地图点分配一个编号写入3D地图的数据中，也可以利用多个3D地图点在内存中的存储顺序隐含表示3D地图点的编号。需要说明的是，3D地图包含的多个3D地图点之间的先后顺序没有实际意义，因此前述编号可以认为是用于标识3D地图点的标识符，以对各个3D地图点加以区分，但该编号并不用于限定多个3D地图点之间的先后顺序，例如，3D地图包含3个3D地图点，其编号分别是1、2和3，对该3个3D地图点进行的处理，可以按照1、2、3的顺序进行，也可以按照3、2、1的顺序进行，还可以按照2、1、3的顺序进行，等等。
在一种可能的实现方式中，3D地图的数据还包括多个区域描述子，该多个区域描述子中的任意一个区域描述子用于描述上述多个3D地图点中的部分或全部3D地图点的特征，即针对多个区域描述子中的任意一个区域描述子，该区域描述子可以用于描述多个3D地图点中的部分或全部3D地图点的特征，这样区域描述子和3D地图点是一对多的关系。而多个3D地图点中的每个3D地图点的特征可以被多个区域描述子中的部分或全部区域描述子描述，这样3D地图点和区域描述子是一对多的关系。由此可见，多个区域描述子和多个3D地图点之间是多对多的关系。区域描述子的生成方法包括但不限于词袋(bag of words，BOW)、局部聚集描述子向量(vector of aggragate locally descriptor，VLAD)等传统方法以及基于NetVLAD、人工智能(artificial intelligence，AI)的新型方法。同理，多个区域描述子也可以用编号进行标识，以对多个区域描述子进行区分，但是该编号同样不是对多个区域描述子之间的先后顺序的限定。
在一种可能的实现方式中，3D地图的数据还包括3D地图点和区域描述子的对应关系，在该对应关系中明确的描述了任意一个区域描述子对应于哪些3D地图点，以及任意一个3D地图点对应于哪些区域描述子。
可选的，上述对应关系可以显式的用区域描述子的编号和3D地图点的编号之间的对应表进行描述，例如，3D地图包含3个区域描述子，编号是T1～T3，6个3D地图点，编号是P1～P6，对应表如表1或表2所示。
表1
Figure PCTCN2021098483-appb-000001
需要说明的是，表1是区域描述子的编号和3D地图点的编号之间的对应表的一个示例，该对应表也可以采用其他的格式或方式呈现，本申请对此不做具体限定。
可选的，上述对应关系也可以隐式的用区域描述子和3D地图点的存储位置进行描述，例如，在内存中先存储T1，跟着存储P1、P2和P3的数据，接着存储T2，跟着存储P2和P3的数据，最后存储T3，跟着存储P3、P4、P5和P6的数据。
图2为本申请实施例提供的电子设备20的一种结构示意图，如图2所示，电子设备20可以是图1所示实施例中的第一电子设备和一个或多个第二电子设备中的至少一者。应该理解的是，图2所示的结构并不构成对电子设备20的具体限定。在本申请另一些实施例中，电子设备20可以包括比图2所示结构更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图2中所示出的各种部件可以在包括一个或多个信号处理和/或专用集成电路在内的硬件、软件、或硬件和软件的组合中实现。
电子设备20可包括：芯片21、存储器22(一个或多个计算机可读存储介质)、用户接口23、显示组件24、摄像头25、传感器26、用于设备定位的定位模块27以及用于通信的收发器28。这些部件之间可以通过一个或多个总线29通信。
芯片21可以集成包括：一个或多个处理器211、时钟模块212以及电源管理模块213。集成于芯片21中的时钟模块212主要用于为处理器211提供数据传输和时序控制所需要的计时器，计时器可以实现数据传输和时序控制的时钟功能。处理器211可以根据指令操作码和时序信号，执行运算，产生操作控制信号，完成取指令和执行指令的控制。集成于芯片21中的电源管理模块213主要用于为芯片21以及电子设备20的其他部件提供稳定的、高精确度的电压。
处理器211又可称为中央处理器(central processing unit，CPU)，处理器211具体可以包括一个或多个处理单元，例如，处理器211可以包括应用处理器(application processor， AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或，神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
在一种可能的实现方式中，处理器211可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或，通用串行总线(universal serial bus，USB)接口等。
存储器22可以和处理器211通过总线29连接，也可以与处理器311耦合在一起，用于存储各种软件程序和/或多组指令。存储器22可以包括高速随机存取的存储器(例如高速缓冲存储器)，也可以包括非易失性存储器，例如一个或多个磁盘存储设备、闪存设备或其他非易失性固态存储设备。存储器22可以存储操作系统，例如，安卓(Android)，苹果移动平台(IOS)，微软窗口操作系统(Windows)或者类UNIX操作系统(Linux)等嵌入式操作系统。存储器22还可以存储数据，例如，图像数据、点云数据、3D地图的数据、位姿数据、坐标系转换信息、地图更新信息等。存储器22还可以存储计算机可执行的程序代码，该计算机可执行的程序代码包括指令，例如，通信程序指令，SLAM系统的相关程序指令等。存储器22还可以存储一个或多个应用程序，例如，AR/VR/MR等虚拟场景应用程序、3D地图类应用程序、图像管理类应用程序、导航与控制类应用程序等。存储器22还可以存储用户界面程序，该用户界面程序可以通过图形化的操作界面将应用程序的内容，例如，AR/VR/MR等虚拟场景中的虚拟物体，形象逼真的显示出来并通过显示组件24呈现，以及实现通过菜单、对话框和按键等输入控件接收用户对应用程序的控制操作。
用户接口23例如可以是触控面板，该触控面板可以检测用户在其上的操作指令，用户接口23也例如可以是小键盘、物理按键或者鼠标等。
电子设备20可以包括一个或多个显示组件24。电子设备20可以通过显示组件24、芯片21中的图形处理器(GPU)和应用处理器(AP)等共同实现显示功能。GPU是实现图像处理的微处理器，其连接显示组件24和应用处理器，GPU执行数学和几何计算以用于图形渲染。显示组件24可以显示电子设备20输出的界面内容，例如，显示AR/VR/MR等虚拟场景中的图像、视频等，界面内容可包括正在运行的应用程序的界面以及系统级别菜单等，具体可以由下述界面元素组成：输入型界面元素，例如按键(Button)，文本输入框(Text)，滑动条(Scroll Bar)，菜单(Menu)等；输出型界面元素，例如视窗(Window)，标签(Label)，图像，视频，动画等。
显示组件24可以是显示面板，镜片(例如VR眼镜)，投影屏等。显示面板也可以称为显示屏，例如，可以是触摸屏、柔性屏、曲面屏等，也可以是其它光学组件。应当理解的，本申请实施例中的电子设备具有的显示屏可以是触摸屏、柔性屏、曲面屏或者其它形式的屏幕，亦即电子设备的显示屏具有显示图像的功能，而对于显示屏的具体材质和形状 不作具体限定。
例如，当显示组件24包括显示面板时，显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。此外，在一种可能的实现方式中，可以将用户接口23中的触控面板和显示组件24中的显示面板耦合在一起，例如，触控面板可设置于显示面板下方，触控面板用于检测用户通过显示面板输入触控操作(例如点击、滑动，触摸等)时作用于显示面板上的触控压力，显示面板用于进行内容显示。
摄像头25可以是单目摄像头或双目摄像头或深度摄像头，用于对环境进行拍摄/录像，以获得图像/视频图像。摄像头25所采集的图像/视频图像例如可作为SLAM系统的一种输入数据，或者可通过显示组件24进行图像/视频显示。
在一种可能的实现方式中，也可以把摄像头25视为一种传感器。摄像头25采集的图像可以是IMG格式，也可以是其他格式类型，本申请实施例对此不做具体限定。
传感器26可以用于采集与电子设备20的状态变化(例如，旋转、摆动、移动、抖动等)相关的数据，传感器26所采集的数据可以作为SLAM系统的一种输入数据。传感器26可以包括一种或多种传感器，例如，惯性测量单元(inertial measurement unit，IMU)，飞行时间(time of flight，TOF)传感器等。IMU中可以包括陀螺仪、加速度计等传感器，陀螺仪用于测量电子设备运动时的角速度，加速度计用于测量电子设备运动时的加速度。TOF传感器中可以包括光发射器和光接收器，光发射器用于向外发射光线，例如，激光、红外线、雷达波等，光接收器用于检测反射的光线，例如，反射的激光、红外线、雷达波等。
需要说明的，传感器26还可以包括更多的其他传感器，例如惯性传感器，气压计，磁力计，轮速计等，本申请实施例对此不做具体限定。
定位模块27用于实现对电子设备20的物理定位，例如，用于获取电子设备20的初始位置。定位模块27可以包括WiFi定位模块、蓝牙定位模块、基站定位模块、卫星定位模块中的一个或多个。卫星定位模块中可以设置有全球导航卫星系统(global navigation satellite system，GNSS)以辅助定位，GNSS不限于北斗系统、全球定位系统(global positioning system，GPS)系统、格洛纳斯(global navigation satellite system，GLONASS)系统、伽利略卫星导航系统(Galileo)系统。
收发器28用于实现电子设备20与其它设备(例如，服务器、其它电子设备等)之间的通信。收发器28集成了发射器和接收器，分别用于发送和接收射频信号。具体实现中，收发器28包括但不限于：天线系统、射频(radio frequency，RF)收发器、一个或多个放大器、调谐器、一个或多个振荡器、数字信号处理器、编译码器(CODEC)芯片、用户身份识别卡(subscriber identification module，SIM)卡和存储介质等。在一种可能的实现方式中，还可以在单独的芯片上实现收发器28。收发器28支持2G/3G/4G/5G等中的至少一种数据网络通信，和/或，支持以下近距无线通信的方式中的至少一种：蓝牙(bluetooth，BT)通信，无线保真(wirelessfidelity，WiFi)通信，近场通信(near field communication， NFC)，红外(infrared，IR)无线通信，超宽带(ultra wide band，UWB)通信，紫蜂协议(ZigBee)通信。
本申请实施例中，处理器211通过运行存储在存储器22的程序代码，从而执行电子设备20的各种功能应用和数据处理。
图3为本申请实施例提供的服务器30的一种结构示意图，如图3所示，服务器30可以是图1所示实施例中的服务器。服务器30包括处理器301、存储器302(一个或多个计算机可读存储介质)和收发器303。这些部件之间可以通过一个或多个总线304通信。
处理器301可以是一个或多个CPU，在处理器301是一个CPU的情况下，该CPU可以是单核CPU，也可以是多核CPU。
存储器302可以和处理器301通过总线304连接，也可以和处理器301耦合在一起，用于存储各种程序代码和/或多组指令、以及数据(例如，地图数据、位姿数据等)。具体实现中，存储器302包括但不限于是随机存储记忆体(Random Access Memory，RAM)、只读存储器(Read-Only Memory，ROM)、可擦除可编程只读存储器(Erasable Programmable Read Only Memory，EPROM)、或便携式只读存储器(Compact Disc Read-Only Memory，CD-ROM)等。
收发器303主要集成了接收器和发射器，其中接收器用于接收电子设备发送的数据(例如，请求、图像等)，发射器用于向电子设备发送数据(例如，地图数据、位姿数据等)。
应当理解，图3所示的服务器30仅为本申请实施例提供的一个示例，服务器30还可以具有比图示更多的部件，本申请实施例对此不做具体限定。
本申请实施例中，处理器301通过运行存储在存储器302中的程序代码，从而执行服务器30的各种功能应用和数据处理。
本申请实施例中使用的术语“耦合”意指直接连接或者通过一个或多个居间组件或电路连接。
图4a为本申请实施例提供的一个应用场景的示意图，如图4a所示，该应用场景是电子设备通过传感器采集视觉信息，结合视觉信息以及来自服务器的3D地图确定电子设备的当前位姿。
3D地图是由服务器提供的，即服务器创建3D地图，然后对该3D地图进行压缩处理，将3D地图的压缩数据传输给电子设备；电子设备收到3D地图的压缩数据后进行解压缩处理，得到3D地图的重构数据，结合采集的视觉信息以及3D地图确定电子设备的当前位姿。位姿即电子设备的位置信息，可以是在世界坐标系中的绝对位姿，也可以是相对于环境中某个点的相对位姿。
本申请实施例中，服务器可以预先创建3D地图，将3D地图进行压缩处理后存储在本地，这样可以节省存储空间。另外，服务器也可以将3D地图的压缩数据传输给其他设备，例如，云存储器。
1、服务器创建3D地图，并压缩得到3D地图的压缩数据存储在本地。
服务器对3D地图进行压缩处理可以节省本地的存储空间。
2、电子设备向服务器发送地图下载请求，该地图下载请求的触发方式有两种：
(1)用户打开安装于电子设备上的3D地图类应用程序，该应用程序向其对应的服务 器上传基于GPS定位或者WiFi定位得到的位置信息，该上传操作可以触发地图下载请求。由于上传的内容包括位置信息，因此服务器可以基于位置信息进行一个初步估计，将位置信息指示的定位点所属区域的3D地图的压缩数据传输给电子设备。位置信息指示的定位点所属区域的范围可以预先设定，例如，该所属区域可以是定位点所在的各个级别的行政区(包括县、市、国家或者行政区等)，也可以是以定位点为中心、设定距离为半径的圆周区域。
(2)用户打开安装于电子设备上的3D地图类应用程序，在该应用程序中主动输入或者选择一个区域，例如，用户主动输入“xx商业中心”，或者从一个列表“A街道、B街道、C街道”中选择了“A街道”。用户的前述操作可以触发地图下载请求。无论是用户输入或是选择包含了一个地理位置，因此服务器将该地理位置的3D地图的压缩数据传输给电子设备。
应当理解的是，本申请实施例中除了上述两种方式还可以采用其他方式触发地图下载请求，例如，电子设备自动检测是否符合下载3D地图或者启动下载3D地图的条件，或者，电子设备通过检测环境光线的变化或者检测环境的变化来启动下载3D地图，以向服务器请求下载一个区域范围内的3D地图，区域范围的大小不做具体限定。
3、服务器向电子设备发送3D地图的压缩数据。
4、电子设备采集视觉信息。
需要说明的是，步骤3和步骤4是相互独立的，不限定先后顺序。
5、电子设备对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
6、电子设备根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
电子设备在收到3D地图的压缩数据后，不需要立即解压缩，只需要在基于视觉信息进行定位之前解压缩得到3D地图的重构数据即可。例如，用户可以以下载“离线地图”的方式预先下载一个区域范围的3D地图的压缩数据，而在需要定位时，才对3D地图的压缩数据进行解压缩。
图4b为本申请实施例提供的一个应用场景的示意图，如图4b所示，该应用场景是电子设备通过传感器采集视觉信息，服务器结合来自电子设备的视觉信息以及3D地图确定电子设备的当前位姿。
3D地图是由服务器提供的，即服务器创建3D地图，然后对该3D地图进行压缩处理，将3D地图的压缩数据存储在本地。当收到来自电子设备的视觉信息时，服务器进行解压缩处理，得到3D地图的重构数据，结合视觉信息以及3D地图确定电子设备的当前位姿。
1、服务器创建3D地图，并压缩得到3D地图的压缩数据存储在本地。
2、电子设备采集视觉信息。
3、电子设备向服务器发送视觉信息。
4、服务器对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
应当理解的，服务器对3D地图进行压缩处理是为了节省存储空间。
5、服务器根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
6、服务器向电子设备发送位姿。
图4c为本申请实施例提供的一个应用场景的示意图，如图4c所示，该应用场景是电子设备通过传感器采集视觉信息，结合视觉信息以及3D地图确定电子设备的当前位姿。
3D地图是由电子设备提供的，即电子设备创建3D地图，然后对该3D地图进行压缩处理，将3D地图的压缩数据存储在本地。当采集到的视觉信息时，电子设备进行解压缩处理，得到3D地图的重构数据，结合采集的视觉信息以及3D地图确定电子设备的当前位姿。
1、电子设备创建3D地图，并压缩得到3D地图的压缩数据存储在本地。
应当理解的，电子设备对3D地图进行压缩处理是为了节省存储空间。
2、电子设备通过传感器采集得到视觉信息。
3、电子设备对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
4、电子设备根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
图4d为本申请实施例提供的一个应用场景的示意图，如图4b所示，该应用场景是第二电子设备通过传感器采集视觉信息，结合视觉信息以及来自服务器的3D地图确定第二电子设备的当前位姿。
3D地图是由第一电子设备创建的，即第一电子设备创建3D地图，对该3D地图进行压缩处理，然后将3D地图的压缩数据发送给服务器，服务器再把3D地图的压缩数据发送给第二电子设备，第二电子设备进行解压缩处理，得到3D地图的重构数据，结合采集的视觉信息以及3D地图确定第二电子设备的当前位姿。
本申请实施例中，第一电子设备可以预先创建3D地图，将3D地图进行压缩处理后传输给服务器，这样可以减少传输带宽。
1、第一电子设备创建3D地图，并压缩得到3D地图的压缩数据。
2、第一电子设备向服务器发送3D地图的压缩数据。
第一电子设备对3D地图进行压缩处理后再传输，可以减少传输带宽，提高传输效率。
3、第二电子设备向服务器发送地图下载请求。
第二电子设备发送地图下载请求也可以基于图4a所示的触发方式。
4、服务器向第二电子设备发送3D地图的压缩数据。
5、第二电子设备对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
6、第二电子设备通过传感器采集得到视觉信息。
7、第二电子设备根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
图4e为本申请实施例提供的一个应用场景的示意图，如图4c所示，该应用场景是第二电子设备通过传感器采集视觉信息，服务器结合来自第二电子设备的视觉信息以及来自第一电子设备的3D地图确定第二电子设备的当前位姿。
3D地图是由第一电子设备创建的，即第一电子设备创建3D地图，对该3D地图进行压缩处理，然后将3D地图的压缩数据发送给服务器。服务器进行解压缩处理，得到3D地图的重构数据，结合来自第二电子设备的视觉信息以及3D地图确定第二电子设备的当前位姿。
1、第一电子设备创建3D地图，并压缩得到3D地图的压缩数据。
2、第一电子设备向服务器发送3D地图的压缩数据。
3、第二电子设备通过传感器采集得到视觉信息。
4、第二电子设备向服务器发送定位请求，该定位请求中携带视觉信息。
5、服务器对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
6、服务器根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
7、服务器向第二电子设备发送定位得到的位姿。
图4f为本申请实施例提供的一个应用场景的示意图，如图4d所示，该应用场景是第二电子设备通过传感器采集视觉信息，结合视觉信息以及来自第一电子设备的3D地图确定第二电子设备的当前位姿。
3D地图是由第一电子设备创建的，即第一电子设备创建3D地图，对该3D地图进行压缩处理，然后将3D地图的压缩数据发送给第二电子设备，第二电子设备进行解压缩处理，得到3D地图的重构数据，结合采集的视觉信息以及来自第一电子设备的3D地图确定第二电子设备的当前位姿。
1、第一电子设备创建3D地图，并压缩得到3D地图的压缩数据存储在本地。
2、第二电子设备向第一电子设备发送地图下载请求。
3、第一电子设备向第二电子设备发送3D地图的压缩数据。
4、第二电子设备对3D地图的压缩数据进行解压缩以得到3D地图的重构数据。
5、第二电子设备通过传感器采集得到视觉信息。
6、第二电子设备根据视觉信息在3D地图中进行定位以得到与视觉信息对应的位姿。
图4a～图4f所示实施例中，所采用的定位算法可以包括：
(1)从视觉信息中提取待检区域索描述子，提取待检区域索描述子使用的算法和从3D地图中提取区域描述子的算法一致。
(2)从视觉信息中提取待检索3D地图点，并获取待检索3D地图点空间位置和待检索3D地图点描述子，提取待检索3D地图点描述子的算法和从3D地图中提取3D地图点描述子的算法一致。
(3)根据待检索区域描述子在3D地图的数据包含的多个区域描述子中进行检索以得到多个候选区域描述子。
本申请实施例中，可以计算待检索区域描述子分别与多个区域描述子中的每一个区域描述子的距离，该距离可以包括汉明距离、曼哈顿距离或者欧氏距离等，然后选择符合条件(例如距离小于阈值)的至少一个区域描述子，即为候选区域描述子。
(4)使用待检索3D地图点描述子分别与多个候选区域描述子所对应的3D地图点描述子进行匹配，匹配亦即计算待检索3D地图点描述子分别与多个候选区域描述子对应的3D地图点描述子的相似度，找出最相似的3D地图点。
(5)根据匹配到的3D地图点，使用透视n点相机姿态估计(pespective-n-point，PnP)、高效透视n点相机姿态估计(efficient perspective-n-point camera pose estimation，EPnP)等位姿求解算法计算得到电子设备的位姿。
而图4a～图4f所示实施例中，均涉及到对3D地图进行压缩处理，本申请实施例提供了多个用于进行前述压缩处理的装置框架，以下对该多个装置框架进行说明。
在图4a～图4f任一应用场景中，基于本申请实施例的3D地图进行定位，以得到电子设备当前的位姿，该位姿可以应用于AR导航、AR人机交互、辅助驾驶、自动驾驶等领域。以基于该位姿进行AR导航为例，图4g为本申请实施例提供的电子设备显示的一种用户界面的示意图，电子设备可以基于该位姿，显示如图4g所示的用户界面，该用户界面上可以包括导航前往会议室2的导航箭头指示，该导航前往会议室2的导航箭头指示可 以是基于该位姿从服务器获取的或者从本地获取的虚拟物体。该用户界面上还可以包括传感器采集的视觉信息，例如，如图4g所示的建筑物。用户参照电子设备的如图4g所示的用户界面，前往会议室2。
需要说明的是，本申请实施例所涉及的解压缩得到的3D地图的重构数据，也可以称之为3D地图的重建数据。
图5为本申请实施例提供的3D地图的解码装置50的一个结构图，如图5所示，该解码装置50可以应用于上述实施例中的服务器或电子设备中，尤其是有接收并解压缩3D地图的需求的设备，例如，图4a所示实施例中的电子设备，或者，图4d和图4f所示实施例中的第二电子设备，或者，图4e所示实施例中的服务器。
本申请实施例中，3D地图的解码装置50包括：传输模块51和解压缩模块52；其中，传输模块51，用于接收3D地图的码流，3D地图包括多个3D地图点；解压缩模块52，用于对3D地图的码流进行解压缩处理以得到3D地图的重建数据，3D地图的重建数据包括多个3D地图点的重建数据。由此可见，传输模块51的数据是3D地图的码流，解压缩模块52用于对3D地图的码流进行解压缩以得到3D地图的重建数据。
解压缩模块52可以支持对3D地图的压缩数据进行解压缩处理，亦即，编解码系统可以支持对3D地图的数据进行压缩/解压缩处理，这样可以减少3D地图的数据量，在需要传输3D地图的场景下，相较于传输原始的3D地图的数据，传输3D地图的压缩数据，一方面可以减少传输的数据量，例如，将数据量从TB级压缩至GB级，另一方面可以减少传输中占用的带宽，提高3D地图的传输效率。
需要说明的是，本申请实施例中，解压缩处理可以包括反量化处理和预测处理中的至少一者，其中，
量化处理是指将待处理数据映射为一个或多个量化索引，每个量化索引对应一个量化中心，量化索引的比特数通常会明显少于原始数据的比特数，从而达到节省存储或传输带宽的目的。量化处理的方法包括但不限于：标量量化、矢量量化、乘积量化等。基于此，反量化处理是前述量化处理的反向处理，即根据一个或多个量化索引恢复出对应于原始数据的重建数据，重建数据的比特数多于量化索引的比特数。
预测处理是指使用已处理数据和待处理数据的残差数据进行预测以得到待处理数据。参考数据的选择可以是预先约定的，例如，固定使用前一个处理的数据做参考数据，此时可以不用在码流中标识参考数据；又例如，采用任意一个已处理数据，此时要在码流中写入参考数据的标识信息，包括参考数据的编号，或者其他能够推断出参考数据的信息。
在上述关于3D地图的数据的描述中，可知3D地图包含的多个3D地图点之间的先后顺序没有实际意义，因此解码端如果涉及对3D地图点的解压缩或解码处理，并不限定多个3D地图点的先后顺序，可以按照码流的先后顺序对多个3D地图点分别进行解压缩或解码处理。
另外，在编码端，对3D地图的数据的处理包括封装处理，目的是为了将待编码的数据封装成码流，封装处理可以采用任意一种编码算法，例如熵编码。熵编码是一种数据的无损压缩方法，熵编码算法包括但不限于：哈夫曼(huffman)编码、算术编码，由LZ77压缩算法改进的压缩/解压(lempel-ziv-markov chain-algorithm，LZMA)算法，提供数据压 缩用的函数库(zlib)算法等。相应的，在解码端，解压缩处理还可以包括解封装处理，目的是为了将码流解封装成待解压缩的数据，解封装处理可以采用与编码端采用的编码算法对应的解码算法，例如熵解码。
应当理解的是，由于量化是一种有损的压缩方式，因此解码端解压缩之后得到的3D地图的数据与编码端压缩处理之前的3D地图的原始数据不完全一致。基于此，本申请实施例中解压缩模块52输出的3D地图的数据可以称之为3D地图的重建数据，而3D地图的重建数据可以参照上述3D地图的数据的描述，此处不再赘述。
在图5所示实施例的基础上，本申请实施例提供了解压缩模块的多个可实施的结构示例，通过以下的实施例进行说明。
在一种可能的实现方式中，图6a为本申请实施例提供的3D地图的解码装置60-1的一个结构图，如图6a所示，该解码装置60-1可以应用于上述实施例中的服务器或电子设备中，尤其是有接收并解压缩3D地图的需求的设备，例如，图4a所示实施例中的电子设备，或者，图4d和图4f所示实施例中的第二电子设备，或者，图4e所示实施例中的服务器。
结合图5所示实施例，本申请实施例中，解码装置60-1包括传输模块61-1和解压缩模块62-1，解压缩模块62-1包括：解封装模块621-1和反量化模块623-1。其中，
解封装模块621-1，用于对输入的3D地图的码流进行处理以输出第一数据；反量化模块623-1，用于对输入的第三数据进行反量化处理以输出第三数据的反量化数据；
上述第一数据为第三数据；第三数据的反量化数据为3D地图的重建数据。由此可见，解封装模块621-1的输入数据是3D地图的码流，输出数据是3D地图的量化数据；反量化模块623-1的输入数据是解封装模块621-1输出的3D地图的量化数据，输出数据是3D地图的重建数据。
在一种可能的实现方式中，图6b为本申请实施例提供的3D地图的解码装置60-2的一个结构图，如图6b所示，该解码装置60-2可以应用于上述实施例中的服务器或电子设备中，有接收并解压缩3D地图的需求的设备，例如，图4a所示实施例中的电子设备，或者，图4d和图4f所示实施例中的第二电子设备，或者，图4e所示实施例中的服务器。
结合图5所示实施例，本申请实施例中，解码装置60-2包括传输模块61-2和解压缩模块62-2，解压缩模块62-2包括：解封装模块621-2和预测模块622-1。其中，
解封装模块621-2，用于对输入的3D地图的码流进行处理以输出第一数据；预测模块622-1，用于对输入的第二数据的残差数据进行预测处理以输出第二数据。
上述第一数据为第二数据的残差数据；第二数据为3D地图的重建数据。由此可见，解封装模块621-2的输入数据是3D地图的码流，输出数据是3D地图的残差数据；预测模块622-1的输入数据是解封装模块621-2输出的3D地图的残差数据，输出数据是3D地图的重建数据。
在一种可能的实现方式中，图6c为本申请实施例提供的3D地图的解码装置60-3的一个结构图，如图6c所示，该解码装置60-3可以应用于上述实施例中的服务器或电子设备中，尤其是有接收并解压缩3D地图的需求的设备，例如，图4a所示实施例中的电子设备，或者，图4d和图4f所示实施例中的第二电子设备，或者，图4e所示实施例中的服务器。
结合图5所示实施例，本申请实施例中，解码装置60-3包括传输模块61-3和解压缩模块62-3，解压缩模块61-3包括：解封装模块621-3、预测模块622-2和反量化模块623-2。其中，
解封装模块621-3，用于对输入的3D地图的码流进行处理以输出第一数据；预测模块622-2，用于对输入的第二数据的残差数据进行预测处理以输出第二数据；反量化模块623-2，用于对输入的第三数据进行反量化处理以输出第三数据的反量化数据。
上述第一数据为第二数据的残差数据或者第三数据；第二数据为3D地图的重建数据或者第三数据；第三数据的反量化数据为3D地图的重建数据。由此可见，解封装模块621-3的输入数据是3D地图的码流，输出数据是3D地图的残差数据；预测模块622-2的输入数据是解封装模块621-3输出的3D地图的残差数据，输出数据是3D地图的量化数据；反量化模块623-2的输入数据是预测模块622-2输出的3D地图的量化数据，输出数据是3D地图的重建数据。
根据上述对3D地图的数据的描述，3D地图的重建数据可以包括多个区域描述子的重建数据和多个3D地图点的重建数据，因此在图6a～图6c所示实施例中，解压缩模块中除解封装模块之外的其它模块均可以拆分为用于处理区域描述子的第一模块和用于处理3D地图点的数据的第二模块，例如，反量化模块可以包括第一反量化模块和第二反量化模块，预测模块可以包括第一预测模块和第二预测模块。而区分后的第一模块和第二模块与上述实施例的区别在于，输入数据分别对应于区域描述子和3D地图点的数据。
图6d为本申请实施例提供的3D地图的解码装置60-4的一个结构图，如图6d所示，该解码装置60-4可以应用于上述实施例中的服务器或电子设备中，尤其是有接收并解压缩3D地图的需求的设备，例如，图4a所示实施例中的电子设备，或者，图4d和图4f所示实施例中的第二电子设备，或者，图4e所示实施例中的服务器。
结合图6c所示实施例，本申请实施例中，解码装置60-4包括传输模块61-4和解压缩模块62-4，解压缩模块62-4包括：解封装模块621-4、预测模块622-3和反量化模块623-3，预测模块622-3包括第一预测模块622a和第二预测模块622b，反量化模块623包括第一反量化模块623a和第二反量化模块623b。
其中，传输模块61-4用于接收上述3D地图的码流。
解封装模块621-4的输入数据是3D地图的码流，解封装模块621-4对3D地图的码流进行解封装处理得到区域描述子的残差数据和3D地图点的残差数据。第一预测模块622a的输入数据是区域描述子的残差数据，输出数据是区域描述子的量化数据；第一反量化模块623a的输入数据是区域描述子的量化数据，输出数据是区域描述子的重建数据。第二预测模块622b的输入数据是3D地图点的残差数据，输出数据是3D地图点的量化数据；第二反量化模块623b的输入数据是3D地图点的量化数据，输出数据是3D地图点的重建数据。
根据上述对3D地图的数据的描述，3D地图的重建数据可以包括多个区域描述子的重建数据，以及多个3D地图点的3D地图点描述子的重建数据和3D地图点空间位置的重建数据，因此在图6a～图6c所示实施例中，解压缩模块中除解封装模块之外的其它模块均可以拆分为用于处理区域描述子的第一模块、用于处理3D地图点描述子的第二模块以及用于处理3D地图点空间位置的第三模块，例如，反量化模块可以包括第一反量化模块、 第二反量化模块和第三反量化模块，预测模块可以包括第一预测模块、第二预测模块和第三预测模块。而区分后的第一模块、第二模块和第三模块与上述实施例的区别在于，输入数据分别对应于区域描述子、3D地图点描述子和3D地图点空间位置。
图6e为本申请实施例提供的3D地图的解码装置60的一个结构图，如图6e所示，该解码装置60可以应用于上述实施例中的服务器或电子设备中，尤其是有接收并解压缩3D地图的需求的设备，例如，图4a所示实施例中的电子设备，或者，图4d和图4f所示实施例中的第二电子设备，或者，图4e所示实施例中的服务器。
结合图6c所示实施例，本申请实施例中，解码装置60-5包括传输模块61-5和解压缩模块62-5，解压缩模块62-5包括：解封装模块621-5、预测模块622-4和反量化模块623-4，预测模块622-4包括第一预测模块622a、第二预测模块622b和第三预测模块622c，反量化模块623-4包括第一反量化模块623a、第二反量化模块623b和第三反量化模块623c。
其中，传输模块61-5用于接收上述3D地图的码流。
解封装模块621-5的输入数据是3D地图的码流，解封装模块621-5对3D地图的码流进行解封装处理得到区域描述子的残差数据、3D地图点描述子的残差数据和3D地图点空间位置的残差数据。第一预测模块622a的输入数据是区域描述子的残差数据，输出数据是区域描述子的量化数据；第一反量化模块623a的输入数据是区域描述子的量化数据，输出数据是区域描述子的重建数据。第二预测模块622b的输入数据是3D地图点描述子的残差数据，输出数据是3D地图点描述子的量化数据；第二反量化模块623b的输入数据是3D地图点描述子的量化数据，输出数据是3D地图点描述子的重建数据。第三预测模块622b的输入数据是3D地图点空间位置的残差数据，输出数据是3D地图点空间位置的量化数据；第三反量化模块623b的输入数据是3D地图点空间位置的量化数据，输出数据是3D地图点空间位置的重建数据。
需要说明的是，图6d和图6e分别示出了3D地图的解码装置的一种示例性的结构，是基于3D地图的码流包含的内容得到的一个结构，但该结构并不构成对解码装置的限定，解码装置可以包括比图6d或图6e所示实施例更多样的模块。例如，结合图6a或图6b所示实施例，基于3D地图的码流包含的内容得到其它的结构。如图6a～图6c所示实施例，预测模块和反量化模块是和/或的关系，即解压缩模块中可以包括预测模块和反量化模块的其中一者，也可以预测模块和反量化模块均包括。因此对于区分了第一模块和第二模块的情况，针对区域描述子的处理模块和针对3D地图点的数据的处理模块可以独立的设置处理方法，二者不必完全一致，例如，针对区域描述子，可以分别通过第一预测模块和第一反量化模块进行处理，针对3D地图点的数据可以通过第二预测模块进行处理。而对于区分了第一模块、第二模块和第三模块的情况，针对区域描述子的处理模块、针对3D地图点描述子的处理模块以及针对3D地图点空间位置的处理模块可以独立的设置处理方法，三者不必完全一致，例如，针对区域描述子，可以分别通过第一预测模块和第一反量化模块进行处理，针对3D地图点描述子，可以分别第二预测模块进行处理，针对3D地图点空间位置，可以通过第三反量化模块进行处理。本申请对解压缩模块的具体实施方式不做具体限定。
根据上述对3D地图的数据的描述，3D地图的重建数据可以包括多个区域描述子的重建数据和多个3D地图点的重建数据。
在一种可能的实现方式中，图7a为本申请实施例提供的3D地图的解码装置70-1的一个结构图，如图7a所示，该解码装置70-1可以应用于上述实施例中的服务器或电子设备中，尤其是有接收并解压缩3D地图的需求的设备，例如，图4a所示实施例中的电子设备，或者，图4d和图4f所示实施例中的第二电子设备，或者，图4e所示实施例中的服务器。
结合图5所示实施例，本申请实施例中，解码装置70-1包括传输模块71-1和解压缩模块72-1，解压缩模块72-1包括：第一解压缩子模块721-1和第二解压缩子模块722-1。其中，
第一解压缩子模块721-1，用于对输入的第十四数据的码流进行解压缩处理以输出第十四数据；第二解压缩子模块722-1，用于对输入的第十五数据的码流进行压缩处理以输出第十五数据；
其中，第十四数据为多个区域描述子的其中之一的重建数据；第十五数据为多个3D地图点的其中之一的重建数据。由此可见，第一解压缩子模块721-1的输入数据是区域描述子的码流，输出数据是区域描述子的重建数据，第二解压缩子模块722-1的输入数据是3D地图点的码流，输出数据是3D地图点的重建数据。3D地图的码流包括多个区域描述子的码流和多个3D地图点的码流。
上述第一解压缩子模块721-1和第二解压缩子模块722-1可以参照图6a～图6e所示实施例中的解压缩模块的结构。需要说明的是，第一解压缩子模块721-1和第二解压缩子模块722-1是相互独立的，二者采用的结构可以相同，也可以不同，即用于处理区域描述子的第一解压缩子模块721-1的结构与处理3D地图点的数据的第二解压缩子模块722-1的结构相同或者不同，相应的，对区域描述子所采用的解压缩处理的步骤和对3D地图点的数据所采用的解压缩处理的步骤也可以相同或者不同。
例如，第一解压缩子模块包括第一解封装模块和第一预测模块，这样区域描述子的码流输入第一解压缩子模块后，先经第一解封装模块解封装以得到区域描述子的残差数据，残差数据再经第一预测模块处理以得到区域描述子的重建数据；第二解压缩子模块包括第二解封装模块、第二预测模块和第二反量化模块，这样3D地图点的码流输入第二解压缩子模块后，先经第二解封装模块处理以得到3D地图点的残差数据，残差数据再经第二预测模块处理以得到3D地图点的量化数据，然后量化数据经第二反量化模块处理以得到3D地图点的重建数据。
又例如，第一解压缩子模块包括第一解封装模块、第一预测模块和第一反量化模块，这样区域描述子的码流输入第一解压缩子模块后，分别经第一解封装模块处理以得到区域描述子的残差数据，残差数据经第一预测模块处理以得到区域描述子的量化数据，量化数据经第一反量化模块处理以得到区域描述子的重建数据；第二解压缩子模块包括第二解封装模块和第二预测模块，这样3D地图点的描述输入第二解压缩子模块后，经第二解封装模块处理以得到3D地图点的残差数据，残差数据经第二预测模块处理以得到3D地图点的重建数据。
应当理解的是，上述仅对第一解压缩子模块和第二解压缩子模块的结构进行示例性说明，但并不对二者的结构构成限定，这两个子模块可以包括比示例中更多或更少的模块，具体可以参照图6a～图6e所示实施例中的解压缩模块的结构，本申请实施例对此不做具 体限定。
在一种可能的实现方式中，图7b为本申请实施例提供的3D地图的解码装置70-2的一个结构图，如图7b所示，该解码装置70-2可以应用于上述实施例中的服务器或电子设备中，尤其是有接收并解压缩3D地图的需求的设备，例如，图4a所示实施例中的电子设备，或者，图4d和图4f所示实施例中的第二电子设备，或者，图4e所示实施例中的服务器。
结合图7a所示实施例，本申请实施例中，解码装置70-2包括传输模块71-2和解压缩模块72-2，解压缩模块72-2包括第一解压缩子模块721-2和第二解压缩子模块722-2，第一解压缩子模块721-2包括第一解封装模块7211、第一预测模块7212和第一反量化模块7213，第二解压缩子模块722-2包括第二解封装模块7221、第二预测模块7222和第二反量化模块7223。其中，
第一解封装模块7211，用于对输入的区域描述子的码流进行处理以得到区域描述子的残差数据，第一预测模块7212，用于对输入的区域描述子的残差数据进行预测处理以得到区域描述子的量化数据，第一反量化模块7213，用于对输入的区域描述子的量化数据进行反量化处理以得到区域描述子的重建数据。第二解封装模块7221，用于对输入的3D地图点的码流进行处理以得到3D地图点的残差数据，第二预测模块7222，用于对输入的3D地图点的残差数据进行预测处理以得到3D地图点的量化数据，第二反量化模块7223，用于对输入的3D地图点的量化数据进行反量化处理以得到3D地图点的重建数据。
应当理解的是，图7b所示实施例仅对第一解压缩子模块和第二解压缩子模块的结构进行示例性说明，但并不对二者的结构构成限定，这两个子模块可以包括比示例中更多或更少的模块，具体可以参照图6a～图6e所示实施例中的解压缩模块62的结构，本申请实施例对此不做具体限定。
根据上述对3D地图的数据的描述，3D地图的重建数据可以包括多个区域描述子的重建数据、多个3D地图点的3D地图点描述子的重建数据和3D地图点空间位置的重建数据。
在一种可能的实现方式中，图8a为本申请实施例提供的3D地图的解码装置80-1的一个结构图，如图8a所示，该解码装置80-1可以应用于上述实施例中的服务器或电子设备中，尤其是有接收并解压缩3D地图的需求的设备，例如，图4a所示实施例中的电子设备，或者，图4d和图4f所示实施例中的第二电子设备，或者，图4e所示实施例中的服务器。
结合图5所示实施例，本申请实施例中，解码装置80-1包括传输模块81-1和解压缩模块82-1，解压缩模块82-1包括：第一解压缩子模块821-1、第二解压缩子模块822-1和第三解压缩子模块823-1。其中，
第一解压缩子模块821-1，用于对输入的第二十二数据的码流进行解压缩处理以输出第二十二数据；第二解压缩子模块822-1，用于对输入的第二十三数据的码流进行压缩处理以输出第二十三数据；第三解压缩子模块823-1，用于对输入的第二十四数据的码流进行压缩处理以输出第二十四数据。
其中，第二十二数据为多个区域描述子的其中之一的重建数据；第二十三数据为多个3D地图点的其中之一的3D地图点描述子的重建数据；第二十四数据为多个3D地图点的 其中之一的3D地图点空间位置的重建数据。由此可见，第一解压缩子模块821-1的输入数据是区域描述子的码流，输出数据是区域描述子的重建数据，第二解压缩子模块812-1的输入数据是3D地图点描述子的码流，输出数据是3D地图点描述子的重建数据，第三解压缩子模块813-1的输入数据是3D地图点空间位置的码流，输出数据是3D地图点空间位置的重建数据。3D地图的码流包括多个区域描述子的码流、多个3D地图点描述子的码流和多个3D地图点空间位置的码流。
上述第一解压缩子模块821-1、第二解压缩子模块822-1和第三解压缩子模块823-1可以参照图6a～图6e所示实施例中的解压缩模块的结构。需要说明的是，第一解压缩子模块821-1、第二解压缩子模块822-1和第三解压缩子模块823-1是相互独立的，三者采用的结构可以相同，也可以不同，即用于处理区域描述子的第一解压缩子模块821-1的结构、用于处理3D地图点描述子的第二解压缩子模块822-1以及用于处理3D地图点空间位置的第三解压缩子模块823-1的结构相同或者不同，相应的，对区域描述子所采用的解压缩处理的步骤、对3D地图点描述子所采用的解压缩处理的步骤以及对3D地图点空间位置所采用的解压缩处理的步骤也可以相同或者不同。
在一种可能的实现方式中，图8b为本申请实施例提供的3D地图的解码装置80-2的一个结构图，如图8b所示，该解码装置80-2可以应用于上述实施例中的服务器或电子设备中，尤其是有接收并解压缩3D地图的需求的设备，例如，图4a所示实施例中的电子设备，或者，图4d和图4f所示实施例中的第二电子设备，或者，图4e所示实施例中的服务器。
结合图8a所示实施例，本申请实施例中，解码装置80-2包括传输模块81-2和解压缩模块82-2，解压缩模块82-2包括第一解压缩子模块821-2、第二解压缩子模块822-2和第三解压缩子模块823-2，第一解压缩子模块821-2包括第一解封装模块8211、第一预测模块8212和第一反量化模块8213，第二解压缩子模块822-2包括第二解封装模块8221、第二预测模块8222和第二反量化模块8223，第三解压缩子模块823-2包括第三解封装模块8231、第三预测模块8232和第三反量化模块8233。其中，
第一解封装模块8211，用于对输入的区域描述子的码流进行处理以得到区域描述子的残差数据，第一预测模块8212，用于对输入的区域描述子的残差数据进行预测处理以得到区域描述子的量化数据，第一反量化模块8213，用于对输入的区域描述子的量化数据进行反量化处理以得到区域描述子的重建数据。第二解封装模块8221，用于对输入的3D地图点描述子的码流进行处理以得到3D地图点描述子的残差数据，第二预测模块8222，用于对输入的3D地图点描述子的残差数据进行预测处理以得到3D地图点描述子的量化数据，第二反量化模块8223，用于对输入的3D地图点描述子的量化数据进行反量化处理以得到3D地图点描述子的重建数据。第三解封装模块8231，用于对输入的3D地图点空间位置的码流进行处理以得到3D地图点空间位置的残差数据，第三预测模块8232，用于对输入的3D地图点空间位置的残差数据进行预测处理以得到3D地图点空间位置的量化数据，第三反量化模块8233，用于对输入的3D地图点空间位置的量化数据进行反量化处理以得到3D地图点空间位置的重建数据。
应当理解的是，图8b所示实施例仅对第一解压缩子模块、第二解压缩子模块和第三解压缩子模块的结构进行示例性说明，但并不对三者的结构构成限定，这三个子模块可以 包括比示例中更多或更少的模块，具体可以参照图6a～图6e所示实施例中的解压缩模块62的结构，本申请实施例对此不做具体限定。
图9为本申请实施例提供的3D地图的解码方法的过程900的一个流程图，如图9所示，过程900可由上述实施例中的解码装置执行。过程900描述为一系列的步骤或操作，应当理解的是，过程900可以以各种顺序执行和/或同时发生，不限于图9所示的执行顺序。假设3D地图的码流在解码装置中进行解压缩处理以得到3D地图的数据，执行包括如下步骤的过程900来处理当前正在处理的3D地图的码流。
步骤901、接收3D地图的码流。
解码装置可以采用通信链路接收3D地图的码流。
步骤902、对3D地图的码流进行解压缩处理以得到3D地图的重建数据。
3D地图和3D地图的重建数据可以参照上述描述，此处不再赘述。
本申请实施例中，对3D地图的码流进行的解压缩处理可以包括解封装处理，以及预测处理和/或反量化处理。以上各种处理可以参照上述实施例的描述，此处不再赘述。
图10为本申请实施例提供的3D地图的解码装置100的一个结构图，如图10所示，该解码装置100可以应用于上述实施例中的服务器或电子设备中，尤其是有存储并解压缩3D地图的需求的设备，例如，图4b所示实施例中的服务器，或者，图4c所示实施例中的电子设备。
本申请实施例中，3D地图的解码装置100包括：存储器101，用于存储比特流形式的经压缩得到的3D地图，所述3D地图包括多个3D地图点，所述3D地图的数据包括所述多个3D地图点的数据；解码器102，用于对所述比特流形式的经压缩得到的3D地图进行解压缩处理，得到重建3D地图，所述重建3D地图包括所述多个3D地图点，所述重建3D地图的数据包括所述多个3D地图点的重建数据。由此可见，存储器101的存储的是比特流形式的经压缩得到的3D地图，解码器102的输入数据是比特流形式的经压缩得到的3D地图，输出数据是重建3D地图。
解码器102可以支持对比特流形式的经压缩得到的3D地图进行解压缩处理，亦即，3D地图的解码装置可以支持对3D地图的数据的压缩/解压缩处理，这样可以减少3D地图的数据量，在需要存储3D地图的场景下，相较于存储原始的3D地图的数据，存储比特流形式的经压缩得到的3D地图，可以节省存储3D地图的数据所占用的空间。
本申请实施例中，解码器102可以参照图5～图8b所示实施例中的解压缩模块，区别在于该解码器不包含解封装模块。
图11a为本申请实施例提供的3D地图的编码码流的一个示例性的结构图，如图11a所示，3D地图的编码码流可以包括：文件头、区域描述子1、3D地图点1、区域描述子2、3D地图点2、……、区域描述子m、3D地图点n以及其它信息。
本申请实施例中，假设3D地图包含m个区域描述子和n个3D地图点，编码码流中区域描述子和3D地图点可以是交叉排列的。
图11b为本申请实施例提供的3D地图的编码码流的一个示例性的结构图，如图11b所示，3D地图的编码码流可以包括：文件头、区域描述子1、区域描述子2、……、区域描述子m、3D地图点1、3D地图点2、……、3D地图点n以及其它信息。
本申请实施例中，假设3D地图包含m个区域描述子和n个3D地图点，编码码流中区域描述子和3D地图点是分开排列的，先是m个区域描述子，再是n个3D地图点，或者先是n个3D地图点，再是m个区域描述子。
应当理解的是，本申请实施例中的3D地图的编码码流还可以采用其他的结构，具体结构与3D地图包含的数据内容相关联，例如，3D地图的编码码流中不包含上述文件头、区域描述子和其它信息中的一个或多个信息。
图11c为本申请实施例提供的3D地图的编码码流的一个示例性的结构图，如图11c所示，3D地图的编码码流可以包括：区域描述子1、区域描述子2和区域描述子3。该编码码流的结构对应于上述解压缩模块不包含预测模块的情况，因此不需要在码流中填入预测用的参考信息。
图11d为本申请实施例提供的3D地图的编码码流的一个示例性的结构图，如图11d所示，3D地图的编码码流可以包括：区域描述子1、区域描述子2的参考区域描述子的索引和区域描述子2的残差数据。该编码码流的结构对应于上述解压缩模块包含预测模块的情况，因此需要在码流中填入预测用的参考信息，例如，区域描述子2的参考区域描述子的索引，并且区域描述子2是填入了其残差数据，而不是原始数据。
图11e为本申请实施例提供的3D地图的编码码流的一个示例性的结构图，如图11e所示，3D地图的编码码流可以包括：3D地图点1、3D地图点2和3D地图点3。以3D地图点2为例，该3D地图点2的码流包括3D地图点2的3D地图点描述子和3D地图点2的3D地图点空间位置，如果该编码码流的结构对应于上述解压缩模块包含预测模块的情况，那么3D地图点描述子的码流包括3D地图点描述子的参考索引和3D地图点描述子的残差数据，3D地图点空间位置的码流包括3D地图点空间位置的参考索引和3D地图点空间位置的残差数据。
应当理解的是，本申请实施例中的3D地图的编码码流根据对3D地图的数据进行的压缩/解压缩处理过程的不同，3D地图的码流包含的内容也有区别。例如，若压缩/解压缩处理过程包含量化/反量化，则3D地图的码流包括量化数据；若压缩/解压缩处理过程包含量化/反量化和预测，则3D地图的码流包括参考索引和残差数据；若压缩/解压缩处理过程包含预测，则3D地图的码流包括参考索引和残差数据。因此本申请实施例对3D地图的编码码流的结构不做具体限定。
在实现过程中，上述方法实施例的各步骤可以通过处理器中的硬件的集成逻辑电路或者软件形式的指令完成。处理器可以是通用处理器、数字信号处理器(digital signal processor,DSP)、特定应用集成电路(application-specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array,FPGA)或其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。本申请实施例公开的方法的步骤可以直接体现为硬件编码处理器执行完成，或者用编码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器，处理器读取存储器中的信息，结合其硬件完成上述方法的步骤。
上述各实施例中提及的存储器可以是易失性存储器或非易失性存储器，或可包括易失 性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(dynamic RAM，DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data rate SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。应注意，本文描述的系统和方法的存储器旨在包括但不限于这些和任意其它适合类型的存储器。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (43)

  1. 一种3D地图的编解码系统，其特征在于，包括：编码装置和解码装置，所述编码装置和所述解码装置通信连接；其中，
    所述编码装置，用于对3D地图的数据进行压缩处理以得到所述3D地图的码流，并将所述3D地图的码流发送给所述解码装置，所述3D地图包括多个3D地图点，所述3D地图的数据包括所述多个3D地图点的数据；
    所述解码装置，用于接收所述3D地图的码流，并对所述3D地图的码流进行解压缩处理以得到所述3D地图的重构数据。
  2. 根据权利要求1所述的系统，其特征在于，所述编码装置为云端服务器，所述解码装置为电子设备；或者，所述编码装置为第一电子设备，所述解码装置为第二电子设备；
    所述解码装置，还用于向所述解码装置发送3D地图下载请求，所述3D地图下载请求包括位置指示信息；
    所述编码装置，还用于接收3D地图下载请求，并根据所述3D地图下载请求向所述解码装置发送与所述位置指示信息对应的3D地图的码流。
  3. 根据权利要求1或2所述的系统，其特征在于，所述编码装置为电子设备，所述解码装置为云端服务器；
    所述编码设备，具体用于在创建3D地图后，向所述解码装置发送所述3D地图的码流。
  4. 根据权利要求1-3中任一项所述的系统，其特征在于，所述3D地图的数据还包括多个区域描述子，所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
  5. 根据权利要求1或4所述的系统，其特征在于，所述多个3D地图点中的任意一个3D地图点的数据包括3D地图点描述子和3D地图点空间位置。
  6. 根据权利要求1-5中任一项所述的系统，其特征在于，所述编码装置，还用于创建所述3D地图。
  7. 根据权利要求1-6中任一项所述的系统，其特征在于，所述解码装置，还用于根据所述3D地图进行定位。
  8. 一种3D地图的解码装置，其特征在于，包括：传输模块和解压缩模块；其中，
    所述传输模块，用于接收3D地图的码流，所述3D地图包括多个3D地图点；
    所述解压缩模块，用于对所述3D地图的码流进行解压缩处理以得到所述3D地图的重建数据，所述3D地图的重建数据包括所述多个3D地图点的重建数据。
  9. 根据权利要求8所述的装置，其特征在于，所述3D地图的重建数据还包括多个区域描述子的重建数据，所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
  10. 根据权利要求9所述的装置，其特征在于，所述多个3D地图点中的任意一个3D地图点的重建数据包括3D地图点描述子的重建数据和3D地图点空间位置的重建数据。
  11. 根据权利要求8-10中任一项所述的装置，其特征在于，所述3D地图的解码装置为云端服务器或者电子设备；
    所述传输模块，还用于发送3D地图下载请求，所述3D地图下载请求包括位置指示信息；接收与所述位置指示信息对应的所述3D地图的码流。
  12. 根据权利要求8-10中任一项所述的装置，其特征在于，所述3D地图的解码装置为云端服务器；
    所述传输模块，具体用于接收由电子设备创建的所述3D地图的码流。
  13. 根据权利要求8-12中任一项所述的装置，其特征在于，所述解压缩模块包括解封装模块，以及预测模块和/或反量化模块；
    所述解封装模块，用于对输入的所述3D地图的码流进行处理以输出第一数据；
    所述预测模块，用于对输入的第二数据的残差数据进行预测处理以输出所述第二数据；
    所述反量化模块，用于对输入的第三数据进行反量化处理以输出所述第三数据的反量化数据；
    其中，所述第一数据为所述第二数据的残差数据或者所述第三数据；所述第二数据为所述3D地图的重建数据或者所述第三数据；所述第三数据的反量化数据为所述3D地图的重建数据。
  14. 根据权利要求13所述的装置，其特征在于，所述预测模块包括第一预测模块和第二预测模块；和/或，所述反量化模块包括第一反量化模块和第二反量化模块；
    所述第一预测模块，用于对输入的第四数据的残差数据进行预测处理以输出所述第四数据；
    所述第一反量化模块，用于对输入的第五数据进行反量化处理以输出所述第五数据的反量化数据；
    其中，所述第四数据为多个区域描述子的其中之一的重建数据或者所述第五数据；所述第五数据的反量化数据为所述多个区域描述子的其中之一的重建数据；
    所述第二预测模块，用于对输入的第六数据的残差数据进行预测处理以输出所述第六数据；
    所述第二反量化模块，用于对输入的第七数据进行反量化处理以输出所述第七数据的反量化数据；
    其中，所述第六数据为所述多个3D地图点的其中之一的重建数据或者所述第七数据；所述第七数据的反量化数据为所述多个3D地图点的其中之一的重建数据。
  15. 根据权利要求13所述的装置，其特征在于，所述预测模块包括第一预测模块、第二预测模块和第三预测模块；和/或，所述反量化模块包括第一反量化模块、第二反量化模块和第三反量化模块；
    所述第一预测模块，用于对输入的第八数据的残差数据进行预测处理以输出所述第八数据；
    所述第一反量化模块，用于对输入的第九数据进行反量化处理以输出所述第九数据的反量化数据；
    其中，所述第八数据为多个区域描述子的其中之一的重建数据或者所述第九数据；所述第九数据的反量化数据为所述多个区域描述子的其中之一的重建数据；
    所述第二预测模块，用于对输入的第十数据的残差数据进行预测处理以输出所述第十数据；
    所述第二反量化模块，用于对输入的第十一数据进行反量化处理以输出所述第十一数据的反量化数据；
    其中，所述第十数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据或者所述第十一数据；所述第十一数据的反量化数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据。
    所述第三预测模块，用于对输入的第十二数据的残差数据进行预测处理以输出所述第十二数据；
    所述第三反量化模块，用于对输入的第十三数据进行反量化处理以输出所述第十三数据的反量化数据；
    其中，所述第十二数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据或者所述第十三数据；所述第十三数据的反量化数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据。
  16. 根据权利要求8-12中任一项所述的装置，其特征在于，所述解压缩模块包括第一解压缩子模块和第二解压缩子模块；
    所述第一解压缩子模块，用于对输入的第十四数据的码流进行解压缩处理以输出所述第十四数据；
    所述第二解压缩子模块，用于对输入的第十五数据的码流进行压缩处理以输出所述第十五数据；
    其中，所述第十四数据为多个区域描述子的其中之一的重建数据；所述第十五数据为所述多个3D地图点的其中之一的重建数据。
  17. 根据权利要求16所述的装置，其特征在于，所述第一解压缩子模块包括第一解封装模块，以及第一预测模块和/或第一反量化模块；所述第二解压缩子模块包括第二解封装模块，以及第二预测模块和/或第二反量化模块；
    所述第一解封装模块，用于对输入的所述3D地图的码流进行处理以输出第十六数据；
    所述第一预测模块，用于对输入的第十七数据的残差数据进行预测处理以输出所述第十七数据；
    所述第一反量化模块，用于对输入的第十八数据进行反量化处理以输出所述第十八数据的反量化数据；
    其中，所述第十六数据为所述第十七数据的残差数据或者所述第十八数据；所述第十七数据为所述多个区域描述子的其中之一的重建数据或者所述第十八数据；所述第十八数据的反量化数据为所述多个区域描述子的其中之一的重建数据；
    所述第二解封装模块，用于对输入的所述3D地图的码流进行处理以输出第十九数据；
    所述第二预测模块，用于对输入的第二十数据的残差数据进行预测处理以输出所述第二十数据；
    所述第二反量化模块，用于对输入的第二十一数据进行反量化处理以输出所述第二十一数据的反量化数据；
    其中，所述第十九数据为所述第二十数据的残差数据或者所述第二十一数据；所述第二十数据为所述多个3D地图点的其中之一的重建数据或者所述第二十一数据；所述第二十一数据的反量化数据为所述多个3D地图点的其中之一的重建数据。
  18. 根据权利要求8-12中任一项所述的装置，其特征在于，所述解压缩模块包括第一解压缩子模块、第二解压缩子模块和第三解压缩子模块；
    所述第一解压缩子模块，用于对输入的第二十二数据的码流进行解压缩处理以输出所述第二十二数据；
    所述第二解压缩子模块，用于对输入的第二十三数据的码流进行压缩处理以输出所述第二十三数据；
    所述第三解压缩子模块，用于对输入的第二十四数据的码流进行压缩处理以输出所述第二十四数据；
    其中，所述第二十二数据为多个区域描述子的其中之一的重建数据；所述第二十三数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据；所述第二十四数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据。
  19. 根据权利要求18所述的装置，其特征在于，所述第一解压缩子模块包括第一解封装模块，以及第一预测模块和/或第一反量化模块；所述第二解压缩子模块包括第二解封装模块，以及第二预测模块和/或第二反量化模块；所述第三解压缩子模块包括第三解封装模块，以及第三预测模块和/或第三反量化模块；
    所述第一解封装模块，用于对输入的所述3D地图的码流进行处理以输出第二十五数据；
    所述第一预测模块，用于对输入的第二十六数据的残差数据进行预测处理以输出所述第二十六数据；
    所述第一反量化模块，用于对输入的第二十七数据进行反量化处理以输出所述第二十七数据的反量化数据；
    其中，所述第二十五数据为所述第二十六数据的残差数据或者所述第二十七数据；所述第二十六数据为所述多个区域描述子的其中之一的重建数据或者所述第二十七数据；所述第二十七数据的反量化数据为所述多个区域描述子的其中之一的重建数据；
    所述第二解封装模块，用于对输入的所述3D地图的码流进行处理以输出第二十八数据；
    所述第二预测模块，用于对输入的第二十九数据的残差数据进行预测处理以输出所述第二十九数据；
    所述第二反量化模块，用于对输入的第三十数据进行反量化处理以输出所述第三十数据的反量化数据；
    其中，所述第二十八数据为所述第二十九数据的残差数据或者所述第三十数据；所述第二十九数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据或者所述第三十数据；所述第三十数据的反量化数据为所述多个3D地图点的其中之一的3D地图点描述子的重建数据；
    所述第三解封装模块，用于对输入的所述3D地图的码流进行处理以输出第三十一数据；
    所述第三预测模块，用于对输入的第三十二数据的残差数据进行预测处理以输出所述第三十二数据；
    所述第三反量化模块，用于对输入的第三十三数据进行反量化处理以输出所述第三十 三数据的反量化数据；
    其中，所述第三十一数据为所述第三十二数据的残差数据或者所述第三十三数据；所述第三十二数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据或者所述第三十三数据；所述第三十三数据的反量化数据为所述多个3D地图点的其中之一的3D地图点空间位置的重建数据。
  20. 一种3D地图的解码方法，其特征在于，包括：
    接收3D地图的码流，所述3D地图包括多个3D地图点；
    对所述3D地图的码流进行解压缩处理以得到所述3D地图的重建数据，所述3D地图的重建数据包括所述多个3D地图点的重建数据。
  21. 根据权利要求20所述的方法，其特征在于，所述3D地图的重建数据还包括多个区域描述子的重建数据，所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
  22. 根据权利要求21所述的方法，其特征在于，所述多个3D地图点中的任意一个3D地图点的重建数据包括3D地图点描述子的重建数据和3D地图点空间位置的重建数据。
  23. 根据权利要求20-22中任一项所述的方法，其特征在于，还包括：
    发送3D地图下载请求，所述3D地图下载请求包括位置指示信息；
    所述接收3D地图的码流，包括：
    接收与所述位置指示信息对应的所述3D地图的码流。
  24. 根据权利要求20-22中任一项所述的方法，其特征在于，所述接收3D地图的码流，包括：
    接收由电子设备创建的所述3D地图的码流。
  25. 根据权利要求20-24中任一项所述的方法，其特征在于，所述对所述3D地图的码流进行解压缩处理以得到所述3D地图的重建数据，包括：
    对所述3D地图的码流进行处理以得到第一数据；
    对第二数据的残差数据进行预测处理以得到所述第二数据，和/或，对第三数据进行反量化处理以得到所述第三数据的反量化数据；
    其中，所述第一数据为所述第二数据的残差数据或者所述第三数据；所述第二数据为所述3D地图的重建数据或者所述第三数据；所述第三数据的反量化数据为所述3D地图的重建数据。
  26. 一种3D地图的编码装置，其特征在于，包括：
    一个或多个处理器；
    存储器，用于存储一个或多个程序；
    当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求20-25中任一项所述的方法。
  27. 一种3D地图的解码装置，其特征在于，包括：
    存储器，用于存储比特流形式的经压缩得到的3D地图，所述3D地图包括多个3D地图点，所述3D地图的数据包括所述多个3D地图点的数据；
    解码器，用于对所述比特流形式的经压缩得到的3D地图进行解压缩处理，得到重建3D地图，所述重建3D地图包括所述多个3D地图点，所述重建3D地图的数据包括所述 多个3D地图点的重建数据。
  28. 根据权利要求27所述的装置，其特征在于，所述重建3D地图的数据还包括多个区域描述子的重建数据，所述多个区域描述子中的任意一个区域描述子用于描述所述多个3D地图点中的部分或全部3D地图点的特征。
  29. 根据权利要求27或28所述的装置，其特征在于，所述多个3D地图点中的任意一个3D地图点的重建数据包括3D地图点描述子的重建数据和3D地图点空间位置的重建数据。
  30. 根据权利要求27-29中任一项所述的装置，其特征在于，所述3D地图的编码装置为云端服务器或者电子设备。
  31. 一种计算机可读存储介质，其特征在于，包括计算机程序，所述计算机程序在计算机上被执行时，使得所述计算机执行权利要求20-25中任一项所述的方法。
  32. 一种计算机程序产品，其特征在于，所述计算机程序产品包括计算机程序代码，当所述计算机程序代码在计算机上运行时，使得计算机执行权利要求20-25中任一项所述的方法。
  33. 一种3D地图的编码码流，其特征在于，所述3D地图的编码码流包括多个3D地图点的码流，所述3D地图包括所述多个3D地图点。
  34. 根据权利要求33所述的码流，其特征在于，其特征在于，所述3D地图的编码码流还包括多个区域描述子的码流，所述多个区域描述子中的任意一个区域描述子和所述多个3D地图点中的至少一个3D地图点对应。
  35. 根据权利要求34所述的码流，其特征在于，所述多个3D地图点中的任意一个3D地图点的码流包括3D地图点描述子的码流和3D地图点空间位置的码流。
  36. 根据权利要求33所述的方法，其特征在于，所述多个3D地图点中的任意一个3D地图点的码流包括3D地图点的残差数据。
  37. 根据权利要求35所述的码流，其特征在于，所述3D地图点描述子的码流包括所述3D地图点描述子的残差数据；和/或，所述3D地图点空间位置的码流包括所述3D地图点空间位置的残差数据。
  38. 根据权利要求36或37所述的码流，其特征在于，所述多个3D地图点中的任意一个3D地图点的码流还包括参考3D地图点的指示信息，所述多个3D地图点包括所述参考3D地图点，且所述参考3D地图点是在编码所述任意一个3D地图点之前已编码的3D地图点。
  39. 根据权利要求33所述的方法，其特征在于，所述多个3D地图点中的任意一个3D地图点的码流包括3D地图点的量化数据。
  40. 根据权利要求35所述的码流，其特征在于，所述3D地图点描述子的码流包括所述3D地图点描述子的量化数据；和/或，所述3D地图点空间位置的码流包括所述3D地图点空间位置的量化数据。
  41. 根据权利要求34所述的码流，其特征在于，所述多个区域描述子中的任意一个区域描述子的码流包括区域描述子的残差数据。
  42. 根据权利要求41所述的码流，其特征在于，所述多个区域描述子中的任意一个区域描述子的码流还包括参考区域描述子的指示信息，所述多个区域描述子包括所述参考区 域描述子。
  43. 根据权利要求34所述的码流，其特征在于，所述多个区域描述子中的任意一个区域描述子的码流包括区域描述子的量化数据。
PCT/CN2021/098483 2021-06-04 2021-06-04 3d地图的解码装置和方法、及3d地图的编码码流 WO2022252235A1 (zh)

Priority Applications (22)

Application Number Priority Date Filing Date Title
CN202180099016.0A CN117461057A (zh) 2021-06-04 2021-06-04 3d地图的解码装置和方法、及3d地图的编码码流
KR1020247000378A KR20240017921A (ko) 2021-06-04 2021-06-04 3d 맵을 디코딩하기 위한 장치 및 방법, 및 3d 맵의 인코딩된 비트스트림
PCT/CN2021/098483 WO2022252235A1 (zh) 2021-06-04 2021-06-04 3d地图的解码装置和方法、及3d地图的编码码流
EP21943596.3A EP4336456A1 (en) 2021-06-04 2021-06-04 Decoding apparatus and method for 3d map, and encoded code stream of 3d map
CN202180098947.9A CN117413523A (zh) 2021-06-04 2021-07-05 3d地图的编解码方法及装置
PCT/CN2021/104601 WO2022252337A1 (zh) 2021-06-04 2021-07-05 3d地图的编解码方法及装置
PCT/CN2021/105624 WO2022252346A1 (zh) 2021-06-04 2021-07-09 3d地图的检索方法和装置
PCT/CN2021/105623 WO2022252345A1 (zh) 2021-06-04 2021-07-09 3d地图的压缩、解压缩方法和装置
KR1020247000354A KR20240017918A (ko) 2021-06-04 2021-07-09 3d 지도 검색 방법 및 장치
CN202180099015.6A CN117461056A (zh) 2021-06-04 2021-07-09 3d地图的检索方法和装置
PCT/CN2021/105625 WO2022252347A1 (zh) 2021-06-04 2021-07-09 3d地图的检索方法和装置
KR1020237045074A KR20240013238A (ko) 2021-06-04 2021-07-09 3d 지도를 위한 압축 및 압축해제 방법들 및 장치들
CN202180099018.XA CN117460972A (zh) 2021-06-04 2021-07-09 3d地图的检索方法和装置
CN202180098601.9A CN117397242A (zh) 2021-06-04 2021-07-09 3d地图的压缩、解压缩方法和装置
EP21943704.3A EP4336454A1 (en) 2021-06-04 2021-07-09 3d map retrieval method and apparatus
KR1020247000191A KR20240017906A (ko) 2021-06-04 2021-07-09 3d 지도 검색 방법 및 장치
EP21943702.7A EP4339881A1 (en) 2021-06-04 2021-07-09 Compression and decompression methods and apparatuses for 3d map
EP21943703.5A EP4336223A1 (en) 2021-06-04 2021-07-09 3d map retrieval method and apparatus
US18/526,846 US20240111793A1 (en) 2021-06-04 2023-12-01 Method and apparatus for retrieving 3d map
US18/526,801 US20240095265A1 (en) 2021-06-04 2023-12-01 Method and apparatus for retrieving 3d map
US18/526,734 US20240112372A1 (en) 2021-06-04 2023-12-01 Method and apparatus for compressing and decompressing 3d map
US18/526,677 US20240119639A1 (en) 2021-06-04 2023-12-01 Apparatus and method for decoding 3d map, and encoded bitstream of 3d map

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/098483 WO2022252235A1 (zh) 2021-06-04 2021-06-04 3d地图的解码装置和方法、及3d地图的编码码流

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/526,677 Continuation US20240119639A1 (en) 2021-06-04 2023-12-01 Apparatus and method for decoding 3d map, and encoded bitstream of 3d map

Publications (1)

Publication Number Publication Date
WO2022252235A1 true WO2022252235A1 (zh) 2022-12-08

Family

ID=84323760

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/098483 WO2022252235A1 (zh) 2021-06-04 2021-06-04 3d地图的解码装置和方法、及3d地图的编码码流

Country Status (5)

Country Link
US (1) US20240119639A1 (zh)
EP (1) EP4336456A1 (zh)
KR (1) KR20240017921A (zh)
CN (1) CN117461057A (zh)
WO (1) WO2022252235A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101290228A (zh) * 2007-04-17 2008-10-22 北京协进科技发展有限公司 一种基于移动终端的地图数据处理方法、系统及移动终端
CN102168983A (zh) * 2010-02-12 2011-08-31 三星电子株式会社 用于发送和接收地图数据的方法和装置
CN110869981A (zh) * 2016-12-30 2020-03-06 迪普迈普有限公司 用于自主车辆的高清晰度地图数据的向量数据编码
CN111639147A (zh) * 2020-04-24 2020-09-08 视辰信息科技(上海)有限公司 一种地图压缩方法、系统及计算机可读存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101290228A (zh) * 2007-04-17 2008-10-22 北京协进科技发展有限公司 一种基于移动终端的地图数据处理方法、系统及移动终端
CN102168983A (zh) * 2010-02-12 2011-08-31 三星电子株式会社 用于发送和接收地图数据的方法和装置
CN110869981A (zh) * 2016-12-30 2020-03-06 迪普迈普有限公司 用于自主车辆的高清晰度地图数据的向量数据编码
CN111639147A (zh) * 2020-04-24 2020-09-08 视辰信息科技(上海)有限公司 一种地图压缩方法、系统及计算机可读存储介质

Also Published As

Publication number Publication date
US20240119639A1 (en) 2024-04-11
CN117461057A (zh) 2024-01-26
KR20240017921A (ko) 2024-02-08
EP4336456A1 (en) 2024-03-13

Similar Documents

Publication Publication Date Title
WO2021088498A1 (zh) 虚拟物体显示方法以及电子设备
CN111026314B (zh) 控制显示设备的方法及便携设备
US20130329061A1 (en) Method and apparatus for storing image data
WO2021088497A1 (zh) 虚拟物体显示方法、全局地图更新方法以及设备
WO2022252235A1 (zh) 3d地图的解码装置和方法、及3d地图的编码码流
WO2018054114A1 (zh) 一种图片编码方法及终端
WO2022252234A1 (zh) 3d地图的编码装置和方法
WO2022252337A1 (zh) 3d地图的编解码方法及装置
WO2022252238A1 (zh) 3d地图的压缩、解压缩方法和装置
WO2022252237A1 (zh) 3d地图的编解码方法及装置
WO2022252236A1 (zh) 3d地图的编解码方法及装置
WO2022253228A1 (zh) 3d地图的压缩、解压缩方法和装置
WO2022140971A1 (zh) 图像传输方法、装置、平台、设备及计算机可读存储介质
CN116055798A (zh) 视频处理方法、装置及电子设备
CN116506645A (zh) 视频压缩方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21943596

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023574743

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 2021943596

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2021943596

Country of ref document: EP

Effective date: 20231208

ENP Entry into the national phase

Ref document number: 20247000378

Country of ref document: KR

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 1020247000378

Country of ref document: KR

NENP Non-entry into the national phase

Ref country code: DE