CN116880848A - Protobuf-based improved coding and decoding method and device - Google Patents
Protobuf-based improved coding and decoding method and device Download PDFInfo
- Publication number
- CN116880848A CN116880848A CN202310608851.3A CN202310608851A CN116880848A CN 116880848 A CN116880848 A CN 116880848A CN 202310608851 A CN202310608851 A CN 202310608851A CN 116880848 A CN116880848 A CN 116880848A
- Authority
- CN
- China
- Prior art keywords
- data
- identification
- coding
- protobuf
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 80
- 238000006243 chemical reaction Methods 0.000 claims abstract description 28
- 238000007405 data analysis Methods 0.000 claims abstract description 7
- 230000008569 process Effects 0.000 claims abstract description 5
- 238000003491 array Methods 0.000 claims description 9
- 238000005516 engineering process Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 7
- 230000005540 biological transmission Effects 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 14
- 230000011664 signaling Effects 0.000 description 19
- 238000004891 communication Methods 0.000 description 4
- 238000004590 computer program Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/06—Notations for structuring of protocol data, e.g. abstract syntax notation one [ASN.1]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
- G06F8/44—Encoding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The application discloses an improved coding and decoding method and device based on Protobuf, comprising the following steps: acquiring original data, analyzing the meaning, the corresponding data type and the arrangement sequence of each data value according to a data analysis protocol, and distributing uniform identification Id; creating a coding format table in a database, defining a field name for each data value and storing the field name; converting the original data into a byte array by a parallel conversion method to realize data coding; and extracting the identification Id according to the byte array, distinguishing the identification Id during decoding, and restoring the data type and the data value through the inverse process of the conversion method and displaying the data. The application realizes the mutual independence of data structure definition and data processing, and can also realize the centralized and unified processing of message data with different data formats when a Protobuf compiling plug-in is not used, thereby improving the data processing efficiency and the utilization efficiency of system resources.
Description
Technical Field
The application relates to the technical field of power grid data exchange processing, in particular to an improved coding and decoding method and device based on Protobuf.
Background
With the continuous expansion of the scale of the power grid equipment, the generated power grid data volume is increased, and the transmission and interaction of the power grid data are more frequent. The Protobuf technology has high serialization and anti-serialization speeds, low space overhead in the transmission process, supports multiple programming languages and is widely applied to the field of power grid data exchange processing.
In the Protobuf related technology, a data structure is defined in a Proto file, and a Protobuf compiling plug-in automatically generates related processing codes according to the names and types of data fields defined in the Proto file, and an application program calls the codes to complete serialization and deserialization of data.
However, the Protobuf serialized data can be read only after reverse serialization analysis is performed by combining the Proto file written code, and the readability is poor. Most of application program users in the power grid scene do not have code development capability, and difficulty is brought to data review. In addition, the data after Protobuf serialization lacks identity information, so that the data in each data format can only be processed independently and cannot be processed in a compatible way, and the consumption of system resources is increased to a certain extent.
Disclosure of Invention
This section is intended to outline some aspects of embodiments of the application and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section as well as in the description of the application and in the title of the application, which may not be used to limit the scope of the application.
The present application has been made in view of the above-described problems occurring in the prior art.
Therefore, the application aims to provide an improved coding and decoding method and device based on Protobuf, which solve the problems that in the field of power grid data exchange processing, the data after Protobuf serialization lacks identity information, so that the data in each data format can be processed independently and compatible processing can not be realized, and the consumption of system resources is increased.
In order to solve the technical problems, the application provides the following technical scheme:
in a first aspect, an embodiment of the present application provides an improved coding and decoding method based on Protobuf, including:
acquiring original data, analyzing the meaning, the corresponding data type and the arrangement sequence of each data value according to a data analysis protocol, and distributing uniform identification Id;
creating a coding format table in a database, and defining a field name for each data value;
storing the identification Id, the data field name and the data value meaning in the coding format table in sequence;
converting the original data into a byte array which comprises identification Id, data type and data value and is alternately arranged in sequence by a parallel conversion method, so as to realize data coding;
extracting an identification Id according to the byte array, distinguishing the data in a plurality of coding formats during mixed decoding, and restoring the data type and the data value through the inverse process of the conversion method;
and according to the identification Id, a specific coding format is queried in the database, and the data display is performed by combining the restored data type and the restored data value.
As an improved coding and decoding method based on Protobuf, the application comprises the following steps: the acquisition of the raw data may include,
receiving original data, and obtaining the original data by means of uploading an equipment terminal, forwarding a message middleware and reading an Http request interface;
and simulating the original data, namely simulating the original data generated in the original data format through the creation and splicing operation of the character strings in the application program.
As an improved coding and decoding method based on Protobuf, the application comprises the following steps: when raw data of multiple sources are processed simultaneously, the identification Id is used to distinguish between sequence numbers of various raw data sources, which are one byte and several bytes.
As an improved coding and decoding method based on Protobuf, the application comprises the following steps: the data encoding is accomplished by parallel conversion of the original data into an array of bytes containing identification Id, data type and data value, and arranged alternately in sequence,
placing the identification Id in the array header, mapping the data type directly into bytes;
mapping the data type into integer numbers, taking the mapped integer numbers as parameters, calling a corresponding conversion method, and converting the data value into bytes.
As an improved coding and decoding method based on Protobuf, the application comprises the following steps: the parallel conversion method comprises the steps of,
converting the data value of the basic data type into binary data through binary operation;
caching the original data in batches into the memory, and grouping each piece of original data according to the data type based on the size of the memory after the set number is reached;
after all the original data in the cache are grouped, adopting a multithreading technology to convert a plurality of groups of original data simultaneously;
after each piece of original data is subjected to grouping conversion, a plurality of sub-byte arrays are generated, and the sub-byte arrays are spliced into a complete byte array by identifying Id, so that conversion is completed.
As an improved coding and decoding method based on Protobuf, the application comprises the following steps: extracting an identification Id for distinguishing when data of a plurality of encoding formats are mixedly decoded according to the byte array, and restoring the data type and the data value comprises,
the first bit or the first few bytes of the byte array are used for storing the identification Id, and the identification Id can be directly extracted and used for distinguishing when the data in a plurality of coding formats are decoded simultaneously;
according to the coding protocol, the arrangement sequence and meaning of the original data in the byte array are determined, and the bytes are restored into the original data by adopting a restoring method as follows:
restoring the data type of the first data value from the byte array, selecting a restoring method of the data value according to the data type, and restoring the first data value;
all data types and data values in the byte array are restored in turn in this way.
As an improved coding and decoding method based on Protobuf, the application comprises the following steps: according to the identification Id, a specific coding format is queried in a database, and the data display is carried out by combining the restored data type and the data value,
according to the identification Id, a unique corresponding coding format is inquired from a coding format table in the database;
and sequentially matching the decoded data with the names of the data fields and the meanings of the data fields in the coding format to form a data display character string, and displaying the data display character string.
In a second aspect, embodiments of the present application provide an improved Protobuf-based encoding and decoding apparatus comprising,
the coding data transmitting module (100) acquires original data, analyzes the meaning of each data value, the corresponding data type and the arrangement sequence according to a data analysis protocol, distributes uniform identification Id, creates a coding format table in a database, defines a field name for each data value, sequentially stores the identification Id, the data field name and the meaning of the data value in the coding format table, converts the original data into a byte array which comprises the identification Id, the data type and the data value and is alternately arranged in sequence through a parallel conversion method, and realizes data coding;
converting the original data into a byte array which comprises identification Id, data type and data value and is alternately arranged in sequence, so as to realize data coding;
message middleware (200) for receiving the encoded data in the encoded data transmission module (100);
the data decoding and displaying function module (300) is used for acquiring byte arrays from the message middleware (200), extracting identification Id to distinguish when data in a plurality of coding formats are mixed and decoded, and restoring data types and data values through the inverse process of the conversion method;
according to the identification Id, a specific coding format is inquired in a database, and data display is carried out by combining the restored data type and the restored data value;
the terminal console (400) refers to the data presentation character string in the data decoding and presentation function module (300).
In a third aspect, embodiments of the present application provide a computing device comprising:
a memory and a processor;
the memory is configured to store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to implement the Protobuf-based improved encoding and decoding methods according to any of the embodiments of the present application.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the Protobuf-based improved encoding and decoding method.
The application has the beneficial effects that: the improved coding and decoding method and device based on Protobuf provided by the application realize that the definition of the data structure and the data processing are mutually independent, and can also develop the application program under the condition of compiling without using a Protobuf compiling plug-in, thereby reducing the threshold of development in the aspects of data coding and decoding; the message data with different data formats are intensively and uniformly processed, so that the data processing efficiency and the utilization efficiency of system resources are improved; the universal display module with the data does not need to create different data display methods according to different data formats, and reduces the development workload in the aspect of data display.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:
fig. 1 is a general flow diagram of an improved coding and decoding method based on Protobuf according to the present application.
Fig. 2 is a message format diagram of an improved coding and decoding method based on Protobuf according to the present application.
Fig. 3 is a message data encoding flow chart of an improved Protobuf-based encoding and decoding method of the present application.
Fig. 4 is a message data decoding flow chart of an improved coding and decoding method based on Protobuf of the present application.
Detailed Description
In order that the above-recited objects, features and advantages of the present application will become more readily apparent, a more particular description of the application will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application, but the present application may be practiced in other ways other than those described herein, and persons skilled in the art will readily appreciate that the present application is not limited to the specific embodiments disclosed below.
Further, reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic can be included in at least one implementation of the application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments.
Further, in describing the embodiments of the present application in detail, the cross-sectional view of the device structure is not partially enlarged to a general scale for convenience of description, and the schematic is only an example, which should not limit the scope of protection of the present application. In addition, the three-dimensional dimensions of length, width and depth should be included in actual fabrication.
Example 1
Referring to fig. 1 to 4, for one embodiment of the present application, there is provided an improved coding and decoding method based on Protobuf, including:
as shown in fig. 1 and 2, the specific flow and the coding flow of the present application are as follows:
s1: and acquiring the original data, analyzing the meaning, the corresponding data type and the arrangement sequence of each data value according to a data analysis protocol, and distributing uniform identification Id. It should be noted that:
acquiring the original data comprises receiving or simulating the original data;
specifically, the original data refers to a string of the agreed data type, which contains the data value and can be parsed. The agreed data types are basic data types including byte, int32, int64, float, double, and string.
Further, receiving the original data means that the original data is obtained through modes of uploading by a device terminal, forwarding by a message middleware, reading by an Http request interface and the like.
Further, the simulation of the original data refers to the simulation of the original data generated in the original data format through operations such as creation, splicing and the like of character strings in an application program.
Further, the data parsing protocol includes meaning of data value, data type, data arrangement order, and method of parsing data.
Further, when raw data of multiple sources are processed simultaneously, the Id is a sequence number for distinguishing the sources of the raw data, which is one byte or several bytes.
S2: an encoding format table is created in the database defining a field name for each data value. It should be noted that:
an encoding format table is created in the database defining a field name and a field meaning for each data value. The Id, data field name, and data field meaning are stored in the coding format table in order. Each identification Id uniquely identifies a coding format.
The database comprises a relational database, a distributed database and an internal memory database.
S3: the Id, data field name, and meaning of the data value are stored in the coding format table in order.
It should be noted that:
each identification Id uniquely identifies a coding format;
s4: the original data is converted into a byte array which comprises identification Id, data type and data value and is alternately arranged in sequence, so that data coding is realized. It should be noted that:
further, for the method of converting Protobuf strip by strip, the embodiment provides a parallel conversion method. And caching the original data in batches into the memory, and grouping each piece of original data according to the data type based on the size of the memory after the set number is reached. After all the original data in the buffer memory are grouped, adopting a multithreading technology to convert a plurality of groups of original data simultaneously. After each piece of original data is subjected to grouping conversion, a plurality of sub-byte arrays are generated, and the sub-byte arrays are spliced into a complete byte array by identifying Id, so that conversion is completed.
Specifically, converting the data value of the basic data type into binary data through binary operation; the method comprises the steps of taking 8-bit data from the lowest bit to be stored into a byte array as one byte, shifting binary data to the right by 8 bits through bit operation, and taking 8-bit data to be stored into the byte array as one byte.
And sequentially executing until all binary data are stored in the byte array.
According to the size of the occupied memory, the basic data types can be divided into four data types based on the memory size, namely, occupied one byte: byte type, occupies four bytes: type int32 and float, occupy 8 bytes: int64 and double, no fixed size: string type.
Wherein float and double are converted into int32 and int64 types by the flotattoinitbs and double tolongbits methods provided in Java, respectively. The data values of float and double can be considered as the data values of int32 and int 64.
Each data value of the data type based on the memory size corresponds to a conversion method for converting the data value into bytes.
Further, as shown in FIG. 3, further, a byte array is created, placing the identification Id in the array header.
The data type is mapped directly into bytes. The data value is converted into bytes according to the method of selecting the conversion from the serialization tools according to the data type.
And after the Id is identified, the bytes of the data type and the data value are orderly and alternately arranged in sequence to form a byte array, so that data coding is realized.
As shown in fig. 4, the decoding flow of the present application is as follows:
s5: and extracting the identification Id according to the byte array, distinguishing the data in a plurality of coding formats during mixed decoding, and restoring the data types and the data values. It should be noted that:
it should be noted that during the decoding phase, the first or first few bytes of the byte array are used to store the Id, so that the Id can be directly extracted for distinguishing when data in multiple encoding formats is decoded simultaneously.
According to the coding protocol, the arrangement sequence and meaning of the original data in the byte array are determined, the bytes are restored into the original data by adopting a restoring method, and the original data are stored in the application program memory for the subsequent functional modules to use.
The method of restoring bytes into basic type data is independently stripped from the Protobuf related technology, and the restoring method used in the application is modified.
The data type of the first data value is restored from the byte array, and a restoring method of the data value is selected according to the data type, so that the first data value is restored. All data types and data values in the byte array are restored in turn and stored in the application memory for later use.
S6: and according to the identification Id, a specific coding format is queried in the database, and the data display is performed by combining the restored data type and the restored data value. It should be noted that:
according to the identification Id, a unique corresponding coding format is inquired from a coding format table in the database;
and sequentially matching the decoded data with the names of the data fields and the meanings of the data fields in the coding format to form a data display character string, displaying the data, and sending the data to a terminal console or a local log file for reference.
The embodiment also provides an improved encoding and decoding device based on Protobuf, which comprises:
the coded data sending module 100 obtains the original data, analyzes the meaning of each data value, the corresponding data type and the arrangement sequence according to the data analysis protocol, distributes uniform identification Id, creates a coded format table in a database, defines a field name for each data value, sequentially stores the identification Id, the data field name and the meaning of the data value in the coded format table, converts the original data into a byte array which comprises the identification Id, the data type and the data value and is alternately arranged in sequence, and realizes data coding;
message middleware 200 for receiving the encoded data in the encoded data transmission module 100;
the data decoding and displaying function module 300 is configured to obtain a byte array from the message middleware 200, extract an Id to distinguish between the data types and the data values when the data in the multiple encoding formats are mixed and decoded, and restore the data types and the data values;
according to the identification Id, a specific coding format is inquired in a database, and data display is carried out by combining the restored data type and the restored data value;
the terminal console 400 refers to the data presentation character string in the data decoding and presentation function module 300.
Further, the data coding and data sending functions are integrated, a coded data sending module is realized, and data is sent to the message middleware. Message middleware mainly includes Kafka, activeMQ, rabbitMQ, rocketMQ, zeroMQ and the like.
The present embodiment also provides a computing device adapted for use in the case of a Protobuf-based improved encoding and decoding method, including:
a memory and a processor; the memory is used for storing computer executable instructions and the processor is used for executing the computer executable instructions to implement the improved coding and decoding method based on Protobuf as proposed in the above embodiments.
The computer device may be a terminal comprising a processor, a memory, a communication interface, a display screen and input means connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
The present embodiment also provides a storage medium having stored thereon a computer program which, when executed by a processor, implements an improved encoding and decoding method based on Protobuf as proposed by the above embodiments.
The storage medium according to the present embodiment belongs to the same inventive concept as the data storage method according to the above embodiment, and technical details not described in detail in the present embodiment can be seen in the above embodiment, and the present embodiment has the same advantageous effects as the above embodiment.
Example 2
For another embodiment of the present application, which is different from the first embodiment, a verification test of an improved coding and decoding method based on Protobuf is provided, and the technical effects adopted in the method are verified and explained.
In the grid data processing, there are two types of data, remote signaling and telemetry, and the data structures are shown in table 1 and table 2 respectively:
table 1: telemetry data structure table
| yc_Id | Telemetry data Id |
| yc_value | Telemetry data values |
| yc_status | Status of telemetry data |
Table 2: remote signaling data structure table
| yx_Id | Remote signaling data Id |
| yx_value | Remote signaling data value |
| yx_status | Status of remote signaling data |
The use of conventional Protobuf techniques for serialized transmission requires the following steps:
a1: the definition of the telemetry and telemetry data structures to create the Proto file and to make the telemetry and telemetry data structures in the file is as follows:
telemetry data structure
Remote signaling data structure
A2: protobuf plug-ins are introduced into telemetry and telemetry data processing applications, respectively.
A3: the telemetry Proto file is compiled through the Protobuf plug-in, a conversion method capable of converting yc_id, yc_value and yc_status into bytes and a reduction method capable of reducing the bytes into yc_id, yc_value and yc_status are automatically generated.
A4: the remote signaling Proto file is compiled through the Protobuf plug-in, a conversion method capable of converting the yx_id, the yx_value and the yx_status into bytes and a reduction method capable of reducing the bytes into the yx_Id, the yx_value and the yx_status are automatically generated.
A5: the data to be processed of telemetry and remote signaling can be converted into byte arrays to be transmitted in a network through a conversion method.
A6: and analyzing the telemetry information and telemetry data to be processed by a reduction method, and processing or displaying the telemetry information and the telemetry data.
A7: when data processing and display are carried out, telemetry and remote signaling data are required to be respectively developed and displayed application programs.
The following steps are required to use my Protobuf-based improved encoding and decoding methods:
b1: for telemetry and remote signaling data, an identification Id is allocated, for example, the telemetry data identification Id is 01, and the remote signaling data identification Id is 02.
B2: from the telemetry data structure, yc_id, yc_value, yc_status data types are int64, float, and int32, respectively. Encoding and decoding are performed in connection with the encoding and decoding methods referred to in the present application. The remote signaling data processing method is similar.
B3: if the telemetry and remote signaling data need to be mixed, the data are processed. In the decoding stage, remote signaling and telemetry data can be distinguished according to identification Id distinction, so that different processing methods are selected.
B4: if data presentation is required, a table is created in the database to store telemetry and telemetry data structures, as shown in the table below.
Table 3: telemetry and telemetry data structure table
| Identification Id | Field name | Meaning of field |
| 01 | yc_id | Telemetry Id |
| 01 | yc_value | Telemetry value |
| 01 | yc_status | Telemetry status |
| 02 | yx_id | Remote signaling Id |
| 02 | yx_value | Remote signaling value |
| 02 | yx_status | Remote signaling value |
After the telemetry and remote signaling data are uniformly decoded by using the decoding method, the corresponding field name and field meaning are inquired from the database according to the identification Id, and the data display is completed by combining the decoded telemetry and remote signaling data types and data values.
It should be noted that the above embodiments are only for illustrating the technical solution of the present application and not for limiting the same, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present application may be modified or substituted without departing from the spirit and scope of the technical solution of the present application, which is intended to be covered in the scope of the claims of the present application.
Claims (10)
1. An improved coding and decoding method based on Protobuf, which is characterized in that: comprising the steps of (a) a step of,
acquiring original data, analyzing the meaning, the corresponding data type and the arrangement sequence of each data value according to a data analysis protocol, and distributing uniform identification Id;
creating a coding format table in a database, and defining a field name for each data value;
storing the identification Id, the data field name and the data value meaning in the coding format table in sequence;
converting the original data into a byte array which comprises identification Id, data type and data value and is alternately arranged in sequence by a parallel conversion method, so as to realize data coding;
extracting an identification Id according to the byte array, distinguishing the data in a plurality of coding formats during mixed decoding, and restoring the data type and the data value through the inverse process of the conversion method;
and according to the identification Id, a specific coding format is queried in the database, and the data display is performed by combining the restored data type and the restored data value.
2. An improved Protobuf-based encoding and decoding method according to claim 1, wherein: the acquisition of the raw data may include,
receiving original data, and obtaining the original data by means of uploading an equipment terminal, forwarding a message middleware and reading an Http request interface;
and simulating the original data, namely simulating the original data generated in the original data format through the creation and splicing operation of the character strings in the application program.
3. An improved Protobuf-based encoding and decoding method according to claim 1, wherein: when raw data of multiple sources are processed simultaneously, the identification Id is used to distinguish between sequence numbers of various raw data sources, which are one byte and several bytes.
4. An improved Protobuf-based encoding and decoding method according to claim 1, wherein: the data encoding is accomplished by parallel conversion of the original data into an array of bytes containing identification Id, data type and data value, and arranged alternately in sequence,
placing the identification Id in the array header, mapping the data type directly into bytes;
mapping the data type into integer numbers, taking the mapped integer numbers as parameters, calling a corresponding conversion method, and converting the data value into bytes.
5. The improved Protobuf-based encoding and decoding method of claim 4, wherein: the parallel conversion method comprises the steps of,
converting the data value of the basic data type into binary data through binary operation;
caching the original data in batches into the memory, and grouping each piece of original data according to the data type based on the size of the memory after the set number is reached;
after all the original data in the cache are grouped, adopting a multithreading technology to convert a plurality of groups of original data simultaneously;
after each piece of original data is subjected to grouping conversion, a plurality of sub-byte arrays are generated, and the sub-byte arrays are spliced into a complete byte array by identifying Id, so that conversion is completed.
6. An improved Protobuf-based encoding and decoding method according to claim 1, wherein: extracting an identification Id for distinguishing when data of a plurality of encoding formats are mixedly decoded according to the byte array, and restoring the data type and the data value comprises,
the first bit or the first few bytes of the byte array are used for storing the identification Id, and the identification Id can be directly extracted and used for distinguishing when the data in a plurality of coding formats are decoded simultaneously;
according to the coding protocol, the arrangement sequence and meaning of the original data in the byte array are determined, and the bytes are restored into the original data by adopting a restoring method as follows:
restoring the data type of the first data value from the byte array, selecting a restoring method of the data value according to the data type, and restoring the first data value;
all data types and data values in the byte array are restored in turn in this way.
7. The improved Protobuf-based encoding and decoding method of claim 6, wherein: according to the identification Id, a specific coding format is queried in a database, and the data display is carried out by combining the restored data type and the data value,
according to the identification Id, a unique corresponding coding format is inquired from a coding format table in the database;
and sequentially matching the decoded data with the names of the data fields and the meanings of the data fields in the coding format to form a data display character string, and displaying the data display character string.
8. An improved Protobuf-based encoding and decoding apparatus, comprising,
the coding data transmitting module (100) acquires original data, analyzes the meaning of each data value, the corresponding data type and the arrangement sequence according to a data analysis protocol, distributes uniform identification Id, creates a coding format table in a database, defines a field name for each data value, sequentially stores the identification Id, the data field name and the meaning of the data value in the coding format table, converts the original data into a byte array which comprises the identification Id, the data type and the data value and is alternately arranged in sequence through a parallel conversion method, and realizes data coding;
converting the original data into a byte array which comprises identification Id, data type and data value and is alternately arranged in sequence, so as to realize data coding;
message middleware (200) for receiving the encoded data in the encoded data transmission module (100);
the data decoding and displaying function module (300) is used for acquiring byte arrays from the message middleware (200), extracting identification Id to distinguish when data in a plurality of coding formats are mixed and decoded, and restoring data types and data values through the inverse process of the conversion method;
according to the identification Id, a specific coding format is inquired in a database, and data display is carried out by combining the restored data type and the restored data value;
the terminal console (400) refers to the data presentation character string in the data decoding and presentation function module (300).
9. A computing device, comprising:
a memory and a processor;
the memory is configured to store computer executable instructions that, when executed by a processor, implement the steps of the Protobuf-based improved encoding and decoding method of any of claims 1 to 7.
10. A computer readable storage medium storing computer executable instructions which when executed by a processor implement the steps of the Protobuf-based improved encoding and decoding method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310608851.3A CN116880848A (en) | 2023-05-26 | 2023-05-26 | Protobuf-based improved coding and decoding method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310608851.3A CN116880848A (en) | 2023-05-26 | 2023-05-26 | Protobuf-based improved coding and decoding method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116880848A true CN116880848A (en) | 2023-10-13 |
Family
ID=88263182
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310608851.3A Pending CN116880848A (en) | 2023-05-26 | 2023-05-26 | Protobuf-based improved coding and decoding method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116880848A (en) |
-
2023
- 2023-05-26 CN CN202310608851.3A patent/CN116880848A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN1166072C (en) | Data compaction, transmission, storage and program transmission | |
| CN105450232A (en) | Encoding method, decoding method, encoding device and decoding device | |
| US8265407B2 (en) | Method for coding and decoding 3D data implemented as a mesh model | |
| CN101223699A (en) | Methods and devices for compressing and decompressing structured documents | |
| CN101526963A (en) | Method for identifying web page coding, device and terminal equipment | |
| CN111131403A (en) | Message coding and decoding method and device for Internet of things equipment | |
| CN102255687B (en) | Rate matching method and device thereof | |
| CN104113390A (en) | Serialization method of mobile network data | |
| JP5551660B2 (en) | Computer-implemented method for encoding text into matrix code symbols, computer-implemented method for decoding matrix code symbols, encoder for encoding text into matrix code symbols, and decoder for decoding matrix code symbols | |
| CN113722533B (en) | Information pushing method and device, electronic equipment and readable storage medium | |
| Abas et al. | Increasing data storage of coloured QR code using compress, multiplexing and multilayered technique | |
| CN114064308A (en) | Multi-data sending and receiving method, device and equipment based on column type data scanning | |
| CN116880848A (en) | Protobuf-based improved coding and decoding method and device | |
| CN101686568B (en) | Methods and terminals for transmitting and displaying text information | |
| CN1127276C (en) | Handwriting inputting, sending and receiving method and system for mobile terminal | |
| CN115167869A (en) | Method, electronic device and medium for serialization and deserialization of Java object | |
| CN102185612A (en) | Run-length coding and decoding methods and devices | |
| CN102289358A (en) | A computer-implemented method, a computer program product and an embedded system for displaying data more efficiently | |
| CN110855727A (en) | System and method for transmitting data objects based on byte serialization and deserialization | |
| CN100558085C (en) | A kind of automatic identification system uses the method for Chinese character exchange message | |
| CN114070901A (en) | Data sending and receiving method, device and equipment based on multi-data alignment | |
| CN114039698A (en) | Data sending and receiving method, device and equipment based on multiple frequencies | |
| CN111931205A (en) | Method and device for encrypting shader file | |
| CN115001628B (en) | Data encoding method and device, data decoding method and device and data structure | |
| CN117807036A (en) | File import and export method, system and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |