CN107992624B - Method, device, storage medium and terminal equipment for analyzing serialized data - Google Patents
Method, device, storage medium and terminal equipment for analyzing serialized data Download PDFInfo
- Publication number
- CN107992624B CN107992624B CN201711407433.9A CN201711407433A CN107992624B CN 107992624 B CN107992624 B CN 107992624B CN 201711407433 A CN201711407433 A CN 201711407433A CN 107992624 B CN107992624 B CN 107992624B
- Authority
- CN
- China
- Prior art keywords
- serialized
- data
- serialization
- function
- acquisition function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
Abstract
The invention provides a method, a device, a storage medium and a terminal device for analyzing serialized data, wherein the method comprises the following steps: responding to a deserialization request for analyzing the serialized text, and acquiring a serialized acquisition function of the serialized text; the serialization acquisition function comprises a mapping relation between a data structure and a data identifier, wherein the data structure is used for defining the data structure of the serialization data corresponding to the data identifier mapped by the data structure; acquiring a data structure mapped by a data identifier of the serialized data to be subjected to the deserialization operation through the serialization acquisition function; and according to the acquired data structure, performing deserialization operation on the serialized data to be deserialized. By adopting the invention, the efficiency of analyzing the serialized data can be improved.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a method and an apparatus for downloading parsed serialized data, a storage medium, and a terminal device.
Background
Protobuf is a Google open source similar to the Json, XM L data exchange format, the internal data of the Protobuf is in a pure binary format, does not depend on languages and platforms, and has the advantages of simplicity, small data volume, quickness and the like.
In a daily reading or querying mode of the PB log, a Proto configuration file of the PB log is generally obtained through metadata describing access information of the PB log, a log analysis engine dynamically serializes the Proto configuration file in each reading task performed on PB log data, directly obtains class compiling information of the Proto configuration file, and deserializes the PB log data by using the serialized Proto configuration file, thereby obtaining specific content in the deserialized data and completing the reading task. The scheme has the advantages of high universality and low coupling degree, and data reading can be completed without pre-compiling the Proto configuration file. However, the time consumption and the length of the process of deserializing the PB log are too long and the efficiency is low because the configuration file needs to be dynamically serialized.
Disclosure of Invention
The embodiment of the invention provides a method, a system, a device, a storage medium and terminal equipment for analyzing serialized data, and the efficiency of analyzing the serialized data is improved.
In a first aspect, an embodiment of the present invention provides a method for parsing serialized data, where the method includes:
responding to a deserialization request for analyzing the serialized text, and acquiring a serialized acquisition function of the serialized text; the serialization acquisition function comprises a mapping relation between a data structure and a data identifier, wherein the data structure is used for defining the data structure of the serialization data corresponding to the data identifier mapped by the data structure;
acquiring a data structure mapped by a data identifier of the serialized data to be subjected to the deserialization operation through the serialization acquisition function; and
and according to the acquired data structure, performing deserialization operation on the serialized data to be deserialized.
With reference to the first aspect, in a first implementation manner of the first aspect, before the obtaining the serialization obtaining function of the serialized text, the method further includes:
acquiring a configuration file of the serialized text; wherein the configuration file is used for defining a data structure of the serialized data in the serialized text;
generating an acquisition function of the serialized texts according to the configuration file; and
and executing serialization operation on the acquisition function to generate a serialized acquisition function.
With reference to the first aspect, in a second implementation manner of the first aspect, the generating an obtaining function of the serialized text according to the configuration file includes:
extracting a parent data identifier corresponding to a data structure for defining the serialized data in the serialized text from the configuration file as a function name of an acquisition function;
compiling the configuration file to generate a configuration source code; and
configuring and generating an acquisition function of the serialized text according to the configuration source code and the function name of the acquisition function; wherein the obtaining function includes a mapping relationship between a source code of the configuration source code for defining a data structure of the serialized data and a source code of a data identifier of the serialized data
With reference to the first aspect, in a third implementation manner of the first aspect, the performing a serialization operation on the obtaining function includes:
invoking a serialization tool for performing a serialization operation and generating the serialized text; and
and controlling the serialization tool to execute the serialization operation on the acquisition function.
In a second aspect, an embodiment of the present invention provides an apparatus for parsing serialized data, including:
the function acquisition module is used for responding to an anti-serialization request for analyzing the serialized texts and acquiring the serialized acquisition function of the serialized texts; the serialization acquisition function comprises a mapping relation between a data structure and a data identifier, wherein the data structure is used for defining the data structure of the serialization data corresponding to the data identifier mapped by the data structure;
the data structure acquisition module is used for acquiring a data structure mapped by the data identifier of the serialized data to be subjected to the deserialization operation through the serialization acquisition function according to the data identifier of the serialized data to be subjected to the deserialization operation; and
and the deserializing operation module is used for performing deserializing operation on the serialized data to be deserialized according to the acquired data structure.
With reference to the second aspect, the present invention provides, in a first embodiment of the second aspect, the apparatus further comprising:
the configuration file acquisition module is used for acquiring a configuration file of the serialized text before acquiring a serialized acquisition function of the serialized text; wherein the configuration file is used for defining a data structure of the serialized data in the serialized text;
the function generating module is used for generating an obtaining function of the serialized texts according to the configuration file; and
and the function serialization module is used for executing serialization operation on the acquisition function to generate a serialized acquisition function.
With reference to the second aspect, in a second implementation manner of the second aspect, the function generation module includes:
the function name definition unit is used for extracting a parent data identifier corresponding to a data structure for defining the serialized data in the serialized text from the configuration file as a function name of the acquisition function;
the source code compiling unit is used for compiling the configuration file to generate a configuration source code; and
the function configuration unit is used for configuring and generating the acquisition function of the serialized text according to the configuration source code and the function name of the acquisition function; the obtaining function comprises a mapping relation between a source code of a data structure used for defining the serialized data in the configuration source code and a source code of a data identifier of the serialized data.
With reference to the second aspect, in a third implementation manner of the second aspect, the function serialization module includes:
the tool calling unit is used for calling a serialization tool for executing the serialization operation and generating the serialization text; and
and the serialization execution unit is used for controlling the serialization tool and executing the serialization operation on the acquisition function.
The functions of the device can be realized by hardware, and can also be realized by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions.
In one possible design, the structure of the apparatus for parsing serialized data includes a processor and a memory, the memory is used for storing a program supporting the apparatus for parsing serialized data to execute the method for parsing serialized data in the first aspect, and the processor is configured to execute the program stored in the memory. The means for parsing the serialized data can also include a communication interface, the means for parsing the serialized data in communication with another device or a communication network.
In a third aspect, an embodiment of the present invention provides a computer-readable storage medium for storing computer software instructions for an apparatus for parsing serialized data, which includes a program for executing the method for parsing serialized data in the first aspect described above as an apparatus for parsing serialized data.
Any one of the above technical solutions has the following advantages or beneficial effects:
when the deserialization request of the serialized text is responded, the serialized acquisition function of the serialized text is acquired; because the serialization acquisition function comprises the mapping relation between the data structure and the data identifier, and the data structure is used for defining the data structure of the serialization data corresponding to the data identifier mapped by the data structure, when the serialization data to be executed is inquired, the data structure mapped by the data identifier of the serialization data to be executed is acquired directly through the serialization acquisition function; and according to the acquired data structure, performing deserialization operation on the serialized data to be deserialized, and further querying the content of the obtained serialized data.
The foregoing summary is provided for the purpose of description only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the present invention will be readily apparent by reference to the drawings and following detailed description.
Drawings
In the drawings, like reference numerals refer to the same or similar parts or elements throughout the several views unless otherwise specified. The figures are not necessarily to scale. It is appreciated that these drawings depict only some embodiments in accordance with the disclosure and are therefore not to be considered limiting of its scope.
FIG. 1 is a schematic flow chart diagram illustrating one embodiment of a method for parsing serialized data provided by the present invention;
FIG. 2 is a flow chart illustrating one embodiment of a process for generating a serialized acquisition function of the method for parsing serialized data provided by the present invention;
FIG. 3 is a flow chart illustrating one embodiment of a process for generating an acquisition function of a method for parsing serialized data provided by the present invention;
FIG. 4 is a schematic flow chart diagram illustrating another embodiment of a method for parsing serialized data provided by the present invention;
FIG. 5 is a schematic structural diagram of an embodiment of an apparatus for parsing serialized data provided by the present invention;
fig. 6 is a schematic structural diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
In the following, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.
Referring to fig. 1, an embodiment of the present invention provides a method for parsing serialized data, where the method may be executed by an deserializer, or may be executed by a processor, where the deserializer is invoked and controls the deserializer to perform corresponding operations, or is directly executed by the processor, and the deserializer may be an engine, for example: the wind-query engine, the deserializing tool is also a compiler in nature, and further the parsing process of the embodiment of the present invention can be regarded as a process of compiling serialized data, and conversely, the process of generating serialized data can be regarded as a compiling process. The technical solution provided by the embodiment of the present invention will be described in detail below with reference to fig. 1:
the method provided by the embodiment of the invention comprises the following steps S100 to S300:
step S100, responding to a deserialization request for analyzing a serialized text, and acquiring a serialized acquisition function of the serialized text; the serialization acquisition function comprises a mapping relation between a data structure and a data identifier, wherein the data structure is used for defining the data structure of the serialization data corresponding to the data identifier mapped by the data structure.
In the embodiment of the present invention, the serialized text refers to a binary string written structured data or object, and the present invention describes the serialized file of the embodiment of the present invention by taking a mixed language data standard developed by Google corporation, Google Protocol Buffer (Protocol for short) as an example. Protobuf is a lightweight, efficient, structured data storage format that can be used to serialize, or serialize, structured data or objects. Due to the characteristics of small data volume and quick transmission, the format is generally adopted for storing and transmitting the system logs for recording the running of various software in a system, which is called PB logs for short. Generally, the PB log is stored in metadata, which describes information such as a storage location of the PB log, information of a configuration file, and a right to ask a question, and thus, in an embodiment of the present invention, a serialized text may be acquired through the metadata.
In the embodiment of the present invention, the serialized acquisition function of the serialized text may be generated according to a configuration file of the serialized text, or may be generated according to the metadata directly.
In the embodiment of the invention, when the deserialization request for analyzing the serialized text is responded, whether a serialized acquisition function of the serialized text exists or not is inquired; if so, directly acquiring a serialization acquisition function of the serialization text; if not, the configuration file is required to be acquired from the metadata, and the configuration file is generated according to the configuration file. Thus, in order to avoid that the serialized acquisition function of the serialized text cannot be queried, the serialized acquisition function can be generated and stored in advance before the serialized acquisition function of the serialized text is acquired or before an anti-serialization request for parsing the serialized text is responded. Compared with the existing scheme, the scheme has the advantages that in the process of deserializing the serialized text, the configuration file of the serialized text is directly compiled, and the data structure of the serialized text is obtained in real time to execute the dynamic deserializing mode of deserializing. As shown in fig. 2, the process of generating the serialized acquisition function provided in the embodiment of the present invention includes steps S110 to S130:
step S110, acquiring a configuration file of the serialized text; wherein the configuration file is used for defining a data structure of the serialized data in the serialized text; specifically, a configuration file of the serialized text may be obtained from the metadata base.
In the embodiment of the present invention, protobuf is taken as an example, the serialized text is protobuf formatted text, which includes a plurality of serialized data, and then the configuration file defines how the data structure of the serialized text and each serialized data in the serialized text is, and a suffix of the configuration file of the protobuf formatted text is generally a proto, which is abbreviated as proto file, and the proto file is similar to a data definition of java or C language. In protobuf's terminology, one piece of serialized data in the serialized text is referred to as a Message. Wherein the serialized document and the serialized data, and the serialized data can be preceded by a nested form.
And step S120, generating an acquisition function of the serialized texts according to the configuration file.
In the embodiment of the present invention, since the configuration file describes how the data structure of the Message is, an obtaining function may be generated according to a mapping relationship between the identifier of the Message and the data structure of the Message.
Referring to fig. 3, the generating process of the obtaining function according to the embodiment of the present invention can be specifically divided into the following steps S121 to S123:
step S121, extracting, from the configuration file, a parent data identifier corresponding to a data structure for defining serialized data in the serialized text as a function name of the acquisition function. Because the data structure of the serialized texts is in a nested format, the identifier of the message at the top layer, namely the identifier of the parent-level data, is taken as the function name of the acquisition function, and the corresponding serialized acquisition functions can be conveniently selected corresponding to different serialized texts. Generally, the identification of the top-most message is the same as the text identification of the serialized text. Of course, the method is not limited to using the parent data identifier as the obtaining function, and may be set accordingly according to the compiling habit.
And step S122, compiling the configuration file to generate a configuration source code. Preferably, the format of the configuration source code is C + +, and other compiling formats, such as C, JAVA, etc., are also possible. In the embodiment of the present invention, since the obtained function is generated based on the configuration file, and since the original configuration file cannot directly generate the obtained function, the configuration file needs to be compiled into the source code by the compiler, and the configuration file can be generated according to the configuration source code.
Step S123, configuring and generating the acquiring function of the serialized text according to the function name of the configuration source code and the acquiring function; because the configuration file or the configuration source code is a file defining a data structure of the serialized data, what the data structure of a certain serialized data is can be queried from the file by setting a function, and the source code of the acquisition function at least comprises a mapping relation between a source code of the data structure defining the serialized data in the configuration source code and a source code of a data identifier of the serialized data. And calling the acquisition function to perform query, namely calling the source code of the acquisition function to perform query.
Step S130, performing a serialization operation on the acquisition function to generate a serialized acquisition function.
In the embodiment of the present invention, since the serialized data to be subjected to the deserialization operation is in the serialized format, the source code of the acquisition function needs to be converted into the serialized format, so that the deserialization operation can be performed. That is, any of the serialized data in the serialized text can be deserialized by the serialization acquisition function.
Preferably, the specific implementation process of step S130 in the embodiment of the present invention may be:
invoking a serialization tool for performing a serialization operation and generating the serialized text; and
and controlling the serialization tool to execute the serialization operation on the acquisition function.
In the implementation of the present invention, the acquisition function is serialized by using a serialization tool that is native to the serialized file, so that the data structure of the serialized acquisition function can be consistent with the data structure of the serialized file. Of course, other serialization tools may be used to perform the serialization operation, and it is only necessary to achieve that the data structure of the serialization acquisition function is consistent with the data structure of the serialization file.
After the serialization acquisition function is obtained, the task of performing the anti-serialization operation on the serialization text can be divided into N anti-serialization tasks, and the process of executing each anti-serialization task can call the serialization acquisition function to execute the following step S200 and step S300. In contrast, in the embodiment of the present invention, the data structure is read through a function, that is, a reflection acquisition mechanism, and the data structure can be dynamically retrieved from the serialization acquisition function according to the requirement of deserialization to perform deserialization operation, thereby further improving the parsing efficiency of deserialization.
Step S200, acquiring a data structure of data identifier mapping of the serialized data to be subjected to the deserialization operation through the serialization acquisition function; and
and step S300, performing deserialization operation on the serialized data to be deserialized according to the acquired data structure.
The processes of step S200 and step S300 of the present invention are described in colloquially, that is, a key is extracted by using the serialization acquisition function in the embodiment of the present invention, and the extracted key is used to parse the corresponding serialization data in the serialization text.
The technical scheme for analyzing the serialized data by the static reverse solution and reflection acquisition mechanism and the technical scheme provided by the prior art are respectively adopted to carry out the experiment for analyzing the serialized data, and the efficiency for analyzing the compressed file or the uncompressed file by the technical scheme provided by the embodiment of the invention is higher than the efficiency for analyzing the compressed file or the uncompressed file by the technical scheme provided by the prior art, wherein the efficiency for the uncompressed file can be improved by 13 times, and the efficiency for the compressed file can be improved by 2 times.
Referring to fig. 4, the embodiment of the present invention uses the serialized text as a PB log, a processor as an execution main body, a Wing query engine as an anti-serialization tool, and a PB compiler as a serialization tool, and explains a specific implementation process of the embodiment of the present invention:
and step 51, acquiring a configuration file of the PB log from the metadata, and compiling the configuration file into a configuration source code with a format of C + + through a PB compiler.
And step 52, acquiring the main message identification name of the configuration file according to the content of the configuration file, and taking the main message identification name as the function name of the acquisition function.
And 53, configuring the acquisition function into an executable acquisition function according to the configuration source code, wherein the executable acquisition function is in a bc file format which can be identified by a Wing query engine. When the main message identification name is typed in the obtaining function, the data structure which is associated with the main message identification name in the configuration source code can be obtained.
And step 54, importing the executable acquisition function into a Wing query engine, and calling a PB compiler by a rear-end optimization module of the Wing query engine to serialize the executable acquisition function to obtain a serialized acquisition function.
And step 55, importing the serialization acquisition function into a map end of the Wing query engine, and controlling the map end to acquire a data structure corresponding to the identification name of the serialization data of the PB log according to the acquisition function.
And 56, controlling the map end to perform deserialization on the serialized data according to the data structure.
Referring to fig. 5, an embodiment of the present invention provides an apparatus for parsing serialized data, which can implement any flow method in the foregoing embodiments, and specifically includes:
a function obtaining module 10, configured to respond to an deserialization request for parsing a serialized text, and obtain a serialized obtaining function of the serialized text; the serialization acquisition function comprises a mapping relation between a data structure and a data identifier, wherein the data structure is used for defining the data structure of the serialization data corresponding to the data identifier mapped by the data structure;
a data structure obtaining module 20, configured to obtain, according to a data identifier of serialized data to be subjected to deserialization operation, a data structure mapped by the data identifier of the serialized data to be subjected to deserialization operation through the serialization obtaining function; and
and the deserializing operation module 30 is configured to perform deserializing operation on the serialized data to be deserialized according to the acquired data structure.
With further improvement in combination with the foregoing embodiment, the apparatus further includes:
the configuration file acquisition module is used for acquiring a configuration file of the serialized text before acquiring a serialized acquisition function of the serialized text; wherein the configuration file is used for defining a data structure of the serialized data in the serialized text;
the function generating module is used for generating an obtaining function of the serialized texts according to the configuration file; and
and the function serialization module is used for executing serialization operation on the acquisition function to generate a serialized acquisition function.
With reference to the foregoing embodiment, in further improvement, the function generation module includes:
the function name definition unit is used for extracting a parent data identifier corresponding to a data structure for defining the serialized data in the serialized text from the configuration file as a function name of the acquisition function;
the source code compiling unit is used for compiling the configuration file to generate a configuration source code; and
the function configuration unit is used for configuring and generating the acquisition function of the serialized text according to the configuration source code and the function name of the acquisition function; the obtaining function comprises a mapping relation between a source code of a data structure used for defining the serialized data in the configuration source code and a source code of a data identifier of the serialized data.
In combination with the foregoing embodiment, in a further improvement, the function serialization module includes:
the tool calling unit is used for calling a serialization tool for executing the serialization operation and generating the serialization text; and
and the serialization execution unit is used for controlling the serialization tool and executing the serialization operation on the acquisition function.
The functions of the device can be realized by hardware, and can also be realized by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions.
An embodiment of the present invention further provides a terminal device, as shown in fig. 6, where the terminal device includes: a memory 21 and a processor 22, the memory 21 having stored therein a computer program operable on the processor 22. The processor 22, when executing the computer program, implements the method of parsing serialized data in the embodiments described above. The number of the memory 21 and the processor 22 may be one or more.
The apparatus further comprises:
a communication interface 23 for communication between the memory 21 and the processor 22.
The memory 21 may comprise a high-speed RAM memory, and may further include a non-volatile memory (non-volatile memory), such as at least one disk memory.
If the memory 21, the processor 22 and the communication interface 23 are implemented independently, the memory 21, the processor 22 and the communication interface 23 may be connected to each other through a bus and perform communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 6, but this is not intended to represent only one bus or type of bus.
Optionally, in a specific implementation, if the memory 21, the processor 22 and the communication interface 23 are integrated on a chip, the memory 21, the processor 22 and the communication interface 23 may complete mutual communication through an internal interface.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
The computer readable medium described in embodiments of the present invention may be a computer readable signal medium or a computer readable storage medium or any combination of the two. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM). Additionally, the computer-readable storage medium may even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
In embodiments of the present invention, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, input method, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, Radio Frequency (RF), etc., or any suitable combination of the preceding.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a separate product, may also be stored in a computer readable storage medium. The storage medium may be a read-only memory, a magnetic or optical disk, or the like.
The above description is only for the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various changes or substitutions within the technical scope of the present invention, and these should be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (10)
1. A method of parsing serialized data, comprising:
responding to a deserialization request for analyzing the serialized text, and acquiring a serialized acquisition function of the serialized text; the serialization acquisition function comprises a mapping relation between a data structure and a data identifier, wherein the data structure is used for defining the data structure of the serialization data corresponding to the data identifier mapped by the data structure;
acquiring a data structure mapped by a data identifier of the serialized data to be subjected to the deserialization operation through the serialization acquisition function; and
according to the acquired data structure, performing deserialization operation on the serialized data to be deserialized;
the method further comprises the following steps: acquiring a configuration file of the serialized text; wherein the configuration file is used for defining data results of the serialized data in the serialized text;
and generating a serialized acquisition function in advance according to the configuration file.
2. The method of parsing serialized data as recited in claim 1, wherein generating the serialized acquisition function in advance from a configuration file comprises:
generating an acquisition function of the serialized texts according to the configuration file; and
and executing serialization operation on the acquisition function to generate a serialized acquisition function.
3. The method of parsing serialized data as recited in claim 2, wherein the generating an acquisition function for the serialized text from the configuration file comprises:
extracting a parent data identifier corresponding to a data structure for defining the serialized data in the serialized text from the configuration file as a function name of an acquisition function;
compiling the configuration file to generate a configuration source code; and
configuring and generating an acquisition function of the serialized text according to the configuration source code and the function name of the acquisition function; the obtaining function comprises a mapping relation between a source code of a data structure used for defining the serialized data in the configuration source code and a source code of a data identifier of the serialized data.
4. The method of parsing serialized data as recited in claim 2, wherein the performing a serialization operation on the get function comprises:
invoking a serialization tool for performing a serialization operation and generating the serialized text; and
and controlling the serialization tool to execute the serialization operation on the acquisition function.
5. An apparatus for parsing serialized data, comprising:
the function acquisition module is used for responding to an anti-serialization request for analyzing the serialized texts and acquiring the serialized acquisition function of the serialized texts; the serialization acquisition function comprises a mapping relation between a data structure and a data identifier, wherein the data structure is used for defining the data structure of the serialization data corresponding to the data identifier mapped by the data structure;
the data structure acquisition module is used for acquiring a data structure mapped by the data identifier of the serialized data to be subjected to the deserialization operation through the serialization acquisition function according to the data identifier of the serialized data to be subjected to the deserialization operation; and
the deserializing operation module is used for performing deserializing operation on the serialized data to be deserialized according to the acquired data structure;
the device further comprises: the configuration file acquisition module is used for acquiring a configuration file of the serialized text before acquiring a serialized acquisition function of the serialized text; wherein the configuration file is used to define a data structure of serialized data in the serialized text,
the apparatus is further configured to: and generating a serialized acquisition function in advance according to the configuration file.
6. The apparatus for parsing serialized data as recited in claim 5, the apparatus further comprising:
the function generating module is used for generating an obtaining function of the serialized texts according to the configuration file; and
and the function serialization module is used for executing serialization operation on the acquisition function to generate a serialized acquisition function.
7. The apparatus for parsing serialized data as recited in claim 6, wherein the function generation module comprises:
the function name definition unit is used for extracting a parent data identifier corresponding to a data structure for defining the serialized data in the serialized text from the configuration file as a function name of the acquisition function;
the source code compiling unit is used for compiling the configuration file to generate a configuration source code; and
the function configuration unit is used for configuring and generating the acquisition function of the serialized text according to the configuration source code and the function name of the acquisition function; the obtaining function comprises a mapping relation between a source code of a data structure used for defining the serialized data in the configuration source code and a source code of a data identifier of the serialized data.
8. The apparatus for parsing serialized data as recited in claim 6, wherein the function serialization module comprises:
the tool calling unit is used for calling a serialization tool for executing the serialization operation and generating the serialization text; and
and the serialization execution unit is used for controlling the serialization tool and executing the serialization operation on the acquisition function.
9. A terminal device for parsing serialized data, the terminal device comprising:
one or more processors;
storage means for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of parsing serialized data as recited in any of claims 1-4.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method of parsing serialized data as claimed in any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711407433.9A CN107992624B (en) | 2017-12-22 | 2017-12-22 | Method, device, storage medium and terminal equipment for analyzing serialized data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711407433.9A CN107992624B (en) | 2017-12-22 | 2017-12-22 | Method, device, storage medium and terminal equipment for analyzing serialized data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107992624A CN107992624A (en) | 2018-05-04 |
| CN107992624B true CN107992624B (en) | 2020-08-04 |
Family
ID=62041565
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711407433.9A Active CN107992624B (en) | 2017-12-22 | 2017-12-22 | Method, device, storage medium and terminal equipment for analyzing serialized data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107992624B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109117209B (en) * | 2018-07-23 | 2021-10-01 | 广州多益网络股份有限公司 | Serialization and deserialization method and device |
| CN109753048B (en) * | 2018-12-12 | 2020-12-29 | 中国铁道科学研究院集团有限公司通信信号研究所 | Automatic test driving engine system of high-speed rail signal equipment |
| CN109699023B (en) * | 2018-12-12 | 2022-02-22 | 中国铁道科学研究院集团有限公司通信信号研究所 | Automatic test data packaging and processing method for high-speed rail signal equipment |
| CN110377289A (en) * | 2019-07-01 | 2019-10-25 | 北京字节跳动网络技术有限公司 | A kind of data analysis method, device, medium and electronic equipment |
| CN110515893B (en) * | 2019-07-26 | 2022-12-09 | 济南浪潮数据技术有限公司 | Data storage method, device, equipment and computer readable storage medium |
| CN110688530B (en) * | 2019-08-19 | 2022-04-26 | 天津开心生活科技有限公司 | Json data processing method and device |
| WO2021097785A1 (en) * | 2019-11-22 | 2021-05-27 | 云图技术有限公司 | Data serialization and data deserialization methods and apparatuses, and computer device |
| CN110990411B (en) * | 2019-12-18 | 2024-02-06 | 上海米哈游网络科技股份有限公司 | Data structure generation method and device, and calling method and device |
| CN111931200B (en) * | 2020-07-13 | 2024-02-23 | 车智互联（北京）科技有限公司 | Data serialization method, mobile terminal and readable storage medium |
| CN112947938B (en) * | 2021-03-23 | 2024-04-30 | 广州趣丸网络科技有限公司 | File conversion method and device, electronic equipment and storage medium |
| CN113746842A (en) * | 2021-09-03 | 2021-12-03 | 唯品会(广州)软件有限公司 | Message sending method based on Protobuf protocol dynamic analysis |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103036961A (en) * | 2012-12-07 | 2013-04-10 | 蓝盾信息安全技术股份有限公司 | Distributed collection and storage method of journal |
| CN103389992A (en) * | 2012-05-09 | 2013-11-13 | 北京百度网讯科技有限公司 | Structured data storage method and device |
| CN104199879A (en) * | 2014-08-21 | 2014-12-10 | 广州华多网络科技有限公司 | Data processing method and device |
| CN105677805A (en) * | 2015-12-31 | 2016-06-15 | 北京奇艺世纪科技有限公司 | Data storing and reading method and device using protobuf |
| CN105760534A (en) * | 2016-03-10 | 2016-07-13 | 上海晶赞科技发展有限公司 | User-defined serializable data structure, hadoop cluster, server and application method thereof |
| CN105787128A (en) * | 2016-03-29 | 2016-07-20 | 四川秘无痕信息安全技术有限责任公司 | Method for recovering Java serialized file data |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8111814B2 (en) * | 2006-03-20 | 2012-02-07 | Microsoft Corporation | Extensible alert types |
-
2017
- 2017-12-22 CN CN201711407433.9A patent/CN107992624B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103389992A (en) * | 2012-05-09 | 2013-11-13 | 北京百度网讯科技有限公司 | Structured data storage method and device |
| CN103036961A (en) * | 2012-12-07 | 2013-04-10 | 蓝盾信息安全技术股份有限公司 | Distributed collection and storage method of journal |
| CN104199879A (en) * | 2014-08-21 | 2014-12-10 | 广州华多网络科技有限公司 | Data processing method and device |
| CN105677805A (en) * | 2015-12-31 | 2016-06-15 | 北京奇艺世纪科技有限公司 | Data storing and reading method and device using protobuf |
| CN105760534A (en) * | 2016-03-10 | 2016-07-13 | 上海晶赞科技发展有限公司 | User-defined serializable data structure, hadoop cluster, server and application method thereof |
| CN105787128A (en) * | 2016-03-29 | 2016-07-20 | 四川秘无痕信息安全技术有限责任公司 | Method for recovering Java serialized file data |
Non-Patent Citations (2)
| Title |
|---|
| Google Protobuf在Linux Socket通讯中的应用;李纪欣 等;《电脑开发与应用》;20130425(第2013年04期);第1-5页 * |
| 基于Protobuf的RPC系统的设计与实现;王斌斌;《中国优秀硕士学位论文全文数据库 信息科技辑》;20170215(第2017年第2期);第I138-769页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107992624A (en) | 2018-05-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107992624B (en) | Method, device, storage medium and terminal equipment for analyzing serialized data | |
| CN107688538B (en) | Script execution method and device and computing equipment | |
| US8156505B2 (en) | Protocol processing including converting messages between SOAP and application specific formats | |
| US20050144137A1 (en) | Protocol processing device and method | |
| CN111737564B (en) | Information query method, device, equipment and medium | |
| EP1071016A2 (en) | Debugger protocol generator | |
| CN111625585B (en) | Access method, device, host and storage medium of hardware acceleration database | |
| CN107526620B (en) | User mode input and output equipment configuration method and device | |
| CN112182036A (en) | Data sending and writing method and device, electronic equipment and readable storage medium | |
| CN109669787B (en) | Data transmission method and device, storage medium and electronic equipment | |
| CN111586097A (en) | Network request processing method, computing device and storage medium | |
| CN111309751A (en) | Big data processing method and device | |
| CN113703862A (en) | Configuration-based interface calling method, device, equipment and storage medium | |
| CN110674205B (en) | Single table query method, device, terminal and readable storage medium | |
| CN114564509A (en) | Serialization method, deserialization method, device, chip and medium | |
| CN111221852A (en) | Mixed query processing method and device based on big data | |
| CN106484375B (en) | Instruction block loading method, soft switch equipment and system | |
| CN111221860A (en) | Mixed query optimization method and device based on big data | |
| CN111221841A (en) | Real-time processing method and device based on big data | |
| CN111984679B (en) | Access method, device, host, system and medium of hardware acceleration database | |
| CN112817922B (en) | Log dynamic printing method and device, computer equipment and storage medium | |
| CN110471708B (en) | Method and device for acquiring configuration items based on reusable components | |
| CN110908619A (en) | Method and device for processing orders | |
| CN116886747A (en) | Front-end-based back-end service function calling method, device, equipment and medium | |
| US11537457B2 (en) | Low latency remoting to accelerators |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |