WO2017059798A1

WO2017059798A1 - Methods, apparatus, and system for serialization and deserialization, and electronic devices

Info

Publication number: WO2017059798A1
Application number: PCT/CN2016/101448
Authority: WO
Inventors: 董石鸣
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2015-10-10
Filing date: 2016-10-08
Publication date: 2017-04-13
Also published as: CN106570018A; CN106570018B

Abstract

A method and apparatus, and electronic device for data object serialization; a method, apparatus, and electronic device for data stream deserialization; and a system for serialization and deserialization. The method for data object serialization comprises: obtaining a metadata description file for a data object to be serialized (S101); on the basis of an attribute operating class name provided by each attribute description in the metadata description file, obtaining a corresponding attribute operating class (S103); on the basis of a reading method provided by each attribute operating class, reading each attribute value in the data object to be serialized (S105); according to the order of the sequence numbers of the attribute descriptions, writing each read attribute value into a resulting data stream for the data object (S107). The described method solves compatibility problems arising between systems upon changes to object attributes. In the method, object metadata and object values are separated, thereby economizing on storage space and transmission traffic during storage and transmission operations. The method also allows for a reduction in system processing overhead during deserialization.

Description

Method, device, system and electronic device for serialization and deserialization

The present application claims priority to Chinese Patent Application No. 201510650, 688, filed on Jan. 10, 2015, entitled Serialization and De-Sequencing, Method, Apparatus, and Electronic Apparatus, the entire contents of In this application.

Technical field

The present invention relates to the field of computer communication technologies, and in particular, to a method for serializing data objects, a device for serializing data objects, and an electronic device; a method for deserializing a data stream, and a pair of data A device for stream deserialization and an electronic device; a serialization and deserialization system.

Background technique

The object serialization technique is a technique for converting object data into another form of persistent data, that is, a process of converting state information of an object into a form that can be stored or transmitted. During serialization, an object writes its current state to a temporary or persistent store. In order for the stored and transmitted data to be able to change back to objects in memory, such conversions must be reversible. The object can be recreated by reading or deserializing the state of the object from the store.

In data transmission and application between networks, it is usually necessary to convert information into data streams for data transmission and application. However, under the existing serialization and deserialization scheme, if the attributes in the object are increased or decreased, the side that performs the deserialization will cause the deserialization to fail if the attributes in the object are not synchronized. problem. When an attribute in an object is added, reduced, or modified, and the original serialized file or data is deserialized, an error or serialization failure occurs. This is the compatibility issue of object serialization.

At present, serialization and deserialization are usually implemented in the prior art using Google's Protobuf (Protocol Buffers) data exchange format. Protobuf is a data exchange format that serializes existing information into binary streams according to different types according to different types, for saving information or data transmission. Upon reading the binary information or receiving the binary information, the serialized binary stream is deserialized to obtain the original information or data. Although Protobuf can reduce the metadata content in the serialization result during serialization, for compatibility, a small amount of metadata (attribute type, attribute number) is retained, and in the case of deserialization, in addition to obtaining a binary stream, It is also necessary to know the type used in serialization and to perform corresponding deserialization according to the type.

It can be seen that in the prior art, since the metadata is retained during serialization, the compression ratio of the serialization is low, and It wastes space and traffic when saving information or data transmission; and when deserializing, the process of obtaining the serialization type used when serializing objects is complicated, increasing processing overhead.

Summary of the invention

The present application provides a method for serializing a data object, a device for serializing a data object, an electronic device, a method for deserializing a data stream, and a device for deserializing a data stream, An electronic device and a serialization and deserialization system to solve the problem of low serialization compression rate in the prior art.

The application provides a method for serializing data objects, and the method for serializing data objects includes:

Obtaining a metadata description file of the data object to be serialized, where the metadata description file includes an attribute description identified by a serial number and an attribute operation class name corresponding to the attribute in the data object;

Obtaining, according to the attribute operation class name provided by each attribute description in the metadata description file, an attribute operation class corresponding to each attribute;

Reading the attribute values corresponding to the respective attribute descriptions in the data objects to be serialized according to the reading methods provided by the attribute operation classes corresponding to the respective attributes;

The read attribute values are written to the result data stream of the data object in the order of the attribute description sequence number, and the result data stream is used as a serialization result of the data object.

Optionally, the attribute description further includes: an attribute type.

Optionally, after the step of reading the attribute value corresponding to each attribute description in the data object to be serialized, the method includes:

Gets the attribute type of the attribute description that cannot be read because the attribute operation class of the corresponding attribute cannot be obtained;

Determining, according to the attribute type, a default attribute operation class corresponding to the attribute type;

According to the default attribute operation class, the corresponding attribute is given a predetermined default value for the attribute description.

Optionally, the metadata description file of the data object to be serialized is:

And searching, according to the name of the data object to be serialized, a metadata description file of the data object to be serialized corresponding to the data object to be serialized and having the largest version number value from the metadata repository.

Optionally, before the step of writing the read attribute value into the result data stream of the data object in the order of the attribute description sequence number, the method includes:

Reading a version number of the metadata description file of the data object to be serialized and writing it to the head of the result data stream of the data object to be serialized.

Optionally, before the step of acquiring the metadata description file of the data object to be serialized, the method further includes:

Receiving a modification operation on an attribute of the data object to be serialized; the modifying operation includes: adding, deleting, or changing an attribute of the data object to be serialized;

Creating an attribute operation class for the newly added attribute of the data object to be serialized based on the modification operation of the data object to be serialized;

A new version of the metadata description file is created according to the modification operation, and the attribute description in the metadata description file is changed according to the modification operation based on the attribute description in the original version metadata description file.

Optionally, the metadata description file is created according to the modifying operation, and the attribute description in the metadata description file is performed according to the attribute description in the original version metadata description file according to the modification operation. After the steps to change, include:

And traversing a metadata description file set for the data object to be serialized in a metadata repository storing the metadata description file according to the name of the data object to be serialized;

Obtaining, in the traversing process, a value having a maximum version number of the metadata description file of the data object to be serialized in the metadata repository;

Adding the value to one, and adding the added value as the version number of the new version of the metadata description file;

The metadata description file in which the version number is recorded is saved in the metadata repository.

Optionally, after the step of reading the attribute value corresponding to each attribute description in the data object to be serialized according to the reading method provided by the attribute operation class corresponding to each attribute, the method includes:

The read attribute values are compressed.

Correspondingly, the present application further provides an apparatus for serializing data objects, where the apparatus for serializing data objects includes: a description file acquisition unit, an attribute operation class acquisition unit, an attribute reading unit, and a serialization write unit. ;

The description file obtaining unit is configured to acquire a metadata description file of the data object to be serialized, where the metadata description file includes an attribute description identified by a serial number and an attribute operation class name corresponding to an attribute in the data object;

The attribute operation class obtaining unit is configured to obtain an attribute operation class name corresponding to each attribute according to the attribute operation class name provided by each attribute description in the metadata description file;

The attribute reading unit is configured to read, according to a reading method provided by the attribute operation class corresponding to each attribute, an attribute value corresponding to each attribute description in the data object to be serialized;

The serialization writing unit is configured to write the read attribute values into the result data stream of the data object in the order of the attribute description sequence number, and the result data stream is used as a serialization result of the data object.

Optionally, the metadata description file of the data object to be serialized acquired by the description file acquiring unit further includes an attribute type.

Optionally, the attribute reading unit further includes: an attribute type reading subunit, a default attribute operation class obtaining subunit, and a default value reading subunit;

The attribute type reading subunit is configured to read an attribute value corresponding to each attribute description in the data object to be serialized, and obtain an attribute description attribute that cannot be read because the attribute operation class of the corresponding attribute cannot be obtained. Types of;

The default attribute operation class acquisition sub-unit is configured to retrieve, according to the attribute type, a default attribute operation class corresponding to the attribute type;

The default value reading subunit is configured to assign a predetermined default value to the corresponding attribute of the attribute description according to the default attribute operation class.

Optionally, the description file acquiring unit is configured to: search, according to the name of the data object to be serialized, the to-be-sequence corresponding to the data object to be serialized and having the largest version number value from the metadata repository. Metadata description file for the data object.

Optionally, the serialized writing unit further includes: a version number writing subunit;

The version number is written into the subunit, and is configured to read the data object to be serialized before the attribute value to be read is written into the result data stream of the data object in the order of the attribute description sequence number. The metadata describes the version number of the file and writes it to the head of the resulting data stream of the data object to be serialized.

Optionally, the device for serializing the data object further includes: a modification operation receiving unit, an operation class creation unit, and a description file creation unit;

The modifying operation receiving unit is configured to receive, before the acquiring the metadata description file of the data object to be serialized, a modification operation on the attribute of the data object to be serialized; the modifying operation includes: Adding, deleting, or changing the attributes of the serialized data object;

The operation class creation unit is configured to create an attribute operation class for the newly added attribute of the data object to be serialized based on the modification operation of the data object to be serialized;

a description file creation unit, configured to create a new version of the metadata description file according to the modification operation, where the attribute description in the metadata description file is based on the attribute description in the original version metadata description file, according to the modification operation The change was made.

Optionally, the device for serializing the data object further includes: a traversal unit, a value obtaining unit, a version number generating unit, and a saving unit;

The traversing unit is configured to create a new version of the metadata description file according to the modification operation, where the attribute description in the metadata description file is based on the attribute description in the original version metadata description file, according to the modification After the operation is changed, the metadata description file set for the data object to be serialized is traversed in the metadata repository storing the metadata description file according to the name of the data object to be serialized;

The value obtaining unit is configured to acquire, during the traversal process, a maximum value of a version number of the metadata description file of the data object to be serialized in the metadata repository;

The version number generating unit is configured to add the value by one, and use the added value as the version number of the metadata version file of the new version;

The saving unit is configured to save the metadata description file with the version number recorded in the metadata repository.

Optionally, the device for serializing the data object further includes: an attribute value compression unit;

The attribute value compression unit is configured to read the attribute value after the attribute value corresponding to each attribute description in the data object to be serialized according to the reading method provided by the attribute operation class corresponding to each attribute Compress.

In addition, an embodiment of the present application further provides an electronic device, where the electronic device includes:

monitor;

processor;

a memory for storing a serialization program, the program, when being read and executed by the processor, performing an operation of: acquiring a metadata description file of a data object to be serialized, where the metadata description file includes a serial number identifier The attribute description and the attribute operation class name corresponding to the attribute in the data object; the attribute operation class name provided according to each attribute description in the metadata description file, and the attribute operation class corresponding to each attribute are obtained; according to the corresponding attribute The reading method provided by the attribute operation class reads the attribute value corresponding to each attribute description in the data object to be serialized; and writes the read attribute value into the result data of the data object in the order of the attribute description serial number. Flow, the resulting data stream as a serialization result of the data object.

In addition, the present application also provides a method for deserializing a data stream, and the method for deserializing the data stream includes:

Obtaining a result data stream;

Obtaining a metadata description file of the local data object, where the metadata description file includes an attribute description identified by a serial number and an attribute operation class name corresponding to an attribute in the data object;

According to the attribute operation class name provided by each attribute description in the metadata description file, obtain corresponding to each The attribute action class of the attribute;

According to the reading method provided by the attribute operation class corresponding to each attribute, the attribute values in the result data stream are sequentially read in the order of the attribute description sequence number;

The read attribute values are written into the attributes of the local data object corresponding to the respective attribute descriptions according to the write method provided by the attribute operation class described by each attribute.

Optionally, the attribute description further includes: an attribute type.

Optionally, in the step of reading the attribute value in the result data stream in the order of the attribute description sequence number, the method further includes:

For the attribute description of the attribute operation class, the default attribute operation class corresponding to the attribute type is obtained according to the attribute type described by the attribute;

Obtaining the reading method provided by the default attribute operation class, and reading the data in the result data stream according to the sequence of the attribute description sequence number;

The data in the result data stream read is discarded.

Optionally, the obtaining the metadata description file of the local data object includes:

Obtaining the name of the local data object;

Parsing the result data stream to obtain a first value in the result data stream;

And traversing a metadata description file set for the local data object in a metadata repository storing the metadata description file according to the name of the local data object;

In the traversal process, a metadata description file of the local data object having the same name as the local data object and having the same version number as the first value in the result data stream is obtained.

Optionally, after the step of sequentially reading the attribute values in the result data stream according to the sequence of the attribute description sequence according to the reading method provided by the attribute operation class corresponding to each attribute, the method includes:

The attribute values read from the resulting data stream are decompressed.

Correspondingly, the present application further provides an apparatus for deserializing a data stream, where the apparatus for deserializing the data stream includes: a data stream obtaining unit, a description file acquiring unit, an attribute operation class acquiring unit, and an attribute value reading. The unit and the attribute value are written into the unit;

The data stream obtaining unit is configured to obtain a result data stream;

The description file obtaining unit is configured to acquire a metadata description file of the local data object, where the metadata description file includes an attribute description identified by a serial number and an attribute operation class name corresponding to an attribute in the data object;

The attribute value reading unit is configured to sequentially read the attribute values in the result data stream according to the sequence of the attribute description sequence according to the reading method provided by the attribute operation class corresponding to each attribute;

The attribute value writing unit is configured to write the read attribute value into the attribute of the local data object corresponding to each attribute description according to the writing method provided by the attribute operation class described by each attribute .

Optionally, the description file obtaining unit, the obtained metadata description file of the local data object further includes an attribute type.

Optionally, the attribute value reading unit includes: an attribute type obtaining subunit, a data reading subunit, and a discarding subunit;

The attribute type obtaining subunit is configured to sequentially read the attribute values in the result data stream according to the sequence of the attribute description serial number in the writing method provided by the attribute operation class according to each attribute description. For the attribute description of the attribute operation class, the default attribute operation class corresponding to the attribute type is obtained according to the attribute type described by the attribute;

The data reading subunit is configured to acquire a reading method provided by the default attribute operation class, and read data in the result data stream according to an order of the attribute description sequence number;

The discarding a single sub-unit is configured to discard the data in the read result data stream.

Optionally, the description file obtaining unit includes: a local data object name obtaining subunit, a result data stream parsing subunit, a description file traversing subunit, and a description file acquiring subunit;

The local data object name obtaining subunit is configured to acquire a name of the local data object;

The result data stream parsing subunit is configured to parse the result data stream to obtain a first value in the result data stream;

The description file traversing subunits for traversing a metadata description file set for the local data object in a metadata repository in which the metadata description file is stored according to the name of the local data object;

The description file obtaining subunit, configured to acquire metadata of the local data object having the same name as the local data object and having the same version number as the first value in the result data stream during the traversal process Description file.

Optionally, the device for deserializing the data stream further includes: an attribute value decompression unit;

The attribute value decompressing unit is configured to sequentially read the attribute values in the result data stream according to the order of the attribute description sequence in the reading method provided by the attribute operation class corresponding to each attribute, and then From the stated As a result, the attribute values read out in the data stream are decompressed.

monitor;

processor;

a memory for storing a deserialization program, the program, when being read and executed by the processor, performing an operation of: acquiring a result data stream, acquiring a metadata description file of the local data object, the metadata description The file includes an attribute description identified by a sequence number and an attribute operation class name corresponding to the attribute in the data object, and an attribute operation class name corresponding to each attribute description in the metadata description file is obtained, and an attribute operation class corresponding to each attribute is obtained, According to the reading method provided by the attribute operation class corresponding to each attribute, the attribute values in the result data stream are sequentially read in the order of the attribute description sequence number, and the writing method provided by the attribute operation class according to each attribute description is The read attribute values are written into the attributes of the local data object corresponding to the respective attribute descriptions.

In addition, the present application also provides a serialization and deserialization system, the serialization and deserialization system comprising:

A device for serializing a data object according to any of the preceding claims; and the device for deserializing the data stream according to any of the preceding claims.

Compared with the prior art, the method and apparatus for serializing a data object provided by the present application obtains a metadata description file of a data object to be serialized, and the metadata description file includes an attribute description identified by a serial number, and each The attribute description includes the attribute operation class name of the corresponding data object attribute; according to the attribute operation class name, the attribute operation class of the corresponding attribute can be obtained, and the reading method provided by the attribute operation class corresponding to each attribute may be Reading the attribute values corresponding to the respective attribute descriptions in the data objects to be serialized, and writing the read attribute values into the result data stream of the data object in the order of the attribute description sequence numbers, where the result data stream is used as the The serialization result of the data object.

The method and apparatus for deserializing a data stream provided by the present application obtains a metadata description file of the local data object by acquiring a result data stream, where the metadata description file includes an attribute description identified by a serial number and corresponding to the description The attribute operation class name of the attribute in the data object, according to the attribute operation class name provided by each attribute description in the metadata description file, obtaining the attribute operation class corresponding to each attribute, and reading according to the attribute operation class corresponding to each attribute The method sequentially reads the attribute values in the result data stream according to the sequence of the attribute description sequence numbers, and writes the read attribute values into each attribute according to the writing method provided by the attribute operation class described by each attribute. Describe the properties of the corresponding local data object.

The above technical solution solves the compatibility problem of serialization and deserialization between systems after changing the attributes of the object, It also saves storage space and transmission traffic when saving information and data transmission by completely splitting the metadata of the object and the value of the object, and reduces the processing overhead of the system when deserializing.

DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings to be used in the embodiments or the prior art description will be briefly described below. Obviously, the drawings in the following description are only These are some of the embodiments described in this application, and other figures can be obtained from those of ordinary skill in the art in view of these drawings.

1 shows a flow chart of a method of serializing data objects provided in accordance with an embodiment of the present application;

2 shows a flowchart of creating a metadata description file provided in accordance with an embodiment of the present application;

FIG. 3 illustrates a flow chart of saving a metadata description file provided in accordance with an embodiment of the present application; FIG.

4 shows a flow chart of a method of reading default values at serialization provided in accordance with an embodiment of the present application.

FIG. 5 shows a schematic diagram of an apparatus for serializing data objects provided in accordance with an embodiment of the present application; FIG.

FIG. 6 shows a schematic diagram of an electronic device provided in accordance with an embodiment of the present application; FIG.

7 shows a flow diagram of a method of deserializing a data stream, provided in accordance with an embodiment of the present application;

FIG. 8 is a flowchart showing a metadata description file for acquiring the local data object according to an embodiment of the present application;

FIG. 9 is a flowchart of a method for discarding a default value at the time of deserialization according to an embodiment of the present application; FIG.

10 shows a schematic diagram of an apparatus for deserializing a data stream, provided in accordance with an embodiment of the present application;

FIG. 11 shows a schematic diagram of an electronic device provided in accordance with an embodiment of the present application; FIG.

FIG. 12 shows a schematic diagram of a serialization and deserialization system provided in accordance with an embodiment of the present application.

detailed description

The above objects, features and advantages of the present application will be more clearly understood from the following description of the appended claims. It should be noted that the embodiments in the present application and the features in the embodiments may be combined with each other without conflict.

Numerous specific details are set forth in the description below in order to provide a thorough understanding of the application. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar promotion without departing from the scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed below.

Embodiments of the present application provide a method for serializing a data object, a device for serializing a data object, a method for deserializing a data stream, a device for deserializing a data stream, and a device A serialization and deserialization system, the embodiment of the present application simultaneously provides an electronic device. Detailed description will be made one by one in the following embodiments.

The technical solution of the present application is directed to data transmission or storage between systems in a distributed system environment, but the application field thereof is not limited to a distributed system, and in an existing system or application, It is possible to perform serialization and deserialization methods, and the methods provided by the present application can be used, and corresponding beneficial effects can be obtained.

Before describing the specific steps of the embodiment in detail, the serialization method and the deserialization method involved in the technical solution are briefly described.

The object serialization technique is a technique for converting object data into another form of persistent data, that is, a process of converting state information of an object into a form that can be stored or transmitted. During serialization, an object writes its current state to a temporary or persistent store, which can be recreated by reading or deserializing the state of the object from the store. In order for the stored and transmitted data to be able to change back to objects in memory, such conversions must be reversible. Serialization methods and deserialization methods are widely used in data transmission and application between networks. It is usually necessary to convert information into data streams for data transmission and application. However, when data is transferred and stored, if the attributes in the object are increased or decreased, the side that performs deserialization will not cause the deserialization to fail if the attributes in the object are not synchronized. When an attribute in an object is added, reduced, or modified, and the original serialized file or data is deserialized, an error or serialization failure occurs. This is the compatibility issue of object serialization. Usually all fields of an object are serialized, which means that the data is represented as serialized data. Similarly, deserialization extracts data from the serialized representation and sets the object state directly.

Embodiments of the present application provide a method of serializing data objects. The method for serializing data objects is as follows:

Please refer to FIG. 1 , which shows a flowchart of a method for serializing data objects according to an embodiment of the present application.

The method for serializing data objects includes:

Step S101: Obtain a metadata description file of a data object to be serialized, where the metadata description file includes an attribute description identified by a sequence number and an attribute operation class name corresponding to an attribute in the data object.

In this embodiment, the obtaining a metadata description file of the data object to be serialized may be implemented by searching for a corresponding sequence from the metadata repository according to the name of the data object to be serialized. Chemical And a metadata description file of the data object to be serialized with the largest version number value, the metadata description file includes an attribute description identified by a sequence number and an attribute operation class name corresponding to the attribute in the data object.

It should be noted that the metadata warehouse is a database for storing metadata and metadata related information (such as a metadata description file), and the metadata warehouse may generally adopt a relational database.

In this embodiment, the method of serializing the data object is to understand the structural composition of the data according to the description semantics of the metadata description file. It can be understood that, when the metadata description file of the data object to be serialized is obtained in step S101, the to-serialization corresponding to the data object to be serialized and having the largest version number value is searched from the metadata repository. The metadata description file of the data object, that is, the metadata version description file obtained in this step is the latest version.

Specifically, the metadata description file includes a plurality of attribute descriptions corresponding to attributes in the data object, each attribute description is identified by a sequence number, and includes an attribute operation class name corresponding to the attribute in the data object.

In order to satisfy the compatibility problem of the attributes in the metadata description file and the data object in the serialization, the technical solution of the embodiment provides a preferred implementation manner. In a preferred mode, each attribute description in the metadata description file is described. Also includes: the attribute type described for the attribute of the strip.

For example, the metadata description file can be as follows:

1:com.taobao.galaxy.io.UserIdFieldDef:VARINT

2: com.taobao.galaxy.io.AgeFieldDef:VARINT

The numbers "1" and "2" are the sequence numbers of each attribute description, the UserIdFieldDef is the attribute operation class name corresponding to the attribute UserId in the data object; the AgeFieldDef is the attribute operation class name corresponding to the attribute Age in the data object; VARINT is the attribute type described by this attribute.

It should be noted that the metadata description file is data describing other data, or structural data. Specifically, the metadata description file is data describing an object such as an information resource or data. Once the metadata description file is created, it can be shared. The development and utilization environment of metadata description files is often a changing distributed environment. Since the metadata description file is also data, it can be stored and retrieved in the metadata repository by a similar data method, which is convenient for sharing and use in a distributed environment.

In the following, for the method for serializing data objects, the present application provides a specific example to explain the method for serializing data objects:

The data object to be serialized is (the attribute value is not shown):

Class User{

Int userId;

String name;

Int age;

}

Defining, according to the name User of the data object to be serialized, a metadata description file of the data object to be serialized corresponding to the data object User to be serialized and having the highest version number value, for example, from the metadata repository, for example : The metadata description file of the data object to be serialized is:

1:com.taobao.test.io.UserIdFieldDef:VARINT

2: com.taobao.test.io.NameFieldDef:VARCHAR

3: com.taobao.test.io.PhoneFieldDef:VARINT

4: com.taobao.test.io.AgeFieldDef:VARINT

The data object described in the metadata description file described above does not exactly match the example User of the aforementioned data object. This is because the metadata description file is the latest version and is determined according to the upgraded version of User.

It should be noted that the data object is a mapping of entities such as people, things, objects, etc. existing in the objective world in computer logic. For example, when the object is a book, the attributes thereof include a book number, an author, a publisher, a comment, and the like, and after serializing the book, a binary data stream corresponding to the book is generated, the binary data stream having attributes with the book. Corresponding attribute values, thereby representing entities such as people, things, objects, etc. that exist in the objective world through computer language. The data object can be any entity, thing, object, etc. that exists in the objective world. The data object in this example is the user, that is, User. The object includes at least one attribute. The name of the attribute is called the attribute name. The value corresponding to the attribute is called the attribute value. The attribute value can be an integer type such as an integer type, a character type, a floating point type, or a string type, for example, an example of the above data object. The attribute userId is an integer, the attribute name is a string type, and the attribute age is an integer.

Before the step S101 obtains the metadata description file of the data object to be serialized, the metadata description file needs to be defined or created, and specifically includes steps S100-1 to S100-3, which will be further described below with reference to FIG. 2 .

Please refer to FIG. 2, which shows a flowchart of creating a metadata description file according to an embodiment of the present application.

Step S100-1: Receive a modification operation on an attribute of the data object to be serialized; the modifying operation includes: adding, deleting, or changing an attribute of the data object to be serialized.

The receiving a modification operation on an attribute of the data object to be serialized includes: receiving an addition, deletion, or change of an attribute in the data object to be serialized. For example, add the attribute address of type string to the data object User, delete the age attribute of the data object User, or modify the name of the userId attribute in the data object to id.

Step S100-2: Create an attribute operation class for the newly added attribute of the data object to be serialized based on the modification operation of the data object to be serialized.

This step creates an attribute operation class for the newly added attribute according to the modification operation of the attribute of the data object to be serialized received in step S100-1. For example, if the attribute address of type string is added in step S100-1, an attribute operation class named AddressFieldDef is created for the attribute address. The AddressFieldDef class encapsulates the getter and setter methods.

It should be noted that, if the modification operation is to delete an attribute, the attribute operation class of the corresponding attribute may not be deleted; if the modification operation is to modify the attribute, the original attribute is deleted, and the modified attribute is created, and the modified attribute is modified. The properties of the corresponding property action class are created.

Step S100-3: Create a new version of the metadata description file according to the modification operation, and the attribute description in the metadata description file is changed according to the modification operation based on the attribute description in the original version metadata description file.

In this embodiment, the new version of the metadata description file is created according to the modification operation, and the attribute description in the metadata description file is performed according to the modification operation on the basis of the attribute description in the original version metadata description file. The change may be implemented by acquiring the attribute of the data object to be serialized after receiving the modification operation, generating the sequence number of the attribute description in the order defined by the attribute in the data object, and acquiring the data corresponding to the serialization to be serialized. The attribute of the attribute in the object manipulates the class name, and obtains the attribute type of the attribute in the data object to be serialized, and writes the attribute operation class name and the attribute type into each attribute description in sequence according to the order defined by the attribute in the data object. In the middle, all the generated attribute descriptions are used as the metadata version file of the new version.

Creating a new version of the metadata description file according to the modification operation, and the attribute description in the metadata description file is changed according to the attribute description in the original version metadata description file, according to the modification operation, The new version of the metadata description file is distinguished from the old version of the metadata description file, and can be shared and used in a distributed environment. The technical solution of this embodiment provides a preferred embodiment. In a preferred manner, Steps S100-4 to S100-7 are included, which will be further described below in conjunction with FIG. 3.

Please refer to FIG. 3, which shows a flowchart of saving a metadata description file according to an embodiment of the present application.

Step S100-4, traversing a metadata description file set for the data object to be serialized in a metadata repository in which the metadata description file is stored according to the name of the data object to be serialized.

For example, the name of the data object to be serialized is User, and the metadata description file set for the data object User is traversed in the metadata repository in which the metadata description file is stored.

Step S100-5, in the traversal process, acquiring the data object to be serialized in the metadata repository The metadata describes the version number of the file with the largest value.

For example, in the traversal process, it is found that there are six metadata description files set for the data object User in the metadata repository, and the version number is 1-6, and the value obtained in this step S100-5 is 6.

In step S100-6, the value is incremented by one, and the added value is used as the version number of the metadata version file of the new version.

In this embodiment, the added value is mapped to the new version of the metadata description file created in step S100-3.

Step S100-7, storing the metadata description file with the version number recorded in the metadata repository.

It should be noted that the seven steps of step S100-1 to step S100-7 may be created in advance, or may be configured on site when performing the method of serializing data objects, which is not limited.

Step S103: Obtain an attribute operation class name corresponding to each attribute description in the metadata description file, and obtain an attribute operation class corresponding to each attribute.

In this embodiment, the attribute operation class name provided by each attribute description in the metadata description file is obtained, and the attribute operation class corresponding to each attribute is obtained, which may be implemented in the following manner: according to the metadata description file Each attribute describes the provided attribute action class name, retrieves and retrieves the same attribute action class as the attribute action class name provided by each attribute description.

The processing of this step will be further described below using the specific example of the above method for serializing data objects.

The data object to be serialized is (the attribute value is not shown):

Class User{

Int userId;

String name;

Int age;

}

The metadata description file of the data object to be serialized is:

1:com.taobao.test.io.UserIdFieldDef:VARINT

2: com.taobao.test.io.NameFieldDef:VARINT

3: com.taobao.test.io.PhoneFieldDef:VARINT

4: com.taobao.test.io.AgeFieldDef:VARINT

Obtaining the attribute operation provided by each attribute description from the metadata description file of the data object to be serialized The class name, including UserIdFieldDef, NameFieldDef, PhoneFieldDef, and AgeFieldDef, retrieves and retrieves the same attribute operation class as the attribute operation class name provided by each attribute description. In this example, the obtained attribute operation class includes: UserIdFieldDef, NameFieldDef, and AgeFieldDef. .

Step S105: Read the attribute values corresponding to the respective attribute descriptions in the data objects to be serialized according to the reading method provided by the attribute operation class corresponding to each attribute.

In this embodiment, according to the step S103, the reading operation method provided by the attribute operation class of each attribute description in the attribute operation class corresponding to each attribute is obtained, and the data object to be serialized is respectively corresponding to each attribute description. Property value.

In a specific implementation, the getter method encapsulated with the same attribute operation class name provided by each attribute description is called, and the attribute value of the corresponding attribute of the data object to be serialized is read.

In order to make the attribute value in the read data object to be serialized, the storage space is saved when the result data stream of the data object is written or the data transmission amount is reduced during the transmission, the technical solution of the embodiment provides a In a preferred embodiment, the read attribute values are compressed prior to writing the result data stream of the data object.

In the specific implementation, the attribute value can be compressed by using out.writeVarint(), or by using more popular LZ and series variants (LZW, LZR, LZX) compression algorithms.

It should be noted that, in step S103, when the attribute operation class name provided by each attribute description in the metadata description file is obtained, and the attribute operation class corresponding to each attribute is obtained, the following two cases occur:

In the first case, the attribute operation class with the same number of the attribute operation class name provided by each attribute description in the metadata description file can be obtained in the code library, and the data object and metadata to be serialized are described. The version of the description file matches, that is, the data object to be serialized is compatible with the metadata description file version, so compatibility problems do not occur.

It should be noted that, in the first case, it is possible that the number and name of the attribute operation classes in the code base are the same as the attribute operation classes described in the respective attribute descriptions in the metadata description file; The number of attribute operation classes in the code library is greater than the attribute operation class described by each attribute description in the metadata description file, but the code library includes all attribute operation classes described by each attribute description in the metadata description file, At the time of serialization, the attribute operation class corresponding to the attribute operation class name in the metadata description file is loaded according to the metadata description file, and the attribute operation class not used in the code library is not used, so A compatibility issue will occur.

In the second case, only the part of the code base and the description of each attribute in the metadata description file can be obtained in the code base. The attribute operation class with the same attribute operation class name is provided, indicating that the data object to be serialized does not match the version of the metadata description file, that is, the data object is a data object generated under the old version, and the metadata description file is the latest. Version, compatibility issues occur during serialization.

In order to solve the problem of the compatibility of the metadata description file and the data object during the serialization, the technical solution of the present embodiment provides a preferred embodiment. In a preferred manner, the method specifically includes the steps S106-1 to S106-3. Figure 4 is further illustrated.

Please refer to FIG. 4, which shows a flowchart of a method for reading default values at the time of serialization according to an embodiment of the present application.

Step S106-1: Acquire an attribute type of the attribute description that cannot be read because the attribute operation class of the corresponding attribute cannot be obtained.

The obtaining the attribute type of the attribute description that cannot be read because the attribute operation class of the corresponding attribute cannot be obtained may be implemented as follows: from the metadata description file, determining that the attribute value cannot be read because the corresponding attribute operation class cannot be obtained The attribute description, that is, the attribute operation class described by the attribute description does not exist, and the attribute type described by the attribute is obtained.

It should be noted that the attribute value cannot be read because the corresponding attribute operation class cannot be obtained, because the code library is not updated or other reasons, and the attribute operation class is not stored. Since the code base is generally stored locally on various terminals, it is likely that it is not updated in time according to the latest version; the metadata description file used is the latest update version.

Step S106-2, according to the attribute type, retrieve a default attribute operation class corresponding to the attribute type.

In this embodiment, the attribute type includes: an integer type, a character type, a floating point type, a string type, and the like.

For example, the obtained property type is VARINT, and the default property operation class of the type DefaultVarintFieldDef is retrieved.

Step S106-3, according to the default attribute operation class, assign a predetermined default value to the attribute corresponding to the attribute description.

Loading the retrieved default attribute operation class, calling the getter method encapsulated in the default attribute operation class, and reading the default value given in advance.

The predetermined default value may adopt different default values according to different attribute types. For example, the default value of the integer type is 0.

The data objects to be serialized are:

Class User{

Int userId;

String name;

Int age;

}

The specific attribute value of the data object is stored in the corresponding database table, for example, a specific data object User is:

userIduserId	nameName	ageAge
11	aaaAaa	2020

It should be noted that the User table given above is only schematic. In a specific implementation, the attribute data in the object User is usually more. In this example, only a set of data is given to illustrate the processing of this step. .

The metadata description file of the obtained data object to be serialized is:

1:com.taobao.test.io.UserIdFieldDef:VARINT

2: com.taobao.test.io.NameFieldDef: VARCHART

3: com.taobao.test.io.PhoneFieldDef:VARINT

4: com.taobao.test.io.AgeFieldDef:VARINT

In step S103, the obtained attribute operation class includes: UserIdFieldDef, NameFieldDef, and AgeFieldDef, wherein the attribute type in the third attribute description is obtained, because the attribute operation class with the same name as PhoneFieldDef is not acquired according to the third attribute description, that is, Get the VARINT type, retrieve the default property operation class DefaultVarintFieldDef of the corresponding type VARINT by obtaining the VARINT type, obtain the property operation class DefaultVarintFieldDef, and execute the getter method in UserIdFieldDef according to the sequence number of the attribute description in the metadata description file. Read the attribute value of userId, execute the getter method in NameFieldDef to read the attribute value aaa of name, execute the getter method in DefaultVarintFieldDef to read the default value of 0, and execute the getter method in AgeFieldDef to read the attribute value 20 of age.

Take UserIdFieldDef and DefaultVarintFieldDef as examples. The detailed code example is as follows: UserIdFieldDef:

Public class UserIdFieldDef<Item>implements FieldDef<T>{

Public void setter(User user,CodedInputStream in){

user.setUserId(in.readVarint());

}

Public void getter(User user,CodedOutputStream out){

out.writeVarint(user.getUserId());

}

DefaultVarintFieldDef:

Public class DefaultVarintFieldDef<Object>implements FieldDef<T>{

Public void setter(Object o,CodedInputStream in){

in.readVarint();

}

Public void getter(Object o,CodedOutputStream out){

out.writeVarint(0);

}

When executing different attribute operation classes, you only need to modify the corresponding parameters, which will not be described here.

Step S107, the read attribute values are written into the result data stream of the data object in the order of the attribute description sequence number, and the result data stream is used as the serialization result of the data object.

In order to make the result data stream obtain the metadata description file used in the serialization when the deserialization is performed later, the compatibility problem is ensured, and the obtained attribute value is written into the result data of the data object in the order of the attribute description sequence number. Before the flow steps, include:

In a specific implementation, the version number of the metadata description file of the data object to be serialized is first obtained, and the value of the version number is written into the position of the first byte of the result data stream, and the sequence is written. The pointers are correspondingly moved backward, and the attribute values read in step S105 are written into the result data stream of the data object in the order of the attribute description sequence numbers.

It should be noted that the serialized result data stream may be binary data, but is not limited to binary data.

A method for serializing a data object provided by the embodiment of the present application, by acquiring a metadata description file, serializing the data object according to the metadata description file, so that the generated result data stream does not include an addition except the data. Data saves storage space and transmission traffic when saving information and data transmission, and can reduce system processing overhead when deserializing.

In the above embodiment, a method for serializing a data object is provided. Corresponding to the above method for serializing a data object, the present application further provides an apparatus for serializing a data object. Since the embodiment of the device is substantially similar to the embodiment of the method, the description is relatively simple, and the relevant portions can be referred to the description of the method embodiment. The device embodiments described below are merely illustrative. The device for serializing the data object is implemented as follows:

Please refer to FIG. 5, which shows a schematic diagram of an apparatus for serializing data objects according to an embodiment of the present application.

The device for serializing the data object includes: a description file obtaining unit 501, an attribute operation class obtaining unit 503, an attribute reading unit 505, and a serialization writing unit 507;

The description file obtaining unit 501 is configured to acquire a metadata description file of a data object to be serialized, where the metadata description file includes an attribute description identified by a serial number and an attribute operation class name corresponding to an attribute in the data object;

The attribute operation class obtaining unit 503 is configured to obtain an attribute operation class name corresponding to each attribute according to each attribute description in the metadata description file, and obtain an attribute operation class corresponding to each attribute;

The attribute reading unit 505 is configured to read, according to a reading method provided by the attribute operation class corresponding to each attribute, an attribute value corresponding to each attribute description in the data object to be serialized;

The serialization writing unit 507 is configured to write the read attribute values into the result data stream of the data object in the order of the attribute description sequence number, and the result data stream is used as a serialization result of the data object.

Optionally, the metadata description file of the data object to be serialized acquired by the description file acquiring unit 501 further includes an attribute type.

Optionally, the attribute reading unit 505 further includes: an attribute type reading subunit, a default attribute operation class obtaining subunit, and a default value reading subunit;

Optionally, the description file obtaining unit 501 is specifically configured to use, according to the name of the data object to be serialized, Finding, from the metadata repository, a metadata description file of the data object to be serialized corresponding to the data object to be serialized and having the highest version number value.

Optionally, the serialization writing unit 507 further includes: a version number writing subunit;

a description file creation unit, configured to create a new version of the metadata description file according to the modification operation, where the attribute description in the metadata description file is based on the attribute description in the original version metadata description file, according to the modification operation Changed

The attribute value compression unit is configured to read according to a reading method provided by an attribute operation class corresponding to each attribute After the attribute values corresponding to the respective attribute descriptions in the data objects to be serialized are respectively, the read attribute values are compressed.

In the above embodiments, a method for serializing data objects and a device for serializing data objects are provided. In addition, the present application further provides an electronic device; the electronic device implementation is as follows:

Please refer to FIG. 6 , which shows a schematic diagram of an electronic device provided in accordance with an embodiment of the present application.

The electronic device includes: a display 601; a processor 603; a memory 605;

The memory 605 is configured to store a serialization program, where the program, when being read and executed by the processor, performs an operation of: acquiring a metadata description file of a data object to be serialized, where the metadata description file includes The attribute operation class identified by the serial number and the attribute operation class name corresponding to the attribute in the data object; the attribute operation class name provided according to each attribute description in the metadata description file, and the attribute operation class corresponding to each attribute are obtained; The reading method provided by the attribute operation class of each attribute reads the attribute value corresponding to each attribute description in the data object to be serialized; and writes the read attribute value into the data object according to the order of the attribute description serial number The resulting data stream, the resulting data stream being the serialized result of the data object.

In the above embodiments, a method for serializing a data object, a device for serializing a data object, and an electronic device are provided. Further, the present application further provides a deserialization of the data stream. Method; the method for deserializing the data stream is implemented as follows:

Please refer to FIG. 7, which illustrates a flow diagram of a method of deserializing a data stream, in accordance with an embodiment of the present application.

The method for deserializing a data stream includes:

Step S701, obtaining a result data stream.

In this embodiment, the obtaining the result data stream includes: acquiring the result data stream by network transmission or obtaining the compressed result data stream from the local.

It should be noted that the result data stream may be binary data, but is not limited to binary data.

Step S703: Obtain a metadata description file of the local data object, where the metadata description file includes an attribute description identified by a sequence number and an attribute operation class name corresponding to an attribute in the data object.

In this embodiment, the acquiring a metadata description file of the local data object, where the metadata description file includes an attribute description identified by a serial number and an attribute operation class name corresponding to an attribute in the data object, specifically including steps S703-1 to S703-4, which will be further described below in conjunction with FIG.

Please refer to FIG. 8 , which shows a flowchart of acquiring a metadata description file of the local data object according to an embodiment of the present application.

Step S703-1, acquiring a name of the local data object.

For example, if the name of the local data object is User, the name of the local data object obtained is User.

Step S703-2, parsing the result data stream to obtain a first value in the result data stream.

Parsing the result data stream to obtain the first byte in the result data stream.

For example, the value obtained is 6.

Step S703-3, traversing the metadata description file set for the local data object in a metadata repository in which the metadata description file is stored according to the name of the local data object.

For example, according to the local data object User, a metadata description file set for the local data object User is traversed in a metadata repository in which a metadata description file is stored.

Step S703-4, in the traversal process, acquiring a metadata description file of the local data object having the same name as the local data object and having the same version number as the first value in the result data stream.

For example, in the traversal process, if there are six metadata description files set for the data object User in the metadata repository, the version number is 1-6, and the version number is obtained in step S603-4. 6 metadata description file.

In order to satisfy the compatibility problem between the metadata description file in the deserialization and the attributes in the local data object, the technical solution of the embodiment provides a preferred embodiment. In a preferred manner, each of the metadata description files The attribute description also includes: the attribute type corresponding to the attribute description.

For the method for deserializing the data stream, the present application provides a specific example to facilitate the method for deserializing the data stream:

The local data object to be written to the attribute value is:

Class User{

Int userId;

String name;

Int phone;

}

And traversing, according to the local data object User, a metadata description file set for the local data object User in a metadata repository storing a metadata description file, if the first value in the result data stream is 6, Then get the metadata description file with version number 6. For example, the metadata description file of the obtained local data object is:

1:com.taobao.test.io.UserIdFieldDef:VARINT

2: com.taobao.test.io.NameFieldDef: VARCHART

3: com.taobao.test.io.PhoneFieldDef:VARINT

4: com.taobao.test.io.AgeFieldDef:VARINT

It should be noted that, since the local data object is determined according to the value of the header of the write result data stream when the metadata description file is obtained, the metadata used during serialization and deserialization is used. The description files are the same, ie the metadata description files are compatible.

Step S705: Acquire an attribute operation class name provided by each attribute description in the metadata description file, and obtain an attribute operation class corresponding to each attribute.

The specific example of the above method for deserializing the data stream is still used below, and the processing of this step is further explained.

The local data object to be written to the attribute value is:

Class User{

Int userId;

String name;

Int phone;

}

The metadata description file of the local data object is:

1:com.taobao.test.io.UserIdFieldDef:VARINT

2: com.taobao.test.io.NameFieldDef:VARINT

3: com.taobao.test.io.PhoneFieldDef:VARINT

4: com.taobao.test.io.AgeFieldDef:VARINT

From the metadata description file of the local data object, obtain the attribute operation class name provided by each attribute description, including UserIdFieldDef, NameFieldDef, PhoneFieldDef, and AgeFieldDef, and retrieve and obtain the same attribute operation as the attribute operation class name provided by each attribute description. Class, in this example, the obtained property action classes include: UserIdFieldDef, NameFieldDef, and PhoneFieldDef.

Step S707, according to the reading method provided by the attribute operation class corresponding to each attribute, sequentially reading the attribute values in the result data stream in the order of the attribute description sequence number.

In this embodiment, according to the step S705, the reading operation method provided by the attribute operation class of each attribute description in the attribute operation class corresponding to each attribute is obtained, and the attribute value in the result data stream is sequentially described according to the attribute description number. The order is read.

In order to ensure the integrity of the data, the technical solution of the embodiment provides a preferred embodiment. In a preferred manner, after the attribute value is read from the result data stream, the result data stream is read from the result data stream. The extracted attribute values are decompressed.

In a specific implementation, the in.readVarint() method of the attribute operation class described by each attribute is called, the attribute value of the corresponding attribute of the serialized data object is read, and the read attribute value is decompressed.

It should be noted that, in step S705, when the attribute operation class name provided by each attribute description in the metadata description file is obtained, and the attribute operation class corresponding to each attribute is obtained, the following two cases occur:

In the first case, the attribute operation class with the same number of the attribute operation class name provided by each attribute description in the metadata description file can be obtained in the code library, and the local data object and the metadata description file are described. The version matches, ie the local data object is compatible with the metadata description file version, so no compatibility issues arise.

It should be noted that, in the first case, it is possible that the number and name of the attribute operation classes in the code base are the same as the attribute operation classes described in the respective attribute descriptions in the metadata description file; The number of attribute operation classes in the code base is greater than the attribute operation class described in each attribute description file in the metadata description file, but the attribute operation class in the code library includes attributes described by each attribute description in the metadata description file. The operation class, but since the attribute operation class corresponding to the attribute operation class name in the metadata description file is loaded according to the metadata description file at the time of serialization, a compatibility problem does not occur.

In the second case, only the attribute operation class with the same name as the attribute operation class provided by each attribute description in the metadata description file can be obtained in the code library, indicating the local data object and the metadata description file version. Mismatch, ie the metadata description file is a newer version and compatibility issues occur during serialization.

In order to meet the compatibility problem between the metadata description file and the local data object in the deserialization, the technical solution of the present embodiment provides a preferred implementation manner. In a preferred manner, the method specifically includes steps S707-1 to S707-3. This will be further described below in conjunction with FIG.

Please refer to FIG. 9 , which shows a flowchart of a method for discarding default values during deserialization according to an embodiment of the present application.

In step S707-1, for the attribute description of the attribute operation class, the default attribute operation class corresponding to the attribute type is obtained according to the attribute type described by the attribute.

For the attribute description that cannot obtain the attribute operation class, obtaining the default attribute operation class corresponding to the attribute type according to the attribute type described by the attribute may be implemented as follows: from the metadata description file, determining that the attribute cannot be obtained The attribute description of the operation class, that is, the attribute operation class described by the attribute description does not exist, the attribute corresponding to the attribute operation class does not exist in the local data object, and the attribute type described by the attribute is obtained, and according to the attribute The attribute type described, and obtain the default attribute operation class corresponding to the attribute type.

For example, the obtained attribute type is VARINT, and the default attribute operation class of type DefaultVarintFieldDef is obtained.

In step S707-2, the reading method provided by the default attribute operation class is obtained, and the data in the result data stream is read in the order of the attribute description sequence number.

Loading the obtained default attribute operation class, calling the setter method encapsulated in the default attribute operation class, and reading out the attribute value in the result data stream.

Step S707-3, discarding the data in the read result data stream.

It should be noted that, the discarding the data in the read result data stream refers to: after the attribute value in the result data stream is read out, no writing is performed, that is, the data is not to be The attribute value is written to the local data object.

In a specific implementation, in this step S707, after reading an attribute value from the result data stream, the deserialization pointer is correspondingly moved to the position of the next attribute value.

The local data object to be written to the attribute value is:

Class User{

Int userId;

String name;

Int phone;

}

The metadata description file of the local data object is:

1:com.taobao.test.io.UserIdFieldDef:VARINT

2: com.taobao.test.io.NameFieldDef:VARINT

3: com.taobao.test.io.PhoneFieldDef:VARINT

4: com.taobao.test.io.AgeFieldDef:VARINT

In step S705, the obtained attribute operation class includes: UserIdFieldDef, NameFieldDef, and PhoneFieldDef, wherein the attribute type in the fourth attribute description is obtained, because the attribute operation class having the same name as the AgeFieldDef is not acquired according to the fourth attribute description, that is, : Get the VARINT type, retrieve the default property operation class DefaultVarintFieldDef of the corresponding type VARINT by obtaining the VARINT type, obtain the property operation class DefaultVarintFieldDef, and execute the in. in the UserIdFieldDef according to the sequence number of the attribute description in the metadata description file. The readVarint() method reads the data 1 at the corresponding position of the result data stream, and decompresses the data, moves the deserialized pointer accordingly to the position of the next attribute value, and executes in.readVarint() in the NameFieldDef. The method reads the data aaa of the corresponding position of the result data stream, decompresses the data, moves the deserialized pointer to the position of the next attribute value, and executes the in.readVarint() method in the PhoneFieldDef. The result data stream corresponds to the data of the location 0, and the number Decompresses the in.readVarint deserialization pointer position corresponding to the next attribute value, and performing the DefaultVarintFieldDef () method reads, discarding 20 the attribute value attribute value after decompression.

Public class UserIdFieldDef<Item>implements FieldDef<T>{

Public void setter(User user,CodedInputStream in){

user.setUserId(in.readVarint());

}

Public void getter(User user,CodedOutputStream out){

out.writeVarint(user.getUserId());

}

DefaultVarintFieldDef:

Public class DefaultVarintFieldDef<Object>implements FieldDef<T>{

Public void setter(Object o,CodedInputStream in){

in.readVarint();

}

Public void getter(Object o,CodedOutputStream out){

out.writeVarint(0);

}

Step S709: Write the read attribute values into the attributes of the local data objects corresponding to the respective attribute descriptions according to the writing method provided by the attribute operation class described by each attribute.

In a specific implementation, the attribute value in the result data stream read by the in.readVarint() method in step S707 is decompressed, and then the attribute value is written according to the attribute description number by a setter method. Corresponding to the properties of the local data object.

For example, the user.setUserId(in.readVarint()) in the UserIdFieldDef is executed to write the attribute value read from the result data stream into the attribute of the local data object corresponding to the userId.

In the above embodiments, a method for deserializing a data stream is provided. Corresponding to the above method for deserializing a data stream, the present application further provides an apparatus for deserializing a data stream. Since the embodiment of the device is substantially similar to the embodiment of the method, the description is relatively simple, and the relevant portions can be referred to the description of the method embodiment. The device embodiments described below are merely illustrative. The apparatus for deserializing the data stream is implemented as follows:

Please refer to FIG. 10, which shows a schematic diagram of an apparatus for deserializing a data stream according to an embodiment of the present application.

The device for deserializing the data stream includes: a data stream obtaining unit 1001, a description file acquiring unit 1003, an attribute operation class obtaining unit 1005, an attribute value reading unit 1007, and an attribute value writing unit 1009;

The data stream obtaining unit 1001 is configured to acquire a result data stream;

The description file obtaining unit 1003 is configured to acquire a metadata description file of the local data object, where the metadata description file includes an attribute description identified by a serial number and an attribute operation class name corresponding to an attribute in the data object;

The attribute operation class obtaining unit 1005 is configured to obtain, according to the attribute operation class name provided by each attribute description in the metadata description file, an attribute operation class corresponding to each attribute;

The attribute value reading unit 1007 is configured to sequentially read the attribute values in the result data stream according to the sequence of the attribute description sequence according to the reading method provided by the attribute operation class corresponding to each attribute;

The attribute value writing unit 1009 is configured to write the read attribute value into the attribute of the local data object corresponding to each attribute description according to the writing method provided by the attribute operation class described by each attribute. in.

Optionally, the description file obtaining unit 1003, the obtained metadata description file of the local data object further includes an attribute type.

Optionally, the attribute value reading unit 1007 includes: an attribute type acquiring subunit, and a data reading subunit. And discarding subunits;

Optionally, the description file obtaining unit 1003 includes: a local data object name obtaining subunit, a result data stream parsing subunit, a description file traversing subunit, and a description file acquiring subunit;

The attribute value decompressing unit is configured to sequentially read the attribute values in the result data stream according to the order of the attribute description sequence in the reading method provided by the attribute operation class corresponding to each attribute, and then The attribute values read from the result data stream are decompressed.

In the above embodiments, a method for serializing a data object, a device for serializing a data object, an electronic device, a method for deserializing a data stream, and a pair of data streams are provided The apparatus for deserialization, in addition, the present application also provides yet another electronic device; the electronic device implementation is as follows:

Please refer to FIG. 11 , which shows a schematic diagram of an electronic device provided in accordance with an embodiment of the present application.

The electronic device includes: a display 1101; a processor 1103; a memory 1105;

The memory 1105 is configured to store a deserialization program, where the program, when being read and executed by the processor, performs an operation of: acquiring a result data stream, acquiring a metadata description file of the local data object, The metadata description file includes an attribute description identified by a sequence number and an attribute operation class name corresponding to the attribute in the data object, the root According to the attribute operation class name provided by each attribute description in the metadata description file, the attribute operation class corresponding to each attribute is obtained, and the read data provided by the attribute operation class corresponding to each attribute is sequentially used in the result data stream. The attribute values are read in the order of the attribute description sequence numbers, and the read attribute values are written into the local data objects corresponding to the respective attribute descriptions according to the writing method provided by the attribute operation class described by each attribute. In the attribute.

In the above embodiments, a method for serializing a data object, a device for serializing a data object, an electronic device, a method for deserializing a data stream, and a pair of data streams are provided. Deserializing device and yet another electronic device, in addition, the present application also provides a serialization and deserialization system; the serialization and deserialization system implementation is as follows:

Referring to Figure 12, a schematic diagram of a serialization and deserialization system provided in accordance with an embodiment of the present application is shown.

The serialization and deserialization system includes: a device 1201 for serializing a data object and a device 1203 for deserializing the data stream;

The device 1201 for serializing the data object is configured to obtain a metadata description file, obtain an attribute operation class corresponding to each attribute according to the metadata description file, and read each attribute and each attribute of the data object to be serialized. Describe the corresponding attribute value, and write the read attribute value to the result data stream of the data object in the order of the attribute description sequence number;

The device 1203 for deserializing the data stream is configured to acquire a result data stream, obtain a metadata description file, obtain an attribute operation class corresponding to each attribute according to the metadata description file, and sequentially perform the attribute data stream in the result data stream. The attribute values are read in the order of the attribute description sequence numbers, and the read attribute values are written in the attributes of the local data objects corresponding to the respective attribute descriptions.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory. Memory is an example of a computer readable medium.

1. Computer readable media including both permanent and non-persistent, removable and non-removable media may be implemented by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, Magnetic box magnetic A tape, magnetic tape storage or other magnetic storage device or any other non-transportable medium can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media, such as modulated data signals and carrier waves.

2. Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system, or computer program product. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment in combination of software and hardware. Moreover, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

The present application is disclosed in the above preferred embodiments, but it is not intended to limit the present application, and any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application. The scope of protection should be based on the scope defined by the claims of the present application.

Claims

A method for serializing data objects, comprising:

Obtaining a metadata description file of the data object to be serialized, where the metadata description file includes an attribute description identified by a serial number and an attribute operation class name corresponding to the attribute in the data object;

Obtaining, according to the attribute operation class name provided by each attribute description in the metadata description file, an attribute operation class corresponding to each attribute;

Reading the attribute values corresponding to the respective attribute descriptions in the data objects to be serialized according to the reading methods provided by the attribute operation classes corresponding to the respective attributes;

The read attribute values are written to the result data stream of the data object in the order of the attribute description sequence number, and the result data stream is used as a serialization result of the data object.
The method for serializing data objects according to claim 1, wherein the attribute description further comprises: an attribute type.
The method for serializing a data object according to claim 2, wherein after the step of reading the attribute value corresponding to each attribute description in the data object to be serialized, the method includes:

Gets the attribute type of the attribute description that cannot be read because the attribute operation class of the corresponding attribute cannot be obtained;

Determining, according to the attribute type, a default attribute operation class corresponding to the attribute type;

According to the default attribute operation class, the corresponding attribute is given a predetermined default value for the attribute description.
The method for serializing a data object according to claim 1, wherein the metadata description file of the data object to be serialized is:

And searching, according to the name of the data object to be serialized, a metadata description file of the data object to be serialized corresponding to the data object to be serialized and having the largest version number value from the metadata repository.
The method for serializing data objects according to claim 4, wherein before said step of writing the read attribute values into the result data stream of said data object in the order of attribute description numbers, :

Reading a version number of the metadata description file of the data object to be serialized and writing it to the head of the result data stream of the data object to be serialized.
The method for serializing data objects according to claim 1, wherein before the step of acquiring the metadata description file of the data object to be serialized, the method further comprises:

Receiving a modification operation on an attribute of the data object to be serialized; the modifying operation includes: adding, deleting, or changing an attribute of the data object to be serialized;

Creating an attribute operation class for the newly added attribute of the data object to be serialized based on the modification operation of the data object to be serialized;

A new version of the metadata description file is created according to the modification operation, and the attribute description in the metadata description file is changed according to the modification operation based on the attribute description in the original version metadata description file.
The method for serializing data objects according to claim 6, wherein in the modifying operation, a new version of the metadata description file is created, and the attribute description in the metadata description file is described in the original version metadata. After the steps of the attribute description in the file are changed according to the modification operation, including:

And traversing a metadata description file set for the data object to be serialized in a metadata repository storing the metadata description file according to the name of the data object to be serialized;

Obtaining, in the traversing process, a value having a maximum version number of the metadata description file of the data object to be serialized in the metadata repository;

Adding the value to one, and adding the added value as the version number of the new version of the metadata description file;

The metadata description file in which the version number is recorded is saved in the metadata repository.
The method for serializing a data object according to claim 1, wherein in the reading method provided by the attribute operation class corresponding to each attribute, reading the data object to be serialized and each attribute description After the steps corresponding to the attribute values, include:

The read attribute values are compressed.
An apparatus for serializing a data object, comprising: a description file acquisition unit, an attribute operation class acquisition unit, an attribute reading unit, and a serialization writing unit;

The description file obtaining unit is configured to acquire a metadata description file of the data object to be serialized, where the metadata description file includes an attribute description identified by a serial number and an attribute operation class name corresponding to an attribute in the data object;

The attribute operation class obtaining unit is configured to obtain an attribute operation class name corresponding to each attribute according to the attribute operation class name provided by each attribute description in the metadata description file;

The attribute reading unit is configured to read, according to a reading method provided by the attribute operation class corresponding to each attribute, an attribute value corresponding to each attribute description in the data object to be serialized;

The serialization writing unit is configured to write the read attribute values into the result data stream of the data object in the order of the attribute description sequence number, and the result data stream is used as a serialization result of the data object.
The apparatus for serializing a data object according to claim 9, wherein the metadata description file of the data object to be serialized acquired by the description file obtaining unit further includes an attribute type.
The apparatus for serializing data objects according to claim 10, wherein the attribute reading unit further comprises: an attribute type reading subunit, a default attribute operation class obtaining subunit, and a default value reading. Subunit

The attribute type reading subunit is configured to read an attribute value corresponding to each attribute description in the data object to be serialized, and obtain an attribute description attribute that cannot be read because the attribute operation class of the corresponding attribute cannot be obtained. Types of;

The default attribute operation class acquisition sub-unit is configured to retrieve, according to the attribute type, a default attribute operation class corresponding to the attribute type;

The default value reading subunit is configured to assign a predetermined default value to the corresponding attribute of the attribute description according to the default attribute operation class.
The apparatus for serializing a data object according to claim 9, wherein the description file obtaining unit is specifically configured to: search for a corresponding to the waiting from the metadata repository according to the name of the data object to be serialized A metadata description file of the serialized data object and the serial number value of the data object to be serialized.
The apparatus for serializing a data object according to claim 12, wherein the serialization writing unit further comprises: a version number writing subunit;

The version number is written into the subunit, and is configured to read the data object to be serialized before the attribute value to be read is written into the result data stream of the data object in the order of the attribute description sequence number. The metadata describes the version number of the file and writes it to the head of the resulting data stream of the data object to be serialized.
The apparatus for serializing a data object according to claim 9, further comprising: a modification operation receiving unit, an operation class creation unit, and a description file creation unit;

The modifying operation receiving unit is configured to receive, before the acquiring the metadata description file of the data object to be serialized, a modification operation on the attribute of the data object to be serialized; the modifying operation includes: Adding, deleting, or changing the attributes of the serialized data object;

The operation class creation unit is configured to create an attribute operation class for the newly added attribute of the data object to be serialized based on the modification operation of the data object to be serialized;

a description file creation unit, configured to create a new version of the metadata description file according to the modification operation, where the attribute description in the metadata description file is based on the attribute description in the original version metadata description file, according to the modification operation The change was made.
The apparatus for serializing data objects according to claim 14, further comprising: Calendar unit, value acquisition unit, version number generation unit, and save unit;

The traversing unit is configured to create a new version of the metadata description file according to the modification operation, where the attribute description in the metadata description file is based on the attribute description in the original version metadata description file, according to the modification After the operation is changed, the metadata description file set for the data object to be serialized is traversed in the metadata repository storing the metadata description file according to the name of the data object to be serialized;

The value obtaining unit is configured to acquire, during the traversal process, a maximum value of a version number of the metadata description file of the data object to be serialized in the metadata repository;

The version number generating unit is configured to add the value by one, and use the added value as the version number of the metadata version file of the new version;

The saving unit is configured to save the metadata description file with the version number recorded in the metadata repository.
The apparatus for serializing data objects according to claim 9, further comprising: an attribute value compression unit;

The attribute value compression unit is configured to read the attribute value after the attribute value corresponding to each attribute description in the data object to be serialized according to the reading method provided by the attribute operation class corresponding to each attribute Compress.
An electronic device, comprising:

monitor;

processor;

a memory for storing a serialization program, the program, when being read and executed by the processor, performing an operation of: acquiring a metadata description file of a data object to be serialized, where the metadata description file includes a serial number identifier The attribute description and the attribute operation class name corresponding to the attribute in the data object; the attribute operation class name provided according to each attribute description in the metadata description file, and the attribute operation class corresponding to each attribute are obtained; according to the corresponding attribute The reading method provided by the attribute operation class reads the attribute value corresponding to each attribute description in the data object to be serialized; and writes the read attribute value into the result data of the data object in the order of the attribute description serial number. Flow, the resulting data stream as a serialization result of the data object.
A method for deserializing a data stream, comprising:

Obtaining a result data stream;

Obtaining a metadata description file of the local data object, where the metadata description file includes an attribute description identified by a serial number and an attribute operation class name corresponding to an attribute in the data object;

Obtaining, according to the attribute operation class name provided by each attribute description in the metadata description file, an attribute operation class corresponding to each attribute;

According to the reading method provided by the attribute operation class corresponding to each attribute, the attribute values in the result data stream are sequentially read in the order of the attribute description sequence number;

The read attribute values are written into the attributes of the local data object corresponding to the respective attribute descriptions according to the write method provided by the attribute operation class described by each attribute.
The method for deserializing a data stream according to claim 18, wherein the attribute description further comprises: an attribute type.
The method for deserializing a data stream according to claim 19, wherein the attribute value in the result data stream is sequentially sequentially performed in the writing method provided by the attribute operation class described according to each attribute Among the steps of reading in the order of the attribute description serial number, it also includes:

For the attribute description of the attribute operation class, the default attribute operation class corresponding to the attribute type is obtained according to the attribute type described by the attribute;

Obtaining the reading method provided by the default attribute operation class, and reading the data in the result data stream according to the sequence of the attribute description sequence number;

The data in the result data stream read is discarded.
The method for deserializing a data stream according to claim 18, wherein the obtaining the metadata description file of the local data object comprises:

Obtaining the name of the local data object;

Parsing the result data stream to obtain a first value in the result data stream;

And traversing a metadata description file set for the local data object in a metadata repository storing the metadata description file according to the name of the local data object;

In the traversal process, a metadata description file of the local data object having the same name as the local data object and having the same version number as the first value in the result data stream is obtained.
The method for deserializing a data stream according to claim 18, wherein, in the reading method provided by the attribute operation class corresponding to each attribute, the attribute values in the result data stream are sequentially attributed according to the attribute After the step of describing the sequential reading of the serial numbers, includes:

The attribute values read from the resulting data stream are decompressed.
An apparatus for deserializing a data stream, comprising: a data stream obtaining unit, a description file acquiring unit, an attribute operation class acquiring unit, an attribute value reading unit, and an attribute value writing unit;

The data stream obtaining unit is configured to obtain a result data stream;

The description file obtaining unit is configured to acquire a metadata description file of the local data object, where the metadata description file includes an attribute description identified by a serial number and an attribute operation class name corresponding to an attribute in the data object;

The attribute operation class obtaining unit is configured to obtain an attribute operation class name corresponding to each attribute according to the attribute operation class name provided by each attribute description in the metadata description file;

The attribute value reading unit is configured to sequentially read the attribute values in the result data stream according to the sequence of the attribute description sequence according to the reading method provided by the attribute operation class corresponding to each attribute;

The attribute value writing unit is configured to write the read attribute value into the attribute of the local data object corresponding to each attribute description according to the writing method provided by the attribute operation class described by each attribute .
The apparatus for deserializing a data stream according to claim 23, wherein the description file obtaining unit acquires the metadata description file of the local data object further includes an attribute type.
The device for deserializing a data stream according to claim 24, wherein the attribute value reading unit comprises: an attribute type obtaining subunit, a data reading subunit, and a discarding subunit;

The attribute type obtaining subunit is configured to sequentially read the attribute values in the result data stream according to the sequence of the attribute description serial number in the writing method provided by the attribute operation class according to each attribute description. For the attribute description of the attribute operation class, the default attribute operation class corresponding to the attribute type is obtained according to the attribute type described by the attribute;

The data reading subunit is configured to acquire a reading method provided by the default attribute operation class, and read data in the result data stream according to an order of the attribute description sequence number;

The discarding a single sub-unit is configured to discard the data in the read result data stream.
The apparatus for deserializing a data stream according to claim 23, wherein the description file obtaining unit comprises: a local data object name obtaining subunit, a result data stream parsing subunit, a description file traversing subunit, and Descriptive file acquisition subunit;

The local data object name obtaining subunit is configured to acquire a name of the local data object;

The result data stream parsing subunit is configured to parse the result data stream to obtain a first value in the result data stream;

The description file traversing subunits for traversing a metadata description file set for the local data object in a metadata repository in which the metadata description file is stored according to the name of the local data object;

The description file obtaining subunit is configured to acquire a name of the local data object during the traversal process And a metadata description file of the local data object having the same version number as the first value in the result data stream.
The apparatus for deserializing a data stream according to claim 23, further comprising: an attribute value decompression unit;

The attribute value decompressing unit is configured to sequentially read the attribute values in the result data stream according to the order of the attribute description sequence in the reading method provided by the attribute operation class corresponding to each attribute, and then The attribute values read from the result data stream are decompressed.
An electronic device, comprising:

monitor;

processor;

a memory for storing a deserialization program, the program, when being read and executed by the processor, performing an operation of: acquiring a result data stream, acquiring a metadata description file of the local data object, the metadata description The file includes an attribute description identified by a sequence number and an attribute operation class name corresponding to the attribute in the data object, and an attribute operation class name corresponding to each attribute description in the metadata description file is obtained, and an attribute operation class corresponding to each attribute is obtained, According to the reading method provided by the attribute operation class corresponding to each attribute, the attribute values in the result data stream are sequentially read in the order of the attribute description sequence number, and the writing method provided by the attribute operation class according to each attribute description is The read attribute values are written into the attributes of the local data object corresponding to the respective attribute descriptions.
A serialization and deserialization system, characterized in that the system comprises:

Apparatus for serializing data objects according to any one of claims 9 to 16; and apparatus for deserializing a data stream according to any one of claims 23 to 27.