WO2017092580A1 - 一种结构化数据的序列化、反序列化方法及装置 - Google Patents

一种结构化数据的序列化、反序列化方法及装置 Download PDF

Info

Publication number
WO2017092580A1
WO2017092580A1 PCT/CN2016/106566 CN2016106566W WO2017092580A1 WO 2017092580 A1 WO2017092580 A1 WO 2017092580A1 CN 2016106566 W CN2016106566 W CN 2016106566W WO 2017092580 A1 WO2017092580 A1 WO 2017092580A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
records
specific
serialized
domain
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/106566
Other languages
English (en)
French (fr)
Inventor
李勇勇
蔡瀛
王升功
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Publication of WO2017092580A1 publication Critical patent/WO2017092580A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/258—Data format conversion from or to a database

Definitions

  • the present application relates to the field of computer technology, and in particular, to a serialization and deserialization method and apparatus for structured data.
  • Serialization of structured data refers to the conversion of raw structured data into serialized data in a specific format to facilitate operations such as transmission based on serialized data.
  • Deserialization is the inverse of serialization, which can restore serialized data with a specific format to the original structured data.
  • a commonly used serialization method is based on the protobuf protocol proposed by Google Inc., based on the protocol to serialize at least one structured data, the specific serialization process includes: each structured data has a corresponding set of attribute values, Converting all structured data into a message object, wherein the message object includes at least one field, each field includes tag information (tag) and value information (value), and each structured value is stored in the value of each field A collection of attribute values for the data.
  • the structured data order1 has a set of attribute values: a1, a2, and a3, and the structured data order2 has a set of attribute values: b1, b2, and b3, and converts the structured data order1 and order2 into a message object according to the provisions of the protobuf protocol.
  • the message object a has 6 fields, and the values of the 6 fields store a1, a2, a3, b1, b2, and b3, respectively.
  • the technical problem to be solved by the present application is to provide a serialization and deserialization method and device for structured data. To solve the problem of serialization that cannot be achieved due to the size limit of the message object.
  • the application provides a serialization method for structured data, including:
  • the serialized data includes n records, wherein the ith record includes mi+1 fields, 1 ⁇ i ⁇ n, mi ⁇ 1, and the value information of the first mi fields stores the ith record a data group, the data group is an attribute value group corresponding to the i-th structured data in the n structured data, and the first specific identifier is stored in the label information of the mi+1-th domain, and is used to identify the The end of the ith record; the serialized data further includes a specific domain after the n records, and the tag information of the specific domain stores a second specific identifier for identifying the end of the serialized data.
  • the n records respectively store check values corresponding to the respective records; wherein, in the ith record, the value information of the mi+1th domain stores the corresponding corresponding to the i th record a check value, where the check value corresponding to the ith record is obtained according to the attribute value group corresponding to the ith structured data.
  • the specific domain includes a first specific domain and/or a second specific domain
  • the tag information of the first specific domain is stored with a first sub-identity, and the value information is stored with a total check value, where the total check value is obtained according to the check value corresponding to the n records respectively;
  • the tag information of the second specific domain stores a second sub-identity, and the value information stores a total number of records, and the total number of records is specifically n.
  • the first specific identifier and the second specific identifier are both values greater than a preset threshold, and the preset threshold is determined according to a maximum available range of the domain identifier of the label information.
  • the application provides a deserialization method for serialized data, including:
  • the serialized data includes n records, wherein the ith record includes mi+1 domains, 1 ⁇ i ⁇ n, n ⁇ 1, mi ⁇ 1, values of the first mi domains
  • the data group of the ith record is stored in the information, and the first specific identifier is stored in the label information of the mi+1th field, and is used to identify the end of the ith record, where the serialized data is located.
  • the n records further include a specific domain, and the tag information of the specific domain stores a second specific identifier for identifying the end of the serialized data;
  • it also includes:
  • the serialized data is verified according to the attribute value group corresponding to the n structured data and the check value corresponding to the n records respectively.
  • the method further includes: obtaining a total check value from the value information of the first specific domain, according to the n records Verifying the serialized data by respectively corresponding check values and the total check values;
  • the method further includes: obtaining a total number of records from the value information of the second specific domain, according to the total number of records and the serialization The number of records included in the data, and the serialized data is verified;
  • the first sub-identity is stored in the label information of the first specific domain
  • the second sub-ID is stored in the label information of the second specific domain.
  • the first specific identifier and the second specific identifier are both values greater than a preset threshold, and the preset threshold is determined according to a maximum available range of the domain identifier of the label information.
  • the application provides a method for saving serialized data, including:
  • the server creates a session
  • each data block being associated with the session identifier, each data block including one or more sets of serialized data;
  • the server saves the plurality of data blocks.
  • any one of the one or more sets of serialized data includes n records, wherein the ith record includes mi+1 domains, 1 ⁇ i ⁇ n, n ⁇ 1,
  • the value information of the first mi fields stores the data group of the ith record, and the tag information of the m+1th domain stores the first specific identifier for identifying the end of the ith record.
  • the set of serialized data further includes a specific domain after the n records, and the tag information of the specific domain stores a second specific identifier for identifying the end of the set of serialized data.
  • it also includes:
  • the server sends the number to the client According to the block missing list, the data block missing list is used to identify a data block that belongs to the data block holding list and does not belong to the multiple data blocks;
  • the server receives the data block identified in the data block missing list sent by the client.
  • the application provides a method for downloading serialized data, comprising: sending, by a server, a total number of saved serialized data to a client;
  • the server receives the download information sent by the client, and the download information indicates the group number identifier of the serialized data to be downloaded by the client;
  • the server distributes the serialized data corresponding to the group number identifier to the client in a distributed manner.
  • any one of the saved serialized data includes n records, wherein the ith record includes mi+1 domains, 1 ⁇ i ⁇ n, n ⁇ 1, mi ⁇ 1
  • the value information of the first mi fields stores the data group of the ith record
  • the tag information of the m+1th domain stores a first specific identifier, which is used to identify the end of the ith record, the group
  • the serialized data further includes a specific domain after the n records, and the tag information of the specific domain stores a second specific identifier for identifying the end of the set of serialized data.
  • the application provides a serialization device, including:
  • a data acquisition unit configured to acquire n structured data, n ⁇ 1;
  • An attribute value obtaining unit configured to acquire an attribute value group corresponding to the n structured data respectively;
  • a data generating unit configured to generate a set of serialized data according to the attribute value groups respectively corresponding to the n structured data
  • the serialized data includes n records, wherein the ith record includes mi+1 fields, 1 ⁇ i ⁇ n, mi ⁇ 1, and the value information of the first mi fields stores the ith record a data group, the data group is an attribute value group corresponding to the i-th structured data in the n structured data, and the first specific identifier is stored in the label information of the mi+1-th domain, and is used to identify the The end of the ith record; the serialized data further includes a specific domain after the n records, and the tag information of the specific domain stores a second specific identifier for identifying the end of the serialized data.
  • the n records respectively store check values corresponding to the respective records; wherein, in the ith record, the value information of the mi+1th domain stores the corresponding corresponding to the i th record a check value, where the check value corresponding to the ith record is obtained according to the attribute value group corresponding to the ith structured data.
  • the specific domain includes a first specific domain and/or a second specific domain
  • the tag information of the first specific domain is stored with a first sub-identity, and the value information is stored with a total check value, where the total check value is obtained according to the check value corresponding to the n records respectively;
  • the tag information of the second specific domain stores a second sub-identity, and the value information stores a total number of records, and the total record
  • the number of records is specifically n.
  • the first specific identifier and the second specific identifier are both values greater than a preset threshold, and the preset threshold is determined according to a maximum available range of the domain identifier of the label information.
  • the application provides a deserialization device, including:
  • a data obtaining unit configured to acquire a set of serialized data, where the serialized data includes n records, wherein the ith record includes mi+1 domains, 1 ⁇ i ⁇ n, n ⁇ 1, mi ⁇ 1,
  • the value information of the first mi fields stores the data group of the ith record, and the tag information of the m+1th domain stores a first specific identifier for identifying the end of the ith record.
  • the serialized data further includes a specific domain after the n records, and the tag information of the specific domain stores a second specific identifier for identifying the end of the serialized data;
  • An attribute value obtaining unit configured to obtain, from the serialized data, an attribute value group corresponding to the n pieces of structured data, where the attribute value group corresponding to the i-th structured data is from the data group recorded by the ith piece Obtained
  • the data generating unit is configured to generate the n structured data according to the attribute value groups respectively corresponding to the n structured data.
  • it also includes:
  • a check value obtaining unit configured to obtain, from the serialized data, a check value corresponding to each of the n records, wherein the check value corresponding to the ith record is from the i+th record Obtained in the value information of 1 domain;
  • the first checking unit is configured to check the serialized data according to the attribute value group corresponding to the n structured data and the check value corresponding to the n records respectively.
  • the apparatus further includes: a second check unit, configured to obtain a total check from the value information of the first specific domain And verifying the serialized data according to the check value corresponding to the n records and the total check value;
  • the apparatus further includes: a third check unit, configured to obtain a total number of records from the value information of the second specific domain, according to the The total number of records and the number of records included in the serialized data, and the serialized data is verified;
  • the first sub-identity is stored in the label information of the first specific domain
  • the second sub-ID is stored in the label information of the second specific domain.
  • the first specific identifier and the second specific identifier are both values greater than a preset threshold, and the preset threshold is determined according to a maximum available range of the domain identifier of the label information.
  • the application provides a server, including:
  • a sending unit configured to send the session identifier of the session to the client
  • a receiving unit configured to receive, by the client, a plurality of data blocks that are distributedly transmitted, each data block being associated with the session identifier, each data block including one or more sets of serialized data, and receiving the a data block save list sent by the client, where the data block save list is used to identify all data blocks to be saved;
  • a saving unit configured to save the plurality of data blocks if the plurality of data blocks match the data block save list.
  • any one of the one or more sets of serialized data includes n records, wherein the ith record includes mi+1 domains, 1 ⁇ i ⁇ n, n ⁇ 1,
  • the value information of the first mi fields stores the data group of the ith record, and the tag information of the m+1th domain stores the first specific identifier for identifying the end of the ith record.
  • the set of serialized data further includes a specific domain after the n records, and the tag information of the specific domain stores a second specific identifier for identifying the end of the set of serialized data.
  • the sending unit is further configured to: if the multiple data blocks do not match the data block save list, send a data block missing list to the client, where the data block missing list is used to identify a data block belonging to the data block holding list and not belonging to the plurality of data blocks;
  • the receiving unit is further configured to receive a data block identified in the missing list of the data block that is sent by the client.
  • the application provides a server, including: a sending unit and a receiving unit;
  • the sending unit is configured to send the total number of saved serialized data to the client;
  • the receiving unit is configured to receive download information sent by the client, where the download information indicates a group number identifier of the serialized data to be downloaded by the client;
  • the sending unit is further configured to distribute the serialized data corresponding to the group number identifier to the client in a distributed manner.
  • any one of the saved serialized data includes n records, wherein the ith record includes mi+1 domains, 1 ⁇ i ⁇ n, n ⁇ 1, mi ⁇ 1
  • the value information of the first mi fields stores the data group of the ith record
  • the tag information of the m+1th domain stores a first specific identifier, which is used to identify the end of the ith record, the group
  • the serialized data further includes a specific domain after the n records, and the tag information of the specific domain stores a second specific identifier for identifying the end of the set of serialized data.
  • the n structured data is converted into a set of serialized data, wherein the set of serialized data includes two parts, and the first part includes n records, and each record corresponds to one structured data.
  • the i-th record includes mi+1 fields, and the i-th node is stored in the value of the first mi fields Attributed value group corresponding to the structured data, the first specific identifier is stored in the tag of the mi+1 field, and is used to identify the end of the ith record; and the second part, that is, after the n records A specific field is also included, and a second specific identifier is stored in the tag of the specific field for identifying the end of the serialized data.
  • the structure of the message object is no longer used in the present application, but the n structured data is converted into n records, and the last field of each record is used to identify the end of the record, thereby being able to distinguish different records. And the end of the serialized data is identified by a specific field after n records. Therefore, the problem that serialization cannot be achieved due to the size limitation of the message object is solved.
  • FIG. 1 is a schematic structural diagram of a field in a protobuf protocol
  • FIG. 2 is a schematic flow chart of an embodiment of a serialization method provided by the present application.
  • FIG. 3 is a schematic structural diagram of providing serialized data according to the present application.
  • FIG. 4 is a schematic flow chart of an embodiment of a deserialization method provided by the present application.
  • FIG. 5 is a schematic flowchart diagram of an embodiment of a saving method provided by the present application.
  • FIG. 6 is a schematic flowchart diagram of an embodiment of a downloading method provided by the present application.
  • FIG. 7 is a schematic structural diagram of an apparatus embodiment of a serialization apparatus provided by the present application.
  • FIG. 8 is a schematic structural diagram of an apparatus embodiment of a deserialization apparatus provided by the present application.
  • FIG. 9 is a schematic structural diagram of an apparatus embodiment of a server provided by the present application.
  • FIG. 10 is a schematic structural diagram of another apparatus embodiment of a server provided by the present application.
  • a common serialization method is based on the protobuf protocol proposed by Google Inc., which is described below.
  • the protobuf protocol is based on the structure of the message object, and the message object includes at least one field. As shown in Figure 1, a field includes a tag and a value.
  • the length of the tag is 4 bytes, including the domain identifier (field_number) and the value type (value_t ype).
  • the field_number is used to identify the field
  • the value_type is used to describe the data type in the value
  • the value is used to store the data.
  • the structured data order1 has a set of attribute values: a1, a2, and a3
  • the structured data order2 has a set of attribute values: b1, b2, and b3, and converts the structured data order1 and order2 into a message object according to the provisions of the protobuf protocol.
  • the message object a has 6 fields, and the values of the 6 fields store a1, a2, a3, b1, b2, and b3, respectively.
  • the protobuf protocol does not support data verification during the serialization process, so data security cannot be guaranteed.
  • the embodiment of the present application provides a serialization and deserialization method and device for structured data, so as to solve the problem that serialization cannot be achieved due to size limitation of a message object.
  • the present application provides an embodiment of a method for serializing structured data, where the method includes:
  • each attribute value group includes at least one attribute value.
  • the structured data order 1 corresponds to the attribute value group a
  • the attribute value group a specifically includes: a1, a2, and a3
  • the structured data order 2 corresponds to the attribute value group b
  • the attribute value group b specifically includes: b1, b2, and b3.
  • the serialized data includes two parts.
  • the first part consists of n records, namely Row 1, Row 2, ..., Row n in the first row of Figure 3.
  • Each record corresponds to one structured data, that is, each record stores a set of attribute values of the structured data corresponding to the record.
  • the ith record 1 ⁇ i ⁇ n
  • the ith record includes mi+1 fields, mi ⁇ 1
  • the data of the ith record is stored in the value of the first mi fields
  • the ith The data group of the record is the attribute value group corresponding to the i-th structured data in the n structured data
  • the first specific identifier is stored in the tag of the mi+1 field, and the first specific identifier is used to identify the The end of the record of the i-th record.
  • the first record (ie, Row1) includes m1+1 columns (Column), that is, Column 1, Column 2, ..., Column m1, Checksum 1, and one column actually
  • the identifier of the column can be represented by the field_number of the corresponding field.
  • the identifier of the first m1 column can be incremented from 0 to m1-1, or it can be out of order.
  • the data group of the first record is stored in the value of the first m1 columns, that is, the attribute value group corresponding to the first structured data (for example, a1, a2, and a3).
  • the tag of the m1+1th column (ie, the checksum 1 shown in FIG. 3) stores a first specific identifier for identifying the end of the first record.
  • the first specific identifier is stored in the field_number of the m1+1th field, and can be distinguished from the field_number in the field for storing data.
  • the field_number in the first mi fields is incremented by 0, for example, 0, 1, 2, ..., mi-1, respectively, and the first specific identifier may be a relatively large number. That is, the number of fields_number in the field for storing data is impossible to reach.
  • the first specific identifier may be a value greater than a preset threshold, and the preset threshold is determined according to a maximum available range of the field_number of the tag.
  • the first specific identifier may be 2 to the 25th power minus 1, ie (2 ⁇ 25)-1.
  • the serialized data further includes a second part, that is, a specific field, a tag of the specific field stores a second specific identifier, and the second specific identifier is used to identify the end of the serialized data.
  • the first specific identifier and the second specific identifier are different.
  • the second specific identifier may be a value greater than a preset threshold, and the preset threshold is determined according to a maximum available range of the field_number of the tag.
  • the second specific identifier may be (2 ⁇ 25)-1024 and/or (2 ⁇ 25)-2.
  • the serialized data further includes a specific field, that is, a record Checksum n and a record Total Row Count, wherein the record Checksum n and the record are recorded.
  • the Total Row Count corresponds to a field.
  • the field_number of the record Checksum n and the field_number of the record Total Row Count are taken as the second specific identifier.
  • the specific field shown in FIG. 3 may also include only one of the record Checksum n and the record Total Row Count.
  • the number of fields of each record in the serialized data may be different, but the missing field is added in the deserialization, so that the number of fields of each record is consistent, where The value in the filled field is set to null and you don't have to encode it.
  • the n structured data is converted into a set of serialized data, wherein the set of serialized data includes two parts, and the first part includes n records, and each record corresponds to one structured data.
  • the ith record includes mi+1 fields
  • the value of the first mi fields stores the attribute value group corresponding to the i-th structured data
  • the first specific identifier is stored in the tag of the mi+1 field
  • the second part that is, the n records, further includes a specific field, where the tag of the specific field stores a second specific identifier for identifying the The end of serializing the data.
  • the structure of the message object is no longer used in the present application, but the n structured data is converted into n records, and the last field of each record is used to identify the end of the record, thereby being able to distinguish different records. And the end of the serialized data is identified by a specific field after n records. Therefore, the problem that serialization cannot be achieved due to the size limitation of the message object is solved.
  • a check code may also be added to the serialized data, thereby supporting data verification of the serialization process, and improving data security. The details are described below.
  • the check values corresponding to the respective records are respectively stored in the n records.
  • the value corresponding to the i-th record is stored in the value of the mi+1 field
  • the check value corresponding to the i-th record is based on the attribute value corresponding to the i-th structured data. Group get. For example, according to each attribute value in the attribute value group corresponding to the i-th structured data, respectively calculate a corresponding redundancy check code, such as crc32, and calculate the sum of each redundant check code as the i-th
  • the corresponding check value is recorded and stored in the value of the mi+1 field, for example, the value of Checksum 1 shown in FIG.
  • the value_type of the mi+1 field can be 32-bit defined by the protobuf protocol.
  • a check value is stored in the specific field.
  • the specific field in the serialized data may be one field or multiple fields.
  • the particular field includes a first field and/or a second field.
  • the first field identifier is stored in the tag of the first field, and the total check value is stored in the value, and the total check value is obtained according to the check value corresponding to the n records, for example, corresponding to each record.
  • the second field identifier is stored in the tag of the second field, and the total number of records is stored in the value, and the total number of records is specifically n.
  • the first field may be a field corresponding to the record Checksum n, and the field_number of the field may be (2 ⁇ 25)-1024, and the value_type may be a 32-bit defined by the protobuf protocol.
  • the second field may be a field corresponding to the Total Row Count, the field_number of the field may be (2 ⁇ 25)-2, and the value_type may be 64-bit defined by the protobuf protocol.
  • the first sub The second specific identifier if the specific field includes the second field and does not include the first field, the second sub-ID is used as the second specific identifier, if the specific field includes the first field and includes the first field The second field, the first sub-identity and the second sub-identity are used as the second specific identifier.
  • the present application provides an embodiment of a method for deserializing serialized data.
  • the method in this embodiment includes:
  • the serialized data is specifically the serialized data generated by the embodiment corresponding to FIG. 2 .
  • the serialized data includes two parts, and the first part includes n records.
  • the i-th record includes mi+1 fields, 1 ⁇ i ⁇ n, n ⁇ 1, mi ⁇ 1, and the data group of the i-th record is stored in the value of the first mi fields, the mi+1
  • the first specific identifier is stored in the tag of the field, and the first specific identifier is used to identify the end of the ith record.
  • the serialized data further includes a second part, that is, a specific field, the tag of the specific field stores a second specific identifier, and the second specific identifier is used to identify the serialized data.
  • the data group recorded by the ith column may be directly used as the attribute value group corresponding to the i-th structured data.
  • the serialized data in the embodiment of the present application no longer uses the structure of the message object, but stores the attribute value groups of the n structured data through n records, and the last field of each record is used to identify The end of the record, so that different records can be distinguished, and the end of the serialized data is identified by a specific field after the n records. Therefore, the attribute values in the attribute value group of the structured data are no longer limited by the number, which solves the problem that the serialization cannot be realized due to the size limitation of the message object.
  • the first specific identifier and the second specific identifier are both values greater than a preset threshold, and the preset threshold is determined according to a maximum available range of the field_number of the tag.
  • data verification may also be performed according to the check code in the serialized data to improve data security. The details are described below.
  • the method further includes: obtaining, from the serialized data, a check value corresponding to each of the n records, wherein the check value corresponding to the ith record is from the ith record Obtaining the value of the mi+1 fields; respectively, the attribute value group corresponding to the n structured data and the check value corresponding to the n records respectively, The serialized data is verified. If all the n records are successfully verified, it indicates that the serialized data verification is successful.
  • the corresponding redundancy check code is respectively calculated according to each attribute value in the attribute value group corresponding to the i-th structured data, and the sum of each redundant check code is calculated, and The sum of the calculated redundancy check codes is the same as the check value corresponding to the i-th record, indicating that the ith record verification is successful.
  • the method further includes: obtaining a total check value from the value of the first specific field, according to the n records respectively
  • the serialized data is verified by a corresponding check value and the total check value. For example, the sum of the check values corresponding to the n records is calculated. If the sum of the check values is consistent with the total check value, it indicates that the serialized data check is successful.
  • the method further includes: obtaining a total number of records from the value of the second specific field, according to the total number of records and the serialized data The number of records included, and the serialized data is verified. For example, the number of records included in the serialized data is consistent with the total number of records, indicating that the serialized data verification is successful.
  • the first sub-identity is stored in the tag of the first specific field
  • the second sub-identity is stored in the tag of the second specific field.
  • structured data processing platforms such as Google's bigquery platform and Amazon's redshift platform. These platforms are capable of converting structured data into serialized data for transport operations such as uploads and downloads.
  • a method for saving serialized data and a server are further provided to implement uploading serialized data in parallel, thereby reducing transmission time.
  • the present application provides an embodiment of a method for saving serialized data, which is applied to a server.
  • the method of this embodiment includes: 501, 502, 503, 504, and 505.
  • the server can set the state of the session to be on.
  • the state of the session is on, it indicates that the data corresponding to the session can be stored.
  • the server sends a session ID of the session to a client.
  • the state of the session may be shared between the server and the client, and therefore, the server may also notify the client of the status of the session.
  • the client receives a session identifier sent by a server.
  • the client segments the serialized data to obtain a plurality of data blocks after the segmentation.
  • each data block is associated with the session identifier, in fact, is associated with the session.
  • Each data block includes one or more sets of serialized data. If each data block includes multiple sets of serialized data, an end identifier is added at the end of the multiple sets of serialized data to identify the end of the data block. .
  • the client distributes the plurality of data blocks after the splitting.
  • the server since the server establishes a session and the session is associated with the plurality of data blocks, the client can be supported to distribute a plurality of data blocks distributed, thereby reducing transmission time.
  • the server receives a plurality of data blocks that are sent by the client in a distributed manner.
  • the client sends a data block save list.
  • the data block save list is used to identify all data blocks to be saved. In fact, the data block save list actually identifies all the data blocks that the client has sliced the serialized data.
  • the server receives a data block save list sent by the client.
  • steps 503 and 504 are not limited, that is, 503 may be performed 504 and 504 may be performed first, or 503 may be performed 503, or 503 and 504 may be performed at the same time.
  • 505 can include sub-steps 5051 and 5052.
  • 5051 Since the data needs to ensure strong consistency, that is to say, the received multiple data blocks are all saved or not saved. Therefore, after receiving the data block and the data block save list, the server matches the received multiple data blocks with the list, which is actually determining all the data blocks and receiving of the list identifier. Whether the multiple data blocks that arrive are consistent. If they match, the match is considered successful. At this time, 5052 is executed.
  • the server commits a plurality of received data blocks, and actually saves the received multiple data blocks.
  • the server can also set the state of the session to off. When the state of the session is off, it indicates that the data corresponding to the session cannot be stored. The server can also notify the client of the status of the session.
  • the server is to the guest
  • the client sends a data block missing list, where the data block missing list is used to identify a data block that belongs to the data block holding list and does not belong to the multiple data blocks; the server receives the client and sends the distributed The data block is missing the data block identified in the list.
  • the server can perform a rematch based on the newly received data block. If the same data block is received multiple times, the last received one will prevail.
  • the session is associated with a plurality of data blocks into which the serialized data is divided, and the server receives the data block storage list, indicating all the data to be saved.
  • the server is configured to receive a plurality of data blocks that are sent by the client in a distributed manner, and determine, according to the received plurality of data blocks and the list, whether the received plurality of data blocks and the list match, that is, Whether the server has received all the data blocks identified by the list. If they match, the server has received all the data blocks identified by the list, so the server saves the received multiple data blocks.
  • the server receives a plurality of data blocks and a data block save list that are sent by the client in a distributed manner, and saves the received multiples only when the received multiple data blocks match the data block save list.
  • the data block guarantees the consistency of the data block.
  • any one of the one or more sets of serialized data may be the specific structure shown in FIG. 3.
  • the n records are included, wherein the ith record includes mi+1 fields, 1 ⁇ i ⁇ n, n ⁇ 1, mi ⁇ 1, and the i-th record is stored in the value of the first mi fields.
  • the tag of the mi+1 field is stored with a first specific identifier for identifying the end of the ith record
  • the set of serialized data further includes a specific field after the n records.
  • the tag of the specific field stores a second specific identifier for identifying the end of the set of serialized data.
  • the embodiment of the present application further provides a method for downloading serialized data, which can implement parallel transmission.
  • the present application provides an embodiment of a method for downloading serialized data, which is applied to a server.
  • the method of this embodiment includes: 701, 702, 703, and 704.
  • the server creates a session.
  • the session may be associated with an index generated by a subsequent process, so that the index is not repeatedly established.
  • the server can also set the state of the session to on. When the state of the session is on, it indicates that the data corresponding to the session can be read.
  • the state of the session may be shared between the server and the client, and therefore, the server may also notify the client of the status of the session.
  • the server may not need to establish a session.
  • the server sends the total number of saved serialized data to the client.
  • the server sends to the client, and currently stores a total of 50 sets of serialized data information.
  • the client receives the total number of groups sent by the server, and generates download information according to the total number of groups, where the download information indicates the group number identifier of the serialized data to be downloaded.
  • the download information may include binary group information (offset, count), where offset indicates the number of starting groups of serialized data to be downloaded, and count indicates the total number of serialized data to be downloaded. For example, if the binary group information is (10, 10), it means that the serialized data of the 10th group to the 19th group is downloaded.
  • the server may establish an index, and sort all the serialized data, so that the client can specify the binary information for downloading, wherein the index is associated with the session established by the server.
  • the client sends the download information to the server.
  • the server receives the download information sent by the client.
  • the server sends the serialized data corresponding to the group number identifier to the client in a distributed manner.
  • the server distributes serialized data to the client distributedly, thereby reducing the time of data transmission.
  • any of the set of serialized data in the saved serialized data may have the structure shown in FIG. Specifically, the n records are included, wherein the ith record includes mi+1 fields, 1 ⁇ i ⁇ n, n ⁇ 1, mi ⁇ 1, and the i-th record is stored in the value of the first mi fields. a data group, the tag of the mi+1 field is stored with a first specific identifier for identifying an end of the ith record, the set of serialized data further including a specific field after the n records, A second specific identifier is stored in the tag of the specific field for identifying the end of the set of serialized data.
  • the group of serialized data reference may be made to the related content in the method embodiment corresponding to FIG. 2, and details are not described herein again.
  • the bigquery platform and the redshift platform only provide some data transfer functions.
  • the redshift platform cannot support direct uploading of serialized data to the platform, and neither the bigquery platform nor the redshift platform supports serialized data download from the platform.
  • the corresponding embodiments of FIG. 5 and FIG. 6 respectively enable uploading and downloading of serialized data, which makes up for the deficiencies of the above platform.
  • the corresponding embodiment of FIG. 5 and FIG. 6 can be implemented based on the HTTP protocol, that is, the client and the server communicate by using an HTTP request.
  • the embodiment of the present application further provides an apparatus embodiment corresponding to the foregoing method embodiment, which is specifically described below.
  • an embodiment of the present application provides an apparatus embodiment of a serialization apparatus, and this embodiment corresponds to The method embodiment shown in Figure 2.
  • the apparatus of this embodiment includes: a data acquisition unit 701, an attribute value acquisition unit 702, and a data generation unit 703.
  • the data acquisition unit 701 is configured to acquire n structured data, n ⁇ 1.
  • the attribute value obtaining unit 702 is configured to acquire an attribute value group respectively corresponding to the n structured data.
  • the structured data order 1 corresponds to the attribute value group a
  • the attribute value group a specifically includes: a1, a2, and a3
  • the structured data order 2 corresponds to the attribute value group b
  • the attribute value group b specifically includes: b1, b2, and b3.
  • the data generating unit 703 is configured to generate a set of serialized data according to the attribute value groups respectively corresponding to the n structured data.
  • the serialized data includes two parts.
  • the first part consists of n records, namely Row 1, Row 2, ..., Row n in the first row of Figure 3.
  • Each record corresponds to one structured data, that is, each record stores a set of attribute values of the structured data corresponding to the record.
  • the ith record 1 ⁇ i ⁇ n
  • the ith record includes mi+1 fields, mi ⁇ 1
  • the data of the ith record is stored in the value of the first mi fields
  • the ith The data group of the record is the attribute value group corresponding to the i-th structured data in the n structured data
  • the first specific identifier is stored in the tag of the mi+1 field, and the first specific identifier is used to identify the The end of the record of the i-th record.
  • the first specific identifier is stored in the field_number of the m1+1th field, and can be distinguished from the field_number in the field for storing data.
  • the first specific identifier may be a relatively large number, that is, a number that cannot be reached by the field_number in the field for storing data.
  • the first specific identifier may be a value greater than a preset threshold, and the preset threshold is determined according to a maximum available range of the field_number of the tag.
  • the first specific identifier may be 2 to the 25th power minus 1, ie (2 ⁇ 25)-1.
  • the serialized data further includes a second part, that is, a specific field, a tag of the specific field stores a second specific identifier, and the second specific identifier is used to identify the end of the serialized data.
  • the first specific identifier and the second specific identifier are different.
  • the second specific identifier may be a value greater than a preset threshold, and the preset threshold is determined according to a maximum available range of the field_number of the tag.
  • the second specific identifier may be (2 ⁇ 25)-1024 and/or (2 ⁇ 25)-2.
  • the number of fields of each record in the serialized data may be different, but the missing field is added in the deserialization, so that the number of fields of each record is consistent, where The value in the filled field is set to null and you don't have to encode it.
  • the school when the serialized data is generated, the school may also be added to the serialized data.
  • the code is verified to support the data verification of the serialization process and improve the security of the data. The details are described below.
  • the n records respectively store check values corresponding to the respective records; wherein, in the ith record, the value information of the mi+1th domain stores the corresponding corresponding to the i th record a check value, where the check value corresponding to the ith record is obtained according to the attribute value group corresponding to the ith structured data.
  • the specific domain includes a first specific domain and/or a second specific domain.
  • the tag information of the first specific domain stores a first sub-identity
  • the value information stores a total check value, where the total check value is obtained according to the check values corresponding to the n records respectively.
  • the tag information of the second specific domain stores a second sub-identity
  • the value information stores a total number of records, and the total number of records is specifically n.
  • an embodiment of the present application provides an apparatus embodiment of a deserialization apparatus, and this embodiment corresponds to the method embodiment shown in FIG.
  • the apparatus of this embodiment includes: a data acquisition unit 801, an attribute value acquisition unit 802, and a data generation unit 803.
  • the data obtaining unit 801 is configured to acquire a set of serialized data.
  • the serialized data is specifically the serialized data generated by the embodiment corresponding to FIG. 2 .
  • the serialized data includes two parts, and the first part includes n records.
  • the i-th record includes mi+1 fields, 1 ⁇ i ⁇ n, n ⁇ 1, mi ⁇ 1, and the data group of the i-th record is stored in the value of the first mi fields, the mi+1
  • the first specific identifier is stored in the tag of the field, and the first specific identifier is used to identify the end of the ith record.
  • the serialized data further includes a second part, that is, a specific field, the tag of the specific field stores a second specific identifier, and the second specific identifier is used to identify the serialized data.
  • the attribute value obtaining unit 802 is configured to obtain, from the serialized data, an attribute value group corresponding to the n structured data, where the attribute value group corresponding to the i-th structured data is recorded from the ith data Get in the group.
  • the data group recorded by the ith column may be directly used as the attribute value group corresponding to the i-th structured data.
  • the data generating unit 803 is configured to generate the n structured data according to the attribute value groups respectively corresponding to the n structured data.
  • the first specific identifier and the second specific identifier are both values greater than a preset threshold, and the preset threshold is determined according to a maximum available range of the field_number of the tag.
  • data verification may also be performed according to the check code in the serialized data to improve data security. The details are described below.
  • the embodiment further includes: a check value obtaining unit, configured to obtain, from the serialized data, a check value corresponding to each of the n records, where the check value corresponding to the ith record is from The mi+1 in the ith record
  • the first check unit is configured to perform the serialized data according to the attribute value group corresponding to the n structured data and the check value corresponding to the n records respectively. check. If all the n records are successfully verified, it indicates that the serialized data verification is successful.
  • the apparatus further includes: a second check unit, configured to obtain a total check from the value information of the first specific domain The value is verified according to the check value corresponding to the n records and the total check value.
  • the apparatus further includes: a third check unit, configured to obtain a total number of records from the value information of the second specific domain, according to the The total number of records and the number of records included in the serialized data, and the serialized data is verified;
  • the first sub-identity is stored in the label information of the first specific domain
  • the second sub-ID is stored in the label information of the second specific domain.
  • an embodiment of the present application provides an apparatus embodiment of a server, and this embodiment corresponds to the method embodiment shown in FIG. 5 .
  • the server of this embodiment includes a creating unit 901, a transmitting unit 902, a receiving unit 903, and a saving unit 904.
  • a creating unit 901 is used to create a session.
  • the server can set the state of the session to be on.
  • the state of the session is on, it indicates that the data corresponding to the session can be stored.
  • the sending unit 902 is configured to send the session identifier of the session to the client.
  • the state of the session may be shared between the server and the client, and therefore, the server may also notify the client of the status of the session.
  • the receiving unit 903 is configured to receive a plurality of data blocks that are sent by the client in a distributed manner.
  • each data block is associated with the session identifier, in fact, is associated with the session.
  • Each data block includes one or more sets of serialized data. If each data block includes multiple sets of serialized data, an end identifier is added at the end of the multiple sets of serialized data to identify the end of the data block. .
  • the server since the server establishes a session and the session is associated with the plurality of data blocks, the client can be supported to distribute a plurality of data blocks distributed, thereby reducing transmission time.
  • the receiving unit 903 is further configured to receive a data block save list sent by the client.
  • the data block save list is used to identify all data blocks to be saved. In fact, the data block save list actually identifies all the data blocks that the client has sliced the serialized data.
  • the saving unit 904 is configured to save the plurality of data blocks if the plurality of data blocks match the data block save list.
  • the server matches the received plurality of data blocks with the list, and actually determines all the data identified by the list. Whether the block is consistent with the received multiple data blocks, if the matching is successful, the matching unit is considered to be successful.
  • the saving unit 904 commits the received multiple data blocks, and actually saves the received multiple data blocks.
  • the server can also set the state of the session to off. When the state of the session is off, it indicates that the data corresponding to the session cannot be stored. The server can also notify the client of the status of the session.
  • the sending unit 902 is further configured to send, to the client, a data block missing list, where the data block missing list is used to identify that the data belongs to the data
  • the block saves the list and does not belong to the data block of the plurality of data blocks
  • the receiving unit 903 is further configured to receive the data block identified in the missing list of the data block that is sent by the client.
  • the server can perform a rematch based on the newly received data block. If the same data block is received multiple times, the last received one will prevail.
  • any one of the one or more sets of serialized data may be the specific structure shown in FIG. 3.
  • the n records are included, wherein the ith record includes mi+1 fields, 1 ⁇ i ⁇ n, n ⁇ 1, mi ⁇ 1, and the i-th record is stored in the value of the first mi fields.
  • the tag of the mi+1 field is stored with a first specific identifier for identifying the end of the ith record
  • the set of serialized data further includes a specific field after the n records.
  • the tag of the specific field stores a second specific identifier for identifying the end of the set of serialized data.
  • an embodiment of the present application provides another embodiment of a server.
  • This embodiment corresponds to the method embodiment shown in FIG. 6.
  • the server of this embodiment includes: a transmitting unit 1001 and a receiving unit 1002.
  • the sending unit 1001 is configured to send the total number of groups of the saved serialized data to the client;
  • the transmitting unit 1001 transmits to the client, and currently stores a total of 50 sets of serialized data information.
  • the receiving unit 1002 is configured to receive download information sent by the client, where the download information indicates a group number identifier of the serialized data to be downloaded by the client.
  • the download information may include binary information (offset, count), where offset indicates The number of starting groups of serialized data to be downloaded, and count indicates the total number of serialized data to be downloaded. For example, if the binary group information is (10, 10), it means that the serialized data of the 10th group to the 19th group is downloaded.
  • the sending unit 1001 is further configured to distribute the serialized data corresponding to the group number identifier to the client in a distributed manner.
  • the server distributes serialized data to the client distributedly, thereby reducing the time of data transmission.
  • the server may establish an index, and sort all the serialized data, so that the client can specify the binary information for downloading, wherein the index is associated with the session established by the server.
  • any of the set of serialized data in the saved serialized data may have the structure shown in FIG. Specifically, the n records are included, wherein the ith record includes mi+1 fields, 1 ⁇ i ⁇ n, n ⁇ 1, mi ⁇ 1, and the i-th record is stored in the value of the first mi fields. a data group, the tag of the mi+1 field is stored with a first specific identifier for identifying an end of the ith record, the set of serialized data further including a specific field after the n records, A second specific identifier is stored in the tag of the specific field for identifying the end of the set of serialized data.
  • the group of serialized data reference may be made to the related content in the method embodiment corresponding to FIG. 2, and details are not described herein again.
  • FIG. 9 and FIG. 10 can be implemented based on the HTTP protocol, that is, the client and the server communicate by using an HTTP request.
  • the disclosed system, apparatus, and method may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to implement the solution of the embodiment. purpose.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
  • the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium.
  • a computer readable storage medium A number of instructions are included to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present application.
  • the foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请提供了一种结构化数据的序列化、反序列化方法及装置，所述方法包括：获取n个结构化数据分别对应的属性值组；根据所述n个结构化数据分别对应的属性值组，生成一组序列化数据；所述序列化数据包括n条记录，其中，第i条记录包括mi+1个域，前mi个域的值信息中存储有第i个结构化数据对应的属性值组，第mi+1个域的标签信息中存储有第一特定标识，用于标识所述第i条记录的结束；所述序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息中存储有第二特定标识，用于标识所述序列化数据的结束。可见，本申请中不再采用message对象的结构，解决了由于message对象的大小限制而导致的无法实现序列化的问题。

Description

一种结构化数据的序列化、反序列化方法及装置
本申请要求2015年11月30日递交的申请号为201510857451.1、发明名称为“一种结构化数据的序列化、反序列化方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，尤其是涉及一种结构化数据的序列化、反序列化方法及装置。
背景技术
随着大数据时代的到来，越来越多的服务商发布了结构化数据的处理平台，这些处理平台能够对结构化数据提供序列化和反序列化处理。结构化数据的序列化指的是将原始的结构化数据转换成具有特定格式的序列化数据，以方便根据序列化数据进行传输等操作。而反序列化是序列化的逆过程，能够将具有特定格式的序列化数据恢复成原始的结构化数据。
目前，一种常用的序列化方法基于谷歌公司提出的protobuf协议，基于该协议将至少一个结构化数据进行序列化时，具体的序列化过程包括：各个结构化数据分别具有对应的属性值集合，将所有结构化数据转换成一个message对象，其中，该message对象包括至少一个域(field)，每个field包括标签信息(tag)和值信息(value)，在各个field的value中存储各个结构化数据的属性值集合。
例如，结构化数据order1具有属性值集合：a1、a2和a3，结构化数据order2具有属性值集合：b1、b2和b3，根据protobuf协议的规定，将结构化数据order1和order2转换为message对象a，message对象a具有6个field，这6个field的value中分别存储有a1、a2、a3、b1、b2和b3。
然而，在这种序列化方式中，由于需要将结构化数据转换成message对象，而message对象的大小具有一定的限制，例如需要满足小于一定的字节，上述限制可能会导致无法实现序列化。
发明内容
本申请解决的技术问题在于提供一种结构化数据的序列化、反序列化方法及装置， 以解决由于message对象的大小限制而导致的无法实现序列化的问题。
为此，本申请解决技术问题的技术方案是：
本申请提供了一种结构化数据的序列化方法，包括：
获取n个结构化数据，n≥1；
获取所述n个结构化数据分别对应的属性值组；
根据所述n个结构化数据分别对应的属性值组，生成一组序列化数据；
所述序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，mi≥1，前mi个域的值信息中存储有所述第i条记录的数据组，所述数据组为所述n个结构化数据中第i个结构化数据对应的属性值组，第mi+1个域的标签信息中存储有第一特定标识，用于标识所述第i条记录的结束；所述序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息中存储有第二特定标识，用于标识所述序列化数据的结束。
可选的，所述n条记录中分别存储有各个记录对应的校验值；其中，所述第i条记录中，第mi+1个域的值信息存储有所述第i条记录对应的校验值，所述第i条记录对应的校验值根据所述第i个结构化数据对应的属性值组获取。
可选的，所述特定域包括第一特定域和/或第二特定域；
所述第一特定域的标签信息存储有第一子标识，值信息存储有总校验值，所述总校验值根据所述n条记录分别对应的校验值获取；
所述第二特定域的标签信息存储有第二子标识，值信息存储有总记录数，所述总记录数具体为n。
可选的，所述第一特定标识和所述第二特定标识均为大于预设阈值的数值，所述预设阈值根据标签信息的域标识的最大可用范围确定。
本申请提供了一种序列化数据的反序列化方法，包括：
获取一组序列化数据，所述序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息中存储有所述第i条记录的数据组，第mi+1个域的标签信息中存储有第一特定标识，用于标识所述第i条记录的结束，所述序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识所述序列化数据的结束；
从所述序列化数据中获取n个结构化数据分别对应的属性值组，其中，第i个结构化数据对应的属性值组从所述第i条记录的数据组中获取；
根据所述n个结构化数据分别对应的属性值组，生成所述n个结构化数据。
可选的，还包括：
从所述序列化数据中获取n条记录分别对应的校验值，其中，所述第i条记录对应的校验值从所述第i条记录中第mi+1个域的值信息中获取；
根据所述n个结构化数据分别对应的属性值组以及所述n条记录分别对应的校验值，对所述序列化数据进行校验。
可选的，若在所述n条记录之后的特定域包括第一特定域，所述方法还包括：从所述第一特定域的值信息中获取总校验值，根据所述n条记录分别对应的校验值以及所述总校验值，对所述序列化数据进行校验；
若在所述n条记录之后的特定域包括第二特定域，所述方法还包括：从所述第二特定域的值信息中获取总记录数，根据所述总记录数以及所述序列化数据包括的记录数，对所述序列化数据进行校验；
其中，所述第一特定域的标签信息中存储有第一子标识，所述第二特定域的标签信息中存储有第二子标识。
可选的，所述第一特定标识和所述第二特定标识均为大于预设阈值的数值，所述预设阈值根据标签信息的域标识的最大可用范围确定。
本申请提供了一种序列化数据的保存方法，包括：
服务器创建会话；
所述服务器将所述会话的会话标识发送至客户端；
所述服务器接收所述客户端分布式地发送的多个数据块，每个数据块与所述会话标识相关联，每个数据块包括一组或多组序列化数据；
所述服务器接收所述客户端发送的数据块保存列表，所述数据块保存列表用于标识出待保存的所有数据块；
若所述多个数据块与所述数据块保存列表匹配，所述服务器保存所述多个数据块。
可选的，所述一组或多组序列化数据中的任一组序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息存储有所述第i条记录的数据组，第mi+1个域的标签信息存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识该组序列化数据的结束。
可选的，还包括：
若所述多个数据块与所述数据块保存列表不匹配，所述服务器向所述客户端发送数 据块缺失列表，所述数据块缺失列表用于标识出属于所述数据块保存列表且不属于所述多个数据块的数据块；
所述服务器接收所述客户端分布式地发送的所述数据块缺少列表中标识的数据块。
本申请提供了一种序列化数据的下载方法，包括：服务器将保存的序列化数据的总组数发送至客户端；
所述服务器接收所述客户端发送的下载信息，所述下载信息指示出所述客户端待下载的序列化数据的组数标识；
所述服务器将所述组数标识对应的序列化数据分布式地发送至所述客户端。
可选的，所述保存的序列化数据中的任一组序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息存储有所述第i条记录的数据组，第mi+1个域的标签信息存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识该组序列化数据的结束。
本申请提供了一种序列化装置，包括：
数据获取单元，用于获取n个结构化数据，n≥1；
属性值获取单元，用于获取所述n个结构化数据分别对应的属性值组；
数据生成单元，用于根据所述n个结构化数据分别对应的属性值组，生成一组序列化数据；
所述序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，mi≥1，前mi个域的值信息中存储有所述第i条记录的数据组，所述数据组为所述n个结构化数据中第i个结构化数据对应的属性值组，第mi+1个域的标签信息中存储有第一特定标识，用于标识所述第i条记录的结束；所述序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息中存储有第二特定标识，用于标识所述序列化数据的结束。
可选的，所述n条记录中分别存储有各个记录对应的校验值；其中，所述第i条记录中，第mi+1个域的值信息存储有所述第i条记录对应的校验值，所述第i条记录对应的校验值根据所述第i个结构化数据对应的属性值组获取。
可选的，所述特定域包括第一特定域和/或第二特定域；
所述第一特定域的标签信息存储有第一子标识，值信息存储有总校验值，所述总校验值根据所述n条记录分别对应的校验值获取；
所述第二特定域的标签信息存储有第二子标识，值信息存储有总记录数，所述总记 录数具体为n。
可选的，所述第一特定标识和所述第二特定标识均为大于预设阈值的数值，所述预设阈值根据标签信息的域标识的最大可用范围确定。
本申请提供了一种反序列化装置，包括：
数据获取单元，用于获取一组序列化数据，所述序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息中存储有所述第i条记录的数据组，第mi+1个域的标签信息中存储有第一特定标识，用于标识所述第i条记录的结束，所述序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识所述序列化数据的结束；
属性值获取单元，用于从所述序列化数据中获取n个结构化数据分别对应的属性值组，其中，第i个结构化数据对应的属性值组从所述第i条记录的数据组中获取；
数据生成单元，用于根据所述n个结构化数据分别对应的属性值组，生成所述n个结构化数据。
可选的，还包括：
校验值获取单元，用于从所述序列化数据中获取n条记录分别对应的校验值，其中，所述第i条记录对应的校验值从所述第i条记录中第mi+1个域的值信息中获取；
第一校验单元，用于根据所述n个结构化数据分别对应的属性值组以及所述n条记录分别对应的校验值，对所述序列化数据进行校验。
可选的，若在所述n条记录之后的特定域包括第一特定域，所述装置还包括：第二校验单元，用于从所述第一特定域的值信息中获取总校验值，根据所述n条记录分别对应的校验值以及所述总校验值，对所述序列化数据进行校验；
若在所述n条记录之后的特定域包括第二特定域，所述装置还包括：第三校验单元，用于从所述第二特定域的值信息中获取总记录数，根据所述总记录数以及所述序列化数据包括的记录数，对所述序列化数据进行校验；
其中，所述第一特定域的标签信息中存储有第一子标识，所述第二特定域的标签信息中存储有第二子标识。
可选的，所述第一特定标识和所述第二特定标识均为大于预设阈值的数值，所述预设阈值根据标签信息的域标识的最大可用范围确定。
本申请提供了一种服务器，包括：
创建单元，用于创建会话；
发送单元，用于将所述会话的会话标识发送至客户端；
接收单元，用于接收所述客户端分布式地发送的多个数据块，每个数据块与所述会话标识相关联，每个数据块包括一组或多组序列化数据，以及接收所述客户端发送的数据块保存列表，所述数据块保存列表用于标识出待保存的所有数据块；
保存单元，用于若所述多个数据块与所述数据块保存列表匹配，保存所述多个数据块。
可选的，所述一组或多组序列化数据中的任一组序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息存储有所述第i条记录的数据组，第mi+1个域的标签信息存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识该组序列化数据的结束。
可选的，所述发送单元还用于，若所述多个数据块与所述数据块保存列表不匹配，向所述客户端发送数据块缺失列表，所述数据块缺失列表用于标识出属于所述数据块保存列表且不属于所述多个数据块的数据块；
所述接收单元还用于，接收所述客户端分布式地发送的所述数据块缺少列表中标识的数据块。
本申请提供了一种服务器，包括：发送单元和接收单元；
所述发送单元，用于将保存的序列化数据的总组数发送至客户端；
所述接收单元，用于接收所述客户端发送的下载信息，所述下载信息指示出所述客户端待下载的序列化数据的组数标识；
所述发送单元，还用于将所述组数标识对应的序列化数据分布式地发送至所述客户端。
可选的，所述保存的序列化数据中的任一组序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息存储有所述第i条记录的数据组，第mi+1个域的标签信息存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识该组序列化数据的结束。
通过上述技术方案可知，本申请实施例中，将n个结构化数据转换成一组序列化数据，其中该组序列化数据包括两部分，第一部分包括n条记录，每条记录对应一个结构化数据，其中，第i条记录包括mi+1个field，前mi个field的value中存储有第i个结 构化数据对应的属性值组，第mi+1个field的tag中存储有第一特定标识，用于标识所述第i条记录的结束；而第二部分，也就是所述n条记录之后还包括特定field，所述特定field的tag中存储有第二特定标识，用于标识所述序列化数据的结束。可见，本申请中不再采用message对象的结构，而是将n个结构化数据转换成n条记录，每条记录的最后1个field用来标识该条记录的结束，从而能够区分不同的记录，而n条记录的之后通过特定field标识所述序列化数据的结束。因此，解决了由于message对象的大小限制而导致的无法实现序列化的问题。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，还可以根据这些附图获得其它的附图。
图1为protobuf协议中field的结构示意图；
图2为本申请提供的序列化方法的一种实施例的流程示意图；
图3为本申请提供序列化数据的结构示意图；
图4为本申请提供的反序列化方法的一种实施例的流程示意图；
图5为本申请提供的保存方法的一种实施例的流程示意图；
图6为本申请提供的下载方法的一种实施例的流程示意图；
图7为本申请提供的序列化装置的一种装置实施例的结构示意图；
图8为本申请提供的反序列化装置的一种装置实施例的结构示意图；
图9为本申请提供的服务器的一种装置实施例的结构示意图；
图10为本申请提供的服务器的另一种装置实施例的结构示意图。
具体实施方式
一种常用的序列化方法基于谷歌公司提出的protobuf协议，下面对该协议进行说明。
protobuf协议基于message对象的结构之上，而message对象包括至少一个field，如图1所示，一个field包括一个tag和value。
其中，tag的长度为4个字节，包括域标识(field_number)和值类型(value_t ype)，field_number用于标识field，value_type用于说明value中的数据类型，value中用于存储数据。
基于protobuf协议将至少一个结构化数据进行序列化时，将所有结构化数据转换成一个message对象，在其中一个message对象的各个value中存储对应的结构化数据的属性值集合。例如，结构化数据order1具有属性值集合：a1、a2和a3，结构化数据order2具有属性值集合：b1、b2和b3，根据protobuf协议的规定，将结构化数据order1和order2转换为message对象a，message对象a具有6个field，这6个field的value中分别存储有a1、a2、a3、b1、b2和b3。
然而，在这种序列化方式中，由于需要将结构化数据转换成message对象，而message对象的大小具有一定的限制，例如需要满足小于一定的字节，上述限制可能会导致无法实现序列化。
此外，protobuf协议也不支持序列化过程的数据校验，因此数据的安全性无法得到保障。
本申请实施例提供一种结构化数据的序列化、反序列化方法及装置，以解决由于message对象的大小限制而导致的无法实现序列化的问题。
为了使本技术领域的人员更好地理解本申请中的技术方案，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都应当属于本发明保护的范围。
请参阅图2，本申请提供了结构化数据的序列化方法的一种方法实施例，本实施例的所述方法包括：
201：获取n个结构化数据，n≥1。
例如，获取两个结构化数据order1和order2。
202：获取所述n个结构化数据分别对应的属性值组。其中，每个属性值组中包括至少一个属性值。
例如，结构化数据order1对应属性值组a，属性值组a具体包括：a1、a2和a3；结构化数据order2对应属性值组b，属性值组b具体包括：b1、b2和b3。
203：根据所述n个结构化数据分别对应的属性值组，生成一组序列化数据。
如图3所示，所述序列化数据包括两部分。第一部分包括n条记录，即图3第1行中的Row 1、Row 2、……、Row n。每条记录对应一个结构化数据，即每条记录存储有该条记录对应的结构化数据的属性值组。
以第i条记录进行说明，1≤i≤n，第i条记录包括mi+1个field，mi≥1，前mi个field的value中存储有第i条记录的数据组，所述第i条记录的数据组为所述n个结构化数据中第i个结构化数据对应的属性值组，第mi+1个field的tag中存储有第一特定标识，第一特定标识用于标识所述第i条记录的结束。
例如图3所示，第1条记录(即Row1)包括m1+1个列(Column)，即图3所示的Column 1、Column 2、……、Column m1、Checksum 1，而一个列实际上就对应一个field，列的标识可以通过对应的field的field_number表示，前m1个列的标识可以从0至m1-1等差递增，或者也可以为乱序。在第1条记录中，前m1个Column的value中存储有第1条记录的数据组，即第1个结构化数据对应的属性值组(例如a1、a2和a3)。第m1+1个Column(即图3所示的Checksum 1)的tag中存储有第一特定标识，用于标识所述第1条记录的结束。
其中，第一特定标识存储在第m1+1个field的field_number中，与用于存储数据的field中的field_number能互相区分开。例如第i条记录中，前mi个field中的field_number从0开始等差递增，例如分别为0，1，2,……，mi-1，则第一特定标识可以为一个比较大的数，也就是用于存储数据的field中的field_number不可能达到的数。具体地，所述第一特定标识可以为大于预设阈值的数值，该预设阈值根据tag的field_number的最大可用范围确定。例如，第一特定标识可以为2的25次方减1，即(2^25)-1。
在第一部分即n条记录之后，所述序列化数据还包括第二部分，即特定field，特定field的tag中存储有第二特定标识，第二特定标识用于标识所述序列化数据的结束。其中，所述第一特定标识和所述第二特定标识不同。具体地，所述第二特定标识可以为大于预设阈值的数值，该预设阈值根据tag的field_number的最大可用范围确定。例如，第二特定标识可以为(2^25)-1024和/或(2^25)-2。
例如图3所示，在n条记录即Row 1、Row 2、……、Row n之后，所述序列化数据还包括特定field，即记录Checksum n和记录Total Row Count，其中记录Checksum n和记录Total Row Count各对应一个field。记录Checksum n的field_number和记录Total Row Count的field_number即作为所述第二特定标识。其中，图3所示的特定field也可以只包括记录Checksum n和记录Total Row Count中的任一个。
需要说明的是，本申请实施例中，序列化数据中各个记录的field的个数可以不同，但是反序列化时会补上缺少的field，使得各个记录的field的个数一致，其中，补上的field中的value设置为null，不用对其进行编码。
通过上述技术方案可知，本申请实施例中，将n个结构化数据转换成一组序列化数据，其中该组序列化数据包括两部分，第一部分包括n条记录，每条记录对应一个结构化数据，其中，第i条记录包括mi+1个field，前mi个field的value中存储有第i个结构化数据对应的属性值组，第mi+1个field的tag中存储有第一特定标识，用于标识所述第i条记录的结束；而第二部分，也就是所述n条记录之后还包括特定field，所述特定field的tag中存储有第二特定标识，用于标识所述序列化数据的结束。可见，本申请中不再采用message对象的结构，而是将n个结构化数据转换成n条记录，每条记录的最后1个field用来标识该条记录的结束，从而能够区分不同的记录，而n条记录之后通过特定field标识所述序列化数据的结束。因此，解决了由于message对象的大小限制而导致的无法实现序列化的问题。
在本申请实施例中，在生成所述序列化数据时，还可以在所述序列化数据中加入校验码，从而支持序列化过程的数据校验，提高数据的安全性。下面具体说明。
可选的，所述n条记录中分别存储有各个记录对应的校验值。其中，在第i条记录中，第mi+1个field的value中存储有第i条记录对应的校验值，第i条记录对应的校验值根据第i个结构化数据对应的属性值组获取。例如，根据第i个结构化数据对应的属性值组中的各个属性值，分别计算对应的冗余校验码，例如crc32，将计算出的各个冗余校验码之和，作为第i条记录对应的校验值，存储在第mi+1个field的value中，例如图3所示的Checksum 1的value中。第mi+1个field的value_type可以为protobuf协议定义的32-bit。
可选的，所述特定field中存储有校验值。需要说明的是，所述序列化数据中的所述特定field的可以为一个field，也可以为多个field。例如，所述特定field包括第一field和/或第二field。其中，第一field的tag中存储有第一子标识，value中存储有总校验值，所述总校验值根据所述n条记录分别对应的校验值获取，例如为各个记录对应的校验值之和。第二field的tag中存储有第二子标识，value中存储有总记录数，所述总记录数具体为n。
例如图3所示，第一field可以为记录Checksum n对应的field，该field的field_number可以为(2^25)-1024，value_type可以为protobuf协议定义的32-bit。第二field可以为记录Total Row Count对应的field，该field的field_number可以为(2^25)-2，value_type可以为protobuf协议定义的64-bit。
其中，若所述特定field包括第一field且不包括第二field，则所述第一子标识作为 所述第二特定标识，若所述特定field包括第二field且不包括第一field，则所述第二子标识作为所述第二特定标识，若所述特定field包括第一field且包括第二field，则所述第一子标识和所述第二子标识作为所述第二特定标识。
下面对上述实施例对应的反序列化过程进行说明。
请参阅图4，本申请提供了序列化数据的反序列化方法的一种方法实施例，本实施例的所述方法包括：
401：获取一组序列化数据。
其中，所述序列化数据具体为图2对应的实施例所生成的序列化数据。如图3所示，所述序列化数据包括两部分，第一部分包括n条记录。其中，第i条记录包括mi+1个field，1≤i≤n，n≥1，mi≥1，前mi个field的value中存储有所述第i条记录的数据组，第mi+1个field的tag中存储有第一特定标识，第一特定标识用于标识所述第i条记录的结束。在第一部分即n条记录之后，所述序列化数据还包括第二部分，即特定field，所述特定field的tag存储有第二特定标识，第二特定标识用于标识所述序列化数据的结束。
402：从所述序列化数据中获取n个结构化数据分别对应的属性值组。其中，第i个结构化数据对应的属性值组从所述第i条记录的数据组中获取。
在本实施例中，可以直接将所述第i条记录的数据组作为第i个结构化数据对应的属性值组。
403：根据所述n个结构化数据分别对应的属性值组，生成所述n个结构化数据。
通过上述技术方案可知，本申请实施例的序列化数据不再采用message对象的结构，而是通过n条记录存储n个结构化数据的属性值组，每条记录的最后1个field用来标识该条记录的结束，从而能够区分不同的记录，而n条记录的之后通过特定field标识所述序列化数据的结束。因此，结构化数据的属性值组中的属性值不再受到个数的限制，解决了由于message对象的大小限制而导致的无法实现序列化的问题。
可选的，所述第一特定标识和所述第二特定标识均为大于预设阈值的数值，所述预设阈值根据tag的field_number的最大可用范围确定。
在本申请实施例中，还可以根据序列化数据中的校验码进行数据校验，提高数据的安全性。下面具体说明。
可选的，所述方法还包括：从所述序列化数据中获取n条记录分别对应的校验值，其中，所述第i条记录对应的校验值从所述第i条记录中第mi+1个field的value中获取；根据所述n个结构化数据分别对应的属性值组以及所述n条记录分别对应的校验值，对 所述序列化数据进行校验。若所述n条记录全部校验成功，则表示所述序列化数据校验成功。
以第i条记录的验证进行说明，根据第i个结构化数据对应的属性值组中的各个属性值，分别计算对应的冗余校验码，并计算各个冗余校验码之和，若计算出的各个冗余校验码之和与第i条记录对应的校验值一致，则表示第i条记录校验成功。
可选的，若在所述n条记录之后的特定field包括第一特定field，所述方法还包括：从所述第一特定field的value中获取总校验值，根据所述n条记录分别对应的校验值以及所述总校验值，对所述序列化数据进行校验。例如，计算n条记录分别对应的校验值之和，若所述校验值之和与所述总校验值一致，则表示所述序列化数据校验成功。
若在所述n条记录之后的特定field包括第二特定field，所述方法还包括：从所述第二特定field的value中获取总记录数，根据所述总记录数以及所述序列化数据包括的记录数，对所述序列化数据进行校验。例如，获取所述序列化数据包括的记录数，若与所述总记录数一致，则表示所述序列化数据校验成功。
其中，所述第一特定field的tag中存储有第一子标识，所述第二特定field的tag中存储有第二子标识。
随着大数据时代的到来，越来越多的服务商发布了结构化数据的处理平台，例如谷歌公司的bigquery平台和亚马逊公司的redshift平台。这些平台能够将结构化数据转换成序列化数据，从而进行上传和下载等传输操作。
目前，这些平台在对序列化数据进行上传时，不能实现all or nothing原则，即不能实现要么全部上传成功，要么全部都不上传，使得上传过程中数据的一致性不能得到保证。
因此，在本申请实施例中还提供了一种序列化数据的保存方法，及服务器，以实现并行地上传序列化数据，从而减少传输时间。
请参阅图5，本申请提供了序列化数据的保存方法的一种方法实施例，应用于服务器端。本实施例的所述方法包括：501、502、503、504和505。
501：服务器创建会话(session)。
其中，所述服务器可以将会话的状态设置为开启。当会话的状态为开启时，表示可以对该会话对应的数据进行存储操作。
502：所述服务器将所述会话的会话标识(session ID)发送至客户端。
其中，所述会话的状态可以在服务器和客户端共享，因此，服务器还可以将所述会话的状态通知所述客户端。
601：所述客户端接收服务器发送的会话标识。
602：所述客户端对序列化数据进行切分，得到切分后的多个数据块(block)。
其中，每个数据块与所述会话标识相关联，实际上也就是与所述会话相关联。每个数据块包括一组或多组序列化数据，若每个数据块包括多组序列化数据，则在该多组序列化数据的结尾，需要加入结束标识，用于标识该数据块的结束。
603：所述客户端分布式地发送切分后的多个数据块。
在本实施例中，由于所述服务器建立了会话，并且所述会话与所述多个数据块相关联，因此能够支持所述客户端分布式的发送多个数据块，从而减少传输的时间。
503：所述服务器接收所述客户端分布式地发送的多个数据块。
604：所述客户端发送数据块保存列表。
其中，所述数据块保存列表用于标识出待保存的所有数据块。实际上，所述数据块保存列表实际上标识出，客户端对序列化数据切分后的所有数据块。
504：所述服务器接收所述客户端发送的数据块保存列表。
需要说明的是，步骤503和504的执行顺序不受限定，也就是说可以先执行503再执行504，或者先执行504再执行503，或者同时执行503和504。
505：若接收到的多个数据块与所述数据块保存列表匹配，所述服务器保存接收到的多个数据块。
505可以包括子步骤5051和5052。
5051：由于数据需要保证强一致性，也就是说接收到的多个数据块要么全部保存，要么就一个都不保存。因此，所述服务器接收到多个数据块和所述数据块保存列表后，会将接收到的多个数据块和所述列表进行匹配，实际上就是判断所述列表标识的所有数据块与接收到的多个数据块是否一致，如果一致则认为匹配成功，此时执行5052。
5052：所述服务器提交(commit)接收到的多个数据块，实际上即保存接收到的多个数据块。
所述服务器还可以将所述会话的状态设置为关闭。当会话的状态为关闭时，表示不可以对该会话对应的数据进行存储操作。服务器还可以将所述会话的状态通知所述客户端。
若步骤5051中，判断出所述多个数据块和所述列表不匹配，则所述服务器向所述客 户端发送数据块缺失列表，所述数据块缺失列表用于标识出属于所述数据块保存列表且不属于所述多个数据块的数据块；所述服务器接收所述客户端分布式地发送的所述数据块缺少列表中标识的数据块。所述服务器可以根据新接收到的数据块进行重新匹配。如果同一个数据块接收到多次，以最后一次接收到的为准。
通过上述技术方案可知，在本申请实施例中，通过建立会话，将会话与序列化数据切分成的多个数据块相关联，并且服务器会接收到数据块保存列表，指示出待保存的所有数据块，使得服务器能够接收客户端分布式地发送的多个数据块，并根据接收到的多个数据块和所述列表，判断接收到的多个数据块和所述列表是否匹配，也就是说所述服务器是否已经接收到所述列表标识的所有数据块，如果匹配，则说明服务器已经接收到所述列表标识的所有数据块，因此服务器保存接收到的多个数据块。可见，本申请实施例中服务器接收客户端分布式地发送的多个数据块以及数据块保存列表，只有当接收到的多个数据块与数据块保存列表匹配时，才保存接收到的多个数据块，保证了数据块的一致性。
本实施例可以用于对本申请实施例提供的序列化数据进行上传，也可以对目前任一种序列化数据进行上传。具体地，所述一组或多组序列化数据中的任一组序列化数据可以为图3所示的具体结构。具体地，包括n条记录，其中，第i条记录包括mi+1个field，1≤i≤n，n≥1，mi≥1，前mi个field的value中存储有所述第i条记录的数据组，第mi+1个field的tag中存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定field，所述特定field的tag存储有第二特定标识，用于标识该组序列化数据的结束。该组序列化数据可以参照图2对应的方法实施例中的相关内容，这里不再赘述。
本申请实施例还提供了一种对序列化数据的下载方法，能够实现并行的传输。
请参阅图6，本申请提供了序列化数据的下载方法的一种方法实施例，应用于服务器端。本实施例的所述方法包括：701、702、703和704。
701：服务器创建会话。
其中，所述会话可以与后续过程生成的索引相关联，从而不用反复建立索引。
所述服务器还可以将所述会话的状态设置为开启。当会话的状态为开启时，表示可以对该会话对应的数据进行读取操作。其中，所述会话的状态可以在服务器和客户端共享，因此，服务器还可以将所述会话的状态通知所述客户端。
需要说明的是，本申请实施例中，所述服务器也可以不用建立会话。
701：所述服务器将保存的序列化数据的总组数发送至客户端。
例如，服务器向客户端发送，目前总共存储有50组序列化数据的信息。
801：所述客户端接收到服务器发送的总组数，根据所述总组数生成下载信息，所述下载信息中指示出待下载的序列化数据的组数标识。
例如，所述下载信息中可以包括二元组信息(offset，count)，其中，offset指示出待下载的序列化数据的起始组数，count指示出待下载的序列化数据的总组数。例如，所述二元组信息为(10，10)，则表示下载第10组-第19组的序列化数据。
其中，所述服务器可以建立索引，对所有的序列化数据进行排序，方便客户端指定二元组信息进行下载，其中所述索引与所述服务器建立的会话相关联。
802：所述客户端向所述服务器发送所述下载信息。
703：所述服务器接收所述客户端发送的下载信息。
704：所述服务器将所述组数标识对应的序列化数据分布式地发送至所述客户端。
在本申请实施例中，服务器向客户端分布式地发送序列化数据，从而减少了数据传输的时间。
本实施例可以用于对本申请实施例提供的序列化数据进行下载，也可以对目前任一种序列化数据进行下载。具体地，所述保存的序列化数据中的任一组序列化数据可以如图3所示的结构。具体地，包括n条记录，其中，第i条记录包括mi+1个field，1≤i≤n，n≥1，mi≥1，前mi个field的value中存储有所述第i条记录的数据组，第mi+1个field的tag存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定field，所述特定field的tag中存储有第二特定标识，用于标识该组序列化数据的结束。该组序列化数据可以参照图2对应的方法实施例中的相关内容，这里不再赘述。
目前，bigquery平台和redshift平台只提供了部分数据传输功能，例如redshift平台不能支持直接将序列化数据上传到该平台上，并且bigquery平台和redshift平台都不支持将序列化数据从该平台下载，而通过本申请实施例中，图5和图6分别对应的实施例，则能够实现对序列化数据的上传和下载，弥补了上述平台的不足。其中，图5和图6分别对应的实施例，可以基于HTTP协议实现，即客户端和服务器之间通过HTTP请求进行通信。
本申请实施例还提供了与上述方法实施例对应的装置实施例，下面具体说明。
请参阅图7，本申请实施例提供了序列化装置的一种装置实施例，本实施例对应于 图2所示的方法实施例。本实施例的所述装置包括：数据获取单元701、属性值获取单元702和数据生成单元703。
数据获取单元701用于获取n个结构化数据，n≥1。
例如，获取两个结构化数据order1和order2。
属性值获取单元702用于获取所述n个结构化数据分别对应的属性值组。
例如，结构化数据order1对应属性值组a，属性值组a具体包括：a1、a2和a3；结构化数据order2对应属性值组b，属性值组b具体包括：b1、b2和b3。
数据生成单元703用于根据所述n个结构化数据分别对应的属性值组，生成一组序列化数据。
如图3所示，所述序列化数据包括两部分。第一部分包括n条记录，即图3第1行中的Row 1、Row 2、……、Row n。每条记录对应一个结构化数据，即每条记录存储有该条记录对应的结构化数据的属性值组。
以第i条记录进行说明，1≤i≤n，第i条记录包括mi+1个field，mi≥1，前mi个field的value中存储有第i条记录的数据组，所述第i条记录的数据组为所述n个结构化数据中第i个结构化数据对应的属性值组，第mi+1个field的tag中存储有第一特定标识，第一特定标识用于标识所述第i条记录的结束。
其中，第一特定标识存储在第m1+1个field的field_number中，与用于存储数据的field中的field_number能够相区分开。例如第一特定标识可以为一个比较大的数，也就是用于存储数据的field中的field_number不可能达到的数。具体地，所述第一特定标识可以为大于预设阈值的数值，该预设阈值根据tag的field_number的最大可用范围确定。例如，第一特定标识可以为2的25次方减1，即(2^25)-1。
在第一部分即n条记录之后，所述序列化数据还包括第二部分，即特定field，特定field的tag中存储有第二特定标识，第二特定标识用于标识所述序列化数据的结束。其中，所述第一特定标识和所述第二特定标识不同。具体地，所述第二特定标识可以为大于预设阈值的数值，该预设阈值根据tag的field_number的最大可用范围确定。例如，第二特定标识可以为(2^25)-1024和/或(2^25)-2。
需要说明的是，本申请实施例中，序列化数据中各个记录的field的个数可以不同，但是反序列化时会补上缺少的field，使得各个记录的field的个数一致，其中，补上的field中的value设置为null，不用对其进行编码。
在本申请实施例中，在生成所述序列化数据时，还可以在所述序列化数据中加入校 验码，从而支持序列化过程的数据校验，提高数据的安全性。下面具体说明。
可选的，所述n条记录中分别存储有各个记录对应的校验值；其中，所述第i条记录中，第mi+1个域的值信息存储有所述第i条记录对应的校验值，所述第i条记录对应的校验值根据所述第i个结构化数据对应的属性值组获取。
可选的，所述特定域包括第一特定域和/或第二特定域。
所述第一特定域的标签信息存储有第一子标识，值信息存储有总校验值，所述总校验值根据所述n条记录分别对应的校验值获取。所述第二特定域的标签信息存储有第二子标识，值信息存储有总记录数，所述总记录数具体为n。
请参阅图8，本申请实施例提供了反序列化装置的一种装置实施例，本实施例对应于图4所示的方法实施例。本实施例的所述装置包括：数据获取单元801、属性值获取单元802和数据生成单元803。
数据获取单元801，用于获取一组序列化数据。
其中，所述序列化数据具体为图2对应的实施例所生成的序列化数据。如图3所示，所述序列化数据包括两部分，第一部分包括n条记录。其中，第i条记录包括mi+1个field，1≤i≤n，n≥1，mi≥1，前mi个field的value中存储有所述第i条记录的数据组，第mi+1个field的tag中存储有第一特定标识，第一特定标识用于标识所述第i条记录的结束。在第一部分即n条记录之后，所述序列化数据还包括第二部分，即特定field，所述特定field的tag存储有第二特定标识，第二特定标识用于标识所述序列化数据的结束。
属性值获取单元802，用于从所述序列化数据中获取n个结构化数据分别对应的属性值组，其中，第i个结构化数据对应的属性值组从所述第i条记录的数据组中获取。
在本实施例中，可以直接将所述第i条记录的数据组作为第i个结构化数据对应的属性值组。
数据生成单元803，用于根据所述n个结构化数据分别对应的属性值组，生成所述n个结构化数据。
可选的，所述第一特定标识和所述第二特定标识均为大于预设阈值的数值，所述预设阈值根据tag的field_number的最大可用范围确定。
在本申请实施例中，还可以根据序列化数据中的校验码进行数据校验，提高数据的安全性。下面具体说明。
可选的，本实施例还包括：校验值获取单元，用于从所述序列化数据中获取n条记录分别对应的校验值，其中，所述第i条记录对应的校验值从所述第i条记录中第mi+1 个域的值信息中获取；第一校验单元，用于根据所述n个结构化数据分别对应的属性值组以及所述n条记录分别对应的校验值，对所述序列化数据进行校验。若所述n条记录全部校验成功，则表示所述序列化数据校验成功。
可选的，若在所述n条记录之后的特定域包括第一特定域，所述装置还包括：第二校验单元，用于从所述第一特定域的值信息中获取总校验值，根据所述n条记录分别对应的校验值以及所述总校验值，对所述序列化数据进行校验。
若在所述n条记录之后的特定域包括第二特定域，所述装置还包括：第三校验单元，用于从所述第二特定域的值信息中获取总记录数，根据所述总记录数以及所述序列化数据包括的记录数，对所述序列化数据进行校验；
其中，所述第一特定域的标签信息中存储有第一子标识，所述第二特定域的标签信息中存储有第二子标识。
请参阅图9，本申请实施例提供了服务器的一种装置实施例，本实施例对应于图5所示的方法实施例。本实施例的所述服务器包括：创建单元901、发送单元902、接收单元903和保存单元904。
创建单元901，用于创建会话。
其中，所述服务器可以将会话的状态设置为开启。当会话的状态为开启时，表示可以对该会话对应的数据进行存储操作。
发送单元902，用于将所述会话的会话标识发送至客户端。
其中，所述会话的状态可以在服务器和客户端共享，因此，服务器还可以将所述会话的状态通知所述客户端。
接收单元903，用于接收所述客户端分布式地发送的多个数据块。
其中，每个数据块与所述会话标识相关联，实际上也就是与所述会话相关联。每个数据块包括一组或多组序列化数据，若每个数据块包括多组序列化数据，则在该多组序列化数据的结尾，需要加入结束标识，用于标识该数据块的结束。
在本实施例中，由于所述服务器建立了会话，并且所述会话与所述多个数据块相关联，因此能够支持所述客户端分布式的发送多个数据块，从而减少传输的时间。
接收单元903，还用于接收所述客户端发送的数据块保存列表。
其中，所述数据块保存列表用于标识出待保存的所有数据块。实际上，所述数据块保存列表实际上标识出，客户端对序列化数据切分后的所有数据块。
保存单元904，用于若所述多个数据块与所述数据块保存列表匹配，保存所述多个数据块。
由于数据需要保证强一致性，也就是说接收到的多个数据块要么全部保存，要么就一个都不保存。因此，接收单元903接收到多个数据块和所述数据块保存列表后，所述服务器会将接收到的多个数据块和所述列表进行匹配，实际上就是判断所述列表标识的所有数据块与接收到的多个数据块是否一致，如果一致则认为匹配成功，此时保存单元904提交(commit)接收到的多个数据块，实际上即保存接收到的多个数据块。
所述服务器还可以将所述会话的状态设置为关闭。当会话的状态为关闭时，表示不可以对该会话对应的数据进行存储操作。服务器还可以将所述会话的状态通知所述客户端。
可选的，若所述多个数据块和所述列表不匹配，发送单元902还用于，向所述客户端发送数据块缺失列表，所述数据块缺失列表用于标识出属于所述数据块保存列表且不属于所述多个数据块的数据块；接收单元903还用于，接收所述客户端分布式地发送的所述数据块缺少列表中标识的数据块。所述服务器可以根据新接收到的数据块进行重新匹配。如果同一个数据块接收到多次，以最后一次接收到的为准。
本实施例可以用于对本申请实施例提供的序列化数据进行上传，也可以对目前任一种序列化数据进行上传。具体地，所述一组或多组序列化数据中的任一组序列化数据可以为图3所示的具体结构。具体地，包括n条记录，其中，第i条记录包括mi+1个field，1≤i≤n，n≥1，mi≥1，前mi个field的value中存储有所述第i条记录的数据组，第mi+1个field的tag中存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定field，所述特定field的tag存储有第二特定标识，用于标识该组序列化数据的结束。该组序列化数据可以参照图2对应的方法实施例中的相关内容，这里不再赘述。
请参阅图10，本申请实施例提供了服务器的另一种装置实施例，本实施例对应于图6所示的方法实施例。本实施例的所述服务器包括：发送单元1001和接收单元1002。
发送单元1001，用于将保存的序列化数据的总组数发送至客户端；
例如，发送单元1001向客户端发送，目前总共存储有50组序列化数据的信息。
接收单元1002，用于接收所述客户端发送的下载信息，所述下载信息指示出所述客户端待下载的序列化数据的组数标识。
例如，所述下载信息中可以包括二元组信息(offset，count)，其中，offset指示出 待下载的序列化数据的起始组数，count指示出待下载的序列化数据的总组数。例如，所述二元组信息为(10，10)，则表示下载第10组-第19组的序列化数据。
发送单元1001，还用于将所述组数标识对应的序列化数据分布式地发送至所述客户端。
在本申请实施例中，服务器向客户端分布式地发送序列化数据，从而减少了数据传输的时间。
其中，所述服务器可以建立索引，对所有的序列化数据进行排序，方便客户端指定二元组信息进行下载，其中所述索引与所述服务器建立的会话相关联。
本实施例可以用于对本申请实施例提供的序列化数据进行下载，也可以对目前任一种序列化数据进行下载。具体地，所述保存的序列化数据中的任一组序列化数据可以如图3所示的结构。具体地，包括n条记录，其中，第i条记录包括mi+1个field，1≤i≤n，n≥1，mi≥1，前mi个field的value中存储有所述第i条记录的数据组，第mi+1个field的tag存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定field，所述特定field的tag中存储有第二特定标识，用于标识该组序列化数据的结束。该组序列化数据可以参照图2对应的方法实施例中的相关内容，这里不再赘述。
其中，图9和图10分别对应的实施例，可以基于HTTP协议实现，即客户端和服务器之间通过HTTP请求进行通信。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的 目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (26)

  1. 一种结构化数据的序列化方法，其特征在于，包括：
    获取n个结构化数据，n≥1；
    获取所述n个结构化数据分别对应的属性值组；
    根据所述n个结构化数据分别对应的属性值组，生成一组序列化数据；
    所述序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，mi≥1，前mi个域的值信息中存储有所述第i条记录的数据组，所述数据组为所述n个结构化数据中第i个结构化数据对应的属性值组，第mi+1个域的标签信息中存储有第一特定标识，用于标识所述第i条记录的结束；所述序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息中存储有第二特定标识，用于标识所述序列化数据的结束。
  2. 根据权利要求1所述的序列化方法，其特征在于，所述n条记录中分别存储有各个记录对应的校验值；其中，所述第i条记录中，第mi+1个域的值信息存储有所述第i条记录对应的校验值，所述第i条记录对应的校验值根据所述第i个结构化数据对应的属性值组获取。
  3. 根据权利要求2所述的序列化方法，其特征在于，所述特定域包括第一特定域和/或第二特定域；
    所述第一特定域的标签信息存储有第一子标识，值信息存储有总校验值，所述总校验值根据所述n条记录分别对应的校验值获取；
    所述第二特定域的标签信息存储有第二子标识，值信息存储有总记录数，所述总记录数具体为n。
  4. 根据权利要求1所述的序列化方法，其特征在于，所述第一特定标识和所述第二特定标识均为大于预设阈值的数值，所述预设阈值根据标签信息的域标识的最大可用范围确定。
  5. 一种序列化数据的反序列化方法，其特征在于，包括：
    获取一组序列化数据，所述序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息中存储有所述第i条记录的数据组，第mi+1个域的标签信息中存储有第一特定标识，用于标识所述第i条记录的结束，所述序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识所述序列化数据的结束；
    从所述序列化数据中获取n个结构化数据分别对应的属性值组，其中，第i个结构 化数据对应的属性值组从所述第i条记录的数据组中获取；
    根据所述n个结构化数据分别对应的属性值组，生成所述n个结构化数据。
  6. 根据权利要求5所述的反序列化方法，其特征在于，还包括：
    从所述序列化数据中获取n条记录分别对应的校验值，其中，所述第i条记录对应的校验值从所述第i条记录中第mi+1个域的值信息中获取；
    根据所述n个结构化数据分别对应的属性值组以及所述n条记录分别对应的校验值，对所述序列化数据进行校验。
  7. 根据权利要求5所述的反序列化方法，其特征在于，
    若在所述n条记录之后的特定域包括第一特定域，所述方法还包括：从所述第一特定域的值信息中获取总校验值，根据所述n条记录分别对应的校验值以及所述总校验值，对所述序列化数据进行校验；
    若在所述n条记录之后的特定域包括第二特定域，所述方法还包括：从所述第二特定域的值信息中获取总记录数，根据所述总记录数以及所述序列化数据包括的记录数，对所述序列化数据进行校验；
    其中，所述第一特定域的标签信息中存储有第一子标识，所述第二特定域的标签信息中存储有第二子标识。
  8. 根据权利要求5所述的反序列化方法，其特征在于，所述第一特定标识和所述第二特定标识均为大于预设阈值的数值，所述预设阈值根据标签信息的域标识的最大可用范围确定。
  9. 一种序列化数据的保存方法，其特征在于，包括：
    服务器创建会话；
    所述服务器将所述会话的会话标识发送至客户端；
    所述服务器接收所述客户端分布式地发送的多个数据块，每个数据块与所述会话标识相关联，每个数据块包括一组或多组序列化数据；
    所述服务器接收所述客户端发送的数据块保存列表，所述数据块保存列表用于标识出待保存的所有数据块；
    若所述多个数据块与所述数据块保存列表匹配，所述服务器保存所述多个数据块。
  10. 根据权利要求9所述的保存方法，其特征在于，所述一组或多组序列化数据中的任一组序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息存储有所述第i条记录的数据组，第mi+1个域的标签信息 存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识该组序列化数据的结束。
  11. 根据权利要求9所述的保存方法，其特征在于，还包括：
    若所述多个数据块与所述数据块保存列表不匹配，所述服务器向所述客户端发送数据块缺失列表，所述数据块缺失列表用于标识出属于所述数据块保存列表且不属于所述多个数据块的数据块；
    所述服务器接收所述客户端分布式地发送的所述数据块缺少列表中标识的数据块。
  12. 一种序列化数据的下载方法，其特征在于，包括：服务器将保存的序列化数据的总组数发送至客户端；
    所述服务器接收所述客户端发送的下载信息，所述下载信息指示出所述客户端待下载的序列化数据的组数标识；
    所述服务器将所述组数标识对应的序列化数据分布式地发送至所述客户端。
  13. 根据权利要求12所述的下载方法，其特征在于，所述保存的序列化数据中的任一组序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息存储有所述第i条记录的数据组，第mi+1个域的标签信息存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识该组序列化数据的结束。
  14. 一种序列化装置，其特征在于，包括：
    数据获取单元，用于获取n个结构化数据，n≥1；
    属性值获取单元，用于获取所述n个结构化数据分别对应的属性值组；
    数据生成单元，用于根据所述n个结构化数据分别对应的属性值组，生成一组序列化数据；
    所述序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，mi≥1，前mi个域的值信息中存储有所述第i条记录的数据组，所述数据组为所述n个结构化数据中第i个结构化数据对应的属性值组，第mi+1个域的标签信息中存储有第一特定标识，用于标识所述第i条记录的结束；所述序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息中存储有第二特定标识，用于标识所述序列化数据的结束。
  15. 根据权利要求14所述的装置，其特征在于，所述n条记录中分别存储有各个 记录对应的校验值；其中，所述第i条记录中，第mi+1个域的值信息存储有所述第i条记录对应的校验值，所述第i条记录对应的校验值根据所述第i个结构化数据对应的属性值组获取。
  16. 根据权利要求15所述的装置，其特征在于，所述特定域包括第一特定域和/或第二特定域；
    所述第一特定域的标签信息存储有第一子标识，值信息存储有总校验值，所述总校验值根据所述n条记录分别对应的校验值获取；
    所述第二特定域的标签信息存储有第二子标识，值信息存储有总记录数，所述总记录数具体为n。
  17. 根据权利要求14所述的装置，其特征在于，所述第一特定标识和所述第二特定标识均为大于预设阈值的数值，所述预设阈值根据标签信息的域标识的最大可用范围确定。
  18. 一种反序列化装置，其特征在于，包括：
    数据获取单元，用于获取一组序列化数据，所述序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息中存储有所述第i条记录的数据组，第mi+1个域的标签信息中存储有第一特定标识，用于标识所述第i条记录的结束，所述序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识所述序列化数据的结束；
    属性值获取单元，用于从所述序列化数据中获取n个结构化数据分别对应的属性值组，其中，第i个结构化数据对应的属性值组从所述第i条记录的数据组中获取；
    数据生成单元，用于根据所述n个结构化数据分别对应的属性值组，生成所述n个结构化数据。
  19. 根据权利要求18所述的装置，其特征在于，还包括：
    校验值获取单元，用于从所述序列化数据中获取n条记录分别对应的校验值，其中，所述第i条记录对应的校验值从所述第i条记录中第mi+1个域的值信息中获取；
    第一校验单元，用于根据所述n个结构化数据分别对应的属性值组以及所述n条记录分别对应的校验值，对所述序列化数据进行校验。
  20. 根据权利要求18所述的装置，其特征在于，若在所述n条记录之后的特定域包括第一特定域，所述装置还包括：第二校验单元，用于从所述第一特定域的值信息中获取总校验值，根据所述n条记录分别对应的校验值以及所述总校验值，对所述序列化 数据进行校验；
    若在所述n条记录之后的特定域包括第二特定域，所述装置还包括：第三校验单元，用于从所述第二特定域的值信息中获取总记录数，根据所述总记录数以及所述序列化数据包括的记录数，对所述序列化数据进行校验；
    其中，所述第一特定域的标签信息中存储有第一子标识，所述第二特定域的标签信息中存储有第二子标识。
  21. 根据权利要求18所述的装置，其特征在于，所述第一特定标识和所述第二特定标识均为大于预设阈值的数值，所述预设阈值根据标签信息的域标识的最大可用范围确定。
  22. 一种服务器，其特征在于，包括：
    创建单元，用于创建会话；
    发送单元，用于将所述会话的会话标识发送至客户端；
    接收单元，用于接收所述客户端分布式地发送的多个数据块，每个数据块与所述会话标识相关联，每个数据块包括一组或多组序列化数据，以及接收所述客户端发送的数据块保存列表，所述数据块保存列表用于标识出待保存的所有数据块；
    保存单元，用于若所述多个数据块与所述数据块保存列表匹配，保存所述多个数据块。
  23. 根据权利要求22所述的服务器，其特征在于，所述一组或多组序列化数据中的任一组序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息存储有所述第i条记录的数据组，第mi+1个域的标签信息存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识该组序列化数据的结束。
  24. 根据权利要求22所述的服务器，其特征在于，
    所述发送单元还用于，若所述多个数据块与所述数据块保存列表不匹配，向所述客户端发送数据块缺失列表，所述数据块缺失列表用于标识出属于所述数据块保存列表且不属于所述多个数据块的数据块；
    所述接收单元还用于，接收所述客户端分布式地发送的所述数据块缺少列表中标识的数据块。
  25. 一种服务器，其特征在于，包括：发送单元和接收单元；
    所述发送单元，用于将保存的序列化数据的总组数发送至客户端；
    所述接收单元，用于接收所述客户端发送的下载信息，所述下载信息指示出所述客户端待下载的序列化数据的组数标识；
    所述发送单元，还用于将所述组数标识对应的序列化数据分布式地发送至所述客户端。
  26. 根据权利要求25所述的服务器，其特征在于，所述保存的序列化数据中的任一组序列化数据包括n条记录，其中，第i条记录包括mi+1个域，1≤i≤n，n≥1，mi≥1，前mi个域的值信息存储有所述第i条记录的数据组，第mi+1个域的标签信息存储有第一特定标识，用于标识所述第i条记录的结束，该组序列化数据在所述n条记录之后还包括特定域，所述特定域的标签信息存储有第二特定标识，用于标识该组序列化数据的结束。
PCT/CN2016/106566 2015-11-30 2016-11-21 一种结构化数据的序列化、反序列化方法及装置 Ceased WO2017092580A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510857451.1 2015-11-30
CN201510857451.1A CN106815238B (zh) 2015-11-30 2015-11-30 一种结构化数据的序列化、反序列化方法及装置

Publications (1)

Publication Number Publication Date
WO2017092580A1 true WO2017092580A1 (zh) 2017-06-08

Family

ID=58796278

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/106566 Ceased WO2017092580A1 (zh) 2015-11-30 2016-11-21 一种结构化数据的序列化、反序列化方法及装置

Country Status (2)

Country Link
CN (1) CN106815238B (zh)
WO (1) WO2017092580A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112214516A (zh) * 2020-10-29 2021-01-12 株洲中车时代电气股份有限公司 数据序列化和反序列化的方法及装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101827111A (zh) * 2010-05-12 2010-09-08 中兴通讯股份有限公司 Tcp链接方法、网络系统、客户端和服务器
CN102043657A (zh) * 2011-02-01 2011-05-04 苏州同元软控信息技术有限公司 物理建模语言Modelica的模型库文件序列化方法
CN103092894A (zh) * 2011-11-08 2013-05-08 阿里巴巴集团控股有限公司 一种结构化信息检索方法和系统
CN104794231A (zh) * 2015-05-04 2015-07-22 罗敬宁 一种基于Map/Reduce的非结构化数据的处理方法和装置

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7441185B2 (en) * 2005-01-25 2008-10-21 Microsoft Corporation Method and system for binary serialization of documents
CN101192148B (zh) * 2006-12-01 2012-02-01 深圳迈瑞生物医疗电子股份有限公司 兼容新旧应用程序的数据处理方法及其数据存储方法
US7752192B2 (en) * 2007-03-02 2010-07-06 International Business Machines Corporation Method and system for indexing and serializing data
CN101477532B (zh) * 2008-12-23 2011-09-28 北京畅游天下网络技术有限公司 实现数据存储、读取的方法、装置及系统
CN101876990A (zh) * 2009-10-27 2010-11-03 用友软件股份有限公司 传递树形结构对象的方法
CN103389992A (zh) * 2012-05-09 2013-11-13 北京百度网讯科技有限公司 一种结构化数据存储方法及装置
CN102981884B (zh) * 2012-11-22 2016-03-30 用友优普信息技术有限公司 序列化装置和序列化方法
CN104077335B (zh) * 2013-05-07 2017-05-03 腾讯科技（深圳）有限公司 一种结构化数据的序列化、反序列化方法、装置和系统
CN104199927B (zh) * 2014-09-03 2016-11-30 腾讯科技（深圳）有限公司 数据处理方法及数据处理装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101827111A (zh) * 2010-05-12 2010-09-08 中兴通讯股份有限公司 Tcp链接方法、网络系统、客户端和服务器
CN102043657A (zh) * 2011-02-01 2011-05-04 苏州同元软控信息技术有限公司 物理建模语言Modelica的模型库文件序列化方法
CN103092894A (zh) * 2011-11-08 2013-05-08 阿里巴巴集团控股有限公司 一种结构化信息检索方法和系统
CN104794231A (zh) * 2015-05-04 2015-07-22 罗敬宁 一种基于Map/Reduce的非结构化数据的处理方法和装置

Also Published As

Publication number Publication date
CN106815238A (zh) 2017-06-09
CN106815238B (zh) 2020-10-20

Similar Documents

Publication Publication Date Title
US12282921B2 (en) Blockchain-based data processing method and apparatus, device, readable storage medium and computer program product
JP6686033B2 (ja) メッセージをプッシュするための方法および装置
CN113300875B (zh) 回源数据校验方法、服务器、系统及存储介质
WO2021073156A1 (zh) 短链接的生成方法、服务器、存储介质及计算机设备
US11361110B2 (en) File verification method, file verification system and file verification server
CN106250270A (zh) 一种云计算平台下的数据备份方法
CN106603198A (zh) 具有网络编码的区块链分布式存储方法及系统
CN104202387B (zh) 一种元数据恢复方法及相关装置
TW200415894A (en) Atomic message division
CN108282339A (zh) 一种基于智能合约和自社交媒体的数字身份找回方法
CN105468472A (zh) 基于iOS操作系统的数据备份、恢复方法及装置
US12335038B2 (en) Ingesting and storing data in a massive data ingestion system
CN112015806A (zh) 区块链存储数据的方法及装置
CN105786629A (zh) 基于消息队列的数据处理方法
CN114092252A (zh) 一种区块链交易执行方法、装置、设备及可读存储介质
CN106878246A (zh) 一种资源管理方法及装置
CN103152606B (zh) 视频文件处理方法及装置、系统
CN105554136B (zh) 备份数据还原方法、装置及系统
US10437849B2 (en) Method and apparatus for implementing storage of file in IP disk
CN111414239A (zh) 基于麒麟云计算平台的虚拟机镜像管理方法、系统及介质
WO2017092580A1 (zh) 一种结构化数据的序列化、反序列化方法及装置
CN109710502B (zh) 日志传输方法、装置及存储介质
CN103729269A (zh) 一种基于云架构的网络考试数据双缓存方法
CN107317742A (zh) 一种消息处理方法及系统
JP6364727B2 (ja) 情報処理システム、分散処理方法、及び、プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16869890

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16869890

Country of ref document: EP

Kind code of ref document: A1