CN113704320B

CN113704320B - Data processing method, device, electronic equipment and storage medium

Info

Publication number: CN113704320B
Application number: CN202110909460.6A
Authority: CN
Inventors: 樊庆; 曹舰航; 贾思超
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2021-08-09
Filing date: 2021-08-09
Publication date: 2024-01-02
Anticipated expiration: 2041-08-09
Also published as: CN113704320A

Abstract

The disclosure discloses a data processing method, a data processing device, electronic equipment and a storage medium, and belongs to the technical field of computers. The data processing method includes receiving a pattern registration request for a first data source; wherein the pattern registration request includes a data source identifier corresponding to the first data source; according to the mode registration request, a data mode corresponding to the first data source is acquired; describing a data mode according to a preset description form, and generating first mode information; storing first mode information into a mode information table according to the data source identification so as to realize the registration of a first data source; wherein the first pattern information is used to process data associated with the first data source. By adopting the data processing method, the device, the electronic equipment and the storage medium provided by the disclosure, at least the problem of high cost of data management and processing of the heterogeneous data sources in the prior art is solved.

Description

Data processing method, device, electronic equipment and storage medium

Technical Field

The disclosure relates to the technical field of computers, and in particular relates to a data processing method, a data processing device, electronic equipment and a storage medium.

Background

Heterogeneous data sources are, in a broad sense, a plurality of data sources that differ in data structure, access manner, and form. When data synchronization is performed between two heterogeneous data sources with different data modes, for example, data in the data source A is synchronized to the data source B, a manager of the data source B, namely, a data consumer of the data source A needs to acquire the data mode of the data source A, so that the data output by the data source A can be analyzed according to the data mode, and the data consumer can always understand the data synchronized by the data source A.

In the prior art, because the data modes corresponding to different data sources are different, in order to facilitate the unified processing of the data output by the different data sources in the later period, data consumers need to respectively adopt different data standardization steps for unifying the data output by the different data sources, so that the cost of data management and processing is higher.

Disclosure of Invention

An object of an embodiment of the present disclosure is to provide a data processing method, apparatus, electronic device, and storage medium, so as to at least solve the problem of high cost of data management and processing of heterogeneous data sources in the prior art.

The technical scheme of the present disclosure is as follows:

according to a first aspect of embodiments of the present disclosure, there is provided a data processing method, the method may include:

receiving a mode registration request for a first data source; wherein the pattern registration request includes a data source identifier corresponding to the first data source;

according to the mode registration request, a data mode corresponding to the first data source is acquired;

describing a data mode according to a preset description form, and generating first mode information;

storing first mode information into a mode information table according to the data source identification so as to realize the registration of a first data source; wherein the first pattern information is used to process data associated with the first data source.

In one embodiment, the mode registration request includes a type of data, describes a data mode according to a preset description form, and generates first mode information, including:

describing a serialization mode of a first data source according to a preset description form under the condition that the type of the data is serialization data, and generating first mode information;

and under the condition that the type of the data is non-serialized data, describing the table structure of the first data source according to a preset description form, and generating first mode information.

Based on this, in one embodiment, in the case where the type of data is serialized data, the pattern registration request further includes a serialization manner and an associated data source identifier corresponding to the first data source;

according to the mode registration request, acquiring a data mode corresponding to the first data source, including:

determining a second data source associated with the first data source according to the associated data source identification, the second data source being different from the first data source in data mode;

acquiring second mode information corresponding to a second data source under the condition that the second data source is a registered data source; the second mode information is generated mode information which describes a data mode corresponding to the second data source according to a preset description form;

and generating a serialization mode corresponding to the first data source according to the second mode information in a serialization mode.

In one embodiment, in the case that the type of the data is serialized data, after acquiring the data mode corresponding to the first data source according to the mode registration request, the method further includes:

and storing the serialization mode corresponding to the first data source into a serialization mode table according to the data source identification.

Based on this, in one embodiment, after storing the serialization pattern corresponding to the first data source into the serialization pattern table according to the data source identifier, the method further includes:

receiving a mode acquisition request of a requester for a first data source; the mode acquisition request comprises a data source identifier corresponding to the first data source;

responding to a mode acquisition request, and acquiring a serialization mode corresponding to the data source identifier from a serialization mode table;

a serialization pattern is sent to the requestor, the serialization pattern being usable to process data associated with the first data source.

Additionally, in one embodiment, after storing the first schema information in the schema information table based on the data source identification, the method further comprises:

receiving a data query request of a requester for a first data source; the data query request comprises a data source identifier corresponding to the first data source;

responding to a data query request, and acquiring first mode information corresponding to a data source identifier from a mode information table;

first schema information is sent to the requesting party, the first schema information being usable to query a data structure of the first data source.

In one embodiment, after storing the first pattern information in the pattern information table according to the data source identification, the method further comprises:

Receiving a mode update request for a first data source;

responding to a mode updating request, describing a data mode updated by the first data source according to a preset description form, and generating third mode information;

according to the first mode information and the third mode information, compatibility checking is carried out on the mode updating;

under the condition that the mode updating is compatible according to the checking result, updating the stored mode information corresponding to the first data source according to the third mode information;

and under the condition that the mode updating is not compatible according to the checking result, refusing to update the stored mode information corresponding to the first data source.

Based on this, in one embodiment, the mode registration request further includes a type of compatibility check corresponding to the first data source;

according to the first mode information and the third mode information, compatibility checking is carried out on the mode updating, and the method comprises the following steps:

acquiring a type of compatibility check corresponding to the first data source;

comparing the first mode information with the third mode information to obtain updated information;

and according to the type of the compatibility check, carrying out the compatibility check on the mode update according to the update information.

In one embodiment, the types of compatibility checks include forward compatibility, backward compatibility, or full compatibility;

according to the type of the compatibility check, the compatibility check is carried out on the mode update according to the update information, and the method comprises the following steps:

if the type of the compatibility check is forward compatibility, if the update information is an added field or a deleted optional field, determining that the mode is updated and compatible; if the update information is not an added field or a deleted optional field, determining that the mode update is incompatible;

if the type of the compatibility check is backward compatibility, if the update information is an optional field or a deleted field, determining that the mode is updated and compatible; if the update information is not an optional field or a deleted field, determining that the mode update is not compatible;

under the condition that the type of the compatibility check is full compatibility, if the update information is an optional field added or deleted, the mode update compatibility is determined; if the update information is not the addition or deletion of the optional field, the mode update incompatibility is determined.

In addition, in one embodiment, after updating the stored mode information corresponding to the first data source according to the third mode information, the method further includes:

Determining a third data source associated with the first data source according to the associated data source identifier corresponding to the first data source; the third data source is a registered data source for storing target data, and the target data is data taken from the first data source;

acquiring registration information of a third data source;

under the condition that the non-serialized data stored in the third data source is determined according to the registration information, generating a table structure update statement corresponding to the third data source according to the first mode information and the third mode information;

and sending a table structure update statement to the third data source, wherein the table structure update statement is used for indicating updating of the table structure of the third data source.

According to a second aspect of embodiments of the present disclosure, there is provided a data processing apparatus, the apparatus may comprise:

a registration request receiving module configured to perform receiving a pattern registration request for a first data source; wherein the pattern registration request includes a data source identifier corresponding to the first data source;

the data mode acquisition module is configured to acquire a data mode corresponding to the first data source according to the mode registration request;

the data pattern description module is configured to execute description of a data pattern according to a preset description form and generate first pattern information;

The mode information registration module is configured to store first mode information into a mode information table according to the data source identification so as to register the first data source; wherein the first pattern information is used to process data associated with the first data source.

In one embodiment, the mode registration request includes a type of data, and the data mode description module includes:

the first description sub-module is configured to execute a serialization mode for describing a first data source according to a preset description form and generate first mode information under the condition that the type of the data is the serialization data;

and the second description submodule is configured to execute the table structure describing the first data source according to a preset description form and generate first mode information under the condition that the type of the data is non-serialized data.

a data pattern acquisition module comprising:

a data source determination sub-module configured to perform determining a second data source associated with the first data source based on the associated data source identification, the second data source being different from the data pattern of the first data source;

The mode acquisition sub-module is configured to acquire second mode information corresponding to the second data source when the second data source is a registered data source; the second mode information is generated mode information which describes a data mode corresponding to the second data source according to a preset description form;

and the pattern generation sub-module is configured to execute the serialization pattern corresponding to the first data source according to the second pattern information in a serialization mode.

In one embodiment, in the case that the type of data is serialized data, the apparatus further includes:

and the serialization mode storage module is configured to store the serialization mode corresponding to the first data source into the serialization mode table according to the data source identifier after acquiring the data mode corresponding to the first data source according to the mode registration request.

Based on this, in one of the embodiments, the apparatus further comprises:

the acquisition request receiving module is configured to receive a mode acquisition request of a requester for the first data source after storing a serialization mode corresponding to the first data source into a serialization mode table according to the data source identifier; the mode acquisition request comprises a data source identifier corresponding to the first data source;

A serialization pattern acquisition module configured to perform acquisition of a serialization pattern corresponding to the data source identifier from a serialization pattern table in response to the pattern acquisition request;

and a serialization mode transmitting module configured to perform a transmission of a serialization mode to the requestor, the serialization mode being operable to process data associated with the first data source.

Additionally, in one embodiment, the apparatus further comprises:

a query request receiving module configured to receive a data query request from a requester for a first data source after storing first pattern information into a pattern information table according to a data source identification; the data query request comprises a data source identifier corresponding to the first data source;

a mode information acquisition module configured to perform acquisition of first mode information corresponding to a data source identifier from a mode information table in response to a data query request;

and a mode information transmitting module configured to perform transmission of first mode information to the requester, the first mode information being usable for querying a data structure of the first data source.

On the basis of the above embodiments, in one embodiment, the apparatus further includes:

an update request receiving module configured to receive a pattern update request for a first data source after storing first pattern information into a pattern information table according to a data source identification;

The mode updating description module is configured to execute the description of the data mode updated by the first data source according to a preset description form in response to the mode updating request, and generate third mode information;

the updating compatibility checking module is configured to perform compatibility checking on the mode updating according to the first mode information and the third mode information;

a mode update registration module configured to perform updating of stored mode information corresponding to the first data source according to the third mode information in a case where it is determined that the current mode update is compatible according to the inspection result;

and the mode updating refusing module is configured to execute refusing to update the stored mode information corresponding to the first data source under the condition that the mode updating is not compatible according to the checking result.

updating the compatibility checking module, comprising:

a type acquisition sub-module configured to perform acquiring a type of compatibility check corresponding to the first data source;

the information acquisition sub-module is configured to execute comparison between the first mode information and the third mode information and acquire updated information;

And the compatibility checking sub-module is configured to perform compatibility checking on the mode update according to the type of the compatibility checking and the update information.

Based on this, in one embodiment, the type of compatibility check includes forward compatibility, backward compatibility, or full compatibility;

a compatibility check sub-module, comprising:

a first checking unit configured to perform, if the type of the compatibility check is forward compatibility, determining that the mode update is compatible if the update information is an add field or a delete optional field; if the update information is not an added field or a deleted optional field, determining that the mode update is incompatible;

a second checking unit configured to perform, if the type of the compatibility check is backward compatibility, if the update information is an add optional field or a delete field, determining that the mode update is compatible; if the update information is not an optional field or a deleted field, determining that the mode update is not compatible;

a third checking unit configured to perform, if the type of the compatibility check is full compatibility, determining that the mode update is compatible if the update information is an add or delete optional field; if the update information is not the addition or deletion of the optional field, the mode update incompatibility is determined.

Additionally, in one embodiment, the apparatus further comprises:

a data source determining module configured to determine a third data source associated with the first data source according to an associated data source identifier corresponding to the first data source after updating the stored pattern information corresponding to the first data source according to the third pattern information; the third data source is a registered data source for storing target data, and the target data is data taken from the first data source;

a registration information acquisition module configured to perform acquisition of registration information of the third data source;

an update statement generation module configured to perform generation of a table structure update statement corresponding to the third data source based on the first pattern information and the third pattern information in a case where it is determined that the non-serialized data is stored in the third data source based on the registration information;

and an update statement transmitting module configured to perform transmission of a table structure update statement to the third data source, the table structure update statement indicating updating of the table structure of the third data source.

According to a third aspect of embodiments of the present disclosure, there is provided an electronic device, which may include:

A processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement a data processing method as shown in any of the embodiments of the first aspect.

According to a fourth aspect of embodiments of the present disclosure, there is provided a storage medium, which when executed by a processor of a data processing apparatus, causes the data processing apparatus to implement a data processing method as shown in any one of the embodiments of the first aspect.

According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product comprising a computer program stored in a readable storage medium, at least one processor of the apparatus reading and executing the computer program from the storage medium, causing the apparatus to perform the data processing method shown in any one of the embodiments of the first aspect.

The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:

in the embodiment of the disclosure, the mode registration service is provided for the plurality of heterogeneous data sources, that is, in the process of registering the first data source in the plurality of heterogeneous data sources, the data mode corresponding to the first data source is described according to the preset description form, the first mode information is generated, and then the first mode information is stored in the mode information table, so that the registration of the first data source is realized, and when the data associated with the first data source needs to be processed, as the first data source adopts the uniform description form to generate the universal mode information as the other heterogeneous data sources during registration, for the data consumer of the first data source, the universal mode information is adopted to process the data of the first data source and the data of the other heterogeneous data sources, and different data standardization steps are not required to be respectively adopted for the data output by different data sources to be unified.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure and do not constitute an undue limitation on the disclosure.

FIG. 1 is an architecture diagram illustrating one type of data processing according to an example embodiment;

FIG. 2 is a schematic diagram of a data synchronization scenario based on heterogeneous data sources, according to an example embodiment;

FIG. 3 is a schematic diagram of another heterogeneous data source based data synchronization scenario, shown in accordance with an exemplary embodiment;

FIG. 4 is a flowchart illustrating a method of data processing according to an exemplary embodiment;

FIG. 5 is a flowchart illustrating another data processing method according to an exemplary embodiment;

FIG. 6 is a flowchart illustrating yet another data processing method according to an exemplary embodiment;

FIG. 7 is a flowchart illustrating yet another data processing method according to an exemplary embodiment;

FIG. 8 is a schematic diagram of an application scenario of a data processing method according to an exemplary embodiment;

FIG. 9 is a schematic diagram of an application scenario illustrating another data processing method according to an example embodiment;

FIG. 10 is a block diagram of a data processing apparatus according to an exemplary embodiment;

fig. 11 is a block diagram illustrating a structure of an electronic device according to an exemplary embodiment.

Detailed Description

In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.

The data processing method provided by the present disclosure may be applied to the architecture shown in fig. 1, and is specifically described in detail with reference to fig. 1.

FIG. 1 is an architecture diagram illustrating a data process according to an example embodiment.

As shown in fig. 1, the architecture diagram may include at least one first electronic device 10 including a data source and a second electronic device 11 including a pattern registration service. The second electronic device 11 may respectively establish a connection with at least one first electronic device 10 through a network and perform information interaction, and the plurality of first electronic devices 10 may also establish a connection through the network and perform data transmission. The first electronic device 10 may be a device with a communication function, such as a mobile phone, a tablet computer, or an all-in-one device, or may be a device simulated by a virtual machine or a simulator, or may be a device with a storage and calculation function, such as a cloud server or a server cluster, and the second electronic device 11 may be a device with a storage and calculation function, such as a cloud server or a server cluster.

Here, a plurality of heterogeneous data sources having different data patterns may exist in different first electronic devices 10, respectively, and each data source may perform pattern registration in the second electronic device 11 including the pattern registration service through the network. Upstream data sources and downstream data sources may be distinguished based on the direction of data transfer between the data sources. For example, if data source a sends data to data source B, data source a is an upstream data source with respect to data source B, and data source B is a downstream data source with respect to data source a; if data source B sends data to data source C, then data source B is an upstream data source with respect to data source C and data source C is a downstream data source with respect to data source B.

When data synchronization is performed among a plurality of heterogeneous data sources, a downstream data source needs to acquire a data mode of an upstream data source, so that data transmitted by the upstream data source can be analyzed, and the downstream data source can always understand the data transmitted by the upstream data source. In the prior art, the data modes corresponding to heterogeneous data sources are different, so that the downstream data sources need to unify the data according to different data standardization steps for the data synchronized by different data sources, so that unified processing is performed when the data synchronized by different data sources are used, and the cost of data management and processing is high.

In view of the above problems, in the embodiments of the present disclosure, a pattern registration service is provided for a heterogeneous data source, and a data pattern of each registered data source is described in a unified manner, so that when data associated with a registered data source needs to be processed, for example, when data synchronization is performed between data sources, a complex data normalization step is not required for a data source downstream of the registered data source, and data synchronized by a plurality of upstream heterogeneous data sources can be processed by using the common pattern information, so that costs for data management and processing of heterogeneous data sources can be reduced.

In addition, the general mode information can be utilized to provide data query service. Thus, the inquirer can inquire the data structure contained in the data source, such as the field name and the data type contained in the data source, through the general mode information corresponding to the data source without analyzing the data in the data source.

Meanwhile, after the data mode is updated by the upstream data source, the updated data mode can be described according to a unified description form to generate new mode information, compatibility check is carried out on the new mode information and the old mode information, and the upstream data source is allowed to carry out mode updating registration under the condition of determining compatibility, so that the downstream data source can be ensured to analyze all data from the upstream data source, the change of the compatibility is prevented from being damaged, and a safe mode evolution strategy is provided.

Based on the above architecture, the embodiments of the present disclosure may be applicable to data synchronization scenarios based on heterogeneous data sources.

One case is to synchronize the data of the data source of the serialized data to the data source of the non-serialized data. Illustratively, as shown in FIG. 2, a data source 21 comprising serialized data, a data source 22 comprising non-serialized data, and a pattern registration service 23, wherein data stored in the data source 21 of serialized data may be synchronized to the data source 22 of non-serialized data. At this time, at least the data source 21 of the serialized data may be subjected to pattern registration in the pattern registration service 23, so that the serialized data acquired from the data source 21 of the serialized data is subjected to deserialization processing according to the serialized pattern, and the deserialized data is stored in the data source 22 of the non-serialized data, before the data is stored in the data source 22 of the non-serialized data, which is acquired from the pattern registration service 23.

In addition, when the data source 21 of the serialized data registers, the serialized mode may be described according to a preset description form to obtain corresponding mode information, and if the data source 21 of the serialized data needs to be queried later, the corresponding mode information may be obtained from the mode registration service 23, so as to query the data structure of the data source 21 of the serialized data without analyzing the data according to the mode information.

In another case, data in a data source of non-serialized data is synchronized to a data source of serialized data. Illustratively, as shown in FIG. 3, a data source 31 comprising non-serialized data, a data source 32 comprising serialized data, and a pattern registration service 33, wherein data stored in the data source 31 comprising non-serialized data may be synchronized to the data source 32 comprising serialized data. At this time, the pattern registration service 33 may perform pattern registration on the non-serialized data source 31 and the serialized data source 32, describe the table structure definition of the non-serialized data source 31 according to the preset description form, generate corresponding pattern information, and automatically generate a serialized pattern according to the pattern information corresponding to the non-serialized data source 31 according to the serialization mode corresponding to the serialized data source 32 in the process of registering the serialized data source 32, so as to synchronize the data in the non-serialized data source 31 to the serialized data source 32 according to the serialized pattern, and then perform serialization processing on the data extracted from the non-serialized data source 31 to obtain the serialized data and then synchronize the serialized data to the serialized data source 32.

In addition, if the data query is required for the non-serialized data source 31 or the serialized data source 32, the corresponding pattern information may be obtained from the pattern registration service 33, so as to query the data structure of the non-serialized data source 31 or the serialized data source 32 according to the pattern information.

Thus, on one hand, the embodiment of the present disclosure can cope with the challenge of heterogeneous data sources in terms of data management, that is, the mode registration service provides a common service, and the common mode information can be used in different data sources to describe respective data modes, so that the cost of data management and processing is greatly simplified. For example, all applications may use unified "website click event" mode information. The upstream application only needs to generate events in a contracted format, while downstream data processing no longer requires complex normalization steps. On the other hand, the embodiment of the disclosure can also be beneficial to data discovery and improve the data query efficiency, namely, a data researcher or developer can check the data structure in the data source through the data query function without depending on a data producer.

According to the above architecture and application scenario, a data processing method provided by the embodiments of the present disclosure is described in detail below with reference to fig. 4 to 9, and the data processing method may be performed by the second electronic device 11 including the mode registration service shown in fig. 1.

FIG. 4 is a flowchart illustrating a method of data processing according to an exemplary embodiment.

As shown in FIG. 4, the data processing method may specifically include the following steps 410-440.

Step 410, receiving a mode registration request for a first data source; wherein the pattern registration request includes a data source identification corresponding to the first data source.

Step 420, according to the mode registration request, a data mode corresponding to the first data source is obtained.

And step 430, describing the data mode according to a preset description form, and generating first mode information.

In one embodiment, the preset description form may be used to describe a pattern format of the data pattern, for example, field information included in the data pattern.

Step 440, according to the data source identification, storing the first mode information into a mode information table to realize the registration of the first data source; wherein the first pattern information is used to process data associated with the first data source.

Therefore, by providing the mode registration service for the heterogeneous data sources, that is, describing the data mode corresponding to the first data source according to the preset description form in the process of registering the first data source in the heterogeneous data sources, generating the first mode information, storing the first mode information in the mode information table, and realizing the registration of the first data source, when the data associated with the first data source needs to be processed, since the first data source adopts the uniform description form to generate the common mode information as the other heterogeneous data sources in the process of registering, for the data consumer of the first data source, the common mode information can be adopted to process the data of the first data source and the data of the other heterogeneous data sources, and different data standardization steps are not required to be respectively adopted for the data output by different data sources to be unified.

The above steps are described in detail below, and are specifically described below.

Referring to step 410, embodiments of the present disclosure are directed to a scenario in which data is transmitted between a plurality of heterogeneous data sources having different data patterns, where, for a data source that sequences data, the data patterns may include serialization patterns, and the serialization patterns may be used to serialize or deserialize the data, and the different serialization patterns correspond to the different serialization patterns. Here, serialization means include, but are not limited to, protobuf, avro, json, etc. Whereas for data sources that are non-serialized data, the data schema may include table structures, such as field names, types, etc.

Specifically, the first data source may be any one of a plurality of heterogeneous data sources, and the first data source may be a data source storing serialized data or a data source storing non-serialized data. For example, a producer of data stored in a first data source may send a pattern registration request for the first data source to a second electronic device at which the pattern registration service is located, such that the second electronic device may receive the pattern registration request for the first data source.

For example, in the case where the serialized data is stored in the first data source, the received mode registration request may carry information such as a data source type, a data source identifier, a serialization manner, and the like of the first data source; in the case that the first data source stores non-serialized data, the received mode registration request may carry information such as a data source type, a data source identifier, a table structure, and the like of the first data source. The data source of the serialized data may be, for example, redis, kafka, etc., and the data source of the non-serialized data may be, for example, mysql, hive, etc.

In one embodiment, the pattern registration request may further include an associated data source identifier corresponding to the first data source, the associated data source identifier being usable to obtain an associated data source corresponding to the first data source. Specifically, when the second electronic device 11 in fig. 1 receives the mode registration request for the first data source, it may determine, according to the associated data source identifier included in the mode registration request, that, in a case where the associated data source corresponding to the first data source is registered, mode information of the associated data source corresponding to the first data source is acquired, and generate the first mode information of the first data source according to the mode information of the associated data source corresponding to the first data source.

Referring to steps 420 and 430, the data patterns may be acquired in different manners for the data sources storing different types of data, and the manner in which the first pattern information is generated may also be different. Here, the data pattern may be acquired from information carried in the pattern registration request, or from another address based on address information carried in the pattern registration request, or the data pattern may be generated based on pattern information of another data source registered in association with the data pattern. In addition, the preset description form may be described by using avro uniformly, and the correspondingly generated first mode information may include information such as column name, type, alias, etc. for describing a data mode of the first data source.

In an alternative embodiment, the mode registration request may include a type of data, that is, a type of data stored in the first data source, including serialized data or non-serialized data, and the step 430 may specifically include:

Here, the serialization pattern and table structure may be used to describe data, for example, to describe that the data included in the first data source is Zhang three and its corresponding telephone number, lifour and its corresponding telephone number, etc., and the first pattern information may be used to describe the serialization pattern or table structure, for example, to describe that fields including names, telephones, etc., and information of field types, aliases, etc. corresponding to these fields are included in the first data source.

In this way, the advantage of describing the serialization mode or the table structure of the first data source by using the preset description form is that by describing the serialization mode or the table structure by adopting a uniform format, even if the downstream data source aims at a plurality of upstream heterogeneous data sources with different data modes, the data of different data sources can be understood by adopting common mode information, that is, the data of a plurality of upstream heterogeneous data sources including the first data source can be uniformly processed without complex standardized steps, so that the cost of data management and processing is reduced.

In addition, in the case that the type of data in the first data source is serialized data, the serialization mode of the first data source may be obtained in three ways: one way is to register a serialization pattern of the second electronic device that the applicant enters through text at the time of registration and sends to the pattern registration service corresponding to the pattern registration request together with the pattern registration request; one way is a serialization pattern pulled from other storage systems to which the address information points, such as a git system, through the address information carried in the pattern registration request; yet another way is to generate a serialization pattern based on pattern information of other registered heterogeneous data sources.

In the last manner, in a case that the type of data in the first data source is serialized data, in an alternative embodiment, the mode registration request may further include a serialized manner corresponding to the first data source and an associated data source identifier. Accordingly, the step 420 may specifically include:

Here, the associated data source identifier may be a data source identifier corresponding to a data source associated with the first data source, where the data source associated with the first data source, i.e. the second data source, may be an upstream data source or a downstream data source of the first data source. In addition, the second data source may be a different data source than the first data source stores, for example, a data source that may be non-serialized data.

For example, before registering the first data source, the second data source may be registered first to generate second mode information corresponding to the second data source, and further, according to the second mode information, a corresponding serialization mode is generated according to a serialization mode corresponding to the first data source. In addition, the first data source may be registered first, after a registration request for requesting to register the second data source is received, the first data source associated with the second data source may be determined according to an associated data source identifier included in the registration request, further, after a data pattern corresponding to the second data source is described according to a preset description form, second pattern information is generated, and after the first data source is determined to be a registered data source, a serialization mode corresponding to the first data source is obtained, and according to the serialization mode, a serialization pattern corresponding to the first data source is generated according to the second pattern information.

The serialization manners of different data sources in the data sources of the serialization data may be different, and specifically, the serialization manners may include protobuf, avro, json, and the like.

In a specific example, firstly, an identifier of a data source associated with a first data source, for example, an identifier corresponding to an upstream data source or a downstream data source of the first data source, which is input by a user, is received, then, whether a second data source corresponding to the identifier is a registered data source is judged according to the identifier, and if the second data source is the registered data source, if the serialization manner corresponding to the first data source is json, a json type serialization pattern can be generated according to second pattern information corresponding to the second data source. Here, the second pattern information is information obtained by describing the data pattern corresponding to the second data source in a general description form, so that the second pattern information can be used as a bridge for mutual conversion between different data patterns corresponding to the first data source and the second data source, thereby generating the data pattern required by the first data source. Specifically, when the serialization mode corresponding to the first data source is json, and the second mode information includes fields such as a name and a phone, and types corresponding to the fields, a serialization mode of the json type may be generated according to the second mode information, where the serialization mode includes specific information such as the name and the phone corresponding to the specific information.

The method has the advantages that the automatic generation of the serialization mode corresponding to the first data source can be realized through the mode information of other registered heterogeneous data sources, a developer does not need to manually write a specific serialization mode in the process of data synchronization among the heterogeneous data sources, and the general second mode information is generated through registering the second data source, so that the serialization mode corresponding to the first data source can be automatically generated according to the second mode information when the first data source related to the second data source is registered.

In the case that the type of the data in the first data source is nonsequential data, the mode registration request may further include a table structure corresponding to the first data source and an associated data source identifier, where the table structure may be used as a data mode corresponding to the first data source to generate the first mode information, and the associated data source identifier may be used to update or register the data mode of the data source corresponding to the associated data source identifier after updating or registering the data mode of the first data source.

In addition, in another optional implementation manner, in a case where the type of data in the first data source is serialized data, after step 420, the data processing method provided in the embodiment of the present disclosure may further include:

For example, the data source identifier may be stored in the serialization mode table in correspondence with the serialization mode, such that the serialization mode corresponding to the data source identifier may be queried from the serialization mode table according to the data source identifier. Specifically, for a serialization schema of the protobuf type, the protoc tool needs to be used to compile the proto file to generate binary descriptor content for storage. In one embodiment, storing the data source identifier in correspondence with the serialization mode refers to storing the correspondence between the data source identifier and the serialization mode; or the data source identification and the serialization mode are stored in a table, and the data source identification and the serialization mode have a corresponding relation; the data source identifier and the serialization mode can be stored in two tables respectively, and the data source identifier and the serialization mode have a corresponding relationship.

Of course, only one pattern table may be set in the pattern registration service, and accordingly, the first pattern information and the serialization pattern may be stored in the same pattern table in association with the data source identifier, so that the serialization pattern and the first pattern information corresponding to the data source identifier may be queried from the pattern table according to the data source identifier, which is not limited herein.

In addition, the data mode corresponding to the data source of the non-serialized data is a table structure definition, so that when the database is registered, the generated mode information and the data source identifier can be correspondingly stored in a preset mode information table, and thus, the mode information corresponding to the data source identifier can be queried from the mode information table according to the data source identifier.

Therefore, by storing the serialization pattern corresponding to the first data source storing the serialization data into the serialization pattern table, the data producer or the data consumer of the first data source can conveniently inquire and acquire the serialization pattern corresponding to the first data source, so that the data producer can conveniently conduct serialization processing on the data to be stored into the first data source, and the data consumer can conveniently conduct deserialization processing on the data taken out from the first data source.

Referring to step 440, the data associated with the first data source may be, for example, data obtained from the first data source or data that needs to be synchronized to the first data source. In addition, the mode registration service includes a mode information table, and in the process of performing mode registration on the data source, the generated mode information can be stored in the mode information table, thereby completing the registration process on the data source.

Based on this, in the case where the first data source stores the serialized data, in a possible embodiment, as shown in fig. 5, after storing, according to the data source identifier, the serialized mode corresponding to the first data source into the serialized mode table, the data processing method provided by the embodiment of the disclosure may further include: steps 4501 through 4503.

Step 4501, receiving a mode acquisition request of a requester for a first data source; the mode acquisition request comprises a data source identifier corresponding to the first data source.

Here, the requestor may be a downstream data source of the first data source, that is, the data consumer, for example, when the requestor needs to perform deserialization processing on the serialized data output in the first data source in the case where the requestor obtains the data from the first data source, the requestor may send a mode obtaining request for obtaining a serialized mode corresponding to the first data source to a second electronic device where the mode registration service is located; in addition, the requester may be an upstream data source of the first data source, that is, the data producer may send a mode acquisition request for acquiring a serialization mode corresponding to the first data source to the second electronic device where the mode registration service is located, for example, when the requester needs to perform serialization processing on data synchronized from the database to the first data source when synchronizing the data to the first data source by the requester; moreover, the requester may also be a user who needs to view the data content of the first data source, for example, in a data query scenario, when the detailed content of the data in the first data source needs to be queried, a mode acquisition request for acquiring a serialization mode corresponding to the first data source may be sent to a second electronic device where the mode registration service is located.

In response to the mode acquisition request, a serialization mode corresponding to the data source identification is acquired from the serialization mode table, step 4502.

Here, since the data source identification corresponding to the first data source and the serialization pattern are stored in correspondence in the serialization pattern table, the serialization pattern stored in correspondence with the data source identification can be obtained by looking up the serialization pattern table.

Step 4503, a serialization pattern is sent to the requestor, the serialization pattern being usable to process data associated with the first data source.

Here, the data associated with the first data source may be, for example, data acquired from the first data source, or data that needs to be synchronized to the first data source.

For example, the serialization pattern may be sent to the requestor such that the requestor performs a de-serialization process on the serialized data obtained from the first data source or a serialization process on the data synchronized from the database to the first data source according to the serialization pattern.

Of course, a data synchronization service may be set on the second electronic device where the mode registration service is located, and the serialized data may be obtained from the first data source through the data synchronization service, and after the data is deserialized according to the serialization mode, the obtained deserialized data may be sent to the requester, or the data to be synchronized may be obtained from the database through the data synchronization service, and after the data is serialized according to the serialization mode, the obtained serialized data may be sent to the requester.

Therefore, by performing mode registration on the first data source, when the requester needs to process the data associated with the first data source, the serialization mode corresponding to the data source identifier can be directly obtained from the serialization mode table, and then the data is serialized or deserialized according to the serialization mode, so that data standardization processing is not needed, the data processing process is simplified, and the cost of data management and processing is reduced.

In another possible embodiment, as shown in fig. 6, in addition to the above steps 410-440, after step 440, the data processing method provided by the embodiments of the present disclosure may further include: step 4601 to step 4603.

Step 4601, receiving a data query request from a requestor for a first data source; the data query request comprises a data source identifier corresponding to the first data source.

In step 4602, first schema information corresponding to the data source identification is obtained from the schema information table in response to the data query request.

Step 4603, sending first schema information to the requestor, the first schema information capable of being used to query a data structure of the first data source.

Here, the second electronic device may provide a data query service while providing a mode registration service, so as to provide corresponding mode information when the user needs to query data in the registered data source.

In an exemplary case that a user queries a data structure in a first data source, the device may search, after receiving a data query request sent to a device where a data query service is located, corresponding first mode information in a corresponding mode information table according to a carried data source identifier, and send the corresponding first mode information to a requester, so that the requester queries the data structure of the first data source according to the first mode information. The data structure may include, among other things, column names, types, aliases, etc.

Of course, after the first mode information is acquired, the data structure of the first data source may be determined according to the first mode information through the data query service, and the data structure may be sent to the requester.

Therefore, because the mode information can embody the data structure of the data source, a developer can check the data structure in the data source through the data query function without depending on a data producer through the data query service, thereby being beneficial to data discovery and improving the efficiency of collaborative development of data.

In addition, in one possible embodiment, as shown in fig. 7, in addition to the steps 410 to 440, after the step 440, the data processing method provided by the embodiments of the present disclosure may further include: step 4701 to step 4704.

Step 4701, receiving a mode update request for a first data source;

step 4702, responding to the mode updating request, describing the data mode updated by the first data source according to a preset description form, and generating third mode information;

step 4703, performing compatibility check on the mode update according to the first mode information and the third mode information;

step 4704, determining whether the mode updating is compatible according to the checking result, if yes, executing step 4705, and if not, executing step 4706;

step 4705, updating the stored mode information corresponding to the first data source according to the third mode information;

step 4706, the stored pattern information corresponding to the first data source is denied for updating.

For example, when the data mode of the first data source changes, the mode update registration may be performed on the first data source again, for example, a new data mode is input, and an update button is clicked, so that a mode update request for the first data source may be sent to a second electronic device where the mode registration service is located. After receiving the mode update request of the first data source, the second electronic device where the mode registration service is located can describe the data mode updated by the first data source according to a preset description form, and generate third mode information. Then, the first mode information and the third mode information are subjected to compatibility checking, such as forward compatibility, backward compatibility and full compatibility checking. If the incompatible modification is detected in the process, the mode evolution is not allowed, namely the updating of the mode information stored correspondingly by the first data source is refused, otherwise, the updating of the mode information stored correspondingly by the first data source is carried out, for example, the corresponding mode table is updated: and updating the updated serialization mode to the corresponding serialization mode table, and updating the third mode information to the corresponding mode information table.

In this way, downstream data consumers can be assured that upstream producers will not send them data that cannot be processed. In this case, the shared pattern table may ensure that the user can deserialize all data from upstream writing, preventing changes that disrupt compatibility, thus making the data pipeline more robust.

Based on this, in an alternative embodiment, the mode registration request may further include a type of compatibility check corresponding to the first data source, and accordingly, the step 4705 may specifically include:

acquiring a type of compatibility check corresponding to the first data source;

Here, different types of compatibility checks may be set according to actual requirements when different data sources are registered, so that when a subsequent data source needs to change its data mode, the compatibility check may be performed according to the corresponding type of compatibility check.

It should be noted that the execution sequence of the first step and the second step in the above three steps is not limited. Specifically, the update information of the third mode information compared with the first mode information may be determined by comparing the data structure corresponding to the first mode information with the data structure corresponding to the third mode information, where the update information may include, for example, an add field, a delete field, and the like. For example, the third mode information includes three fields of "name", "phone" and "age", and the first mode information includes two fields of "name" and "phone", and the update information is an added field as known by comparison.

Because the results of the compatibility check obtained for different front and back update information are different in different types of compatibility check, different check strategies can be adopted to determine whether the mode update is compatible.

Therefore, the compatibility check is carried out on the mode update by combining the update information according to the types of the compatibility check of different data sources, so that the data sources can carry out compatible mode update according to actual requirements, and the mode update requirements of the data sources under different scenes are met.

The type of the compatibility check may include forward compatibility, backward compatibility or full compatibility, and correspondingly, the step of performing the compatibility check on the current mode update according to the update information according to the type of the compatibility check may specifically include:

Here, the optional field may be a field using a defined default value when there is no value.

Illustratively, in the case that the type of the compatibility check is forward compatible, judging whether the third mode information is added with a new field or deleted with an optional field compared with the first mode information, if so, determining that the mode update is compatible, and if not, not; if the type of the compatibility check is backward compatibility, judging whether the third mode information is added with optional fields or deleted fields compared with the first mode information, if so, determining that the mode update is compatible, and if not, not; and judging whether the third mode information is added with optional fields or deleted with the optional fields compared with the first mode information under the condition that the type of the compatibility check is fully compatible, if so, determining that the mode update is compatible, and if not, not compatible.

Therefore, the compatibility inspection aiming at different compatibility inspection types can be realized, and the diversity of the compatibility inspection is ensured, so that different compatibility inspection mechanisms can be set for each data source according to actual requirements, and the personalized requirements of the data source updating process are met.

In addition, in an optional embodiment, after updating the stored mode information corresponding to the first data source according to the third mode information, the data processing method may further include:

acquiring registration information of a third data source;

Here, the third data source may be a downstream data source associated with the first data source. The registration information may be information stored at the time of registration of the third data source, including, but not limited to, data source type, data source identification, table structure description information, serialization manner, serialization pattern, and the like.

Specifically, whether the data source stores serialized data or non-serialized data may be determined according to the data source type, table structure description, or serialization manner. For example, if the type of the data source in the registration information is redis, kafka, etc. or the registration information includes a serialization manner, it may be determined that the data source stores the serialized data; if the registration information includes a table structure description, it may be determined that non-serialized data is stored in the data source.

In addition, in a specific example, the table structure update statement may be, for example, an sql statement, and correspondingly, the third data source may be, for example, a mysql database. In this way, in the heterogeneous data source, when the data mode of the upstream data source is changed, after compatibility is determined, mode update registration is performed on the upstream data source, and when the downstream data source is a data source of non-serialized data, a table structure update statement corresponding to the downstream data source is generated according to the newly generated third mode information, so that the table structure of the downstream data source can be automatically updated according to the table structure update statement, when the data mode of the upstream data source is changed, the downstream data source does not need to be notified, and the downstream data source can automatically analyze the data of the new data mode without code update.

In order to better understand the data processing method provided by the embodiments of the present disclosure, the following specific description is provided with reference to several practical application scenarios.

As shown in fig. 8, after the data producer serializes the produced data in accordance with the required serialization pattern, the data producer transmits the serialized data to kafka, and can register the data pattern of the data source kafka. Specifically, the data source kafka sends a mode registration request to the second electronic device where the mode registration service is located, after receiving the mode registration request, the second electronic device may first determine whether the data source kafka is a registered data source, if not, store, through the mode registration service, a serialization mode used when serializing data into a corresponding serialization mode table, and simultaneously generate a general mode (i.e., mode information) according to the serialization mode, and store the general mode into a corresponding general mode table (i.e., mode information table); if the data source is registered, rejecting the mode registration request.

When the data consumer needs to synchronize the serialized data in the kafka to the downstream database hive, a mode acquisition request can be sent to the second electronic device, the mode registration service in the second electronic device can acquire the corresponding serialized mode from the serialized mode table according to the data source identifier of the kafka carried in the request and send the serialized mode to the data consumer, and the data consumer can obtain the non-serialized data by performing anti-serialization processing on the serialized data acquired from the kafka by using the serialized mode and store the non-serialized data in the database hive.

In addition, as shown in fig. 8, in the data query scenario, the data querier may send a data query request to the second electronic device, and the data query service in the second electronic device may obtain, according to the data source identifier of kafka carried in the request, a corresponding generic pattern from the generic pattern table, and send the generic pattern to the data querier, so that the data querier queries, without parsing the data, which fields and field types are included in the kafka according to the generic pattern. Of course, the data query service in the second electronic device may obtain, according to the data source identifier of kafka carried in the request, a corresponding serialization pattern from the serialization pattern table, and send the serialization pattern to the data querier, so that the data querier uses the serialization pattern to parse the data in kafka.

As shown in fig. 9, in the case of synchronizing data stored in the database hive to the message queue redis, after the data producer produces data and stores it to the database hive, the database hive may be registered. Specifically, the data source hive sends a mode registration request to a second electronic device where the mode registration service is located, after receiving the mode registration request, the second electronic device can firstly judge whether the second electronic device is a registered data source, if the second electronic device is not the registered data source, a general mode is generated according to a table structure definition through the mode registration service, and the general mode is stored in a corresponding general mode table; if the data source is registered, rejecting the mode registration request.

When the data consumer needs to synchronize the data in the database hive to the downstream message queue redis, the data consumer can firstly send a mode registration request to the second electronic device where the mode registration service is located, wherein the request can carry a data source identifier of the redis and a data source identifier of the hive associated with the redis, acquire a general mode corresponding to the database hive according to the data source identifier through the mode registration service, further generate a serialization mode corresponding to the serialization mode of the redis according to the general mode, and store the serialization mode into a serialization mode table.

Then, the data consumer sends a mode acquisition request to the second electronic device, the mode registration service in the second electronic device can acquire a corresponding serialization mode from the serialization mode table according to the data source identifier of the redis carried in the request, and send the corresponding serialization mode to the data consumer, and the data consumer can obtain the serialized data by using the serialization mode to carry out serialization processing on the data read from hive, and synchronize the serialized data to the redis.

It should be noted that, the application scenario described in the foregoing embodiments of the present disclosure is for more clearly describing the technical solution of the embodiments of the present disclosure, and does not constitute a limitation to the technical solution provided by the embodiments of the present disclosure, and those skilled in the art can know that, with the appearance of a new application scenario, for example, the case of data transmission between data sources of the same type but different data modes, or between any data sources of the same data mode, the technical solution provided by the embodiments of the present disclosure is also applicable.

Based on the same inventive concept, the present disclosure also provides a data processing apparatus. This is described in detail with reference to fig. 10.

Fig. 10 is a schematic diagram showing a structure of a data processing apparatus according to an exemplary embodiment.

As shown in fig. 10, the data processing apparatus 100 may specifically include:

a registration request receiving module 1001 configured to perform receiving a pattern registration request for a first data source; wherein the pattern registration request includes a data source identifier corresponding to the first data source;

a data pattern obtaining module 1002 configured to obtain a data pattern corresponding to the first data source according to the pattern registration request;

a data pattern description module 1003 configured to perform description of a data pattern in a preset description form, generating first pattern information;

a mode information registration module 1004 configured to perform storing first mode information to a mode information table according to a data source identification to enable registration of a first data source; wherein the first pattern information is used to process data associated with the first data source.

The data processing apparatus 100 will be described in detail below, specifically as follows:

in one embodiment, the mode registration request includes a type of data, and the data mode description module 1003 specifically may include:

the data pattern obtaining module 1002 may specifically include:

In one embodiment, in the case where the type of data is serialized data, the data processing apparatus 100 described above may further include:

Based on this, in one embodiment, the data processing apparatus 100 as referred to above may further include:

In addition, in one embodiment, the data processing apparatus 100 described above may further include:

On the basis of the above embodiments, in one embodiment, the data processing apparatus 100 described above may further include:

Based on this, in one embodiment, the pattern registration request may further include a type of compatibility check corresponding to the first data source;

the update compatibility checking module may specifically include:

the above-mentioned compatibility checking sub-module may specifically include:

Therefore, the mode registration service is provided for the heterogeneous data sources, that is, in the process of registering the first data source in the heterogeneous data sources, the data mode corresponding to the first data source is described according to the preset description form, the first mode information is generated, and then the first mode information is stored in the mode information table, so that the registration of the first data source is realized, when the data associated with the first data source needs to be processed, as the first data source adopts the uniform description form to generate the universal mode information like other heterogeneous data sources during the registration, the data consumer of the first data source can adopt the universal mode information to process the data of the first data source and the data of other heterogeneous data sources, and the data output by different data sources do not need to be unified by adopting different data standardization steps respectively.

Based on the same inventive concept, the embodiment of the present disclosure further provides an electronic device, which is specifically described in detail with reference to fig. 11. Fig. 11 is a block diagram illustrating a structure of an electronic device according to an exemplary embodiment.

As shown in fig. 11, the electronic device 110 is capable of implementing a data processing method and a block diagram of an exemplary hardware architecture of the electronic device of the data processing apparatus according to the embodiment of the present disclosure.

The electronic device 110 may include a processor 1101 and a memory 1102 storing computer program instructions.

In particular, the processor 1101 may comprise a Central Processing Unit (CPU), or an application specific integrated circuit (application specific integrated circuit, ASIC), or may be configured to implement one or more integrated circuits of embodiments of the present application.

Memory 1102 may include mass storage for information or instructions. By way of example, and not limitation, memory 1102 may comprise a Hard Disk Drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or universal serial bus (universal serial bus, USB) drive, or a combination of two or more of these. Memory 1102 may include removable or non-removable (or fixed) media where appropriate. Memory 1102 may be internal or external to the integrated gateway device, where appropriate. In a particular embodiment, the memory 1102 is a non-volatile solid state memory. In a particular embodiment, the memory 1102 includes Read Only Memory (ROM). The ROM may be mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically Erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these, where appropriate.

The processor 1101 reads and executes the computer program instructions stored in the memory 1102 to perform the steps of:

a processor 1101 that performs receiving a mode registration request for a first data source; wherein the pattern registration request includes a data source identifier corresponding to the first data source; according to the mode registration request, a data mode corresponding to the first data source is acquired; describing a data mode according to a preset description form, and generating first mode information; storing first mode information into a mode information table according to the data source identification so as to realize the registration of a first data source; wherein the first pattern information is used to process data associated with the first data source.

In one embodiment, the processor 1101 specifically executes, in a case where the type of data is serialized data, describing a serialization pattern of the first data source according to a preset description form, and generating first pattern information; and under the condition that the type of the data is non-serialized data, describing the table structure of the first data source according to a preset description form, and generating first mode information.

In one embodiment, the processor 1101 specifically further performs determining a second data source associated with the first data source based on the associated data source identification, the second data source being different from the data pattern of the first data source; acquiring second mode information corresponding to a second data source under the condition that the second data source is a registered data source; the second mode information is generated mode information which describes a data mode corresponding to the second data source according to a preset description form; and generating a serialization mode corresponding to the first data source according to the second mode information in a serialization mode.

In one embodiment, the processor 1101 specifically further stores the serialization pattern corresponding to the first data source in a serialization pattern table according to the data source identifier.

In addition, in one embodiment, the processor 1101 is further configured to receive a mode acquisition request from a requesting party for the first data source; the mode acquisition request comprises a data source identifier corresponding to the first data source; responding to a mode acquisition request, and acquiring a serialization mode corresponding to the data source identifier from a serialization mode table; a serialization pattern is sent to the requestor, the serialization pattern being usable to process data associated with the first data source.

In one embodiment, the processor 1101 specifically further executes a data query request by a receiving requestor for a first data source; the data query request comprises a data source identifier corresponding to the first data source; responding to a data query request, and acquiring first mode information corresponding to a data source identifier from a mode information table; first schema information is sent to the requesting party, the first schema information being usable to query a data structure of the first data source.

Based on this, in one embodiment, the above-described involves the processor 1101 also performing receiving a mode update request for the first data source; responding to a mode updating request, describing a data mode updated by the first data source according to a preset description form, and generating third mode information; according to the first mode information and the third mode information, compatibility checking is carried out on the mode updating; under the condition that the mode updating is compatible according to the checking result, updating the stored mode information corresponding to the first data source according to the third mode information; and under the condition that the mode updating is not compatible according to the checking result, refusing to update the stored mode information corresponding to the first data source.

In one embodiment, the foregoing relates to the processor 1101 further performing a type of acquiring a compatibility check corresponding to the first data source; comparing the first mode information with the third mode information to obtain updated information; and according to the type of the compatibility check, carrying out the compatibility check on the mode update according to the update information.

In one embodiment, the processor 1101 further performs determining that the mode update is compatible if the update information is an add field or a delete optional field if the type of the compatibility check is forward compatibility; if the update information is not an added field or a deleted optional field, determining that the mode update is incompatible; if the type of the compatibility check is backward compatibility, if the update information is an optional field or a deleted field, determining that the mode is updated and compatible; if the update information is not an optional field or a deleted field, determining that the mode update is not compatible; under the condition that the type of the compatibility check is full compatibility, if the update information is an optional field added or deleted, the mode update compatibility is determined; if the update information is not the addition or deletion of the optional field, the mode update incompatibility is determined.

In one embodiment, the processor 1101 further performs determining a third data source associated with the first data source based on an associated data source identifier corresponding to the first data source; the third data source is a registered data source for storing target data, and the target data is data taken from the first data source; acquiring registration information of a third data source; under the condition that the non-serialized data stored in the third data source is determined according to the registration information, generating a table structure update statement corresponding to the third data source according to the first mode information and the third mode information; and sending a table structure update statement to the third data source, wherein the table structure update statement is used for indicating updating of the table structure of the third data source.

In one example, the electronic device 110 may also include a transceiver 1103 and a bus 1104. As shown in fig. 11, the processor 1101, the memory 1102, and the transceiver 1103 are connected to each other via a bus 1104 and perform communication with each other.

Bus 1104 includes hardware, software, or both. By way of example, and not limitation, the buses may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a micro channel architecture (MCa) bus, a Peripheral Control Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus, or a combination of two or more of the above. Bus 1104 may include one or more buses, where appropriate. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

The embodiments of the present disclosure also provide a computer storage medium having stored therein computer executable instructions for implementing the data processing method described in the embodiments of the present disclosure.

In some possible implementations, aspects of the methods provided by the present disclosure may also be implemented in the form of a program product comprising program code for causing a computer device to carry out the steps of the methods described in the foregoing specification in accordance with the various exemplary implementations of the disclosure, when the program product is run on the computer device, e.g., the computer device may carry out the data processing methods described in the examples of the disclosure.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus and computer program products according to the disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable information processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable information processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable information processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable information processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present disclosure without departing from the spirit or scope of the disclosure. Thus, the present disclosure is intended to include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. A method of data processing, comprising:

acquiring a data mode corresponding to the first data source according to the mode registration request;

describing the data mode according to a preset description form, and generating first mode information;

Storing the first mode information into a mode information table according to the data source identification so as to realize the registration of the first data source; wherein the first pattern information is used for processing data associated with the first data source;

the mode registration request comprises a data type, and the mode registration request further comprises a serialization mode and an associated data source identifier corresponding to the first data source when the data type is serialization data;

the obtaining the data mode corresponding to the first data source according to the mode registration request includes:

determining a second data source associated with the first data source according to the associated data source identification, wherein the second data source is different from the first data source in data mode;

acquiring second mode information corresponding to the second data source under the condition that the second data source is a registered data source; the second mode information is mode information generated by describing a data mode corresponding to the second data source according to the preset description form;

and generating a serialization mode corresponding to the first data source according to the second mode information according to the serialization mode.

2. The method of claim 1, wherein the describing the data pattern in a predetermined description form generates first pattern information, including:

describing a serialization mode of the first data source according to the preset description form under the condition that the data type is serialization data, and generating first mode information;

and under the condition that the data type is non-serialized data, describing the table structure of the first data source according to the preset description form, and generating first mode information.

3. The method according to claim 2, wherein in the case where the type of data is serialized data, after acquiring the data pattern corresponding to the first data source according to the pattern registration request, the method further comprises:

and storing the serialization mode corresponding to the first data source into a serialization mode table according to the data source identifier.

4. A method according to claim 3, wherein after storing the serialization pattern corresponding to the first data source into a serialization pattern table according to the data source identification, the method further comprises:

receiving a mode acquisition request of a requester for the first data source; the mode acquisition request comprises a data source identifier corresponding to the first data source;

Responding to the mode acquisition request, and acquiring a serialization mode corresponding to the data source identifier from the serialization mode table;

the serialization schema is transmitted to the requestor, the serialization schema being usable to process data associated with the first data source.

5. The method of claim 1, wherein after storing the first pattern information in a pattern information table according to the data source identification, the method further comprises:

receiving a data query request of a requester for the first data source; the data query request comprises a data source identifier corresponding to the first data source;

responding to the data query request, and acquiring first mode information corresponding to the data source identifier from the mode information table;

the first schema information is sent to the requestor, which can be used to query a data structure of the first data source.

6. The method according to any one of claims 1-5, wherein after storing the first pattern information into a pattern information table according to the data source identification, the method further comprises:

Receiving a mode update request for the first data source;

responding to the mode updating request, describing the data mode updated by the first data source according to the preset description form, and generating third mode information;

according to the first mode information and the third mode information, compatibility checking is conducted on the mode updating;

7. The method of claim 6, wherein the pattern registration request further includes a type of compatibility check corresponding to the first data source;

and performing compatibility checking on the mode update according to the first mode information and the third mode information, including:

acquiring a type of compatibility check corresponding to the first data source;

8. The method of claim 7, wherein the type of compatibility check includes forward compatibility, backward compatibility, or full compatibility;

and according to the type of the compatibility check, carrying out the compatibility check on the mode update according to the update information, wherein the compatibility check comprises the following steps:

if the type of the compatibility check is forward compatibility, determining that the mode is compatible with the update if the update information is an added field or a deleted optional field; if the update information is not an added field or a deleted optional field, determining that the mode update is incompatible;

if the type of the compatibility check is backward compatibility, determining that the mode is compatible with the update if the update information is an optional field or a deleted field; if the update information is not an optional field or a deleted field, determining that the mode update is not compatible;

if the type of the compatibility check is full compatibility, determining that the mode is compatible with the update if the update information is an optional field added or deleted; and if the updating information is not adding or deleting optional fields, determining that the mode updating is not compatible.

9. The method of claim 6, wherein after updating the stored mode information corresponding to the first data source according to the third mode information, the method further comprises:

determining a third data source associated with the first data source according to the associated data source identifier corresponding to the first data source; the third data source is a registered data source for storing target data, and the target data is data fetched from the first data source;

acquiring registration information of the third data source;

generating a table structure update statement corresponding to the third data source according to the first mode information and the third mode information under the condition that the non-serialized data stored in the third data source is determined according to the registration information;

and sending the table structure update statement to the third data source, wherein the table structure update statement is used for indicating to update the table structure of the third data source.

10. A data processing apparatus, comprising:

the data pattern description module is configured to execute the description of the data pattern according to a preset description form and generate first pattern information;

a mode information registration module configured to perform storing the first mode information to a mode information table according to the data source identification to realize registration of the first data source; wherein the first pattern information is used for processing data associated with the first data source;

the data pattern acquisition module includes:

the mode acquisition sub-module is configured to acquire second mode information corresponding to the second data source when the second data source is a registered data source; the second mode information is mode information generated by describing a data mode corresponding to the second data source according to the preset description form;

And the mode generation sub-module is configured to execute the serialization mode corresponding to the first data source according to the second mode information according to the serialization mode.

11. The apparatus of claim 10, wherein the data pattern description module comprises:

the first description submodule is configured to execute a serialization mode of describing the first data source according to the preset description form and generate first mode information when the type of the data is the serialization data;

and the second description submodule is configured to execute the table structure describing the first data source according to the preset description form and generate first mode information under the condition that the type of the data is non-serialized data.

12. The apparatus of claim 11, wherein in the event that the type of data is serialized data, the apparatus further comprises:

and the serialization mode storage module is configured to store the serialization mode corresponding to the first data source into a serialization mode table according to the data source identifier after acquiring the data mode corresponding to the first data source according to the mode registration request.

13. The apparatus of claim 12, wherein the apparatus further comprises:

an acquisition request receiving module configured to receive a mode acquisition request of a requester for the first data source after storing a serialization mode corresponding to the first data source into a serialization mode table according to the data source identifier; the mode acquisition request comprises a data source identifier corresponding to the first data source;

a serialization pattern acquisition module configured to perform acquisition of a serialization pattern corresponding to the data source identifier from the serialization pattern table in response to the pattern acquisition request;

a serialization mode transmitting module configured to perform transmitting the serialization mode to the requestor, the serialization mode being operable to process data associated with the first data source.

14. The apparatus of claim 10, wherein the apparatus further comprises:

a query request receiving module configured to receive a data query request from a requester for the first data source after storing the first pattern information into a pattern information table according to the data source identification; the data query request comprises a data source identifier corresponding to the first data source;

A mode information acquisition module configured to perform acquisition of first mode information corresponding to the data source identifier from the mode information table in response to the data query request;

and a mode information sending module configured to send the first mode information to the requester, wherein the first mode information can be used for querying a data structure of the first data source.

15. The apparatus according to any one of claims 10-14, wherein the apparatus further comprises:

an update request receiving module configured to receive a mode update request for the first data source after storing the first mode information to a mode information table according to the data source identification;

the mode updating description module is configured to execute the description of the data mode updated by the first data source according to the preset description form in response to the mode updating request, and generate third mode information;

an update compatibility checking module configured to perform compatibility checking on the current mode update according to the first mode information and the third mode information;

And the mode updating refusing module is configured to execute refusing to update the stored mode information corresponding to the first data source when the mode updating is determined to be incompatible according to the checking result.

16. The apparatus of claim 15, wherein the pattern registration request further comprises a type of compatibility check corresponding to the first data source;

the update compatibility checking module includes:

an information acquisition sub-module configured to perform comparison of the first mode information with the third mode information to acquire update information;

and the compatibility checking sub-module is configured to execute compatibility checking on the current mode update according to the type of the compatibility checking and the update information.

17. The apparatus of claim 16, wherein the type of compatibility check comprises forward compatibility, backward compatibility, or full compatibility;

the compatibility check sub-module includes:

A second checking unit configured to perform, if the type of the compatibility check is backward compatibility, determining that the mode update is compatible if the update information is an add optional field or a delete field; if the update information is not an optional field or a deleted field, determining that the mode update is not compatible;

a third checking unit configured to perform, if the type of the compatibility check is full compatibility, determining that the mode update is compatible if the update information is an add or delete optional field; and if the updating information is not adding or deleting optional fields, determining that the mode updating is not compatible.

18. The apparatus of claim 15, wherein the apparatus further comprises:

a data source determining module configured to determine a third data source associated with the first data source according to an associated data source identifier corresponding to the first data source after updating stored mode information corresponding to the first data source according to the third mode information; the third data source is a registered data source for storing target data, and the target data is data fetched from the first data source;

an update statement generation module configured to perform generation of a table structure update statement corresponding to the third data source according to the first pattern information and the third pattern information in a case where it is determined that non-serialized data is stored in the third data source according to the registration information;

and the updating statement sending module is configured to send the table structure updating statement to the third data source, wherein the table structure updating statement is used for indicating updating of the table structure of the third data source.

19. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the data processing method of any of claims 1 to 9.

20. A computer readable storage medium, characterized in that instructions in the computer readable storage medium, when executed by a processor of a data processing apparatus, cause the data processing apparatus to implement the data processing method of any one of claims 1 to 9.