WO2021204041A1

WO2021204041A1 - Data processing

Info

Publication number: WO2021204041A1
Application number: PCT/CN2021/084306
Authority: WO
Inventors: 孙鹏
Original assignee: 支付宝(杭州)信息技术有限公司
Priority date: 2020-04-08
Filing date: 2021-03-31
Publication date: 2021-10-14
Also published as: CN111414369B; CN111414369A; CN117951139A

Abstract

A data processing method, apparatus and device. The method comprises: obtaining configuration information of data of data source providers; configuring, according to the configuration information, data information provided by data sources, and storing the configured data information into a database of a data platform; obtaining a data invoking request of a user of the data platform; and according to the invoking request, providing data provided by the data source provider for the user on the basis of customized configuration information of the user and the configured data information in the data platform.

Description

data processing

Technical field

One or more embodiments of this specification relate to the field of computer technology, and in particular to a data processing method, device, and equipment.

Background technique

At present, with the development of Internet information technology, data has been applied to every industry and business intelligence field. Each user organization generates massive amounts of data every day, and people need to mine and use massive amounts of data. For example, in the financial field, each institution generates a large amount of transaction data every day. Generally speaking, in order to ensure the normal operation of multiple institutions, each institution needs to obtain the required data through the data platform to achieve intelligent risk control, user profile, and intelligent marketing. And so on, so as to be able to accurately evaluate the transaction users, in order to better provide users with services. For example, a credit institution obtains user-corresponding data from the corresponding data provider through the data platform, thereby evaluating the user's business, and providing the user with the corresponding service based on the evaluation result.

The data platform is a platform that integrates application interfaces such as data access, data processing, data storage, query retrieval, analysis and mining. Each user can call the required data through the data platform according to their own needs.

However, when users currently call data through a data platform, in order to meet the user's business processing needs, they often need to obtain data from data sources of multiple institutions. At this time, according to the user's data call request, it is necessary to call the required data through the query engine corresponding to the data source of each organization, resulting in a large workload and cumbersome call. At this time, when large-scale, multi-tenant spaces need to use open data from multiple data sources for big data calculations, how to configure and customize the use of open data from multiple data sources becomes an urgent need to solve Technical issues.

Summary of the invention

In view of this, one or more embodiments of this specification provide a data processing method, device, and equipment, which are used to implement unified configuration of data from multiple data sources while satisfying users' customized use of data.

In order to solve the above technical problems, the embodiments of this specification are implemented as follows:

The embodiment of this specification provides a data processing method, which is applied to a data platform, and includes: obtaining data configuration information of each data source provider; the data configuration information includes basic information and interface information; The configuration information configures the data information provided by each of the data sources, and saves the configured data information in the database of the data platform; obtains the call of the data provided by the data source provider by the user of the data platform Request; based on the user's customized configuration information and the configured data information, the data provided by the data source provider is provided to the user.

The embodiment of this specification provides a data processing device, which is applied to a data platform and includes: a configuration information acquisition module for acquiring data configuration information of each data source provider; the data configuration information includes basic information And interface information; a configuration module, used to configure the data information provided by each of the data sources according to the configuration information, and save the configured data information in the database of the data platform; the data call request acquisition module is used In order to obtain the call request of the user of the data platform for the data provided by the data source provider; the customized data providing module is used to convert all the data based on the customized configuration information of the user and the configured data information. The data provided by the data source provider is provided to the user.

A data processing device provided by an embodiment of this specification includes: at least one processor; and a memory communicatively connected with the at least one processor. Wherein, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can: Configuration information; the configuration information of the data includes basic information and interface information; configures the data information provided by each data source according to the configuration information, and saves the configured data information in the database of the data platform; Obtain the call request of the user of the data platform for the data provided by the data source provider; based on the customized configuration information of the user and the configured data information, provide the data provided by the data source provider To the user.

An embodiment of this specification provides a computer-readable medium having computer-readable instructions stored thereon, and the computer-readable instructions can be executed by a processor to implement a data processing method.

An embodiment of this specification can achieve the following beneficial effects: by acquiring the configuration information of the data of each data source provider, configuring the data information provided by each of the data sources according to the configuration information, and saving the configured data information to the data platform In order to realize the unified configuration of data information provided by multiple data sources on the data platform, thereby improving the efficiency of data source access and use; the user who obtains the data platform requests data from the data source provider based on The customized configuration information of the user provides the data provided by the data source provider to the user, and the data can be customized according to the needs of the user, so as to meet the personalized customization needs of the user using the data.

Description of the drawings

The drawings described here are used to provide a further understanding of one or more embodiments of this specification, and constitute a part of one or more embodiments of this specification. The exemplary embodiments of this specification and their descriptions are used to explain one or more embodiments of this specification. The multiple embodiments do not constitute an improper limitation of one or more embodiments in this specification. In the attached picture:

FIG. 1 is a schematic diagram of an overall scheme of a data processing method provided by an embodiment of this specification;

2 is a schematic flowchart of a data processing method provided by an embodiment of this specification;

3 is a schematic diagram of a configuration information configuration interface in a data processing method provided by an embodiment of this specification;

4 is a schematic diagram of a user-customized configuration information configuration interface in a data processing method provided by an embodiment of this specification;

FIG. 5 is a schematic structural diagram of a data processing device corresponding to FIG. 2 provided by an embodiment of this specification;

FIG. 6 is a schematic structural diagram of a data processing device corresponding to FIG. 2 provided by an embodiment of the specification.

Detailed ways

In order to make the purpose, technical solutions and advantages of one or more embodiments of this specification clearer, the technical solutions of one or more embodiments of this specification will be clearly and completely described below in conjunction with specific embodiments of this specification and the corresponding drawings. . Obviously, the described embodiments are only a part of the embodiments in this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of one or more embodiments of this specification.

The technical solutions provided by the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

Fig. 1 is a schematic diagram of an overall scheme of a data processing method provided by an embodiment of this specification. As shown in FIG. 1, it includes a data source provider 101, a data platform 103, and a user 105. It should be noted that the user 105 here may refer to a device used by the user, such as a company device, a bank device, or a personal device, and so on. The data source provider can provide data corresponding to multiple data sources. For example, Figure 1 includes n data sources, data source 1 provides data A, data source 2 provides data B, and data source 3 provides data C, ..., data Source n provides data m. The data platform 103 does not store data, but only stores the data information corresponding to the data. The data information corresponding to the data provided by these n data sources needs to be uniformly configured on the data platform 103, which can be understood as the data platform 103 needs The data information corresponding to the data provided by the n data sources is configured and saved in the database of the data platform 103. When storing, the data information corresponding to the data provided by a data source can be stored in a database. As shown in Figure 1, the data information a of data A provided by the data source 1 can be stored in the database 1 of the data platform 103. , Store the data information b of the data B provided by the data source 2 in the database 2 of the data platform 103, ..., store the data information m of the data M provided by the data source n in the database n of the data platform 103. The device used by the user 105 can send a data call request to the data platform to make a customized call to the data provided by the data source provider. The customized call of the user 105 is based on the configuration information and customized configuration information of the data platform 103. The interface provided by the party performs data query to meet the individual needs of users using data.

Next, a data processing method provided in the embodiment of the specification will be described in detail in conjunction with the accompanying drawings:

Fig. 2 is a schematic flowchart of a data processing method provided by an embodiment of the specification. From a program perspective, the execution subject of a process can be a program or an application client loaded on an application server.

As shown in Figure 2, the process can include the following steps:

Step 202: Obtain data configuration information of each data source provider; the data configuration information may include basic information and interface information.

The data source provider can refer to the database or database server used by the database application. For example, data providers can provide data to the data platform through the database or database server used by their own open applications.

Configuration information can represent variables that can be edited by the user in the configuration table or configuration interface. After the data platform obtains the data provided by multiple data sources from the data source provider, it can match the data configuration information of each data source provider. The data is configured.

Step 204: Configure the data information provided by each of the data sources according to the configuration information, and save the configured data information in the database of the data platform.

The data information provided by the data source provider may include the data access mode, query mode, usage mode, and the identity of the data source, etc. For example, the data X provided by data source A is transaction data. At this time, the transaction data may include the transaction scenario, access address information for obtaining the transaction data, query method information, usage format information, and the transaction corresponding to the transaction data X Type, transaction name, transaction time, specific transaction content, etc.

The data platform can be a platform that integrates application interfaces such as data access, data processing, data storage, query retrieval, analysis and mining. The data platform can store data information provided by various data sources to realize data resource sharing for various institutions. The data platform can provide various open data display and application functions. After applying for the use of data through the data platform, you can access the query data through its application development. The data platform may include multiple databases for storing data information provided by various data source providers. In the specific storage, the data information can be classified and stored, and one type of data information can be stored in the same type of database; it can also be stored according to the data source, and the data information provided by a data source can be stored in the same type of database; Of course, in practical applications, you can also choose to store the data in one database or multiple databases according to the amount of data. The specific storage can be set according to the actual application.

Step 206: Obtain a call request of the user of the data platform for the data provided by the data source provider.

It should be noted that the “user” mentioned in this manual can refer to the specific device used by the user.

In practical applications, the data platform can sign contracts with some institutions and cooperate with each other, and provide the contracted institutions with the required data according to the needs of the contracted institutions. In addition, the data platform can also be open to all institutions, and each institution can retrieve and query data through the data platform, apply for the use of data, and so on.

When users of the data platform need to use the data provided by the data source provider corresponding to the data information on the data platform, they can send a data call request to the data platform, requesting the data platform to call the data call interface of the corresponding data source provider Data, and provide the data obtained by the call.

Step 208: Based on the customized configuration information of the user and the configured data information, provide the data provided by the data source provider to the user.

The customized configuration information can indicate the user's demand for the use of data, and can also include the calling parameters of the designated part of the interface.

The method in Figure 2 obtains the configuration information of the data of each data source provider, configures the data information provided by each data source according to the configuration information, and saves the configured data information in the database of the data platform to Realize the unified configuration of data information provided by multiple data sources on the data platform, thereby improving the efficiency of data source access; the user who obtains the data platform requests the data of the data source provider based on the customized configuration of the user Information, the data provided by the data source provider is provided to the user, the data can be customized according to the needs of the user, and the personalized customization needs of the user's use of the data can be met.

The examples of this specification also provide some specific implementations of the method, which are described below based on FIG. 2.

In actual use requirements, when the data platform integrates data information provided by multiple data sources into the data platform, the business operations involved may include data source configuration and online and data source access detection. After multiple data sources are configured, test verification is required, and the verification can be stored on the data platform for each user to use.

Specifically, before obtaining the data configuration information of each data source provider, it may further include: obtaining configuration information for the data provided by the data source provider; for any piece of configuration information, testing the configuration information to obtain the test Result; when the test result indicates that the configuration information is configured correctly, the configuration information is saved on the data platform, and the configuration information is used online to perform the data information provided by the data source provider according to the configuration information Configuration. The configuration information provided by the data source provider is tested, and only after the test is passed can it go online and provide the equipment to each user for use.

In the specific test, you can test according to the set configuration information format, fill in the corresponding interface address and parameters in the configuration information in the test interface, click the test button to test, and view the returned test results. If the test fails, then Modify the configuration information and reset the test parameters. If the test is correct, you can request to go online.

When the configuration information is tested and adjusted, the test can be performed according to the dynamic script. If the test fails, the parameters can be adjusted according to the test result, for example, the signature calculation logic of the query interface can be adjusted.

Through the above method, before configuring the data information provided by the data source provider according to the configuration information, the configuration information can be tested first, and the configuration information that has passed the test is saved on the data platform. When the data provided by the data source provider is obtained After the data information, the data information can be configured directly according to the configuration information saved on the data platform, thereby improving the efficiency of the configuration.

The unified configuration of data provided by multiple data source providers mentioned in this manual is actually to save the data information provided by multiple data source providers in the database of the same data platform according to the corresponding configuration information. Specifically, when storing data information provided by multiple data source providers on the same data platform, it is necessary to clarify in which databases the data information of each data source should be stored. The specific implementation method can be as follows:

The basic information includes at least the provider information of the data provided by the data source; the data information provided by each data source is configured according to the configuration information, and the configured data information is saved to the data platform The database specifically includes: determining the provider information of the data information provided by the data source according to the configuration information; determining the database type corresponding to the provider information; storing the data information provided by the data source in the provider In the database corresponding to the party information.

It should be noted that when the data information provided by the data source provider is stored in the data platform, the provider information can be used to determine which database the data information provided by the data source provider should be stored. For example: Data source provider A provides data X. At this time, it can be based on the name or identification of data source provider A, the classification, application scenarios, coverage of data X, and the open mode of the data provided by data source provider A. Wait for information to determine that the data information corresponding to the data X should be stored in the R-type database. Among them, one database type can correspond to one database or multiple databases, and the number of databases corresponding to the database type can be set according to actual conditions.

Through the above method, the storage database type of the data information of each data source on the data platform is determined according to the provider information of the data provided by the data source, which can improve the configuration efficiency of the data information.

In practical applications, it can be that the data source provider actively sends the data information to the data platform for configuration storage, or the data platform sends a request for data information to each data source provider, requesting each data source provider to send the data The information is placed on the data platform for configuration storage. When the user needs to call data through the data platform, the user can send a data call request to the data platform. The data platform provides the user with a call interface for data call. The user calls the required data from the call interface. The interface information in the configuration information can at least include the interface Address information and transmission protocol information; when specifically acquiring data, it may include: determining the interface address of the data source provider according to the interface address information; accessing the interface address according to the transmission protocol corresponding to the transmission protocol information to obtain the data Data provided by the source.

In practical applications, the configuration information can be filled in through the interface provided in Figure 3. Specifically, it can be described in detail with reference to FIG. 3. As shown in Figure 3, the configuration information may include basic information and interface information. Basic information can include data source provider information, such as: the name or logo of the data source provider, data classification, application scenarios, coverage (for example: national, international, etc.) and the open mode of the data provided by the data source provider (For example: dedicated to designated institutions or for use by all institutions) and other information.

The interface information may include interface address information, transmission protocol information, parameter information, and so on. Among them, the interface address information can be filled in the interface address of the specified request data source, and the transmission protocol information can be filled in the transmission protocol used when the data platform and the interface of the specified request data source perform data transmission, such as: Transmission Control Protocol Protocol, TCP for short), User Datagram Protocol (UDP for short), Internet Protocol (IP for short), etc.

The parameter information may include parameter type, parameter name, parameter description information, parameter setting mode, and so on. It should be noted that for some parameters, it is necessary to call other interfaces of the data provider to obtain the parameter value. At this time, it is necessary to configure the data source information of the interface for obtaining the parameter value, and then configure the data source information for obtaining the data.

In the above method, the interface address provided by the data source provider can be determined from the interface address information, and the data provided by the data source provider can be obtained by accessing the interface address. When transferring data between different databases, the corresponding transmission protocol is adopted. Therefore, the transmission protocol between the data source provider and the data platform can be obtained from the interface information, and the interface address of the data source provider can be accessed according to the transmission protocol. To obtain the data provided by the data source provider.

When storing the data information provided by the data source provider in the database corresponding to the database type determined corresponding to the data source provider information, if the database does not have the required data format, the data information can be directly converted to its own data format Stored in the database, data format conversion is not required; however, if the database has data format requirements when storing, at this time, the data format of the data information provided by the data source provider needs to be converted into the data format required by the database. Before it can be stored. Specifically, the following methods can be used to achieve:

Before storing the data information provided by the data source in the database corresponding to the determined database type corresponding to the data source provider information, the method may further include: obtaining data format information of the database; according to the data format The information format converts the data information provided by the data source to obtain the data information after the format conversion. The storing the data information provided by the data source in the database corresponding to the determined database type corresponding to the data source provider information specifically includes: storing the data information after the format conversion in the corresponding database type In the database.

It should be noted that the data format can be an arrangement format in which data is stored in a file or record. Generally, when using an interface to transmit data, the data format can include: lightweight text data exchange format (JavaScript Object Notation, JSON for short), eXtensible Markup Language (XML for short), and data serialization format (Yet Another Markup Language, YAML for short).

For example, the data format of data information X provided by data source provider A is XML format, the data format of data information Y provided by data source provider B is YAML format, and the data format of data information Z provided by data source provider C is JSON) format. Through analysis, data information X needs to be stored in a type I database, data information Y needs to be stored in a type H database, and data information Z needs to be stored in a type J database. Assuming that the type I database, The stored data format corresponding to the H-type database is XML format, and the stored data format corresponding to the J-type database is YAML format. At this time, the data information X provided by the data source provider A can be directly stored in the I-type database; The data format of data information Y provided by data source provider B is converted from YAML format to XML format, and the data information Y in XML format is stored in a type H database; the data format of data information Z provided by data source provider C is changed by The JSON format is converted into YAML format, and then the Z data information in YAML format is stored in the J-type database.

In the actual application scenario, after the data information provided by each data source provider is configured on the same data platform, the data platform can provide each user with a data query interface and provide the user with the required data, for example: Bank A is processing In a large-sum loan transaction, it is necessary to evaluate the creditworthiness of the borrower, the property condition of the borrower, and the repayment ability, etc. At this time, Bank A needs to conduct a user portrait based on the basic portrait data of the borrower provided by other institutions For example, it is necessary to use the borrower’s company registration data provided by the Industrial and Commercial Bureau, obtain the borrower’s real estate data from the real estate management office, and the borrower’s deposit data and loan data in other banks. After the data information of these data is uniformly configured by the data platform, Bank A can obtain the data calling interface for calling the basic profile data of the borrower through the data platform, and customize the data obtained by calling it according to its own usage needs. Specifically, the providing the data in the database to the user based on the user's customized configuration information may specifically include: obtaining a user's request for data calling; and according to the user's customized configuration information, Determine the target database that stores the data information corresponding to the data called by the calling request; determine the calling interface corresponding to the data called by the calling request according to the data information in the target database; access the calling interface and call The data is provided to the user.

In practical applications, customized configuration information may refer to the customized use requirements of individual users for data. The customized configuration information for the user can be filled in through the interface provided in FIG. 4. Specifically, it can be described in detail with reference to FIG. 4.

Fig. 4 is a schematic diagram of a user-customized configuration information configuration interface in a data processing method provided by an embodiment of the specification. As shown in Figure 4, the user's customized configuration information can include the identification parameters of the calling interface and the signature method; it can also include the specified fields, the data format of the usage data, and the combination or routing of multiple data sources. For example, only a few fields in the data need to be obtained, or data in XML format needs to be used, or data needs to be obtained from a specified database.

It should be noted that the specified part of the interface here can represent the target database, and the identification parameters of the calling interface can represent the identification information of the target database; the signature method can represent the signature method of data encryption in the target database.

For example: the user needs to use data X. At this time, the customized configuration information for the user to use data X can at least include the data format of data X, the use field of data X, and the designated invocation interface of data X. The designated invocation interface can be determined according to the designated invocation interface. The target database for storing data X. After getting the data from the corresponding database according to the user's configuration information, the data is converted into data that the user can use to meet the user's personalized use needs.

The data platform can obtain the user's call request for data, for example: obtain the bank A's call request for the borrower 1's basic profile data.

After obtaining the user's data call request, the target database for storing the data that the user needs to call can be determined according to the user's customized information. In the application scenario, the user can specify the database to retrieve the data, or retrieve the data from all the databases that store the required data, which can be described according to the following two methods:

Method 1: Obtain the data information corresponding to the required data from the specified database.

More specifically, the customized configuration information includes a database identifier, and the database identifier is used to specify a target database for obtaining data information corresponding to the data called by the user.

The determining the target database storing the data information corresponding to the data called by the calling request according to the customized configuration information of the user may specifically include:

The database corresponding to the database identifier is determined as the target database.

The customized configuration information can include the database identifier, for example: the configuration information of user A specifies database Q and database R. At this time, when the data platform obtains the call request of user A for data X, it directly from database Q and database R Take out the data X and send it to user A.

Through the above method, according to the user's customized needs, it can be ensured that effective data is provided to the user in a more targeted manner, thereby improving the user's use efficiency of the data on the data platform.

Method 2: Get data information from all databases containing data information corresponding to the required data.

The determining the target database storing the data information corresponding to the data called by the calling request according to the customized configuration information of the user may specifically include: determining the location of the calling request according to the customized configuration information of the user. The data information type corresponding to the called data; determining all databases storing the data information of the data information type; determining all databases storing the data information of the data information type as the target database.

It should be noted that the user's customized configuration information may include the data information type corresponding to the data that needs to be called. Therefore, after obtaining the user's data calling request, the data information type corresponding to the data that the user needs to call can be determined. The data information type here can be classified according to the business scenario of the data, such as: asset data, identity data, credit data, and so on.

The steps in the second method above are mainly to meet business requirements. In actual applications, users may need to combine data provided by multiple data sources to make better use of the data for subsequent business operations. For example: a banking institution needs to complete a user portrait of a registered user, and needs some online and offline data about the registered user. At this time, it needs to combine the data provided by the online data source and the offline data source , Comprehensively determine the user portrait of the registered user.

The data of each data source is correspondingly stored in the corresponding database. Therefore, the data required by the user can be obtained from multiple databases to meet the complex scenarios of the user using the data.

According to the data type, all the databases storing the data of the data type can be determined. This type of data can be provided to users from these databases. For example, the data platform receives a call request from company S for employee A's identity data, and according to company S's configuration information for employee identity data, it can be determined that the data information type of employee A's identity data is basic profile data. Determine all the databases storing the basic profile data type of user A, namely database 1, database 2, and database 3. At this time, the data platform can obtain the corresponding data from each database according to the data information in database 1, database 2 and database 3. The source provider calls the data and provides it to company S so that company S can conduct a comprehensive evaluation of employee A.

Through the above method, the user is provided with the required data as comprehensively as possible according to the user's customized configuration information, so as to better meet the user's demand for data use and improve the user's experience of using the data on the data platform.

In practical applications, the data platform may have contracted targeted cooperative users, and these targeted cooperative users can sign an agreement to order data sources with the data platform. After signing the agreement, the user can obtain key parameter values such as the identity of the corresponding data source. After signing the contract, targeted cooperative users can customize and use related data sources. When querying data, the query engine provided on the data platform will also perform data query operations according to the user's customized configuration information.

For these targeted cooperative users, after the data platform obtains the data information from the data source provider, the data information can be configured in advance according to the configuration information of the targeted cooperative user and stored on the data platform. Specifically, the following steps may be included:

Before the request of the user who obtains the data platform to call the data provided by the data source provider, it may further include: obtaining data information corresponding to the data provided by the data source provider; The data information is preprocessed; the preprocessed data information is stored in the database of the data platform.

It should be noted that the preprocessing here may mean that the data information provided by the data source provider is configured in advance according to the configuration information of the user. The configured data information can be designated to be stored in the corresponding database of the data platform. Of course, the stored database may not be specified, and the specific implementation method can be set according to actual needs.

In the actual application process, according to the contract agreement, the data provided by the corresponding data source provider can be periodically sent to the targeted cooperative user, or after receiving the data call request of the targeted cooperative user, the data call interface can be provided to the user. Directed cooperating users to make data calls.

The customized configuration information includes the fields specified by the user. The providing the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information may specifically include: obtaining data provided by the data source provider; A field specified by the user is selected from the data; the content corresponding to the field is provided to the user.

In order to meet the data platform users’ data customization needs, accurate data can be provided to users according to user needs. Specifically, suppose that user A requires to call field A1 and field A2. These two fields belong to data X. At this time, you can Find the field A1 and the field A2 from the data X and send them to the user A.

In addition to specifying fields, users can also specify the data format information of the required data. The customized configuration information also includes data format information of the data required by the user. The providing the data provided by the data source provider to the user based on the customized configuration information of the user may specifically include: converting the data format of the data provided by the data source provider into the data The data format corresponding to the format information obtains the converted data; the converted data is provided to the user.

In practical applications, the data format of the data provided by the data source provider may be inconsistent with the data format of the data that the user needs to use. At this time, the data format of the data provided by the data source provider needs to be converted, which will meet the needs of the user. The data used in the data format is consistent with the data sent to the user.

For example: the user needs to call data X and data Y, the data format of data X is XML format, and the data format of Y is JSON format; and the data format required by the user to use data is XML format, at this time, the format of data Y needs to be changed from Convert JSON format to XML format, and then provide Y data in XML format to users.

Of course, after the user obtains the corresponding data from the data platform and uses the obtained data, the data usage log can also be counted, and the data usage and data quality can be displayed on the device used by the user after analysis and processing.

The data platform can provide data information service application programming interface (Application Programming Interface, API for short) for users to use, and it can also provide an open portal for external users to consult data sources and contract to use data sources.

After the data platform uniformly configures the data of each data source, when the user wants to call the required data from the data platform, it can provide the user with an interface for data call or query. This interface can be called a data view. The user's customized configuration information can be filled in the interface, for example, the user's identity, signature method, and usage format of the requested data can be filled in. In addition, in this interface, you can also fill in multiple data source information according to the application scenario to ensure that the user can call the data of multiple data sources for use at the same time.

The data processing method provided in the embodiment of this specification greatly improves the efficiency of data source access and use by uniformly configuring and controlling the data of multiple data sources, and providing users with customized usage data according to the individual needs of users. , It can also meet the needs of users. In addition, when the data platform uniformly configures the data, in order to ensure the quality of the configuration online, the configuration information has passed the process of test verification, process-based application online, and grayscale release, reducing the risk of production. In the face of more complex business scenarios, this solution also supports multiple data source combination queries, filtering and then displaying the data query results, which can solve the scenario of large-scale, multi-tenant space using open data from multiple data sources for big data calculations , The efficiency of large-scale data access, and how to ensure that it is customized and used according to business needs after access.

More specifically, the method provided in the embodiments of this specification can specifically achieve the following technical effects:

1) Unified access of multiple data sources and customization during use. While improving the efficiency of data source access, it also supports personalized customization requirements and meets the requirements of data localization and safe use.

2) Configuration and process management: data from multiple data sources is unified access to the data platform for configuration → testing → audit → online process. When users customize the data, they also need to carry out testing and verification processes, which reduces the cost of access At the same time, it can also guarantee the quality of data usage.

3) Configuration multi-version support, gray-scale release and online: Test and verify the configuration information, which can meet the requirements for adjusting the configuration information after the test fails, and at the same time, the revised configuration information may be brought online by gradually increasing the grayscale. The impact and loss of the project are controlled to a minimum.

4) Multi-data source routing and combined use: In order to meet business needs, multiple data sources can be combined for application scenarios, which can better meet users' data usage needs.

Based on the same idea, the embodiment of this specification also provides a device corresponding to the above method. FIG. 5 is a schematic structural diagram of a data processing device corresponding to FIG. 2 provided by an embodiment of the specification. As shown in Figure 5, the device can be applied to a data platform and can include:

The configuration information obtaining module 502 is used to obtain the configuration information of the data of each data source provider; the configuration information of the data includes basic information and interface information;

The configuration module 504 is configured to configure the data information provided by each of the data sources according to the configuration information, and save the configured data information in the database of the data platform;

The data call request obtaining module 506 is configured to obtain a call request of a user of the data platform for the data provided by the data source provider;

The customized data providing module 508 is configured to provide the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information.

Optionally, the basic information includes at least provider information of the data provided by the data source. The configuration module 504 may specifically include: a data source provider information determining unit for determining the provider information of the data information provided by the data source according to the configuration information; a database type determining unit for determining the provider The type of database corresponding to the information; the data storage unit is used to store the data information provided by the data source in the database corresponding to the provider information.

Optionally, the device may further include: a configuration information obtaining module, which is used to obtain configuration information for data provided by a data source provider; a configuration information test module, which is used to check the configuration information for any configuration information The test is performed to obtain the test result; the configuration information saving module is used to save the configuration information on the data platform when the test result indicates that the configuration information is correctly configured.

Optionally, the interface information includes at least interface address information and transmission protocol information. The device may further include: an interface address acquisition module, configured to determine the interface address of the data source provider according to the interface address information; and a data acquisition module, configured to access the interface according to the transmission protocol corresponding to the transmission protocol information Address to obtain the data provided by the data source.

Optionally, the customized data providing module 508 may specifically include: a data call request obtaining unit, configured to obtain a user's call request for data; and a target database obtaining unit, configured according to the customized configuration information of the user, Determining the target database storing the data information corresponding to the data called by the call request; a data calling interface determining unit, configured to determine the calling interface corresponding to the data called by the calling request according to the data information in the target database; The data providing unit is used to access the calling interface and provide the called data to the user.

Optionally, the customized configuration information includes a database identifier, and the database identifier is used to specify a target database for obtaining data information corresponding to the data called by the user. The target database acquiring unit may be specifically used to determine the database corresponding to the database identifier as the target database.

Optionally, the target database obtaining unit may be specifically configured to: determine the type of data information corresponding to the data called by the call request according to the customized configuration information of the user; determine that the data of the type of data information is stored All databases of information; all databases storing data information of the data information type are determined as target databases.

Optionally, the device may further include: a data acquisition module for acquiring data information corresponding to the data provided by the data source provider; a data preprocessing module for evaluating the data information according to the customized information of the user Perform preprocessing; a preprocessed data storage module is used to store the preprocessed data information in the database of the data platform.

Optionally, the customized configuration information includes fields designated by the user. The customized data providing module 508 may specifically include: a data obtaining unit for obtaining data provided by a data source provider; a designated field selecting unit for selecting a user-specified field from the data; a field providing unit , Used to provide the content corresponding to the field to the user.

Optionally, the customized configuration information further includes data format information of the data required by the user. The customized data providing module 508 may specifically include: a data format conversion unit for converting the data format of the data provided by the data source provider into the data format corresponding to the data format information to obtain the converted data ; A data providing unit for providing the converted data to the user.

Based on the same idea, the embodiment of this specification also provides a device corresponding to the above method. FIG. 6 is a schematic structural diagram of a data processing device corresponding to FIG. 2 provided by an embodiment of the specification. As shown in FIG. 6, the device 600 may include: at least one processor 610; and a memory 630 communicatively connected with the at least one processor. The memory 630 stores an instruction 620 that can be executed by the at least one processor 610, and the instruction is executed by the at least one processor 610.

The instructions may enable the at least one processor 610 to: obtain data configuration information of each data source provider; the data configuration information includes basic information and interface information; according to the configuration information, each data source Configure the provided data information, save the configured data information in the database of the data platform; obtain the call request of the user of the data platform for the data provided by the data source provider; based on the user’s customization The data provided by the data source provider is provided to the user through the customized configuration information and the configured data information.

Based on the same idea, the embodiments of this specification also provide a computer-readable medium corresponding to the above method. The computer-readable medium stores computer-readable instructions, and the computer-readable instructions can be executed by the processor to implement the following method: obtaining data configuration information of each data source provider; the data configuration information includes basic information and Interface information; configure the data information provided by each of the data sources according to the configuration information, and save the configured data information in the database of the data platform; obtain the data provided by the user of the data platform Request for data provided by the party; based on the customized configuration information of the user and the configured data information, the data provided by the data source provider is provided to the user.

The foregoing describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than in the embodiments and still achieve desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown in order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

In the 1990s, the improvement of a technology can be clearly distinguished between hardware improvements (for example, improvements in circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements in method flow). However, with the development of technology, the improvement of many methods and processes of today can be regarded as a direct improvement of the hardware circuit structure. Designers almost always get the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be realized by the hardware entity module. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user's programming of the device. It is programmed by the designer to "integrate" a digital system on a piece of PLD, without requiring chip manufacturers to design and manufacture dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly realized with "logic compiler" software, which is similar to the software compiler used in program development and writing, but before compilation The original code must also be written in a specific programming language, which is called Hardware Description Language (HDL), and there is not only one type of HDL, but many types, such as ABEL (Advanced Boolean Expression Language) , AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description), etc., currently most commonly used It is VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It should also be clear to those skilled in the art that only a little bit of logic programming of the method flow in the above-mentioned hardware description languages and programming into an integrated circuit, the hardware circuit that implements the logic method flow can be easily obtained.

The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program codes (such as software or firmware) executable by the (micro)processor. , Logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers and embedded microcontrollers. Examples of controllers include but are not limited to the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicon Labs C8051F320, the memory controller can also be implemented as part of the memory control logic. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, it is completely possible to program the method steps to make the controller use logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded logic. The same function can be realized in the form of a microcontroller or the like. Therefore, such a controller can be regarded as a hardware component, and the devices included in it for realizing various functions can also be regarded as a structure within the hardware component. Or even, a device for realizing various functions can be regarded as both a software module for realizing the method and a structure within a hardware component.

The systems, devices, modules, or units illustrated in the above embodiments may be specifically implemented by computer chips or entities, or implemented by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cell phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or Any combination of these devices.

For the convenience of description, when describing the above device, the functions are divided into various units and described separately. Of course, when implementing one or more embodiments of this specification, the functions of each unit may be implemented in the same one or more software and/or hardware.

Those skilled in the art should understand that one or more embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of this specification may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may adopt computer programs implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The form of the product.

One or more embodiments of this specification are described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to one or more embodiments of this specification. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are generated It is a device that realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device. The device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment. The instructions provide steps for implementing the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

In a typical configuration, the computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include non-permanent memory in computer readable media, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer readable media.

Computer-readable media includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, Magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

It should also be noted that the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or equipment including a series of elements not only includes those elements, but also includes Other elements that are not explicitly listed, or also include elements inherent to such processes, methods, commodities, or equipment. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, commodity, or equipment that includes the element.

One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification can also be practiced in distributed computing environments. In these distributed computing environments, tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, as for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment.

The above descriptions are only the embodiments of this specification, and are not used to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification should be included in the scope of the claims of one or more embodiments of this specification.

Claims

A data processing method, which is applied to a data platform, includes:

Acquiring data configuration information of each data source provider; the data configuration information includes basic information and interface information;

Configure the data information provided by each of the data sources according to the configuration information, and save the configured data information in the database of the data platform;

Obtaining a call request of the user of the data platform for the data provided by the data source provider;

Based on the customized configuration information of the user and the configured data information, the data provided by the data source provider is provided to the user.
The method according to claim 1, before said obtaining the data configuration information of each data source provider, the method further comprises:

Obtain the configuration information for the data provided by the data source provider;

For any piece of configuration information, test the configuration information to obtain a test result;

When the test result indicates that the configuration information is configured correctly, the configuration information is saved on the data platform.
The method according to claim 1, wherein the basic information includes at least provider information of the data provided by the data source;

The configuring the data information provided by each of the data sources according to the configuration information, and saving the configured data information in the database of the data platform, specifically includes:

According to the configuration information, determine the provider information of the data information provided by the data source;

Determine the database type corresponding to the provider information;

The data information provided by the data source is stored in a database corresponding to the provider information.
The method according to claim 1, wherein the interface information includes at least interface address information and transmission protocol information;

The method further includes:

Determining the interface address of the data source provider according to the interface address information;

Access the interface address according to the transmission protocol corresponding to the transmission protocol information to obtain the data provided by the data source.
The method according to claim 3, before storing the data information provided by the data source in the database corresponding to the database type determined corresponding to the data source provider information, the method further comprises:

Acquiring data format information of the database;

Format conversion of the data information provided by the data source according to the data format information to obtain data information after the format conversion;

The storing the data information provided by the data source in the database corresponding to the determined database type corresponding to the data source provider information specifically includes:

The data information after the format conversion is stored in a database corresponding to the database type.
The method according to claim 1, wherein the providing the data in the database to the user based on the customized configuration information of the user specifically includes:

Get the user's call request for data;

According to the customized configuration information of the user, determine the target database storing the data information corresponding to the data called by the calling request;

Determining the calling interface corresponding to the data called by the calling request according to the data information in the target database;

Access the calling interface and provide the called data to the user.
7. The method of claim 6, wherein the customized configuration information includes a database identifier, and the database identifier is used to specify a target database for obtaining data information corresponding to the data called by the user;

The determining the target database storing the data information corresponding to the data called by the calling request according to the customized configuration information of the user specifically includes:

The database corresponding to the database identifier is determined as the target database.
7. The method according to claim 6, wherein the determining a target database storing data information corresponding to the data called by the calling request according to the customized configuration information of the user specifically includes:

Determining the type of data information corresponding to the data called by the calling request according to the customized configuration information of the user;

Determine all databases storing data information of the data information type;

All the databases storing the data information of the data information type are determined as the target database.
8. The method according to claim 1, before said obtaining a call request of a user of the data platform for the data provided by the data source provider, the method further comprises:

Obtain data information corresponding to the data provided by the data source provider;

Preprocessing the data information according to the customized information of the user;

The preprocessed data information is stored in the database of the data platform.
5. The method of claim 1, wherein the customized configuration information includes a field specified by the user;

The providing the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information specifically includes:

Obtain the data provided by the data source provider;

Select fields specified by the user from the data;

The content corresponding to the field is provided to the user.
The method of claim 1, wherein the customized configuration information further includes data format information of the data required by the user;

The providing the data provided by the data source provider to the user based on the customized configuration information of the user specifically includes:

Converting the data format of the data provided by the data source provider into the data format corresponding to the data format information to obtain the converted data;

Provide the converted data to the user.
A data processing device, which is applied to a data platform, includes:

The configuration information obtaining module is used to obtain the configuration information of the data of each data source provider; the configuration information of the data includes basic information and interface information;

The configuration module is configured to configure the data information provided by each of the data sources according to the configuration information, and save the configured data information in the database of the data platform;

A data call request obtaining module, which is used to obtain a call request of a user of the data platform for the data provided by the data source provider;

The customized data providing module is configured to provide the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information.
11. The device of claim 12, wherein the basic information includes at least provider information of the data provided by the data source;

The configuration module specifically includes:

A data source provider information determining unit, configured to determine the provider information of the data information provided by the data source according to the configuration information;

The database type determining unit is used to determine the database type corresponding to the provider information;

The data storage unit is configured to store the data information provided by the data source in the database corresponding to the provider information.
The device according to claim 12, wherein the interface information includes at least interface address information and transmission protocol information;

The device further includes:

The interface address obtaining module is configured to determine the interface address of the data source provider according to the interface address information;

The data acquisition module is configured to access the interface address according to the transmission protocol corresponding to the transmission protocol information to obtain the data provided by the data source.
The device according to claim 12, the customized data providing module specifically includes:

The data call request obtaining unit is used to obtain the user's call request for data;

The target database acquiring unit is configured to determine the target database storing the data information corresponding to the data called by the calling request according to the customized configuration information of the user;

The data calling interface determining unit is configured to determine the calling interface corresponding to the data called by the calling request according to the data information in the target database;

The data providing unit is used to access the calling interface and provide the called data to the user.
15. The device according to claim 15, wherein the customized configuration information includes a database identifier, and the database identifier is used to specify a target database for obtaining data information corresponding to the data called by the user;

The target database obtaining unit is specifically used for:

The database corresponding to the database identifier is determined as the target database.
The device according to claim 15, wherein the target database obtaining unit is specifically configured to:

Determining the type of data information corresponding to the data called by the calling request according to the customized configuration information of the user;

Determine all databases storing data information of the data information type;

All the databases storing the data information of the data information type are determined as the target database.
The device according to claim 12, further comprising:

The data acquisition module is used to acquire data information corresponding to the data provided by the data source provider;

The data preprocessing module is used to preprocess the data information according to the customized information of the user;

The preprocessed data saving module is used to save the preprocessed data information in the database of the data platform.
11. The device of claim 12, wherein the customized configuration information includes a field specified by the user;

The customized data providing module specifically includes:

The data acquisition unit is used to acquire the data provided by the data source provider;

A designated field selection unit for selecting a field designated by the user from the data;

The field providing unit is used to provide the content corresponding to the field to the user.
A data processing device, including:

At least one processor; and,

A memory communicatively connected with the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can:

Acquiring data configuration information of each data source provider; the data configuration information includes basic information and interface information;

Configure the data information provided by each of the data sources according to the configuration information, and save the configured data information in the database of the data platform;

Obtaining a call request of the user of the data platform for the data provided by the data source provider;

Based on the customized configuration information of the user and the configured data information, the data provided by the data source provider is provided to the user.
A computer-readable medium having computer-readable instructions stored thereon, and the computer-readable instructions can be executed by a processor to implement the data processing method according to any one of claims 1 to 11.