CN117951139A

CN117951139A - Data processing method, device and equipment

Info

Publication number: CN117951139A
Application number: CN202410202708.9A
Authority: CN
Inventors: 孙鹏
Original assignee: Alipay Hangzhou Information Technology Co Ltd
Current assignee: Alipay Hangzhou Information Technology Co Ltd
Priority date: 2020-04-08
Filing date: 2020-04-08
Publication date: 2024-04-30
Also published as: WO2021204041A1; CN111414369B; CN111414369A

Abstract

The embodiment of the specification provides a data processing method, a device and equipment. The scheme comprises the following steps: acquiring configuration information of data of each data source provider; configuring the data information provided by each data source according to the configuration information, and storing the configured data information into a database of a data platform; acquiring a call request of a user of a data platform for data; and according to the call request, providing the data provided by the data source provider for the user based on the customized configuration information of the user and the data information configured by the data platform.

Description

Data processing method, device and equipment

The invention is a divisional application of an invention application of which the application date is 04 month 08 in 2020, the application number is 2020102776436 and the invention name is 'a data processing method, a device and equipment'.

Technical Field

One or more embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a data processing method, apparatus, and device.

Background

At present, with the development of internet information technology, data are already applied to each industry and business intelligence field, and each user organization can generate massive data every day, so people need to mine and use the massive data. For example, in the financial field, each institution generates a large amount of transaction data every day, generally, in order to ensure the normal operation of a plurality of institutions, each institution needs to acquire required data through a data platform to realize intelligent wind control, user image, intelligent marketing and the like, so that a transaction user can be accurately evaluated, and services can be better provided for the user. Such as: the credit agency acquires data corresponding to the user from the corresponding data provider through the data platform, so that the service of the user is evaluated, and corresponding service is provided for the user according to the evaluation result.

The data platform is a platform integrating application interfaces such as data access, data processing, data storage, query retrieval, analysis mining and the like, and each user can call the required data through the data platform according to own requirements.

However, when a user invokes data through a data platform, in order to meet the service processing requirements of the user, the data needs to be acquired from data sources of multiple institutions. At this time, according to the data call request of the user, the required data needs to be called through the query engine corresponding to the data source of each mechanism, which results in large workload and complicated call. At this time, in a scenario where large-scale and multi-tenant space needs to use open data of multiple data sources to perform big data calculation, how to configure and customize the open data of multiple data sources is a technical problem to be solved.

Disclosure of Invention

In view of this, one or more embodiments of the present disclosure provide a data processing method, apparatus, and device, which are used to implement unified configuration of data of multiple data sources while satisfying customized use of data by a user.

In order to solve the above technical problems, the embodiments of the present specification are implemented as follows:

the embodiment of the specification provides a data processing method, which is applied to a data platform and comprises the following steps:

acquiring configuration information of data of each data source provider; the configuration information of the data comprises basic information and interface information;

configuring the data information provided by each data source according to the configuration information, and storing the configured data information into a database of the data platform;

Acquiring a call request of a user of the data platform for data provided by the data source provider;

And providing the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information.

The embodiment of the present disclosure provides a data processing apparatus, where the apparatus is applied to a data platform, including:

The configuration information acquisition module is used for acquiring configuration information of data of each data source provider; the configuration information of the data comprises basic information and interface information;

the configuration module is used for configuring the data information provided by each data source according to the configuration information and storing the configured data information into a database of the data platform;

The data calling request acquisition module is used for acquiring a calling request of a user of the data platform for data provided by the data source provider;

and the customized data providing module is used for providing the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information.

The embodiment of the present specification provides a data processing apparatus, including:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to:

Embodiments of the present disclosure provide a computer readable medium having computer readable instructions stored thereon, the computer readable instructions being executable by a processor to implement a data processing method.

One embodiment of the present disclosure can achieve the following beneficial effects: the configuration information of the data provided by each data source provider is obtained, the data information provided by each data source is configured according to the configuration information, and the configured data information is stored in a database of a data platform, so that the data information provided by a plurality of data sources is uniformly configured on the data platform, and the access use efficiency of the data sources is further improved; the method comprises the steps of obtaining a call request of a user of a data platform for data of a data source provider, providing the data provided by the data source provider for the user based on customized configuration information of the user, and customizing the data according to the requirement of the user so as to meet personalized customization requirement of the user for using the data.

Drawings

The accompanying drawings, which are included to provide a further understanding of one or more embodiments of the specification, illustrate and explain one or more embodiments of the specification, and are not an undue limitation on the one or more embodiments of the specification. In the drawings:

Fig. 1 is an overall schematic diagram of a data processing method according to an embodiment of the present disclosure;

FIG. 2 is a schematic flow chart of a data processing method according to an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a configuration interface of configuration information in a data processing method according to an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a configuration interface of user-customized configuration information in a data processing method according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a data processing apparatus corresponding to FIG. 2 according to an embodiment of the present disclosure;

Fig. 6 is a schematic structural diagram of a data processing apparatus corresponding to fig. 2 according to an embodiment of the present disclosure.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of one or more embodiments of the present specification more clear, the technical solutions of one or more embodiments of the present specification will be clearly and completely described below in connection with specific embodiments of the present specification and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present specification. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without undue burden, are intended to be within the scope of one or more embodiments herein.

The following describes in detail the technical solutions provided by the embodiments of the present specification with reference to the accompanying drawings.

Fig. 1 is an overall schematic diagram of a data processing method according to an embodiment of the present disclosure. As shown in fig. 1, includes a data source provider 101, a data platform 103, and a user 105. It should be noted that, here, the user 105 may represent a device used by the user, for example: corporate devices, banking devices or personal devices, etc. The data source provider can provide data corresponding to a plurality of data sources, for example, fig. 1 includes n data sources, data source 1 provides data a, data source 2 provides data B, data source 3 provides data C, … …, data source n provides data m, the data platform 103 does not store data, only stores data information corresponding to data, and needs to uniformly configure the data information corresponding to the data provided by the n data sources to the data platform 103, which can be understood that the data platform 103 needs to configure the data information corresponding to the data provided by the n data sources and store the data information in a database of the data platform 103. In the storing process, the data information corresponding to the data provided by one data source may be stored in one database, as shown in fig. 1, the data information a of the data a provided by the data source 1 may be stored in the database 1 of the data platform 103, the data information B of the data B provided by the data source 2 may be stored in the database 2 of the data platform 103, … …, and the data information M of the data M provided by the data source n may be stored in the database n of the data platform 103. The device used by the user 105 may send a data call request to the data platform, and perform a customized call on the data provided by the data source provider, where the customized call of the user 105 requests to perform a data query to an interface provided by the provider according to the configuration information and the customized configuration information of the data platform 103, so as to meet the personalized requirement of the user for using the data.

Next, a data processing method provided for the embodiments of the specification will be specifically described with reference to the accompanying drawings:

fig. 2 is a flow chart of a data processing method according to an embodiment of the present disclosure. From the program perspective, the execution subject of the flow may be a program or an application client that is installed on an application server.

As shown in fig. 2, the process may include the steps of:

Step 202: acquiring configuration information of data of each data source provider; the configuration information of the data may include basic information and interface information.

A data source provider may refer to a database or database server used by a database application. Such as: the data provider may provide data to the data platform through a database or database server used by its own open application.

The configuration information may represent a variable that enables a user to edit in a configuration table or a configuration interface, and after the data platform obtains data provided by a plurality of data sources from the data source provider, the data may be configured according to the configuration information of the data of each data source provider.

Step 204: and configuring the data information provided by each data source according to the configuration information, and storing the configured data information into a database of the data platform.

The data information provided by the data source provider may include an access mode, a query mode, a use mode, an identity of the data source, and the like. Such as: the data X provided by the data source a is transaction data, where the transaction data may include a transaction scenario, access address information for obtaining the transaction data, query mode information, usage format information, and a transaction type, a transaction name, a transaction time, a specific transaction content, etc. corresponding to the transaction data X.

The data platform can be a platform integrating application interfaces such as data access, data processing, data storage, query and retrieval, analysis and mining, and the like, and can store data information provided by each data source to realize data resource sharing for each mechanism. The data platform can provide the functions of displaying and applying for use of various open data, and after the data platform applies for the use of the data, the data can be accessed and inquired through application development of the data. The data platform may include a plurality of databases for storing data information provided by the respective data source providers. When the data information is specifically stored, the data information can be stored in a classified mode, and one type of data information is stored in the same type of database; the data information provided by one data source is stored in the same type of database; of course, in practical applications, the data may be stored in one database or multiple databases according to the size of the data volume. The specific storage may be set according to the actual application.

Step 206: and acquiring a call request of a user of the data platform for data provided by the data source provider.

It should be noted that, in this specification, reference to "a user" may refer to a specific device used by the user.

In practical application, the data platform can sign up with part of institutions, directionally cooperate, and provide required data for the signing institutions according to the requirements of the signing institutions. In addition, the data platform can also face all institutions, and each institution can search query data, apply for using data and the like through the data platform.

When a user of the data platform needs to use the data provided by the data source provider corresponding to the data information on the data platform, a data calling request can be sent to the data platform, the data platform is requested to call the data from the data calling interface of the corresponding data source provider, and the data obtained by calling is provided.

Step 208: and providing the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information.

The customized configuration information may represent user requirements for the data and may also include call parameters specifying the portion of the interface.

In the method in fig. 2, configuration information of data provided by each data source provider is obtained, the data information provided by each data source is configured according to the configuration information, and the configured data information is stored in a database of a data platform, so that unified configuration of the data information provided by a plurality of data sources on the data platform is realized, and further, the efficiency of access use of the data sources is improved; the method comprises the steps of obtaining a call request of a user of a data platform for data of a data source provider, providing the data provided by the data source provider for the user based on customized configuration information of the user, and customizing the data according to the requirement of the user so as to meet personalized customization requirement of the user for using the data.

The examples of the present specification also provide some specific embodiments of the method based on the method of fig. 2, which is described below.

In practical use requirements, when the data platform uniformly accesses data information provided by a plurality of data sources into the data platform, related business operations can comprise data source configuration online and data source access detection. After a plurality of data sources are configured, test verification is needed, and the data sources can be stored on a data platform for each user to use after verification is passed.

Specifically, before the obtaining the data configuration information of each data source provider, the method may further include:

acquiring configuration information for data provided by a data source provider;

Testing the configuration information aiming at any one piece of configuration information to obtain a test result;

And when the test result shows that the configuration information is configured correctly, storing the configuration information on a data platform, and using the configuration information on line so as to configure the data information provided by the data source provider according to the configuration information. The configuration information provided by the data source provider is tested, and the device provided by each user can be online for use after the test passes.

When in specific test, the test can be carried out according to the set configuration information format, the corresponding interface address, parameters and the like in the configuration information are filled in the test interface, the test button is clicked to carry out the test, the returned test result is checked, if the test fails, the configuration information is modified, and the test parameters are reset. If the test is correct, an online request may be made.

When the configuration information is tested and adjusted, the test can be performed according to dynamic scenarios. Assuming test failure, parameters may be adjusted for test results, such as: the signature computation logic of the query interface is adjusted.

By the method, before the data information provided by the data source provider is configured according to the configuration information, the configuration information can be tested, the configuration information passing the test is stored on the data platform, and after the data information provided by the data source provider is obtained, the data information can be configured directly according to the configuration information stored on the data platform, so that the configuration efficiency is improved.

The unified configuration of data provided by multiple data source providers is that data information provided by multiple data source providers is actually stored in databases of the same data platform according to corresponding configuration information. Specifically, when data information provided by a plurality of data source providers is stored on the same data platform, it is required to determine which databases the data information of each data source should be stored in, and the specific implementation method may be as follows:

The basic information at least comprises provider information of data provided by the data source; the configuring the data information provided by each data source according to the configuration information, and storing the configured data information into a database of the data platform, specifically including:

Determining provider information of data information provided by the data source according to the configuration information;

determining a database type corresponding to the provider information;

and storing the data information provided by the data source into a database corresponding to the provider information.

It should be noted that, when data information provided by a data source provider is stored in a data platform, it may be determined, according to provider information, which database the data information provided by the data source provider should be stored in. Such as: the data source provider a provides the data X, and at this time, it may determine that the data information corresponding to the data X should be stored in the R-type database according to the name or identifier of the data source provider a, the classification of the data X, the application scenario, the coverage area, and the open mode of the data provided by the data source provider a. One database type can correspond to one database or a plurality of databases, and the number of the databases corresponding to the database type can be set according to actual conditions.

By the method, the type of the storage database of the data information of each data source on the data platform is determined according to the provider information of the data provided by the data source, so that the configuration efficiency of the data information can be improved.

In practical application, the data source provider may actively send the data information to the data platform for configuration storage, or the data platform may send a request for obtaining the data information to each data source provider, so as to request each data source provider to place the data information on the data platform for configuration storage. When a user needs to call data through a data platform, the user can send a data call request to the data platform, the data platform provides a call interface for the user to call the required data from the call interface, and interface information in the configuration information at least can comprise interface address information and transmission protocol information; specifically, when data is acquired, the method may include:

determining an interface address of a data source provider according to the interface address information;

And accessing the interface address according to a transmission protocol corresponding to the transmission protocol information to acquire the data provided by the data source.

In practical applications, the configuration information may be filled in through the interface provided in fig. 3. Specifically, the following details can be described with reference to fig. 3:

Fig. 3 is a schematic diagram of a configuration information configuration interface in a data processing method according to an embodiment of the present disclosure. As shown in fig. 3, the configuration information may include basic information as well as interface information. The base information may include data source provider information such as: the name or identification of the data source provider, the classification of the data, the application scenario, the coverage (e.g., national, international, etc.), and the open mode (e.g., dedicated to a specified organization or for use by all organizations) of the data provided by the data source provider.

The interface information may include interface address information, transport protocol information, parameter information, and the like. The interface address information may be filled with an interface address specifying the source of the request data, and the transmission protocol information may be filled with a transmission protocol used when the data platform performs data transmission with an interface specifying the source of the request data, for example: transmission control protocol (Transmission Control Protocol, TCP for short), user datagram protocol (User Datagram Protocol, UDP for short), internet protocol (Internet Protocol, IP for short), etc.

The parameter information may include a parameter type, a parameter name, parameter description information, a setting manner of the parameter, and the like. It should be noted that, for some parameters, other interfaces of the data provider need to be called first to obtain parameter values, and then data source information of the interface for obtaining the parameter values needs to be configured first, and then the data source information of the obtained data needs to be configured.

In the above method, the interface address of the data provided by the data source provider can be determined from the interface address information, and the data provided by the data source provider can be obtained by accessing the interface address. When data is transmitted between different databases, corresponding transmission protocols are adopted, so that the transmission protocols of the data source provider and the data platform can be acquired from the interface information, and the interface address of the data source provider is accessed according to the transmission protocols so as to acquire the data provided by the data source provider.

When the data information provided by the data source provider is stored in a database corresponding to the type of the database determined by the data source provider information, if the database has no required data format, the data information can be directly stored in the database according to the data format of the database without data format conversion; however, if the database is required to be stored in a data format, the data format of the data information provided by the data source provider needs to be converted into the data format required by the database to be stored. The method can be realized by the following steps:

Before the data information provided by the data source is stored in the database corresponding to the database type determined by the data source provider information, the method may further include:

Acquiring data format information of the database;

Carrying out format conversion on the data information provided by the data source according to the data format information to obtain data information after format conversion;

The step of storing the data information provided by the data source into a database corresponding to the database type determined by the data source provider information, specifically includes:

and storing the data information after format conversion into a database corresponding to the database type.

It should be noted that, the data format may be a format of data stored in a file or a record, and generally when the interface is used to transmit data, the data format may include: lightweight text data interchange format (JavaScript Object Notation, JSON for short), extensible markup language (eXtensible Markup Language, XML for short), data serialization format (Yet Another Markup Language, YAML for short).

Such as: the data format of the data information X provided by the data source provider A is an XML format, the data format of the data information Y provided by the data source provider B is a YAML format, the data format of the data information Z provided by the data source provider C is a JSON) format, the data information X needs to be stored in the I-type database through analysis, the data information Y needs to be stored in the H-type database, the data information Z needs to be stored in the J-type database, the storage data formats corresponding to the known I-type database and the H-type database are all XML formats, and the storage data format corresponding to the J-type database is a YAML format, and at the moment, the data information X provided by the data source provider A can be directly stored in the I-type database; converting the data format of the data information Y provided by the data source provider B from YAML format to XML format, and storing the data information Y in the XML format into an H-type database; the data format of the data information Z provided by the data source provider C is converted from JSON format to YAML format, and then the Z data information in YAML format is stored in the J-class database.

In an actual application scenario, after the data information provided by each data source provider is configured on the same data platform, the data platform may provide a data query interface for each user, so as to provide the user with required data, for example: when a bank a processes a large-amount loan transaction, it needs to evaluate the credit of a borrower, the property condition, repayment capability and the like of the borrower, and at this time, the bank a needs to perform user portrait according to basic portrait data about the borrower provided by other institutions, for example: it is necessary to use company registration data of the lender provided by the business bureau, real estate data of the lender obtained from a real estate management department, deposit data and lending data of the lender at other banks, and the like. After the data information of the data is configured uniformly by the data platform, the bank a may acquire a data calling interface for calling the basic portrait data of the lender through the data platform, and customize the data obtained by calling according to the own use requirement, and specifically, the providing the data in the database to the user based on the customized configuration information of the user may specifically include:

Acquiring a call request of a user for data;

determining a target database storing data information corresponding to the data called by the calling request according to the customized configuration information of the user;

determining a calling interface corresponding to the data called by the calling request according to the data information in the target database;

and accessing the calling interface and providing the called data to the user.

In practical applications, the customized configuration information may refer to customized usage requirements of each user for data. The customized configuration information for the user may be filled in through the interface provided in fig. 4.

Specifically, the following details can be described with reference to fig. 4:

Fig. 4 is a schematic diagram of a configuration interface of user-customized configuration information in a data processing method according to an embodiment of the present disclosure. As shown in fig. 4, the customized configuration information of the user may include an identification parameter of the calling interface and a signature mode; and may also include specified fields, data formats for use with data, and combined or routed use of multiple data sources, etc. Such as: only a few fields in the data need be acquired, or data in XML format need be used, or data need be acquired from a specified database, etc.

It should be noted that, the designated part interface may represent a target database, and the identity recognition parameter of the calling interface may represent the identification information of the target database; the signature manner may represent a signature manner of data encryption in the target database.

For example: the user needs to use the data X, and at this time, the customized configuration information of the user using the data X may at least include a data format of the data X, a use field of the data X, and a designated call interface of the data X, and according to the designated call interface, a designated target database storing the data X may be determined. After the data are taken from the corresponding database according to the configuration information of the user, the data are converted into data which can be used by the user, so that the personalized use requirement of the user is met.

The data platform may obtain a call request for data from a user, such as: a request for a bank a to call the basic image data of the lender 1 is acquired.

After the call request of the user for the data is obtained, a target database storing the data required to be called by the user can be determined according to the customization information of the user. In the application scenario, the user may specify a database for taking data, or may take data from all databases storing required data, which may be specifically described according to the following two methods:

The method I obtains the data information corresponding to the required data from the appointed database.

More specifically, the customized configuration information includes a database identifier, where the database identifier is used to designate a target database for acquiring data information corresponding to the data called by the user.

The determining, according to the customized configuration information of the user, a target database storing data information corresponding to the data invoked by the invocation request may specifically include:

and determining the database corresponding to the database identifier as a target database.

Database identifications may be included in the customized configuration information, such as: the configuration information of the user A designates a database Q and a database R, and when the data platform acquires a call request of the user A for the data X, the data X is directly taken out of the database Q and the database R and sent to the user A.

According to the method, according to the customized requirements of the user, the effective data can be provided for the user more pertinently, so that the use efficiency of the user on the data platform is improved.

And secondly, taking the data information from all databases containing the data information corresponding to the required data.

Determining a data information type corresponding to the data called by the call request according to the customized configuration information of the user;

determining all databases storing data information of the data information type;

All databases storing the data information of the data information type are determined as target databases.

It should be noted that, the customized configuration information of the user may include a data information type corresponding to the data to be invoked, so after the user's request for invoking the data is obtained, the data information type corresponding to the data to be invoked by the user may be determined. The data information type may be classified according to the service scenario of the data, for example: asset data, identity data, credit data, and the like.

In the second method, the step is mainly to attach the service requirement, and in practical application, the user may need to use the data provided by multiple data sources in combination to better utilize the data to perform subsequent service operations. Such as: a banking institution needs to complete the user portrait of a registered user, needs some online and offline data about the registered user, and needs to comprehensively determine the user portrait of the registered user by combining data provided by online data sources and offline data sources.

The data of each data source is correspondingly stored in the corresponding database, so that the data required by the user can be obtained from a plurality of databases to meet the complex scene of using the data by the user.

From the data type, all databases storing data of that data type may be determined. From these databases, this type of data can be provided to the user. Such as: the data platform receives a call request of the company S for the identity data of the staff A, and the data information type of the identity data of the staff A can be determined to be basic portrait data according to the configuration information of the company S for the identity data of the staff A. All databases of the basic portrait data types stored in the user A are determined, namely a database 1, a database 2 and a database 3, and at this time, the data platform can call data from the data source provider corresponding to each database according to the data information in the database 1, the database 2 and the database 3 and provide the data to the company S so that the company S can comprehensively evaluate the staff A.

By the method, the required data are provided for the user as comprehensively as possible according to the customized configuration information of the user, so that the use requirement of the user on the data can be better met, and the use experience of the user on the data platform is improved.

In practical application, the data platform may have subscribed directional cooperative users, and these directional cooperative users and the data platform may sign a protocol for subscribing to the data source, and after signing the protocol, the user may obtain key parameter values such as identity identification using the corresponding data source. After signing, the oriented collaboration user can customize and use the oriented collaboration user for relevant data sources, and when data query is performed, a query engine provided on the data platform can also execute data query operation according to the customized configuration information of the user.

For these targeted collaboration users, after the data platform obtains the data information from the data source provider, the data information may be configured in advance according to the configuration information of the targeted collaboration users and stored on the data platform. Specifically, the method may include the steps of:

before the call request of the user of the data platform for the data provided by the data source provider is obtained, the method may further include:

acquiring data information corresponding to data provided by a data source provider;

preprocessing the data information according to the customized information of the user;

and storing the preprocessed data information in a database of the data platform.

It should be noted that, preprocessing herein may mean that data information provided by a data source provider is configured in advance according to configuration information of a user. The configured data information may be designated for storage in a corresponding database of the data platform. Of course, the stored database may not be specified, and the specific implementation manner may be set according to actual requirements.

In the practical application process, according to the subscription protocol, the data provided by the corresponding data source provider can be sent to the directional cooperative user at regular time, or after receiving the call request of the directional cooperative user for the data, a data call interface is provided for the user for the directional cooperative user to call the data.

The customized configuration information comprises a field appointed by the user;

the providing the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information may specifically include:

acquiring data provided by a data source provider;

Selecting a field designated by a user from the data;

And providing the content corresponding to the field for the user.

In order to meet the customization requirement of the user of the data platform for data, accurate data can be provided for the user according to the user requirement, specifically, it is assumed that the user A requires to call the field A1 and the field A2, the two fields belong to the data X, and at the moment, the field A1 and the field A2 can be found out from the data X and sent to the user A.

The customization requirement of the user can specify the data format information of the required data besides the field;

The customized configuration information also comprises data format information of data required by the user;

The providing the data provided by the data source provider to the user based on the customized configuration information of the user may specifically include:

Converting the data format of the data provided by the data source provider into a data format corresponding to the data format information to obtain converted data;

and providing the converted data to the user.

In practical applications, the data format of the data provided by the data source provider may not be consistent with the data format of the data required to be used by the user, and at this time, the data format of the data provided by the data source provider needs to be converted, and the data consistent with the data format of the data required to be used by the user is sent to the user.

Such as: the user needs to call data X and data Y, the data format of the data X is an XML format, and the data format of the data Y is a JSON format; and the data format required by the user to use the data is an XML format, at this time, the format of the data Y is required to be converted from the JSON format to the XML format, and then the Y data in the XML format is provided for the user.

Of course, after the user obtains the corresponding data from the data platform and uses the obtained data, the data use log can be counted, and after analysis processing, the data use condition, the data quality and the like are displayed on the equipment used by the user.

The data platform can provide a data information service application programming interface (Application Programming Interface, abbreviated as API) for users to use, and can also provide an open portal for external users to review data sources and sign up to use the data sources.

After the data platform uniformly configures the data of each data source, when the user wants to call the required data from the data platform, an interface for calling or querying the data can be provided for the user, the interface can be called a data view, and customized configuration information of the user can be filled in the interface, for example: information such as the identity of the user, the signature mode, the use format of the data requested to be invoked and the like can be filled in. In addition, the interface can be filled with a plurality of data source information according to the application scene, so that the user can call the data of a plurality of data sources for use.

According to the data processing method provided by the embodiment of the specification, the data of the data sources are uniformly configured and controlled, customized use data are provided for the user according to the personalized requirements of the user, the efficiency of accessing and using the data sources is greatly improved, and the use requirements of the user can be met. In addition, when the data platform performs unified configuration on the data, in order to ensure the online quality of configuration, configuration information passes through the processes of test verification, flow application online and gray scale release, so that the production risk is reduced. In the face of more complex service scenes, the scheme also supports multi-data source combined query and screening, and then displays the data query result, so that the problem of large-scale access efficiency of data and the problem of how to guarantee customized use according to service requirements after access in the scene of large-scale and multi-tenant space using open data of multiple data sources for large-data calculation can be solved.

More specifically, the method provided in the embodiment of the present specification may specifically achieve the following technical effects:

1) The multi-data source is accessed in a unified way, customization is performed during use, personalized customization requirements are supported while the data source access efficiency is improved, and meanwhile, the requirements of localization and safety use of data are met.

2) Configuration and flow management: the data of multiple data sources are uniformly accessed into a data platform to perform configuration, testing, auditing and online flow, and when the user customizes the data, the processes of testing, verifying and the like are also required, so that the access cost is reduced, and the quality of the data can be ensured.

3) Configuration multi-version support and gray level release online: the configuration information is tested and verified, the requirement for adjusting the configuration information after test failure can be met, and meanwhile, the influence and loss possibly brought by release on line are controlled to be minimum through gradual gray scale release aiming at the modified configuration information.

4) Multiple data source routing and combined use: in order to meet business requirements, a plurality of data sources can be combined for application scenes, and the use requirements of users on data can be better met.

Based on the same thought, the embodiment of the specification also provides a device corresponding to the method. Fig. 5 is a schematic structural diagram of a data processing apparatus corresponding to fig. 2 according to an embodiment of the present disclosure. As shown in fig. 5, the apparatus may be applied to a data platform, and may include:

A configuration information obtaining module 502, configured to obtain configuration information of data of each data source provider; the configuration information of the data comprises basic information and interface information;

the configuration module 504 is configured to configure data information provided by each data source according to the configuration information, and store the configured data information into a database of the data platform;

a data call request acquiring module 506, configured to acquire a call request of a user of the data platform for data provided by the data source provider;

And the customized data providing module 508 is configured to provide the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information.

Optionally, the basic information at least includes provider information of data provided by the data source;

The configuration module 504 may specifically include:

a data source provider information determining unit, configured to determine provider information of data information provided by the data source according to configuration information;

A database type determining unit, configured to determine a database type corresponding to the provider information;

And the data storage unit is used for storing the data information provided by the data source into a database corresponding to the provider information.

Optionally, the apparatus may further include:

the configuration information acquisition module is used for acquiring configuration information of data provided by a data source provider;

the configuration information testing module is used for testing any one configuration information to obtain a testing result;

and the configuration information storage module is used for storing the configuration information to a data platform when the test result indicates that the configuration information is configured correctly.

Optionally, the interface information at least includes interface address information and transmission protocol information;

The apparatus may further include:

The interface address acquisition module is used for determining the interface address of the data source provider according to the interface address information;

and the data acquisition module is used for accessing the interface address according to a transmission protocol corresponding to the transmission protocol information and acquiring the data provided by the data source.

Optionally, the customized data providing module 508 may specifically include:

The data call request acquisition unit is used for acquiring a call request of a user for data;

the target database acquisition unit is used for determining a target database storing data information corresponding to the data called by the calling request according to the customized configuration information of the user;

The data calling interface determining unit is used for determining a calling interface corresponding to the data called by the calling request according to the data information in the target database;

and the data providing unit is used for accessing the calling interface and providing the called data to the user.

Optionally, the customized configuration information includes a database identifier, where the database identifier is used to designate a target database for acquiring data information corresponding to the data called by the user;

The target database acquisition unit may specifically be configured to:

Optionally, the target database obtaining unit may be specifically configured to:

Optionally, the apparatus may further include:

the data acquisition module is used for acquiring data information corresponding to the data provided by the data source provider;

The data preprocessing module is used for preprocessing the data information according to the customized information of the user;

The preprocessing data storage module is used for storing the preprocessed data information in a database of the data platform.

Optionally, the customized configuration information includes a field specified by the user;

the customized data providing module 508 may specifically include:

A data acquisition unit for acquiring data provided by a data source provider;

A specified field selection unit for selecting a field specified by a user from the data;

And the field providing unit is used for providing the content corresponding to the field for the user.

Optionally, the customized configuration information further includes data format information of the data required by the user;

the customized data providing module 508 may specifically include:

A data format conversion unit, configured to convert a data format of data provided by the data source provider into a data format corresponding to the data format information, so as to obtain converted data;

And the data providing unit is used for providing the converted data to the user.

Based on the same thought, the embodiment of the specification also provides equipment corresponding to the method. Fig. 6 is a schematic structural diagram of a data processing apparatus corresponding to fig. 2 according to an embodiment of the present disclosure. As shown in fig. 6, the apparatus 600 may include:

at least one processor 610; and

A memory 630 communicatively coupled to the at least one processor; wherein,

The memory 630 stores instructions 620 executable by the at least one processor 610, the instructions being executable by the at least one processor 610.

The instructions may enable the at least one processor 610 to:

Based on the same thought, the embodiment of the specification also provides a computer readable medium corresponding to the method. Computer readable instructions stored on a computer readable medium, the computer readable instructions being executable by a processor to perform a method of:

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable GATE ARRAY, FPGA)) is an integrated circuit whose logic functions are determined by user programming of the device. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented with "logic compiler (logic compiler)" software, which is similar to the software compiler used in program development and writing, and the original code before being compiled is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but HDL is not just one, but a plurality of kinds, such as ABEL(Advanced Boolean Expression Language)、AHDL(Altera Hardware Description Language)、Confluence、CUPL(Cornell University Programming Language)、HDCal、JHDL(Java Hardware Description Language)、Lava、Lola、MyHDL、PALASM、RHDL(Ruby Hardware Description Language), and VHDL (Very-High-SPEED INTEGRATED Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application SPECIFIC INTEGRATED Circuits (ASICs), programmable logic controllers, and embedded microcontrollers, examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing one or more embodiments of the present description.

One skilled in the art will appreciate that one or more embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

One or more embodiments of the present specification are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to one or more embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

One or more embodiments of the present specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. One or more embodiments of the present description may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing description is illustrative of embodiments of the present disclosure and is not to be construed as limiting one or more embodiments of the present disclosure. Various modifications and alterations to one or more embodiments of this description will be apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of one or more embodiments of the present disclosure, are intended to be included within the scope of the claims of one or more embodiments of the present disclosure.

Claims

1. A data processing method, the method being applied to a data platform, comprising:

Configuring the data information provided by each data source according to the configuration information, and storing the configured data information into a database corresponding to the provider information of the data source provider in the data platform;

Providing the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information; the customized configuration information represents the user's need for use of the data.

2. The method of claim 1, further comprising, prior to the obtaining the data configuration information for each data source provider:

and when the test result shows that the configuration information is configured correctly, storing the configuration information on a data platform.

3. The method of claim 1, the base information comprising at least provider information of data provided by the data source;

The configuring the data information provided by each data source according to the configuration information, and storing the configured data information into a database corresponding to the provider information of the data source provider in the data platform, specifically including:

determining a database type corresponding to the provider information;

4. The method of claim 1, the interface information comprising at least interface address information, transport protocol information;

The method further comprises the steps of:

5. The method of claim 3, further comprising, before storing the data information provided by the data source in the database corresponding to the database type determined by the data source provider information,:

Acquiring data format information of the database;

6. The method according to claim 1, wherein the providing the data in the database to the user based on the customized configuration information of the user specifically comprises:

Acquiring a call request of a user for data;

and accessing the calling interface and providing the called data to the user.

7. The method of claim 6, wherein the customized configuration information includes a database identifier, the database identifier is used for designating a target database for acquiring data information corresponding to the data called by the user;

the determining, according to the customized configuration information of the user, a target database storing data information corresponding to the data called by the call request specifically includes:

8. The method of claim 6, wherein the determining, according to the customized configuration information of the user, a target database storing data information corresponding to the data invoked by the invocation request, specifically includes:

9. The method of claim 1, further comprising, prior to the obtaining the call request by the user of the data platform for the data provided by the data source provider:

10. The method of claim 1, wherein the customized configuration information includes the user-specified field therein;

The step of providing the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information specifically includes:

acquiring data provided by a data source provider;

Selecting a field designated by a user from the data;

And providing the content corresponding to the field for the user.

11. The method of claim 1, the customized configuration information further comprising data format information of the user-desired data;

The step of providing the data provided by the data source provider to the user based on the customized configuration information of the user specifically includes:

and providing the converted data to the user.

12. A data processing apparatus, the apparatus being applied to a data platform, comprising:

The configuration module is used for configuring the data information provided by each data source according to the configuration information, and storing the configured data information into a database corresponding to the provider information of the data source provider in the data platform;

A customized data providing module, configured to provide the data provided by the data source provider to the user based on the customized configuration information of the user and the configured data information; the customized configuration information represents the user's need for use of the data.

13. A data processing apparatus comprising:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein,