CN111831713A - A data processing method, device and equipment - Google Patents

Abstract

The present application provides a data processing method, apparatus and device, the method includes: obtaining a data processing request, the data processing request including a first input format and a second output format; obtaining a target processing unit, the target processing unit of the target processing unit The conversion information is first conversion information, and the first conversion information is used to realize the conversion of the first input format and the second output format; the first input format of the first input format is obtained from the data source according to the data processing request. data, and output the first data to the target processing unit, so that the target processing unit converts the first data into the second data in the second output format using the first conversion information; obtain the first data from the target processing unit Second data, and output the second data. Through the technical solution of the present application, the computing resources of the data lake analysis system can be saved, and the processing performance can be improved.

Description

A data processing method, device and equipment

technical field

The present application relates to the field of Internet technologies, and in particular, to a data processing method, apparatus, and device.

Background technique

Data Lake Analytics is used to provide users with serverless query and analysis services, which can analyze and query massive data in any dimension, and can support high concurrency and low latency (millisecond response). ), real-time online analysis, massive data query and other functions.

The data lake analysis system includes storage clusters and computing clusters. The storage clusters include different types of data sources, and these data sources use different data formats. A computing cluster includes multiple computing nodes, and different computing nodes can use different data formats. Usually, the data format used by the data source is different from the data format used by the computing node. Therefore, the data format needs to be converted.

For example, read data in data format A1 from a data source, convert the data in data format A1 into data in data format B1, output the data in data format B1 to the computing node, and the computing node uses the data in data format B1 for processing. Since different types of data sources use different data formats, and different computing nodes also use different data formats, the data lake analysis system needs to support the conversion of various data formats, and the data lake analysis system needs to provide a large number of computing resources. Resources realize the conversion of data formats, and as the number of users increases, the demand for computing resources also increases.

SUMMARY OF THE INVENTION

The application provides a data processing method, the method includes:

obtaining a data processing request, where the data processing request includes a first input format and a second output format;

Acquiring a target processing unit, where target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion of the first input format and the second output format;

Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the first conversion information to convert the converting the first data into second data in a second output format;

The second data is acquired from the target processing unit, and the second data is output.

The present application provides a data processing method, which is applied to a data lake analysis system, and the data lake analysis system is used to provide users with serverless data processing services, and the method includes:

obtaining a data processing request, where the data processing request includes a first input format and a second output format;

Obtain a target processing unit from multiple processing units of the data lake analysis system; wherein, target conversion information of the target processing unit is first conversion information, and the first conversion information is used to implement the first input format conversion with the second output format;

Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the first conversion information to convert the converting the first data into second data in a second output format;

Obtain the second data from the target processing unit, and output the second data;

Wherein, the data source includes a cloud database provided by the data lake analysis system.

The application provides a data processing method, the method includes:

obtaining a data processing request, where the data processing request includes a first input format and a second output format;

Acquiring first data in a first input format from a data source according to the data processing request;

outputting the first data in the first input format to a target processing unit, so that the target processing unit converts the first data into second data in a second output format;

The second data is acquired from the target processing unit, and the second data is output.

The present application provides a data processing method, which is applied to a data lake analysis system. For a processing unit in a plurality of processing units of the data lake analysis system, the processing unit includes a plurality of different conversion information, and the different conversion information uses For realizing data conversion in different formats, the method includes:

The processing unit obtains the first data in the first input format;

If the target conversion information of the processing unit is the first conversion information, and the first conversion information is used to realize the conversion of the first input format and the second output format, use the first conversion information to convert The first data is converted into second data in a second output format;

The processing unit outputs the second data.

The present application provides a data processing device, the device comprising:

an acquisition module, configured to acquire a data processing request, where the data processing request includes a first input format and a second output format; acquire a target processing unit, the target conversion information of the target processing unit is first conversion information, the first conversion information The conversion information is used to realize the conversion of the first input format and the second output format;

a processing module, configured to obtain first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the first data a conversion information to convert the first data into second data in a second output format;

The second data is acquired from the target processing unit, and the second data is output.

This application provides a data processing device, including:

A processor and a machine-readable storage medium, where several computer instructions are stored on the machine-readable storage medium, and the processor performs the following processing when executing the computer instructions:

obtaining a data processing request, where the data processing request includes a first input format and a second output format;

Acquiring a target processing unit, where target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion of the first input format and the second output format;

Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the first conversion information to convert the converting the first data into second data in a second output format;

The second data is acquired from the target processing unit, and the second data is output.

Based on the above technical solutions, in this embodiment of the present application, by setting the target conversion information of the target processing unit as the first conversion information, the target processing unit uses the first conversion information to convert the first data in the first input format into the second output The second data in the format, that is, the data format conversion is realized by the target processing unit, and the target processing unit is usually realized by a logic chip. The logic chip has high processing performance. Therefore, the computing resources (such as CPU) of the data lake analysis system can be saved. (Central Processing Unit, central processing unit) resources, etc.), and improve the overall processing performance of the data lake analysis system, improve the overall use efficiency and experience of the data lake analysis system, accelerate data processing and computing performance, and combine hardware acceleration technology to process Data docking of storage clusters and providing data interfaces to computing clusters.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings that need to be used in the description of the embodiments of the present application or the prior art. Obviously, the drawings in the following description These are just some embodiments described in this application, and for those of ordinary skill in the art, other drawings can also be obtained according to these drawings in the embodiments of this application.

1 is a schematic flowchart of a data processing method in an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a data lake analysis system in an embodiment of the present application;

3A-3E are schematic diagrams of a data scanning cluster in an embodiment of the present application;

4 is a schematic diagram of data format conversion in an embodiment of the present application;

5A and 5B are structural diagrams of a data scanning cluster in an embodiment of the present application;

6 is a schematic flowchart of a data processing method in an embodiment of the present application;

7 is a schematic structural diagram of a data processing apparatus in an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a data processing device in an embodiment of the present application.

Detailed ways

The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than limiting the present application. As used in this application and the claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information without departing from the scope of the present application. Furthermore, the use of the word "if" can be interpreted as "at the time of" or "when" or "in response to determining", depending on the context.

A data processing method is proposed in this embodiment of the present application, which can be applied to any device, such as any device of a data lake analysis system. Referring to FIG. 1 , which is a flowchart of the method, the method may include:

Step 101: Obtain a data processing request, where the data processing request includes a first input format (ie, the format of the data in the data source) and a second output format (ie, the format of the data to be output).

Step 102: Acquire a target processing unit, the target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize the conversion of the first input format and the second output format. Based on the first conversion information, the target processing unit can Convert data in the first input format to data in the second output format.

Optionally, in an example, acquiring the target processing unit may include, but is not limited to: arbitrarily selecting a processing unit from multiple processing units in the data lake analysis system, and using the selected processing unit as the target processing unit. Alternatively, target conversion information of multiple processing units of the data lake analysis system may be acquired, and a processing unit may be selected from the plurality of processing units by using the target conversion information of each processing unit, and the selected processing unit may be used as the target processing unit.

In one example, for each processing unit of the data lake analysis system, the processing unit may be: a processing unit that is currently not working (that is, the processing unit is not currently performing data conversion operations), or a processing unit that is currently working (that is, the processing unit is currently performing a data transformation operation).

In one example, using the target conversion information of each processing unit to select a processing unit from the plurality of processing units, and using the selected processing unit as the target processing unit, may include, but is not limited to: if there is target conversion information for the first The processing unit of the conversion information (for realizing the conversion of the first input format and the second output format), then the processing unit whose target conversion information is the first conversion information is determined as the target processing unit; or, if there is no target conversion information, it is For the processing unit of the first conversion information, a processing unit is arbitrarily selected from the plurality of processing units, and the selected processing unit is determined as the target processing unit.

Optionally, in an example, after acquiring the target processing unit, it may also include but is not limited to: if the target conversion information of the target processing unit is the first conversion information, maintaining the target processing according to the first input format and the second output format. The target conversion information of the unit is unchanged; or, if the target conversion information of the target processing unit is the second conversion information (the second conversion information is not used to realize the conversion of the first input format and the second output format), then according to the first input format and the second output format modifies the target conversion information of the target processing unit to the first conversion information.

Optionally, in an example, for step 102, it can also be determined whether the data lake analysis system supports the conversion of the first input format and the second output format. If yes, that is, the processing unit of the data lake analysis system supports the conversion of the first input format and the second output format, the target processing unit is obtained from the multiple processing units of the data lake analysis system. If no, that is, all processing units of the data lake analysis system do not support the conversion between the first input format and the second output format, the traditional process is used for processing.

Optionally, in an example, for step 102, the data processing request may further include the number of shards, the number of target processing units is determined according to the number of shards, and the number of target processing units is acquired.

Step 103: Obtain the first data in the first input format from the data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data using the first conversion information is the second data in the second output format, and the conversion process will not be described again.

Step 104: Acquire the second data from the target processing unit, and output the second data, for example, the second data may be output to the computing node, so that the computing node uses the second data for processing.

In one example, the data processing request may further include a service mode. If the service mode is a traffic mode, the total amount of data may be obtained, virtual resource information (such as cost information) may be determined according to the total amount of data, and virtual resource information may be output. Alternatively, if the service mode is the instance mode, the number of target processing units may be acquired, and virtual resource information may be determined according to the number of target processing units, and the virtual resource information may be output.

In the above embodiment, the target processing unit includes a plurality of different conversion information, and the different conversion information is used to realize data conversion in different formats; the target processing unit is realized by a logic chip, and the logic chip may include but not limited to: FPGA (Field Programmable Gate Array, field programmable logic gate array), CPLD (Complex Programmable Logic Device, complex programmable logic device), ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), etc., which are not limited.

In an example, the above execution sequence is only an example given for the convenience of description. In practical applications, the execution sequence between steps may also be changed, and the execution sequence is not limited. Moreover, in other embodiments, the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification, and the methods may include more or less steps than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

Based on the above technical solutions, in the embodiment of the present application, the conversion of data formats is implemented by the target processing unit. The target processing unit is usually implemented by a logic chip and has high processing performance, which can save the computing resources (such as CPU resources) of the data lake analysis system. etc.), improve the overall processing performance of the data lake analysis system, improve the overall use efficiency and experience of the data lake analysis system, accelerate data processing and computing performance, and combine hardware acceleration technology to process the data docking of storage clusters and provide data interfaces to computing clusters.

Based on the same application concept as the above method, the embodiment of the present application also proposes another data processing method, which can be applied to a data lake analysis system (such as a cloud computing platform in the data lake analysis system), and the data lake analysis system is used for Provide users with serverless data processing services, the method includes:

Obtain a data processing request, the data processing request includes a first input format and a second output format; obtain a target processing unit from multiple processing units of the data lake analysis system, and target conversion information of the target processing unit is the first conversion information, and the first conversion information is the first conversion information. A conversion information is used to realize the conversion of the first input format and the second output format. Obtain the first data in the first input format from the data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the first conversion information to convert the first data into the second output the second data in the format; obtain the second data from the target processing unit, and output the second data; wherein, the data source includes a cloud database provided by the data lake analysis system.

The above data source may include a cloud database provided by the data lake analysis system, and the cloud database may be used to provide serverless query and analysis services. The data lake analysis system can be a storage-type cloud platform that mainly focuses on data storage, or a computing-type cloud platform that focuses on data processing, or a comprehensive cloud computing platform that takes both computing and data storage and processing into consideration. No restrictions.

The cloud database provided by the data lake analysis system can be used to provide users with serverless query and analysis services, which can analyze and query massive data in any dimension, and support high concurrency and low latency (millisecond level). response), real-time online analysis, massive data query and other functions.

In one example, the data lake analysis system is specifically: a data lake analysis system with separate storage and computing; the data lake analysis system includes a storage cluster and a computing cluster, the storage cluster includes multiple data sources with different input formats, and the computing cluster includes Multiple compute nodes with different output formats. Further, the data lake analysis system may also include a data scanning cluster, and the data scanning cluster includes multiple processing units; the data scanning cluster, as a built-in module of the computing cluster, is deployed on the same node as the computing resources of the computing cluster; The independent modules of the computing cluster are deployed on different nodes from the computing resources of the computing cluster; or, the data scanning cluster is used as an independent cluster different from the computing cluster.

Based on the same application concept as the above method, an embodiment of the present application also proposes a data processing method, which may include: acquiring a data processing request, where the data processing request may include a first input format and a second output format; The data processing request obtains the first data in the first input format from the data source, and outputs the first data to the target processing unit, so that the target processing unit converts the first data into the second data in the second output format; The unit acquires the second data and outputs the second data.

Based on the same application concept as the above method, an embodiment of the present application also proposes a data processing method, which is applied to a data lake analysis system. The data lake analysis system includes a plurality of processing units, for each of the plurality of processing units a processing unit, the processing unit includes a plurality of different conversion information, and the different conversion information is used to realize the data conversion of different formats, and the method includes:

The processing unit obtains the first data in the first input format; if the target conversion information of the processing unit is the first conversion information, and the first conversion information is used to realize the first input format and the second output format conversion, the processing unit converts the first data into second data in a second output format by using the first conversion information; the processing unit outputs the second data.

In an example, before the processing unit converts the first data into the second data in the second output format by using the first conversion information, if the target conversion information of the processing unit is not the first conversion information, Then the processing unit modifies the target conversion information of the processing unit to the conversion information.

The above data processing method will be further described below with reference to specific application scenarios.

Referring to Figure 2, which is a schematic diagram of the structure of the Data Lake Analytics system, the Data Lake Analytics system may include a client, a load balancing device, a front node (also referred to as a front-end server), a computing node ( A compute node (also referred to as a computing server) and a database, of course, the data lake analysis system may also include other servers, which is not limited.

In FIG. 2 , three front-end nodes are taken as an example. In practical applications, the number of front-end nodes may also be other numbers, which are not limited. In FIG. 2 , four computing nodes are taken as an example. In practical applications, the number of computing nodes may also be other numbers, which are not limited. Since the processing flow of each front-end node is the same, and the processing flow of each computing node is the same, for convenience of description, in the following embodiments, the processing flow of one front-end node is taken as an example, and the processing flow of one computing node is example.

In FIG. 2 , five databases are taken as an example. In practical applications, the number of databases may also be other numbers, which are not limited, and these databases are the data sources. In this embodiment, it may be a scenario for heterogeneous data sources, that is, these databases may be databases of the same type, or may be databases of different types. These databases can be relational or non-relational.

Further, for each database, the type of the database may also include but is not limited to: OSS (Object Storage Service, object storage service), TableStore (table storage), HBase (HadoopDatabase, Hadoop database), HDFS (Hadoop Distributed) File System, Hadoop distributed file system), MySQL (that is, relational database), RDS (Relational Database Service, relational database service), DRDS (Distribute Relational Database Service, distributed relational database service), RDBMS (Relational Database Management) System, relational database management system), SQLServer (i.e. relational database), PostgreSQL (i.e. object-relational database), MongoDB (i.e. database based on distributed file storage), etc. Of course, the above are just a few examples of database types. The type of this database is not limited.

Among them, the database is used to store various types of data, which is not limited, such as user data, commodity data, map data, video data, image data, audio data, and the like.

The client may be an APP (Application) included in a terminal device (such as a PC (Personal Computer, personal computer), a notebook computer, a mobile terminal, etc.), or a browser included in the terminal device, which is not limited . The load balancing device is used to load balance the data request of the client. For example, after receiving the data request, load balancing the data request to each front-end node.

In one example, multiple front-end nodes can be used to provide the same function, forming a resource pool of front-end nodes. For each front-end node in the resource pool, it is used to receive the data request sent by the client, perform SQL (Structured Query Language, Structured Query Language) parsing on the data request, generate multiple execution plans according to the parsing results, and process these Implementation plan. For example, the front-end node can send these execution plans to one or more computing nodes, and the computing nodes process the execution plans.

In one example, multiple computing nodes are used to provide the same function, forming a resource pool of computing nodes. For each computing node in the resource pool, if the computing node receives the execution plan sent by the front-end node, the computing node can process the execution plan and return the processing result to the front-end node.

To sum up, the data lake analysis system adopts an architecture that separates storage and computing, and computing nodes read data from different data sources, which are various types of databases.

In one example, the data lake analysis system is specifically an architecture in which storage and computing are separated, that is, the data lake analysis system includes a storage cluster and a computing cluster, the storage cluster includes multiple data sources (ie databases) using different input formats, and the computing cluster includes Multiple compute nodes with different output formats. On this basis, in this embodiment of the present application, the data lake analysis system may further include a data scanning cluster, and the data scanning cluster may include multiple processing units, for example, processing units implemented by FPGAs.

Referring to Figure 3A, the data scanning cluster can be used as an independent module of the computing cluster, and the computing resources (such as CPU resources, etc.) of the computing cluster are deployed on different nodes, that is, the processing units of the data scanning cluster are deployed in the computing cluster, but The computing resources (such as CPU resources, etc.) of the computing cluster are deployed on different nodes. Specifically, in the data lake analysis system in which storage and computing are separated, the data scanning cluster, as a module in the computing cluster, is a functional module in the computing cluster that directly faces the storage cluster.

Referring to Fig. 3B, the data scanning cluster can be used as a built-in module of the computing cluster, and is deployed on the same node as the computing resources (such as CPU resources, etc.) of the computing cluster, that is, the processing units of the data scanning cluster are deployed in the computing cluster, as The built-in module of the computing node in the computing cluster is located on the same node as the CPU-based operator. The computing task scheduling determines whether to enable the data scanning cluster to perform data format conversion. If not, the CPU software module based on the computing node realizes data format conversion. .

Referring to Figure 3C, the data scanning cluster can be used as an independent cluster different from the computing cluster. In the data lake analysis system where storage and computing are separated, the data scanning cluster is used as a functional module oriented to the computing cluster, and the data scanning cluster is used as a storage-oriented cluster. function module. The data scanning cluster is a completely independent cluster on the cloud, which can concurrently respond to data scanning requests from different computing clusters on the cloud in the form of a service. The data scanning cluster runs completely independently, and has its own cluster elastic management and capacity expansion .

For convenience of description, in subsequent embodiments, the data scanning cluster is taken as an independent cluster as an example.

In one example, the data lake analysis system may include multiple computing clusters, each computing cluster including multiple computing nodes. For each computing cluster, it can be a computing cluster oriented to SQL (Structured Query Language, Structured Query Language) computing, a computing cluster oriented to machine learning, or a computing cluster oriented to deep learning (Deep Learning). , there is no restriction on this.

Specifically, as shown in Figure 3D, these computing clusters may include but are not limited to: Presto-based computing clusters, Spark-based computing clusters, Hadoop-based computing clusters, Flink-based computing clusters, TensorFlow-based computing clusters, and PyTorch-based computing clusters computing clusters, etc.

For a Presto-based computing cluster, a data access interface adapted to Presto is provided, that is, the data output to the computing cluster is data that matches the Presto data format. For a Spark-based computing cluster, a data access interface adapted to Spark is provided, that is, the data output to the computing cluster is data that matches the Spark data format. For a Hadoop-based computing cluster, a data access interface adapted to Hadoop is provided, that is, the data output to the computing cluster is data that matches the Hadoop data format. For a Flink-based computing cluster, a data access interface adapted to Flink is provided, that is, the data output to the computing cluster is data that matches the Flink data format. For a TensorFlow-based computing cluster, a data access interface adapted to TensorFlow is provided, that is, the data output to the computing cluster is data that matches the TensorFlow data format. For a PyTorch-based computing cluster, a data access interface adapted to PyTorch is provided, that is, the data output to the computing cluster is data that matches the PyTorch data format, and so on.

In one example, the data lake analysis system may include a storage cluster, and the storage cluster includes multiple data sources, and the data source may be a database, such as a cloud database, and the cloud database is used to provide users with serverless (Serverless) query and analysis services , can analyze and query massive data in any dimension, support high concurrency, low latency (millisecond response), real-time online analysis, massive data query, etc.

In one example, these data sources may include, but are not limited to: OSS-based data sources, TableStore-based data sources, HBase-based data sources, HDFS-based data sources, MySQL-based data sources, RDS-based data sources, DRDS-based data sources, RDBMS-based data sources, PostgreSQL-based data sources, etc. Of course, the above is just an example, which is not limited.

Referring to Figure 3E, due to the different types of data sources, the data formats of the data in the data sources are also different. For example, the data formats may include but are not limited to: parquet data format, orc data format, text data format, json data format , kv data format, rcfile data format, avro data format, arrow data format, etc. Of course, the above is just an example, and other data formats are also possible, which is not limited.

To sum up, since the data format of the data source is different from that of the computing cluster, it is necessary to convert the data format so that the computing cluster can process the data correctly. For example, if the data format of the data source is json data format and the computing cluster is a Presto-based computing cluster, the data in json data format needs to be converted into data that matches the Presto data format.

In the embodiment of the present application, the data format conversion is realized by providing a data scanning cluster, that is, the data format conversion is realized by a processing unit (such as an FPGA, etc.) in the data scanning cluster.

In an example, in order to realize the conversion of the data format, the conversion information can be configured in the processing unit (such as FPGA, etc.), and the processing unit can use the conversion information to realize the conversion of the data format. The content of the conversion information is not limited, as long as the processing unit The conversion of the data format can be realized by using the conversion information.

For example, the conversion information A1 is configured in the processing unit in advance, and based on the conversion information A1, the processing unit can convert the data in the json data format into data matching the Presto data format.

In one example, a plurality of different conversion information may be configured in a processing unit (eg, FPGA, etc.), and different conversion information is used to realize data conversion in different formats. For example, the conversion information A1, the conversion information A2, the conversion information A3, and the conversion information A4 are configured in the processing unit, and so on. Based on the conversion information A1, the processing unit can convert the data in the json data format into data matching the Presto data format. Based on the conversion information A2, the processing unit can convert the data in the json data format into data matching the Spark data format. Based on the conversion information A3, the processing unit can convert the data in the text data format into data matching the Presto data format. Based on the conversion information A4, the processing unit can convert the data in the text data format into data matching the Spark data format, and so on.

Of course, the above is just an example of conversion information. In practical applications, more conversion information can be configured in the processing unit to realize conversion of various data formats. See FIG. 4 , which is a schematic diagram of data format conversion. The first column indicates the data format supported by the data source, and the first row indicates the data format supported by the computing cluster. "Yes" in Figure 4 indicates that the conversion of these two data formats is supported, and "No" in Figure 4 indicates that the conversion of these two data formats is not supported. Based on this, a plurality of conversion information can be configured in the processing unit, so that the processing unit supports the conversion of two data formats corresponding to "Yes" through the conversion information.

To sum up, since a plurality of different conversion information is configured in the processing unit, and the different conversion information is used to realize data conversion of different formats, the computing capability of the processing unit can be fully utilized, and the utilization rate of the processing unit can be improved. For example, if the processing unit is configured with conversion information A1, the processing unit is configured to convert the data in the json data format into data matching the Presto data format. When there is no task of "converting the data in the json data format into data that matches the Presto data format", the processing unit is in an idle state, wasting the computing power of the processing unit. If the processing unit is configured with conversion information A1 and conversion information A2, the processing unit is used to convert the data in the json data format into data matching the Presto data format, and convert the data in the json data format into data matching the Spark data format . When there is no task of "converting data in json data format into data matching the Presto data format", the processing unit can also convert the data in json data format into data matching the Spark data format, so as to avoid the processing unit in the The idle state increases the computing power of the processing unit.

In one example, the usage of the processing units in the data scanning cluster is relatively fixed for accelerating data scanning tasks of different computing clusters. Referring to FIG. 5A , the data scanning cluster may include basic modules such as instruction storage, data storage, constant storage, register group, data storage linked list, and instruction execution. Further, as shown in FIG. 5B , the data scanning cluster may also include multiple processing units (such as FPGAs, etc.), each processing unit is used to convert different data formats, in addition, the data scanning cluster may also include scheduling and Management modules, input modules, and output modules, etc.

In the above application scenario, referring to FIG. 6 , the flowchart of the data processing method proposed in this embodiment of the present application can be applied to a data scanning cluster of a data lake analysis system, and the method may include:

In step 601, a data processing request, such as a data scan request, is obtained.

Specifically, the client can send a data processing request to the data lake analysis system through the load balancing device, so that the data scanning cluster of the data lake analysis system can obtain the data processing request. For example, the scheduling and management module of the data scanning cluster can obtain the data processing request.

Step 602: Determine whether the data lake analysis system supports data format conversion corresponding to the data processing request. If yes, go to step 603; if no, prompt that the data processing request is not supported.

Specifically, the data processing request may include the input data format (that is, the format of the data in the data source, for the convenience of distinction, the first input format is taken as an example, such as the json data format) and the output target format (that is, the format of the data to be outputted) Format, for the convenience of distinction, the second output format is used as an example in the following, such as Presto data format), therefore, it can be judged whether the data lake analysis system supports the conversion of the first input format and the second output format, and if so, go to step 603; If not, prompt that the data processing request is not supported.

For example, the scheduling and management module of the data scanning cluster can obtain the first input format and the second output format from the data processing request, and query whether the data lake analysis system supports the conversion of the first input format and the second output format. Specifically, it is assumed that the data lake analysis system includes a capability registry, and the capability registry is used to record the conversion of all data formats supported by the data lake analysis system, and the capability registry is shown in FIG. 4 . If the capability registry does not have the first input format and/or the second output format, it is determined that the data lake analysis system does not support the conversion of the first input format and the second output format; if the capability registry has the first input format and the second output format Output format, and the corresponding value of the first input format and the second output format is "No", it is determined that the data lake analysis system does not support the conversion of the first input format and the second output format; if the capability registry has the first input format and The second output format, and the corresponding value of the first input format and the second output format is "Yes", it is determined that the data lake analysis system supports the conversion of the first input format and the second output format.

Step 603, select a target processing unit from multiple processing units of the data lake analysis system.

Wherein, for each processing unit in the plurality of processing units, the processing unit may be: a processing unit that is currently not working (that is, the processing unit does not currently perform a data conversion operation), or a processing unit that is currently working (that is, the processing unit is currently performing a data transformation operation).

For example, the target processing unit can be selected from processing unit 1, processing unit 2, and processing unit 3. Processing unit 1 can be the currently inactive processing unit or the currently active processing unit, and processing unit 2 can be the currently inactive processing unit unit or currently working processing unit, and so on.

In an example, the data processing request may also include a service mode. If the service mode is a traffic mode, it means that the user adopts the total amount of data billing. Based on this, the user can share the processing unit with other users. Therefore, the data lake analysis The multiple processing units of the system can be the currently not working processing units or the currently working processing units, that is to say, the currently not working processing units can be used as the target processing units, or the currently working processing units can be used as the target processing units. target processing unit. If the service mode is the instance mode, it means that the user is billed by the number of processing units. Based on this, the user uses the processing unit alone. Therefore, the multiple processing units of the data lake analysis system can be the processing units that are not currently working, that is, Say, the processing unit that is not currently working can be the target processing unit.

In one example, a processing unit may be arbitrarily selected from a plurality of processing units of the data lake analysis system, and the selected processing unit may be used as a target processing unit. Alternatively, the target conversion information of multiple processing units of the data lake analysis system may be acquired, and the target conversion information of each processing unit may be used to select a processing unit from the plurality of processing units, and the selected processing unit may be used as the target processing unit .

For example, assuming that the scheduling and management module of the data scanning cluster needs to select the target processing unit from processing unit 1, processing unit 2 and processing unit 3, the following method is adopted: the processing unit can be randomly selected from these processing units, such as selecting a processing unit 1, and use processing unit 1 as the target processing unit; or, according to the target conversion information of processing unit 1, processing unit 2 and processing unit 3, select a processing unit from these processing units, such as selecting processing unit 2, and set the processing unit 2 as the target processing unit.

Wherein, using the target conversion information of each processing unit to select a processing unit from a plurality of processing units, and using the selected processing unit as the target processing unit, may include but not limited to: if there is target conversion information, the first conversion information (using In order to realize the processing unit of the conversion of the first input format and the second output format), the processing unit whose target conversion information is the first conversion information can be determined as the target processing unit; or, if there is no target conversion information, it is the first conversion information. information processing unit, a processing unit may be randomly selected from a plurality of processing units, and the selected processing unit may be determined as the target processing unit.

The target conversion information is the conversion information currently enabled by the processing unit, that is, the conversion information currently being used by the processing unit. For example, the processing unit configures conversion information A1 (used to convert data in json data format to data matching the Presto data format) and conversion information A2 (used to convert data in json data format to data matching the Spark data format) data), if the target conversion information is conversion information A1, it means that the processing unit is currently used to convert the data in the json data format into data that matches the Presto data format, but it is not used to convert the data in the json data format into data with Data that matches the Spark data format. If the target conversion information is conversion information A2, it means that the processing unit is currently used to convert the data in the json data format into data that matches the Spark data format, and so on.

Assuming that the first input format is the json data format and the second output format is the Presto data format, the first conversion information is the conversion information A1, that is, the first conversion information is used to implement the conversion between the json data format and the Presto data format. If the target conversion information of the processing unit 1 is the conversion information A1, the target conversion information of the processing unit 1 is the first conversion information, and the processing unit 1 may be determined as the target processing unit.

In one example, the data processing request may further include the number of shards, which indicates the number of processing units that the user needs to use. Therefore, the number of target processing units can also be determined according to the number of shards, and then the data lake can be analyzed from the data lake. The number of target processing units is selected from among the processing units.

For example, if the number of shards is 5, that is, the number of processing units is 5, it is necessary to select 5 target processing units from the multiple processing units of the data lake analysis system. For the specific selection method, refer to the above embodiment.

Step 604: Set the target conversion information of the target processing unit (eg, one or more target processing units, eg, 5 target processing units) as the first conversion information according to the first input format and the second output format.

Wherein, the first conversion information is used to realize the conversion of the first input format and the second output format, that is, used to convert the data of the first input format into the data of the second output format.

Specifically, if the target conversion information of the target processing unit is the first conversion information, then keep the target conversion information of the target processing unit unchanged according to the first input format and the second output format; or, if the target conversion information of the target processing unit is the second conversion information (the second conversion information is not used to realize the conversion of the first input format and the second output format), then according to the first input format and the second output format, the target conversion information of the target processing unit is converted from the second conversion information Modified to the first conversion information.

For example, assuming that the first input format is the json data format and the second output format is the Presto data format, the first conversion information is conversion information A1, that is, the first conversion information is used to implement the conversion between the json data format and the Presto data format. Further, if the target conversion information of the target processing unit is the conversion information A1, the target conversion information of the target processing unit can be kept unchanged, that is, the target conversion information is still the conversion information A1. If the target conversion information of the target processing unit is conversion information A2 (for realizing the conversion between json data format and Spark data format), the target conversion information of the target processing unit can be modified to conversion information A1, so that the target processing unit no longer It is used to realize the conversion between json data format and Spark data format, but it is used to realize the conversion between json data format and Presto data format.

Wherein, steps 601 to 604 may be performed by the scheduling and management module of the data scanning cluster.

Step 605: Acquire the first data in the first input format from the data source according to the data processing request (the data in the data source may be referred to as the first data), and output the first data to the target processing unit.

Specifically, the data processing request may include information of the data source. Based on the information of the data source, the first data may be obtained from the data source, and the data format of the first data is the first input format, and the obtaining process will not be repeated. Then, the first data in the first input format can be output to the target processing unit.

For example, the input module of the data scanning cluster may acquire the first data in the first input format from the data source, and output the first data in the first input format to the target processing unit.

Step 606 , the target processing unit converts the first data into second data in the second output format by using the first conversion information (the converted data is referred to as the second data), and the conversion process will not be repeated.

Specifically, referring to the above embodiment, the target conversion information of the target processing unit is the first conversion information, such as conversion information A1, and the conversion information A1 is used to realize the conversion between the json data format and the Presto data format. Assuming that the first input format is the json data format, and the second output format is the Presto data format, based on this, the data format of the first data is the json data format, and the target processing unit can use the conversion information A1 to convert the first data format of the json data format. The data is converted into the second data in the Presto data format.

Step 607: Acquire second data in the second output format from the target processing unit, and output the second data.

For example, the output module of the data scanning cluster obtains the second data in the second output format from the target processing unit, such as the second data in the Presto data format, and outputs the second data in the Presto data format to the computing node, such as Presto-based computing Compute nodes within the cluster. Since the output to the computing node is the second data in the Presto data format, the computing node can use the second data for processing.

In an example, the data processing request may also include a service mode. If the service mode is a traffic mode (ie, a shared service type), it means that the user adopts the total amount of data billing, and based on this, the user can share the processing unit with other users. , therefore, the total amount of data (that is, the total amount of data read from the data source) can be obtained, and the virtual resource information (such as cost information) can be determined according to the total amount of data, and the virtual resource information can be output, such as outputting virtual resources to users information. If the service mode is an instance mode (ie, an exclusive instance type), it means that the user uses the number of processing units for billing. Based on this, the user uses the processing unit alone. Therefore, the number of target processing units can be obtained, and the virtual resource information (such as cost information), and output virtual resource information, such as outputting virtual resource information to the user.

In an example, the above execution sequence is only an example given for the convenience of description. In practical applications, the execution sequence between steps may also be changed, and the execution sequence is not limited. Moreover, in other embodiments, the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification, and the methods may include more or less steps than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

Based on the above technical solutions, in the embodiment of the present application, the data format conversion is realized by the target processing unit. The target processing unit is usually realized by a logic chip, which has high processing performance and saves the computing resources (such as CPU resources, etc.) of the data lake analysis system. ), improve the overall processing performance of the data lake analysis system, improve the overall use efficiency and experience of the data lake analysis system, accelerate data processing and computing performance, and combine hardware acceleration technology to process data connection of storage clusters and provide data interfaces to computing clusters. The data scanning cluster in this embodiment has better versatility and productized application capabilities, greatly improves the application range of computing clusters for interconnection and acceleration, greatly improves the productization capabilities of cloud products, and provides FPGA data scanning acceleration services in multiple modes. , a general FPGA data scanning engine is proposed, which can build input and output support for multiple data formats, and develop a specific FPGA data scanning computing acceleration core for a specific computing engine.

Based on the same application concept as the above method, an embodiment of the present application further provides a data processing apparatus. As shown in FIG. 7 , which is a structural diagram of the data processing apparatus, the data processing apparatus includes:

The obtaining module 71 is configured to obtain a data processing request, where the data processing request includes a first input format and a second output format; obtain a target processing unit, the target conversion information of the target processing unit is the first conversion information, and the first conversion information is the first conversion information. A conversion information is used to realize the conversion of the first input format and the second output format;

A processing module 72, configured to obtain first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the The first conversion information converts the first data into second data in a second output format;

The second data is acquired from the target processing unit, and the second data is output.

When the acquisition module 71 acquires the target processing unit, it is specifically used to: acquire target conversion information of multiple processing units of the data lake analysis system, and use the target conversion information to select a processing unit from the plurality of processing units as the target processing unit. unit.

In one example, the processing module 72 is further configured to:

If the target conversion information of the target processing unit is the first conversion information, keep the target conversion information of the target processing unit unchanged according to the first input format and the second output format; or,

If the target conversion information of the target processing unit is the second conversion information, the target conversion information of the target processing unit is modified into the first conversion information according to the first input format and the second output format.

Based on the same application concept as the above method, an embodiment of the present application further provides a data processing device, including: a processor and a machine-readable storage medium, where several computer instructions are stored on the machine-readable storage medium, and the processor The following processes are performed when executing the computer instructions:

obtaining a data processing request, where the data processing request includes a first input format and a second output format;

Acquiring a target processing unit, where target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion of the first input format and the second output format;

Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the first conversion information to convert the converting the first data into second data in a second output format;

The second data is acquired from the target processing unit, and the second data is output.

Embodiments of the present application also provide a machine-readable storage medium, where several computer instructions are stored on the machine-readable storage medium; when the computer instructions are executed, the following processing is performed:

obtaining a data processing request, where the data processing request includes a first input format and a second output format;

Acquiring a target processing unit, where target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion of the first input format and the second output format;

Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the first conversion information to convert the converting the first data into second data in a second output format;

The second data is acquired from the target processing unit, and the second data is output.

Referring to FIG. 8 , which is a structural diagram of a data processing device proposed in an embodiment of the application, the data processing device 80 may include: a processor 81 , a network interface 82 , a bus 83 , and a memory 84 . Memory 84 may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the memory 84 may be: RAM (Radom Access Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid state drives, any type of storage disks (such as optical disks) , DVD, etc.).

The systems, devices, modules or units described in the above embodiments may be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementing device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smart phone, personal digital assistant, media player, navigation device, email sending and receiving device, game control desktop, tablet, wearable device, or a combination of any of these devices.

For the convenience of description, when describing the above device, the functions are divided into various units and described respectively. Of course, when implementing the present application, the functions of each unit may be implemented in one or more software and/or hardware.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

Furthermore, these computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer readable memory result in an article of manufacture comprising the instruction means, The instruction means implements the functions specified in a flow or flows of the flowcharts and/or a block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The above descriptions are merely examples of the present application, and are not intended to limit the present application. Various modifications and variations of this application are possible for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims (16)

1. a data processing method, is characterized in that, described method comprises: obtaining a data processing request, where the data processing request includes a first input format and a second output format; Acquiring a target processing unit, where target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion of the first input format and the second output format; Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the first conversion information to convert the converting the first data into second data in a second output format; The second data is acquired from the target processing unit, and the second data is output. 2. The method according to claim 1, wherein The acquisition target processing unit includes: Obtain target conversion information of multiple processing units of the data lake analysis system, and use the target conversion information to select a processing unit from the plurality of processing units as a target processing unit. 3. The method according to claim 2, wherein the selecting a processing unit from the plurality of processing units by using the target conversion information as the target processing unit comprises: If there is a processing unit whose target conversion information is the first conversion information, the processing unit whose target conversion information is the first conversion information is determined as the target processing unit; or, If there is no processing unit whose target conversion information is the first conversion information, a processing unit is selected from the plurality of processing units, and the selected processing unit is determined as the target processing unit. 4. The method of claim 1, wherein After the acquisition of the target processing unit, the method further includes: If the target conversion information of the target processing unit is the first conversion information, keep the target conversion information of the target processing unit unchanged according to the first input format and the second output format; or, If the target conversion information of the target processing unit is the second conversion information, the target conversion information of the target processing unit is modified into the first conversion information according to the first input format and the second output format. 5. The method of claim 1, wherein The acquisition target processing unit includes: Determine whether the data lake analysis system supports the conversion between the first input format and the second output format; If yes, get the target processing unit from the multiple processing units of the data lake analytics system. 6. The method of claim 1, wherein The data processing request further includes the number of shards, and the acquisition target processing unit includes: Determine the number of target processing units according to the number of shards; Obtain the number of target processing units. 7. The method of claim 1, wherein the method further comprises: The data processing request further includes a service mode, and if the service mode is a traffic mode, the total amount of data is obtained, and virtual resource information is determined according to the total amount of data, and the virtual resource information is output; If the service mode is the instance mode, the number of target processing units is acquired, virtual resource information is determined according to the number of target processing units, and the virtual resource information is output. 8. A data processing method, characterized in that it is applied to a data lake analysis system, and the data lake analysis system is used to provide users with serverless data processing services, the method comprising: obtaining a data processing request, where the data processing request includes a first input format and a second output format; Obtain a target processing unit from multiple processing units of the data lake analysis system; wherein, target conversion information of the target processing unit is first conversion information, and the first conversion information is used to implement the first input format conversion with the second output format; Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the first conversion information to convert the converting the first data into second data in a second output format; Obtain the second data from the target processing unit, and output the second data; Wherein, the data source includes a cloud database provided by the data lake analysis system. 9. The method of claim 8, wherein: The data lake analysis system is specifically: a data lake analysis system in which storage and computing are separated; the data lake analysis system includes a storage cluster and a computing cluster, the storage cluster includes multiple data sources using different input formats, and the computing A cluster includes multiple compute nodes with different output formats; The data lake analysis system further includes a data scanning cluster, and the data scanning cluster includes a plurality of processing units; the data scanning cluster, as a built-in module of the computing cluster, is deployed on the same node as the computing resources of the computing cluster; Alternatively, the data scanning cluster acts as an independent module of the computing cluster, and is deployed on a different node from the computing resources of the computing cluster; or, the data scanning cluster acts as an independent cluster different from the computing cluster. 10. A data processing method, wherein the method comprises: obtaining a data processing request, where the data processing request includes a first input format and a second output format; Acquiring first data in a first input format from a data source according to the data processing request; outputting the first data in the first input format to a target processing unit, so that the target processing unit converts the first data into second data in a second output format; The second data is acquired from the target processing unit, and the second data is output. 11. A data processing method, characterized in that, when applied to a data lake analysis system, for a processing unit in a plurality of processing units of the data lake analysis system, the processing unit includes a plurality of different conversion information, different The conversion information is used to realize data conversion in different formats, and the method includes: The processing unit obtains the first data in the first input format; If the target conversion information of the processing unit is the first conversion information, and the first conversion information is used to realize the conversion of the first input format and the second output format, use the first conversion information to convert the converting the first data into second data in a second output format; The processing unit outputs the second data. 12 . The method according to claim 11 , wherein before converting the first data into the second data in the second output format by using the first conversion information, the method further comprises: 12 . If the target conversion information of the processing unit is not the first conversion information, the processing unit modifies the target conversion information of the processing unit to the first conversion information. 13. A data processing device, wherein the device comprises: an acquisition module, configured to acquire a data processing request, where the data processing request includes a first input format and a second output format; acquire a target processing unit, the target conversion information of the target processing unit is first conversion information, the first conversion information The conversion information is used to realize the conversion of the first input format and the second output format; a processing module, configured to obtain first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the first data a conversion information to convert the first data into second data in a second output format; The second data is acquired from the target processing unit, and the second data is output. 14. The apparatus of claim 13, wherein When the acquisition module acquires the target processing unit, it is specifically used for: Obtain target conversion information of multiple processing units of the data lake analysis system, and use the target conversion information to select a processing unit from the plurality of processing units as a target processing unit. 15. The apparatus according to claim 13, wherein the processing module is further configured to: If the target conversion information of the target processing unit is the first conversion information, keep the target conversion information of the target processing unit unchanged according to the first input format and the second output format; or, If the target conversion information of the target processing unit is the second conversion information, the target conversion information of the target processing unit is modified into the first conversion information according to the first input format and the second output format. 16. A data processing device, comprising: A processor and a machine-readable storage medium, where several computer instructions are stored on the machine-readable storage medium, and the processor performs the following processing when executing the computer instructions: obtaining a data processing request, where the data processing request includes a first input format and a second output format; Acquiring a target processing unit, where target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion of the first input format and the second output format; Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit uses the first conversion information to convert the converting the first data into second data in a second output format; The second data is acquired from the target processing unit, and the second data is output.

Images