CN112860777B

CN112860777B - Data processing method, device and equipment

Info

Publication number: CN112860777B
Application number: CN202110301981.3A
Authority: CN
Inventors: 李陆彪; 张世维; 杨绍鹏
Original assignee: Shenzhen Tencent Information Technology Co Ltd
Current assignee: Shenzhen Tencent Information Technology Co Ltd
Priority date: 2021-03-22
Filing date: 2021-03-22
Publication date: 2024-03-15
Anticipated expiration: 2041-03-22
Also published as: CN112860777A

Abstract

The application discloses a data processing method, device and equipment, and belongs to the technical field of computers. The method comprises the following steps: acquiring configuration information of a data export task; based on the data source information, calling the source data corresponding to the data export task; obtaining a format conversion library file corresponding to a data export format; performing format conversion processing on the source data based on the format conversion library file to generate export data conforming to a data export format; and sending the export data to at least one target machine corresponding to the data export address according to the data export address. In the technical scheme provided by the embodiment of the application, the source data corresponding to the called data export task is subjected to format conversion to obtain the export data conforming to the preset format, and the export data is automatically sent to the target machine, so that the data export of multiple formats is realized, the needs of multiple business scenes are met, the data export cost is reduced, and the data export efficiency is improved.

Description

Data processing method, device and equipment

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, apparatus, and device.

Background

With the rise of the internet, the capacity of people to produce and collect data is greatly enhanced, and new values are expected to be obtained from the data. Big data traffic is constantly emerging, data export being a very important service in databases.

In related data export schemes, the data export task is typically dominated by the user, who traverses various data in the database, and then exports the read data to a fixed server. The user needs to design a set of independent export system according to the service scene, so that the needed data can be exported from the database. If the business scene of the user changes, the user can redesign a set of export systems according to the changed business scene, and if the user wants to change an export format, the export format can be changed by modifying the export systems.

In the related art, the data export is low in efficiency, high in cost and single in applicable scene.

Disclosure of Invention

The embodiment of the application provides a data processing method, a device and equipment, which can reduce the data export cost, improve the data export efficiency and have wide application scenes.

In one aspect, an embodiment of the present application provides a data processing method, where the method includes:

Acquiring configuration information of a data export task, wherein the configuration information comprises data source information, a data export format and a data export address;

invoking source data corresponding to the data export task based on the data source information;

obtaining a format conversion library file corresponding to the data export format;

performing format conversion processing on the source data based on the format conversion library file to generate export data conforming to the data export format;

and according to the data export address, sending the export data to at least one target machine corresponding to the data export address.

In another aspect, an embodiment of the present application provides a data processing apparatus, including:

the configuration acquisition module is used for acquiring configuration information of the data export task, wherein the configuration information comprises data source information, a data export format and a data export address;

the data calling module is used for calling the source data corresponding to the data export task based on the data source information;

the library file acquisition module is used for acquiring a format conversion library file corresponding to the data export format;

the format conversion module is used for carrying out format conversion processing on the source data based on the format conversion library file and generating export data conforming to the data export format;

And the data transmitting module is used for transmitting the exported data to at least one target machine corresponding to the data exported address according to the data exported address.

In another aspect, embodiments of the present application provide a computer device, where the computer device includes a processor and a memory, where at least one instruction, at least one program, a code set, or an instruction set is stored in the memory, where the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the data processing method described above.

In yet another aspect, embodiments of the present application provide a computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, the at least one instruction, the at least one program, the set of codes, or the set of instructions being loaded and executed by a processor to implement the above-described data processing method.

In yet another aspect, embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-described data processing method.

The technical scheme provided by the embodiment of the application can bring the following beneficial effects:

the source data corresponding to the data export task is called through the configuration information of the data export task, the format conversion of the source data is supported, the export data conforming to the preset format is obtained, finally the export data can be automatically sent through the export address, the data export of multiple formats can be supported, the method and the device are applicable to the business scene of multiple data exports, a data export system is not required to be customized, the data export cost is reduced, and the data export efficiency is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of an application execution environment provided by one embodiment of the present application;

FIG. 2 is a flow chart of a data processing method provided by one embodiment of the present application;

FIG. 3 illustrates a schematic diagram of a data export flow in a data export system;

FIG. 4 illustrates a schematic diagram of a data flow for processing by the data processing module;

FIG. 5 is a flow chart of a data processing method provided by one embodiment of the present application;

FIG. 6 illustrates a schematic technical architecture of a data export scheme;

FIG. 7 is a flow chart of a data processing method provided by one embodiment of the present application;

FIG. 8 illustrates a schematic diagram of a data export system;

FIG. 9 is a block diagram of a data processing apparatus provided in one embodiment of the present application;

FIG. 10 is a block diagram of a computer device according to one embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

The embodiment of the application relates to the technical field of cloud and blockchain, and is briefly described below:

cloud technology (Cloud technology): the cloud computing business model application-based network technology, information technology, integration technology, management platform technology, application technology and the like can be collectively called to form a resource pool, and the resource pool is flexible and convenient as required. Cloud computing technology will become an important support. Background services of technical networking systems require a large amount of computing, storage resources, such as video websites, picture-like websites, and more portals. Along with the high development and application of the internet industry, each article possibly has an own identification mark in the future, the identification mark needs to be transmitted to a background system for logic processing, data with different levels can be processed separately, and various industry data needs strong system rear shield support and can be realized only through cloud computing.

Cloud computing (Cloud computing): refers to the delivery and usage mode of the IT infrastructure, meaning that the required resources are obtained in an on-demand, easily scalable manner through the network; generalized cloud computing refers to the delivery and usage patterns of services, meaning that the required services are obtained in an on-demand, easily scalable manner over a network. Such services may be IT, software, internet related, or other services. Cloud Computing is a product of fusion of traditional computer and network technology developments such as Grid Computing (Grid Computing), distributed Computing (Distributed Computing), parallel Computing (Parallel Computing), utility Computing (Utility Computing), network storage (Network StorageTechnologies), virtualization (Virtualization), load balancing (Load balancing), and the like.

Blockchain (Blockchain) is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanisms, encryption algorithms, and the like. The blockchain is essentially a decentralised database, which is a series of data blocks generated by cryptographic methods, each data block containing a batch of information of network transactions for verifying the validity (anti-counterfeiting) of the information and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, and an application services layer. In a narrow sense, a blockchain is a chain data structure that combines blocks of data in a sequential manner in time order, and cryptographically guaranteed, non-tamperable and non-counterfeitable, distributed ledgers, i.e., the data in the blockchain will be irreversible once recorded.

Referring to fig. 1, a schematic diagram of an application running environment provided in one embodiment of the present application is shown. The application execution environment may include: a terminal 10 and a server 20.

The terminal 10 may be an electronic device such as a cell phone, tablet computer, game console, electronic book reader, multimedia playing device, wearable device, PC (Personal Computer ) or the like. A client in which an application program can be installed in the terminal 10.

In the embodiment of the present application, the application may be any application capable of generating data. Typically, the application is a gaming application such as a multiplayer online tactical game (Multiplayer Online Battle Arena, MOBA) game, a big escape survival (BR) game, a Third-party shooting game (TPS), a First-party shooting game (First-Person Shooting Game, FPS), and a multiplayer gunfight survival game, among others. Of course, data may be generated in other types of applications in addition to game applications. For example, virtual Reality (VR) class applications, augmented Reality (Augmented Reality, AR) class applications, three-dimensional map applications, social class applications, interactive entertainment class applications, statistics class applications, business management class applications, and the like, to which embodiments of the present application are not limited. In addition, the data generated by the application program may be different from one application program to another, and the corresponding data characteristics and data attributes may be different from one application program to another, which is not limited in the embodiments of the present application. Optionally, a client of the above application program is running in the terminal 10. In some embodiments, the application may upload user data to the server 20 or derive user-desired data from the server 20.

The server 20 is used to provide background services for the terminal 10. For example, the server 20 may be a background server of an application installed in the terminal 10. The server 20 may be a server, a server cluster comprising a plurality of servers, or a cloud computing service center. Alternatively, the server 20 provides background services for applications in a plurality of terminals 10 at the same time. The server 20 may store a large amount of data, which is generally accumulated through a transaction system, an internet service, an internet of things terminal, and the like. The data collection in the server 20 can be generally divided into an online collection and an offline collection. On-line acquisition refers to directly monitoring changes in the data source, acquiring the new data generated in real-time or near real-time, and loading the new data into the data system. The loading process may be in a "push" mode, i.e., the data source (which may be stored in the terminal 10) actively writes the data to the system, or in a "pull" mode, i.e., the data distribution service (which may be running in the server 20) actively looks at the data changes and obtains the data, facilitating real-time analysis of the data. Offline acquisition refers to a manner in which data is uploaded from a data source (which may be stored in terminal 10) to a large data system (which may be run in server 20) at regular intervals, which has less impact on the production system (i.e., the system that generates the data) and is less difficult to implement.

Alternatively, the terminal 10 and the server 20 may communicate with each other via the network 30.

The method embodiments provided in the present application will be described in detail below. First, terms that may be involved in the method embodiments provided in the present application are briefly described to facilitate understanding by those skilled in the art.

A Database (DB) generally refers to a collection of data information, and may also be considered a software container or repository that stores and organizes information data according to a data structure. The database and its management software constitute a database management system (Database Management System, DBMS) to realize the functions of data management and use. The Database management System and the software and hardware environment, operators, manual documents and other contents of the Database management System and the operators form a complete System, which is called a Database System (DBS). Databases typically include "relational databases" and "non-relational databases" (nosqls), both of which are actually specific forms of database management systems, used to manage data of different characteristics, and used to support different business logic.

The data model is an abstraction of the real world. The real world is defined as entities, attributes and associations in the relational data model, and there are mainly the following abstractions. (1) Entity (Entity): refers to concrete or abstract things in the real world. For example: a student, a teacher, a course. (2) Entity Set (Entity Set): a group of entities having the same characteristics constitutes a set of entities. For example: students, teachers, and courses. Entity sets can be divided into strong and weak entity sets. (3) Entity Type (Entity Type): is a collection of physical features that have common elements. (4) Attribute (Attribute): refers to describing characteristics of an entity, such as a student's number, class, name, etc. Attributes generally require atomicity, i.e., no repartition. Attributes have both value range and data type characteristics. (5) entity identifier: an attribute capable of uniquely identifying an entity is referred to as an entity identifier, such as the student's number, i.e., the concept of a key in a database implementation. An entity set is a strong entity set if all identifier sources in the entity set are independent, and a weak entity set if part of the identifiers in the entity set are from (are attributes of) other entity sets. (6) contact (Relation): refers to relationships between entities, and relationships between attributes within an entity. Relationships, the concept of a data table. A Tuple (Tuple) can be seen as a set of entity attributes, i.e. a row in a data table. Column, attribute of corresponding entity or contact, has atomicity, attribute element in column has same type and value Domain (Domain).

The Key-Value pair mode is also called Key-Value mode. In this data structure, each actual row in the data table has only two basic contents, a row Key (Key) and a Value (Value). The value may be considered as a single memory area, possibly of any type, or even an array. In an actual software implementation, information such as a timestamp, a column name, etc. may be stored, that is, each value may have a different column name, and the values corresponding to different keys may be completely different contents (completely different columns). Therefore, the structure of the table (columns included in the table, value ranges thereof, etc.) cannot be designed in advance, that is, the table of such a key-value pattern is unstructured. When applied, the rows of the same row key are considered to belong to the same logical row (a tuple-like concept). The key-value mode is suitable for quickly positioning data according to keys, and can also realize faster data positioning by sequencing and partitioning the keys.

Extensible markup language (Extensible Markup Language, XML), a subset of standard generic markup language, abbreviated XML. Is a markup language for marking electronic documents to be structured. In an electronic computer, a mark refers to an information symbol that the computer can understand, and by such mark, various information such as articles and the like can be processed between the computers. It can be used to mark data, define data types, and is a source language that allows users to define their own mark-up language. It is well suited for web transport, providing a unified approach to describe and exchange structured data independent of applications or vendors.

JSON (Java Script Object Notation) is a lightweight data exchange language. The method for organizing the data by JSON is independent of language, has self-descriptive property, is simpler and has less strict requirements on the structure. The elements in JSON can be regarded as a description of key-value pairs in the manner ": "interval, with keys in front and values in back, the keys need to be included with double quotation marks. It is generally considered that describing the same data structure, JSON is more compact than XML, and storage and processing efficiency of JSON is higher. JSON supports some simple data types and is therefore more convenient when describing data. JSON does not reserve words and does not require a strict tree structure. JSON data can be parsed very conveniently with JavaScript, python and other common high-level languages.

Library files generally refer to a class of files on a computer, two types, one being static library files and the other being dynamically linked library (Dynamic Link Library, DLL) files, also known as dynamically linked library files. The distinction between static and dynamic libraries is: the static library is copied into the program in the linking stage of the program; the dynamic library is not copied into the program during the linking phase, but rather the program is dynamically loaded into memory by the system for program invocation at runtime. The advantage of using dynamic libraries is that the system only needs to load the dynamic library once, and different programs can obtain copies of the same dynamic library in the memory, so that a lot of memory is saved, and the use of the dynamic library is convenient for modularized updating programs. Many applications loaded on an operating system are not a complete executable file, they are partitioned into relatively independent dynamically linked library files that are placed in the system. When a program is executed, the corresponding DLL file is called. One application may use multiple DLL files, one DLL file may also be used by a different application, such DLL files are referred to as shared DLL files. The library file may simply be viewed as a code repository that provides the user with a number of variables, functions or classes that may be directly accessed.

Referring to fig. 2, a flowchart of a data processing method according to an embodiment of the present application is shown. The method can be applied to a computer device, wherein the computer device is an electronic device with data computing and processing capabilities, and the execution subject of each step can be the server 20 in the application running environment shown in fig. 1. The method may include the following steps (210-240).

Step 210, obtain configuration information of the data export task.

The data export task is a task of exporting part or all of the data in the stored data according to the user demand. The configuration information of the data export task is used to indicate the specific task content of the data export task, and can be simply understood that the configuration information specifies what data is exported from where, what data is exported, and what way the data is exported. The user can set the relevant configuration information to complete the setting of the data export task.

Optionally, the configuration information includes data source information, a data export format, and a data export address. Optionally, the data source information includes a source database, a source data table, or some or all of source data in the source data table corresponding to the data export task. The data source information is used to indicate the location from which the data source was derived. In the case that the data source indicated by the data source information is a database, the database is referred to as a source database of the data deriving task. And when the data source indicated by the data source information is a data table, the data table is called as a source data table of the data deriving task.

Step 220, retrieving source data corresponding to the data export task based on the data source information.

Optionally, in the case that the data source is a database, source data corresponding to the data export task is invoked from the source database. Optionally, in the case that the data source is a data table, source data corresponding to the data export task is invoked from the source data table.

In a possible implementation manner, the configuration information further comprises a data filtering condition, wherein the data filtering condition is used for filtering data in the data source, and excluding data irrelevant to the data export task. The above step 220 may also be performed in the following manner.

Determining a data source of the data export task based on the data source information; and filtering the data in the data source according to the data filtering condition to obtain source data corresponding to the data export task.

In step 230, a format conversion library file corresponding to the data export format is obtained.

The data export format has a corresponding relation with the format conversion library file, and the format conversion library file comprises variables, functions or classes for format conversion and is used for converting source data into a data format corresponding to the format conversion library file for export.

Optionally, a format conversion library file corresponding to the data export format is retrieved from memory according to the data export format to perform a subsequent format conversion process.

The format conversion library file includes, but is not limited to, a universal format conversion library file and a custom format conversion library file. The universal format conversion library file can be a library file which is deployed in advance in a terminal or a server, and is convenient to use in a conventional application scene. The format conversion library file may be a custom format conversion library file, and may be uploaded to a terminal or a server by a user.

And under the condition that the data export format is a custom data format, acquiring a custom format conversion library file corresponding to the custom data format. The custom data format refers to a private data format formulated for the target application scenario.

And when the data export format is the universal data format, acquiring a universal format conversion library file corresponding to the universal data format. The above-mentioned general data formats include conventional data formats such as JSON, XML.

Step 240, performing format conversion processing on the source data based on the format conversion library file, and generating export data conforming to the data export format.

The format conversion process described above is used to convert data from one format to another. Optionally, the source data is subjected to format conversion processing according to a format conversion function, and derived data conforming to a data derived format is generated. The format conversion function reflects the mapping relation between the two data formats before and after conversion.

And under the condition that the data export format is the custom data format, carrying out format conversion processing on the source data based on the custom format conversion library file to generate export data conforming to the data export format.

When the data export format is a general data format, format conversion processing is performed on the source data based on the general format conversion library file, and export data conforming to the data export format is generated.

In an exemplary embodiment, format unification processing is performed on the source data, so as to obtain source data with unified format. Typically, the data in the database is often encoded and compressed data, and sometimes encrypted data, and the format unification process is used to convert the format of the source data into a standard format, where the standard format refers to a data format that can be processed by a standard interface in the memory. Correspondingly, format conversion processing is carried out on the source data with the unified format, and export data conforming to the data export format is generated.

In an exemplary embodiment, the data export format comprises a custom data format, and step 240 may be implemented as follows.

Acquiring a custom format conversion library file corresponding to a custom data format; and carrying out format conversion processing on the source data based on the custom format conversion library file to generate derived data conforming to the custom data format.

In the data processing method provided in the embodiment of the present application, the user may customize in what format the data is derived, and the data format defined by the user is also referred to herein as a custom data format. The custom data format may be customized according to the business scenario of the user. The user can upload a custom format conversion library file customized by the user to the data export system, wherein the custom format conversion library file comprises variables, functions or classes for format conversion and is used for converting source data into a custom data format for export.

Alternatively, the custom format conversion library file is stored in a server of the data export system, and may be obtained by obtaining the custom format conversion library file from the server when the custom format conversion library file is needed.

The data export system calls related format conversion functions in the custom format conversion library file, performs format conversion processing on the source data, and generates export data in a custom data format.

In an exemplary embodiment, field identification processing is performed on the source data, so as to obtain source data after field identification.

In general, the source data has low readability, and the user cannot intuitively see the content which the user wants to know from the source data. And therefore, performing field identification processing on all fields in the data record of the source data to identify the field type. Optionally, the data record includes multiple levels of fields therein. For example, an address book entry in which a plurality of mail, telephone numbers, and addresses are nested and recorded, and the number of entries is not certain. The structure of mail, phone and address is nested, and by field identification, it can be determined that the field type in the entry, such as "xxx@xx.com", is mail.

Optionally, field identification processing is performed on the source data with the unified format to obtain source data with the field identified.

In one possible implementation manner, the source data after field identification comprises an identification result of a field type, so that the data is conveniently exported into a visualized text in the subsequent export, the readability of the exported data is improved, and the user can conveniently use the exported data.

Correspondingly, format conversion processing is carried out on the source data after field identification, and the textualized export data conforming to the data export format is generated.

Optionally, the derived data comprises textual derived data. The text data refers to data checking in a text mode, so that data attributes and data values can be visually seen, and conclusion proof, state display and auxiliary decision making are facilitated. Optionally, the export data comprises an export data table. Alternatively, the export data table can be viewed directly in text. The textual export data has high readability, and the data attribute of each data is marked in the textual export data. The user can intuitively see the meaning represented by the data through the labels.

Step 250, according to the data export address, sending export data to at least one target machine corresponding to the data export address.

The data export address is used to indicate the address of the destination of the data export, and the data export address may be a network protocol address, a port address, or a storage address corresponding to a folder.

Optionally, the export data is sent to the data export address via network transmission. Optionally, the export data is sent by way of a copy of the data. The embodiment of the application does not limit the sending mode of the derived data, and can select and adjust according to the specific service scene derived from the data.

In one example, as shown in FIG. 3, a schematic diagram of a data export flow in a data export system is illustratively shown. The management node 31 is a node for managing the entire database export system, the storage node is a node for actually storing user data, and includes a main storage node 32 and a backup storage node 34, and the resource configuration library 33 includes a dynamic link library or an executable file written by a user, and configuration information of each data export task. The management node 31 manages export plans for all data tables. By the time data export is required, the management node 31 first notifies the corresponding backup storage node 34 to prepare data, and the data source may be either real-time user data or current day's backup data. The real-time data requires a certain processing time, the preparation time of the cold standby data is faster, and what data to use is specified by the user when the task is exported by the configuration data. Then, after receiving the instruction from the management node 31, the data processing module 35 obtains the full-capacity backup data 36, and performs data processing according to the user-specified mode to derive data in a custom format, or directly performs data generic format derivation. In addition, the data processing module 35 can refine all the fields in the data table, including multi-level fields, not only identify the field type, but also perform various computation processes on the data to generate the textual export data.

In one example, as shown in FIG. 4, a data processing module processing data flow diagram is schematically illustrated. The data processing module 41 obtains the full-capacity cold-standby data 42, and traverses each record in the full-capacity cold-standby data 42 to convert the obtained cold-standby data into a data format which can be processed through a standard interface in the memory. For example, the decoding process or the decryption process is performed on the backup data. The data processing module 41 may directly locate a plurality of fields in a piece of data by calling an open interface, and the function may process the data in the fields, for example, perform data general mathematical operations, perform data filtering, perform string processing, etc. The user may add a custom format conversion library file to the user custom library 43, and the data processing module may perform data format conversion by calling a function in the library file, and finally export the data in a data format that the user can understand, i.e. according to the custom data format. In addition, the data processing module 41 may also export data into a generic data format by calling a library file in a standard library of the data export system. Common export formats include JSON, BSON (Binary Serialized Document Format ), base64 (representing binary data based on 64 printable characters), XML, CSV (Comma-Separated Values), and the like. Finally, the exported data needs to be transmitted to the user machine, and can be exported in an automatic copying mode, such as a remote file copying command in a Linux system; automatic uploading mode export is also supported, for example, various transmission modes such as uploading interfaces of cloud storage are used. In addition, the export scheme supports the transmission of one copy of data to multiple user machines, and also supports the sharing of one copy of data to multiple user machines for storage, since one copy of data is made up of many files, and it is highly likely that one user machine will not have all of these files present. The scheme supports a user to provide a plurality of machines, and the data export system can equally transmit the files to the user machines.

In summary, according to the technical scheme provided by the embodiment of the application, the source data corresponding to the data export task is called through the configuration information of the data export task, the format conversion of the source data is supported, the export data conforming to the preset format is obtained, finally the export data can be automatically sent through the export address, the data export of multiple formats can be supported, the method and the device can be applied to various data export business scenes, the data export system is not required to be customized, the data export cost is reduced, and the data export efficiency is improved.

In addition, the technical scheme provided by the embodiment of the application can perform format conversion processing on the data according to the user-defined library file, support the data of the user-defined format to be exported, meet the personalized custom data export scene, and effectively reduce the labor cost and the equipment cost of data export without independently designing a data export system by a user.

In addition, the technical scheme provided by the embodiment of the application can be used for carrying out field identification on the source data to obtain the field type corresponding to the data, so that the text data is exported, the readability of the exported data is improved, and the application of a user is facilitated.

Referring to fig. 5, a flowchart of a data processing method according to an embodiment of the present application is shown. The method can be applied to a computer device, wherein the computer device is an electronic device with data computing and processing capabilities, and the execution subject of each step can be the server 20 in the application running environment shown in fig. 1. The method may include the following steps (510-590).

Step 510, in response to a setting operation for the data export task, generates configuration information for the data export task.

The user can set the data export task in the front-end interactive interface or the client corresponding to the data export system, for example, set parameters such as the data source position, the data source name, the data source filtering condition and the like, and the data export system can generate configuration information of the export task according to the configuration parameters set by the user.

Optionally, the setting operation includes at least one of:

setting a source database corresponding to a data export task; setting a source data table corresponding to a data export task; setting a data export time corresponding to the data export task; setting data filtering conditions corresponding to data export tasks; setting a data export address corresponding to the data export task; setting a data export format corresponding to the data export task; setting operation of library files corresponding to data export tasks; setting an export period corresponding to the data export task; and (5) setting.

Optionally, the configuration information of the data export task includes at least one of:

source database identification, source data table identification, data export time, data filtering conditions, data export format, library file identification and export period.

Step 520, obtain configuration information of the data export task.

Optionally, the configuration information further comprises a library file identification for determining a unique library file in the data export system. Optionally, the configuration information includes a data-derived time.

Step 530, retrieving source data corresponding to the data export task based on the data source information.

Optionally, format unification processing is performed on the source data, so as to obtain source data with unified format. The procedure of the format unification process is described in the above embodiment, and will not be described here again.

In an exemplary embodiment, the above-described step 530 may be implemented as follows.

If the data export time is reached, acquiring cold standby data; and calling the source data corresponding to the data export task from the cold standby data based on the data source information.

And starting a data export process according to the preset data export time in the data export task. Optionally, the backup data is generated based on backup data of the source database, and the backup data of the source database can be kept consistent with the original data, so that real-time data can be obtained based on the backup data of the source database, real-time service of the database is not affected, and database service resources of a server where the original data is located are not occupied.

The cold standby data refers to a complete data derived when the database is shut down. Alternatively, the cold standby data may be provided by a backup storage node in the data export system. The backup data may be data collected from backup storage data of the original storage data at a certain set time, or data obtained from backup storage data in real time. In one possible implementation manner, when the cold standby node in the data export system does the full amount of cold standby data, the full amount of data file is in a completely static state at the cold standby starting time point, and the full amount of data is backed up by adopting byte copy, so that the problem of consistency is completely avoided. And during the cold standby period, the front-end read-write is not affected at all, the new request will write a small modification set, and the request will combine the full data and the small modification set. Finally, the backup data is generated, and the generation mode of the backup data is not limited in the application.

Optionally, source data in a source database corresponding to the data export task is invoked from the cold standby data. Optionally, the source data in the source data table corresponding to the data export task is invoked from the cold standby data.

Step 540, determining the library file corresponding to the library file identification.

And determining the library file corresponding to the library file identifier according to the library file identifier.

In step 550, in the case that the library file corresponding to the library file identifier is the custom data operation library file, the custom data operation library file is obtained.

And under the condition that the library file corresponding to the library file identifier is the custom data operation library file, acquiring the custom data operation library file from the user custom library.

Step 560, according to the custom data operation library file, performing data operation processing on the source data to obtain the source data after data operation.

The data operation process is used for performing numerical operations on data in the source data, and the operations on the data are completed by means of, for example, merging, intersection, difference, division, projection, selection, cartesian product and the like in relational algebra.

And carrying out data operation processing on the source data according to the custom format conversion function provided in the custom data operation library file to obtain the source data after the data operation.

Optionally, according to the custom data operation library file, performing data operation processing on the source data with the unified format to obtain the source data after data operation.

In step 570, when the library file corresponding to the library file identifier is the standard data operation library file, the data operation processing is performed on the source data according to the standard data operation library file, so as to obtain the source data after the data operation.

And calling the standard data operation library file in the standard library file under the condition that the library file corresponding to the library file identifier is the standard data operation library file, and performing data operation processing on the source data to obtain the source data after data operation.

Optionally, when the library file corresponding to the library file identifier is a standard data operation library file, calling the standard data operation library file in the standard library file, and performing data operation processing on the source data with the unified format to obtain the source data after data operation.

In step 580, format conversion processing is performed on the source data after the data operation, so as to generate export data conforming to the data export format.

Step 590, transmitting the export data according to the data export address.

In an exemplary embodiment, the above step 590 may be implemented as follows.

Determining at least one target machine corresponding to the data export address; the derived data is sent to at least one target machine.

The configuration information comprises at least one data export address, and the target machine corresponding to each data export address is determined.

In one possible implementation, the export data may also be data files of other open source databases. The data in the databases are directly exported as the data files of other open source databases, so that the data can be directly loaded and used by other open source databases, and the migration speed of the data from one database to another database is greatly increased. Meanwhile, the read-write interfaces of other databases can be called while data export operation is performed, the data exported in real time can be directly written into the target database, the data can be quickly migrated from one database to another database, and the data migration efficiency is improved.

Optionally, a complete piece of export data is sent to at least one target machine.

In an exemplary embodiment, the above step 590 may also be implemented as follows.

Determining at least one target machine corresponding to the data export address; decomposing the export data to generate at least one part of decomposed export data; and transmitting at least one part of decomposed derived data to at least one target machine.

Optionally, according to the number of the target machines, the export data is equally divided, and a complete export data is decomposed into export data decomposition files with the same number as the target machines.

And sending each derived data decomposition file to each target machine according to the corresponding relation between the derived data decomposition file and the target machine.

In one example, as shown in fig. 6, a schematic technical architecture diagram of a data export scheme is illustratively shown. The user configures information of the data export task, such as a source data table, export time, data filtering conditions, general format, information of export target machine, etc., on the management page through the browser 61. If the user needs to define the export format by himself or additionally needs to perform additional computation and processing on the exported data, the custom library file may be uploaded to the user custom library 62 of the data export system, and the management node 63 may invoke the custom library file in the user custom library 62 to conduct data export, thereby completing the user-customized data export service application. Alternatively, the user-defined library 62 may also be in the management node 63. Alternatively, the data export task may be persisted to the resource configuration library 64, and all nodes in the overall data export system have backup nodes, which may enable the overall data export scheme to be highly available.

In summary, according to the technical scheme provided by the embodiment of the application, the data operation processing can be performed on the source data to obtain the data required by the user, the data after the user-defined special operation is supported to be exported, the personalized customized data export scene is satisfied, the user does not need to independently design a data export system, and the labor cost and the equipment cost of data export are effectively reduced.

In addition, the data source of the technical scheme provided by the embodiment of the application comes from the cold backup data, the cold backup data is generated by the backup storage data, no influence is caused on the data reading and writing of the main storage system in the whole data export process, the database is not required to be closed for export, and the data export is not required to be specially carried out at the service low peak. And flexible setting of export time and export times is supported, and flexibility of data export tasks is improved.

In addition, the technical scheme provided by the embodiment of the application supports direct transmission of the exported data to the target machine, so that data migration can be realized, a user does not need to reproduce a customized data import tool, and the data export efficiency is effectively improved.

Referring to fig. 7, a flowchart of a data processing method according to an embodiment of the present application is shown. The method can be applied to a computer device, wherein the computer device is an electronic device with data computing and processing capabilities, and the execution subject of each step can be the server 20 in the application running environment shown in fig. 1. The method may include the following steps (701-714).

Step 701, obtaining a data export time corresponding to at least one data export task.

In step 702, when the target data export time is reached, at least one target data export task corresponding to the target data export time is determined.

The target data export time is the data export time corresponding to any data export task.

Step 703, for each target data export task, obtaining configuration information of the target data export task.

Step 704, source data corresponding to the target data export task is invoked from the cold standby data.

Step 705, performing format unification processing on the source data to obtain source data with unified format.

Typically, the data in the database is often encoded and compressed data, and sometimes encrypted data, and the format unification process is used to convert the format of the source data into a standard format, where the standard format refers to a data format that can be processed by a standard interface in the memory.

Step 706, determining a library file corresponding to the library file identification.

The library file includes information such as functions and variables for data processing, and the data can be processed by calling the library file. Optionally, the library file includes a library file of the data export system itself, and further includes a custom library file uploaded by the user. The custom library file is a library file designated by a user according to an actual service export task. In one possible implementation manner, the user may decide to disclose the self-uploaded custom library file to all users for other users, and the user may encrypt the self-uploaded custom library file, which is not limited in this embodiment of the present application.

Step 707, obtaining the custom data operation library file when the library file corresponding to the library file identifier is the custom data operation library file.

Step 708, according to the custom data operation library file, performing data operation processing on the source data with unified format to obtain source data after data operation.

Step 709, in the case that the library file corresponding to the library file identifier is the standard data operation library file, performing data operation processing on the source data with unified format according to the standard data operation library file, to obtain the source data after data operation.

And 710, performing field identification processing on the source data after the data operation to obtain the source data after the field identification.

In step 711, when the data export format is a custom data format, a custom format conversion library file corresponding to the custom data format is obtained.

Step 712, based on the custom format conversion library file, format conversion processing is performed on the source data after field identification, and the textually exported data conforming to the custom data format is generated.

In step 713, in the case where the data export format is the target general data format, format conversion processing is performed on the source data identified by the field, and the textually exported data conforming to the target general data export format is generated.

Step 714, transmitting the textually exported data to at least one target machine corresponding to the data export address according to the data export address.

In summary, according to the technical scheme provided by the embodiment of the application, by monitoring the data export time of each data export task in the data export system, when the preset time is reached, the corresponding data export task is executed, so that the management efficiency of the data export task is improved, parallel processing of multiple export tasks is supported, transparency of the data export process to users is ensured, and the efficiency of data export is improved. Therefore, the technical scheme provided by the embodiment of the application can be used for data text export, and is convenient for service data viewing, service data analysis, service debugging loopholes and other scenes.

In another possible embodiment, shown in fig. 8, a schematic diagram of a data export system is schematically illustrated. All nodes of the data export system are provided with backup nodes, and particularly, whether direct relation export tasks are successfully executed or not is judged for the critical management nodes and storage nodes in the system. In order to realize high availability of the whole export system, all nodes are provided with redundant backups, so that smooth and timed execution of export tasks is ensured, and the situation that data export can be performed on a certain day and data export cannot be performed on a certain day is avoided. The management node is composed of a work management node 81 and a standby (standby) management node 82. The storage nodes consist of a primary storage node 83 and a backup storage node 84. The data processing module 86 also has a backup data processing module 87 corresponding thereto. The job management node 81 is responsible for normal system management job, and the standby management node 82 is not responsible for system management job, but it accepts reports of all storage nodes. For example, the backup storage node 84 discovers that the work management node 81 is down, reports the result to the backup management node 82, and the backup management node 82 connects to the whole storage system, and notifies the work management node 81 to be down, and the backup management node 82 upgrades itself to the work management node, so as to implement the down hot switch. The data source of the data export task is provided by the cold standby data in the backup storage node 84 without affecting the serving primary storage node 83. If one backup storage node 84 is down, the export task may be completed by the same set of additional backup storage nodes 85, ensuring that the export task is completed.

In an exemplary embodiment, in the process of executing the data export task, the resource configuration library for storing the configuration information of the data export task may be down, and in this case, if the configuration information of the data export task is to be obtained, the configuration information of the data export task needs to be obtained from the backup resource configuration library. Therefore, the embodiment of the application further comprises the following steps.

And step A, obtaining backup configuration information of the data export task in response to receiving a switching instruction for the configuration information transmission line.

The backup configuration information and the configuration information are the same in content, and the storage positions of the backup configuration information and the configuration information are different.

In an exemplary embodiment, in the process of executing the data export task, the data export system may be down in the backup storage node for storing the user data, and in this case, if the data source is to be acquired, the backup source data corresponding to the data export task needs to be called to another backup node. Therefore, the embodiment of the application further comprises the following steps.

And step B, in response to receiving a switching instruction for the source data transmission line, calling the backup source data corresponding to the data export task based on the data source information.

The backup source data and the source data have the same content, and the storage positions of the backup source data and the source data are different. Optionally, the source data is stored in a backup storage node and the backup source data is stored in another backup storage node. In one possible implementation, the data source may be stored locally (i.e., the node to which the data was first written, such as the primary storage node), on another node of the own chassis (backup storage node 1), and on a node on another chassis (backup storage node 2).

In an exemplary embodiment, in the process of executing a data export task, a data processing module for executing format conversion may be down, and in this case, if format conversion processing is to be performed on source data, conversion processing is required to be performed on the source data by a backup data processing module. Therefore, the embodiment of the application further comprises the following steps.

And step C, in response to receiving a switching instruction for the format conversion processing process, performing second format conversion processing on the source data, and generating export data conforming to the data export format.

The second format conversion process is the same as the process procedure of the format conversion process, and the second format conversion process is different from the execution area of the format conversion process.

In summary, in order to achieve high availability of the whole export system, the technical scheme provided by the embodiment of the application backs up data and data processing resources in the system, ensures smooth and timed execution of export tasks, and ensures stability and accuracy of data export.

After the scheme is used, the database user can be conveniently liberated from the heavy user data deriving task, and the database helps to complete the function. The functions of configurable export tasks, computable export processes, custom export results, repeatable import of export data, high availability of the whole data export system and the like are realized.

The following are device embodiments of the present application, which may be used to perform method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

Referring now to FIG. 9, shown is a block diagram of a data processing apparatus provided in one embodiment of the present application. The device has the function of realizing the data processing method, and the function can be realized by hardware or can be realized by executing corresponding software by hardware. The device may be a computer device or may be provided in a computer device. The apparatus 900 may include: a configuration acquisition module 910, a data retrieval module 920, a library file acquisition module 930, a format conversion module 940, and a data transmission module 950.

A configuration obtaining module 910, configured to obtain configuration information of the data export task, where the configuration information includes data source information, a data export format, and a data export address.

The data retrieving module 920 is configured to retrieve source data corresponding to the data export task based on the data source information.

A library file acquisition module 930, configured to acquire a format conversion library file corresponding to the data export format.

And a format conversion module 940, configured to perform format conversion processing on the source data, and generate export data that conforms to the data export format.

A data sending module 950, configured to send the derived data according to the data derived address.

In an exemplary embodiment, the data export format comprises a custom data format, and the apparatus 900 further comprises: and a library file receiving module.

And the library file receiving module is used for receiving the custom format conversion library file corresponding to the custom data format, which is uploaded by the terminal.

The format conversion module 940 is configured to:

and carrying out format conversion processing on the source data based on the custom format conversion library file to generate export data conforming to the custom data format.

In an exemplary embodiment, the configuration information further includes a library file identification, and the library file acquisition module 930 is further configured to:

and under the condition that the library file corresponding to the library file identifier is a custom data operation library file, acquiring the custom data operation library file.

The apparatus 900 further comprises: and the data operation module.

And the data operation module is used for carrying out data operation processing on the source data according to the custom data operation library file to obtain source data after data operation.

The format conversion module 940 is further configured to:

and carrying out format conversion processing on the source data after the data operation to generate export data conforming to the data export format.

In an exemplary embodiment, the data operation module is further configured to:

and under the condition that the library file corresponding to the library file identifier is a standard data operation library file, performing data operation processing on the source data according to the standard data operation library file to obtain source data after data operation.

The format conversion module 940 is further configured to:

In an exemplary embodiment, the apparatus 900 further comprises: and a field identification module.

And the field identification module is used for carrying out field identification processing on the source data to obtain source data after field identification.

The export data includes textual export data, and the format conversion module 940 is further configured to:

and carrying out format conversion processing on the source data identified by the field to generate text export data conforming to the data export format.

In an exemplary embodiment, the data sending module 950 is further configured to:

determining at least one target machine corresponding to the data export address;

decomposing the export data to generate at least one part of decomposed export data;

and sending the at least one piece of decomposed derived data to the at least one target machine.

In an exemplary embodiment, the configuration information includes a data export moment, and the data retrieval module 920 is configured to:

if the data export time is reached, acquiring cold standby data;

and calling source data corresponding to the data export task from the cold standby data based on the data source information.

In an exemplary embodiment, the apparatus 900 further comprises: and a configuration generation module.

A configuration generating module, configured to generate configuration information of the data export task in response to a setting operation for the data export task, where the setting operation includes at least one of:

setting a source database corresponding to the data export task;

setting a source data table corresponding to the data export task;

setting a data export time corresponding to the data export task;

setting data filtering conditions corresponding to the data export task;

setting a data export address corresponding to the data export task;

setting a data export format corresponding to the data export task;

and setting the operation of the library file corresponding to the data export task.

In an exemplary embodiment, the apparatus 900 further comprises:

and the time acquisition module is used for acquiring the data export time corresponding to at least one data export task.

And the task determining module is used for determining at least one target data export task corresponding to the target data export time when the target data export time is reached, wherein the target data export time is the data export time corresponding to any data export task.

And the task execution module is used for exporting tasks for each target data, and the task execution module starts to execute the steps of exporting the configuration information of the tasks from the acquired data.

Referring to fig. 10, a block diagram of a computer device according to an embodiment of the present application is shown. The computer device may be a server for performing the data processing method described above. Specifically, the present invention relates to a method for manufacturing a semiconductor device.

The computer apparatus 1000 includes a central processing unit (Central Processing Unit, CPU) 1001, a system Memory 1004 including a random access Memory (Random Access Memory, RAM) 1002 and a Read Only Memory (ROM) 1003, and a system bus 1005 connecting the system Memory 1004 and the central processing unit 1001. Computer device 1000 also includes a basic Input/Output system (I/O) 1006, which helps to transfer information between various devices within the computer, and a mass storage device 1007 for storing an operating system 1013, application programs 1014, and other program modules 1012.

The basic input/output system 1006 includes a display 1008 for displaying information and an input device 1009, such as a mouse, keyboard, etc., for the user to enter information. Wherein the display 1008 and the input device 1009 are connected to the central processing unit 1001 through an input output controller 1010 connected to a system bus 1005. The basic input/output system 1006 may also include an input/output controller 1010 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input output controller 1010 also provides output to a display screen, a printer, or other type of output device.

The mass storage device 1007 is connected to the central processing unit 1001 through a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable media provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include a computer readable medium (not shown) such as a hard disk or CD-ROM (Compact Disc Read-Only Memory) drive.

Computer readable media may include computer storage media and communication media without loss of generality. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory, electrically erasable programmable read-only memory), flash memory or other solid state memory technology, CD-ROM, DVD (Digital Video Disc, high density digital video disc) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the ones described above. The system memory 1004 and mass storage devices 1007 described above may be collectively referred to as memory.

According to various embodiments of the present application, the computer device 1000 may also operate by being connected to a remote computer on a network, such as the Internet. I.e., the computer device 1000 may be connected to the network 1012 through a network interface unit 1011 connected to the system bus 1005, or other types of networks or remote computer systems (not shown) may be connected using the network interface unit 1011.

The memory also includes a computer program stored in the memory and configured to be executed by the one or more processors to implement the data processing method described above.

In an exemplary embodiment, a computer readable storage medium is also provided, in which at least one instruction, at least one program, a set of codes or a set of instructions is stored, which when executed by a processor, implement the above-mentioned data processing method.

Alternatively, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory ), SSD (Solid State Drives, solid state disk), or optical disk, etc. The random access memory may include ReRAM (Resistance Random Access Memory, resistive random access memory) and DRAM (Dynamic Random Access Memory ), among others.

In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-described data processing method.

It should be understood that references herein to "a plurality" are to two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship. In addition, the step numbers described herein are merely exemplary of one possible execution sequence among steps, and in some other embodiments, the steps may be executed out of the order of numbers, such as two differently numbered steps being executed simultaneously, or two differently numbered steps being executed in an order opposite to that shown, which is not limited by the embodiments of the present application.

The foregoing description of the exemplary embodiments of the present application is not intended to limit the invention to the particular embodiments disclosed, but on the contrary, the intention is to cover all modifications, equivalents, alternatives, and alternatives falling within the spirit and scope of the invention.

Claims

1. A method of data processing, the method comprising:

transmitting the export data to at least one target machine corresponding to the data export address according to the data export address;

wherein the configuration information further includes a library file identification, the method further comprising:

acquiring the custom data operation library file under the condition that the library file corresponding to the library file identifier is the custom data operation library file;

According to the custom data operation library file, carrying out data operation processing on the source data to obtain source data after data operation;

the format conversion processing is performed on the source data to generate export data conforming to the data export format, including:

2. The method of claim 1, wherein the data export format comprises a custom data format, the method further comprising:

receiving a custom format conversion library file corresponding to the custom data format uploaded by a terminal;

the step of performing format conversion processing on the source data based on the format conversion library file to generate export data conforming to the data export format comprises the following steps:

3. The method according to claim 1, wherein the method further comprises:

under the condition that the library file corresponding to the library file identifier is a standard data operation library file, carrying out data operation processing on the source data according to the standard data operation library file to obtain source data after data operation;

4. The method according to claim 1, wherein the method further comprises:

performing field identification processing on the source data to obtain field-identified source data;

the export data includes textual export data, the performing format conversion processing on the source data to generate export data conforming to the data export format includes:

5. The method according to any of claims 1 to 4, wherein said sending said derived data to at least one target machine corresponding to said data derived address based on said data derived address comprises:

6. The method according to any one of claims 1 to 4, wherein the configuration information includes a data export time, and the retrieving source data corresponding to the data export task based on the data source information includes:

if the data export time is reached, acquiring cold standby data;

7. The method according to any one of claims 1 to 4, further comprising:

generating configuration information of the data export task in response to a setting operation for the data export task, the setting operation comprising at least one of:

setting a source database corresponding to the data export task;

setting a source data table corresponding to the data export task;

setting a data export time corresponding to the data export task;

setting data filtering conditions corresponding to the data export task;

setting a data export address corresponding to the data export task;

Setting a data export format corresponding to the data export task;

8. A data processing apparatus, the apparatus comprising:

a data transmitting module, configured to transmit the derived data to at least one target machine corresponding to the data derived address according to the data derived address;

wherein the configuration information further includes a library file identifier, and the library file acquisition module is further configured to: acquiring the custom data operation library file under the condition that the library file corresponding to the library file identifier is the custom data operation library file;

The apparatus further comprises: the data operation module is used for carrying out data operation processing on the source data according to the custom data operation library file to obtain source data after data operation;

the format conversion module is further configured to: and carrying out format conversion processing on the source data after the data operation to generate export data conforming to the data export format.

9. The apparatus of claim 8, wherein the data export format comprises a custom data format, the apparatus further comprising: a library file receiving module;

the library file receiving module is used for receiving a custom format conversion library file corresponding to the custom data format, which is uploaded by a terminal;

the format conversion module is used for carrying out format conversion processing on the source data based on the custom format conversion library file to generate export data conforming to the custom data format.

10. The apparatus of claim 8, wherein the data operation module is further configured to perform data operation processing on the source data according to the standard data operation library file to obtain source data after data operation if the library file corresponding to the library file identifier is the standard data operation library file;

The format conversion module is further used for carrying out format conversion processing on the source data after the data operation to generate export data conforming to the data export format.

11. The apparatus of claim 8, wherein the apparatus further comprises: a field identification module;

the field identification module is used for carrying out field identification processing on the source data to obtain source data after field identification;

the export data comprises text export data, and the format conversion module is further used for performing format conversion processing on the source data identified by the field to generate text export data conforming to the data export format.

12. The apparatus according to any of claims 8-11, wherein the data transmission module is further configured to determine at least one target machine to which the data export address corresponds; decomposing the export data to generate at least one part of decomposed export data; and sending the at least one piece of decomposed derived data to the at least one target machine.

13. The apparatus according to any one of claims 8-11, wherein the configuration information includes a data export moment, and the data retrieval module is configured to obtain the cold standby data if the data export moment is reached; and calling source data corresponding to the data export task from the cold standby data based on the data source information.

14. The apparatus according to any one of claims 8-11, wherein the apparatus further comprises: a configuration generation module;

the configuration generating module is used for responding to the setting operation of the data export task and generating the configuration information of the data export task, and the setting operation comprises at least one of the following steps:

setting a source database corresponding to the data export task;

setting a source data table corresponding to the data export task;

setting a data export time corresponding to the data export task;

setting data filtering conditions corresponding to the data export task;

setting a data export address corresponding to the data export task;

setting a data export format corresponding to the data export task;

15. A computer device comprising a processor and a memory having stored therein at least one instruction, at least one program, code set or instruction set, the at least one instruction, at least one program, code set or instruction set being loaded and executed by the processor to implement the data processing method of any of claims 1 to 7.

16. A computer readable storage medium having stored therein at least one instruction, at least one program, code set, or instruction set, the at least one instruction, the at least one program, the code set, or instruction set being loaded and executed by a processor to implement the data-based processing method of any of claims 1 to 7.

17. A computer program product, characterized in that it comprises computer instructions stored in a computer-readable storage medium, from which computer instructions a processor of a computer device reads, which processor executes the computer instructions, so that the computer device performs the data-based processing method according to any of claims 1 to 7.