CN114201538A

CN114201538A - Data processing method and device, intelligent equipment and computer storage medium

Info

Publication number: CN114201538A
Application number: CN202010910100.3A
Authority: CN
Inventors: 陈瑶
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-09-02
Filing date: 2020-09-02
Publication date: 2022-03-18

Abstract

The embodiment of the application discloses a data processing method, a data processing device, intelligent equipment and a computer storage medium, wherein the method comprises the following steps: analyzing the acquired data construction task, obtaining source data from source data environment equipment by executing a data selection script associated with the data construction task, storing files in the source data environment equipment, storing the source data in various databases, facilitating management and reading, and adding logs related to storage and reading of each data in the source data if necessary; loading target data corresponding to the source data into a target data environment device by executing a data loading script associated with the data construction task; and feeding back a notification message about the target data loaded to the target data environment equipment to the management platform so as to process the target data in the target data environment equipment. By the method and the device, data construction time of a user is saved, and automation requirements of data construction are met.

Description

Data processing method and device, intelligent equipment and computer storage medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an intelligent device, and a computer storage medium.

Background

With the development of computer technology and communication technology, people have entered into the digital society, and people generate and utilize various data in daily work, life, and study. Various schemes and products related to functions of data acquisition, storage, management, transmission and the like are widely applied to various industries. Meanwhile, mass data can be stored in the cloud server through a cloud technology, and the data stored in the cloud server can be applied to industries such as medical treatment, finance, banking, government affairs, games, internet of things and the like.

For scattered data, the data can be converted into a usable form by means of data construction, such as data construction by means of a data table. The purpose of constructing data through different data constructing modes is to meet the requirements of different users on the characteristic data. In the process of constructing data for some new data requirements, the existing technical scheme is that a user is required to manually select data from a data storage environment and then perform a series of manual operations such as data copying and unloading, which consumes time and is low in efficiency.

Disclosure of Invention

The embodiment of the application provides a data processing method and device, intelligent equipment and a computer storage medium, which can automatically select data from a source data environment and load the data into a target data environment based on a script.

In one aspect, an embodiment of the present application provides a data processing method, including:

acquiring a data construction task, wherein the data construction task is generated according to a detected data construction event;

analyzing the data construction task, acquiring a data selection script associated with the data construction task, and executing the data selection script to obtain source data, wherein the data selection script is used for acquiring the source data from source data environment equipment;

acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is obtained according to the source data;

and feeding back a notification message about the target data loaded to the target data environment equipment to a management platform, wherein the notification message is used for instructing the management platform to respond to the data construction event and process the target data in the target data environment equipment.

On the other hand, an embodiment of the present application further provides a data processing apparatus, including:

the data construction system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a data construction task, and the data construction task is generated according to a detected data construction event;

the processing module is used for analyzing the data construction task, acquiring a data selection script related to the data construction task, and executing the data selection script to obtain source data, wherein the data selection script is used for acquiring the source data from source data environment equipment; acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is obtained according to the source data;

and the notification module is used for feeding back a notification message about that the target data is loaded to the target data environment equipment to the management platform, wherein the notification message is used for indicating the management platform to respond to the data construction event and process the target data in the target data environment equipment.

Correspondingly, an embodiment of the present application further provides an intelligent device, which is characterized in that the intelligent device includes a storage device and a processor, wherein a computer program is stored in the storage device, and the processor calls the computer program to implement the data processing method.

Accordingly, the embodiment of the present application further provides a computer storage medium, in which a computer program is stored, and the data processing method described above is implemented in the process of being executed by a processor.

According to the data loading method and device, the data selection script is automatically operated, part of required data can be obtained from the source data environment equipment, the obtained data can be automatically loaded into the target environment through the data loading script, the automation requirement of data construction can be met, and the personalized data requirement of a user can be met. In the whole process, a user only needs to compile a part of scripts and configure the scripts to the data construction task, the data construction task is executed through the application corresponding to the application, automatic selection and loading of the data can be achieved, manual participation of the user is not needed in the selection and loading, the time of the user is saved, and the automation requirement of data construction is met.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic flow chart of a data construction process according to an embodiment of the present application;

FIG. 2 is a block diagram of a framework for data construction according to an embodiment of the present application;

FIG. 3 is a schematic illustration of the construction and management of a data processing-related environment of an embodiment of the present application;

FIG. 4 is a flow chart illustrating a data processing method according to an embodiment of the present application;

FIG. 5a is a flow chart illustrating a data processing method according to an embodiment of the present application;

FIG. 5b is a schematic flow chart illustrating the blocking of source data to facilitate the synchronous extraction and transformation of data according to an embodiment of the present application;

FIG. 6 is a flow chart of an interactive method of data processing according to an embodiment of the present application;

FIG. 7 is a schematic illustration of a configuration interface according to an embodiment of the present application;

FIG. 8 is a block diagram of a data processing apparatus according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an intelligent device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In the embodiment of the application, for the data construction task, after the data construction task is obtained, the source data can be automatically obtained from the source data environment equipment based on the data selection script and the data loading script indicated by the task configuration information in the data construction task, and the target data obtained according to the source data is automatically loaded into the target data environment equipment, so that some function platforms or applications capable of obtaining data from the target data environment equipment can conveniently obtain the required target data. After the data selection script is executed, partial source data can be acquired from the source data environment equipment, the data which are needed can be customized in the data selection script, and meanwhile, the data loading script can load the target data corresponding to the selected source data into the target data environment for use, so that the automation requirement of data construction can be met, and the personalized data requirement of a user can be met. In the whole process, a user only needs to compile a script, automatic processing of data selection and loading can be achieved through the application corresponding to the application based on the compiled script, manual participation of the user is not needed in the selection and loading process, user time is saved, and the automatic requirement of data construction is met.

The servers (for example, the intelligent devices and the devices included in the management platform mentioned in the embodiments) related to the embodiments of the present application may be independent physical servers, may also be a server cluster or a distributed system formed by a plurality of physical servers, and may also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms. The terminal may be, but is not limited to, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal and the server may be directly or indirectly connected through wired or wireless communication, and the application is not limited herein.

The embodiment of the application can realize related data processing operation in a data construction process based on a Cloud technology, and the Cloud technology refers to a hosting technology for unifying series resources such as hardware, software and networks in a wide area network or a local area network to realize calculation, storage, processing and sharing of data. Cloud technology (Cloud technology) is based on a general term of network technology, information technology, integration technology, management platform technology, application technology and the like applied in a Cloud computing business model, can form a resource pool, is used as required, and is flexible and convenient. Cloud computing technology will become an important support. Background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture-like websites and more web portals. With the high development and application of the internet industry, each article may have its own identification mark and needs to be transmitted to a background system for logic processing, data in different levels are processed separately, and various industrial data need strong system background support and can only be realized through cloud computing.

The source data and the target data related to the application can be stored in the cloud end in a cloud storage mode, and the source data environment equipment and the target data environment equipment can also be equipment in the cloud end, such as some servers for realizing a data storage function. A distributed cloud storage system (hereinafter, referred to as a storage system) refers to a storage system that integrates a large number of storage devices (storage devices are also referred to as storage nodes) of different types in a network through application software or application interfaces to cooperatively work by using functions such as cluster application, grid technology, and a distributed storage file system, and provides a data storage function and a service access function to the outside. The application can select the source data through the software or the application interfaces, and load the source data into a new data environment for different data users to use.

The storage method of the cloud storage system comprises the following steps: logical volumes are created, and when created, each logical volume is allocated physical storage space, which may be the disk composition of a certain storage device or of several storage devices. The client stores data on a certain logical volume, that is, the data is stored on a file system, the file system divides the data into a plurality of parts, each part is an object, the object not only contains the data but also contains additional information such as data identification (ID, ID entry), the file system writes each object into a physical storage space of the logical volume, and the file system records storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

The process of allocating physical storage space for the logical volume by the storage system specifically includes: physical storage space is divided in advance into stripes according to a group of capacity measures of objects stored in a logical volume (the measures often have a large margin with respect to the capacity of the actual objects to be stored) and Redundant Array of Independent Disks (RAID), and one logical volume can be understood as one stripe, thereby allocating physical storage space to the logical volume.

Database (Database), which can be regarded as an electronic file cabinet in short, a place for storing electronic files, a user can add, query, update, delete, etc. to data in files. A "database" is a collection of data that is stored together in a manner that can be shared by multiple users, has as little redundancy as possible, and is independent of the application.

A Database Management System (DBMS) is a computer software System designed for managing a Database, and generally has basic functions such as storage, interception, security assurance, and backup. The database management system may classify the database according to the database model it supports, such as relational, XML (Extensible Markup Language); or classified according to the type of computer supported, e.g., server cluster, mobile phone; or sorted according to the Query Language used, such as SQL (Structured Query Language), XQuery, or sorted according to performance impulse emphasis, such as max size, maximum operating speed, or other sorting.

In the embodiment of the present application, the data construction process is divided into multiple stages, which mainly include selection (Select), extraction (Extract), transformation (Transform), and loading (Load). The selecting (Select) refers to selecting source data, and the source data may be data in environments such as a database, a cache, a file, a network, and the like. Select scripts (i.e., data selection scripts) can be written to obtain data from the above types of source data. Extraction (Extract) refers to extracting data obtained from source data. Such as extracting data for certain fields in the database. An Extract script (i.e., a data extraction script) can also be written, so as to Extract the required data from the source data selected by the Select script. The conversion (Tranfrom) refers to converting the data extracted by the Extract script, and a specific conversion process is implemented according to the user-defined Tranfrom script (i.e. data conversion script), and the conversion process of the data includes, for example, deletion of some unnecessary data items, conversion between some data and data, and the like. Loading (Load) refers to importing the converted data into a target database, or loading the converted data into a target data environment, so that a corresponding data demand end reads data from the target database or the target data environment. Data loading is accomplished by a user-defined Load script (i.e., a data loading script).

In one embodiment, a process diagram of data construction is shown in fig. 1, the process of data construction is divided into 4 stages, and the construction of data can be automatically completed by using a user-defined script in each of the 4 stages. It should be noted that, when a user does not define a custom script at a certain stage, no processing is performed on data at this stage, for example, when an Extract script is not defined, data extraction is not performed on source data acquired through a Select script, and for example, when a Tranfrom script is not defined, no data conversion processing exists.

Referring to fig. 2 again, it is a schematic diagram of a framework for data construction according to an embodiment of the present application, fig. 2 only illustrates one possible implementation manner, and the embodiment of the present application does not limit a specific architecture, for example, in fig. 2, a management platform and an intelligent device for data processing are two different components, but in other embodiments, the management platform and the intelligent device for data processing may be the same device or belong to the same system.

Various types of source data are stored in the source data environment device, and in fig. 2, cc (configuration center) type source data, bc (business center) type source data, Key system type source data, CMDB (configuration management database) type source data, CKV (Cloud Key-Value) (a memory-level, persistent, low-cost, distributed storage system Key) type source data, DB (database) type source data are shown, and of course, other types of source data may be included. The source data environment device may be a cloud server or a dedicated storage device.

In fig. 2, the core service module and the underlying facility module may belong to the same management platform, and provide a device or a device group of specific services in the management platform. The management platform can select the desired source data from the source data environment equipment on one hand, and can load the acquired source data into the target data environment equipment after processing the acquired source data through the data construction agent on the other hand. The data construction agent may be installed in an intelligent device (e.g., a server) in the management platform, or may be installed in the target data environment device, and the data acquired from the management platform is processed and stored in the CC source data, the BC source data, the Key source data, the CMDB source data, the CKV source data, and the DB source data configured in the target data environment device. The standard data environment device may also be one of the intelligent devices (e.g., on a server) in the management platform.

The management platform performs data interaction with other external systems or platforms by providing an external interface. Fig. 2 shows that data is interacted with the environment management system through the environment data construction interface, data is interacted with the automation test system through the service data construction interface, data is interacted with the account management system through the account management interface, and of course, data interaction may also be performed through other special or general interfaces, which is not limited in this application. The data interacted with may include a data construction event initiated by a corresponding system or platform, and the management platform responds to the data loaded by the data construction agent in the target data environment device provided by the data construction event, for example, a target data environment including CC source data, BC source data, Key source data, CMDB source data, CKV source data, and DB source data included in the target data environment device in fig. 2 is provided to the environment management system as the interaction data together.

Referring to fig. 3 again, it is a schematic diagram of the construction and management of the data processing related environment according to the embodiment of the present application, which mainly includes: resource initialization, where resources include installation packages for machines, services, and the like. At this stage, initialization preparation is carried out on a machine, an environment for running system services and the like are installed, an installation package of each service to be installed and a mirror image of a container are prepared, when a new environment and a target data environment are built, services such as mysql and the like can be installed in a mode of downloading the installation package, or the services can be packaged into the mirror image in advance, and then the mirror image is downloaded and the container is started to complete deployment of the services. And arranging and networking containers, and finishing the installation and deployment of each service module at the stage. Specifically, the mirror image is pulled, the container is started, and the configuration of each service is completed at the same time. And (3) data construction, namely constructing and generating configuration data, service data (such as account numbers) and the like required by the service at the stage. And checking the service state, namely checking whether the running state of the service is normal. And environment verification, namely verifying whether the whole environment is available or not and whether the whole environment can be delivered to a user or not through a use case. The environment verification is mainly used for verifying the established target data environment, the test case is operated in the environment in a test case mode, and the environment is correctly available when the test case is executed to an expected value. The source data environment device mentioned later in the embodiment of the present application refers to a device such as a server loaded with a source data environment, the target data environment device refers to a device such as a built target data environment server, services and data are included in the data environment, and corresponding data (source data or target data) is included in mysql, file, ckv, and the like.

For the data construction steps mentioned in fig. 3, in the embodiment of the present application, on the basis of fig. 2, the data construction process substantially includes: triggering of a data construction task. The triggering of tasks is typically performed by the management platform or other third party system invoking an interface of the data build. And preparing a construction task, namely writing the relevant script constructed by the data into the DB or the cache by the data construction core Server at the stage, creating the data construction task, and sending the task to the data construction Agent. Executing a construction task, analyzing the data construction task by a data construction Agent, downloading a custom script required by the execution process, starting the execution of the task, and executing according to 4 stages, wherein the execution is as follows: 4 stage steps of Select, Extract, Transform, Load, etc. And reporting the execution result. The data construction Agent reports the process and the result of data construction execution to the data construction core Server, and the data construction core Server updates the execution state of the task.

Referring to fig. 4, it is a schematic flow chart of a data processing method according to an embodiment of the present application, where the embodiment of the present application may be executed by an intelligent device, and the intelligent device may implement the method by running the above-mentioned data construction agent. The intelligent device may be a single server or the like, or may be a part of the management platform. The method comprises the following steps.

S401: acquiring a data construction task, wherein the data construction task is generated according to the detected data construction event. The data construction task may be sent by a core service module, such as in fig. 2, and the core service module generates the data construction task and sends the data construction task to the intelligent device according to the embodiment of the present application when receiving a data construction event initiated by a user or receiving the data construction event from an API. The data construction task comprises a data selection script, a data recording script, a device identifier of source data environment equipment, an identifier of target data environment equipment and the like which are set according to a data construction event. In other embodiments, the data construction tasks may also include data extraction scripts, data transformation scripts, and the like.

S402: and analyzing the data construction task, acquiring a data selection script associated with the data construction task, and executing the data selection script to obtain source data, wherein the data selection script is used for acquiring the source data from source data environment equipment. The data selection script can be directly written by related users, or can be written according to a provided script template in the process of configuring a data construction task or before sending a data construction event. And by running the data selection script, the source data of the specified type can be acquired from the specified equipment. The source data acquired by the data selection script comprises: any one or more of CC-type source data, BC-type source data, Key-type source data, CMDB-type source data, CKV-type source data, and DB-type source data, for example, by running a data selection script, the DB source data stored in the source data environment device can be acquired, and all data stored in the DB can be acquired. In one embodiment, in the process of running the data selection script in S402, the core service module obtains the required source data from the source data environment device.

S403: and acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is obtained according to the source data. The data loading script can be directly written by related users, or can be written according to a provided script template in the process of configuring a data construction task or before sending a data construction event. By executing the data loading script, all the data of the source data acquired in S402 can be loaded into the data environment of the target data environment device according to the environment requirement of the target data environment device, for example, all the data of the DB source data acquired from the source data environment device shown in fig. 2 in S403 can be loaded into the DB database in the target data environment device in fig. 2 to constitute the DB source data in the target data environment device.

S404: and feeding back a notification message about the target data loaded to the target data environment equipment to a management platform, wherein the notification message is used for instructing the management platform to respond to the data construction event and process the target data in the target data environment equipment. After the loading is completed, a notification message may be fed back to the management platform that sends a data construction task downward, so that a core service module in the management platform (for example, in fig. 2) processes target data, where the processing of the target data includes directly acquiring target data included in a data environment in target data environment equipment, and the acquired target data may also be sent to other systems or platforms through an API (Application Programming Interface) Interface or the like according to an actual situation.

Please refer to fig. 5a, which is a flowchart illustrating a data processing method according to an embodiment of the present application, where the embodiment of the present application can be executed by the above-mentioned smart device. The method comprises the following steps.

S501: acquiring a data construction task, wherein the data construction task is generated according to the detected data construction event. The data construction event can be detected after a user submits task configuration information on a configuration interface; or the data construction event may be detected after receiving a data acquisition request initiated by an external system or platform through an interface. In different ways, different processing is performed after the construction of the data is completed. For the specific representation and usage of the configuration interface, please refer to the description of the embodiments described later.

S502: and analyzing the data construction task, acquiring a data selection script associated with the data construction task, and executing the data selection script to obtain source data, wherein the data selection script is used for acquiring the source data from source data environment equipment. In one embodiment, the identity of the source data environment device may also be configured on the configuration interface. The type of source data may refer to the description in the foregoing embodiments.

S503: and acquiring a data extraction script associated with the data construction task, and extracting intermediate data meeting screening conditions from the acquired source data by executing the data extraction script. The filtering condition is defined by the user in the data extraction script. For example, the source data selected in S502 may be a table in which a large number of types of data are recorded, and it is defined in the script that only a part of the types of data are acquired.

S504: and acquiring a data conversion script associated with the data construction task, and converting the intermediate data into target data meeting conversion conditions by executing the data conversion script. The data conversion script is mainly to convert part or all of the intermediate data into data meaningful to a system or platform reading data from a target data environment device, for example, in the intermediate data, calculation result data and calculation raw data exist, the calculation result data is calculated based on one or more calculation raw data, if only the calculation result data and the calculation raw data are directly loaded into the target data environment, it is possible for the platform or system extracting data from the target data environment that the calculation result data and the calculation raw data do not have any meaning and the relationship therebetween is not clear, and therefore, in the data conversion script, script contents regarding the calculation process obtaining the calculation result data can be added, so that, for the calculation result data and the calculation raw data, the target data satisfying the conversion condition obtained by conversion includes calculation result data, calculation original data, and calculation rules.

S505: and acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is acquired according to the source data, namely the target data is acquired by performing the data extraction and data conversion on the source data.

The step of obtaining the corresponding script mentioned in S502 to S504 may be to read all scripts of the data construction task from the cache device at one time, so as to perform corresponding running processing, and then sequentially obtain and execute all scripts read and stored in the local. Or sequentially acquiring corresponding scripts from the cache according to the instruction of the data construction task for running processing, namely acquiring and executing the data selection script, acquiring and executing the data extraction script, acquiring and executing the data conversion script, and acquiring and executing the data loading script.

S506: and feeding back a notification message about the target data loaded to the target data environment equipment to a management platform, wherein the notification message is used for instructing the management platform to respond to the data construction event and process the target data in the target data environment equipment.

It will be appreciated that the manner in which data is interacted with as referred to in this application may be implemented via a computer network. Description information of the content and field format of each source data in the source data environment equipment can be displayed, so that a user who writes the above-mentioned data selection script, data extraction script, data conversion script and data loading script writes a corresponding script based on the description information.

In one embodiment, the data construction event is generated in response to a detected user submission received from the configuration interface, and the feedback of the notification message about the target data being loaded to the target data environment device to the management platform includes: generating a notification message, wherein the notification message comprises storage indication information of target data loaded into target data environment equipment, and the storage indication information comprises a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for indicating the management platform to acquire target data from target data environment equipment according to the indication of the storage indication information. That is, if it is a task that the user directly initiates on the configuration interface, the smart device may notify the management platform to directly obtain the target data in the target data environment to provide to other systems or platforms. Different databases are obtained in different manners, so that the database type is added to the storage indication information, so that the management platform can correctly initiate a data request to the storage address to obtain the target data.

In one embodiment, the data construction event is generated in response to a data construction request received from an interface connected to the management platform, and the feeding back a notification message to the management platform about the target data being loaded to the target data environment device includes: generating a notification message, wherein the notification message comprises storage indication information of target data loaded into target data environment equipment, and the storage indication information comprises a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for instructing the management platform to forward the storage indication information to an initiating terminal initiating the data construction request, so that the initiating terminal acquires target data from target data environment equipment according to the indication of the storage indication information. That is, if an external system or platform requests data construction through an interface, the management platform may be triggered to forward the notification message to the originating terminal of the data construction request, so that the originating terminal obtains data from a corresponding storage address based on the storage indication information in a corresponding database. Different databases are obtained in different manners, and therefore, the database type is added to the storage indication information, so that a corresponding platform or system can correctly initiate a data request to a storage address to obtain target data.

The four layers are divided in the data construction process, so that the data construction process can be processed according to different stages and layers. When the data volume required to be constructed is large, the time consumed by the data construction process is also large, so that in the embodiment of the application, the data can be partitioned to facilitate concurrent processing. The concurrent processing mainly involves two stages, namely a stage of data selection by the data selection script and a stage of data conversion by the data conversion script. First, the extracting intermediate data satisfying a filtering condition from the obtained source data by executing the data extraction script may include: partitioning the source data according to the data relation among all data items in the source data to obtain source block data, and adding the source block data into corresponding queues according to the data relation to obtain a plurality of source block queues; respectively and synchronously extracting intermediate data meeting screening conditions from the data of each source block in each source block queue by executing the data extraction script to obtain a plurality of intermediate data block queues;

next, the converting the intermediate data into the target data satisfying the conversion condition by executing the data conversion script may include: and synchronously converting the intermediate data blocks in each intermediate data block queue by executing the data conversion script to respectively obtain target data meeting conversion conditions and output each target data.

Fig. 5b is a schematic flow chart illustrating the process of partitioning the source data to facilitate the synchronous data extraction and conversion. For the acquired source data, data blocking is performed based on the relationship between the source data, for example, for 10 pieces of acquired table data, 3 pieces of tables are related to financial data and can be divided into one piece of source block data, 3 pieces of tables are related to sales data and can be divided into one piece of source block data, 4 pieces of tables are related to cost data and can be divided into one piece of source block data, so that three pieces of source block data can be obtained, extraction processing and conversion processing are performed on the three pieces of source block data respectively, and the processing efficiency is improved.

According to the embodiment of the application, the data selection script, the data extraction script, the data conversion script and the data loading script are automatically operated, so that the conversion and the construction of data are fully automatically realized, the requirement of a user on the automation of the construction of the data is met, and the requirement of the construction of personalized data is also met. In the whole process, a user only needs to compile a part of scripts and configure the scripts to a data construction task, so that the time of the user is saved, the automation requirement of data construction is met, the block processing logic is introduced, the processing concurrency is improved, and the data construction efficiency is further improved.

Referring to fig. 6 again, a flowchart of an interaction method for data processing according to an embodiment of the present application is shown, where the method according to the embodiment of the present application includes the following steps.

S601: the management platform triggers data construction. The management platform may receive a data acquisition request initiated by an environment management system, such as the environment management system in fig. 2, or may detect that a user initiates a confirmation operation on a configuration interface to generate the data acquisition request. After receiving the data acquisition request, the management platform determines that a data construction event is detected, and triggers the core service module to generate a data construction task according to the data acquisition request.

S602: the core service module creates a data construction task and stores a relevant script corresponding to the data construction task into a memory. The memory may be a dedicated database or a cache, so that the intelligent device in which the data construction agent is installed can acquire the corresponding script when executing the data creation task.

In one embodiment, the script stored in S602 includes: the data processing system comprises a data selection script, a data extraction script, a data conversion script and a data loading script. After the storage successfully stores the relevant information including the script in the data construction task, the storage can feed back the information of successful storage.

In one embodiment, the core service module is a part of the management platform, and the core service module may determine that a data construction event exists directly based on a data acquisition request initiated by the environment management system, that is, may generate a data construction task according to the data construction event. Alternatively, the core service module may determine that the data construction event is detected directly after receiving a confirmation operation of the user on the configuration interface, and generate the data construction task based on the task configuration information received by the user on the configuration interface.

Fig. 7 is a schematic diagram of a configuration interface according to an embodiment of the present application, which may be provided to a user by a core service module. The configuration interface comprises the following configuration items:

the configuration item for configuring the source data environment device identification, i.e. the source environment ID item 701 in fig. 7, shows in fig. 7 a specific source environment ID number entered by the user: ENV1597908578T 7279981;

configuration items for configuring the target data environment device identification, namely the target environment ID item 702 in fig. 7, a specific target data environment ID number entered by the user is shown in fig. 7: ENV1517208544T 6275904;

a configuration item for configuring a data type of data to be constructed, that is, the construction data type item 703 in fig. 7, a user may select a CC data type, a BC data type, a key data type, and the like, and may also customize "other" data types;

a configuration item of data indication information, such as configuration item 704 in fig. 7, the data indication information comprising: the source type and/or data association information of the data needing to be selected through the data selection script; as shown in fig. 7, the source type may refer to a "mysql" type, i.e., source data is selected from a database, the source type may refer to a "file" type and source data is selected from a file, and the source type may also refer to an "api" type, i.e., source data is selected through an api interface. The data association information is used to specify a source type so that data can be correctly selected when the data selection script is executed, for example, as shown in fig. 7, when the source type selected by the user is "mysql", the corresponding data association information includes: the method comprises the steps that required source data can be found correctly according to the two pieces of information, data association information can also comprise a user name, a password and a library name, when text is selected to obtain the source data based on the information operation data, source data environment equipment can be obtained to verify the source data obtaining, and after authorization is obtained, the intelligent equipment is allowed to obtain the source data corresponding to the corresponding database address.

Configuration items of data storage information, such as configuration item 705 in FIG. 7, including: the storage type of data loaded by the data loading script and/or data storage information is required. As shown in fig. 7, the storage type may refer to a "mysql" type, that is, source data is stored into a database, the storage type may refer to a "file" type, that is, source data is stored into a certain file, and the storage type may also refer to an "api" type, that is, source data is stored into a device or application associated with an api interface through the api interface. The data storage information is used to specify a storage type so that data can be correctly loaded when the data loading script is executed, for example, as shown in fig. 7, when the storage type selected by the user is "file", the corresponding data storage information includes: and the file path can be used for correctly finding the corresponding file according to the information to determine the required source data, the data storage information can also comprise a user name, a password and a file name, the loading of the source data by the target data environment equipment can be verified when the data loading text is operated to load the source data into a certain file based on the information, and the intelligent equipment is allowed to record the acquired source data into the file corresponding to the file path after the source data is authorized.

When a user clicks a submit button on the configuration interface, a data construction request can be initiated, or a data construction event is used for triggering a core service module to create a data construction task.

In one embodiment, for some data construction requirements, the data in the selected source data can be further filtered, and only part of the data is extracted. Therefore, the configuration interface may further include: uploading configuration items of the data extraction script, wherein the uploading configuration items of the data extraction script comprise:

a recording frame for recording script identification of the uploaded data extraction script; such as record box 706 in fig. 7. An upload input button for acquiring a data extraction script confirmed by a user in response to a user click operation and uploading the acquired data extraction script; such as the "upload script" button in fig. 7. A download input button, such as the "script template download" button in fig. 7, for displaying the data extraction script template to the user in response to a user download operation, and acquiring the data extraction script generated from the data extraction script template and uploading the generated data extraction script.

In an embodiment, the data extracted by the data extraction script or the source data selected by the data selection script may be further converted to obtain target data meeting the data construction requirement, and therefore, the configuration interface may further include: the uploading configuration items of the data conversion script comprise:

a recording frame for recording script identification of the uploaded data conversion script; such as record box 707 in fig. 7. An upload input button for responding to a user click operation to acquire a data conversion script confirmed by the user and uploading the acquired data conversion script; such as the "upload script" button in fig. 7. A download input button, such as the "script template download" button in fig. 7, for displaying the data conversion script template to the user in response to a user download operation, and acquiring the data conversion script generated from the data conversion script template and uploading the generated data conversion script.

S603: and the core service module receives the feedback message of successful storage.

S604: and the core service module issues the data construction task to the intelligent equipment. For the intelligent device, the data construction task is received, and the data processing of the embodiment of the application is started to be executed.

S605: and the intelligent equipment analyzes the data to construct a task. The intelligent device acquires the data construction task and reads task configuration information from the data construction task so as to conveniently select source data, extract data, convert data and record data.

S606: the intelligent device obtains the script. The corresponding script file may be found in the memory based on the identification of the data construction task. In S606, the following are included: the method comprises the steps of obtaining a data selection script related to the data construction task, obtaining a data loading script related to the data construction task, obtaining a data extraction script related to the data construction task and obtaining a data conversion script related to the data construction task.

S607: the memory returns the required script.

S608: the intelligent device creates a Select process;

s609: and the intelligent equipment obtains the source data by executing the data selection script.

S610: the intelligent equipment receives source data returned by the source data environment equipment;

s611: the intelligent equipment creates an Extract process;

s612: and the intelligent equipment extracts the intermediate data meeting the screening condition from the obtained source data by executing the data extraction script. After the Extract process is created, the source data may be partitioned, and a concurrent task may be created for subsequent data processing.

S613: the intelligent device creates a Tranfrom process;

s614: and the intelligent equipment converts the intermediate data into target data meeting conversion conditions by executing the data conversion script. In the process of executing the Extract script, if the source data is blocked in S612, the plurality of blocks extracted by the data extraction script are also converted in S614 to obtain a plurality of target data.

S615: the intelligent equipment creates a load process;

s616: the intelligent equipment loads target data into the target data environment equipment by executing the data loading script;

s617: and the intelligent equipment receives the load result returned by the target data environment equipment.

S618: and the intelligent device feeds back a notification message about the target data loaded to the target data environment device to the core service module, wherein the notification message is used for instructing the management platform to respond to the data construction event and process the target data in the target data environment device.

S619: and the core service module sends a construction result updating notice to the management platform so as to inform the management platform that the corresponding data is successfully constructed.

In the embodiment of the application, a simple and visual configuration interface can be provided for a user, the user can configure the information required by data construction conveniently, the user can upload scripts conveniently or rapidly through a template, the scripts are selected through automatically operating the data configured by the user, and the data extraction scripts, the data conversion scripts and the data loading scripts are fully automatically realized, so that the requirement of the user on automation of data construction is met, and the requirement of personalized data construction is also met. And a block processing logic is introduced, so that the processing concurrency is improved, and the data construction efficiency is further improved.

Referring to fig. 8, it is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention, where the data processing apparatus according to the embodiment of the present invention may be disposed in an intelligent device such as a server, and the intelligent device may be the intelligent device mentioned in the foregoing embodiment. The device comprises the following modules.

An obtaining module 801, configured to obtain a data construction task, where the data construction task is generated according to a detected data construction event;

a processing module 802, configured to parse the data construction task, obtain a data selection script associated with the data construction task, and obtain source data by executing the data selection script, where the data selection script is used to obtain the source data from a source data environment device; acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is obtained according to the source data;

a notification module 803, configured to feed back, to the management platform, a notification message that the target data is loaded to the target data environment device, where the notification message is used to instruct the management platform to process the target data in the target data environment device in response to the data construction event.

In an embodiment, the processing module 802 is further configured to, after executing the data selection script to obtain source data and before executing the data loading script to load target data into target data environment equipment, obtain a data extraction script associated with the data construction task, and extract intermediate data meeting a screening condition from the obtained source data by executing the data extraction script; and acquiring a data conversion script associated with the data construction task, and converting the intermediate data into target data meeting conversion conditions by executing the data conversion script.

In an embodiment, the processing module 802 is configured to block source data according to a data relationship between data items in the source data to obtain source block data, and add the source block data to a corresponding queue according to the data relationship to obtain a plurality of source block queues; respectively and synchronously extracting intermediate data meeting screening conditions from the data of each source block in each source block queue by executing the data extraction script to obtain a plurality of intermediate data block queues; and synchronously converting the intermediate data blocks in each intermediate data block queue by executing the data conversion script to respectively obtain target data meeting conversion conditions and output each target data.

In one embodiment, the data construction task includes task configuration information, and the task configuration information is set on a configuration interface; the configuration interface includes any one or more of the following configuration items:

a configuration item for configuring the source data environment device identification;

a configuration item for configuring the target data environment device identification;

a configuration item for configuring a data type of data to be constructed;

a configuration item of data indication information, the data indication information comprising: the source type and/or data association information of the data needing to be selected through the data selection script;

a configuration item of data storage information, the data storage information comprising: the storage type of data loaded by the data loading script and/or data storage information is required.

In one embodiment, the configuration interface further includes an upload configuration item of the data extraction script, and the upload configuration item of the data extraction script includes:

a recording frame for recording script identification of the uploaded data extraction script;

an upload input button for acquiring a data extraction script confirmed by a user in response to a user click operation and uploading the acquired data extraction script;

and a download input button for displaying the data extraction script template to the user in response to a user download operation, acquiring the data extraction script generated according to the data extraction script template, and uploading the generated data extraction script.

In one embodiment, the configuration interface further includes an upload configuration item of the data conversion script, where the upload configuration item of the data conversion script includes:

a recording frame for recording script identification of the uploaded data conversion script;

an upload input button for responding to a user click operation to acquire a data conversion script confirmed by the user and uploading the acquired data conversion script;

and a download input button for responding to a download operation of a user, displaying the data conversion script template to the user, acquiring the data conversion script generated according to the data conversion script template, and uploading the generated data conversion script.

In one embodiment, the notification module 803 is configured to generate a notification message, where the notification message includes storage indication information of target data loaded into a target data environment device, and the storage indication information includes a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for indicating the management platform to acquire target data from target data environment equipment according to the indication of the storage indication information.

In one embodiment, the notification module 803 is configured to generate a notification message, where the notification message includes storage indication information of target data loaded into a target data environment device, and the storage indication information includes a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for instructing the management platform to forward the storage indication information to an initiating terminal initiating the data construction request, so that the initiating terminal acquires target data from target data environment equipment according to the indication of the storage indication information.

Referring to fig. 9, a schematic structural diagram of an intelligent device according to an embodiment of the present invention is shown, where the intelligent device according to the embodiment of the present invention corresponds to the intelligent device mentioned in the foregoing embodiment, and may be a server. The intelligent device according to the embodiment of the present invention may include a storage 901 and a processor 902, and may also include an interface for data interaction, such as a network interface 903 and a communication interface 904. Interface modules such as a power supply module, a USB data interface, etc. may also be included.

The network interface 903 may be various interfaces for accessing to a computer network, and the communication interface 904 may be a mobile communication interface 904 capable of accessing to a 4G/5G mobile communication network.

The storage 901 may include a volatile memory (volatile memory), such as a random-access memory (RAM); the storage 901 may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a solid-state drive (SSD), or the like; the storage 901 may also comprise a combination of memories of the kind described above.

The processor 902 may be a Central Processing Unit (CPU) 902. The processor 902 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or the like. The PLD may be a field-programmable gate array (FPGA), a General Array Logic (GAL), or the like.

Optionally, the storage 901 is further configured to store a computer program. The processor 902 may invoke the computer program to implement the relevant methods as mentioned in the previous embodiments of the application.

In one embodiment, the processor 902 calls a computer program stored in the storage 901 for obtaining a data construction task, wherein the data construction task is generated according to a detected data construction event; analyzing the data construction task, acquiring a data selection script associated with the data construction task, and executing the data selection script to obtain source data, wherein the data selection script is used for acquiring the source data from source data environment equipment; acquiring a data loading script associated with the data construction task, and loading target data into target data environment equipment by executing the data loading script, wherein the target data is obtained according to the source data; and feeding back a notification message about the target data loaded to the target data environment equipment to a management platform, wherein the notification message is used for instructing the management platform to respond to the data construction event and process the target data in the target data environment equipment.

In one embodiment, the processor 902, after executing the data selection script to obtain source data and before executing the data loading script to load target data into target data environment equipment, is further configured to obtain a data extraction script associated with the data construction task, and extract intermediate data meeting a filtering condition from the obtained source data by executing the data extraction script; and acquiring a data conversion script associated with the data construction task, and converting the intermediate data into target data meeting conversion conditions by executing the data conversion script.

In an embodiment, the processor 902 is configured to block source data according to a data relationship between data items in the source data to obtain source block data, and add the source block data to a corresponding queue according to the data relationship to obtain a plurality of source block queues; respectively and synchronously extracting intermediate data meeting screening conditions from the data of each source block in each source block queue by executing the data extraction script to obtain a plurality of intermediate data block queues; and synchronously converting the intermediate data blocks in each intermediate data block queue by executing the data conversion script to respectively obtain target data meeting conversion conditions and output each target data.

a configuration item for configuring a data type of data to be constructed;

In one embodiment, the configuration interface further includes an upload configuration item of the data extraction script, where the upload configuration item of the data extraction script includes:

In one embodiment, the data construction event is generated in response to a detected user submission received from the configuration interface, and the processor 902 is configured to generate a notification message, where the notification message includes storage indication information of target data loaded into a target data environment device, and the storage indication information includes a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for indicating the management platform to acquire target data from target data environment equipment according to the indication of the storage indication information.

In one embodiment, the data construction event is generated in response to a data construction request received from an interface connected to the management platform, and the processor 902 is configured to generate a notification message, where the notification message includes storage indication information of target data loaded into a target data environment device, and the storage indication information includes a database type and a storage address; and feeding back the notification message to a management platform, wherein the notification message is used for instructing the management platform to forward the storage indication information to an initiating terminal initiating the data construction request, so that the initiating terminal acquires target data from target data environment equipment according to the indication of the storage indication information.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

While the invention has been described with reference to a number of embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A data processing method, comprising:

2. The method of claim 1, wherein after executing the data selection script to obtain source data and before executing the data loading script to load target data into a target data environment device, the method further comprises:

acquiring a data extraction script associated with the data construction task, and extracting intermediate data meeting a screening condition from the acquired source data by executing the data extraction script;

and acquiring a data conversion script associated with the data construction task, and converting the intermediate data into target data meeting conversion conditions by executing the data conversion script.

3. The method of claim 2,

the extracting of the intermediate data satisfying the screening condition from the obtained source data by executing the data extraction script includes:

partitioning the source data according to the data relation among all data items in the source data to obtain source block data, and adding the source block data into corresponding queues according to the data relation to obtain a plurality of source block queues;

respectively and synchronously extracting intermediate data meeting screening conditions from the data of each source block in each source block queue by executing the data extraction script to obtain a plurality of intermediate data block queues;

the converting the intermediate data into target data satisfying a conversion condition by executing the data conversion script includes:

and synchronously converting the intermediate data blocks in each intermediate data block queue by executing the data conversion script to respectively obtain target data meeting conversion conditions and output each target data.

4. The method according to any one of claims 1 to 3, wherein the data construction task comprises task configuration information, and the task configuration information is set on a configuration interface; the configuration interface includes any one or more of the following configuration items:

a configuration item for configuring a data type of data to be constructed;

5. The method of claim 4, wherein the configuration interface further comprises an upload configuration item of the data extraction script, and the upload configuration item of the data extraction script comprises:

6. The method of claim 4, wherein the configuration interface further comprises an upload configuration item of the data conversion script, and the upload configuration item of the data conversion script comprises:

7. The method of claim 4, wherein the data build event is generated in response to detecting a user-submitted operation received from the configuration interface, the feeding back to a management platform a notification message regarding the target data being loaded to a target data environment device comprising:

generating a notification message, wherein the notification message comprises storage indication information of target data loaded into target data environment equipment, and the storage indication information comprises a database type and a storage address;

and feeding back the notification message to a management platform, wherein the notification message is used for indicating the management platform to acquire target data from target data environment equipment according to the indication of the storage indication information.

8. The method of claim 4, wherein the data build event is generated in response to a data build request received from an interface to which the management platform is connected, the feeding back to the management platform a notification message regarding the target data being loaded to a target data environment device comprising:

and feeding back the notification message to a management platform, wherein the notification message is used for instructing the management platform to forward the storage indication information to an initiating terminal initiating the data construction request, so that the initiating terminal acquires target data from target data environment equipment according to the indication of the storage indication information.

9. A data processing apparatus, comprising:

10. An intelligent device, characterized in that it comprises storage means in which a computer program is stored and a processor which invokes said computer program to implement the data processing method according to any one of claims 1 to 8.

11. A computer storage medium, in which a computer program is stored, which computer program, when being executed by a processor, is to carry out a data processing method according to any one of claims 1 to 8.