CN111104450B

CN111104450B - Target data importing method, medium, device and computing equipment

Info

Publication number: CN111104450B
Application number: CN201911311712.4A
Authority: CN
Inventors: 朱云杰; 潘胜一
Original assignee: Hangzhou Netease Zaigu Technology Co Ltd
Current assignee: Hangzhou Netease Zaigu Technology Co Ltd
Priority date: 2019-12-18
Filing date: 2019-12-18
Publication date: 2023-05-02
Anticipated expiration: 2039-12-18
Also published as: CN111104450A

Abstract

The embodiment of the application provides a target data importing method, medium, device and computing equipment. The method comprises the following steps: analyzing the monitored target task to obtain a plurality of data sources corresponding to the target task; and configuring a working node for the target task, wherein the working node is used for acquiring target data from a plurality of data sources and importing the target data into a service system corresponding to the target task. According to the method and the device for processing the data, the data from the multiple data sources can be imported into the service system corresponding to the target task, the data from various data sources can be processed in a generalized mode, data synchronization is achieved, universality, expandability and usability of the system are improved, and labor cost is reduced.

Description

Target data importing method, medium, device and computing equipment

Technical Field

Embodiments of the present application relate to the field of data management technologies, and more particularly, to a target data importing method, medium, apparatus, and computing device.

Background

This section is intended to provide a background or context for embodiments of the present application that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

With the rapid growth of the internet, the explosive growth of data, the efficient organization and management of data from various types of data sources has become increasingly important. Various service systems implementing specific functions also need to obtain data from various types of data sources, for example, search engines need to obtain data from multiple data sources in order to provide search services to users.

Due to the difference of various types of data sources, in the process of importing data into a service system, customization processing needs to be performed on the data for each specific service. The customization process has the following drawbacks:

1) The scalability is poor. For example, when a service is added, it is necessary to newly develop a specific customized processing method for data to be imported for the service.

2) The versatility is poor. For example, for various types of business, a developer is required to know specific business logic.

3) The usability is low. For example, a developer needs to maintain each business individually.

In summary, the customization-based data import requires a large labor cost, and the expandability, versatility and usability of the system are poor.

Disclosure of Invention

The present application desirably provides a target data importing method and device to solve at least one of the above technical problems.

In a first aspect of embodiments of the present application, there is provided a target data importing method, including:

analyzing the monitored target task to obtain a plurality of data sources corresponding to the target task;

and configuring a working node for the target task, wherein the working node is used for acquiring target data from a plurality of data sources and importing the target data into a service system corresponding to the target task.

In one embodiment of the present application, before parsing the monitored target task, the method further includes:

acquiring an index column field corresponding to a target task, wherein the index column field is designated in data fields of a plurality of data sources;

obtaining an associated field according to the index column field, wherein the associated field is used for carrying out merging operation on target data acquired from a plurality of data sources;

and creating the target task according to the service system, the index column field, the data source and the associated field corresponding to the index column field.

In one embodiment of the present application, obtaining target data from a plurality of data sources includes:

acquiring a first data table containing first offline data corresponding to an index column field in a first data source;

acquiring second offline data corresponding to the index column field from a second data source according to a preset period, establishing a second data table in the first data source, and storing the second offline data into the second data table;

And combining the first data table and the second data table through the association field to obtain joint data, and taking the joint data as target data.

In one embodiment of the present application, the merging operation includes:

acquiring a first field value corresponding to the associated field from a first data table, and acquiring a second field value corresponding to the associated field from a second data table;

in the case that the first field value is the same as the second field value, a first record in the first data table containing the first field value is added to a second record in the second data table containing the second field value.

In one embodiment of the present application, obtaining target data from a plurality of data sources further comprises:

and adding real-time data corresponding to the index column field in the second data source updated after the second offline data is acquired into the target data.

In one embodiment of the present application, adding real-time data to the target data includes:

acquiring real-time data from a message system, and adding the acquired real-time data into target data; wherein the messaging system is used to access real-time data.

In one embodiment of the application, the working node includes a data source node and a data consumption node, the data source node is used for acquiring target data from a plurality of data sources, and the data consumption node is used for importing the target data into a service system corresponding to a target task.

In one embodiment of the present application, the data source node is further configured to: and sending the target data to the data consumption node through the data channel.

In a second aspect of the embodiments of the present application, there is provided a target data importing apparatus, including:

the analysis unit is used for analyzing the monitored target task to obtain a plurality of data sources corresponding to the target task;

the configuration unit is used for configuring a working node for the target task, and the working node is used for acquiring target data from a plurality of data sources and importing the target data into a service system corresponding to the target task.

In an embodiment of the present application, the configuration unit is further configured to:

In one embodiment of the present application, the working node is further configured to:

In one embodiment of the present application, the merging operation includes:

In a third aspect of the embodiments of the present application, there is provided a computer-readable medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above-described target data import method.

In a fourth aspect of embodiments of the present application, there is provided a computing device comprising: memory, processor and computer program stored on the memory and executable on the processor, the processor implementing the steps of the target data importing method when executing the program.

According to the target data importing method and device, the data from a plurality of data sources can be imported into the service system corresponding to the target task through the pre-configured working node, the data from various types of data sources can be processed in a generalized mode, data synchronization is achieved, universality, expandability and usability of the system are improved, and labor cost is reduced.

Drawings

The above, as well as additional purposes, features, and advantages of exemplary embodiments of the present application will become readily apparent from the following detailed description when read in conjunction with the accompanying drawings. Several embodiments of the present application are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which:

FIG. 1 schematically illustrates a flow chart of an implementation of a target data importation method according to an embodiment of the present application;

FIG. 2 schematically illustrates a schematic diagram of a target data import system according to an embodiment of the present application;

FIG. 3 schematically illustrates a flow chart for implementing a create target task of a target data import method according to an embodiment of the present application;

FIG. 4 schematically illustrates a data association diagram of a target data import method according to an embodiment of the present application;

FIG. 5 schematically illustrates a distributed working node design of a target data importation method according to an embodiment of the present application;

FIG. 6 schematically illustrates a flow chart of one implementation of a target data import method to obtain target data from multiple data sources according to an embodiment of the present application;

FIG. 7 schematically illustrates a flowchart of an implementation of a data source node acquiring data according to a target data import method according to an embodiment of the present application;

FIG. 8 schematically illustrates a media schematic for a target data import method according to an embodiment of the present application;

FIG. 9 schematically illustrates a schematic structure of a target data importing apparatus according to an embodiment of the present application;

FIG. 10 schematically illustrates a structural schematic diagram of a computing device according to an embodiment of the present application.

In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

Detailed Description

The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are presented merely to enable one skilled in the art to better understand and practice the present application and are not intended to limit the scope of the present application in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Those skilled in the art will appreciate that embodiments of the present application may be implemented as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the following forms, namely: complete hardware, complete software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.

According to an embodiment of the application, a target data importing method, medium, device and computing equipment are provided.

Any number of elements in the figures are for illustration and not limitation, and any naming is used for distinction only, and not for any limiting sense.

The principles and spirit of the present application are explained in detail below with reference to several representative embodiments thereof.

Summary of The Invention

The applicant finds that the existing target data importing method needs to customize data for each specific service, consumes large labor cost, and has poor expandability, universality and usability.

In view of this, the application provides a target data importing method and device, through the work node that is configured in advance, can import the data from a plurality of data sources into the service system that the target task corresponds to, can generalize and handle the data from all kinds of data sources, realize data synchronization, promoted commonality, scalability and the usability of system, reduced the cost of labor.

Having described the basic principles of the present application, various non-limiting embodiments of the present application are specifically described below.

Exemplary method

A target data import method according to an exemplary embodiment of the present application is described below with reference to fig. 1.

Fig. 1 schematically illustrates a flowchart of an implementation of a target data importing method according to an embodiment of the present application. As shown in fig. 1, the target data importing method in the embodiment of the present application includes the following steps:

s11: analyzing the monitored target task to obtain a plurality of data sources corresponding to the target task;

s12: and configuring a working node for the target task, wherein the working node is used for acquiring target data from a plurality of data sources and importing the target data into a service system corresponding to the target task.

Wherein the service system may comprise programs, routines or processes that perform particular system functions. Typically, various service systems implementing a particular function need to obtain data from various types of data sources, e.g., a search engine needs to obtain data from multiple data sources in order to provide a search service to a user.

Fig. 2 schematically illustrates a schematic structure of a target data import system according to an embodiment of the present application. Referring to fig. 1 and 2, in one example, a target data import system structure may include a configuration module and a scheduling module. In one embodiment, the configuration module is used to create a target task corresponding to a service requirement. Taking a service system of a search engine as an example, a search target task can be created through a configuration module when a user has a search requirement. The target task is created by designating which data sources the task obtains target data from, namely a plurality of data sources corresponding to the target task.

In step S11, the scheduling module monitors the created target task, analyzes the target task and obtains a plurality of data sources corresponding to the target task. In step S12, the scheduling module performs pre-configuration for the target task, i.e. configures which working nodes perform the task, and which data sources the working nodes obtain target data from, store the target data in which service system.

For example, the scheduling module monitors that a search target task is created by a user, analyzes the target task, and obtains a data source for acquiring target data, which is designated when the target task is created. In one example, a user may specify that target data be retrieved from a first data source and a second data source when creating a target task. The first data source may be a Hive database and the second data source may be a Mysql database. The scheduling module then performs pre-configuration for the search target task, i.e. configures the working node that performs the search target task, and instructs the working node to acquire target data from the specified Hive database and Mysql database in the configuration information. When the target task is executed, the target data acquisition task and the target data consumption task need to be executed first. The target data acquisition task includes acquiring target data from a plurality of data sources; the target data consumption task includes importing target data into a specified index of the search engine. The worker node may be a machine that performs a target data acquisition task and a target data consumption task. After the search target task is parsed, the scheduling module pre-configures which machines execute tasks that obtain target data from multiple data sources and which machines execute tasks that import target data into a specified index of the search engine.

In this example, the service system corresponding to the search target task created by the user is a search engine, so the configuration information may also instruct the working node to import the target data into the search engine after acquiring the target data. Typically, a search engine can guarantee the query speed of data through an index. For example, an index may be built in a search engine and related data may be stored in the same index. In the above example, the worker node may also be instructed in configuration information to import the target data into a specified index of the search engine.

In one example, an elastiscearch may be employed as the search engine. The elastosearch is a distributed, highly extended, high real-time search and data analysis engine. The method can conveniently enable a large amount of data to have the capabilities of searching, analyzing and exploring, and is used for searching, recommending and other data recall scenes.

And on the basis of the pre-configuration, the scheduling module distributes the target task to the corresponding working node to execute the task. Typically, the formats of the data stores of the different data sources are different. In one embodiment, after the working node obtains the corresponding target data from the designated data source according to the instruction, the target data is subjected to format conversion, and after the target data is converted into a uniform format, the target data is imported into a service system corresponding to the target task through a data channel.

Fig. 3 schematically shows a flowchart for implementing a creation target task of a target data import method according to an embodiment of the present application. As shown in fig. 3, in one possible implementation, step S11 in fig. 1: before analyzing the monitored target task, the method further comprises the following steps:

step S31: acquiring an index column field corresponding to a target task, wherein the index column field is designated in data fields of a plurality of data sources;

step S32: obtaining an associated field according to the index column field, wherein the associated field is used for carrying out merging operation on target data acquired from a plurality of data sources;

step S33: and creating the target task according to the service system, the index column field, the data source and the associated field corresponding to the index column field.

Referring again to FIG. 2, in one example, to facilitate creation of a target task, an operator interface may be provided by the configuration module for a user such that the user may perform related operations on the data source in the operator interface. For example, the user may specify the data source to which the target task corresponds. After the user has specified the data source, schema information may be obtained from the user-specified data source. In databases, schema is the organization and structure of the database, is an abstract representation of the database metadata, and mainly relates to declarations of elements and attributes, etc. The schema information contains the data fields of each data table in the data source. After the schema information of the data source is acquired, the data fields of each data table in the data source can be displayed in the operation interface.

Still taking the search target task as an example, the user may specify an index column field corresponding to the target task, that is, a field required for the search engine index, from the data fields of the data table. In one example, correlating the data sources in the operations interface may include: and setting the field names, indexes, aggregation and other attributes of the data fields in the data table. Wherein, whether index indicates whether the field needs to be used in the search engine, that is, whether the field is designated as an index column field; whether to aggregate or not indicates whether to count the search results for the value of the field. For example, for the "age" field, statistics may be performed on several results searched, such as counting the total number of people in the interval of 20 to 30 years of age.

In step S31, an index column field corresponding to the target task specified by the user in the operation interface is acquired. In step S32, associated fields for performing a merging operation on the respective data tables are determined from index column fields specified in the data tables of the plurality of data sources.

Fig. 4 schematically shows a data association diagram of a target data importing method according to an embodiment of the present application. The reference numerals shown in fig. 4 have the following meanings:

Reference numeral 1 denotes schema information corresponding to an index field in a first data source (e.g., hive database). As can be seen from fig. 4, the index column fields specified by the user in the data table of the Hive database include: ID: string, name: string. Wherein, the field name of the first index column field is ID (identification number), and the field type is character string; the field name of the second index column field is name and the field type is a string.

Reference numeral 2 denotes schema information corresponding to an index field in a second data source (e.g., mysql database). As can be seen from fig. 4, the index column fields specified by the user in the data table of the Mysql database include: ID is String, age is Int. Wherein, the field name of the first index column field is ID (identification number), and the field type is character string; the field name of the second index column field is age and the field type is integer.

As can be seen from the information indicated by the above reference numerals 1 and 2, the above two data tables have a common field ID String. The ID String can be used as an association field, and the two data tables are combined through the ID String. In the embodiment of the application, the combined data is used as target data to be imported into a service system corresponding to the target task.

Reference numeral 3 denotes schema information corresponding to an index column field in a service system corresponding to a target task. As can be seen from fig. 4, the index column field in the service system corresponding to the target task includes: ID: string, name: string, age: int. That is, index column field IDs and names are selected from the Hive data source, and index column field IDs and names are selected from the Mysql data source, which are associated according to the ID field to form the index column field IDs, names, and names of the search engine.

In step S33, the configuration module may create the target task according to the service system, the index column field, the data source and the associated field corresponding to the index column field. In the example of FIG. 4, a search target task may be created from the following information: the service system corresponding to the search target task is a search engine, index column fields corresponding to a first data source (Hive database) are ID (identity) String and name (name) String, index column fields corresponding to a second data source (Mysql database) are ID (identity) String and age (Int), and the associated field is ID (identity) String.

In one embodiment, a timed task may also be created, such as a 7-point-a-day-morning trigger task, to acquire synchronization data.

Referring to fig. 2, an exemplary target data importing system may mainly include: the system comprises a scheduling module, a configuration module, a data source module and a data consumption module. The relevant functions can be realized by using various corresponding middleware in each module, for example, a configuration module can adopt a timing scheduling component, a scheduling module can adopt a Zookeeper, and a data source module can adopt a Kafka-connector. Wherein the timing scheduling component is operative to implement triggering of the timing scheduling task.

The data source module may be comprised of at least one data source node and the data consuming module may be comprised of at least one data consuming node, collectively referred to as a worker node.

In one possible implementation, the working nodes include a data source node and a data consumption node, the data source node is used for acquiring target data from a plurality of data sources (such as Hive database and Mysql database), and the data consumption node is used for importing the target data into a service system (such as elastic search) corresponding to a target task.

Referring to fig. 2, in one possible implementation, the data source node is further configured to: the target Data is sent to the Data consuming node via a Data Channel (Data Channel). The process of transmitting data over the data channel may include: the sender stores the target data appointed to be sent in an appointed area and sets an access path; the receiver acquires the target data according to the access path.

In addition, data may also be sent over the data channel between the data source and the data source module, and between the data consumption module and the elastic search. The related functions can be implemented by using corresponding middleware in each data channel, for example, kafka is a high-throughput distributed publish-subscribe message system, and the data channels can implement the data transmission functions by using Kafka (Kafka).

Referring to fig. 2, the scheduling module adopts a Zookeeper design, and relates to distributed work node mounting, task mounting and task distribution. The Zookeeper is a distributed, open source distributed application coordination service, which is software that provides a consistency service for distributed applications. Fig. 5 schematically illustrates a distributed working node design diagram of a target data importing method according to an embodiment of the present application. As shown in fig. 5, the root node is easy-search. The data-sync in the second-level directory is a data source module, the es-data-sync-consumer is a data consumption module, and the easy-search is a configuration module. In the respective third level directory of each second level directory, the master represents the master node; workbench nodes are used for downloading and mounting working nodes; task nodes are mounted on tasks; the assigntsks mount tasks already allocated to each working node; the deletetasks are mounted with task identifiers for ending the synchronous tasks and are used for recovering system resources; the assgin mounts the work nodes that have been assigned to the respective tasks.

When each module is started, a work node for executing the target task is mounted on the worker node of each module. The working nodes are divided into a master node and a slave node, and the master node is responsible for tracking the state of the slave node and the validity of tasks and distributing the tasks to be executed to the slave node.

The following functions can be realized by adopting a Zookeeper design:

1. and distributing the task, namely informing the data source module and the data consumption module to start the working node thread to start working when the task is newly added in the task node.

2. The task state is maintained. The module is started or crashed in the data synchronization process, and the task can be transferred to other working nodes.

Referring to fig. 2, the configuration module is used to create a task, specify which data source to acquire data from, which data to acquire, whether to acquire data at any time, and so on when creating the task. In addition, tasks may also be triggered by the configuration module, such as manual triggers or timed scheduled task triggers. When a task is triggered, firstly, a task identifier is mounted in a task node; the configuration module master node monitors that the task node of the mounting task identifier changes, triggers the mounting of the target data acquisition task to the data source module, and triggers the mounting of the target data consumption task to the data consumption module.

Referring to fig. 2, in the process of acquiring data, the data source module may perform format conversion on the target data to mask the difference between the data sources, and form a consistent data format for providing to the data consumption module. The master node of the data source module monitors the task node change of the data source module, performs task allocation, starts the working node to acquire target data from the data source, converts the target data from different data sources into a uniform format after processing and processing, and then sends the target data to the data channel.

Referring to fig. 2, the data consumption module is configured to obtain target data from the data channel and write the target data into the search engine. For example, in the case of an update operation for target data, such as adding, deleting, modifying, etc., the update operation may be mapped into the search engine, and the index is added, deleted, modified according to the data in the data channel. The master node of the data consumption module monitors the task node change of the data consumption module, and if the task node change occurs, the target data in the thread consumption data channel is started.

In the example of FIG. 2, a user or timing schedule component initiates creation of a target task through a configuration module, which monitors the created target task, parses the target task, and performs a pre-configuration. The pre-configuration comprises: the target task is performed by which data source module and data consumption module, and from which data source the data source module obtains data, which index the data consumption module stores data into the elastic search, and the data channel for transmitting the target data. The scheduling module assigns the target task to the corresponding data source module and data consumption module, instructs the data source module to acquire data from which data source, the data consumption module to store the data in which index of the elastic search, and the data channel for transmitting the target data. The data source module executes the task according to the received target task, acquires target data from the designated data source, converts the target data format after acquiring the corresponding target data, and sends the target data to the data consumption module through the data channel.

Fig. 6 schematically illustrates a flowchart of one implementation of a target data importing method for obtaining target data from a plurality of data sources according to an embodiment of the present application. As shown in fig. 6, in one possible implementation, obtaining target data from a plurality of data sources includes:

step S61: acquiring a first data table containing first offline data corresponding to an index column field in a first data source;

step S62: acquiring second offline data corresponding to the index column field from a second data source according to a preset period, establishing a second data table in the first data source, and storing the second offline data into the second data table;

step S63: and combining the first data table and the second data table through the association field to obtain joint data, and taking the joint data as target data.

In general, executing a target task requires importing multi-source and multi-relationship target data into a service system corresponding to the target task. Wherein the multi-source target data may include offline data and real-time data. The offline data may include data generated in t+1 time units. For example, T herein may be understood as daily. T+1 can be understood as data before the setting time of today (for example, early morning) is data of T units. The data generated after the new day, that is, the data generated between the setting time of today and the setting time of the next day, that is, the data of (t+1) time units. In one example, mysql data generates a copy of mirrored data in the early morning every day, i.e., offline data generated in T+1 time units. The real-time data may include data that changes in real-time, e.g., the Mysql data source may be able to respond to the change immediately after the data in the Mysql database is modified. The target data for the multiple relations may comprise data tables of different databases, e.g. table a in the Hive database and table B in the Mysql database. On the one hand, before importing the target data into the service system, the target data needs to be denormalized, for example, the target data is subjected to a merging operation. On the other hand, the real-time requirements for obtaining data from a data source are also different. For the above reasons, the data needs to be masked from the difference when being imported, and the synchronization processing is performed. Specific examples of the differences in data are shown in table 1.

TABLE 1

Data source	Index recording	Real-time requirements
			Single hive table	Partial columns of Hive Table	T+1
Single mysql table	Column of Mysql Table	Real time
			Hive+mysql	Hive tablePartial column and column of mysql table	Real time
Hive+Hive	Partial columns of two hive tables	T+1
			Mysql+Mysql	Partial columns of two mysql tables	Real time

Because of the variability of the multi-source target data, the variability needs to be masked during the process of the data source module acquiring the data and the target data needs to be denormalized before the data is imported. Taking joint indexing of offline data and real-time data as an example, fig. 7 schematically illustrates a flowchart of an implementation of a data source node acquiring data according to a target data importing method according to an embodiment of the present application. In the example of fig. 7, the first data source is a Hive database, the second data source is a Mysql database, and the offline data flow is as follows:

in step S61, a first data table of the Hive database including first offline data corresponding to the index column field is obtained. Reference numeral 1 in fig. 7 denotes schema information corresponding to an index column field of table a in the Hive database. The index column fields specified by the user in Table A in the data of the Hive database include: ID: string, name: string. And according to the table A in the Hive database, acquiring a first data table containing first offline data corresponding to the index column field, namely acquiring the offline data of the table A.

In step S62, second offline data corresponding to the index column field is obtained from the Mysql database according to the preset period, a second data table is built in the Hive database, and the second offline data is stored in the second data table. Reference numeral 2 in fig. 7 denotes schema information corresponding to the index column field of table B in the Mysql database. The index column field specified by the user in table B in the Mysql database includes: ID is String, age is Int. The Mysql data source is real-time data, and aiming at a table B in the Mysql database, second offline data corresponding to the index column field is obtained according to a preset period. For example, offline data for Table B in the Mysql database may be generated in the early morning daily, i.e., data for (T+1) time units is generated. The data of the (t+1) time unit may be simply referred to as "t+1 mirror data".

After generating the data of the (T+1) time unit, the offline data of the table B in the Mysql database is stored into a second data table established in the Hive database to form the table B in the Hive database. Reference numeral 4 in fig. 7 denotes schema information corresponding to the index column field of table B in the Hive database. To form a wide table, both tables a and B need to be stored in the Hive database. The broad table comprises database tables with relatively large fields, and generally refers to a database table with related indexes, dimensions and attributes of a business theme associated together. In one embodiment, table B may be stored in the Hive database when a target task is triggered. In another embodiment, table B may also be stored in the Hive database at preset periods.

In step S63, the first data table and the second data table are combined through the association field to obtain joint data, and the joint data is used as target data. Reference numeral 5 in fig. 7 denotes schema information corresponding to an index column field of the joint data obtained by performing the merging operation on the table a and the table B in the Hive database. In this step, table A in the Hive database and table B in the Hive database are denormalized to form a joint data table of table A and table B in the Hive database.

After step S63, the joint data table of tables A and B in the Hive database is imported into the data source module. Referring to fig. 2, finally, the joint data table is imported into the service system corresponding to the target task through the data source module and the data consumption module. For example, for a search target task, an index may be built in a search engine based on the search target task, and then the federated data table is imported into the corresponding index. Reference numeral 3 in fig. 7 denotes schema information corresponding to an index column field in a service system corresponding to a target task.

In one possible implementation, the merging operation includes:

Example data for the first data table (table a), the second data table (table B) and the federated data table are as follows:

table A

Sequence number	ID	name
			1	050104	Li Si
2	050103	Zhang San

Table B

Sequence number	ID	age
			1	050103	13
2	050104	14

Federated data table

Sequence number	ID	name	age
				1	050103	Zhang San	13
2	050104	Li Si	14

The fields in Table A include an ID and a name, and the fields in Table B include an ID and an age. The associated fields of tables A and B are IDs. The first field values 050103 and 050104 corresponding to the associated field ID are obtained from table a and the second field values 050103 and 050104 corresponding to the associated field ID are obtained from table B. In the case where the first field value is the same as the second field value, for example, for field value 050103, the first record in table a containing first field value 050103 is the record in table a with number 2, and the record in table a with number 2 (ID: 050103, name: zhang San) is added to the second record in table B containing second field value 050103, that is, to the record in table B with number 1 (ID: 050103, age: 13), resulting in the record in joint data table with number 1 (ID: 050103, name: zhang San, age: 13).

Referring to fig. 6, in one possible implementation, obtaining target data from a plurality of data sources further includes:

Step S64: and adding real-time data corresponding to the index column field in the second data source updated after the second offline data is acquired into the target data.

Referring to fig. 6 and 7, the target data generated through the offline data flow does not include new data generated by the change of the data source after the table B of the Hive data source is generated, that is, does not include real-time data corresponding to the index column field in the second data source updated after the second offline data is acquired. Therefore, data compensation is needed to be performed, and new data which changes with time is compensated into target data.

In one possible implementation, adding real-time data to the target data includes:

Referring to fig. 6, 7 and 2, in step S64, an exemplary real-time data flow is as follows:

1) And sending the data of which the Mysql data source table B is changed after the T+1 mirror image data is generated to a message system. The data source module acquires the changed data from the message system and sends the changed data to the data consumption module.

2) The received data from the messaging system is imported into the index of the search engine by the data source module and the data consumption module.

After the data is complemented, the index in the search engine can be provided for the user to use. Meanwhile, if the Mysql data source table B changes, the changed data can be imported into the index of the search engine in real time according to the real-time data flow.

Other types of data synchronization include: the offline single-table synchronization, the real-time single-table synchronization, the offline multi-table synchronization and the real-time multi-table synchronization can be adapted according to the offline data flow and/or the real-time data flow.

In summary, the embodiment of the application can import the data from the plurality of data sources into the service system corresponding to the target task, and can generalize and process the data from various data sources to realize data synchronization. When one service is added, a specific customized processing method does not need to be developed again for the data which needs to be imported by the service. For various services, a developer does not need to know specific service logic. The developer does not need to maintain each business separately. The target data importing method improves the universality, expandability and usability of the system and reduces the labor cost.

Exemplary Medium

Having described the method of the exemplary embodiments of the present application, next, a medium of the exemplary embodiments of the present application will be described with reference to fig. 8.

In some possible embodiments, the various aspects of the present application may also be implemented as a computer-readable medium, on which a program is stored, which when executed by a processor is configured to implement the steps in the target data importing method according to the various exemplary embodiments of the present application described in the "exemplary method" section of the present specification.

Specifically, the processor is configured to implement the following steps when executing the program: analyzing the monitored target task to obtain a plurality of data sources corresponding to the target task; and configuring a working node for the target task, wherein the working node is used for acquiring target data from a plurality of data sources and importing the target data into a service system corresponding to the target task.

It should be noted that: the medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

As shown in fig. 8, a medium 80 according to an embodiment of the present application is described that may employ a portable compact disc read-only memory (CD-ROM) and that includes a program and that may run on a device. However, the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take many forms, including, but not limited to: electromagnetic signals, optical signals, or any suitable combination of the preceding. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the context of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN).

Exemplary apparatus

Having described the media of the exemplary embodiments of the present application, next, an apparatus of the exemplary embodiments of the present application will be described with reference to fig. 9.

Fig. 9 schematically illustrates a schematic structure of a target data importing apparatus according to an embodiment of the present application, including:

the parsing unit 901 is configured to parse the monitored target task to obtain a plurality of data sources corresponding to the target task;

the configuration unit 902 is configured to configure a working node for a target task, where the working node is configured to obtain target data from multiple data sources, and import the target data into a service system corresponding to the target task.

In a possible implementation, the configuration unit 902 is further configured to:

In one possible implementation, the working node is further configured to:

In one possible implementation, the merging operation includes:

In one possible implementation, the working node is further configured to:

In one possible implementation, the working node includes a data source node and a data consumption node, the data source node is used for acquiring target data from a plurality of data sources, and the data consumption node is used for importing the target data into a service system corresponding to a target task.

In one possible implementation, the data source node is further configured to: and sending the target data to the data consumption node through the data channel.

Exemplary computing device

Having described the methods, media, and apparatus of exemplary embodiments of the present application, a computing device of exemplary embodiments of the present application is next described with reference to fig. 10.

Those skilled in the art will appreciate that the various aspects of the present application may be implemented as a system, method, or program product. Accordingly, aspects of the present application may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.

In some possible implementations, a computing device according to embodiments of the present application may include at least one processing unit and at least one storage unit. Wherein the storage unit stores program code which, when executed by the processing unit, causes the processing unit to perform the steps in the target data importing method according to various exemplary embodiments of the present application described in the section "exemplary method" above in the present specification.

A computing device 100 according to such an implementation of the present application is described below with reference to fig. 10. The computing device 100 shown in fig. 10 is only one example and should not be taken as limiting the functionality and scope of use of embodiments of the present application.

As shown in fig. 10, the computing device 100 is in the form of a general purpose computing device. Components of computing device 100 may include, but are not limited to: the at least one processing unit 1001 and the at least one memory unit 1002 are connected to a bus 1003 that connects different system components (including the processing unit 1001 and the memory unit 1002).

Bus 1003 includes a data bus, a control bus, and an address bus.

The storage unit 1002 may include readable media in the form of volatile memory, such as Random Access Memory (RAM) 10021 and/or cache memory 10022, and may further include readable media in the form of non-volatile memory, such as Read Only Memory (ROM) 10023.

The storage unit 1002 may also include a program/utility 10025 having a set (at least one) of program modules 10024, such program modules 10024 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

Computing device 100 may also communicate with one or more external devices 1004 (e.g., keyboard, pointing device, etc.). Such communication may occur through an input/output (I/O) interface 1005. Moreover, computing device 100 may also communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 10010. As shown in fig. 10, the network adapter 10010 communicates with other modules of the computing device 100 via the bus 1003. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in connection with computing device 100, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

It should be noted that although in the above detailed description several units/modules or sub-units/sub-modules of the target data importing apparatus are mentioned, such a division is only exemplary and not mandatory. Indeed, the features and functionality of two or more units/modules described above may be embodied in one unit/module according to embodiments of the present application. Conversely, the features and functions of one unit/module described above may be further divided into ones that are embodied by a plurality of units/modules.

Furthermore, although the operations of the methods of the present application are depicted in the drawings in a particular order, this is not required to or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.

While the spirit and principles of this application have been described with reference to several particular embodiments, it is to be understood that this application is not limited to the disclosed particular embodiments nor does it imply that features in the various aspects are not useful in combination, nor are they intended to be in any way useful for the convenience of the description. The application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A target data importing method, comprising:

configuring a working node for the target task, wherein the working node is used for acquiring target data from the plurality of data sources and importing the target data into a service system corresponding to the target task;

Before analyzing the monitored target task, the method further comprises the following steps:

acquiring an index column field corresponding to the target task, wherein the index column field is specified in the data fields of the plurality of data sources, and the index column field corresponding to the target task comprises an index column field corresponding to each data source in the plurality of data sources;

determining a common field of the plurality of data sources according to an index column field corresponding to the target task, and obtaining an associated field based on the common field, wherein the associated field is used for carrying out merging operation on target data acquired from the plurality of data sources;

and creating the target task according to the service system corresponding to the target task, the index column field, the data source corresponding to the index column field and the association field.

2. The method of claim 1, wherein the obtaining target data from the plurality of data sources comprises:

acquiring a first data table containing first offline data corresponding to the index column field in a first data source;

And combining the first data table and the second data table through the association field to obtain joint data, and taking the joint data as the target data.

3. The method of claim 2, wherein the merging operation comprises:

acquiring a first field value corresponding to the associated field from the first data table, and acquiring a second field value corresponding to the associated field from the second data table;

adding a first record containing the first field value in the first data table to the second data table containing the first field value in the case that the first field value is the same as the second field value

In a second record of the second field value.

4. A method according to claim 2 or 3, wherein said obtaining target data from said plurality of data sources further comprises:

5. The method of claim 4, wherein adding real-time data to the target data comprises:

acquiring the real-time data from a message system, and adding the acquired real-time data into the target data; wherein the message system is used for accessing the real-time data.

6. A method according to any one of claims 1 to 3, wherein the working nodes comprise data source nodes for acquiring target data from the plurality of data sources and data consuming nodes for importing the target data into a service system to which the target task corresponds.

7. The method of claim 6, wherein the data source node is further configured to: and sending the target data to the data consumption node through a data channel.

8. A target data importing apparatus, comprising:

the configuration unit is used for configuring a working node for the target task, wherein the working node is used for acquiring target data from the plurality of data sources and importing the target data into a service system corresponding to the target task;

wherein the configuration unit is further configured to:

according to the service system, the index column field and the index column word corresponding to the target task

And the data source corresponding to the segment and the associated field create the target task.

9. The apparatus of claim 8, wherein the working node is further configured to:

10. The apparatus of claim 9, wherein the combining operation comprises:

and adding a first record containing the first field value in the first data table to a second record containing the second field value in the second data table under the condition that the first field value is the same as the second field value.

11. The apparatus of claim 9 or 10, wherein the working node is further configured to:

12. The apparatus of claim 11, wherein the working node is further configured to:

13. The apparatus according to any one of claims 8 to 10, wherein the working node comprises a data source node for acquiring target data from the plurality of data sources and a data consuming node for importing the target data into a service system corresponding to the target task.

14. The apparatus of claim 13, wherein the data source node is further configured to: and sending the target data to the data consumption node through a data channel.

15. A medium storing a computer program, which when executed by a processor performs the method of any one of claims 1-7.

16. A computing device, comprising:

one or more processors;

a storage means for storing one or more programs;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7.