CN111984740A - Business data table processing method and device, computer equipment and storage medium - Google Patents

Business data table processing method and device, computer equipment and storage medium Download PDF

Info

Publication number
CN111984740A
CN111984740A CN202010736508.3A CN202010736508A CN111984740A CN 111984740 A CN111984740 A CN 111984740A CN 202010736508 A CN202010736508 A CN 202010736508A CN 111984740 A CN111984740 A CN 111984740A
Authority
CN
China
Prior art keywords
task
data table
instruction
external
integrated
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010736508.3A
Other languages
Chinese (zh)
Other versions
CN111984740B (en
Inventor
林健昌
邓超
张宗祺
张鹏
姜良雷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhaolian Consumer Finance Co ltd
Original Assignee
Merchants Union Consumer Finance Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Merchants Union Consumer Finance Co Ltd filed Critical Merchants Union Consumer Finance Co Ltd
Priority to CN202010736508.3A priority Critical patent/CN111984740B/en
Publication of CN111984740A publication Critical patent/CN111984740A/en
Application granted granted Critical
Publication of CN111984740B publication Critical patent/CN111984740B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G06F16/285—Clustering or classification
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The application relates to a method and a device for processing a business data table, computer equipment and a storage medium, wherein data information of the business data table to be processed is obtained; searching an integration rule corresponding to the business data table to be processed, and judging an integration processing task type corresponding to the business data table to be processed; then, a first integrated processing task corresponding to the business data table to be processed is newly established; and when a task switching instruction corresponding to the first integrated processing task is received, controlling the first external data integrated system to delete the first integrated processing task and newly establish a second integrated processing task. The method and the device classify the business data tables according to the integration rules corresponding to the business data tables, and then establish corresponding data integration tasks on corresponding data integration systems according to classification results to process the business data tables. When the task processing types need to be switched, the user can perform one-key switching through the corresponding instruction, so that the complexity of the task switching process is reduced, and the processing efficiency of the business data table is improved.

Description

Business data table processing method and device, computer equipment and storage medium
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data updating method, apparatus, computer device, and storage medium.
Background
With the rapid development of the internet and the rapid increase of data, people gradually recognize the strategic importance of data in the development of the whole internet. For the requirement of integrating the business database data into the data warehouse, the general company configures the data integration tasks of the batch on a dispatching system similar to azkaban. However, with the increasing data volume, the batch tasks and data volume of the data warehouse are increased in the early morning, the pressure on the resource requirements in the peak period is increased, and the completion time is also prolonged. Aiming at the problem, most tasks of batch running in the peak period every day can be replaced by quasi real-time tasks, so that the pressure in the peak period in the morning is relieved, and the efficiency of batch running of the tasks with high priority is improved.
However, some special service tables have batch running or end-of-day service tables, which do not meet the requirement of quasi-real time, and can only run tasks off line through batch, so the integration is divided into two task types of quasi-real time and off line, and the two task types can be switched with each other.
For the process of switching from the off-line task to the quasi-real-time task, the tasks of configuring the real-time data path, manually establishing a table view and tasks, and then configuring the off-line task are needed one by one, so that the steps are complicated, and the processing efficiency is low.
Disclosure of Invention
Therefore, in order to solve the above technical problems, it is necessary to provide a method, an apparatus, a computer device, and a storage medium for processing a service data table, which can effectively improve the efficiency of switching between an offline task and a quasi-real-time task.
A method for processing a service data table, the method comprising:
acquiring data information of a to-be-processed business data table;
searching an integration rule corresponding to the business data table to be processed, and judging an integration processing task type corresponding to the business data table to be processed according to the data information;
sending a first task generation instruction to an external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table, wherein the first task generation instruction is used for newly building a first integrated processing task corresponding to the to-be-processed service data table on the external first data integration system, and the first integrated processing task is used for performing integrated processing on the to-be-processed service data table;
when a task switching instruction corresponding to the first integrated processing task is received, according to the task switching instruction, sending a first task deleting instruction to an external first data integration system corresponding to the integrated processing task type, and sending a second task generating instruction to an external second data integration system corresponding to the integrated processing task type, where the first task deleting instruction is used to delete the first integrated processing task in the first external data integration system, and the second task generating instruction is used to newly create a second integrated processing task corresponding to the task switching instruction in the second external data integration system.
In one embodiment, the acquiring data information of the to-be-processed service data table includes:
scanning service metadata in a preset storage position;
identifying a newly added service data table to be processed according to the service metadata;
and acquiring data information corresponding to the to-be-processed service data table.
In one embodiment, the searching for the integration rule corresponding to the to-be-processed service data table, and the determining, according to the data information, the type of the integration processing task corresponding to the to-be-processed service data table includes:
searching an integration rule corresponding to the business data table to be processed;
and sequentially inquiring the integration rules according to the data information, and determining the type of the integration processing task corresponding to the to-be-processed business data table according to the inquiry result.
In one embodiment, the sending the first task generation instruction to the external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table includes:
when the integrated processing task type corresponding to the to-be-processed service data table is a quasi-real-time integrated task, sending a first quasi-real-time task generation instruction to an external real-time data access system, wherein the first quasi-real-time task generation instruction is used for newly adding a quasi-real-time integrated task corresponding to the to-be-processed service data table in the external real-time data access system and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task;
and when the integrated processing task type corresponding to the to-be-processed service data table is an offline integrated task, sending a first offline task generation instruction to an external offline scheduling system, wherein the first offline task generation instruction is used for newly adding an offline scheduling task corresponding to the to-be-processed service data table in the external offline scheduling system and newly building a temporary table and a target table corresponding to the offline integrated task.
In one embodiment, the sending the first task deletion instruction to the external first data integration system corresponding to the integrated processing task type and the sending the second task generation instruction to the external second data integration system corresponding to the integrated processing task type according to the task switching instruction when receiving the task switching instruction corresponding to the first integrated processing task includes:
when a task switching instruction corresponding to a quasi-real-time integrated task is received, sending a first quasi-real-time task deleting instruction to the external real-time data access system, and sending a second offline task generating instruction to the external offline scheduling system, wherein the first quasi-real-time task deleting instruction is used for controlling the external real-time data access system to delete the quasi-real-time integrated task, and the second offline task generating instruction is used for newly adding an offline scheduling task corresponding to the to-be-processed service data table in the external offline scheduling system, and newly building a temporary table and a target table corresponding to the quasi-real-time integrated task;
when a task switching instruction corresponding to an offline integrated task is received, a first offline task deleting instruction is sent to the external offline scheduling system, and a second quasi-real-time task generating instruction is sent to the external real-time data access system, wherein the first offline task deleting instruction is used for controlling the external offline scheduling system to delete the offline integrated task, the second quasi-real-time task generating instruction is used for newly adding a quasi-real-time integrated task corresponding to the to-be-processed service data table in the external real-time data access system, and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task.
In one embodiment, after the sending a first task generation instruction to an external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table, the method further includes:
searching historical data corresponding to the to-be-processed business data table;
and sending an initialization instruction to an external first data integration system corresponding to the type of the integrated processing task according to the historical data corresponding to the to-be-processed service data table, wherein the initialization instruction performs initialization processing on the service data table created by the first integrated processing task.
A traffic data table processing apparatus, the apparatus comprising:
the data acquisition module is used for acquiring data information of a service data table to be processed;
the task type identification module is used for searching an integration rule corresponding to the to-be-processed business data table and judging an integration processing task type corresponding to the to-be-processed business data table according to the data information;
a task creating module, configured to send a first task generating instruction to an external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table, where the first task generating instruction is used to create a first integrated processing task corresponding to the to-be-processed service data table on the external first data integration system, and the first integrated processing task is used to perform integrated processing on the to-be-processed service data table;
and a task switching module, configured to send a first task deletion instruction to an external first data integration system corresponding to the type of the integrated processing task and send a second task generation instruction to an external second data integration system corresponding to the type of the integrated processing task according to the task switching instruction when receiving a task switching instruction corresponding to the first integrated processing task, where the first task deletion instruction is used to delete the first integrated processing task in the first external data integration system, and the second task generation instruction is used to newly create a second integrated processing task corresponding to the task switching instruction in the second external data integration system.
In one embodiment, the data acquisition module is specifically configured to:
scanning service metadata in a preset storage position;
identifying a newly added service data table to be processed according to the service metadata;
and acquiring data information corresponding to the to-be-processed service data table.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
acquiring data information of a to-be-processed business data table;
searching an integration rule corresponding to the business data table to be processed, and judging an integration processing task type corresponding to the business data table to be processed according to the data information;
sending a first task generation instruction to an external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table, wherein the first task generation instruction is used for newly building a first integrated processing task corresponding to the to-be-processed service data table on the external first data integration system, and the first integrated processing task is used for performing integrated processing on the to-be-processed service data table;
when a task switching instruction corresponding to the first integrated processing task is received, according to the task switching instruction, sending a first task deleting instruction to an external first data integration system corresponding to the integrated processing task type, and sending a second task generating instruction to an external second data integration system corresponding to the integrated processing task type, where the first task deleting instruction is used to delete the first integrated processing task in the first external data integration system, and the second task generating instruction is used to newly create a second integrated processing task corresponding to the task switching instruction in the second external data integration system.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
acquiring data information of a to-be-processed business data table;
searching an integration rule corresponding to the business data table to be processed, and judging an integration processing task type corresponding to the business data table to be processed according to the data information;
sending a first task generation instruction to an external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table, wherein the first task generation instruction is used for newly building a first integrated processing task corresponding to the to-be-processed service data table on the external first data integration system, and the first integrated processing task is used for performing integrated processing on the to-be-processed service data table;
when a task switching instruction corresponding to the first integrated processing task is received, according to the task switching instruction, sending a first task deleting instruction to an external first data integration system corresponding to the integrated processing task type, and sending a second task generating instruction to an external second data integration system corresponding to the integrated processing task type, where the first task deleting instruction is used to delete the first integrated processing task in the first external data integration system, and the second task generating instruction is used to newly create a second integrated processing task corresponding to the task switching instruction in the second external data integration system.
The business data table processing method, the device, the computer equipment and the storage medium acquire the data information of the business data table to be processed; searching an integration rule corresponding to the business data table to be processed, and judging an integration processing task type corresponding to the business data table to be processed according to the data information; sending a first task generation instruction to an external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table; when a task switching instruction corresponding to the first integrated processing task is received, a first task deleting instruction is sent to an external first data integration system corresponding to the integrated processing task type according to the task switching instruction, and a second task generating instruction is sent to an external second data integration system corresponding to the integrated processing task type. The business data sheet processing method classifies the business data sheets according to the integration rules corresponding to the business data sheets, and then establishes corresponding data integration tasks on corresponding data integration systems according to classification results to process the business data sheets. When the task processing types need to be switched, the user can perform one-key switching through the corresponding instruction, so that the complexity of the task switching process is reduced, and the processing efficiency of the business data table is improved.
Drawings
FIG. 1 is a diagram of an exemplary implementation of a business data table processing method;
FIG. 2 is a flow chart illustrating a method for processing a business data table according to an embodiment;
FIG. 3 is a schematic sub-flow chart illustrating step 201 of FIG. 2 according to an embodiment;
FIG. 4 is a schematic sub-flow chart of step 203 of FIG. 2 in one embodiment;
FIG. 5 is a flow diagram illustrating an initialization process in one embodiment;
FIG. 6 is a flow diagram of a method for processing a business data table in a more specific embodiment;
FIG. 7 is a block diagram of an embodiment of a business data table processing apparatus;
FIG. 8 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The service data table processing method provided by the application can be applied to the application environment shown in fig. 1. Wherein the spreadsheet processing server 102 may communicate with the first external data integration system 104 and the second external data integration system 106 via a network. The data sheet processing server 102 may first obtain data information of a service data sheet to be processed; searching an integration rule corresponding to the business data table to be processed, and judging an integration processing task type corresponding to the business data table to be processed according to the data information; according to the type of the integrated processing task corresponding to the service data table to be processed, a first integrated processing task corresponding to the service data table to be processed is newly established on a corresponding external first data integrated system 104; when a task switching instruction corresponding to the first integrated processing task is received, controlling the first external data integration system 104 to delete the first integrated processing task according to the task switching instruction, and controlling the second external data integration system 106 to newly establish a second integrated processing task corresponding to the task switching instruction; and controlling the second external data integration system 106 to perform data integration processing on the to-be-processed service data table through the second integration processing task. The server 102, the first external data integration system 104, and the second external data integration system 106 may be implemented by separate servers or a server cluster composed of a plurality of servers.
In one embodiment, as shown in fig. 2, a method for processing a service data table is provided, which is described by taking the method as an example applied to the data table processing server 102 in fig. 1, and includes the following steps:
step 201, acquiring data information of a to-be-processed service data table.
The business data table to be processed is a processing object processed by the business data table, and the method is particularly applied to a data warehouse system and is suitable for integrating the business data contained in the business data table to be processed in a preset business warehouse into the data warehouse. The data information specifically comprises the data type of the service data table to be processed, data increment, whether the single service table has a day end, whether the single service table runs in batch, whether the data table has special purposes and the like.
In particular, the data table processing server 102 may specifically operate on a database within the business system and a database within the data warehouse. The method can be particularly used for data integration, such as controlling which integration type is used to import data of a new generation table in a business system into a database. Specifically, the data table processing server may identify which data tables in the service system are to-be-processed service data tables, and then obtain data information corresponding to the to-be-processed service data tables, where the data information may be data corresponding to the service data tables newly created when the service system makes the service data tables. At this time, the data table processing server can look up the table to obtain the data information. Or some search rules of the data information may be preset, and then the data information of the service data table to be processed, such as daily increment data of the service data table, is obtained according to the search rules and is used as one data information.
Step 203, searching an integration rule corresponding to the to-be-processed service data table, and judging an integration processing task type corresponding to the to-be-processed service data table according to the data information.
The integration rule is a rule for judging how to perform integration processing on the to-be-processed service data table, and specifically includes an offline integration rule and a quasi-real-time integration rule. Whether offline integration is carried out on the business table to be processed or not can be judged through the offline integration rule, whether quasi real-time integration is carried out on the business table to be processed or not can be judged through the quasi real-time integration rule, and the integration rule specifically comprises whether daily increment of a single business table is larger than a preset threshold value or not, whether the business table is an important data table or not, whether the business table has a terminal date or not and the like.
Specifically, the data sheet processing server 102 has pre-stored therein integration rules, which are pre-configured by the integration processing staff according to real-time business needs. The staff can configure the corresponding business data table integration rule to be processed by time intervals or business system types. After the data information of the to-be-processed service data table is obtained, the integration rule corresponding to the to-be-processed service data table can be further searched, and then the integration processing task type of the to-be-processed service data table is determined based on the obtained data information. If the configured integration rule corresponding to the to-be-processed service data table is specifically an offline integration rule, the configured integration rule includes that "when the daily increment of the single table is more than 1000 ten thousand, the corresponding integration processing task type is offline integration", "when the type of the to-be-processed service data table is a non-important table, the corresponding integration processing task type is offline integration", and the like. When the data information of the to-be-processed service data table shows that the to-be-processed service data table meets the integration rules, the type of the integration processing task of the current to-be-processed service data table can be judged to be offline integration processing.
Step 205, according to the type of the integrated processing task corresponding to the to-be-processed service data table, sending a first task generation instruction to an external first data integration system corresponding to the type of the integrated processing task, where the first task generation instruction is used to newly create a first integrated processing task corresponding to the to-be-processed service data table on the external first data integration system, and the first integrated processing task is used to perform integrated processing on the to-be-processed service data table.
The external first data integration system may be specifically a real-time data access system or an offline scheduling system, and these two systems are subsystems included in the data warehouse system and used for data integration processing. The two systems can be respectively connected with the data sheet processing server 102 through a network, the data sheet processing server 102 can establish corresponding integrated processing tasks on the two data integration systems through corresponding control instructions, and then the data integration systems perform integrated processing on the service data in the service data sheet to be processed.
Specifically, after the integrated processing task type corresponding to the to-be-processed service data table is determined, an external first data integration system may be called to establish a corresponding integrated processing task based on the integrated processing task type. If the integrated processing task type corresponding to the to-be-processed service data table is the offline processing type, the external first data integrated system corresponding to the to-be-processed service data table is the offline scheduling system. At this time, the data table processing server 102 may newly create a first integrated processing task for performing integrated processing on the to-be-processed service data table on the offline scheduling system. The first external data integration system can be controlled to perform data integration processing on the business data table to be processed based on the established first integrated processing task.
Step 207, when receiving a task switching instruction corresponding to the first integrated processing task, sending a first task deleting instruction to an external first data integration system corresponding to the integrated processing task type according to the task switching instruction, and sending a second task generating instruction to an external second data integration system corresponding to the integrated processing task type, where the first task deleting instruction is used to delete the first integrated processing task in the first external data integration system, and the second task generating instruction is used to newly create a second integrated processing task corresponding to the task switching instruction in the second external data integration system.
The task switching instruction is used for switching the processing type of the service data table to be processed, for example, switching an offline integrated task into a quasi-real-time integrated task, or switching the quasi-real-time integrated task into an offline integrated task.
Specifically, when the staff finds that the integration type of the to-be-processed service data table needs to be switched, the task type of the integration task may be switched by sending a task switching instruction corresponding to the first integration processing task to the data table processing server 102. When the data sheet processing server 102 receives the task switching instruction, the external data integrated system may be processed based on the task switching instruction. At this time, the first external data integration system performs integrated processing on the data in the to-be-processed service data table based on the first integrated processing task. At this time, the data table processing server 102 may directly delete the first integration task in the service data table to be processed, and clear the integrated data. And then controlling a second external data integration system to newly build a second integration processing task corresponding to the task switching instruction, and continuing integration processing.
The business data table processing method comprises the steps of obtaining data information of a business data table to be processed; searching an integration rule corresponding to the business data table to be processed, and judging an integration processing task type corresponding to the business data table to be processed according to the data information; sending a first task generation instruction to an external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table; when a task switching instruction corresponding to the first integrated processing task is received, a first task deleting instruction is sent to an external first data integration system corresponding to the integrated processing task type according to the task switching instruction, and a second task generating instruction is sent to an external second data integration system corresponding to the integrated processing task type. The business data sheet processing method classifies the business data sheets according to the integration rules corresponding to the business data sheets, and then establishes corresponding data integration tasks on corresponding data integration systems according to classification results to process the business data sheets. When the task processing types need to be switched, the user can perform one-key switching through the corresponding instruction, so that the complexity of the task switching process is reduced, and the processing efficiency of the business data table is improved.
In one embodiment, as shown in FIG. 3, step 201 comprises:
step 302, scanning the service metadata in the preset storage location.
And step 304, identifying the newly added service data table to be processed according to the service metadata.
And step 306, acquiring data information corresponding to the service data table to be processed.
The service metadata is metadata corresponding to the service data table, and the metadata is also called intermediary data and relay data, is data describing data, mainly information describing data attributes, and is used for supporting functions such as indicating storage locations, historical data, resource searching, file recording and the like. The service metadata specifically refers to metadata corresponding to a service data table in a database of the service system. When a service table is newly generated in the service database or data of the service table is changed, the data table processing server 102 may identify newly added data by scanning service metadata in a preset storage location, and then identify a newly added service data table to be processed according to the service metadata; and acquiring data information corresponding to the to-be-processed service data table. In the embodiment, the to-be-processed business data table is identified by scanning the metadata, so that the effect of identifying the to-be-processed business data table in time can be achieved.
In one embodiment, as shown in FIG. 4, step 203 comprises:
step 401, searching an integration rule corresponding to the service data table to be processed.
And 403, sequentially inquiring the integration rules according to the data information, and determining the integration processing task type corresponding to the to-be-processed service data table according to the inquiry result.
Specifically, when the type of the integrated processing task corresponding to the to-be-processed service data table is determined, the integrated rule corresponding to the to-be-processed service data table needs to be searched first, and the determination is performed according to the preset rule. The inquired integration rule is a rule or a plurality of integration rules corresponding to the off-line integration mode, or the inquired integration rule or the plurality of integration rules corresponding to the quasi-real-time integration mode. When a plurality of rules are configured, the rules are in an OR relationship, and the integration mode of the business data table can be determined as long as one of the rules is satisfied. In the prior art, the distinguishing integration processing is integrated in a quasi-real-time mode or an off-line mode, the quasi-real-time operation and the batch operation are identified manually, but the manual classification efficiency is low and errors are easy to occur. In the embodiment, the integration processing task types corresponding to the to-be-processed service data table are sequentially judged based on the integration rules obtained through query, and the integration processing efficiency can be effectively improved compared with manual classification processing.
In one embodiment, the integrated processing task types include an offline integrated task and a quasi-real-time integrated task, the first task generation instruction includes a first quasi-real-time task generation instruction and a first offline task generation instruction, and step 205 includes: when the integrated processing task type corresponding to the business data table to be processed is a quasi-real-time integrated task, sending a first quasi-real-time task generation instruction to an external real-time data access system, wherein the first quasi-real-time task generation instruction is used for newly adding a quasi-real-time integrated task corresponding to the business data table to be processed in the external real-time data access system and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task; and when the integrated processing task type corresponding to the to-be-processed service data table is an offline integrated task, sending a first offline task generation instruction to an external offline scheduling system, wherein the first offline task generation instruction is used for newly adding an offline scheduling task corresponding to the to-be-processed service data table in the external offline scheduling system and newly building a temporary table and a target table corresponding to the offline integrated task.
The kudu table, the view table and the hive table are tables required for executing the quasi-real-time integration task, and the temporary table and the target table are tables required for executing the offline integration task.
Specifically, when it is recognized that the type of the integrated processing task is quasi-real-time automatic integration, the data table processing server 102 may automatically add a quasi-real-time task and the like to the real-time data path system by sending a first quasi-real-time task generation instruction, and newly add a kudu table, a view table and a hive table required for integration. The real-time data path system can then execute the quasi-real-time task, and quasi-real-time integration of the business data table data in the business database into the data warehouse is carried out through the kudu table, the view and the hive table. And when the integration in an offline mode is identified, the offline integrated task can be automatically newly built on the offline scheduling system by sending a first offline task generation instruction, and a temporary table (stg table) and a target table (ods table) required by the offline integrated task are built. In this embodiment, the corresponding data integration system is called according to the type of the integrated processing task, and the corresponding processing task and the task table are processed by creating the new processing task. The processing efficiency of the integrated data service can be effectively improved.
In one embodiment, the first task deleting instruction includes a first quasi-real-time task deleting instruction and a first offline task deleting instruction, the second task generating instruction includes a second offline task generating instruction and a second quasi-real-time task generating instruction, and step 207 includes: when a task switching instruction corresponding to the quasi-real-time integrated task is received, a first quasi-real-time task deleting instruction is sent to an external real-time data access system, and a second offline task generating instruction is sent to an external offline scheduling system, wherein the first quasi-real-time task deleting instruction is used for controlling the external real-time data access system to delete the quasi-real-time integrated task, the second offline task generating instruction is used for newly adding a temporary table and a target table corresponding to the quasi-real-time integrated task in the external offline scheduling system, and the temporary table and the target table correspond to the to-be-processed service data table; when a task switching instruction corresponding to an offline integrated task is received, a first offline task deleting instruction is sent to an external offline scheduling system, and a second quasi-real-time task generating instruction is sent to an external real-time data access system, wherein the first offline task deleting instruction is used for controlling the external offline scheduling system to delete the offline integrated task, the second quasi-real-time task generating instruction is used for newly adding a quasi-real-time integrated task corresponding to a to-be-processed service data table in the external real-time data access system, and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task.
Specifically, when the user needs to switch the task type of the current integrated processing task. When a user wants to switch the quasi-real-time operation to an offline operation, the type of the integrated processing task corresponding to the first integrated processing task is the quasi-real-time integrated task. The user may directly send a task switching instruction corresponding to the first integrated processing task to the spreadsheet processing server 102, and then the spreadsheet processing server controls the service switching. Specifically, the spreadsheet processing server 102 may be connected to a display, and display all the business tables being processed in an integrated manner on the display, and the user may directly find the quasi-real-time integrated task corresponding to the first integrated processing task and switch the quasi-real-time integrated task to an offline state, at this time, the spreadsheet processing server determines that the task switching instruction is received. And then, a first quasi real-time task deleting instruction is sent to the external real-time data access system, a second offline task generating instruction is sent to the external offline scheduling system, so that the quasi real-time tasks corresponding to the offline of the external real-time data access system are controlled, the corresponding intermediate tables are deleted, the offline tasks and the corresponding intermediate tables are newly built in the external offline scheduling system, and the process is not perceived by a user. Similarly, when the user wants to switch the offline task to the quasi-real-time task, the user also supports one-key switching, can directly find the offline integrated task which finds the table, and can click to switch to the quasi-real-time task. At this time, the data table processing server 102 sends a first offline task deleting instruction to the external offline scheduling system, sends a second quasi real-time task generating instruction to the external real-time data access system, drops the offline task and deletes the corresponding intermediate table, and then newly establishes the quasi real-time task and the corresponding intermediate table, which is not perceived by the user.
In one embodiment, as shown in fig. 5, after step 205, the method further includes:
step 502, searching historical data corresponding to the business data table to be processed.
Step 504, an initialization instruction is sent to the external first data integration system corresponding to the type of the integrated processing task according to the historical data corresponding to the business data table to be processed, and the initialization instruction is used for initializing the business data table created by the first integrated processing task.
Specifically, a first integrated processing task corresponding to the to-be-processed service data table is newly built on the external first data integrated system, and since the newly-built first integrated processing task can only synchronize subsequent data in the service data table, in order to ensure that all data in the service data table can be integrated into the data warehouse, the newly-built data table in the first integrated processing task needs to be initialized, and the service data table is initialized according to the existing historical data in the to-be-processed service data table. The synchronism of the data is ensured. Meanwhile, when switching tasks, in this embodiment, when a table corresponding to the integrated task is newly created, initialization processing may also be performed through historical data existing in the service data table to be processed. The business data table created by the first integrated processing task is initialized through the historical data corresponding to the business data table to be processed, so that the business data table created by the first integrated processing task can be effectively ensured to be synchronous with the data in the business data table to be processed.
In a specific embodiment, a specific flowchart of the present application can be shown in fig. 6, wherein the data sheet processing server 102 can be specifically a data transmission console. Firstly, the data transmission console scans the metadata regularly, then finds the integration table needing to be newly added, and configures the corresponding classification rule. And when the configuration rule is not met, distributing the task table to a quasi-real-time acquisition system for data table integrated processing, and when the configuration rule is met, performing the data table integrated processing in a batch acquisition (offline acquisition) mode. Meanwhile, a corresponding integrated processing task can be newly established on the data transmission console, and when data switching is needed, one-key switching can be performed, so that the integrated processing efficiency is improved.
It should be understood that although the various steps in the flow charts of fig. 2-5 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-5 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.
In one embodiment, as shown in fig. 7, there is provided a service data table processing apparatus, including:
a data obtaining module 702, configured to obtain data information of a service data table to be processed;
the task type identification module 704 is configured to search an integration rule corresponding to the to-be-processed service data table, and determine an integration processing task type corresponding to the to-be-processed service data table according to the data information;
a task creating module 706, configured to create a first integrated processing task corresponding to the to-be-processed service data table on a corresponding external first data integrated system according to the integrated processing task type corresponding to the to-be-processed service data table;
the task switching module 708 is configured to, when receiving a task switching instruction corresponding to the first integrated processing task, send a first task deletion instruction to an external first data integration system corresponding to the integrated processing task type according to the task switching instruction, and send a second task generation instruction to an external second data integration system corresponding to the integrated processing task type, where the first task deletion instruction is used to delete the first integrated processing task at the first external data integration system, and the second task generation instruction is used to newly create a second integrated processing task corresponding to the task switching instruction at the second external data integration system.
In one embodiment, the data obtaining module 701 is specifically configured to: scanning service metadata in a preset storage position; identifying a newly added service data table to be processed according to the service metadata; and acquiring data information corresponding to the to-be-processed service data table.
In one embodiment, the task type identification module 703 is specifically configured to: searching an integration rule corresponding to the business data table to be processed; and sequentially inquiring the integration rules according to the data information, and determining the type of the integration processing task corresponding to the to-be-processed service data table according to the inquiry result.
In one embodiment, the integrated processing task types include an offline integrated task and a quasi-real-time integrated task, the first task generation instruction includes a first quasi-real-time task generation instruction and a first offline task generation instruction, and the task creating module 705 is specifically configured to: when the integrated processing task type corresponding to the business data table to be processed is a quasi-real-time integrated task, sending a first quasi-real-time task generation instruction to an external real-time data access system, wherein the first quasi-real-time task generation instruction is used for newly adding a quasi-real-time integrated task corresponding to the business data table to be processed in the external real-time data access system and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task; and when the integrated processing task type corresponding to the to-be-processed service data table is an offline integrated task, sending a first offline task generation instruction to an external offline scheduling system, wherein the first offline task generation instruction is used for newly adding an offline scheduling task corresponding to the to-be-processed service data table in the external offline scheduling system and newly building a temporary table and a target table corresponding to the offline integrated task.
In one embodiment, the first task deleting instruction includes a first quasi-real-time task deleting instruction and a first offline task deleting instruction, the second task generating instruction includes a second offline task generating instruction and a second quasi-real-time task generating instruction, and the task switching module 707 is specifically configured to: when a task switching instruction corresponding to the quasi-real-time integrated task is received, a first quasi-real-time task deleting instruction is sent to an external real-time data access system, and a second offline task generating instruction is sent to an external offline scheduling system, wherein the first quasi-real-time task deleting instruction is used for controlling the external real-time data access system to delete the quasi-real-time integrated task, the second offline task generating instruction is used for newly adding a temporary table and a target table corresponding to the quasi-real-time integrated task in the external offline scheduling system, and the temporary table and the target table correspond to the to-be-processed service data table; when a task switching instruction corresponding to an offline integrated task is received, a first offline task deleting instruction is sent to an external offline scheduling system, and a second quasi-real-time task generating instruction is sent to an external real-time data access system, wherein the first offline task deleting instruction is used for controlling the external offline scheduling system to delete the offline integrated task, the second quasi-real-time task generating instruction is used for newly adding a quasi-real-time integrated task corresponding to a to-be-processed service data table in the external real-time data access system, and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task.
In one embodiment, the data initialization module is further included to: searching historical data corresponding to a to-be-processed business data table; and sending an initialization instruction to an external first data integration system corresponding to the type of the integrated processing task according to the historical data corresponding to the to-be-processed service data table, wherein the initialization instruction is used for initializing the service data table created by the first integrated processing task.
For specific limitations of the service data table processing apparatus, reference may be made to the above limitations on the service data table processing method, which is not described herein again. All or part of each module in the service data table processing device can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 8. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing the business data table processing data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a business data table processing method.
Those skilled in the art will appreciate that the architecture shown in fig. 8 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:
acquiring data information of a to-be-processed business data table;
searching an integration rule corresponding to the business data table to be processed, and judging an integration processing task type corresponding to the business data table to be processed according to the data information;
according to the integrated processing task type corresponding to the to-be-processed business data table, sending a first task generation instruction to an external first data integration system corresponding to the integrated processing task type, wherein the first task generation instruction is used for newly building a first integrated processing task corresponding to the to-be-processed business data table on the external first data integration system, and the first integrated processing task is used for performing integrated processing on the to-be-processed business data table;
when a task switching instruction corresponding to a first integrated processing task is received, a first task deleting instruction is sent to an external first data integrated system corresponding to the integrated processing task type according to the task switching instruction, and a second task generating instruction is sent to an external second data integrated system corresponding to the integrated processing task type, wherein the first task deleting instruction is used for deleting the first integrated processing task in the first external data integrated system, and the second task generating instruction is used for newly building a second integrated processing task corresponding to the task switching instruction in the second external data integrated system.
In one embodiment, the processor, when executing the computer program, further performs the steps of: scanning service metadata in a preset storage position; identifying a newly added service data table to be processed according to the service metadata; and acquiring data information corresponding to the to-be-processed service data table.
In one embodiment, the processor, when executing the computer program, further performs the steps of: searching an integration rule corresponding to the business data table to be processed; and sequentially inquiring the integration rules according to the data information, and determining the type of the integration processing task corresponding to the to-be-processed service data table according to the inquiry result.
In one embodiment, the processor, when executing the computer program, further performs the steps of: when the integrated processing task type corresponding to the business data table to be processed is a quasi-real-time integrated task, sending a first quasi-real-time task generation instruction to an external real-time data access system, wherein the first quasi-real-time task generation instruction is used for newly adding a quasi-real-time integrated task corresponding to the business data table to be processed in the external real-time data access system and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task; and when the integrated processing task type corresponding to the to-be-processed service data table is an offline integrated task, sending a first offline task generation instruction to an external offline scheduling system, wherein the first offline task generation instruction is used for newly adding an offline scheduling task corresponding to the to-be-processed service data table in the external offline scheduling system and newly building a temporary table and a target table corresponding to the offline integrated task.
In one embodiment, the processor, when executing the computer program, further performs the steps of: when a task switching instruction corresponding to the quasi-real-time integrated task is received, a first quasi-real-time task deleting instruction is sent to an external real-time data access system, and a second offline task generating instruction is sent to an external offline scheduling system, wherein the first quasi-real-time task deleting instruction is used for controlling the external real-time data access system to delete the quasi-real-time integrated task, the second offline task generating instruction is used for newly adding a temporary table and a target table corresponding to the quasi-real-time integrated task in the external offline scheduling system, and the temporary table and the target table correspond to the to-be-processed service data table; when a task switching instruction corresponding to an offline integrated task is received, a first offline task deleting instruction is sent to an external offline scheduling system, and a second quasi-real-time task generating instruction is sent to an external real-time data access system, wherein the first offline task deleting instruction is used for controlling the external offline scheduling system to delete the offline integrated task, the second quasi-real-time task generating instruction is used for newly adding a quasi-real-time integrated task corresponding to a to-be-processed service data table in the external real-time data access system, and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task.
In one embodiment, the processor, when executing the computer program, further performs the steps of: searching historical data corresponding to a to-be-processed business data table; and sending an initialization instruction to an external first data integration system corresponding to the type of the integrated processing task according to the historical data corresponding to the to-be-processed service data table, wherein the initialization instruction is used for initializing the service data table created by the first integrated processing task.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
acquiring data information of a to-be-processed business data table;
searching an integration rule corresponding to the business data table to be processed, and judging an integration processing task type corresponding to the business data table to be processed according to the data information;
according to the integrated processing task type corresponding to the to-be-processed business data table, sending a first task generation instruction to an external first data integration system corresponding to the integrated processing task type, wherein the first task generation instruction is used for newly building a first integrated processing task corresponding to the to-be-processed business data table on the external first data integration system, and the first integrated processing task is used for performing integrated processing on the to-be-processed business data table;
when a task switching instruction corresponding to a first integrated processing task is received, a first task deleting instruction is sent to an external first data integrated system corresponding to the integrated processing task type according to the task switching instruction, and a second task generating instruction is sent to an external second data integrated system corresponding to the integrated processing task type, wherein the first task deleting instruction is used for deleting the first integrated processing task in the first external data integrated system, and the second task generating instruction is used for newly building a second integrated processing task corresponding to the task switching instruction in the second external data integrated system.
In one embodiment, the computer program when executed by the processor further performs the steps of: scanning service metadata in a preset storage position; identifying a newly added service data table to be processed according to the service metadata; and acquiring data information corresponding to the to-be-processed service data table.
In one embodiment, the computer program when executed by the processor further performs the steps of: searching an integration rule corresponding to the business data table to be processed; and sequentially inquiring the integration rules according to the data information, and determining the type of the integration processing task corresponding to the to-be-processed service data table according to the inquiry result.
In one embodiment, the computer program when executed by the processor further performs the steps of: when the integrated processing task type corresponding to the business data table to be processed is a quasi-real-time integrated task, sending a first quasi-real-time task generation instruction to an external real-time data access system, wherein the first quasi-real-time task generation instruction is used for newly adding a quasi-real-time integrated task corresponding to the business data table to be processed in the external real-time data access system and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task; and when the integrated processing task type corresponding to the to-be-processed service data table is an offline integrated task, sending a first offline task generation instruction to an external offline scheduling system, wherein the first offline task generation instruction is used for newly adding an offline scheduling task corresponding to the to-be-processed service data table in the external offline scheduling system and newly building a temporary table and a target table corresponding to the offline integrated task.
In one embodiment, the computer program when executed by the processor further performs the steps of: when a task switching instruction corresponding to the quasi-real-time integrated task is received, a first quasi-real-time task deleting instruction is sent to an external real-time data access system, and a second offline task generating instruction is sent to an external offline scheduling system, wherein the first quasi-real-time task deleting instruction is used for controlling the external real-time data access system to delete the quasi-real-time integrated task, the second offline task generating instruction is used for newly adding a temporary table and a target table corresponding to the quasi-real-time integrated task in the external offline scheduling system, and the temporary table and the target table correspond to the to-be-processed service data table; when a task switching instruction corresponding to an offline integrated task is received, a first offline task deleting instruction is sent to an external offline scheduling system, and a second quasi-real-time task generating instruction is sent to an external real-time data access system, wherein the first offline task deleting instruction is used for controlling the external offline scheduling system to delete the offline integrated task, the second quasi-real-time task generating instruction is used for newly adding a quasi-real-time integrated task corresponding to a to-be-processed service data table in the external real-time data access system, and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task.
In one embodiment, the computer program when executed by the processor further performs the steps of: searching historical data corresponding to a to-be-processed business data table; and sending an initialization instruction to an external first data integration system corresponding to the type of the integrated processing task according to the historical data corresponding to the to-be-processed service data table, wherein the initialization instruction is used for initializing the service data table created by the first integrated processing task.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (10)

1. A method for processing a service data table, the method comprising:
acquiring data information of a to-be-processed business data table;
searching an integration rule corresponding to the business data table to be processed, and judging an integration processing task type corresponding to the business data table to be processed according to the data information;
sending a first task generation instruction to an external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table, wherein the first task generation instruction is used for newly building a first integrated processing task corresponding to the to-be-processed service data table on the external first data integration system, and the first integrated processing task is used for performing integrated processing on the to-be-processed service data table;
when a task switching instruction corresponding to the first integrated processing task is received, according to the task switching instruction, sending a first task deleting instruction to an external first data integration system corresponding to the integrated processing task type, and sending a second task generating instruction to an external second data integration system corresponding to the integrated processing task type, where the first task deleting instruction is used to delete the first integrated processing task in the first external data integration system, and the second task generating instruction is used to newly create a second integrated processing task corresponding to the task switching instruction in the second external data integration system.
2. The method according to claim 1, wherein the obtaining data information of the pending service data table comprises:
scanning service metadata in a preset storage position;
identifying a newly added service data table to be processed according to the service metadata;
and acquiring data information corresponding to the to-be-processed service data table.
3. The method according to claim 1, wherein the searching for the integration rule corresponding to the to-be-processed service data table, and the determining, according to the data information, the type of the integration processing task corresponding to the to-be-processed service data table includes:
searching an integration rule corresponding to the business data table to be processed;
and sequentially inquiring the integration rules according to the data information, and determining the type of the integration processing task corresponding to the to-be-processed business data table according to the inquiry result.
4. The method according to claim 1, wherein the integrated processing task types include an offline integrated task and a quasi-real-time integrated task, the first task generation instruction includes a first quasi-real-time task generation instruction and a first offline task generation instruction, and the sending the first task generation instruction to the external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table includes:
when the integrated processing task type corresponding to the to-be-processed service data table is a quasi-real-time integrated task, sending a first quasi-real-time task generation instruction to an external real-time data access system, wherein the first quasi-real-time task generation instruction is used for newly adding a quasi-real-time integrated task corresponding to the to-be-processed service data table in the external real-time data access system and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task;
and when the integrated processing task type corresponding to the to-be-processed service data table is an offline integrated task, sending a first offline task generation instruction to an external offline scheduling system, wherein the first offline task generation instruction is used for newly adding an offline scheduling task corresponding to the to-be-processed service data table in the external offline scheduling system and newly building a temporary table and a target table corresponding to the offline integrated task.
5. The method according to claim 4, wherein the first task deletion instruction includes a first quasi-real-time task deletion instruction and a first offline task deletion instruction, the second task generation instruction includes a second offline task generation instruction and a second quasi-real-time task generation instruction, and when receiving a task switching instruction corresponding to the first integrated processing task, sending the first task deletion instruction to an external first data integration system corresponding to the integrated processing task type and sending the second task generation instruction to an external second data integration system corresponding to the integrated processing task type according to the task switching instruction includes:
when a task switching instruction corresponding to a quasi-real-time integrated task is received, sending a first quasi-real-time task deleting instruction to the external real-time data access system, and sending a second offline task generating instruction to the external offline scheduling system, wherein the first quasi-real-time task deleting instruction is used for controlling the external real-time data access system to delete the quasi-real-time integrated task, and the second offline task generating instruction is used for newly adding an offline scheduling task corresponding to the to-be-processed service data table in the external offline scheduling system, and newly building a temporary table and a target table corresponding to the quasi-real-time integrated task;
when a task switching instruction corresponding to an offline integrated task is received, a first offline task deleting instruction is sent to the external offline scheduling system, and a second quasi-real-time task generating instruction is sent to the external real-time data access system, wherein the first offline task deleting instruction is used for controlling the external offline scheduling system to delete the offline integrated task, the second quasi-real-time task generating instruction is used for newly adding a quasi-real-time integrated task corresponding to the to-be-processed service data table in the external real-time data access system, and newly building a kudu table, a view and a hive table corresponding to the quasi-real-time integrated task.
6. The method according to claim 1, wherein after sending a first task generation instruction to an external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table, the method further includes:
searching historical data corresponding to the to-be-processed business data table;
and sending an initialization instruction to an external first data integration system corresponding to the type of the integrated processing task according to the historical data corresponding to the to-be-processed service data table, wherein the initialization instruction performs initialization processing on the service data table created by the first integrated processing task.
7. An apparatus for processing a service data table, the apparatus comprising:
the data acquisition module is used for acquiring data information of a service data table to be processed;
the task type identification module is used for searching an integration rule corresponding to the to-be-processed business data table and judging an integration processing task type corresponding to the to-be-processed business data table according to the data information;
a task creating module, configured to send a first task generating instruction to an external first data integration system corresponding to the integrated processing task type according to the integrated processing task type corresponding to the to-be-processed service data table, where the first task generating instruction is used to create a first integrated processing task corresponding to the to-be-processed service data table on the external first data integration system, and the first integrated processing task is used to perform integrated processing on the to-be-processed service data table;
and a task switching module, configured to send a first task deletion instruction to an external first data integration system corresponding to the type of the integrated processing task and send a second task generation instruction to an external second data integration system corresponding to the type of the integrated processing task according to the task switching instruction when receiving a task switching instruction corresponding to the first integrated processing task, where the first task deletion instruction is used to delete the first integrated processing task in the first external data integration system, and the second task generation instruction is used to newly create a second integrated processing task corresponding to the task switching instruction in the second external data integration system.
8. The apparatus of claim 7, wherein the data acquisition module is specifically configured to:
scanning service metadata in a preset storage position;
identifying a newly added service data table to be processed according to the service metadata;
and acquiring data information corresponding to the to-be-processed service data table.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
CN202010736508.3A 2020-07-28 2020-07-28 Service data table processing method, device, computer equipment and storage medium Active CN111984740B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010736508.3A CN111984740B (en) 2020-07-28 2020-07-28 Service data table processing method, device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010736508.3A CN111984740B (en) 2020-07-28 2020-07-28 Service data table processing method, device, computer equipment and storage medium

Publications (2)

Publication Number Publication Date
CN111984740A true CN111984740A (en) 2020-11-24
CN111984740B CN111984740B (en) 2023-07-25

Family

ID=73444891

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010736508.3A Active CN111984740B (en) 2020-07-28 2020-07-28 Service data table processing method, device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN111984740B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116431742A (en) * 2023-06-09 2023-07-14 合肥青谷信息科技有限公司 Method and device for processing business data with large data volume and electronic equipment

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110153519A1 (en) * 2009-12-22 2011-06-23 Sap Ag Systems and methods for generating trigger networks corresponding to event-condition-action rules
CN109800090A (en) * 2018-12-19 2019-05-24 北京仁科互动网络技术有限公司 A kind of data integrated system and method
CN110457358A (en) * 2019-07-30 2019-11-15 新华三大数据技术有限公司 A kind of information collecting method, device, server and computer readable storage medium

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110153519A1 (en) * 2009-12-22 2011-06-23 Sap Ag Systems and methods for generating trigger networks corresponding to event-condition-action rules
CN109800090A (en) * 2018-12-19 2019-05-24 北京仁科互动网络技术有限公司 A kind of data integrated system and method
CN110457358A (en) * 2019-07-30 2019-11-15 新华三大数据技术有限公司 A kind of information collecting method, device, server and computer readable storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
胡细平;: "业务驱动型信息分析平台研究", 信息化研究, no. 09 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116431742A (en) * 2023-06-09 2023-07-14 合肥青谷信息科技有限公司 Method and device for processing business data with large data volume and electronic equipment

Also Published As

Publication number Publication date
CN111984740B (en) 2023-07-25

Similar Documents

Publication Publication Date Title
CN108924250B (en) Service request processing method and device based on block chain and computer equipment
CN109040227B (en) Service request response method and device based on block chain and computer equipment
CN104954468A (en) Resource allocation method and resource allocation device
EP3255849A1 (en) Multi-channel communications for sending push notifications to mobile devices
EP3267323A1 (en) Thin client system, connection management device, virtual machine operating device, method, and storage medium
CN110768914A (en) Decentralized Internet of things gateway system based on semantic scene instance migration
CN108965450B (en) Service request response method, device, computer equipment and storage medium
CN111984740A (en) Business data table processing method and device, computer equipment and storage medium
CN110659272A (en) Data cleaning method and system
US10033737B2 (en) System and method for cross-cloud identity matching
CN113220452A (en) Resource allocation method, model training method, device and electronic equipment
CN112199200A (en) Resource scheduling method and device, computer equipment and storage medium
CN115982148A (en) Database table processing method and device, electronic equipment and storage medium
CN114070889B (en) Configuration method, traffic forwarding device, storage medium, and program product
CN114610719B (en) Cross-cluster data processing method and device, electronic equipment and storage medium
CN114564249B (en) Recommendation scheduling engine, recommendation scheduling method and computer readable storage medium
US20200133930A1 (en) Information processing method, information processing system, and non-transitory computer readable storage medium
CN114090911A (en) Interface processing method and device, computer equipment and computer readable storage medium
CN114003717A (en) Data processing method and device, electronic equipment and storage medium
CN109582680B (en) Business processing method based on new product development, electronic device and readable storage medium
CN114327819A (en) Task management method, device, equipment and storage medium
US10887381B1 (en) Management of allocated computing resources in networked environment
US11233847B1 (en) Management of allocated computing resources in networked environment
CN112217849A (en) Task scheduling method and system in SD-WAN (secure digital-Wide area network) system and computer equipment
CN111126604A (en) Model training method, device, server and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CP03 Change of name, title or address

Address after: 518000 Room 201, building A, No. 1, Qian Wan Road, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong (Shenzhen Qianhai business secretary Co., Ltd.)

Patentee after: Zhaolian Consumer Finance Co.,Ltd.

Country or region after: China

Address before: 518000 Room 201, building A, No. 1, Qian Wan Road, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong (Shenzhen Qianhai business secretary Co., Ltd.)

Patentee before: MERCHANTS UNION CONSUMER FINANCE Co.,Ltd.

Country or region before: China

CP03 Change of name, title or address