WO2018014582A1

WO2018014582A1 - Insurance policy data processing method, device, servicer and storage medium

Info

Publication number: WO2018014582A1
Application number: PCT/CN2017/078356
Authority: WO
Inventors: 刘永凡; 罗志权
Original assignee: 平安科技（深圳）有限公司
Priority date: 2016-07-22
Filing date: 2017-03-28
Publication date: 2018-01-25
Also published as: CN106980636B; CN106980636A

Abstract

An insurance policy data processing method, comprising: determining whether batch processing in a plurality of source databases is completed when a synchronization time is reached; if yes, triggering a synchronization operation of source databases and a target database, writing insurance policy data in the plurality of source databases into the target database and obtaining a general data table corresponding to the insurance policy data in the target database; splitting the general data table into multi-dimensioned partitions in the target database; receiving a data query instruction sent by a terminal, the query instruction carrying dimensional information; and querying in a corresponding partition according to the dimensional information to obtain a query result and returning the query result to the terminal, so that the terminal performs a monthly settlement according to the query result.

Description

Policy data processing method, device, server and storage medium

This application is submitted to the Chinese Patent Office on July 22, 2016, and the application number is 2016105852562. The title of the invention is the priority of the Chinese patent application of the 'Policy Data Processing Method and Apparatus', the entire contents of which are hereby incorporated by reference.

[Technical Field]

The present application relates to the field of computer technology, and in particular, to a policy data processing method, apparatus, server, and storage medium.

【Background technique】

With the development of society, insurance has penetrated into people's lives. There are more and more types of insurance. Each type of insurance has a very high requirement for data accuracy, so the computational logic that is processed in the background is complex. For a policy that requires monthly settlement, the amount of data processed in the monthly settlement process is very large due to the large number of policies. Statistics policy data can take a lot of time. How to provide policy statistics quickly and accurately during the monthly settlement process has become a technical problem that needs to be solved.

[Summary of the Invention]

According to various embodiments of the present application, a policy data processing method, apparatus, server, and storage medium are provided.

A policy data processing method, comprising:

When the synchronization time is reached, it is judged whether the batch processing in the plurality of source databases is completed;

If yes, triggering a synchronization operation between the source database and the target database, writing policy data in the plurality of source databases to the target database, and obtaining a data summary table corresponding to the policy data in the target database;

Splitting the data summary table into partitions of multiple dimensions in the target database;

Receiving a data query instruction sent by the terminal, where the query instruction carries dimension information; and

The query is performed in the corresponding partition according to the dimension information, and the query result is obtained, and the query result is returned to the terminal, so that the terminal performs monthly node processing according to the query result.

A policy data processing device comprising:

a judging module, configured to determine whether the batch processing in the plurality of source databases is completed when the synchronization time is reached;

a synchronization module, configured to: if the batch processing in the multiple source databases is completed, trigger a synchronous operation between the source database and the target database, write policy data in the multiple source databases into the target database, and obtain a policy in the target database. a data summary table corresponding to the data;

a splitting module, configured to split the data summary table into partitions of multiple dimensions in the target database;

a receiving module, configured to receive a data query instruction sent by the terminal, where the query instruction carries dimension information;

a query module, configured to perform a query in the corresponding partition according to the dimension information, to obtain a query result; and

And a sending module, configured to return the query result to the terminal, so that the terminal performs monthly node processing according to the query result.

A server comprising a memory and a processor, the memory storing computer executable instructions, the computer executable instructions being executed by the processor, such that the processor performs the following steps:

One or more non-volatile readable storage media storing computer-executable instructions, when executed by one or more processors, cause the one or more processors to perform the following steps:

Details of one or more embodiments of the present application are set forth in the accompanying drawings and description below. Other features, objects, and advantages of the invention will be apparent from the description and appended claims.

[Description of the Drawings]

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings to be used in the embodiments or the prior art description will be briefly described below. Obviously, the drawings in the following description are only It is a certain embodiment of the present invention, and those skilled in the art can obtain drawings of other embodiments according to the drawings without any creative work.

1 is an application scenario diagram of a policy data processing method in an embodiment;

2 is a flow chart of a method for processing policy data in an embodiment;

Figure 3 is a block diagram of a server in one embodiment;

Figure 4 is a block diagram of a policy data processing apparatus in an embodiment;

Figure 5 is a block diagram of a policy data processing apparatus in another embodiment.

【detailed description】

In order to make the objects, technical solutions, and advantages of the present application more comprehensible, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the application and are not intended to be limiting.

The policy data processing method provided in the embodiment of the present application can be applied to the application scenario shown in FIG. 1 . The terminal 102 and the server 104 are connected through a network. Server 104 runs the job and kettle. The job is configured to synchronize time. When the synchronization time is reached, the job is started and the job is used to determine whether the batch processing in the multiple source databases is completed. If it is completed, the kettle is executed to perform the synchronization operation between the source database and the target database. The policy data in the source database is written into the target database, and the data summary table corresponding to the policy data is obtained in the target database. The server 104 splits the data summary table into partitions of multiple dimensions in the target database. The terminal 102 sends a data query instruction to the server 104, and the server 104 receives the data query instruction, performs a query in the corresponding partition according to the dimension information carried in the query instruction, obtains the query result, and returns the query result to the terminal 102. The terminal 102 performs a monthly knot process based on the result of the query. This makes it possible to provide policy statistics quickly and accurately when performing policy monthly settlement processing.

In one embodiment, as shown in FIG. 2, a policy data processing method is provided. It should be understood that although the steps in the flowchart of FIG. 2 are sequentially displayed as indicated by the arrows, these steps are not necessarily in accordance with The order indicated by the arrows is performed in order. Except as explicitly stated herein, the execution of these steps is not strictly limited, and may be performed in other sequences. Moreover, at least some of the steps in FIG. 2 may include a plurality of sub-steps or stages, which are not necessarily performed at the same time, but may be executed at different times, and the order of execution thereof is not necessarily This may be performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of the other steps. The method is applied to the server as an example, and specifically includes the following steps:

Step 202: When the synchronization time is reached, it is determined whether the batch processing in the plurality of source databases is completed; if yes, step 204 is performed; otherwise, the batch processing is completed.

The job and the kettle are running on the server, and the kettle is working on the kettle platform. The synchronization time is the time set according to the need for the monthly settlement process of the policy. For example, 20:00 on the 1st of each month. The synchronization time can be either working time or non-working time. The job is configured for synchronization time.

Multiple databases are deployed on the server, including the source and target databases. Among them, the policy data of the storage organization in the source database. The source database can be the same number as the organization, and each organization is configured with a corresponding source database. The server can be a standalone server or a cluster service.

In one embodiment, determining whether the batch processing in the plurality of databases is completed comprises: starting a job, using a job to obtain a log corresponding to the batch processing; using a job to check a log corresponding to the batch processing, and determining whether the batch processing is performed according to the content recorded in the log carry out.

Batch processing refers to policy-guaranteed settlement and policy monthly snapshot snapshot refreshing. Batch operations are performed in multiple source databases and the logs corresponding to the batch are recorded. The progress of the batch execution is recorded in the log. When the synchronization time is reached, the job is started, the log corresponding to the batch processing is acquired by the job, and the contents of the log record are checked, and the batch processing is checked according to the content of the log record. If the batch processing is not completed, wait for the preset time. For example, wait for 5 minutes, and use the job to obtain the log corresponding to the batch. The log records the execution status of the batch, and thus uses the job to check according to the contents recorded in the log. Whether the batch is complete. Loop through until the job determines that the batch in multiple source databases is complete.

Step 204: trigger a synchronization operation between the source database and the target database, write policy data in the plurality of source databases into the target database, and obtain a data summary table corresponding to the policy data in the target database.

Policy data in the source database can be stored in the form of a data table. Data tables in multiple source databases can be in the same format. The data sheet includes the policy number, insured and premiums. When the job determines that the batch processing in multiple source databases is completed, the kettle is triggered to perform synchronization operations of the multiple source databases with the target database. Specifically, the kettle deletes the existing data in the target database, creates a new conversion, obtains a data table that needs to be synchronized in the source database, and sets multiple nodes in turn after starting the node. The number of nodes can be the same as the number of source databases. Each node is configured with a corresponding thread, and each thread is responsible for executing a source database file output to the target database. Import data tables from multiple source databases into the target database using multiple threads. The merchant data is processed by the merchant, and the full-scale insertion synchronization operation is performed in the target database, thereby quickly obtaining the data summary table corresponding to the policy data in the target database. The synchronization operation is performed concurrently by multiple threads, which effectively improves the synchronization efficiency of policy data in multiple source databases.

By adding a job to check the execution status of the batch in the source database, when the batch processing in the source database is completed, the job is triggered by the job to perform the synchronization operation between the source database and the target data. Thereby effectively reducing the coupling between the synchronous operation and the kettle platform, and improving the synchronization efficiency of the massive policy data.

Step 206: Split the data summary table into partitions of multiple dimensions in the target database.

Since the amount of data in the data master is usually in the order of millions and tens of millions, the data master is too large. It takes too long to perform data query in the data summary table, which also degrades the performance of the target database, which causes the monthly knot processing to be adversely affected.

In order to facilitate monthly processing, the data summary table is split into partitions of multiple dimensions in the target database. Partitioning refers to splitting the data master table into policy data in the data master table and storing it in multiple locations. The policy data for the post-partition data master is hashed to multiple locations in the database.

The server obtains the dimension field in the target database, and splits the data summary table into multiple partitions according to the dimension field. Dimension fields include time and organization. Among them, the time can be one day, one week or one month. The institution can be the identity of the location of the institution. The server can also configure partition thresholds for the amount of data for the partition. For example, the threshold is 10,000. The server splits the data master into multiple partitions based on the dimension field and the partition threshold. For policy data for a dimension that is less than the threshold, a single partition can be constructed. Further, the server can also divide the partition into multiple sub-partitions. For example, the server divides the partition into sub-partitions based on the type of insurance and premiums. Partitioning can effectively reduce the burden on the target database and improve the performance of the target database.

Step 208: Receive a data query instruction sent by the terminal, where the query instruction carries dimension information.

Step 210: Perform a query in the corresponding partition according to the dimension information, obtain a query result, and return the query result to the terminal, so that the terminal performs monthly node processing according to the query result.

The terminal sends a data query instruction to the server, and the query instruction carries the dimension information. One or more dimension fields, such as time and organization, etc., can be included in the dimension information. The server receives the data query instruction, performs a query in the corresponding partition according to the dimension information, and obtains the query result. The server does not need to query in the data summary table, which can effectively improve the query efficiency. The server returns the query result to the terminal, and the terminal performs the policy monthly settlement process according to the query result.

For the partition that is not needed in the policy of the monthly settlement, the terminal may send a delete instruction to the server, and the server deletes the corresponding partition according to the delete instruction. In the traditional method, if the data in the data summary table is deleted, it needs to be deleted line by line. If the deleted data is more, it takes more time. In this embodiment, the server can directly obtain the address of the partition, and delete the entire partition, and the operation is simple and fast.

Further, the terminal can also export the partitions required for the policy of the monthly settlement from the target database, and does not need to export the policy data one by one in the multiple source databases, thereby further improving the work efficiency of the policy monthly settlement processing. The terminal can also perform data cleaning and data compression on the exported policy data. Data cleaning refers to reviewing and verifying policy data, deleting duplicate policy data, and invalid policy data. Data compression refers to the consolidation and compression of policy data of the same type of insurance.

In this embodiment, when the synchronization time is reached, if the batch processing in the plurality of source databases has been completed; the synchronization operation of the source database and the target database is triggered. Thereby, the policy data in the plurality of source databases is written into the target database, and the data summary table corresponding to the policy data is obtained in the target database. Through the synchronization of multiple source databases and the target database, a large amount of policy data is written into the target database, thereby ensuring the accuracy of the policy statistics. Split the data master table into partitions of multiple dimensions in the target database. After receiving the data query instruction sent by the terminal, the query can be quickly performed in the partition according to the dimension information carried in the query instruction, thereby enabling the terminal to perform the policy monthly settlement process according to the query result. Therefore, the policy statistics can be quickly and accurately provided during the policy monthly settlement process.

In an embodiment, after the step of triggering the synchronization operation of the source database and the target database, the method further includes: recording a synchronization operation log; after the estimated time period, checking the synchronization operation log by using the job, according to the content recorded in the synchronization operation log. Determine whether the synchronization operation is completed; if yes, generate a prompt message that the synchronization operation is completed, and send the prompt information to the terminal.

In this embodiment, the server records the synchronization operation log while the kettle performs the synchronization operation of the plurality of source databases and the target database. The execution status of the synchronization operation is recorded in the synchronization operation log. After the estimated time, the server uses the job to check the synchronization operation log to determine whether the synchronization operation is completed. The estimated time is the time required to perform the synchronization operation based on the historical data and is an empirical value. For example, the estimated time can be 1 hour.

After the synchronization operation is triggered, if the job determines that the synchronization operation is not completed, the job waits for the preset time, acquires the synchronization operation log again, checks whether the synchronization operation is completed, and executes the loop until the synchronization operation is completed. When the job determines that the synchronization operation is completed, the server generates a prompt message that the synchronization operation is completed, and sends the prompt information to the terminal. Therefore, the synchronous operation is monitored in real time, and it is convenient for the terminal to know the execution status of the synchronous operation in time.

In one embodiment, the step of splitting the data summary table into the partitions of the plurality of dimensions in the target database comprises: running multiple threads in the target database to perform split task concurrently on the data summary table; and recording the split task splitting Divide the log; when multiple threads exit the target database and then perform the split task again, according to the split log, find the corresponding breakpoint task when multiple threads exit, and multiple threads continue to execute the split concurrently from the breakpoint task. Task until the data master is split into partitions of multiple dimensions.

In this embodiment, since the policy data of the data summary table is too large, the server runs multiple threads in the target database and concurrently performs the splitting task on the data summary table. The server can generate split tasks based on the dimension fields. Different threads execute split tasks for different dimension fields, and the number of threads is less than the number of split tasks. Multiple threads execute the split task concurrently according to the preset logic, which can effectively improve the splitting efficiency of the data summary table.

The server records the split log for the data master table split process, and records the split status of the data master table by splitting the log, thereby monitoring the split process. Once an error occurs during the splitting of the data master, the server can quickly locate the location of the error by calling the split log.

When multiple threads exit from the target database at some point, it may not have been completed for all split tasks. You need to run the thread again to continue the concurrent operation on the split task that has not yet been executed. Specifically, when multiple threads are running again, the corresponding breakpoint task when multiple threads exit from the target database is found according to the execution status of the task recorded in the split log. Starting from the breakpoint task, multiple threads continue to perform concurrent operations again, processing the policy according to the preset logic.

In this embodiment, multiple threads are run in the target database to perform a split task concurrently on the data summary table, and the corresponding split log is recorded. When multiple threads execute the split task again after exiting in the target database, it is not necessary to re-execute all split tasks. The breakpoint task corresponding to the multiple threads at the time of exit may be searched according to the execution status of the split task, so that multiple threads continue to perform concurrent operations from the breakpoint task. This eliminates the need to perform full replenishment of the split task, and finds the breakpoint task to continue execution at the breakpoint task, effectively preventing the missed execution and mis-execution of the split task, and effectively improving the policy data. Processing efficiency.

In one embodiment, the step of executing a split task on the data table concurrently by running multiple threads in the target database includes: obtaining a split task identifier in the target database, grouping the split tasks according to the split task identifier, and obtaining more Task groups; assign corresponding threads to task groups; perform split tasks concurrently on task groups through multiple threads.

In this embodiment, each split task has a unique split task identifier. The server obtains the split task identifier, and groups the split tasks according to the split task identifier to obtain multiple task groups.

In one embodiment, the server obtains the split task identifier, and groups the split tasks according to the preset number of task groups according to the sequence of split task identifiers to obtain multiple task groups. For example, there are a total of 100 split tasks in the target database. Each split task has a corresponding split task ID. The default task group is 10. Each of the 10 tasks is grouped according to the order of the split task identification, thereby obtaining 10 task groups.

In an embodiment, the split task identifier may be a task number, and the server obtains the split task, and obtains the tasks with the same number and tails according to the task number, and divides the split tasks with the same number and tails into one task group to obtain multiple task groups. . For example, there are a total of 100 tasks in the database. Each task has a corresponding task number, such as task 1, task 2, and task 100. Group tasks with the same task number and mantissa. For example, task 1, task 11, task 21, ... task 91 are grouped into one group, thereby obtaining 10 task groups.

The server assigns a corresponding thread to each task group. That is to say, each thread will execute the tasks in the corresponding task group fixedly. For example, there are a total of 10 task groups, 10 tasks in each task group, 5 threads running on the server, and the server assigns thread 1 to task group 1 and task group 3, then thread 1 will execute task group 1 fixedly. 10 tasks in the task until the task in task group 1 is completed. Thread 1 executes task group 3 after executing task group 1. Multiple threads perform concurrent operations based on the corresponding task group to process the policy. Since the thread fixedly executes the task corresponding to a certain task identifier, it is easy to find an abnormality occurring during the execution of the task, and the maintenance cost is low.

In one embodiment, the step of performing a split task concurrently on the task group by using multiple threads includes: multiple threads randomly acquiring the split task and performing concurrent operations; after the split task is executed, the thread randomly acquires the next split The task performs the corresponding operation.

In this embodiment, the thread does not perform a split task fixedly, and the split task can be randomly acquired to execute. Multiple threads can simultaneously acquire multiple split tasks and perform operations concurrently, and split the data summary table. After the thread has processed a split task, it can randomly acquire the next split task to execute. Because it does not require a thread to perform a split task, it can effectively reduce the time required to perform split tasks.

In one embodiment, as shown in FIG. 3, a server is provided that includes a processor coupled via a system bus, an internal memory, a non-volatile storage medium, and a network interface. The non-volatile storage medium of the server stores an operating system, a source database, a target database, and computer executable instructions. The policy data of the organization is stored in the source database, and the data summary table corresponding to the policy data is stored in the target database. Computer executable instructions are used to perform a policy data processing method. The server's processor is used to provide computing and control capabilities that support the operation of the entire server. The server's network interface is used to connect to the terminal and communicate with the terminal. The server can be a separate server or a clustered server. Those skilled in the art can understand that the structure shown in FIG. 3 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the server to which the solution of the present application is applied. A particular server may include more or fewer components than shown, or some components may be combined, or have different component arrangements.

In an embodiment, as shown in FIG. 4, a policy data processing apparatus is provided, including: a determining module 402, a synchronization module 404, a splitting module 406, a receiving module 408, a querying module 410, and a sending module 412, where:

The determining module 402 is configured to determine whether the batch processing in the plurality of source databases is completed when the synchronization time is reached.

The synchronization module 404 is configured to: if the batch processing in the multiple source databases is completed, trigger a synchronous operation between the source database and the target database, write policy data in the plurality of source databases into the target database, and obtain policy data in the target database. Corresponding data summary table.

The splitting module 406 is configured to split the data summary table into partitions of multiple dimensions in the target database.

The receiving module 408 is configured to receive a data query instruction sent by the terminal, where the query instruction carries dimension information.

The query module 410 is configured to perform a query in the corresponding partition according to the dimension information to obtain a query result.

The sending module 412 is configured to return the query result to the terminal, so that the terminal performs monthly node processing according to the query result.

In one embodiment, the determining module 402 is further configured to start a job, use a job to obtain a log corresponding to the batch processing, use a job to check a log corresponding to the batch processing, and determine whether the batch processing is completed according to the content recorded in the log.

In an embodiment, as shown in FIG. 5, the apparatus further includes: a recording module 414, configured to record a synchronization operation log; and the determining module 402 is further configured to: after the estimated time period, use the job to check the synchronization operation log, according to the synchronization operation. The content recorded in the log is used to determine whether the synchronization operation is completed; if yes, the prompt information for completing the synchronization operation is generated; the sending module 412 is further configured to send the prompt information to the terminal.

In one embodiment, the splitting module 406 is further configured to run multiple threads in the target database to perform a split task concurrently on the data summary table; record a split log of the split task; and when multiple threads exit from the target database When the split task is executed again, according to the split log, multiple breakpoint tasks corresponding to multiple threads are exited, and multiple threads continue to execute the split task concurrently from the breakpoint task until the data summary table is split into multiple dimensions. Partition.

In one embodiment, the splitting module 406 is further configured to obtain a split task identifier in the target database, group the split tasks according to the split task identifier, and obtain multiple task groups; assign a corresponding thread to the task group; Multiple threads perform split tasks concurrently on the task group.

The various modules in the policy data processing apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. For example, in hardware implementation, the receiving module 408 is configured to receive a query instruction sent by the terminal through a network interface of the server, and the sending module 412 returns the query result to the terminal through a network interface of the server. The network interface may be an Ethernet card or a wireless network card. The above modules may be embedded in the hardware of the base station or may be stored in the memory of the base station in a software form, so that the processor can call the corresponding operations of the above modules. The processor may be a central processing unit (CPU) or a microprocessor.

In one embodiment, there is provided one or more non-volatile readable storage media having computer-executable instructions that, when executed by one or more processors, cause one or more processors to execute The following steps:

If yes, the synchronization operation between the source database and the target database is triggered, the policy data in the plurality of source databases is written into the target database, and the data summary table corresponding to the policy data is obtained in the target database;

Split the data master table into partitions of multiple dimensions in the target database;

The query is performed in the corresponding partition according to the dimension information, and the query result is obtained, and the query result is returned to the terminal, so that the terminal performs monthly knot processing according to the query result.

In an embodiment, determining whether the batch processing in the plurality of databases is completed includes: starting a job, acquiring a log corresponding to the batch processing by using a job; and checking a log corresponding to the batch processing by using a job, and determining whether the batch processing is performed according to the content recorded in the log carry out.

In one embodiment, when the computer executable instructions are executed by one or more processors, the one or more processors further cause the step of: recording a synchronization operation log; after an estimated period of time, checking the synchronization operation log with a job, Determining whether the synchronization operation is completed according to the content recorded in the synchronization operation log; and if so, generating prompt information for completion of the synchronization operation, and transmitting the prompt information to the terminal.

In one embodiment, splitting the data summary table into multiple dimensions in the target database includes: running multiple threads in the target database to perform split tasks concurrently on the data master table; recording split logs of split tasks And when multiple threads execute the split task after exiting from the target database, find the breakpoint tasks corresponding to the multiple threads at the exit according to the split log, and multiple threads continue to execute the split task concurrently from the breakpoint task. Until the data master is split into partitions of multiple dimensions.

In one embodiment, running the multiple threads in the target database to perform the splitting task on the data table concurrently includes: obtaining the split task identifier in the target database, grouping the split tasks according to the split task identifier, and obtaining multiple tasks a group; assign a corresponding thread to the task group; and perform a split task concurrently on the task group through multiple threads.

One of ordinary skill in the art can understand that all or part of the process of implementing the above embodiments can be completed by a computer program to instruct related hardware, and the program can be stored in a non-volatile computer readable storage medium. Wherein, the program, when executed, may include the flow of an embodiment of the methods as described above. The storage medium may be a magnetic disk, an optical disk, or a read-only storage memory (Read-Only) Memory, ROM), etc.

The technical features of the above embodiments may be arbitrarily combined. For the sake of brevity of description, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, It is considered to be the range described in this specification.

The above embodiments are merely illustrative of several embodiments of the present invention, and the description thereof is more specific and detailed, but is not to be construed as limiting the scope of the invention. It should be noted that a number of variations and modifications may be made by those skilled in the art without departing from the spirit and scope of the invention. Therefore, the scope of the invention should be determined by the appended claims.

Claims

A policy data processing method, comprising:

When the synchronization time is reached, it is judged whether the batch processing in the plurality of source databases is completed;

If yes, triggering a synchronization operation between the source database and the target database, writing policy data in the plurality of source databases to the target database, and obtaining a data summary table corresponding to the policy data in the target database;

Splitting the data summary table into partitions of multiple dimensions in the target database;

Receiving a data query instruction sent by the terminal, where the query instruction carries dimension information; and

The query is performed in the corresponding partition according to the dimension information, and the query result is obtained, and the query result is returned to the terminal, so that the terminal performs monthly node processing according to the query result.
The method according to claim 1, wherein said determining whether the batch processing in the plurality of databases is completed comprises:

Start the job, use the job to get the log corresponding to the batch; and

Use the job to check the log corresponding to the batch, and judge whether the batch is completed according to the contents recorded in the log.
The method according to claim 1, wherein after the synchronization operation of the trigger source database and the target database, the method further comprises:

Record the synchronization operation log; and

After the estimated period of time, the synchronization operation log is checked by using the job, and the synchronization operation is determined according to the content recorded in the synchronization operation log;

If yes, the prompt information of the completion of the synchronization operation is generated, and the prompt information is sent to the terminal.
The method according to claim 1, wherein the partitioning the data summary table into the plurality of dimensions in the target database comprises:

Running a plurality of threads in the target database concurrently performing a splitting task on the data summary table;

Record the split log of the split task; and

When the plurality of threads exit the target database and execute the split task again, the split log is used to find a breakpoint task corresponding to the multiple threads when exiting, and multiple threads continue to execute the split concurrently from the breakpoint task. The tasks are divided until the data summary table is split into partitions of multiple dimensions.
The method according to claim 4, wherein the running the plurality of threads in the target database to perform the splitting task concurrently on the data table comprises:

Obtaining a split task identifier in the target database, grouping the split tasks according to the split task identifier, and obtaining multiple task groups;

Assign a corresponding thread to the task group; and

A split task is performed concurrently on a task group through multiple threads.
A policy data processing device comprising:

a judging module, configured to determine whether the batch processing in the plurality of source databases is completed when the synchronization time is reached;

a synchronization module, configured to: if the batch processing in the multiple source databases is completed, trigger a synchronous operation between the source database and the target database, write policy data in the multiple source databases into the target database, and obtain a policy in the target database. a data summary table corresponding to the data;

a splitting module, configured to split the data summary table into partitions of multiple dimensions in the target database;

a receiving module, configured to receive a data query instruction sent by the terminal, where the query instruction carries dimension information;

a query module, configured to perform a query in the corresponding partition according to the dimension information, to obtain a query result; and

And a sending module, configured to return the query result to the terminal, so that the terminal performs monthly node processing according to the query result.
The device according to claim 6, wherein the determining module is further configured to start a job, use a job to obtain a log corresponding to the batch processing, use a job to check a log corresponding to the batch processing, and determine a batch processing according to the content recorded in the log. Whether it is completed.
The device according to claim 6, further comprising:

a recording module for recording a synchronization operation log;

The determining module is further configured to: after the estimated time period, use the job to check the synchronization operation log, determine whether the synchronization operation is completed according to the content recorded in the synchronization operation log; if yes, generate prompt information that the synchronization operation is completed; and

The sending module is further configured to send the prompt information to the terminal.
The apparatus according to claim 6, wherein the splitting module is further configured to run a plurality of threads in the target database to perform a split task concurrently on the data summary table; and record a split log of the split task; When the plurality of threads exit the target database and execute the split task again, the split log is used to find a breakpoint task corresponding to the multiple threads when exiting, and multiple threads continue to execute the split concurrently from the breakpoint task. The tasks are divided until the data summary table is split into partitions of multiple dimensions.
The device according to claim 9, wherein the splitting module is further configured to acquire a split task identifier in the target database, group the split tasks according to the split task identifier, and obtain multiple task groups. Assign a corresponding thread to the task group; perform split tasks concurrently on the task group through multiple threads.
A server comprising a memory and a processor, the memory storing computer executable instructions, the computer executable instructions being executed by the processor, such that the processor performs the following steps:

When the synchronization time is reached, it is judged whether the batch processing in the plurality of source databases is completed;

If yes, triggering a synchronization operation between the source database and the target database, writing policy data in the plurality of source databases to the target database, and obtaining a data summary table corresponding to the policy data in the target database;

Splitting the data summary table into partitions of multiple dimensions in the target database;

Receiving a data query instruction sent by the terminal, where the query instruction carries dimension information; and

The query is performed in the corresponding partition according to the dimension information, and the query result is obtained, and the query result is returned to the terminal, so that the terminal performs monthly node processing according to the query result.
The server according to claim 11, wherein said determining whether the batch processing in the plurality of databases is completed comprises:

Start the job, use the job to get the log corresponding to the batch; and

Use the job to check the log corresponding to the batch, and judge whether the batch is completed according to the contents recorded in the log.
The server according to claim 11, wherein after the synchronization operation of the trigger source database and the target database, the processor is further caused to perform the following steps:

Record the synchronization operation log;

After the estimated period of time, the job is checked by the job, and the synchronization operation is determined according to the content recorded in the synchronization operation log; and

If yes, the prompt information of the completion of the synchronization operation is generated, and the prompt information is sent to the terminal.
The server according to claim 11, wherein the partitioning the data summary table into the plurality of dimensions in the target database comprises:

Running a plurality of threads in the target database concurrently performing a splitting task on the data summary table;

Record the split log of the split task; and

When the plurality of threads exit the target database and execute the split task again, the split log is used to find a breakpoint task corresponding to the multiple threads when exiting, and multiple threads continue to execute the split concurrently from the breakpoint task. The tasks are divided until the data summary table is split into partitions of multiple dimensions.
The server according to claim 14, wherein the running the plurality of threads in the target database to perform the splitting task concurrently on the data table comprises:

Obtaining a split task identifier in the target database, grouping the split tasks according to the split task identifier, and obtaining multiple task groups;

Assign a corresponding thread to the task group; and

A split task is performed concurrently on a task group through multiple threads.
One or more non-volatile readable storage media storing computer-executable instructions, when executed by one or more processors, cause the one or more processors to perform the following steps:

When the synchronization time is reached, it is judged whether the batch processing in the plurality of source databases is completed;

If yes, triggering a synchronization operation between the source database and the target database, writing policy data in the plurality of source databases to the target database, and obtaining a data summary table corresponding to the policy data in the target database;

Splitting the data summary table into partitions of multiple dimensions in the target database;

Receiving a data query instruction sent by the terminal, where the query instruction carries dimension information; and

The query is performed in the corresponding partition according to the dimension information, and the query result is obtained, and the query result is returned to the terminal, so that the terminal performs monthly node processing according to the query result.
The non-volatile readable storage medium according to claim 16, wherein the determining whether the batch processing in the plurality of databases is completed comprises:

Start the job, use the job to get the log corresponding to the batch; and

Use the job to check the log corresponding to the batch, and judge whether the batch is completed according to the contents recorded in the log.
A non-volatile readable storage medium according to claim 16, wherein said said

When the computer executable instructions are executed by one or more processors, the one or more processors are further caused to perform the following steps:

Record the synchronization operation log;

After the estimated period of time, the job is checked by the job, and the synchronization operation is determined according to the content recorded in the synchronization operation log; and

If yes, the prompt information of the completion of the synchronization operation is generated, and the prompt information is sent to the terminal.
The non-volatile readable storage medium according to claim 16, wherein the partitioning the data summary table into the plurality of dimensions in the target database comprises:

Running a plurality of threads in the target database concurrently performing a splitting task on the data summary table;

Record the split log of the split task; and

When the plurality of threads exit the target database and execute the split task again, the split log is used to find a breakpoint task corresponding to the multiple threads when exiting, and multiple threads continue to execute the split concurrently from the breakpoint task. The tasks are divided until the data summary table is split into partitions of multiple dimensions.
The non-volatile readable storage medium according to claim 19, wherein the running the plurality of threads in the target database to perform the splitting task concurrently on the data table comprises:

Obtaining a split task identifier in the target database, grouping the split tasks according to the split task identifier, and obtaining multiple task groups;

Assign a corresponding thread to the task group; and

A split task is performed concurrently on a task group through multiple threads.