CN117874137A

CN117874137A - Data processing method, device, computer equipment and storage medium

Info

Publication number: CN117874137A
Application number: CN202410035799.1A
Authority: CN
Inventors: 李慎刚
Original assignee: Ping An Property and Casualty Insurance Company of China Ltd
Current assignee: Ping An Property and Casualty Insurance Company of China Ltd
Priority date: 2024-01-09
Filing date: 2024-01-09
Publication date: 2024-04-12

Abstract

The application belongs to the field of big data and the field of financial science and technology, and relates to a data processing method, which comprises the following steps: acquiring incremental data of a target data table in a service main database based on a target processing link and sending the incremental data to a first card cluster; loading the incremental data to a first data table of a first big data platform; acquiring buried point data corresponding to the target data table, and writing the buried point data into a second card cluster; loading the buried data into a second data table of a second big data platform; performing data verification processing on the incremental data and the buried data to generate a data verification result; and executing a corresponding data synchronization business processing flow based on the data verification result. The application also provides a data processing device, computer equipment and a storage medium. In addition, the present application relates to blockchain technology in which incremental data may be stored. The method and the device can be applied to business data verification scenes in the financial field, and effectively improve the processing efficiency of data synchronous processing of the data table.

Description

Data processing method, device, computer equipment and storage medium

Technical Field

The present disclosure relates to the field of big data technologies and financial technologies, and in particular, to a data processing method, a data processing device, a computer device, and a storage medium.

Background

For financial and scientific enterprises, such as insurance enterprises, banks and the like, an Oracle database is generally used to store business data, and the business data of a large data table in the Oracle database is extracted and synchronized to a large data platform, and the business data in the large data platform is used for data processing requirements of data analysis and report statistics. However, the current data synchronization method for the business data of the large data table in the Oracle database is generally a processing strategy for synchronizing all the business data in the large data table in full. However, in the data synchronization process, the processing time of the data synchronization process is low due to the extremely large data volume of the large data table, and the synchronization failure is easy to occur, so that the data analysis is delayed or cannot be used.

Disclosure of Invention

An embodiment of the present application is directed to a data processing method, apparatus, computer device, and storage medium, so as to solve a technical problem of low processing timeliness of data synchronization processing in a data synchronization manner of service data of a large data table in an Oracle database adopted by an existing financial and scientific enterprise.

In order to solve the above technical problems, the embodiments of the present application provide a data processing method, which adopts the following technical schemes:

acquiring incremental data corresponding to a target data table in a service main database based on a target processing link corresponding to a preset service main database, and transmitting the incremental data to a preset first card cluster;

loading the incremental data in the first card cluster into a first data table in a preset first big data platform;

acquiring buried point data corresponding to the target data table in the service main database, and writing the buried point data into a preset second Kaff card cluster;

loading the buried data in the second card cluster into a second data table in a preset second big data platform;

performing data verification processing on the incremental data in the first data table and the buried data in the second data table to generate a data verification result between the incremental data and the buried data;

and executing a corresponding data synchronization business processing flow based on the data verification result.

Further, the step of performing data verification processing on the incremental data in the first data table and the buried data in the second data table to generate a data verification result between the incremental data and the buried data specifically includes:

Performing stripe number verification on the incremental data and the buried point data based on a preset stripe number verification mode;

if the number verification is passed, performing field verification on the incremental data and the buried data based on a preset field verification mode;

if the field verification is passed, generating a first data verification result of the data verification passing of the incremental data and the buried data corresponding to the data;

and if the field verification is not passed, generating a second data verification result that the data verification corresponding to the incremental data and the buried data is not passed.

Further, the step of performing field verification on the incremental data and the buried data based on a preset field verification mode specifically includes:

performing numerical matching processing on all fields of the incremental data and the buried data based on a preset data parallel processing strategy to obtain a corresponding numerical matching result;

judging whether all the numerical value matching results pass through the matching;

if yes, judging that the field verification result between the incremental data and the buried data is that the field verification is passed;

if not, judging that the field verification result between the incremental data and the buried data is that the field verification is not passed.

Further, the step of executing the corresponding data synchronization service processing flow based on the data verification result specifically includes:

if the data verification result is passed, generating reminding information which is corresponding to the incremental data and is successful in data synchronization;

acquiring communication information of a target business person;

and sending the reminding information to the target business personnel based on the communication information.

if the data verification result is that the data verification is not passed, acquiring difference data between the incremental data and the buried data;

performing complement processing on the incremental data based on the difference data to obtain processed target incremental data;

and performing data replacement processing on the incremental data by using the target incremental data.

Further, the step of loading the incremental data in the first kaff card cluster into a first data table in a preset first big data platform specifically includes:

calling a preset distributed computing engine;

acquiring address information of the first big data platform;

And based on the address information, using the distributed computing engine to consume the incremental data in the first Kaff card cluster, and loading the consumed incremental data into a first data table of the first big data platform.

Further, before the step of obtaining the buried point data corresponding to the target data table in the service main database and writing the buried point data into the preset second kaff card cluster, the method further includes:

receiving input buried point demand information;

constructing a corresponding buried point code based on the buried point demand information;

and carrying out point burying processing on the source end system corresponding to the service main database based on the point burying code.

In order to solve the above technical problems, the embodiments of the present application further provide a data processing apparatus, which adopts the following technical schemes:

the first processing module is used for acquiring incremental data corresponding to a target data table in a service main database based on a target processing link corresponding to a preset service main database, and sending the incremental data to a preset first Kaff card cluster;

the first loading module is used for loading the incremental data in the first Kaff card cluster into a first data table in a preset first big data platform;

The second processing module is used for acquiring buried point data corresponding to the target data table in the service main database and writing the buried point data into a preset second card cluster;

the second loading module is used for loading the embedded data in the second card cluster into a second data table in a preset second big data platform;

the verification module is used for carrying out data verification processing on the incremental data in the first data table and the buried data in the second data table, and generating a data verification result between the incremental data and the buried data;

and the execution module is used for executing the corresponding data synchronization business processing flow based on the data verification result.

In order to solve the above technical problems, the embodiments of the present application further provide a computer device, which adopts the following technical schemes:

In order to solve the above technical problems, embodiments of the present application further provide a computer readable storage medium, which adopts the following technical solutions:

Compared with the prior art, the embodiment of the application has the following main beneficial effects:

according to the embodiment of the application, firstly, incremental data corresponding to a target data table in a service main database is obtained based on a target processing link corresponding to the preset service main database, and the incremental data is sent to a preset first Kaff card cluster; then loading the incremental data in the first card cluster into a first data table in a preset first big data platform; then obtaining buried point data corresponding to the target data table in the service main database, and writing the buried point data into a preset second Kaff card cluster; loading the buried data in the second card cluster into a second data table in a preset second big data platform; subsequently, carrying out data verification processing on the incremental data in the first data table and the buried data in the second data table to generate a data verification result between the incremental data and the buried data; and finally, executing a corresponding data synchronization business processing flow based on the data verification result. According to the method and the device for processing the data synchronization of the target data table, the incremental data corresponding to the target data table in the service main database is obtained through the use of the target processing link corresponding to the service main database, and then the data synchronization processing of the target data table is carried out in an incremental synchronization mode of the target data table according to the incremental data, so that the whole table synchronization processing of the target data table can be avoided, and the processing efficiency of the data synchronization processing of the data table is effectively improved. In addition, the embedded data corresponding to the target data table in the service main database is used for verifying the incremental data corresponding to the target data table, and the corresponding data synchronization service processing flow is executed according to the data verification result, so that the processing accuracy of the data synchronization flow corresponding to the target data table is ensured, the situation of synchronization failure is avoided, and further the situation of data analysis delay or incapacitation is caused.

Drawings

For a clearer description of the solution in the present application, a brief description will be given below of the drawings that are needed in the description of the embodiments of the present application, it being obvious that the drawings in the following description are some embodiments of the present application, and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art.

FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;

FIG. 2 is a flow chart of one embodiment of a data processing method according to the present application;

FIG. 3 is a schematic diagram of one embodiment of a data processing apparatus according to the present application;

FIG. 4 is a schematic structural diagram of one embodiment of a computer device according to the present application.

Detailed Description

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the applications herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having" and any variations thereof in the description and claims of the present application and in the description of the figures above are intended to cover non-exclusive inclusions. The terms first, second and the like in the description and in the claims or in the above-described figures, are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.

In order to better understand the technical solutions of the present application, the following description will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings.

As shown in fig. 1, a system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as a web browser application, a shopping class application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc., may be installed on the terminal devices 101, 102, 103.

The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablet computers, electronic book readers, MP3 players (Moving Picture Experts Group Audio Layer III, dynamic video expert compression standard audio plane 3), MP4 (Moving Picture ExpertsGroup Audio Layer IV, dynamic video expert compression standard audio plane 4) players, laptop and desktop computers, and the like.

The server 105 may be a server providing various services, such as a background server providing support for pages displayed on the terminal devices 101, 102, 103.

It should be noted that, the data processing method provided in the embodiments of the present application is generally executed by a server/terminal device, and accordingly, the data processing apparatus is generally disposed in the server/terminal device.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

With continued reference to FIG. 2, a flow chart of one embodiment of a data processing method according to the present application is shown. The order of the steps in the flowchart may be changed and some steps may be omitted according to various needs. The data processing method provided by the embodiment of the application can be applied to any scene needing service data verification, and the data processing method can be applied to products of the scenes, for example, service data verification in the field of financial insurance. The data processing method comprises the following steps:

Step S201, based on a target processing link corresponding to a preset service main database, incremental data corresponding to a target data table in the service main database is obtained, and the incremental data is sent to a preset first Kaff card cluster.

In this embodiment, the electronic device (for example, the server/terminal device shown in fig. 1) on which the data processing method operates may acquire incremental data corresponding to the target data table in the service main database through a wired connection manner or a wireless connection manner. It should be noted that the wireless connection may include, but is not limited to, 3G/4G/5G connection, wiFi connection, bluetooth connection, wiMAX connection, zigbee connection, UWB (ultra wideband) connection, and other now known or later developed wireless connection. The service master database is a master database for storing service data of a target service system, and the master database can be an Oracle database. The target processing link can be an Oracle GG link, oracle GoldenGate (OGG for short) is a heterogeneous data replication technology with high performance and high availability provided by Oracle corporation, and can realize data synchronization between different databases. The Oracle GG link captures the changed data in the source database and transmits the changed data to the target database, so that the data can be synchronized in real time or periodically, and the operations such as real-time backup, data migration, data sharing and the like of the data are realized. The target data table refers to a large data table used for storing business data in an Oracle database. In a business scenario of business data verification of financial science and technology, the target business system may include an insurance system, a banking system, a transaction system, an order system, and the business data may include business data, transaction data, payment data, and the like. In addition, the first kafka cluster is a kafka cluster for caching the incremental data.

Step S202, the incremental data in the first Kaff card cluster are loaded into a first data table in a preset first big data platform.

In this embodiment, the first large data platform may specifically use an hdfs distributed file system, and the first data table is a hive table a in the hdfs distributed file system. The foregoing specific implementation process of loading the incremental data in the first card cluster into the first data table in the preset first big data platform will be described in further detail in the following specific embodiments, which will not be described herein.

Step 203, obtaining buried point data corresponding to the target data table in the service main database, and writing the buried point data into a preset second kaff card cluster.

In this embodiment, a preset buried point code may be configured in a service main database, so as to automatically collect buried point data corresponding to the target data table in the service main database. The second kafka cluster is used for caching the buried point data.

Step S204, loading the embedded point data in the second card cluster into a second data table in a preset second big data platform.

In this embodiment, the second large data platform may specifically use an hdfs distributed file system, and the first data table is a hive table B in the hdfs distributed file system. The process of loading the embedded data in the second card cluster into the second data table in the preset second large data platform may refer to the process of loading the incremental data in the first card cluster into the first data table in the preset first large data platform, which is not described herein in detail.

Step S205, performing data verification processing on the incremental data in the first data table and the buried data in the second data table, and generating a data verification result between the incremental data and the buried data.

In this embodiment, the foregoing performs data verification processing on the incremental data in the first data table and the buried data in the second data table, and generates a data verification result between the incremental data and the buried data. This will be described in further detail in the following embodiments, which will not be explained here too much.

Step S206, executing the corresponding data synchronization business processing flow based on the data verification result.

In this embodiment, the specific implementation process of executing the corresponding data synchronization service processing procedure based on the data verification result is described in further detail in the following specific embodiments, which will not be described herein.

Firstly, acquiring incremental data corresponding to a target data table in a service main database based on a target processing link corresponding to a preset service main database, and transmitting the incremental data to a preset first card cluster; then loading the incremental data in the first card cluster into a first data table in a preset first big data platform; then obtaining buried point data corresponding to the target data table in the service main database, and writing the buried point data into a preset second Kaff card cluster; loading the buried data in the second card cluster into a second data table in a preset second big data platform; subsequently, carrying out data verification processing on the incremental data in the first data table and the buried data in the second data table to generate a data verification result between the incremental data and the buried data; and finally, executing a corresponding data synchronization business processing flow based on the data verification result. According to the method and the device, the incremental data corresponding to the target data table in the service main database is acquired through the use of the target processing link corresponding to the service main database, and then the data synchronization processing of the target data table is carried out in an incremental synchronization mode of the target data table according to the incremental data, so that the full-table synchronization processing of the target data table can be avoided, and the processing efficiency of the data synchronization processing of the data table is effectively improved. In addition, the embedded data corresponding to the target data table in the service main database is used for verifying the incremental data corresponding to the target data table, and the corresponding data synchronization service processing flow is executed according to the data verification result, so that the processing accuracy of the data synchronization flow corresponding to the target data table is ensured, the situation of synchronization failure is avoided, and further the situation of data analysis delay or incapacitation is caused.

In some alternative implementations, step S205 includes the steps of:

and verifying the number of the incremental data and the buried point data based on a preset number verification mode.

In this embodiment, the foregoing preset number verification method includes the steps of: acquiring a first total number of the incremental data in the first data table and a second total number of the buried point data in the second data table; then comparing whether the first total number is the same as the second number; if the number of the pieces of the data is the same, the number of the pieces of the data is judged to pass the verification, otherwise, the number of the pieces of the data is judged to not pass the verification.

And if the number verification is passed, performing field verification on the incremental data and the buried data based on a preset field verification mode.

In this embodiment, the specific implementation process of performing field verification on the incremental data and the buried data based on the preset field verification manner is described in further detail in the following specific embodiments, which will not be described herein.

And if the field verification is passed, generating a first data verification result of the data verification passing of the incremental data and the buried data.

In this embodiment, if the incremental data in the first data table and the embedded point data in the second data table pass through the stripe verification and the field verification, it is determined that the incremental data in the service main database of the main link and the embedded point data corresponding to the target data table in the source client are completely identical to each other, and then a first data verification result that the data verification corresponding to the incremental data and the embedded point data passes is generated.

In this embodiment, if the incremental data in the first data table and the buried point data in the second data table do not pass the number verification or do not pass the field verification, it is determined that the incremental data in the service main database of the main link and the buried point data corresponding to the target data table in the source client are not completely consistent data, and then a second data verification result that the data verification corresponding to the incremental data and the buried point data does not pass is generated.

The incremental data and the buried point data are subjected to stripe number verification in a stripe number verification mode based on a preset stripe number; if the number verification is passed, then carrying out field verification on the incremental data and the buried data based on a preset field verification mode; if the field verification is passed, generating a first data verification result of the data verification passing of the incremental data and the buried data corresponding to the data; and if the field verification is not passed, generating a second data verification result that the data verification corresponding to the incremental data and the buried data is not passed. According to the method and the device, the incremental data in the first data table and the buried data in the second data table are subjected to strip number verification and field verification based on the preset strip number verification mode and field verification mode, so that the data verification result between the incremental data and the buried data can be quickly and accurately generated according to the obtained strip number verification result and the field verification result, the generation efficiency of the data verification result is improved, and the accuracy of the generated data verification result is guaranteed.

In some optional implementations of this embodiment, the performing field verification on the incremental data and the buried data based on a preset field verification manner includes the following steps:

and carrying out numerical matching processing on all fields of the incremental data and the buried data based on a preset data parallel processing strategy to obtain a corresponding numerical matching result.

In the present embodiment, the above-described data parallel processing policy refers to processing for performing parallel data computation using the SIMD model. Specifically, the data alignment operation can be performed on the incremental data and the buried data to ensure the corresponding relation between the fields of the incremental data and the buried data, and then the parallel numerical matching processing of all fields is performed on the incremental data and the buried data subjected to the data alignment operation by utilizing multi-core multithreading, so that the corresponding numerical matching result is obtained.

And judging whether all the numerical value matching results are matched and pass.

In this embodiment, content analysis is performed on each of the numerical matching results to identify whether all the numerical matching results pass through matching. Wherein, the content of the numerical value matching result comprises that the matching is passed or the matching is failed.

If yes, judging that the field verification result between the incremental data and the buried data is that the field verification is passed.

The method comprises the steps of carrying out numerical matching processing on all fields of incremental data and buried data based on a preset data parallel processing strategy to obtain a corresponding numerical matching result; then judging whether all the numerical value matching results are matched and pass; if yes, judging that the field verification result between the incremental data and the buried data is that the field verification is passed; if not, judging that the field verification result between the incremental data and the buried data is that the field verification is not passed. According to the method and the device, the data parallel processing strategy is used for carrying out numerical matching processing on all fields of the incremental data and the buried data, and further content analysis is carried out on the obtained numerical matching result, so that the field verification result between the incremental data and the buried data can be quickly and accurately generated, the generation efficiency of the field verification result is improved, and the data accuracy of the generated field verification result is ensured.

In some alternative implementations, step S206 includes the steps of:

and if the data verification result is passed, generating reminding information which is corresponding to the incremental data and is successful in data synchronization.

In this embodiment, the reminder information is information that is successfully written and generated and includes data synchronization related to incremental data corresponding to the target data table in the service main database.

And acquiring communication information of the target business personnel.

In this embodiment, the target service personnel may refer to operation and maintenance personnel of the service master database. The communication information may include a mail address or a telephone number.

In this embodiment, the reminding information may be sent to a communication terminal corresponding to the target service person according to the communication information.

If the data verification result is detected to pass through the data verification, generating reminding information of successful data synchronization corresponding to the incremental data; then obtaining the communication information of the target business personnel; and then, based on the communication information, sending the reminding information to the target business personnel. According to the method and the device, when the data verification result is detected to pass the data verification, the reminding information of successful data synchronization corresponding to the incremental data is intelligently generated, and the reminding information is sent to the corresponding target business personnel, so that the target business personnel can know the information of successful data synchronization corresponding to the incremental data in time by consulting the reminding information, further follow-up processing measures can be carried out on the incremental data, and the work efficiency and the work experience of the target business personnel are improved.

In some alternative implementations, step S206 includes the steps of:

and if the data verification result is that the data verification is not passed, acquiring difference data between the incremental data and the buried data.

In this embodiment, if the data verification result is that the data verification is not passed, in a process of performing data verification processing on the incremental data in the first data table and the buried data in the second data table, difference data between the incremental data and the buried data is recorded at the same time.

And carrying out complement processing on the incremental data based on the difference data to obtain the processed target incremental data.

In this embodiment, the difference data may include a specified field where a difference exists between the incremental data and the buried point data, and first field data in the incremental data corresponding to the specified field, second field data in the buried point data corresponding to the specified field, and then complement the first field data in the incremental data with the second field data in the buried point data to obtain the complemented incremental data, that is, the target incremental data.

In this embodiment, the data replacement process may be performed on the incremental data in the first data table in the first large data platform by using the target incremental data, so as to ensure that the synchronized incremental data is identical to the corresponding source data, that is, the buried data corresponding to the target data table in the service main database.

If the data verification result is that the data verification is failed, acquiring difference data between the incremental data and the buried data; then, carrying out complement processing on the incremental data based on the difference data to obtain processed target incremental data; and carrying out data replacement processing on the incremental data by using the target incremental data. When the data verification result is detected to be not passed, the increment data is intelligently subjected to complement processing according to the difference data between the increment data and the buried point data, and the processed target increment data is used for carrying out data replacement processing on the increment data, so that synchronous increment data and corresponding source end data, namely buried point data corresponding to the target data table in the service main database, are ensured to be the same, and the accuracy of data synchronous processing on the target data table is ensured.

In some alternative implementations of the present embodiment, step S202 includes the steps of:

and calling a preset distributed computing engine.

In this embodiment, the distributed computing engine may specifically use spark or flink.

And acquiring the address information of the first big data platform.

In this embodiment, the address information may refer to communication address information of the first big data platform, such as url address.

In this embodiment, the incremental data in the first card cluster may be consumed by using spark/flink, and then the incremental data may be landed in the first big data platform and loaded into the first data table of the first big data platform.

The method comprises the steps of calling a preset distributed computing engine; then, obtaining the address information of the first big data platform; and subsequently, based on the address information, using the distributed computing engine to consume the incremental data in the first Kaff card cluster, and loading the consumed incremental data into a first data table of the first big data platform. According to the method and the device, based on the use of the distributed computing engine, consumption processing is carried out on the incremental data in the first Kaff card cluster, and then the consumed incremental data are loaded into the first data table of the first big data platform, so that the data loading processing of the incremental data is completed rapidly, and smooth execution of the data loading of the incremental data is guaranteed.

In some optional implementations of this embodiment, before step S203, the electronic device may further perform the following steps:

and receiving input buried point demand information.

In this embodiment, the embedded point requirement information is information generated by a relevant user according to a service requirement of embedded point data for collecting incremental data corresponding to a big data table in a service main database.

And constructing a corresponding buried point code based on the buried point demand information.

In this embodiment, the corresponding embedded point code may be constructed by filling the embedded point requirement information into a preset embedded point code template. The embedded point code template is constructed according to the actual embedded point requirement.

In this embodiment, the embedded point code is used to perform embedded point processing on the source end system corresponding to the service main database, so that the embedded point data of the incremental data corresponding to the large data table in the service main database can be automatically collected later.

The method comprises the steps of receiving input buried point demand information; then constructing a corresponding buried point code based on the buried point demand information; and carrying out point burying processing on the source end system corresponding to the service main database based on the point burying code. According to the method and the device, the corresponding buried point codes are constructed by utilizing the input buried point demand information, then the buried point processing is carried out on the source end system corresponding to the service main database based on the buried point codes, so that the buried point data of the incremental data corresponding to the large data table in the service main database can be automatically and conveniently collected later, the data verification processing of the incremental data corresponding to the large data table in the service main database can be accurately completed according to the obtained buried point data, and the processing efficiency of the data verification processing is improved.

It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present invention.

It is emphasized that to further ensure the privacy and security of the delta data, the delta data may also be stored in a node of a blockchain.

The blockchain referred to in the application is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm and the like. The Blockchain (Blockchain), which is essentially a decentralised database, is a string of data blocks that are generated by cryptographic means in association, each data block containing a batch of information of network transactions for verifying the validity of the information (anti-counterfeiting) and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, an application services layer, and the like.

The embodiment of the application can acquire and process the related data based on the artificial intelligence technology. Among these, artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use knowledge to obtain optimal results.

Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by computer readable instructions stored in a computer readable storage medium that, when executed, may comprise the steps of the embodiments of the methods described above. The storage medium may be a nonvolatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a random access Memory (Random Access Memory, RAM).

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.

With further reference to fig. 3, as an implementation of the method shown in fig. 2, the present application provides an embodiment of a data processing apparatus, where an embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus is particularly applicable to various electronic devices.

As shown in fig. 3, the data processing apparatus 300 according to the present embodiment includes: a first processing module 301, a first loading module 302, a second processing module 303, a second loading module 304, a verification module 305, and an execution module 306. Wherein:

the first processing module 301 is configured to obtain incremental data corresponding to a target data table in a service main database based on a target processing link corresponding to a preset service main database, and send the incremental data to a preset first kaff card cluster;

the first loading module 302 is configured to load the incremental data in the first kaff card cluster into a first data table in a preset first big data platform;

a second processing module 303, configured to obtain buried point data corresponding to the target data table in the service main database, and write the buried point data into a preset second kaff card cluster;

the second loading module 304 is configured to load the embedded data in the second kaff card cluster into a second data table in a preset second big data platform;

A verification module 305, configured to perform data verification processing on the incremental data in the first data table and the buried data in the second data table, and generate a data verification result between the incremental data and the buried data;

and the execution module 306 is configured to execute a corresponding data synchronization service processing procedure based on the data verification result.

In this embodiment, the operations performed by the modules or units respectively correspond to the steps of the data processing method in the foregoing embodiment one by one, and are not described herein again.

In some alternative implementations of the present embodiment, the verification module 305 includes:

the first verification sub-module is used for verifying the number of the incremental data and the buried point data based on a preset number verification mode;

the second verification sub-module is used for carrying out field verification on the incremental data and the buried data based on a preset field verification mode if a plurality of pieces of data pass the verification;

the first generation sub-module is used for generating a first data verification result of the passing of the data verification corresponding to the incremental data and the buried data if the field verification is passed;

and the second generation sub-module is used for generating a second data verification result that the data verification corresponding to the incremental data and the buried data is not passed if the field verification is not passed.

In some optional implementations of this embodiment, the second verification sub-module includes:

the matching unit is used for carrying out numerical matching processing on all fields of the incremental data and the buried data based on a preset data parallel processing strategy to obtain a corresponding numerical matching result;

the judging unit is used for judging whether all the numerical value matching results pass through the matching;

the first judging unit is used for judging that the field verification result between the incremental data and the buried data is that the field verification is passed if the incremental data and the buried data are positive;

and the second judging unit is used for judging that the field verification result between the incremental data and the embedded data is that the field verification is failed if the incremental data and the embedded data are not judged to be the field verification.

In some alternative implementations of the present embodiment, the execution module 306 includes:

the third generation sub-module is used for generating reminding information of successful data synchronization corresponding to the incremental data if the data verification result passes the data verification;

The first acquisition sub-module is used for acquiring communication information of target service personnel;

and the sending sub-module is used for sending the reminding information to the target business personnel based on the communication information.

the second acquisition sub-module is used for acquiring difference data between the incremental data and the buried data if the data verification result is that the data verification is not passed;

the first processing sub-module is used for carrying out complement processing on the incremental data based on the difference data to obtain processed target incremental data;

and the second processing sub-module is used for performing data replacement processing on the incremental data by using the target incremental data.

In some alternative implementations of the present embodiment, the first loading module 302 includes:

The calling sub-module is used for calling a preset distributed computing engine;

the third acquisition sub-module is used for acquiring the address information of the first big data platform;

and the loading sub-module is used for carrying out consumption processing on the incremental data in the first Kaff card cluster by using the distributed computing engine based on the address information, and loading the consumed incremental data into a first data table of the first big data platform.

In some optional implementations of this embodiment, the data processing apparatus further includes:

the receiving module is used for receiving the input buried point demand information;

the construction module is used for constructing corresponding buried point codes based on the buried point demand information;

and the embedded point module is used for carrying out embedded point processing on the source end system corresponding to the service main database based on the embedded point code.

In order to solve the technical problems, the embodiment of the application also provides computer equipment. Referring specifically to fig. 4, fig. 4 is a basic structural block diagram of a computer device according to the present embodiment.

The computer device 4 comprises a memory 41, a processor 42, a network interface 43 communicatively connected to each other via a system bus. It should be noted that only computer device 4 having components 41-43 is shown in the figures, but it should be understood that not all of the illustrated components are required to be implemented and that more or fewer components may be implemented instead. It will be appreciated by those skilled in the art that the computer device herein is a device capable of automatically performing numerical calculations and/or information processing in accordance with predetermined or stored instructions, the hardware of which includes, but is not limited to, microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASICs), programmable gate arrays (fields-Programmable Gate Array, FPGAs), digital processors (Digital Signal Processor, DSPs), embedded devices, etc.

The computer equipment can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing equipment. The computer equipment can perform man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch pad or voice control equipment and the like.

The memory 41 includes at least one type of readable storage medium including flash memory, hard disk, multimedia card, card memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), programmable Read Only Memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the storage 41 may be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the computer device 4. Of course, the memory 41 may also comprise both an internal memory unit of the computer device 4 and an external memory device. In this embodiment, the memory 41 is typically used to store an operating system and various application software installed on the computer device 4, such as computer readable instructions of a data processing method. Further, the memory 41 may be used to temporarily store various types of data that have been output or are to be output.

The processor 42 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is configured to execute computer readable instructions stored in the memory 41 or process data, such as computer readable instructions for executing the data processing method.

The network interface 43 may comprise a wireless network interface or a wired network interface, which network interface 43 is typically used for establishing a communication connection between the computer device 4 and other electronic devices.

in the embodiment of the application, firstly, incremental data corresponding to a target data table in a service main database is acquired based on a target processing link corresponding to the preset service main database, and the incremental data is sent to a preset first card cluster; then loading the incremental data in the first card cluster into a first data table in a preset first big data platform; then obtaining buried point data corresponding to the target data table in the service main database, and writing the buried point data into a preset second Kaff card cluster; loading the buried data in the second card cluster into a second data table in a preset second big data platform; subsequently, carrying out data verification processing on the incremental data in the first data table and the buried data in the second data table to generate a data verification result between the incremental data and the buried data; and finally, executing a corresponding data synchronization business processing flow based on the data verification result. According to the method and the device for processing the data synchronization of the target data table, the incremental data corresponding to the target data table in the service main database is obtained through the use of the target processing link corresponding to the service main database, and then the data synchronization processing of the target data table is carried out in an incremental synchronization mode of the target data table according to the incremental data, so that the whole table synchronization processing of the target data table can be avoided, and the processing efficiency of the data synchronization processing of the data table is effectively improved. In addition, the embedded data corresponding to the target data table in the service main database is used for verifying the incremental data corresponding to the target data table, and the corresponding data synchronization service processing flow is executed according to the data verification result, so that the processing accuracy of the data synchronization flow corresponding to the target data table is ensured, the situation of synchronization failure is avoided, and further the situation of data analysis delay or incapacitation is caused.

The present application also provides another embodiment, namely, a computer-readable storage medium storing computer-readable instructions executable by at least one processor to cause the at least one processor to perform the steps of the data processing method as described above.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method described in the embodiments of the present application.

It is apparent that the embodiments described above are only some embodiments of the present application, but not all embodiments, the preferred embodiments of the present application are given in the drawings, but not limiting the patent scope of the present application. This application may be embodied in many different forms, but rather, embodiments are provided in order to provide a more thorough understanding of the present disclosure. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described in the foregoing, or equivalents may be substituted for elements thereof. All equivalent structures made by the specification and the drawings of the application are directly or indirectly applied to other related technical fields, and are also within the protection scope of the application.

Claims

1. A method of data processing comprising the steps of:

2. The method for processing data according to claim 1, wherein the step of performing data verification processing on the incremental data in the first data table and the buried data in the second data table to generate a data verification result between the incremental data and the buried data specifically includes:

3. The data processing method according to claim 2, wherein the step of performing field verification on the incremental data and the buried data based on a preset field verification method specifically includes:

4. The method for processing data according to claim 1, wherein the step of executing the corresponding data synchronization business processing flow based on the data verification result specifically comprises:

acquiring communication information of a target business person;

5. The method for processing data according to claim 1, wherein the step of executing the corresponding data synchronization business processing flow based on the data verification result specifically comprises:

6. The method for processing data according to claim 1, wherein the step of loading the incremental data in the first kaff card cluster into a first data table in a preset first large data platform specifically includes:

Calling a preset distributed computing engine;

acquiring address information of the first big data platform;

7. The data processing method according to claim 1, further comprising, before the step of acquiring buried point data corresponding to the target data table in the service master database and writing the buried point data into a predetermined second card cluster:

receiving input buried point demand information;

8. A data processing apparatus, comprising:

9. A computer device comprising a memory having stored therein computer readable instructions which when executed by a processor implement the steps of the data processing method of any of claims 1 to 7.

10. A computer-readable storage medium, having stored thereon computer-readable instructions which, when executed by a processor, implement the steps of the data processing method according to any of claims 1 to 7.