CN116955342B - Service data consistency rate verification method and device - Google Patents

Service data consistency rate verification method and device Download PDF

Info

Publication number
CN116955342B
CN116955342B CN202311213203.4A CN202311213203A CN116955342B CN 116955342 B CN116955342 B CN 116955342B CN 202311213203 A CN202311213203 A CN 202311213203A CN 116955342 B CN116955342 B CN 116955342B
Authority
CN
China
Prior art keywords
task
verification
data
subtask
tested
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202311213203.4A
Other languages
Chinese (zh)
Other versions
CN116955342A (en
Inventor
杨良志
白琳
汪志新
朱彩霞
叶楠
张云康
杜文昊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Caixun Digital Technology Co ltd
Richinfo Technology Co ltd
Original Assignee
Guangzhou Caixun Digital Technology Co ltd
Richinfo Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Caixun Digital Technology Co ltd, Richinfo Technology Co ltd filed Critical Guangzhou Caixun Digital Technology Co ltd
Priority to CN202311213203.4A priority Critical patent/CN116955342B/en
Publication of CN116955342A publication Critical patent/CN116955342A/en
Application granted granted Critical
Publication of CN116955342B publication Critical patent/CN116955342B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The application discloses a service data consistency rate verification method and a device, which relate to a service data consistency rate verification technology, wherein the dynamic splitting verification process in the method comprises the following steps: if the verification workload of the verification task exceeds the current task execution capacity of the processor, splitting the verification task into at least two subtasks according to the verification workload and the current task execution capacity of the processor; splitting the current task execution capacity of the processor into at least two subtask lines, wherein each subtask line comprises a data comparison function, an update task generation function and a data synchronization function; each subtask searches a subtask line meeting the corresponding workload requirement according to the idle condition of the subtask line to be combined to form a task execution group; after each task execution group executes the corresponding subtasks, corresponding task line resources are released, and the problems of low efficiency and high error rate caused by the fact that service data consistency rate verification relies on manpower in the prior art are solved.

Description

Service data consistency rate verification method and device
Technical Field
The application relates to the technical field of service data consistency rate verification, in particular to a service data consistency rate verification method and device.
Background
Most of the traditional service support systems are designed in a single-center and multi-node mode, and when the problem of inconsistent data exists between nodes or between the nodes and the center, the normal operation and expansion of the service are greatly affected.
In the related art, a reconciliation method is generally adopted after reconciliation, a service system uploads data to a main FTP during service leisure, compares the data with corresponding data of the main system, and adjusts inconsistent data through negotiation. The time nodes for determining whether the data are consistent are all of fixed periodicity, and a method for actively initiating detection at any time is lacking. Meanwhile, the data adjustment needs to be manually performed after detection, a large amount of labor hour is consumed, and the risk of errors is brought.
Disclosure of Invention
The application aims to solve the technical problems that the network marketing strategy generation process is complex and highly depends on manpower in the prior art, thereby providing a service data consistency rate verification method and device.
In order to solve the technical problems, the embodiment of the application at least provides a service data consistency rate verification method and device.
In a first aspect, an embodiment of the present disclosure provides a service data consistency rate verification method, for consistency rate verification of batch data to be verified, where the method includes a dynamic split verification process, where the dynamic split verification process includes:
generating a verification task based on the data to be verified;
if the verification workload of the task to be verified exceeds the current task execution capacity of the processor, splitting the verification task into at least two subtasks according to the verification workload and the current task execution capacity of the processor;
splitting the current task execution capacity of a processor into at least two subtask lines, wherein each subtask line comprises a data comparison function, an update task generation function and a data synchronization function;
each subtask searches a subtask line meeting the corresponding workload requirement according to the idle condition of the subtask line and combines the subtask lines to form a task execution group;
and after each task execution group executes the corresponding subtask, releasing the corresponding task line resource.
Optionally, before the dynamic split verification process, the method further includes a front-end configuration process, the front-end configuration process including: acquiring the data to be verified of the tested party system in real time; determining a data verification interface of the tested party system; configuring a unique identification field of the data to be verified; and determining the verification execution time according to the input information of the user.
Optionally, the data synchronization function includes: inquiring target data information corresponding to the unique identification field from a tested party system; and returning the target data information to a verification initiating terminal for field-by-field comparison, searching for difference content, generating an update script taking a statement as a core, and returning to the tested party system so as to enable the tested party system to complete consistency synchronization.
Optionally, the method further comprises: a history verification task recording process, the history verification task recording process comprising: acquiring any history verification task; acquiring verification data corresponding to the history verification task; if the check data and the associated data are not changed again, finishing the rollback operation and returning to the state before updating.
Optionally, the searching for a subtask line meeting the corresponding workload requirement is combined as follows: according to the following formulaSearching a subtask line meeting the corresponding workload requirement for combination, wherein R is the resource ratio allocated by the server to be tested for executing the task, p is the expected of generating execution capacity resources for the server to be tested required by the task to be executed, and->As an adjustable coefficient, determining the influence of the task to be executed on the allocation resource of the server to be tested, wherein the value is greater than 1,/for>For the anticipated occupation of the server resources of the tested party, the value of p is smaller than the value of p, and the value of +.>Whether the core program of the server to be tested is executed or not, and the value is 0 or 1, which respectively represents two states of being executed and not being executed, and the value is +.>Is the remainder of the server resources of the tested party.
Optionally, in the history verification task recording process, the time point of the history verification task is stored in a snapshot form at the history verification task recording process end, and corresponding data is directly changed according to snapshot information during rollback.
Optionally, the method further comprises: if the tested party system is a new system, when a check address dictionary column is added, configuring a new tested party system data check interface address so as to quickly increase the tested party system.
In a second aspect, an embodiment of the present disclosure provides a service data consistency rate verification apparatus, configured to perform consistency rate verification on batch data to be verified, where the apparatus includes a dynamic split verification module, where the dynamic split verification module includes:
the verification task generation sub-module is used for generating a verification task based on data to be verified;
the task splitting sub-module is used for splitting the checking task into at least two sub-tasks according to the checking workload and the current task execution capacity of the processor if the checking workload of the task to be checked exceeds the current task execution capacity of the processor;
the execution capacity splitting sub-module is used for splitting the current task execution capacity of the processor into at least two subtask lines, wherein each subtask line comprises a data comparison function, an update task generation function and a data synchronization function;
the combined execution sub-module is used for searching a sub-task line meeting the corresponding workload requirement according to the idle condition of the sub-task lines for each sub-task to be combined to form a task execution group;
and the resource management sub-module is used for releasing the corresponding task line resources after each task execution group executes and completes the corresponding sub-task.
In a third aspect, the disclosed embodiments of the application also provide a computer device comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication via the bus when the computer device is running, the machine-readable instructions when executed by the processor performing the steps of the first aspect, or any of the possible implementations of the first aspect.
In a fourth aspect, the disclosed embodiments also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the first aspect, or any of the possible implementation manners of the first aspect.
The technical scheme provided by the embodiment of the application has the following beneficial effects:
the method comprises the steps of carrying out consistency rate verification on batch data to be verified, wherein the consistency rate verification comprises a dynamic splitting verification process, and splitting a verification task into at least two subtasks according to the verification workload and the current task execution capacity of a processor when the verification workload of the task to be verified exceeds the current task execution capacity of the processor; splitting the current task execution capacity of the processor into at least two subtask lines, wherein each subtask line comprises a data comparison function, an update task generation function and a data synchronization function; each subtask searches a subtask line meeting the corresponding workload requirement according to the idle condition of the subtask line to be combined to form a task execution group; after each task execution group executes and completes the corresponding subtasks, corresponding task line resources are released, verification is completed quickly through task and function block combination allocation, the method is suitable for batch data verification, automatic completion is achieved, and energy consumption improving efficiency is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present application, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
Fig. 1 shows a flowchart of a service data consistency rate verification method provided by an embodiment of the present disclosure;
FIG. 2 is a flowchart illustrating another method for verifying the consistency rate of service data according to an embodiment of the present disclosure;
fig. 3 is a functional block diagram of a service data consistency rate checking device according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a computer device according to an embodiment of the present disclosure;
fig. 5 is a schematic functional structural diagram of a service data consistency rate checking device according to an embodiment of the present disclosure;
FIG. 6 is a schematic diagram illustrating an implementation of a dynamic split verification process in accordance with an embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying summary.
Example 1
As shown in fig. 1, a flow chart of a service data consistency rate verification method provided by an embodiment of the present application is used for consistency rate verification of batch data to be verified, the method includes a dynamic split verification process, and the dynamic split verification process includes:
s11: and generating a verification task based on the data to be verified.
S12: if the verification workload of the task to be verified exceeds the current task execution capacity of the processor, splitting the verification task into at least two subtasks according to the verification workload and the current task execution capacity of the processor.
S13: the current task execution capability of the processor is split into at least two subtask lines, each subtask line comprising a data comparison function, an update task generation function and a data synchronization function.
S14: and each subtask searches a subtask line meeting the corresponding workload requirement according to the idle condition of the subtask line to be combined to form a task execution group.
S15: and after each task execution group executes and completes the corresponding subtask, releasing the corresponding task line resource.
It can be understood that the technical scheme provided by the embodiment is used for checking the consistency rate of the batch data to be checked, and comprises a dynamic splitting checking process, wherein when the checking workload of the task to be checked exceeds the current task execution capacity of the processor, the dynamic splitting checking process splits the checking task into at least two subtasks according to the checking workload and the current task execution capacity of the processor; splitting the current task execution capacity of the processor into at least two subtask lines, wherein each subtask line comprises a data comparison function, an update task generation function and a data synchronization function; each subtask searches a subtask line meeting the corresponding workload requirement according to the idle condition of the subtask line to be combined to form a task execution group; after each task execution group executes and completes the corresponding subtasks, corresponding task line resources are released, verification is completed quickly through task and function block combination allocation, the method is suitable for batch data verification, automatic completion is achieved, and energy consumption improving efficiency is reduced.
Example 2
As shown in fig. 2, a flowchart of another service data consistency rate verification method provided by an embodiment of the present application is used for consistency rate verification of batch data to be verified, where the method includes:
s21: front end configuration process.
Specifically, in some alternative embodiments, S21 may be implemented by the following procedure (not shown in the figures):
s211: and acquiring data to be verified of the tested party system in real time.
S212: and determining a 4-data check interface of the tested party system.
S213: and configuring a unique identification field of the data to be verified.
S214: and determining the verification execution time according to the input information of the user.
S22: and (5) dynamically splitting the verification process.
Specifically, in some alternative embodiments, S22 may be implemented by the following procedure (not shown in the figures):
s221: and generating a verification task based on the data to be verified.
S222: if the verification workload of the task to be verified exceeds the current task execution capacity of the processor, splitting the verification task into at least two subtasks according to the verification workload and the current task execution capacity of the processor.
S223: the current task execution capability of the processor is split into at least two subtask lines, each subtask line comprising a data comparison function, an update task generation function and a data synchronization function.
S224: and each subtask searches a subtask line meeting the corresponding workload requirement according to the idle condition of the subtask line to be combined to form a task execution group.
S225: and after each task execution group executes and completes the corresponding subtask, releasing the corresponding task line resource.
In some alternative embodiments, the data synchronization function in S223 may be implemented by the following procedure (not shown in the figure):
s2231: and inquiring target data information corresponding to the unique identification field from the tested party system.
S2232: and returning the target data information to the verification initiating terminal for field-by-field comparison, searching for difference content, generating an update script taking the statement as a core, and transmitting the update script back to the system of the tested party so as to finish consistency synchronization of the system of the tested party.
In some alternative embodiments, in S224, finding a subtask line that meets the corresponding workload requirement is combined as follows: according to the following formula
Searching a subtask line meeting the corresponding workload requirement for combination, wherein R is the resource ratio allocated by a tested side server for executing the task, p is the expectation that the tested side server end required by the task to be executed generates execution capacity resources, and R is the resource ratio allocated by the tested side server for executing the task>As an adjustable coefficient, determining the influence of the task to be executed on the allocation resource of the server to be tested, wherein the value is greater than 1,/for>For the anticipated occupation of the server resources of the tested party, the value of p is smaller than the value of p, and the value of +.>Whether the core program of the server to be tested is executed or not, and the value is 0 or 1, which respectively represents two states of being executed and not being executed, and the value is +.>Is the remainder of the server resources of the tested party.
S23: history checking task recording process.
Specifically, in some alternative embodiments, S23 includes (not shown in the figures):
s231: any history checking task is obtained.
S232: and acquiring verification data corresponding to the history verification task.
S233: if the verification data and the associated data are not changed again, finishing the rollback operation and returning to the state before updating.
The time points of the history verification task are stored at the history verification task recording process end in a snapshot mode, and corresponding data are directly changed according to snapshot information during rollback.
In some alternative embodiments, if the system is a new system, when the check address dictionary field is added, the new system data check interface address is configured to quickly increase the system.
It can be understood that the technical scheme provided by the embodiment is used for checking the consistency rate of the batch data to be checked, and comprises a dynamic splitting checking process, wherein when the checking workload of the task to be checked exceeds the current task execution capacity of the processor, the dynamic splitting checking process splits the checking task into at least two subtasks according to the checking workload and the current task execution capacity of the processor; splitting the current task execution capacity of the processor into at least two subtask lines, wherein each subtask line comprises a data comparison function, an update task generation function and a data synchronization function; each subtask searches a subtask line meeting the corresponding workload requirement according to the idle condition of the subtask line to be combined to form a task execution group; after each task execution group executes and completes the corresponding subtasks, corresponding task line resources are released, verification is completed quickly through task and function block combination allocation, the method is suitable for batch data verification, automatic completion is achieved, and energy consumption improving efficiency is reduced.
Example 3
As shown in fig. 3, the embodiment of the present application further provides a service data consistency rate verification device, which is used for consistency rate verification of batch data to be verified, where the device includes a dynamic split verification module 31, and the dynamic split verification module 31 includes:
the verification task generating sub-module 311 is configured to generate a verification task based on the data to be verified.
The task splitting sub-module 312 is configured to split the verification task into at least two sub-tasks according to the verification workload and the current task execution capability of the processor if the verification workload of the task to be verified exceeds the current task execution capability of the processor.
The execution capability splitting submodule 313 is configured to split the current task execution capability of the processor into at least two subtask lines, where each subtask line includes a data comparison function, an update task generation function, and a data synchronization function.
And the combined execution sub-module 314 is configured to search a sub-task line meeting the corresponding workload requirement for each sub-task according to the idle condition of the sub-task line to combine to form a task execution group.
The resource management sub-module 315 is configured to release the corresponding task line resource after each task execution group completes the corresponding sub-task.
In some alternative embodiments, the apparatus further comprises: front end configuration module 32, front end configuration module 32 includes:
and the verification data acquisition sub-module 321 is used for acquiring the data to be verified of the tested party system in real time.
An interface validation sub-module 322 is configured to determine a data verification interface of the system under test.
The identifier configuration submodule 323 is configured to configure a unique identification field of the data to be verified.
A root execution time validation sub-module 324 for determining a verification execution time based on the user input information.
In some alternative embodiments, the execution capability splitting submodule 313 includes:
the data query unit 3131 is configured to query, from the system of the tested party, the target data information corresponding to the unique identification field.
The data comparison unit 3132 is configured to return the target data information to the verification initiator for field-by-field comparison, search for the difference content, generate an update script with a statement as a core, and return the update script to the system of the tested party, so that the system of the tested party completes consistency synchronization.
In some alternative embodiments, the apparatus further comprises: the history verification task recording module 33, the history verification task recording module 33 includes:
the history task obtaining sub-module 331 is configured to obtain any history check task.
The historical data obtaining sub-module 332 is configured to obtain the verification data corresponding to the historical verification task.
The rollback module 333 is configured to complete the rollback operation and return to the state before updating if the check data and the associated data are not changed again.
In some alternative embodiments, the combined execution sub-module 314 is formulated as follows
Searching a subtask line meeting the corresponding workload requirement for combination, wherein R is the resource ratio allocated by the server to be tested for executing the task, p is the expected of generating execution capacity resources for the server to be tested required by the task to be executed, and->As an adjustable coefficient, determining the influence of the task to be executed on the allocation resource of the server to be tested, wherein the value is greater than 1,/for>For the anticipated occupation of the server resources of the tested party, the value of p is smaller than the value of p, and the value of +.>Whether the core program of the server to be tested is executed or not, and the value is 0 or 1, which respectively represents two states of being executed and not being executed, and the value is +.>Is the remainder of the server resources of the tested party.
In some alternative embodiments, in the history verification task recording process, the time points of the history verification task are stored in a snapshot form at the history verification task recording process end, and corresponding data is directly changed according to snapshot information when the record rolls back.
In some alternative embodiments, if the system is a new system, when the check address dictionary field is added, the new system data check interface address is configured to quickly increase the system.
It can be understood that the technical scheme provided by the embodiment is used for checking the consistency rate of the batch data to be checked, and comprises a dynamic splitting checking process, wherein when the checking workload of the task to be checked exceeds the current task execution capacity of the processor, the dynamic splitting checking process splits the checking task into at least two subtasks according to the checking workload and the current task execution capacity of the processor; splitting the current task execution capacity of the processor into at least two subtask lines, wherein each subtask line comprises a data comparison function, an update task generation function and a data synchronization function; each subtask searches a subtask line meeting the corresponding workload requirement according to the idle condition of the subtask line to be combined to form a task execution group; after each task execution group executes and completes the corresponding subtasks, corresponding task line resources are released, verification is completed quickly through task and function block combination allocation, the method is suitable for batch data verification, automatic completion is achieved, and energy consumption improving efficiency is reduced.
Example 4
Based on the same technical concept, the embodiment of the application also provides a computer device, which comprises a memory 1 and a processor 2, as shown in fig. 4, the memory 1 stores a computer program, and the processor 2 implements the service data consistency rate checking method of any one of the above items when executing the computer program.
The memory 1 includes at least one type of readable storage medium including flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 1 may in some embodiments be an internal storage unit of a service data consistency check system, such as a hard disk. The memory 1 may in other embodiments also be an external storage device of a service data consistency check system, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like. Further, the memory 1 may also include both an internal storage unit and an external storage device of the service data consistency rate verification system. The memory 1 may be used not only for storing application software installed in the service data consistency rate verification system and various types of data, such as codes of service data consistency rate verification programs, etc., but also for temporarily storing data that has been output or is to be output.
The processor 2 may in some embodiments be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor or other data processing chip for running program code or processing data stored in the memory 1, e.g. executing a traffic data consistency rate check program or the like.
It can be understood that the technical scheme provided by the embodiment is used for checking the consistency rate of the batch data to be checked, and comprises a dynamic splitting checking process, wherein when the checking workload of the task to be checked exceeds the current task execution capacity of the processor, the dynamic splitting checking process splits the checking task into at least two subtasks according to the checking workload and the current task execution capacity of the processor; splitting the current task execution capacity of the processor into at least two subtask lines, wherein each subtask line comprises a data comparison function, an update task generation function and a data synchronization function; each subtask searches a subtask line meeting the corresponding workload requirement according to the idle condition of the subtask line to be combined to form a task execution group; after each task execution group executes and completes the corresponding subtasks, corresponding task line resources are released, verification is completed quickly through task and function block combination allocation, the method is suitable for batch data verification, automatic completion is achieved, and energy consumption improving efficiency is reduced.
In order to facilitate the reader to understand the technical scheme of the embodiment of the application, the technical details in the scheme are described in detail below through specific examples.
According to the service data consistency rate verification method and device, the service data auditing tool is constructed, the data information to be verified is obtained in real time, the difference information is automatically adjusted, the standardized access specification is defined, new system data detection can be rapidly supported through configuration, verification is rapidly completed through task and function block combination allocation, energy consumption of service data consistency rate verification is effectively reduced, and efficiency is improved.
The method comprises a front-end configuration process, a dynamic splitting verification process and a historical verification task recording process, wherein an operator configures data information to be verified of a verifier through the front-end configuration process, executes a resource allocation method based on a dynamic task, sequentially acquires the data to be verified, then compares the data with a main system to generate a result, judges whether the result has a difference, and automatically adjusts and synchronizes at least the tested party to finish updating by taking the main system as a standard. When the tested party system is a new system, the data dictionary of the new service system is defined in the front-end configuration process, so that different tested party systems can be supported. And (3) a rollback mechanism is provided, and data in each dynamic splitting verification process is reversely operated according to the history verification task recording process.
The operator configures a data verification task according to the information of the tested party, obtains the related data of the tested party after the execution, checks the record one by one, generates a data update task according to the verification result, executes the data update task, records the execution result, and performs reverse operation on the executed result according to any data update task when the rollback history data is required.
The specific steps are as follows:
step 1: the operator configures the data consistency rate checking task, which comprises four aspects of selecting a system data checking interface of a tested party, configuring and comparing checked data, automatically updating or not and executing time.
Data checking interface of the tested party system: and the system is responsible for receiving the unique identification field of the tested data, returning all information of the data, and if the system is a newly added system, configuring the data check interface address of the new tested system at the added check address dictionary column, thereby being capable of rapidly adding the tested system.
Configuration comparison check data: according to the test requirement, configuring the unique identification field of the data to be compared and checked, supporting the importing of single data or batch data, and providing a standard text format template.
Automatic updating: and providing an automatic updating option, defaulting to automatic, and automatically updating the data in the database of the tested system by taking the main system as a standard.
Execution time: the execution time option is provided, the default is immediate execution, and the service can be selected to be executed when idle according to actual conditions.
Step 2: after the data consistency rate verification task is executed, splitting the data consistency rate verification task into a plurality of task lines according to the number of the tasks, searching for an idle task group by a task head, completing data processing by means of comparison carried by the task group, generating a script and synchronous updating functions, specifically, inquiring corresponding data information of a target system according to a unique identifier, returning to a verification starting end to perform field comparison one by one, searching for difference content, further generating an updating script taking a statement as a core, and returning to a tested party system. For example: the data A comprises fields a1, a2, a3, a4 and a5, wherein a unique field is a1, the complete information of the corresponding data A is queried in a system of a tested party according to the a1, the complete information comprises a1, a2, a3, a4 and a5, whether the two a 1-a 5 have differences or not is compared, and an update script is generated according to the differences.
Step 3: the tested side system automatically executes the update script to complete data consistency synchronization with the main system, meanwhile records the data consistency synchronization as a history task, a consistency rate test initiator can select any history task, and when the system determines that the data and the associated data are not changed again, rollback operation can be completed and the data can be returned to a state before consistency rate verification is more displayed.
Fig. 5 is a functional block diagram of the service data consistency rate verification device provided in the present embodiment.
The concrete explanation is as follows:
front end configuration process: the test method comprises three blocks of check data, tested party configuration and execution time, wherein the check data provides a form filled and imported by tested data, and the form is transmitted to a data check interface after being submitted. The configuration of the tested party is to add, delete and edit the system which can be checked by configuring the interface address dictionary, and the added system can be selected when the front end configures the check data. The execution time may configure a specific time node or period of the data consistency rate verification task.
Data checking interface: and acquiring checked data, managing, scheduling and monitoring the data returned to the tested system through a task group in a dynamic splitting checking process, and generating an updating task according to the difference after comparison and checking.
Dynamic splitting and checking process: the system comprises a task group management module and a task related management change module, wherein each task group comprises data comparison, task generation and data synchronization capability updating, when data are verified in batches, the tasks are split, each sub-task carries the task group capability through task scheduling, the task execution condition is monitored in real time, and resources occupied by the tasks during execution are managed and controlled.
History verification task recording process: the method comprises the steps of checking details, changing details and task rollbacks, wherein the checking details record the successful and failed execution conditions of monotone data check in checking tasks, the changing details are synchronous task conditions for guaranteeing the execution of data consistency rate, the task rollbacks are obtained by selecting executed tasks, reversely changing synchronous data processes, recorded time points are stored at a historical checking task recording process end in a snapshot mode, and corresponding data are directly changed according to snapshot information during rollbacks.
Fig. 6 is a schematic process diagram of a dynamic splitting verification process, in order to ensure smooth execution of the whole data consistency rate verification process, splitting tasks according to detected data magnitude, further searching idle task execution capacity for combination, independently executing "data comparison", "change task generation" and "data synchronization" tasks, completing resources occupied by the latter tasks, restoring the task execution capacity, waiting for new splitting tasks to be combined, and distributing and splitting the original linear task execution flow by using a dynamic task execution resource allocation method, so that system resources are effectively utilized, and efficiency is comprehensively improved.
The concrete explanation is as follows:
task splitting and execution capabilities: the service data consistency ratio comparison method is not performed linearly, when the data magnitude of comparison verification is too large, the whole task is split, a certain amount of task execution capacity is generated according to system resources, and single tasks waiting for splitting are combined.
The expected allocation resource of the task execution capacity is determined according to the waiting task quantity of the master check party and the factors such as the current execution task condition of the tested party, the occupation of the server resource of the tested party and the like, and the calculation method comprises the following steps:
wherein R is the resource allocated by the server to execute the taskThe source duty ratio, p is the expectation that the server of the tested party generates the execution capacity resource required by the task to be executed,/I>As an adjustable coefficient, determining the influence of the task to be executed on the allocation resource of the server to be tested, wherein the value is greater than 1,/for>For the anticipated occupation of the server resources of the tested party, the value of p is smaller than the value of p, and the value of +.>Whether the core program of the server to be tested is executed or not, and the value is 0 or 1, which respectively represents two states of being executed and not being executed, and the value is +.>The value of the residual condition of the server resource of the tested party can be debugged.
For example: if the tested side server is executing the main program, the expected execution capacity resource generated by the tested side server side required by the task to be executed is 5,aggressive debugging to 1.2 (representing that the task to be performed has a very large influence on the allocation of resources by the tested party), calculating +.>=13.79%, then the measured party allocates 13.79% of task-capable resources.
For example: if the subject does not execute the main routine, p remains 5,aggressive debugging is 1.2, and the expected q value of the server resource occupation of the tested party is smaller than p and is set to be 4, < ->The adjustment is defined as 2.
And (3) calculating:，
the measured party allocates 80.46% of the task execution capacity resources.
The combination is performed: the split tasks are combined with the execution capacity to form an independent channel, the task does not need to wait for the execution completion of the task, other tasks can be combined with the idle execution capacity to form a new task execution channel, and a distributed task execution mode is realized.
And (3) resource release: after the execution of the single task is completed, removing resources occupied by the split task, recovering occupied data comparison, updating task generation and data synchronization capacity resources, and waiting for further combination of the new split task. The task execution capacity level is determined according to the system hardware resources, and when the hardware resources are enough to cover the split task amount, dynamic task execution resource allocation is not needed in a distributed mode.
In order to ensure the smooth execution of the whole data consistency rate verification process, task splitting is carried out according to the detected data magnitude, idle task execution capacity is further searched for and combined, the 'data comparison', 'change task generation' and 'data synchronization' tasks are independently executed, resources occupied by the latter tasks are completed, the task execution capacity is restored to wait for a new splitting task to be combined, and the dynamic task execution resource allocation method is used for distributing and splitting the original linear task execution flow, so that system resources are effectively utilized, and the efficiency is comprehensively improved.
The disclosed embodiments also provide a computer readable storage medium having a computer program stored thereon, which when executed by a processor performs the steps of the business data consistency rate verification method in the method embodiments described above. Wherein the storage medium may be a volatile or nonvolatile computer readable storage medium.
The computer program product of the service data consistency rate verification method provided by the embodiment of the application comprises a computer readable storage medium storing program codes, and the instructions included in the program codes can be used for executing the steps of the service data consistency rate verification method in the method embodiment, and the specific reference can be made to the method embodiment, and the details are not repeated here.
The disclosed embodiments also provide a computer program which, when executed by a processor, implements any of the methods of the previous embodiments. The computer program product may be realized in particular by means of hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied as a computer storage medium, and in another alternative embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), or the like.
It is to be understood that the same or similar parts in the above embodiments may be referred to each other, and that in some embodiments, the same or similar parts in other embodiments may be referred to.
It should be noted that in the description of the present application, the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Furthermore, in the description of the present application, unless otherwise indicated, the meaning of "plurality" means at least two.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
Those of ordinary skill in the art will appreciate that all or a portion of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, where the program may be stored in a computer readable storage medium, and where the program, when executed, includes one or a combination of the steps of the method embodiments.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing module, or each unit may exist alone physically, or two or more units may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product.
The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present application have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the application, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the application.

Claims (8)

1. The service data consistency rate verification method is characterized by comprising a dynamic splitting verification process, wherein the dynamic splitting verification process comprises the following steps of:
generating a verification task based on the data to be verified;
if the verification workload of the task to be verified exceeds the current task execution capacity of the processor, splitting the verification task into at least two subtasks according to the verification workload and the current task execution capacity of the processor;
splitting the current task execution capacity of a processor into at least two subtask lines, wherein each subtask line comprises a data comparison function, an update task generation function and a data synchronization function;
each subtask searches a subtask line meeting the corresponding workload requirement according to the idle condition of the subtask line and combines the subtask lines to form a task execution group;
after each task execution group executes the corresponding subtask, releasing the corresponding task line resource;
further comprises: a history verification task recording process, the history verification task recording process comprising: acquiring any history verification task; acquiring verification data corresponding to the history verification task; if the check data and the associated data are not changed again, finishing rollback operation, and returning to a state before updating;
the finding of a subtask line meeting the corresponding workload requirement is combined as follows: according to the following formula
Searching a subtask line meeting the corresponding workload requirement for combination, wherein R is the resource ratio allocated by a tested side server for executing the task, p is the expectation that the tested side server end required by the task to be executed generates execution capacity resources, and R is the resource ratio allocated by the tested side server for executing the task>As an adjustable coefficient, determining the influence of the task to be executed on the allocation resource of the server to be tested, wherein the value is greater than 1,/for>For the anticipated occupation of the server resources of the tested party, the value of p is smaller than the value of p, and the value of +.>Whether the core program of the server to be tested is executed or not, and the value is 0 or 1, which respectively represents two states of being executed and not being executed, and the value is +.>Is the remainder of the server resources of the tested party.
2. The traffic data consistency rate verification method according to claim 1, wherein prior to the dynamic split verification process, the method further comprises a front-end configuration process comprising:
acquiring the data to be verified of the tested party system in real time;
determining a data verification interface of the tested party system;
configuring a unique identification field of the data to be verified;
and determining the verification execution time according to the input information of the user.
3. The traffic data consistency rate verification method according to claim 2, wherein the data synchronization function comprises:
inquiring target data information corresponding to the unique identification field from a tested party system;
and returning the target data information to a verification initiating terminal for field-by-field comparison, searching for difference content, generating an update script taking a statement as a core, and returning to the tested party system so as to enable the tested party system to complete consistency synchronization.
4. The business data consistency rate checking method according to claim 3, wherein in the history checking task recording process, the time points of the history checking task are stored in a snapshot form at the history checking task recording end, and corresponding data are changed directly according to snapshot information during rollback.
5. The traffic data consistency rate verification method according to claim 4, further comprising:
if the tested party system is a new system, when a check address dictionary column is added, configuring a new tested party system data check interface address so as to quickly increase the tested party system.
6. The utility model provides a business data consistency rate verifying attachment, its characterized in that is used for the consistency rate check of data that wait to check in batches, the device includes the dynamic split check module, the dynamic split check module includes:
the verification task generation sub-module is used for generating a verification task based on data to be verified;
the task splitting sub-module is used for splitting the checking task into at least two sub-tasks according to the checking workload and the current task execution capacity of the processor if the checking workload of the task to be checked exceeds the current task execution capacity of the processor;
the execution capacity splitting sub-module is used for splitting the current task execution capacity of the processor into at least two subtask lines, wherein each subtask line comprises a data comparison function, an update task generation function and a data synchronization function;
the combined execution sub-module is used for searching a sub-task line meeting the corresponding workload requirement according to the idle condition of the sub-task lines for each sub-task to be combined to form a task execution group;
the resource management sub-module is used for releasing corresponding task line resources after each task execution group executes and completes corresponding sub-tasks;
further comprises: the history check task record module, history check task record module includes: the history task acquisition sub-module is used for acquiring any history verification task; the historical data acquisition sub-module is used for acquiring verification data corresponding to the historical verification task; the rollback module is used for finishing rollback operation and returning to a state before updating if the check data and the associated data are not changed again;
the combined execution submodule is according to the formulaSearching a subtask line meeting the corresponding workload requirement for combination, wherein R is the resource ratio allocated by the server to be tested for executing the task, p is the expected of generating execution capacity resources for the server to be tested required by the task to be executed, and->As an adjustable coefficient, determining the influence of the task to be executed on the allocation resource of the server to be tested, wherein the value is greater than 1,/for>For the anticipated occupation of the server resources of the tested party, the value of p is smaller than the value of p, and the value of +.>Whether the core program of the server to be tested is executed or not, and the value is 0 or 1, which respectively represents two states of being executed and not being executed, and the value is +.>Is the remainder of the server resources of the tested party.
7. A computer device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication via the bus when the computer device is running, the machine-readable instructions when executed by the processor performing the business data consistency rate checking method according to any of claims 1 to 5.
8. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, performs the service data consistency rate verification method according to any of claims 1 to 5.
CN202311213203.4A 2023-09-20 2023-09-20 Service data consistency rate verification method and device Active CN116955342B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311213203.4A CN116955342B (en) 2023-09-20 2023-09-20 Service data consistency rate verification method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311213203.4A CN116955342B (en) 2023-09-20 2023-09-20 Service data consistency rate verification method and device

Publications (2)

Publication Number Publication Date
CN116955342A CN116955342A (en) 2023-10-27
CN116955342B true CN116955342B (en) 2023-12-15

Family

ID=88449569

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311213203.4A Active CN116955342B (en) 2023-09-20 2023-09-20 Service data consistency rate verification method and device

Country Status (1)

Country Link
CN (1) CN116955342B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108271255A (en) * 2016-12-30 2018-07-10 中国移动通信集团上海有限公司 A kind of method and device of business datum shunting
CN113326126A (en) * 2021-05-28 2021-08-31 湘潭大学 Task processing method, task scheduling device and computer equipment
CN114139220A (en) * 2021-12-14 2022-03-04 北京中电普华信息技术有限公司 Data verification method and device
WO2023011157A1 (en) * 2021-08-02 2023-02-09 腾讯科技（深圳）有限公司 Service processing method and apparatus, server, storage medium, and computer program product

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012052775A1 (en) * 2010-10-21 2012-04-26 Bluwireless Technology Limited Data processing systems
CN113127058B (en) * 2021-04-28 2024-01-16 北京百度网讯科技有限公司 Data labeling method, related device and computer program product

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108271255A (en) * 2016-12-30 2018-07-10 中国移动通信集团上海有限公司 A kind of method and device of business datum shunting
CN113326126A (en) * 2021-05-28 2021-08-31 湘潭大学 Task processing method, task scheduling device and computer equipment
WO2023011157A1 (en) * 2021-08-02 2023-02-09 腾讯科技（深圳）有限公司 Service processing method and apparatus, server, storage medium, and computer program product
CN114139220A (en) * 2021-12-14 2022-03-04 北京中电普华信息技术有限公司 Data verification method and device

Also Published As

Publication number Publication date
CN116955342A (en) 2023-10-27

Similar Documents

Publication Publication Date Title
RU2429529C2 (en) Dynamic configuration, allocation and deployment of computer systems
CN109995677B (en) Resource allocation method, device and storage medium
CN111221726A (en) Test data generation method and device, storage medium and intelligent equipment
US8448181B2 (en) Sizing an infrastructure configuration optimized for a workload mix
WO2010028868A1 (en) Method and system for sharing performance data between different information technology product/solution deployments
CN112581018B (en) Method, system, device and storage medium for managing process tasks
CN112463450A (en) Incremental backup management method, system, electronic equipment and storage medium
CN109460358B (en) Configuration management method and system based on Cmdb
CN109597764A (en) A kind of test method and relevant apparatus of catalogue quota
WO2021135742A1 (en) Reconciliation and liquidation method and apparatus
CN116955342B (en) Service data consistency rate verification method and device
CN115984022B (en) Unified account checking method and device for distributed payment system
CN111694748A (en) Pressure testing method and device
CN109389271B (en) Application performance management method and system
TWI236619B (en) Automatic generation system and method for hardware-based testing script
CN115576706A (en) Method and device for interfacing with third-party system, electronic equipment and readable medium
CN113342512A (en) IO task silencing and driving method and device and related equipment
CN113010276A (en) Task scheduling method and device, terminal equipment and storage medium
CN112905461A (en) Method and device for executing automatic interface test case
CN110971664A (en) Interface service management system
CN111930718B (en) Node adjustment method and device for configuration management database
CN113206882B (en) Consensus method, computer device and storage medium
CN109002287B (en) Management method and device for software in cloud data system
CN116303361A (en) Service data migration method, device and system
CN117389978A (en) Automatic importing method and device for interface file, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant