WO2021174696A1

WO2021174696A1 - Data updating method and apparatus, computer device, and storage medium

Info

Publication number: WO2021174696A1
Application number: PCT/CN2020/093327
Authority: WO
Inventors: 周亮
Original assignee: 平安科技（深圳）有限公司
Priority date: 2020-03-06
Filing date: 2020-05-29
Publication date: 2021-09-10
Also published as: CN111475517A; CN111475517B

Abstract

Provided are a data updating method and apparatus, a computer device, and a storage medium, relating to the technical field of big data. The method comprises: acquiring unique data records and write-in times thereof from a configuration table in a front-end database; parsing a synchronous data time range in a data reading instruction, performing traversing and querying to obtain the write-in times that match the synchronous data time range and the unique data records associated with the various matched write-in times, and marking, as data identifiers to be subjected to reading, the unique data records associated with the various write-in times that match the synchronous data time range; querying an interface table for the index data corresponding to each of said data identifiers, determining, according to a segmentation symbol, batches to be subjected to reading, to which the index data corresponding to said data identifiers belongs, and recording all the index data of said batches as newly-added batch index data; and updating the newly-added batch index data of all the batches and storing same in a target data table of a back-end database. Accordingly, the operation performance of a back-end server can be improved.

Description

Data update method, device, computer equipment and storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on March 6, 2020, with the application number 202010152296.4, titled "data update method, device, computer equipment and storage medium", the entire content of which is incorporated by reference In this application.

Technical field

This application relates to the field of data update in the field of big data technology, and in particular to a data update method, device, computer equipment, and storage medium.

Background technique

At present, in the data processing scenario of the database, the front end needs to write the indicator data in an interface table for the back end to obtain and store the indicator data. In order to ensure the integrity of the back-end data, it is necessary to obtain all the indicator data in the interface table. At present, there is a method that synchronizes index data in the database where the back-end is located according to time partition. The inventor realizes that this method may easily lead to a large time span of part of the index data and cause the problem of incomplete data update. Another method is to synchronize according to the indicators, but the inventor realized that this method requires multiple corresponding programs on the back end when there are multiple indicators. As the indicators continue to grow, the back end needs to be synchronized. There will be more and more programs, which will cause problems that are difficult to develop and maintain. Moreover, the inventor found that the back-end of the above two methods synchronizes the entire interface table each time, which will cause the database where the back-end is located to accumulate too much useless data, which will affect the performance of the back-end server and increase the running tasks of the back-end server. Running time. Therefore, those skilled in the art urgently need to find a technical solution to solve the above-mentioned technical problems.

Application content

Based on this, it is necessary to provide a data update method, device, computer equipment, and storage medium for the above technical problems to improve the running performance of the back-end server and reduce the running time of the running task of the back-end server.

A data update method includes:

When receiving the reading instruction, the back-end server obtains all unique data records of the configuration table in the front-end database of the front-end server, and the writing time associated with the unique data record; wherein, the unique data record refers to the The index data written in batches in the interface table of the database, and at the same time the unique identifier of the configuration is updated in the configuration table for the first index data in the batch of the index data, and the unique data The record is stored in the configuration table in association with the write time when the indicator data of the batch is written into the interface table; the indicator data of each batch includes at least one indicator data, and the interface In the table, the index data of each batch is divided by dividing symbols;

The back-end server parses the synchronization data time range in the reading instruction, and traverses queries that match the synchronization data time range among all the write times and the unique data records obtained from the configuration table The writing time and the unique data record associated with each writing time matching the synchronization data time range, and marking the unique data record associated with each writing time matching the synchronization data time range Identifies the data to be read;

The back-end server queries the interface table for index data corresponding to each of the data identifiers to be read, and determines, according to the segmentation symbol in the interface table, that the index data corresponding to the data identifier to be read belongs to Batches to be read, and record all index data of the batches to be read as new batch index data;

After the back-end server obtains the newly added batch index data of all batches from the interface table, it updates the newly-added batch index data of all the read batches to the back-end database stored in the back-end server. In the target data table, the unique data record corresponding to the first index data in the index data of all the batches to be read and the write time stored in association therewith are deleted from the configuration table at the same time.

A data update device includes:

The obtaining module is used to obtain all the unique data records of the configuration table in the front-end database of the front-end server and the writing time associated with the unique data record when the back-end server receives the reading instruction; wherein, the unique data record is Refers to the indicator data written in batches in the interface table of the front-end database, and at the same time update the configuration unique identifier for the first indicator data in the batch of the indicator data in the configuration table, And the unique data record is stored in the configuration table in association with the write time when the indicator data of the batch is written into the interface table; the indicator data of each batch includes at least one indicator Data, the index data of each batch is divided by a division symbol in the interface table;

The first marking module is used for the back-end server to parse the synchronization data time range in the reading instruction, to traverse the query and all the writing times and the unique data records obtained from the configuration table. The write time that matches the synchronization data time range, and the unique data record associated with each write time that matches the synchronization data time range, and each write time that matches the synchronization data time range The unique data record mark associated with time is the data identification to be read;

The recording module is used for the back-end server to query the index data corresponding to each of the data identifiers to be read in the interface table, and determine the data identifiers to be read according to the segmentation symbol in the interface table The batch to be read to which the corresponding index data belongs, and all the index data of the batch to be read are recorded as the new batch index data;

The storage module is configured to update and store the new batch index data of all read batches after the back-end server obtains the new batch index data of all batches from the interface table In the target data table of the back-end database, delete the unique data record corresponding to the first index data in all the index data of the batch to be read and all the data stored in association therewith in the configuration table. The writing time.

A computer device includes a memory, a processor, and computer-readable instructions that are stored in the memory and can run on the processor, and the processor implements the following steps when the processor executes the computer-readable instructions:

One or more readable storage media storing computer readable instructions, when the computer readable instructions are executed by one or more processors, the one or more processors execute the following steps:

The above-mentioned data update method, device, computer equipment and storage medium continuously verify whether the index data of each batch of the interface table is the new batch index data in the interface table of the database, and will meet the conditions (including each batch index The write time corresponding to the data should fall within the synchronization data time range). The new batch indicator data is updated and stored in the target data table where the back-end database is located, for the purpose of incremental update, in the target data table The data will be more in line with the use of the back-end database, and the back-end database where the target data table is located will not accumulate too much useless data (the indicator data of each historical batch that has been synchronized to the target data table, if you re-synchronize , It will cause repeated batches of indicator data in the target data table), the operating performance of the back-end database will not be greatly affected, that is, the back-end server corresponding to the back-end database will not be affected, so through this Applying can improve the running performance of the back-end server and reduce the running time of the running tasks of the back-end server. The details of one or more embodiments of the present application are presented in the following drawings and description, and other features and advantages of the present application will become apparent from the description, drawings and claims.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following will briefly introduce the drawings that need to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

FIG. 1 is a schematic diagram of an application environment of a data update method in an embodiment of the present application;

Figure 2 is a flowchart of a data update method in an embodiment of the present application;

3 is a schematic diagram of the structure of a data update device in an embodiment of the present application;

Fig. 4 is a schematic diagram of a computer device in an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The data update method provided in this application can be applied in the application environment as shown in Fig. 1, in which the client communicates with the back-end server through the network. Among them, the client can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The back-end server can be implemented as an independent server or a server cluster composed of multiple servers.

In an embodiment, as shown in FIG. 2, a method for updating data is provided. The method is applied to the server in FIG. 1 as an example for description, and includes the following steps:

S10. When receiving the reading instruction, the back-end server obtains all unique data records of the configuration table in the front-end database of the front-end server, and the writing time associated with the unique data record; wherein, the unique data record refers to the The index data written in batches in the interface table of the front-end database, and at the same time update the configuration unique identifier for the first index data in the batch of the index data in the configuration table, and The unique data record is stored in the configuration table in association with the write time when the indicator data of the batch is written into the interface table; the indicator data of each batch includes at least one indicator data, so In the interface table, the index data of each batch is divided by a division symbol;

Understandably, both the interface table and the configuration table are located in the front-end database of the front-end server; the index data of each batch is written into the interface table in the front-end server (in batch The data is written in the form of secondary data, and the indicator data in a preset time or a business scenario can be used as a batch of indicator data, such as indicator data within 10 minutes as a batch of indicator data, a data of the company The indicator data in the report is used as a batch of indicator data, etc.), and the indicator data of each batch in the interface table is divided by a dividing symbol (the dividing symbol can be a comma, a slash, and other symbols that have the meaning of dividing); Data record refers to a unique identifier (the unique identifier) generated corresponding to the first indicator data in the indicator data of the newly added batch written in the interface table when the indicator data is added in batches in the interface table. Identifier is the key information extracted from the index data. The key information includes but is not limited to the index type. You can compare the key information table of the preset index data to extract the key information of the index data, and then form the unique identifier); In this embodiment, all index data that has a corresponding unique data record in the configuration table are newly-added index data. Understandably, since the index data is written into the interface table in batches, in a batch When there is a newly-added index data in the index data, it means that all the index data in the batch are newly-added index data.

In this embodiment, since the index data is imported into the interface table in batches, the index data between the two division symbols can be presented in the interface table through the division symbol in the form of index data in one batch, and the index data Importing data in batches can reduce the time required for multiple importing, and it is also convenient to query indicator data in the interface table. Since the only data record in the configuration table is only the first indicator data of a batch of indicator data, In the subsequent data processing process, the load level in the configuration table can be reduced, and the speed and efficiency of data processing can be improved at the same time.

S20. The back-end server parses the synchronization data time range in the reading instruction, and traverses the query and the synchronization data time range from all the write times and the unique data records obtained from the configuration table The matched write time, and the unique data record associated with each write time that matches the synchronization data time range, and the unique data associated with each write time that matches the synchronization data time range The record mark is the identification of the data to be read;

Understandably, the reading instruction is an instruction that the back-end server needs to read the batch index data (the batch index data that needs to be synchronized to the back-end database) in the interface table for a period of time.

S30. The back-end server queries the index data corresponding to each of the data identifiers to be read in the interface table, and determines the index corresponding to the data identifiers to be read according to the segmentation symbol in the interface table The batch to be read to which the data belongs, and all index data of the batch to be read are recorded as new batch index data;

In this embodiment, after analyzing the synchronization data time range contained in the reading instruction, first traverse the write time matching the synchronization data time range in the query configuration table to determine the unique data associated with the successfully matched write time Record, the unique data recording cup is marked as the data to be read identification; then the indicator data of each batch in the interface table is verified according to the data identification and the segmentation symbol to be read out of the query (first determine and read from the interface table Take the index data that matches the data identifier, and then determine the batch to which the index data belongs. Since two segmentation symbols can determine the index data of a batch, the two segmentation symbols that are closest to the index data before and after the index data are identified. It can be determined whether the indicator data between the two symbols is a batch of indicator data, that is, whether the first indicator data in the batch to which the indicator data belongs is associated with the data identifier to be read, and the batch of indicator data is used as a Object, and verify the first indicator data of this object. When it is determined that the first indicator data is associated with the identifier of the data to be read, it can indicate that the indicator data is new indicator data, or the entire object ( One batch of index data) is a new batch of index data, which prevents the back-end server from verifying the index data of all batches, reduces the workload and working time of the back-end server, and because the data to be read only corresponds to one identifier The first indicator data of the batch of indicator data also reduces the number of queries and verifications of the back-end server, and further reduces the workload and working time of the back-end server.

Further, the determining, according to the segmentation symbol in the interface table, the batch to be read to which the indicator data corresponding to the data identifier to be read belongs belongs to:

After recognizing the first unmarked segmentation symbol, mark the segmentation symbol as recognized;

Re-identify the first segmentation symbol that is located after the first segmentation symbol and is not marked, and use the index data between the two identified segmentation symbols as one batch of the index data, Until the two division symbols between the indicator data of each batch are marked and recognized;

Query and determine the index data with the first sorting order from the index data of each successfully identified batch, and verify whether the first index data is successfully associated with the data identification to be read, And after the association is successful, the batch to be read to which the first indicator data belongs is determined.

In this embodiment, two unmarked segmentation symbols are used to determine the index data of a batch, which can improve the identification efficiency of the index data of each batch, and can also improve the first index of the index data of each batch. The data has to determine the efficiency.

S40. After the back-end server obtains the new batch index data of all batches from the interface table, it updates and stores the back-end server's back-end server with the newly-added batch index data of all read batches. In the target data table of the database, delete the unique data record corresponding to the first indicator data in all the indicator data of the batch to be read and the write-in stored in association therewith in the configuration table at the same time time.

In this embodiment, by continuously verifying in the interface table of the database whether the index data of each batch of the interface table is the new batch index data, and meeting the conditions (including the write time requirements for each batch index data) The new batch indicator data that falls within the synchronization data time range) is updated and stored in the target data table where the back-end database is located, for an incremental update purpose, and the data in the target data table will be more consistent The use of the end database, the backend database where the target data table is located will not accumulate too much useless data (the indicator data of each historical batch that has been synchronized to the target data table, if resynchronized, it will cause the target data table Duplicate indicator data of each batch), the operating performance of the back-end database will not be greatly affected; and the unique data record corresponding to the first indicator data in all the indicator data of the batch to be read will be deleted and The reason for the write time associated with the storage is: because the unique data record and write time are the basis for judging whether the indicator data of the batch is the new batch indicator data, the new batch indicator data is updated in the storage backend After the database, in order to avoid affecting the judgment of the new batch index data next time, the unique data record and writing time associated with the newly synchronized batch index data are deleted from the configuration table.

In the embodiment where step S10 to step S40 are located, first, batch A index data and B batch index data can be newly written in the interface table of the front-end database in the front-end server in the form of batch index data (the interface table There are also historically written batch indicator data, such as: C batch indicator data; historically written batch indicator data refers to the batch indicator data that has been transmitted and stored in the target data table), and A batch After the index data of the second batch and the index data of the batch B are newly written into the interface table, the header and the tail of the index data of the A batch and the index data of the B batch are inserted into the dividing symbol ( For example, there is a segmentation symbol in front of the index data of batch A, there is a segmentation symbol between batch A index data and batch B index data, and there is a segmentation symbol between batch B index data and batch C index data), The dividing symbol can separate the index data of batch A, the index data of batch B and the index data of batch C in the interface table.

In the embodiment where step S10 to step S40 are located, all indicator data in batch A indicator data are written into the interface table as a batch addition at 3 pm on October 1, 2019, and batch B indicators All the indicator data in the data are added to the interface table as a batch at 1 p.m. on October 1, 2019. At this time, a new writing interface is added for the indicator data of batch A and the indicator data of batch B. At the same time as the table, the configuration table in the front-end database also needs to be written in the unique data record corresponding to the first indicator data of batch A indicator data and the first indicator data of batch B indicator data (the only data record is Refers to the type and key data attributes of the first indicator data extracted from the first indicator data). At the same time, in the configuration table, the only data record of batch A indicator data and batch A indicator data are written to the interface The write time of the table (that is, at 3 pm on October 1, 2019) is stored in association, and the unique data record of batch B index data and the write time of batch B index data written to the interface table (that is, 1 pm on October 1, 2019) Associated storage. Understandably, the historically written batch indicator data does not have any unique data record or writing time information corresponding to it in the configuration table, and the historically written batch indicator data is transmitted and stored to the target data table After the mediation, the above-mentioned information in the configuration table has been deleted from the configuration table.

Then, the back-end server parses out the synchronization data time range in the reading instruction. For example, if the synchronization data time range is from 2:00 to 4:00 in the afternoon of the same day, at this time, because the write time of the indicator data of batch A is 3 in the afternoon of the same day At 1 o’clock, the writing time of the indicator data of batch B is 1 o’clock in the afternoon of the same day; at this time, the writing time of indicator data of batch A falls within the synchronization data time range (2 o’clock to 4 o’clock), and batch B The write time of indicator data does not belong to the synchronization data time range. Therefore, the back-end server only obtains the batch A indicator data associated with the write time of the indicator data of batch A that falls within the synchronization data time range from the configuration table. And record the unique data record as the data identifier to be read; then query the A batch indicator data corresponding to the data identifier to be read in the interface table. The specific query process is: first determine from the interface table The unique index data that matches the identification of the data to be read is determined, and then the batch to which the unique index data belongs is determined (determined according to the nearest segmentation symbol before and after the unique index data, that is, the batch determined at this time is Is the index data of batch A). After that, it is determined that the index data of batch A is the newly added batch index data that needs to be transmitted to the target data table (understandably, because in this application, the index of each batch The data is written at the same writing time and a batch corresponding to the writing time is formed, and the indicator data is transferred from the interface table of the front-end database to the target data table of the back-end database on the back-end server. Batch transmission, therefore, when there is a new index data in a batch of index data, it means that all the index data of the batch are newly added index data).

Finally, the back-end server obtains the index data of batch A that is determined to be the new batch index data that needs to be transmitted to the target data table from the interface table, and stores it in the target data table of the back-end database in the back-end server , And delete the write time and unique data record associated with the first indicator data of batch A indicator data in the configuration table at the same time. (Because the writing time of batch B index data does not match the time range of the synchronized data, it is a new batch of index data, but it does not correspond to the reading command, so it does not need to be stored in accordance with this reading command. The target data table, the write time and unique data records associated with batch B indicator data are still stored in the configuration table).

Further, the updating and storing the new batch indicator data of all the batches read in the target data table of the back-end database of the back-end server includes:

Determine the index type corresponding to the newly added batch of index data through the preset index type table;

According to the indicator type, the new batch indicator data corresponding to it is updated and stored in the storage partition corresponding to the indicator type in the target data table where the back-end database is located; one storage partition corresponds to one The indicator type.

Understandably, the indicator type is the data type to which the new batch of indicator data belongs. For example, the currently determined indicator data of 180 data types, and each indicator type can correspond to a storage partition in a target data table (the target data table has Data partitioning by indicator type).

In this embodiment, the newly-added batch index data corresponding to the index type is stored in the storage partition corresponding to the index type in the target data table, so that the newly-added index data can be classified and synchronized to be stored for convenience and acquisition.

Further, before the updating and storing the index data of the newly added batches of all the batches read in the target data table of the back-end database of the back-end server, the method further includes:

Judging by index data verification whether the new batch index data of each batch in the interface table has a preset data abnormality;

If the preset data abnormality occurs in the newly-added batch index data of one of the batches, the newly-added batch index data of the batch is marked and a reminder is issued.

In this embodiment, because in the process of writing the newly added batch index data of each batch into the interface table or the back-end server prepares to obtain the newly added batch index data in the interface table, it may be affected. The interference of external factors (such as unstable computer performance, interfering programs, etc.) causes the indicator data in the interface table to have preset data abnormalities (such as blank data, garbled data, etc.), so the abnormal batch indicators can be marked The data also reminds the preset personnel, so that the preset data abnormality can be solved in time, the generation of erroneous data can be reduced, and the accuracy of the data can be guaranteed.

Further, the acquiring new batch index data of all batches from the interface table includes:

After obtaining the idle first processing thread from multiple threads in the order of thread priority, the batch number associated with the newly added batch index data of each batch is associated with each first processing thread, and issued The newly added batch indicator data of each batch is sent to each of the first processing threads associated with its batch number; the batch number is in accordance with the configuration corresponding to the newly added batch indicator data of each batch The sequence of the writing time in the table is generated;

Verifying whether each of the first processing threads corresponding to the newly added batch index data of each batch currently being processed has a preset process abnormality;

If the preset process exception occurs in one of the first processing threads corresponding to the newly added batch indicator data of a batch, the newly added batch indicators of the batch will be executed in the order of the thread priority. The batch number associated with the data is again associated with the idle second processing thread, and the batch numbers and the preset arrangement sequence associated with the new batch index data of each batch are compared to each batch. Arranging the index data of the newly added batches;

Obtain the newly added batch index data of each batch that has been successfully arranged from the interface table.

Understandably, the smaller the number of the batch number, the earlier the new batch index data associated with the batch number is obtained; the processing thread in the multi-thread is used to run the new batches The task of arranging batch index data.

In this embodiment, the batch number generated by each batch (the batch number is generated according to the order of the writing time in the configuration table corresponding to the newly added batch index data of each batch, where: The earlier the writing time, the smaller the batch number. The new batch indicator data with the smaller batch number is the indicator data that needs to be obtained by Xianbei) after it is associated with each first processing thread, then each batch is issued To each first processing thread associated with its batch number. This method can increase the work efficiency of the first processing thread to process the new batch indicator data of each batch (by batch number as the first processing thread). One processing thread arranges the new batch index data data of each batch, and one processing thread can only be associated with one batch number, that is, one processing can only process one batch of newly added batch index data separately, and there will be no first batch. One processing thread processes multiple batches of newly added batch index data. Therefore, as long as a processing thread recognizes the batch number, it can quickly process the newly added batch index data of a batch associated with the batch number. According to the numerical size of the batch number, the first processing threads with priority order are arranged for the new batch index data of each batch. Among them, the first processing thread has a higher priority order, and the first processing thread has priority. The new batch index data with the smaller number of the batch number, so that the first processing thread can process the newly added batch index data according to the number of the batch number, and the first processing thread will not process each batch purposelessly. Add batch index data to speed up the processing efficiency of the first processing thread); by detecting whether each first processing thread corresponding to the new batch index data currently processed for each batch has a preset process abnormality, in the first processing thread When a preset process exception occurs, a second processing thread is reset for the new batch index data of the batch corresponding to the first processing thread with the preset process exception, so as to ensure the new batch index of each batch The data has a processing thread for processing (that is, the new batch index data will be obtained from the interface table) to ensure that the new batch index data of each batch is processed in a timely manner. In this embodiment, the index data of the newly added batch is acquired according to the processing thread, and the progress of acquiring index data of the newly added batch is accelerated.

Further, after acquiring the new batch index data of all batches from the interface table, it further includes:

Judging whether there is a problem of interruption in obtaining the index data of the newly added batch in the interface table;

If the acquisition interruption problem occurs, the write time corresponding to the newly added batch index data of the batch is recorded in the exception log, and the new batch is acquired in the exception log. The work progress of the batch of index data is increased, and when writing is started next time, the newly added batch of index data of the batch is determined according to the writing time in the abnormal log, and according to the abnormal log The indicator data remaining in the batch of new batch indicator data determined by the work progress and the writing time.

In this embodiment, since in the process of acquiring the new batch indicator data of each batch through the processing thread, the acquisition may be interrupted due to the network, server crash, etc., so the subsequent start-up writing can pass The exception log is used to determine the index data of the new batch corresponding to the last acquisition interruption, and it can also determine the work progress of the last acquisition of the new batch index data of the batch, so as to ensure that the data processing is not interrupted.

To sum up, the above provides a data update method, by continuously verifying whether the index data of each batch of the interface table is the new batch index data in the interface table of the database, and will meet the conditions (including each batch The write time corresponding to the indicator data should fall within the time range of the synchronization data) The new batch indicator data update is stored in the target data table where the back-end database is located, for the purpose of incremental update, the target data table The data in the database will be more in line with the use of the back-end database. The back-end database where the target data table is located will not accumulate too much useless data (it has been synchronized to the indicator data of each historical batch in the target data table. Synchronization will cause repeated batches of indicator data in the target data table), the operating performance of the back-end database will not be greatly affected, that is, the back-end server corresponding to the back-end database will not be affected, so through The application can improve the running performance of the back-end server and reduce the running time of the running tasks of the back-end server.

It should be understood that the size of the sequence number of each step in the foregoing embodiment does not mean the order of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

In one embodiment, a data update device is provided, and the data update device corresponds to the data update method in the above-mentioned embodiment one-to-one. As shown in FIG. 3, the data update device includes an acquisition module 11, a first marking module 12, a recording module 13 and a storage module 14. The detailed description of each functional module is as follows:

The obtaining module 11 is used to obtain all the unique data records of the configuration table in the front-end database of the front-end server and the writing time associated with the unique data record when the back-end server receives the reading instruction; wherein, the unique data record Refers to the index data written in batches in the interface table of the front-end database, and at the same time, in the configuration table, update the unique identifier of the configuration for the first index data in the batch of the index data , And the unique data record is stored in the configuration table in association with the write time when the indicator data of the batch is written into the interface table; each batch of the indicator data includes at least one Index data, the index data of each batch is divided by a division symbol in the interface table;

The first marking module 12 is used for the back-end server to parse the synchronization data time range in the reading instruction, and traverse the query and the unique data records from all the writing times and the unique data records obtained from the configuration table. The write time that matches the synchronization data time range, and the unique data record associated with each write time that matches the synchronization data time range, and each write time that matches the synchronization data time range The unique data record mark associated with the entry time is the data to be read identification;

The recording module 13 is used for the back-end server to query the index data corresponding to each of the data identifiers to be read in the interface table, and determine the data to be read according to the segmentation symbol in the interface table Identify the batch to be read to which the corresponding index data belongs, and record all index data of the batch to be read as the newly added batch index data;

The storage module 14 is configured to update and store the new batch index data of all the read batches after the back-end server obtains the new batch index data of all batches from the interface table In the target data table of the back-end database of the server, delete the unique data record corresponding to the first index data in all the index data of the batch to be read and the associated stored data in the configuration table. The write time.

Further, the recording module includes:

The marking sub-module is used to mark the segmentation symbol as recognized after identifying the first segmentation symbol that has not been marked;

The recognition sub-module is used for re-identifying the first segmentation symbol that is located after the first segmentation symbol and is not marked, and uses the index data between the two identified segmentation symbols as a batch The index data of each batch until the two segmentation symbols between the index data of each batch are marked and recognized;

The first determining sub-module is configured to query and determine the index data whose sorting order is the first from the index data of each batch that has been successfully identified, and verify whether the first index data is consistent with the pending index data. The read data identifier is successfully associated, and after the successful association, the batch to be read to which the first indicator data belongs is determined.

Further, the storage module includes:

The second determining sub-module is used to determine the index type corresponding to the newly added batch of index data through the preset index type table;

The storage sub-module is configured to update and store the new batch index data corresponding to the index type to the storage partition corresponding to the index type in the target data table where the back-end database is located; The storage partition corresponds to one indicator type.

Further, the data update device further includes:

The first judgment module is configured to judge whether the new batch index data of each batch in the interface table has a preset data abnormality through index data verification;

The second marking module is configured to mark the newly-added batch index data of the batch and issue a reminder if the preset data abnormality occurs in the newly-added batch index data of one of the batches.

Further, the storage module includes:

The issuing sub-module is used to obtain the idle first processing thread from multiple threads in the order of thread priority, and then associate the batch number associated with the newly added batch indicator data of each batch with each of the first processing threads. A processing thread, and sends the newly added batch indicator data of each batch to each of the first processing threads associated with its batch number; the batch number is in accordance with the newly added batch of each batch The sequence of the writing time in the configuration table corresponding to the indicator data is generated;

The verification sub-module is used to verify whether each of the first processing threads corresponding to the newly added batch index data of each batch currently processed has a preset process abnormality;

The arranging sub-module is configured to: if one of the first processing threads corresponding to the newly added batch indicator data of a batch has the preset process abnormality, then all the batches of the batch will be processed according to the thread priority order. The batch number associated with the newly added batch index data is again associated with the idle second processing thread, and each batch number and preset associated with the newly added batch index data of each batch The arrangement sequence arranges the index data of the newly added batches of each batch;

The obtaining sub-module is used to obtain the newly-added batch index data of each batch that has been successfully arranged from the interface table.

Further, the data update device further includes:

The second judgment module is configured to judge whether there is a problem of interruption in obtaining the index data of the newly added batch in the interface table;

The determination module is configured to record the write time corresponding to the new batch index data of the batch in the abnormal log if the problem of the acquisition interruption occurs, and record the acquisition in the abnormal log The work progress of the batch of newly added batch index data, and the next time the writing is started, the newly added batch index data of the batch is determined according to the writing time in the abnormal log, according to The work progress in the abnormal log and the remaining indicator data of the new batch of indicator data of the batch determined by the writing time.

For the specific limitation of the data update device, please refer to the above limitation of the data update method, which will not be repeated here. Each module in the above-mentioned data update device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. 4. The computer equipment includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer readable instructions in the readable storage medium. The database of the computer equipment is used to store the data involved in the data update method. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer-readable instruction is executed by the processor to realize a data update method. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and capable of running on the processor. When the processor executes the computer-readable instructions, the data in the above-mentioned embodiment is realized. The steps of the update method are, for example, step S10 to step S40 shown in FIG. 2. Alternatively, the processor implements the functions of the modules/units of the data update apparatus in the foregoing embodiment when executing computer-readable instructions, for example, the functions of the modules 11 to 14 shown in FIG. 3. To avoid repetition, I won’t repeat them here.

In one embodiment, one or more readable storage media storing computer readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage. Medium; the readable storage medium stores computer readable instructions, and when the computer readable instructions are executed by one or more processors, the one or more processors implement the steps of the data update method in the above-mentioned embodiment, for example, as shown in FIG. Step S10 to Step S40 shown in 2. Alternatively, when the computer-readable instructions are executed by the processor, the functions of the modules/units of the data updating apparatus in the above-mentioned embodiment are realized, for example, the functions of the modules 11 to 14 shown in FIG. 3. To avoid repetition, I won’t repeat them here.

A person of ordinary skill in the art can understand that all or part of the processes in the methods of the foregoing embodiments can be implemented by instructing relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile computer. In a readable storage medium or a volatile readable storage medium, when the computer readable instruction is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the division of the above functional units and modules is used as an example. In practical applications, the above functions can be allocated to different functional units and modules as needed. Module completion, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, a person of ordinary skill in the art should understand that it can still implement the foregoing The technical solutions recorded in the examples are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be included in Within the scope of protection of this application.

Claims

A data update method, which includes:

When receiving the reading instruction, the back-end server obtains all unique data records of the configuration table in the front-end database of the front-end server, and the writing time associated with the unique data record; wherein, the unique data record refers to the The index data written in batches in the interface table of the database, and at the same time the unique identifier of the configuration is updated in the configuration table for the first index data in the batch of the index data, and the unique data The record is stored in the configuration table in association with the write time when the indicator data of the batch is written into the interface table; the indicator data of each batch includes at least one indicator data, and the interface In the table, the index data of each batch is divided by dividing symbols;

The back-end server parses the synchronization data time range in the reading instruction, and traverses queries that match the synchronization data time range among all the write times and the unique data records obtained from the configuration table The writing time and the unique data record associated with each writing time matching the synchronization data time range, and marking the unique data record associated with each writing time matching the synchronization data time range Identifies the data to be read;

The back-end server queries the interface table for index data corresponding to each of the data identifiers to be read, and determines, according to the segmentation symbol in the interface table, that the index data corresponding to the data identifier to be read belongs to Batches to be read, and record all index data of the batches to be read as new batch index data;

After the back-end server obtains the newly added batch index data of all batches from the interface table, it updates the newly-added batch index data of all the read batches to the back-end database stored in the back-end server. In the target data table, the unique data record corresponding to the first index data in the index data of all the batches to be read and the write time stored in association therewith are deleted from the configuration table at the same time.
The data update method according to claim 1, wherein the determining, according to the segmentation symbol in the interface table, the batch to be read to which the indicator data corresponding to the data identifier to be read belongs belongs to:

After recognizing the first unmarked segmentation symbol, mark the segmentation symbol as recognized;

Re-identify the first segmentation symbol that is located after the first segmentation symbol and is not marked, and use the index data between the two identified segmentation symbols as one batch of the index data, Until the two division symbols between the indicator data of each batch are marked and recognized;

Query and determine the index data with the first sorting order from the index data of each successfully identified batch, and verify whether the first index data is successfully associated with the data identification to be read, And after the association is successful, the batch to be read to which the first indicator data belongs is determined.
The data update method according to claim 1, wherein the updating and storing the new batch indicator data of all the batches read in the target data table of the back-end database of the back-end server comprises:

Determine the index type corresponding to the newly added batch of index data through the preset index type table;

According to the indicator type, the new batch indicator data corresponding to it is updated and stored in the storage partition corresponding to the indicator type in the target data table where the back-end database is located; one storage partition corresponds to one The indicator type.
The data update method according to claim 1, wherein before said updating the index data of the newly added batches of all the read batches in the target data table stored in the back-end database of the back-end server, further include:

Judging by index data verification whether the new batch index data of each batch in the interface table has a preset data abnormality;

If the preset data abnormality occurs in the newly-added batch index data of one of the batches, the newly-added batch index data of the batch is marked and a reminder is issued.
The data update method according to claim 1, wherein said acquiring new batch index data of all batches from the interface table comprises:

After obtaining the idle first processing thread from multiple threads in the order of thread priority, the batch number associated with the newly added batch index data of each batch is associated with each first processing thread, and issued The newly added batch indicator data of each batch is sent to each of the first processing threads associated with its batch number; the batch number is in accordance with the configuration corresponding to the newly added batch indicator data of each batch The sequence of the writing time in the table is generated;

Verifying whether each of the first processing threads corresponding to the newly added batch index data of each batch currently being processed has a preset process abnormality;

If the preset process exception occurs in one of the first processing threads corresponding to the newly added batch indicator data of a batch, the newly added batch indicators of the batch will be executed in the order of the thread priority. The batch number associated with the data is again associated with the idle second processing thread, and the batch numbers and the preset arrangement sequence associated with the new batch index data of each batch are compared to each batch. Arranging the index data of the newly added batches;

Obtain the newly added batch index data of each batch that has been successfully arranged from the interface table.
The data update method according to claim 1, wherein, after acquiring the new batch index data of all batches from the interface table, the method further comprises:

Judging whether there is a problem of interruption in obtaining the index data of the newly added batch in the interface table;

If the acquisition interruption problem occurs, the write time corresponding to the newly added batch index data of the batch is recorded in the exception log, and the new batch is acquired in the exception log. The work progress of the batch of index data is increased, and when writing is started next time, the newly added batch of index data of the batch is determined according to the writing time in the abnormal log, and according to the abnormal log The indicator data remaining in the batch of new batch indicator data determined by the work progress and the writing time.
A data update device, which includes:

The obtaining module is used to obtain all the unique data records of the configuration table in the front-end database of the front-end server and the writing time associated with the unique data record when the back-end server receives the reading instruction; wherein, the unique data record is Refers to the indicator data written in batches in the interface table of the front-end database, and at the same time update the configuration unique identifier for the first indicator data in the batch of the indicator data in the configuration table, And the unique data record is stored in the configuration table in association with the write time when the indicator data of the batch is written into the interface table; the indicator data of each batch includes at least one indicator Data, the index data of each batch is divided by a division symbol in the interface table;

The first marking module is used for the back-end server to parse the synchronization data time range in the reading instruction, to traverse the query and all the writing times and the unique data records obtained from the configuration table. The write time that matches the synchronization data time range, and the unique data record associated with each write time that matches the synchronization data time range, and each write time that matches the synchronization data time range The unique data record mark associated with time is the data identification to be read;

The recording module is used for the back-end server to query the index data corresponding to each of the data identifiers to be read in the interface table, and determine the data identifiers to be read according to the segmentation symbol in the interface table The batch to be read to which the corresponding index data belongs, and all the index data of the batch to be read are recorded as the new batch index data;

The storage module is configured to update and store the new batch index data of all read batches after the back-end server obtains the new batch index data of all batches from the interface table In the target data table of the back-end database, delete the unique data record corresponding to the first index data in all the index data of the batch to be read and all the data stored in association therewith in the configuration table. The writing time.
The data update device according to claim 7, wherein the recording module comprises:

The marking sub-module is used to mark the segmentation symbol as recognized after identifying the first segmentation symbol that has not been marked;

The recognition sub-module is used for re-identifying the first segmentation symbol that is located after the first segmentation symbol and is not marked, and uses the index data between the two identified segmentation symbols as a batch The index data of each batch until the two segmentation symbols between the index data of each batch are marked and recognized;

The first determining sub-module is configured to query and determine the index data whose sorting order is the first from the index data of each batch that has been successfully identified, and verify whether the first index data is consistent with the pending index data. The read data identifier is successfully associated, and after the successful association, the batch to be read to which the first indicator data belongs is determined.
A computer device includes a memory, a processor, and computer-readable instructions that are stored in the memory and can run on the processor, wherein the processor implements the following steps when the processor executes the computer-readable instructions:

When receiving the reading instruction, the back-end server obtains all unique data records of the configuration table in the front-end database of the front-end server, and the writing time associated with the unique data record; wherein, the unique data record refers to the The index data written in batches in the interface table of the database, and at the same time the unique identifier of the configuration is updated in the configuration table for the first index data in the batch of the index data, and the unique data The record is stored in the configuration table in association with the write time when the indicator data of the batch is written into the interface table; the indicator data of each batch includes at least one indicator data, and the interface In the table, the index data of each batch is divided by dividing symbols;

The back-end server parses the synchronization data time range in the reading instruction, and traverses queries that match the synchronization data time range among all the write times and the unique data records obtained from the configuration table The writing time and the unique data record associated with each writing time matching the synchronization data time range, and marking the unique data record associated with each writing time matching the synchronization data time range Identifies the data to be read;

The back-end server queries the interface table for index data corresponding to each of the data identifiers to be read, and determines, according to the segmentation symbol in the interface table, that the index data corresponding to the data identifier to be read belongs to Batches to be read, and record all index data of the batches to be read as new batch index data;

After the back-end server obtains the newly added batch index data of all batches from the interface table, it updates the newly-added batch index data of all the read batches to the back-end database stored in the back-end server. In the target data table, the unique data record corresponding to the first index data in the index data of all the batches to be read and the write time stored in association therewith are deleted from the configuration table at the same time.
9. The computer device according to claim 9, wherein the determining, according to the segmentation symbol in the interface table, the batch to be read to which the indicator data corresponding to the data identifier to be read belongs belongs to:

After recognizing the first unmarked segmentation symbol, mark the segmentation symbol as recognized;

Re-identify the first segmentation symbol that is located after the first segmentation symbol and is not marked, and use the index data between the two identified segmentation symbols as one batch of the index data, Until the two division symbols between the indicator data of each batch are marked and recognized;

Query and determine the index data with the first sorting order from the index data of each successfully identified batch, and verify whether the first index data is successfully associated with the data identification to be read, And after the association is successful, the batch to be read to which the first indicator data belongs is determined.
9. The computer device according to claim 9, wherein said updating and storing the new batch indicator data of all batches read in the target data table of the back-end database of the back-end server comprises:

Determine the index type corresponding to the newly added batch of index data through the preset index type table;

According to the indicator type, the new batch indicator data corresponding to it is updated and stored in the storage partition corresponding to the indicator type in the target data table where the back-end database is located; one storage partition corresponds to one The indicator type.
9. The computer device according to claim 9, wherein before the said new batch indicator data of all batches read is updated in the target data table stored in the back-end database of the back-end server, said The processor also implements the following steps when executing the computer-readable instructions:

Judging by index data verification whether the new batch index data of each batch in the interface table has a preset data abnormality;

If the preset data abnormality occurs in the newly-added batch index data of one of the batches, the newly-added batch index data of the batch is marked and a reminder is issued.
9. The computer device according to claim 9, wherein said acquiring new batch index data of all batches from the interface table comprises:

After obtaining the idle first processing thread from multiple threads in the order of thread priority, the batch number associated with the newly added batch index data of each batch is associated with each first processing thread, and issued The newly added batch indicator data of each batch is sent to each of the first processing threads associated with its batch number; the batch number is in accordance with the configuration corresponding to the newly added batch indicator data of each batch The sequence of the writing time in the table is generated;

Verifying whether each of the first processing threads corresponding to the newly added batch index data of each batch currently being processed has a preset process abnormality;

If the preset process exception occurs in one of the first processing threads corresponding to the newly added batch indicator data of a batch, the newly added batch indicators of the batch will be executed in the order of the thread priority. The batch number associated with the data is again associated with the idle second processing thread, and the batch numbers and the preset arrangement sequence associated with the new batch index data of each batch are compared to each batch. Arranging the index data of the newly added batches;

Obtain the newly added batch index data of each batch that has been successfully arranged from the interface table.
9. The computer device according to claim 9, wherein after the new batch indicator data of all batches is obtained from the interface table, the processor further implements the following steps when executing the computer readable instruction:

Judging whether there is a problem of interruption in obtaining the index data of the newly added batch in the interface table;

If the acquisition interruption problem occurs, the write time corresponding to the newly added batch index data of the batch is recorded in the exception log, and the new batch is acquired in the exception log. The work progress of the batch of index data is increased, and when writing is started next time, the newly added batch of index data of the batch is determined according to the writing time in the abnormal log, and according to the abnormal log The indicator data remaining in the batch of new batch indicator data determined by the work progress and the writing time.
One or more readable storage media storing computer readable instructions, where when the computer readable instructions are executed by one or more processors, the one or more processors execute the following steps:

When receiving the reading instruction, the back-end server obtains all unique data records of the configuration table in the front-end database of the front-end server, and the writing time associated with the unique data record; wherein, the unique data record refers to the The index data written in batches in the interface table of the database, and at the same time the unique identifier of the configuration is updated in the configuration table for the first index data in the batch of the index data, and the unique data The record is stored in the configuration table in association with the write time when the indicator data of the batch is written into the interface table; the indicator data of each batch includes at least one indicator data, and the interface In the table, the index data of each batch is divided by dividing symbols;

The back-end server parses the synchronization data time range in the reading instruction, and traverses queries that match the synchronization data time range among all the write times and the unique data records obtained from the configuration table The writing time and the unique data record associated with each writing time matching the synchronization data time range, and marking the unique data record associated with each writing time matching the synchronization data time range Identifies the data to be read;

The back-end server queries the interface table for index data corresponding to each of the data identifiers to be read, and determines, according to the segmentation symbol in the interface table, that the index data corresponding to the data identifier to be read belongs to Batches to be read, and record all index data of the batches to be read as new batch index data;

After the back-end server obtains the newly added batch index data of all batches from the interface table, it updates the newly-added batch index data of all the read batches to the back-end database stored in the back-end server. In the target data table, the unique data record corresponding to the first index data in the index data of all the batches to be read and the write time stored in association therewith are deleted from the configuration table at the same time.
15. The readable storage medium according to claim 15, wherein the determining, according to the segmentation symbol in the interface table, the batch to be read to which the indicator data corresponding to the data identifier to be read belongs belongs to:

After recognizing the first unmarked segmentation symbol, mark the segmentation symbol as recognized;

Re-identify the first segmentation symbol that is located after the first segmentation symbol and is not marked, and use the index data between the two identified segmentation symbols as one batch of the index data, Until the two division symbols between the indicator data of each batch are marked and recognized;

Query and determine the index data with the first sorting order from the index data of each successfully identified batch, and verify whether the first index data is successfully associated with the data identification to be read, And after the association is successful, the batch to be read to which the first indicator data belongs is determined.
The readable storage medium according to claim 15, wherein said updating and storing the new batch indicator data of all batches read in the target data table of the back-end database of the back-end server includes :

Determine the index type corresponding to the newly added batch of index data through the preset index type table;

According to the indicator type, the new batch indicator data corresponding to it is updated and stored in the storage partition corresponding to the indicator type in the target data table where the back-end database is located; one storage partition corresponds to one The indicator type.
15. The readable storage medium according to claim 15, wherein, before the new batch indicator data of all batches read is updated and stored in the target data table of the back-end database of the back-end server, When the computer-readable instructions are executed by one or more processors, the one or more processors further execute the following steps:

Judging by index data verification whether the new batch index data of each batch in the interface table has a preset data abnormality;

If the preset data abnormality occurs in the newly-added batch index data of one of the batches, the newly-added batch index data of the batch is marked and a reminder is issued.
15. The readable storage medium according to claim 15, wherein said acquiring new batch indicator data of all batches from the interface table comprises:

After obtaining the idle first processing thread from multiple threads in the order of thread priority, the batch number associated with the newly added batch index data of each batch is associated with each first processing thread, and issued The newly added batch indicator data of each batch is sent to each of the first processing threads associated with its batch number; the batch number is in accordance with the configuration corresponding to the newly added batch indicator data of each batch The sequence of the writing time in the table is generated;

Verifying whether each of the first processing threads corresponding to the newly added batch index data of each batch currently being processed has a preset process abnormality;

If the preset process exception occurs in one of the first processing threads corresponding to the newly added batch indicator data of a batch, the newly added batch indicators of the batch will be executed in the order of the thread priority. The batch number associated with the data is again associated with the idle second processing thread, and the batch numbers and the preset arrangement sequence associated with the new batch index data of each batch are compared to each batch. Arranging the index data of the newly added batches;

Obtain the newly added batch index data of each batch that has been successfully arranged from the interface table.
The readable storage medium according to claim 15, wherein after the new batch indicator data of all batches is obtained from the interface table, when the computer-readable instructions are executed by one or more processors, The one or more processors further execute the following steps:

Judging whether there is a problem of interruption in obtaining the index data of the newly added batch in the interface table;

If the acquisition interruption problem occurs, the write time corresponding to the newly added batch index data of the batch is recorded in the exception log, and the new batch is acquired in the exception log. The work progress of the batch of index data is increased, and when writing is started next time, the newly added batch of index data of the batch is determined according to the writing time in the abnormal log, and according to the abnormal log The indicator data remaining in the batch of new batch indicator data determined by the work progress and the writing time.