CN117573777A - Data synchronization method, device and electronic equipment - Google Patents

Abstract

The disclosure relates to a data synchronization method, a device and electronic equipment, and relates to the technical field of data synchronization processing, wherein the method comprises the following steps: acquiring preset configuration information at fixed time, wherein the preset configuration information comprises: periodically starting task time and presetting a processing rule; when the current time is equal to the period starting task time, acquiring initial data in each data source according to corresponding execution parameters by utilizing a plurality of data acquisition programs; and processing the initial data according to a preset processing rule by using a data processing program to obtain processed target data. In this way, the data acquisition from a plurality of data sources is supported simultaneously, and the synchronization time or the control data synchronization frequency can be flexibly set.

Description

Data synchronization method, device and electronic equipment

Technical field

The present application relates to the technical field of data synchronization processing, and in particular to a data synchronization method, device and electronic equipment.

Background technique

In current large-scale system or Internet project development, the system environment usually includes development environment, test environment and production environment. In order to reduce the number of repeated configurations and reduce the error rate of data configuration, basic data or configuration item data are usually configured in the development environment, the data after development is synchronously migrated to the test environment for testing, and finally the tested data is synchronously migrated to the production environment. . In addition, some configuration data in the system needs to be processed by obtaining third-party data and then stored in the database for use.

The implementation of the above-mentioned existing data synchronization methods or systems mainly relies on the change detection of the source database. Therefore, it cannot support multiple data sources to obtain data at the same time; when obtaining the changed data in the source database and sending a message to notify the target database, the target database Executing SQL scripts to update data has reached the consistency of data storage status in the source database and target database, making it impossible to flexibly set the synchronization time or control the frequency of data synchronization.

Contents of the invention

In view of this, this application provides a data synchronization method, device and electronic equipment. The main purpose is to solve the problem that the current existing technology cannot support multiple data sources to obtain data at the same time, and cannot flexibly set the synchronization time or control the data synchronization frequency. question.

According to a first aspect of the present disclosure, a data synchronization method is provided, which method includes:

Regularly obtain preset configuration information, which includes: periodic startup task time and preset processing rules;

When the current time is equal to the cycle startup task time, use multiple data collection programs to collect initial data from each data source according to the corresponding execution parameters;

Use a data processing program to process the initial data according to the preset processing rules to obtain processed target data.

According to a second aspect of the present disclosure, a data synchronization device is provided, which device includes:

The first acquisition module is used to regularly acquire preset configuration information, where the preset configuration information includes: periodic startup task time and preset processing rules;

The collection module is used to use multiple data collection programs to collect initial data from each data source according to the corresponding execution parameters when the current time is equal to the cycle startup task time;

A processing module, configured to use a data processing program to process the initial data according to the preset processing rules to obtain processed target data.

According to a third aspect of the present disclosure, an electronic device is provided, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores information that can be processed by the at least one processor. The instructions are executed by the at least one processor, so that the at least one processor can execute the method of the first aspect.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method of the first aspect.

Compared with the existing technology, the data synchronization method, device and electronic equipment provided by the present disclosure obtain preset configuration information through timing. The preset configuration information includes: periodic startup task time, preset processing rules; at the current time, it is equal to When the task is started in a cycle, multiple data collection programs are used to collect initial data from each data source according to the corresponding execution parameters; the data processing program is used to process the initial data according to preset processing rules to obtain the processed target data. In this way, it not only supports obtaining data from multiple data sources at the same time, but also can flexibly set the synchronization time or control the data synchronization frequency.

The above description is only an overview of the technical solutions of the present application. In order to have a clearer understanding of the technical means of the present application, they can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present application more obvious and understandable. , the specific implementation methods of the present application are specifically listed below.

Description of the drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.

In order to more clearly explain the embodiments of the present application or the technical applications in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those of ordinary skill in the art, It is said that other drawings can be obtained based on these drawings without exerting creative labor.

Figure 1 is a schematic flow chart of a data synchronization method provided by an embodiment of the present disclosure;

Figure 2 is a schematic flowchart of a data synchronization method provided by an embodiment of the present disclosure;

Figure 3 is an overall flow chart of a data synchronization method provided by an embodiment of the present disclosure;

Figure 4 is a periodic task diagram provided by an embodiment of the present disclosure;

FIG. 5 is a schematic structural diagram of a data synchronization device provided by an embodiment of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the present disclosure are included to facilitate understanding and should be considered to be exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness. It should be noted that, as long as there is no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

The following describes the data synchronization method, device and electronic equipment according to the embodiments of the present disclosure with reference to the accompanying drawings.

The present disclosure provides a data synchronization method, device and electronic equipment, which not only supports obtaining data from multiple data sources at the same time, but also can flexibly set the synchronization time or control the data synchronization frequency.

As shown in Figure 1, an embodiment of the present disclosure provides a data synchronization method, where the method may include:

Step 101: Obtain preset configuration information regularly. The preset configuration information includes: periodic startup task time and preset processing rules.

Among them, the preset configuration information can be a series of parameters and settings set for the task in the scheduled task platform. These parameters and settings determine the execution time, execution cycle, execution method, etc. of the task.

The preset configuration information may include the registration of the data collection program, data collection program execution parameters, preset processing rules for data processing, periodic startup task time, whether to retry if the execution fails, the maximum number of failed retries, and other parameter configurations.

For the embodiments of the present disclosure, the execution subject can be a data synchronization device or equipment. Taking the scheduled task platform as a data synchronization device or equipment as an example, the technical solutions in the present disclosure will be described, but this does not constitute a specific limitation to the technical solutions in the present disclosure. .

Among them, the scheduled task platform is mainly responsible for timing monitoring, interacting with the data collection program, providing visual configuration, task program registration, and full-process monitoring and display, allowing users to flexibly set synchronization time and control data synchronization frequency to meet different needs. Scenarios and needs.

Step 102: When the current time is equal to the cycle startup task time, use multiple data collection programs to collect initial data from each data source according to the corresponding execution parameters.

For the embodiments of the present disclosure, in the scheduled task platform, one or more data collection programs can be registered for each data synchronization task, and corresponding execution parameters can be configured so that the platform can identify and schedule the corresponding data collection tasks. These data collection programs are independent of each other and do not affect each other. They can process different types of data sources, such as relational databases, non-relational databases, or data obtained by calling third-party programs.

After all configurations are completed, once the periodic task start time set by the user is reached, the scheduled task platform will start the relevant data collection program to execute the data collection task according to the corresponding execution parameters, and collect initial data from each data source. Among them, the initial The data can be an initial version of the data set collected when the task is started or in a certain cycle.

The periodic start time of the task can be the time point at which the task starts execution at a specified time. For example, if a scheduled task is started on the hour every hour, then its periodic start time is the hour every hour to ensure the accuracy of the data synchronization time point. ;

Execution parameters are used to guide the data collection program on how to collect data, process data, and interact with other components when executing data synchronization tasks. They can include the collection range, frequency of data collection, address of the data source, conditions for the data that needs to be collected, etc. To meet the collection needs of different data sources;

The data collection program is used to read the execution parameters stored in the database. Based on the obtained execution parameters, the data collection program will run the relevant program code to obtain qualified data from the data source.

Step 103: Use a data processing program to process the initial data according to preset processing rules to obtain processed target data.

Among them, the preset processing rules can be the execution rules of a preset data processing program, and the guidelines used to specify how to clean, transform, integrate, etc. the data can include data cleaning, transformation, merging and other operations, so as to collect the collected data. Data is preprocessed to ensure the consistency, accuracy and completeness of the data, thereby ensuring that the data has a high value during storage and analysis;

Data processing programs can be computer programs used to clean, transform, integrate and analyze raw data to meet specific needs or goals;

The target data can be data processed by a data processing program, and its quality, format, and content meet the needs of subsequent analysis, modeling, and application.

In summary, compared with the existing technology, the data synchronization method provided by the present disclosure obtains preset configuration information through timing. The preset configuration information includes: periodic startup task time and preset processing rules; when the current time is equal to periodic startup During task time, multiple data collection programs are used to collect initial data from each data source according to the corresponding execution parameters; the data processing program is used to process the initial data according to preset processing rules to obtain the processed target data. In this way, it not only supports obtaining data from multiple data sources at the same time, but also can flexibly set the synchronization time or control the data synchronization frequency.

Further, as a refinement and expansion of the above embodiments, in order to fully explain the specific implementation process of the disclosed method, the present disclosure provides a specific method as shown in Figure 2, which method includes:

Step 201: Obtain preset configuration information regularly. The preset configuration information includes: periodic startup task time and preset processing rules.

For the embodiments of the present disclosure, existing problems are mainly solved by introducing a scheduled task platform, message queue middleware and database;

As shown in Figure 3, the four parts of the data collection program, data processing program, data storage program and scheduled task platform constitute the entire data synchronization system. This division of labor and cooperation model enables the system to support data acquisition from multiple data sources at the same time, improving the flexibility and compatibility of data synchronization.

Introducing message queue middleware for asynchronous communication between various programs to solve the coupling problem between data processing, data collection, and data storage. This method makes the data processing process independent of the data collection and storage process, improving the system's concurrent processing capabilities and performance.

A database is introduced to store information such as data collection program execution parameters, preset processing rules, and program execution results to facilitate unified management and monitoring.

The data acquisition program is responsible for collecting data from various data sources, storing the data in the cache, and notifying the data processing program; the data processing program processes the data in the cache according to preset processing rules, and sends the processed data to the data storage program for storage. This mode not only ensures real-time data synchronization, but also reduces system resource consumption when data changes are small.

Step 202: When the current time is equal to the cycle startup task time, use multiple data collection programs to collect initial data from each data source according to the corresponding execution parameters.

Among them, programs such as data collection programs can run in the scheduled task platform or on other servers or devices to communicate and collaborate with the scheduled task platform. The scheduled task platform is responsible for scheduling and monitoring the execution of these programs to ensure the smooth progress of data synchronization tasks. At the same time, the operating environment and data processing capabilities of the data collection program can be selected and configured according to actual needs to meet the data synchronization needs in various scenarios.

For the embodiments of the present disclosure, under the scheduling of the scheduled task platform, the data collection program can execute the data collection task according to the parameters and rules set by the user, where different data collection programs can collect different types of data. The data source can be data in a relational database, such as MySQL, Oracle, etc.; it can also be data in a non-relational database, such as MongoDB, Redis, etc.; it can also be data obtained by calling a third-party program.

Each data collection program requires corresponding data source configuration, which can include the address of the data source, access method, data type and other information; the data collection program can collect full data or incremental data through execution parameter configuration, and full data can be all data , and incremental data can be newly added data since the last collection;

The data collection program can also set collection conditions through execution parameters to collect data that meets specific needs. For example, you can set up the collection of data in a specific field, data in a certain time period, etc.

Step 203: Calculate the data volume of the initial data; if the data volume is greater than 0, store the initial data in the cache system to obtain the cached data; use the message middleware to transfer the cached data to the data processing program in the form of a message.

For the embodiment of the present disclosure, if the data collection program does not obtain any data (that is, when the amount of initial data is equal to 0), the flow of the program ends.

If the data is obtained by the data acquisition program, the data can be stored in the cache system. The cache system is used to temporarily store data for subsequent data processing; after caching the data, the data collection program will send a message, and the message middleware is responsible for receiving the message and sending the message to the data processing program to wake up the data processing program to start The new task processes cached data. The data processing program processes the data in the cache based on the data information contained in the message. At the same time, the data collection program will also store relevant information about the collection program execution into the database, such as data collection duration, data collection time, etc. total amount etc.

Among them, the message can contain information related to cached data, such as data source, data type, data volume, etc.; Message Queue (MQ) is a technology used to handle message delivery and communication in distributed systems. An asynchronous communication mechanism that decouples applications by sending messages to a queue.

Step 204: Determine the number of threads in the thread pool; when the data processing program receives the message, it uses the threads in the thread pool to perform data processing on the cached data according to the preset processing rules through the first preset processing algorithm to obtain the processed target data.

Among them, the thread pool can be a tool used to allocate, schedule and manage threads. In the data processing program, the thread pool can allocate tasks among multiple threads, so that the data processing program can process multiple data tasks at the same time. , instead of waiting for one task to complete before starting the next task, it can make full use of the computer's multi-core processor, helping to shorten the total time of data processing and increase processing speed.

For the embodiment of the present disclosure, when the data processing program receives the message sent by the message middleware, it wakes up, reads the preset processing rules from the database, and uses the predefined algorithm (i.e., the first preset processing algorithm) to The original data in the cache (that is, the cache data) is cleaned and processed to obtain the processed target data (that is, the target database storage format data). Among them, the first preset processing algorithm can be an algorithm set in the data processing program, used to convert or process data; the first preset processing algorithm can be stored in the data processing program, and can be used to implement a simple The data model can be converted into another data model, or the original data model can be converted into another data model through complex algorithm operations; the target data can be data that meets the storage requirements of the target database after cleaning and processing.

Preset processing rules can be used to identify and remove outliers, erroneous values, and duplicate values in cached data to improve data quality; they can also be used to convert between different data formats to make the data meet the requirements of the target database or analysis tool ; can also be used to guide the merging, aggregation and correlation of data from different data sources for further analysis.

In the embodiment of the present disclosure, the preset processing rule can be an operation guide, describing the process of how to process cached data; and the first preset processing algorithm can be a specific implementation method, which can be processed according to the preset Rules clean, transform and integrate cached data. The preset processing rules are used to provide guidance for the first preset processing algorithm, and the first preset processing algorithm can perform specific operations according to the preset processing rules. In actual application scenarios, the preset processing rules and the first preset processing algorithm can jointly ensure the quality and accuracy of cached data during processing.

For the embodiments of the present disclosure, determining a reasonably sized thread pool (that is, determining the number of threads in the thread pool) can be the optimal number of threads configured according to factors such as system resources, task load, and performance requirements in a specific application scenario. A large and small thread pool can avoid problems caused by too many or too few threads while ensuring full utilization of system resources.

Specifically, the data processing algorithm also includes an important algorithm for generating a thread pool of reasonable size. The algorithm formula (second preset processing algorithm) is as follows:

In the formula, N _threads represents the thread pool, N _cpu represents the number of processor cores, U _cpu represents the processor resource utilization, W represents the program waiting time, and C represents the program processing time;

Among them, the higher the number of cores of the processor (which can be a CPU), the higher the computing power; when the processor resource usage is high, the number of threads may need to be increased to increase the data processing speed; and when the processor resource usage is high When it is low, the number of threads can be appropriately reduced to reduce the degree of contention for system resources; the program waiting time (can be the length of time the program spends waiting (such as waiting for IO operation results)), and the thread pool algorithm needs to consider the time the program is waiting (such as IO operations). result), the longer the waiting time, the greater the program's overhead in waiting for IO operations, and the number of threads may need to be increased to increase the data processing speed; the program processing time (that is, the time the program actually takes up the processor for calculations), the thread The pool algorithm needs to consider the time that the program actually takes up the processor for calculation. The longer the calculation time, the greater the computational overhead of the program. The number of threads may need to be increased to increase the data processing speed.

You can use the JVisualVM tool to collect monitoring data of the program and calculate the ratio between the waiting time of the program and the processing time of the program. This can help evaluate the relationship between the program's overhead in waiting for IO operations and the actual computing overhead, thereby adjusting the size of the thread pool.

Through this algorithm, the data processing program generates a reasonably sized thread pool that can process data in batches to improve data processing capabilities and speed.

Accordingly, the specific process of determining the number of threads in the thread pool may include: obtaining processor data and program data of the data storage program. The processor data includes the number of processor cores and processor resource usage, and the program data includes the program waiting time. and program processing time;

The number of processor cores, processor resource utilization, program waiting time, and program processing time are substituted into the second preset processing algorithm to determine the number of threads in the thread pool.

Step 205: Send the processed target data to the data storage program for data storage. The data storage program is used to receive the target data and update or insert the target data to implement data storage.

Among them, data storage can be the process of saving target data in a computer system, including data storage, management and maintenance, etc.; data storage program can be a program responsible for storing target data in a database or other storage system.

The data processing program will send the processed target data to the data storage program for storage. The data storage program is responsible for receiving the target data processed by the data processing program and updating or inserting it into the corresponding data storage structure to achieve data efficiency. Storage and management.

After each thread task completes the storage operation, the execution time can be stored in the database, which helps record the task execution and facilitates subsequent analysis, monitoring, and optimization of system performance.

Step 206: Obtain the preset task execution cycle configured in the data collection program; when it is judged that the preset task execution cycle is unreasonable, provide an early warning reminder for the preset task execution cycle, and optimize the preset task execution cycle.

Among them, the preset task execution cycle can be the time interval at which the task is executed at a specified time, that is, the time period from the end of the last execution of the task to the beginning of the next execution; for example, if a scheduled task is executed once every minute, then it The cycle execution time is 1 minute. In practical applications, the preset task execution cycle can be adjusted according to needs to achieve more flexible task scheduling.

For the embodiments of the present disclosure, since the data sources collected by each data synchronization task are different, the data collection and processing rules are different, and the total amount of data collected each time is also different, the data synchronization task can be analyzed and calculated, Continuously adjust and optimize the preset task execution cycle to achieve the best performance for each data synchronization task.

For embodiments of the present disclosure, the method for determining that the preset task execution cycle is unreasonable may include obtaining the number of tasks, the number of executions, and the execution time within the preset task execution cycle;

Calculate the super period of the preset task execution cycle corresponding to all data collection programs, and calculate the deadline corresponding to the data collection program based on the number of executions and execution time;

Substitute the preset task execution period and super-period into the task number calculation formula to obtain the maximum number of tasks within the super-period, and substitute the preset task execution period and execution time into the load calculation formula to obtain the system load;

If at least one of the following: the number of tasks is greater than the maximum number of tasks, the system load is greater than 1, and the execution time is greater than the deadline, it is determined that the task execution cycle is unreasonable.

In the specific application scenario, it is assumed that the scheduled task platform has registered N data collection programs T1, T2,..., Tn, and the preset task execution period configured for the Ti-th data collection program is Pi, then the j-th data collection program The running time is (j-1)Pi (j=1, 2, 3,..., n"), the relative running deadline is D _ij = jP _i , and the least common multiple of the execution cycles of all data collection programs is the super period of the task set Denoted as SLCM. For the data collection program, the situation in each super period is the same, so it only needs to be studied in the range of [0, SLCM]. The maximum number of tasks in the period is:

Among them, the super period can be the least common multiple of the preset task execution cycles of all data acquisition programs; the least common multiple can be the smallest common multiple of two or more integers.

In a specific application scenario, it is assumed that n collection tasks Ti of n data collection programs are executed, with execution time Ci and period Pi. Without considering other auxiliary overhead of the system, the collection program for the deadline priority algorithm can be executed by the processor if it satisfies the following formula.

Among them, the execution time can be the time required to complete a specific task (in this case, the data collection program). Specifically, it can be the time elapsed from the start of task execution to the completion of the task, which can be expressed as: execution time = actual time to complete the task/number of task executions; ρ represents the system load. If the calculated system load is 1.275>1 , that is, unschedulable (overloaded), the system cannot guarantee that all tasks will be completed before the deadline.

In the specific application scenario, t is the current time of the system, Cr is the estimated execution time of the data collection program Ti, and Dij is the deadline for the jth run of the data collection program Ti. If d=D _ij -(t+C _r )≥0, it means that the deadline for running the collection program is currently achievable. Therefore, during scheduling, the scheduled ready task d is calculated. If d is greater than 0, it will be executed, otherwise it will not be executed.

Among them, the Earliest Deadline First (EDF) algorithm can be used in real-time systems to schedule tasks with deadlines. In this algorithm, the priority of a task is dynamically assigned according to its deadline. The earlier the deadline, the earlier the deadline. , the higher the priority. Without considering other auxiliary overhead of the system, if a collection program meets the following conditions, it may be executed by the processor:

The execution time of the collection program does not exceed its deadline, and the collection program completes execution before the deadline, thereby meeting the real-time requirement.

When processing real-time tasks, the deadline priority algorithm will give priority to tasks with approaching deadlines to ensure that they can be completed within the specified time. Such a scheduling strategy can improve the real-time performance of the system and ensure that tasks are executed in the expected time sequence.

In specific application scenarios, for example, the scheduled task platform has registered three data collection programs A, B, and C. The configured task criticality Ki is medium, low, and high respectively, and the configured execution periods Pi are 20ms, 40ms, and 50ms respectively. , according to the executed parameters stored in the database, it is found that the longest duration Ci in the previous executions of data collection programs A, B, and C is 10ms, 15ms, and 20ms respectively, as shown in Figure 4.

For the embodiments of the present disclosure, it can be seen from the above that according to the number of data collection programs registered on the scheduled task platform, the preset task execution cycle and task criticality configured for each data collection program, and the execution of each data collection program stored in the database The duration of the collection task can be used to calculate the maximum number of tasks within the system cycle, whether the program can be executed by the processor, and whether a certain collection task can be executed. If the number of tasks is greater than the maximum number of tasks, the system If at least one of the load is greater than 1 and the execution time is greater than the deadline, it is determined that the configured preset task execution cycle is unreasonable, and the system will send a reminder to modify the task cycle execution time. After receiving the reminder, the task cycle execution time can be modified, thus Achieve the effect of continuously optimizing the execution time of the scheduled task cycle.

For the embodiments of this disclosure, this application proposes the concept of supporting multiple data sources, which is essentially a set of different data collection, processing, and storage programs to support diversified collection requirements. The data source can be a relational database. Data, data in cache database or data obtained by requesting an interface, etc.

This application integrates data synchronization with scheduled tasks, and centrally manages the data synchronization program through the scheduled task management platform. The platform provides visual configuration, which can configure the execution parameters of the collection program, the processing rules of the data processing program, and the execution cycle of the program.

In this application, data processing algorithms and thread pool size generation algorithms are added to the data processing steps, which can process data with different needs and different capacities.

In this application, the timing task execution time of the synchronized data can be calculated based on the algorithm whether it is reasonable. If it is not reasonable, a modification reminder will be sent, and then modified to achieve the optimization effect.

This application optimizes the traditional data synchronization system. A single system can only support a single data source collection requirement. It introduces a scheduled task management platform. The management platform can register different data source collection requirement programs, which can well solve the problem of a single data source and Different collection programs can be managed uniformly, and the timing task execution time of synchronized data can be determined based on the algorithm to determine whether it is reasonable.

In summary, compared with the existing technology, the data synchronization method provided by the present disclosure obtains preset configuration information through timing. The preset configuration information includes: periodic startup task time and preset processing rules; when the current time is equal to periodic startup During task time, multiple data collection programs are used to collect initial data from each data source according to the corresponding execution parameters; the data processing program is used to process the initial data according to preset processing rules to obtain the processed target data. In this way, it not only supports obtaining data from multiple data sources at the same time, but also can flexibly set the synchronization time or control the data synchronization frequency.

Based on the specific implementation of the methods shown in Figures 1 and 2, this embodiment provides a data synchronization device, as shown in Figure 5. The device includes: a first acquisition module 31, a collection module 32, and a processing module 33;

The first acquisition module 31 is used to regularly acquire preset configuration information, where the preset configuration information includes: periodic startup task time and preset processing rules;

The collection module 32 is configured to use multiple data collection programs to collect initial data from each data source according to the corresponding execution parameters when the current time is equal to the cycle startup task time;

The processing module 33 is configured to use a data processing program to process the initial data according to the preset processing rules to obtain processed target data.

In a specific application scenario, as shown in 5, the device also includes: a calculation module 34, a storage module 35, a transfer module 36, and a termination module 37;

Calculation module 34, used to calculate the data volume of the initial data;

The storage module 35 is used to store the initial data in the cache system to obtain cache data when the data amount is greater than 0;

The transfer module 36 is used to transfer the cached data to the data processing program in the form of a message using message middleware;

The termination module 37 is configured to terminate the processing flow when the data amount of the initial data is equal to 0.

In a specific application scenario, the processing module 33 can be used to determine the number of threads in the thread pool; when the data processing program receives the message, use the threads in the thread pool to use the first preset processing algorithm according to The preset processing rules perform data processing on the cached data to obtain the processed target data.

In a specific application scenario, the processing module 33 can be used to obtain processor data and program data of the data storage program. The processor data includes the number of processor cores and processor resource usage. The program data includes Program waiting time and program processing time;

The number of processor cores, the processor resource utilization rate, the program waiting time and the program processing time are substituted into the second preset processing algorithm to determine the number of threads in the thread pool.

In a specific application scenario, as shown in 5, the device also includes: a storage module 38, a second acquisition module 39, and a reminder module 40;

The storage module 38 is used to send the processed target data to a data storage program for data storage. The data storage program is used to receive the target data and update or insert the target data to realize data storage. storage;

The second acquisition module 39 is used to acquire the preset task execution cycle configured in the data collection program;

The reminder module 40 is configured to provide an early warning reminder for the preset task execution cycle and optimize the preset task execution cycle when it is determined that the preset task execution cycle is unreasonable.

In a specific application scenario, the reminder module 40 can be used to obtain the number of tasks, the number of executions, and the execution time within the preset task execution cycle;

Calculate the super period of the preset task execution cycles corresponding to all the data collection programs, and calculate the deadline corresponding to the data collection program based on the number of executions and the execution time;

Substitute the preset task execution period and the super period into the task number calculation formula to obtain the maximum number of tasks within the super period, and substitute the preset task execution period and the execution time into the load calculation formula to obtain system load;

If at least one of the following: the number of tasks is greater than the maximum number of tasks, the system load is greater than 1, and the execution time is greater than the deadline, it is determined that the task execution cycle is unreasonable.

It should be noted that for other corresponding descriptions of the functional units involved in the frame extraction task scheduling device applicable to the edge node side provided in this embodiment, please refer to the corresponding description of the method in Figure 1 and will not be described again here.

Based on the above-mentioned methods shown in Figures 1 and 2, correspondingly, the present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned methods shown in Figures 1 and 2 are implemented. The method shown in 2.

Based on this understanding, the technical solution of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.), including several Instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method of each implementation scenario of the present disclosure.

Based on the above methods shown in Figures 1 and 2, and the virtual device embodiment shown in Figure 5, in order to achieve the above purpose, embodiments of the present disclosure also provide an electronic device that can be configured in a vehicle (such as an electric vehicle) On the end side, the device includes a storage medium and a processor; the storage medium is used to store a computer program; and the processor is used to execute the computer program to implement the above-mentioned methods shown in Figures 1 and 2.

Optionally, the above-mentioned physical devices may also include user interfaces, network interfaces, cameras, radio frequency (Radio Frequency, RF) circuits, sensors, audio circuits, WI-FI modules, etc. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), etc. The optional user interface may also include a USB interface, a card reader interface, etc. Optional network interfaces may include standard wired interfaces, wireless interfaces (such as WI-FI interfaces), etc.

Those skilled in the art can understand that the above-mentioned physical device structure provided by the present disclosure does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or arrange different components.

The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical devices and supports the operation of information processing programs and other software and/or programs. The network communication module is used to realize communication between components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

Through the above description of the embodiments, those skilled in the art can clearly understand that the present disclosure can be implemented by means of software plus a necessary general hardware platform, or can also be implemented by hardware. Compared with the existing technology, the data synchronization method, device and electronic equipment provided by the present disclosure obtain preset configuration information through timing. The preset configuration information includes: periodic startup task time, preset processing rules; at the current time, it is equal to When the task is started in a cycle, multiple data collection programs are used to collect initial data from each data source according to the corresponding execution parameters; the data processing program is used to process the initial data according to preset processing rules to obtain the processed target data. In this way, it not only supports obtaining data from multiple data sources at the same time, but also can flexibly set the synchronization time or control the data synchronization frequency.

It should be noted that in this article, relational terms such as “first” and “second” are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these There is no such actual relationship or sequence between entities or operations. Furthermore, the term "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus including a list of elements includes not only those elements but also other elements not expressly listed, Or it also includes elements inherent to the process, method, article or equipment. Without further limitation, an element qualified by the statement "comprises a..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be practiced in other embodiments without departing from the spirit or scope of the disclosure. Therefore, the present disclosure is not to be limited to the embodiments herein but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims (10)

1. A method of data synchronization, the method comprising:

acquiring preset configuration information at fixed time, wherein the preset configuration information comprises: periodically starting task time and presetting a processing rule;

when the current time is equal to the period starting task time, acquiring initial data in each data source according to corresponding execution parameters by utilizing a plurality of data acquisition programs;

and processing the initial data according to the preset processing rule by using a data processing program to obtain processed target data.

2. The method of claim 1, wherein after said acquiring initial data in each data source in accordance with the corresponding execution parameters using a plurality of data acquisition programs, the method further comprises:

calculating the data quantity of the initial data;

storing the initial data into a cache system to obtain cache data under the condition that the data volume is larger than 0;

and using message middleware to transfer the cache data to the data processing program in the form of a message.

3. The method according to claim 2, wherein the method further comprises:

in the case where the data amount of the initial data is equal to 0, the process flow is terminated.

4. The method according to claim 2, wherein the processing the initial data by the data processing program according to the preset processing rule to obtain the processed target data includes:

determining the number of threads in a thread pool;

when the data processing program receives the message, the data processing program processes the cached data according to the preset processing rule by utilizing threads in the thread pool through a first preset processing algorithm, and the processed target data is obtained.

5. The method of claim 4, wherein determining the number of threads in the thread pool comprises:

acquiring processor data and program data of the data storage program, wherein the processor data comprises a processor core number and a processor resource utilization rate, and the program data comprises a program waiting time and a program processing time;

substituting the processor core number, the processor resource utilization rate, the program waiting time and the program processing time into a second preset processing algorithm to determine the number of threads in the thread pool.

6. The method of claim 4, wherein after said data processing is performed on said buffered data to obtain said target data after processing, said method further comprises:

and sending the processed target data to a data storage program for data storage, wherein the data storage program is used for receiving the target data and carrying out updating or inserting operation on the target data to realize data storage.

7. The method according to claim 1, wherein the method further comprises:

acquiring a preset task execution period configured in the data acquisition program;

and under the condition that the preset task execution period is unreasonable, carrying out early warning reminding on the preset task execution period, and optimizing the preset task execution period.

8. The method of claim 7, wherein the determining that the preset task execution period is unreasonable comprises:

acquiring the task number, the execution times and the execution time in the preset task execution period;

calculating the overcycles of the execution cycles of the preset tasks corresponding to all the data acquisition programs, and calculating the deadlines corresponding to the data acquisition programs according to the execution times and the execution time;

substituting the preset task execution period and the overcycle into a task number calculation formula to obtain the maximum task number in the overcycle, and substituting the preset task execution period and the execution time into a load calculation formula to obtain a system load;

and if at least one of the task number is greater than the maximum task number, the system load is greater than 1, and the execution time is greater than a deadline is met, determining that the task setting execution period is unreasonable.

9. A data synchronization device, the device comprising:

the first acquisition module is used for acquiring preset configuration information at fixed time, and the preset configuration information comprises: periodically starting task time and presetting a processing rule;

the acquisition module is used for acquiring initial data in each data source according to corresponding execution parameters by utilizing a plurality of data acquisition programs when the current time is equal to the period starting task time;

and the processing module is used for processing the initial data according to the preset processing rule by utilizing a data processing program to obtain processed target data.

10. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.