CN111367925A - Data dynamic real-time updating method, device and storage medium - Google Patents
Data dynamic real-time updating method, device and storage medium Download PDFInfo
- Publication number
- CN111367925A CN111367925A CN202010122961.5A CN202010122961A CN111367925A CN 111367925 A CN111367925 A CN 111367925A CN 202010122961 A CN202010122961 A CN 202010122961A CN 111367925 A CN111367925 A CN 111367925A
- Authority
- CN
- China
- Prior art keywords
- data information
- data
- information
- level cache
- rtm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of data processing, and provides a method, a device and a storage medium for dynamically updating data in real time, wherein the method comprises the following steps: s1, receiving data information through a STORM topological structure; s2, the transmitting node of the STORM topological structure sends the received data information to the processing node of the STORM topological structure and carries out data conversion; s3, storing the data information after data conversion into an RTM (resin transfer molding) tree storage engine and storing the data information into a first-level cache; s4, putting the data information in the first-level cache into a task queue according to the type information, and sequentially storing the data information into a second-level cache; and S5, analyzing the data information stored in the secondary cache, storing the analyzed data information into a warehousing preparation data file corresponding to the database, and recording the data information in the warehousing preparation data file into the corresponding database. The data information is stored in the first-level cache and is directly called from the first-level cache, so that the data updating efficiency is greatly improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for dynamically updating data in real time, and a storage medium.
Background
When updated information data appear in the operation of the distributed server, the information data are transmitted to the database, and then the server reads the data information through the cache, the memory and the database in sequence. The storage and the calling of the data information both take more time, so that the load of the server is increased; and when the real-time requirement on the data information is high, the acquired data is the data information which is in the cache or in the memory, and the problem of high error rate exists.
In the prior art, data updating is performed by adopting a timing updating mode, and a preset planning task is automatically called after a set interval period is reached by configuring the interval time of a timer, for example, once every 5 minutes or 10 minutes; the timer triggers the timing task engine once in fixed time, and then executes the data updating script. When the execution time of a certain function task is longer, the system automatically executes the function task at intervals by adopting a timer. The method realizes the purpose of data updating by setting the timer; when the task interface is really triggered, the result can be read only without executing complex logic judgment. However, the above method has the following disadvantages: the real state of the current data cannot be fed back in time due to low timeliness.
Another kind of data updating is realized by adopting a mode of information notification, when one database in the distributed servers has data updating, other servers are informed of data synchronization by a mode of sending notification; and once the data of the database changes, each distributed server is informed to update the memory data, and each distributed server acquires the message for updating. The real-time performance of the method is enhanced, but in the face of some scenes with high real-time requirements on time, the risk of data updating delay still exists.
Therefore, a method for dynamically updating data in real time with high timeliness is needed.
Disclosure of Invention
The invention provides a data dynamic real-time updating method, an electronic device and a computer readable storage medium, which mainly store data information into a primary cache and directly call the data information from the primary cache when a server runs, so that the time delay caused by data warehousing and data updating is reduced, and the data updating time is greatly shortened.
In order to achieve the above object, the present invention further provides a method for dynamically updating data in real time, which is applied to an electronic device, and the method includes:
s1, receiving data information through a preset STORM topological structure;
s2, the transmitting node of the STORM topological structure sends the received data information to the processing node of the STORM topological structure, and data conversion is carried out on the data information through the processing node; wherein the processing node is associated with a RTM tree storage engine;
s3, storing the data information after data conversion into an RTM (resin transfer molding) tree storage engine, and storing the data information stored into the RTM tree storage engine into a primary cache after the information quantity of the data information in the RTM tree storage engine reaches a preset information quantity threshold;
s4, putting the data information in the first-level cache into a task queue according to type information, and sequentially storing the data information stored in the first-level cache into a second-level cache through the task queue;
and S5, analyzing the data information stored in the second-level cache, storing the analyzed data information into a warehousing preparation data file corresponding to the database, inputting the data information in the warehousing preparation data file into the corresponding database, and updating the number of the warehousing second-level cache data files in the task queue. Data information by type information
Further, preferably, after the data information in the RTM tree storage engine reaches the threshold, the method for storing the data information in the primary cache includes:
s110, after receiving the data information after the data conversion, the RTM storage engine calculates the information quantity of the existing data information in the RTM storage engine;
s120, judging whether the information quantity is larger than a preset information quantity threshold value or not, and if so, triggering an RTM (resin transfer molding) tree storage engine to merge the stored data information; if not, repeating the step of S110;
and S130, storing the data information after the merging operation into a first-level cache.
Further, preferably, after receiving the data information after the data conversion, the RTM tree storage engine further includes a type determining process for the data information, where the type determining process includes:
judging whether the type of the received data information is consistent with a set type, if not, discarding the data information, and continuing to receive new data information;
and if the data information is consistent with the threshold value, deleting the original data information of the same type in the first-level cache, and continuously judging the threshold value of the information quantity threshold information quantity of the received data information.
Further, preferably, before the second-level cache receives the data information in the first-level cache, a determination process of a timestamp of the data information in the first-level cache is further included, where the determination process of the timestamp includes:
reading a timestamp of the data information stored in the first-level cache, and calculating the time difference between the timestamp and the existing time;
judging whether the time difference is greater than a preset time threshold, if so, judging that the data information in the primary cache is out of date, and waiting for new data information;
if the data information is less than the preset threshold value, the data information is called when the data information is judged not to be over.
Further, in step S5, after updating the number of the data files that have been put into the second-level cache, the method further includes:
and generating a new secondary cache data file to be written, and waiting for the next writing of the secondary cache data file.
In order to achieve the above object, the present invention further provides a data dynamic real-time updating system, which includes a data information receiving unit, a data information converting unit, a first-level cache storing unit, a second-level cache storing unit and a second-level cache updating unit; wherein the content of the first and second substances,
the data information receiving unit is used for receiving data information through a preset STORM topological structure;
the data information conversion unit is used for sending the data information to the processing node of the STORM topological structure after receiving the data information through the transmitting node of the STORM topological structure, and performing data conversion on the data information through the processing node; wherein the processing node is associated with a RTM tree storage engine;
the first-level cache storing unit is used for storing the data information after data conversion into the RTM storage engine, and storing the data information stored into the RTM storage engine into a first-level cache after the information quantity of the data information in the RTM storage engine reaches a preset information quantity threshold;
the second-level cache storing unit is used for putting the data information in the first-level cache into a task queue according to type information and sequentially storing the data information stored in the first-level cache into a second-level cache through the task queue;
and the second-level cache updating unit is used for analyzing the data information stored in the second-level cache, storing the analyzed data information into the warehousing preparation data file corresponding to the database, inputting the data information in the warehousing preparation data file into the corresponding database, and updating the number of the warehoused second-level cache data files in the task queue. The data information is according to type information.
Further, preferably, the first-level cache storing unit comprises a data information storing module, a data information amount judging module, a data information merging module and a first-level cache storing module; wherein the content of the first and second substances,
the data information storage module is used for calculating the information quantity of the existing data information in the RTM storage engine after the RTM storage engine receives the data information after data conversion;
the information quantity judging module of the data information is used for judging whether the information quantity of the data information storage module is larger than a preset information quantity threshold value or not, and if so, the data information merging module is started;
the data information merging module is used for merging the data information of which the information quantity stored in the data information storage module is larger than a preset information quantity threshold value;
and the first-level cache storing module is used for storing the data information subjected to merging operation by the data information merging module into the first-level cache.
To achieve the above object, the present invention also provides an electronic device, including: the data dynamic real-time updating program is stored in the memory and is executed by the processor to realize the following steps:
s1, receiving data information through a preset STORM topological structure;
s2, the transmitting node of the STORM topological structure sends the received data information to the processing node of the STORM topological structure, and data conversion is carried out on the data information through the processing node; wherein the processing node is associated with a RTM tree storage engine;
s3, storing the data information after data conversion into an RTM (resin transfer molding) tree storage engine, and storing the data information stored into the RTM tree storage engine into a primary cache after the information quantity of the data information in the RTM tree storage engine reaches a preset information quantity threshold;
s4, putting the data information in the first-level cache into a task queue according to type information, and sequentially storing the data information stored in the first-level cache into a second-level cache through the task queue;
and S5, analyzing the data information stored in the second-level cache, storing the analyzed data information into a warehousing preparation data file corresponding to the database, inputting the data information in the warehousing preparation data file into the corresponding database, and updating the number of the warehousing second-level cache data files in the task queue. Data information by type information
Further, preferably, in step S3, after the data information in the RTM tree storage engine reaches the threshold, the method for storing the data information in the first-level cache includes:
s110, after receiving the data information after the data conversion, the RTM storage engine calculates the information quantity of the existing data information in the RTM storage engine;
s120, judging whether the information quantity is larger than a preset information quantity threshold value or not, and if so, triggering an RTM (resin transfer molding) tree storage engine to merge the stored data information; if not, repeating the step of S110;
and S130, storing the data information after the merging operation into a first-level cache.
In addition, to achieve the above object, the present invention further provides a computer readable storage medium, in which a computer program is stored, the computer program including a data dynamic real-time updating program, and when the data dynamic real-time updating program is executed by a processor, the steps of the data dynamic real-time updating method are implemented.
The data dynamic real-time updating method, the electronic device and the computer readable storage medium provided by the invention have the advantages that the data information is stored into the primary cache, and the data information is directly called from the primary cache when the server runs; the beneficial effects are as follows:
(1) updating the cache in time by using a message queue method, and setting expiration time for the cache; even if active updating is problematic, dirty data can be deleted due to expiration;
(2) establishing a second-level cache mechanism; the updated data information is classified according to types and is submitted uniformly according to the types, so that the data updating time is greatly shortened;
(3) setting a refresh cache attribute in the state configuration; when the cache is used, if the query data in the database table is manually modified, dirty data reading can occur, and through the timestamp judgment process, dirty data reading is avoided;
(4) the data are synchronized into the redis with higher service performance, and a secondary cache mechanism is utilized, so that the data updating time is further shortened;
(5) and updating the data state in a server, processing the updating service in one thread, taking out the current object from the redis to replace the synchronization lock in the other thread, and performing the operation through multi-thread synchronization, thereby realizing the remarkable improvement of the system performance.
Drawings
FIG. 1 is a flow chart of a preferred embodiment of a method for dynamically updating data in real time according to the present invention;
FIG. 2 is a flowchart illustrating a preferred embodiment of a method for a RTM tree storage engine to store data information into a first level cache according to the present invention;
FIG. 3 is a flow chart of a preferred embodiment of the present invention for determining a timestamp of data information in an RTM tree storage engine;
FIG. 4 is a schematic diagram of a data dynamic real-time update system according to a preferred embodiment of the present invention;
FIG. 5 is a schematic structural diagram of an electronic device according to a preferred embodiment of the invention;
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the technical field of data acquisition, data updating is inevitable and relates to IP address information, mobile phone blacklists and the like. When the distributed server operates, when updated information data occurs, the information data is firstly transmitted to the database, and then the server reads the data information through cache, a memory and the database once. The storage and the calling of the data information both take more time, so that the load of the server is increased; and when the real-time requirement on the data information is high during the operation of the server, the acquired data is the data information which is too old in the cache or the memory, so that the error rate of the operation of the server is high.
In order to improve the real-time performance of data updating, the invention provides a data dynamic real-time updating method. FIG. 1 shows a flow of a preferred embodiment of a method for dynamic real-time update of data according to the present invention. Referring to fig. 1, the method may be performed by an apparatus, which may be implemented by software and/or hardware.
In this embodiment, the method for dynamically updating data in real time includes: step S1-step S5.
S1, receiving the data information through the preset STORM topology.
S2, after receiving the data information, the transmitting node of the STORM topological structure sends the data information to the processing node of the STORM topological structure, and data conversion is carried out on the data information through the processing node; wherein the processing node is associated with a RTM tree storage engine.
It should be noted that, the RTM Tree (Real-Time Merge Tree) storage engine, where the RTM Tree is a Real-Time computation Merge Tree; the storage engine is associated with a processing node of the STORM topology and is used to store data information after a data conversion operation.
And S3, storing the data information after data conversion into the RTM storage engine, and storing the data information stored into the RTM storage engine into a primary cache after the information quantity of the data information in the RTM storage engine reaches a preset information quantity threshold value.
Specifically, when data information in the RTM storage engine reaches a certain preset information amount threshold value, the data information is transmitted to a first-level cache; and analyzing the received data information in the first-level cache according to the type information, deleting the original data information of the same type in the first-level cache according to the type information, and storing the newly received data information into the first-level cache.
It should be noted that, in order to improve the performance of the project and relieve the server pressure, the secondary cache needs to be started in the project, but if no other processing is performed, dirty data is read frequently. The solution is that the cache is refreshed after the commit operation is executed, and the cache is refreshed by default, so that dirty reading of the database can be avoided, and data can be synchronously updated in terms of service.
S4, the data information in the first-level cache is put into a task queue according to type information, and the data information stored in the first-level cache is sequentially stored into second-level cache data information according to the type information through the task queue.
And S5, analyzing the data information stored in the second-level cache, storing the analyzed data information into a warehousing preparation data file corresponding to the database, inputting the data information in the warehousing preparation data file into the corresponding database, and updating the number of the warehousing second-level cache data files in the task queue.
It should be noted that, by using the configuration management center, the data of the dictionary table is managed, meanwhile, a trigger of the dictionary table is designed in the database, and if there is data update, the java program is called in the trigger and synchronized into the cache.
For steps S3-S5, after the data in the database is updated, the redis cache data is updated immediately, the cache is updated immediately by using methods such as a message queue, and then overtime elimination is performed, the expiration time is set for the cache, and if a problem occurs in active update, dirty data is also expired and deleted.
That is, when new data arrives, the database is searched first and then stored in the redis; and asynchronously updating using the queue.
In a specific embodiment, a new second-level cache data file to be written is generated, and the writing of the next second-level cache data file is waited.
Specifically, the number of files for which the storage of the second-level cache data is finished is marked, and the warehousing thread of each data message judges whether the number of files for which the storage of the second-level cache data is finished is larger than the number of files for which the storage of the second-level cache data is finished.
If not, indicating that no new data information needing to be put in storage exists, and continuing to wait; if so, acquiring the next to-be-warehoused secondary cache data file according to the number of the warehoused secondary cache data files, analyzing the secondary cache data file, storing the analyzed data into the warehousing preparation data file with the corresponding model, calling a data loading tool SQL (structured query language) Loader and a warehousing control file to record the data in the warehousing preparation data file into the corresponding database after the analysis is completed, and meanwhile, updating the number of the corresponding warehoused secondary cache data files in the model queue.
FIG. 2 illustrates a flow diagram of a preferred embodiment of a method for a RTM tree storage engine to store data information in a level one cache in accordance with the present invention; referring to fig. 2, the method for the RTM tree storage engine to store data information into the first level cache includes steps S110 to S130.
S110, after receiving the data information after the data conversion, the RTM storage engine calculates the information quantity of the existing data information in the RTM storage engine;
s120, judging whether the information quantity is larger than a preset information quantity threshold value or not, and if so, triggering an RTM (resin transfer molding) tree storage engine to merge the stored data information; if not, repeating the step of S110;
and S130, storing the data information after the merging operation into a first-level cache.
That is, data information is sent from other servers and received by the primary cache, and data Transmission is realized between the other servers and the primary cache through a TCP (Transmission Control Protocol).
In other words, an information amount threshold is set in the RTM tree storage engine; firstly, after receiving new data information after conversion operation, the RTM engine calculates the information quantity value of the existing data information in the RTM storage engine; then, judging the value of the information quantity and the size of the threshold value of the information quantity through a judgment program; when the information quantity threshold value is not less than the information quantity value, the triggering condition is considered to be reached; at the moment, a merging algorithm is started, and merging operation is carried out on data information in the RTM storage engine; and finally, starting a reading-in function of the first-level cache, and reading the data information after the merging operation into the first-level cache.
In a specific embodiment, the RTM tree storage engine further includes a type determination process for the data information after receiving the data information after the data conversion, where the type determination process includes: judging whether the type of the received data information is consistent with a set type, if not, discarding the data information, and continuing to receive new data information; and if the data information is consistent with the threshold value, deleting the original data information of the same type in the first-level cache, and continuously judging the threshold value of the information quantity threshold information quantity of the received data information.
When data information in the first-level cache is placed in a task queue according to type information, analyzing an information source field in the data information, acquiring a type information mark to which the data information belongs, judging whether the data information corresponding to the type information mark is in the task queue, if not, acquiring the data information with the type information mark and adding the data information into the task queue, and creating a second-level cache data file storage directory corresponding to the data information with the type information mark, a related data file and a control file; calling a thread pool where the task queue is located, starting a warehousing thread of the data information, and writing the data information into a secondary cache data file; and if the data information corresponding to the type information mark is in the task queue, directly writing the data information into the corresponding second-level cache data file.
In a word, after one first-level cache data file is written into the corresponding second-level cache data file, the number of the written first-level cache data files is recorded, all current second-level cache data files to be written are closed and renamed, the number of the second-level cache data files to be written is saved after data information in a task queue is updated, a new second-level cache data file to be written is generated at the same time, and the process of writing data into the second-level cache next time is waited.
FIG. 3 illustrates a flow of a preferred embodiment of the timestamp determination process for data information of the RTM tree storage engine of the present invention;
referring to fig. 3, before the second-level cache receives the data information in the first-level cache, a timestamp determination process of the data information in the first-level cache is further included, where the timestamp determination process includes:
reading a time stamp of the data information, and calculating the time difference between the time stamp and the existing time;
judging whether the time difference is greater than a preset time threshold, if so, judging that the data information in the primary cache is out of date, and waiting for new data information;
if the data information is less than the preset threshold value, judging that the data information is not outdated, and calling the data information.
Specifically, when data information is analyzed or stored, a timestamp mark is added to the data information; when the data information is read, calling a timestamp judgment program, and realizing the following operation when the timestamp judgment program is executed by the processor; firstly, reading a time stamp of data information; then calculating the time difference between the existing time and the timestamp through a calculation function; then, judging the time difference value and a preset time threshold value, and judging according to the time difference value and the preset time threshold value to realize the following operations: when the time difference is larger than the time threshold, the data in the first-level cache is considered to be outdated, the data information is not taken, and the updating of the data information is waited; and when the time difference is smaller than the time threshold, calling the data information when the data in the first-level cache is considered to be not passed.
In summary, the state is updated in the service, and then the state value is determined to determine whether to process the service each time the service enters, but if the same service initiation time is too short for two times, the state is not updated for the first time, and the state is already executed for the second time, so that the state value cannot be updated, and the acquired data is dirty data.
But through the data dynamic real-time updating method of the invention, a cache corresponding to a cache name field and a deadline field is established on the basis of the original cache mechanism by establishing a secondary cache mechanism; writing an acquisition function, loading an updating cache script before loading the cache each time, and then comparing the modification time of the script with the time value in the updating cache. If the time stamp in the updating cache is larger than the modification time of the cache file, the updating cache is called, otherwise, the script file is directly loaded. And simultaneously updating the time cache data and the cache script so as to ensure the cache updating of the cache script. When the data of the database is updated, the memory data operation of updating the distributed service is brought into the same object type, all the objects are waited to be finished, and the objects are submitted uniformly, so that the data updating time can be greatly shortened.
FIG. 4 is a block diagram of a preferred embodiment of the present invention; as shown with reference to figure 4 of the drawings,
a dynamic real-time data updating system 400 comprises a data information receiving unit 410, a data information conversion unit 420, a primary cache storage unit 430, a secondary cache storage unit 440 and a secondary cache updating unit 450; wherein the content of the first and second substances,
a data information receiving unit 410, configured to receive data information through a preset STORM topology;
the data information conversion unit 420 is configured to send the data information to the processing node of the STORM topology after receiving the data information through the transmitting node of the STORM topology, and perform data conversion on the data information through the processing node; wherein the processing node is associated with a RTM tree storage engine;
the first-level cache storing unit 430 is configured to store the data information after data conversion into an RTM tree storage engine, and store the data information stored into the RTM tree storage engine into a first-level cache when an information amount of the data information in the RTM tree storage engine reaches a preset information amount threshold;
a second-level cache storing unit 440, configured to put the data information in the first-level cache into a task queue according to type information, and sequentially store the data information stored in the first-level cache into second-level cache data information according to type information through the task queue;
and the second-level cache updating unit 450 is configured to analyze the data information stored in the second-level cache, store the analyzed data information in the warehousing preparation data file corresponding to the database, record the data information in the warehousing preparation data file into the corresponding database, and update the number of the warehoused second-level cache data files in the task queue.
In a specific embodiment, the first-level cache storing unit 430 includes a data information storing module 431, a data information amount determining module 432, a data information merging module 433, and a first-level cache storing module 434; wherein the content of the first and second substances,
the data information storage module 431 is configured to calculate an information amount of existing data information in the RTM tree storage engine after the RTM tree storage engine receives the data information after the data conversion;
the information amount determination module 432 is configured to determine whether the information amount of the data information storage module is greater than a preset information amount threshold, and if so, enter the data information merging module;
the data information merging module 433 is configured to merge data information, which is stored in the data information storage module and has an information amount larger than a preset information amount threshold;
the first-level cache storing module 434 is configured to store the data information after the merging operation of the data information merging module into the first-level cache.
To sum up, the data information is firstly put into the first-level cache, and the server directly calls the data information from the first-level cache when running.
The invention provides a dynamic real-time data updating method which is applied to an electronic device 5.
FIG. 5 shows an application environment of a preferred embodiment of the dynamic real-time data updating method according to the present invention.
Referring to fig. 5, in the present embodiment, the electronic device 5 may be a terminal device having an arithmetic function, such as a server, a smart phone, a tablet computer, a portable computer, or a desktop computer.
The electronic device 5 includes: a processor 52, a memory 51, a communication bus 53, and a network interface 55.
The memory 51 includes at least one type of readable storage medium. The at least one type of readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory 51, and the like. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 5, such as a hard disk of the electronic device 5. In other embodiments, the readable storage medium may also be an external memory 51 of the electronic device 5, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the electronic device 5.
In the present embodiment, the readable storage medium of the memory 51 is generally used for storing the data dynamic real-time updating program 50 and the like installed in the electronic device 5. The memory 51 may also be used to temporarily store data that has been output or is to be output.
The processor 52 may be a Central Processing Unit (CPU), a microprocessor or other data Processing chip in some embodiments, and is used for executing program codes stored in the memory 51 or Processing data, such as executing the data dynamic real-time updating program 50.
The communication bus 53 is used to realize connection communication between these components.
The network interface 54 may optionally include a standard wired interface, a wireless interface (e.g., a WI-FI interface), and is typically used to establish a communication link between the electronic apparatus 5 and other electronic devices.
Fig. 5 only shows the electronic device 5 with components 51-54, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may alternatively be implemented.
Optionally, the electronic device 5 may further include a user interface, which may include an input unit such as a Keyboard (Keyboard), a voice input device such as a microphone (microphone) or other devices with voice recognition function, a voice output device such as a sound box, a headset, etc., and optionally may also include a standard wired interface or a wireless interface.
Optionally, the electronic device 5 may further include a display, which may also be referred to as a display screen or a display unit. In some embodiments, the display device may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an Organic Light-Emitting Diode (OLED) touch device, or the like. The display is used for displaying information processed in the electronic device 5 and for displaying a visual user interface.
Optionally, the electronic device 5 may further include a Radio Frequency (RF) circuit, a sensor, an audio circuit, and the like, which are not described in detail herein.
In the embodiment of the apparatus shown in fig. 5, the memory 51, which is a kind of computer storage medium, may include therein an operating system, and a data dynamic real-time updating program 50; the processor 52 executes the data dynamic real-time update program 50 stored in the memory 51 to implement the following steps: s1, receiving data information through a preset STORM topological structure; s2, the transmitting node of the STORM topological structure sends the received data information to the processing node of the STORM topological structure, and data conversion is carried out on the data information through the processing node; wherein the processing node is associated with a RTM tree storage engine; s3, storing the data information after data conversion into an RTM (resin transfer molding) tree storage engine, and storing the data information stored into the RTM tree storage engine into a primary cache after the information quantity of the data information in the RTM tree storage engine reaches a preset information quantity threshold; s4, putting the data information in the first-level cache into a task queue according to type information, and sequentially storing the data information stored in the first-level cache into a second-level cache through the task queue; and S5, analyzing the data information stored in the second-level cache, storing the analyzed data information into a warehousing preparation data file corresponding to the database, inputting the data information in the warehousing preparation data file into the corresponding database, and updating the number of the warehousing second-level cache data files in the task queue. Data information by type information
In other embodiments, the data dynamic real-time update program 50 may be divided into one or more modules, and the one or more modules are stored in the memory 51 and executed by the processor 52 to implement the present invention. The modules referred to herein are referred to as a series of computer program instruction segments capable of performing specified functions. The data dynamic real-time update program 50 can be divided into a data information receiving unit 410, a data information conversion unit 420, a first-level buffer storage unit 430, a second-level buffer storage unit 440, and a second-level buffer update unit 450.
In addition, the present invention also provides a computer readable storage medium, which includes a data dynamic real-time updating program, and when the data dynamic real-time updating program is executed by a processor, the data dynamic real-time updating program implements operations such as a data dynamic real-time updating method.
The embodiments of the computer-readable storage medium of the present invention are substantially the same as the embodiments of the method, the system, and the electronic device for dynamically updating data in real time, and thus, the detailed description thereof is omitted here.
In summary, the data dynamic real-time updating method, the electronic device and the computer readable storage medium of the present invention establish a cache corresponding to a cache name field and a deadline field on the basis of an original cache mechanism by establishing a second-level cache mechanism; writing an acquisition function, loading an updating cache script before loading the cache each time, and then comparing the modification time of the script with the time value in the updating cache. If the time stamp in the updating cache is larger than the modification time of the cache file, the updating cache is called, otherwise, the script file is directly loaded. And simultaneously updating the time cache data and the cache script so as to ensure the cache updating of the cache script. When the data of the database is updated, the memory data operation of updating the distributed service is brought into the same object type, and is submitted uniformly according to the type, so that the data updating time is greatly shortened.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments. Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A dynamic real-time data updating method is applied to an electronic device and is characterized by comprising the following steps:
s1, receiving data information through a preset STORM topological structure;
s2, the transmitting node of the STORM topological structure sends the received data information to the processing node of the STORM topological structure, and data conversion is carried out on the data information through the processing node; wherein the processing node is associated with a RTM tree storage engine;
s3, storing the data information after data conversion into an RTM (resin transfer molding) tree storage engine, and storing the data information stored into the RTM tree storage engine into a primary cache after the information quantity of the data information in the RTM tree storage engine reaches a preset information quantity threshold;
s4, putting the data information in the first-level cache into a task queue according to type information, and sequentially storing the data information stored in the first-level cache into a second-level cache through the task queue;
and S5, analyzing the data information stored in the second-level cache, storing the analyzed data information into a warehousing preparation data file corresponding to the database, inputting the data information in the warehousing preparation data file into the corresponding database, and updating the number of the warehousing second-level cache data files in the task queue.
2. The method according to claim 1, wherein in step S3, when the data information in the RTM tree storage engine reaches the threshold, the method for storing the data information in the primary cache includes:
s110, after receiving the data information after the data conversion, the RTM storage engine calculates the information quantity of the existing data information in the RTM storage engine;
s120, judging whether the information quantity is larger than a preset information quantity threshold value or not, and if so, triggering an RTM (resin transfer molding) tree storage engine to merge the stored data information; if not, repeating the step of S110;
and S130, storing the data information after the merging operation into a first-level cache.
3. The method according to claim 2, wherein after receiving the data information after data conversion, the RTM tree storage engine further comprises a type determining process for the data information, the type determining process comprising:
judging whether the type of the received data information is consistent with a set type, if not, discarding the data information, and continuing to receive new data information;
if the data information is consistent with the information quantity threshold value, the original data information of the same type in the first-level cache is deleted, and the received data information is continuously judged according to the information quantity and the information quantity threshold value.
4. The dynamic real-time data updating method according to claim 2 or 3, further comprising a determination process of a time stamp of the data information in the primary cache before the secondary cache receives the data information in the primary cache, wherein the determination process of the time stamp comprises:
reading a timestamp of the data information stored in the first-level cache, and calculating the time difference between the timestamp and the existing time;
judging whether the time difference is greater than a preset time threshold, if so, judging that the data information in the primary cache is out of date, and waiting for new data information;
if the data information is less than the preset threshold value, the data information is called when the data information is judged not to be over.
5. The method for dynamically updating data in real time according to claim 1, wherein in step S5, after updating the number of the data files that have been put into the storage in the secondary cache, the method further comprises:
and generating a new secondary cache data file to be written, and waiting for the next writing of the secondary cache data file.
6. A data dynamic real-time updating system is characterized by comprising a data information receiving unit, a data information conversion unit, a primary cache storage unit, a secondary cache storage unit and a secondary cache updating unit; wherein the content of the first and second substances,
the data information receiving unit is used for receiving data information through a preset STORM topological structure;
the data information conversion unit is used for sending the data information to the processing node of the STORM topological structure after receiving the data information through the transmitting node of the STORM topological structure, and performing data conversion on the data information through the processing node; wherein the processing node is associated with a RTM tree storage engine;
the first-level cache storing unit is used for storing the data information after data conversion into the RTM storage engine, and storing the data information stored into the RTM storage engine into a first-level cache after the information quantity of the data information in the RTM storage engine reaches a preset information quantity threshold;
the second-level cache storing unit is used for putting the data information in the first-level cache into a task queue according to the type information and sequentially storing the data information stored in the first-level cache into a second-level cache through the task queue;
and the second-level cache updating unit is used for analyzing the data information stored in the second-level cache, storing the analyzed data information into the warehousing preparation data file corresponding to the database, inputting the data information in the warehousing preparation data file into the corresponding database, and updating the number of the warehoused second-level cache data files in the task queue.
7. The dynamic real-time data update system of claim 6,
the first-level cache storing unit comprises a data information storage module, a data information quantity judging module, a data information merging module and a first-level cache storing module; wherein the content of the first and second substances,
the data information storage module is used for calculating the information quantity of the existing data information in the RTM storage engine after the RTM storage engine receives the data information after data conversion;
the information quantity judging module of the data information is used for judging whether the information quantity of the data information storage module is larger than a preset information quantity threshold value or not, and if so, the data information merging module is started;
the data information merging module is used for merging the data information of which the information quantity stored in the data information storage module is larger than a preset information quantity threshold value;
and the first-level cache storing module is used for storing the data information subjected to merging operation by the data information merging module into the first-level cache.
8. An electronic device, comprising: the data dynamic real-time updating program is stored in the memory and is executed by the processor to realize the following steps:
s1, receiving data information through a preset STORM topological structure;
s2, the transmitting node of the STORM topological structure sends the received data information to the processing node of the STORM topological structure, and data conversion is carried out on the data information through the processing node; wherein the processing node is associated with a RTM tree storage engine;
s3, storing the data information after data conversion into an RTM (resin transfer molding) tree storage engine, and storing the data information stored into the RTM tree storage engine into a primary cache after the information quantity of the data information in the RTM tree storage engine reaches a preset information quantity threshold;
s4, putting the data information in the first-level cache into a task queue according to type information, and sequentially storing the data information stored in the first-level cache into a second-level cache through the task queue;
and S5, analyzing the data information stored in the second-level cache, storing the analyzed data information into a warehousing preparation data file corresponding to the database, inputting the data information in the warehousing preparation data file into the corresponding database, and updating the number of the warehousing second-level cache data files in the task queue. The data information is according to type information.
9. The electronic device according to claim 8, wherein in step S3, after the data information in the RTM tree storage engine reaches the threshold, the method for storing the data information in the first-level cache includes:
s110, after receiving the data information after the data conversion, the RTM storage engine calculates the information quantity of the existing data information in the RTM storage engine;
s120, judging whether the information quantity is larger than a preset information quantity threshold value or not, and if so, triggering an RTM (resin transfer molding) tree storage engine to merge the stored data information; if not, repeating the step of S110;
and S130, storing the data information after the merging operation into a first-level cache.
10. A computer-readable storage medium, in which a computer program is stored, the computer program comprising a data dynamic real-time update program, which when executed by a processor, implements the steps of the data dynamic real-time update method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010122961.5A CN111367925A (en) | 2020-02-27 | 2020-02-27 | Data dynamic real-time updating method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010122961.5A CN111367925A (en) | 2020-02-27 | 2020-02-27 | Data dynamic real-time updating method, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111367925A true CN111367925A (en) | 2020-07-03 |
Family
ID=71209581
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010122961.5A Pending CN111367925A (en) | 2020-02-27 | 2020-02-27 | Data dynamic real-time updating method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111367925A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112434066A (en) * | 2020-11-24 | 2021-03-02 | 深圳市前海迅智系统科技有限公司 | Multi-channel high-frequency real-time data aggregation method and terminal |
| CN112764908A (en) * | 2021-01-26 | 2021-05-07 | 北京鼎普科技股份有限公司 | Network data acquisition processing method and device and electronic equipment |
| CN113297216A (en) * | 2021-05-17 | 2021-08-24 | 中国人民解放军63920部队 | Real-time storage method for space flight measurement and control data |
| CN113742290A (en) * | 2021-11-04 | 2021-12-03 | 上海闪马智能科技有限公司 | Data storage method and device, storage medium and electronic device |
| CN113760570A (en) * | 2021-01-07 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Data processing method, device, electronic equipment, system and storage medium |
| CN114513556A (en) * | 2021-12-31 | 2022-05-17 | 西安电子科技大学 | Two-stage time-triggered service scheduling method and system based on deterministic network |
| CN115167891A (en) * | 2022-07-07 | 2022-10-11 | 北京蓝天航空科技股份有限公司 | Data updating method, device and equipment of interface control file and storage medium |
| CN116342371A (en) * | 2023-03-24 | 2023-06-27 | 摩尔线程智能科技(北京)有限责任公司 | Method for GPU and secondary cache, GPU and secondary cache |
| CN116832439A (en) * | 2023-05-17 | 2023-10-03 | 广州三七极梦网络技术有限公司 | Processing method, device, equipment and medium of game data refreshing system |
| CN116832439B (en) * | 2023-05-17 | 2024-06-07 | 广州三七极梦网络技术有限公司 | Processing method, device, equipment and medium of game data refreshing system |
-
2020
- 2020-02-27 CN CN202010122961.5A patent/CN111367925A/en active Pending
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112434066A (en) * | 2020-11-24 | 2021-03-02 | 深圳市前海迅智系统科技有限公司 | Multi-channel high-frequency real-time data aggregation method and terminal |
| CN113760570A (en) * | 2021-01-07 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Data processing method, device, electronic equipment, system and storage medium |
| CN112764908B (en) * | 2021-01-26 | 2024-01-26 | 北京鼎普科技股份有限公司 | Network data acquisition processing method and device and electronic equipment |
| CN112764908A (en) * | 2021-01-26 | 2021-05-07 | 北京鼎普科技股份有限公司 | Network data acquisition processing method and device and electronic equipment |
| CN113297216A (en) * | 2021-05-17 | 2021-08-24 | 中国人民解放军63920部队 | Real-time storage method for space flight measurement and control data |
| CN113742290A (en) * | 2021-11-04 | 2021-12-03 | 上海闪马智能科技有限公司 | Data storage method and device, storage medium and electronic device |
| CN114513556A (en) * | 2021-12-31 | 2022-05-17 | 西安电子科技大学 | Two-stage time-triggered service scheduling method and system based on deterministic network |
| CN114513556B (en) * | 2021-12-31 | 2024-02-02 | 西安电子科技大学 | Two-stage time triggering service scheduling method and system based on deterministic network |
| CN115167891A (en) * | 2022-07-07 | 2022-10-11 | 北京蓝天航空科技股份有限公司 | Data updating method, device and equipment of interface control file and storage medium |
| CN115167891B (en) * | 2022-07-07 | 2023-09-05 | 北京蓝天航空科技股份有限公司 | Data updating method, device and equipment of interface control file and storage medium |
| CN116342371A (en) * | 2023-03-24 | 2023-06-27 | 摩尔线程智能科技(北京)有限责任公司 | Method for GPU and secondary cache, GPU and secondary cache |
| CN116342371B (en) * | 2023-03-24 | 2024-05-24 | 摩尔线程智能科技(北京)有限责任公司 | Method for GPU and secondary cache, GPU and secondary cache |
| CN116832439A (en) * | 2023-05-17 | 2023-10-03 | 广州三七极梦网络技术有限公司 | Processing method, device, equipment and medium of game data refreshing system |
| CN116832439B (en) * | 2023-05-17 | 2024-06-07 | 广州三七极梦网络技术有限公司 | Processing method, device, equipment and medium of game data refreshing system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111367925A (en) | Data dynamic real-time updating method, device and storage medium | |
| CN112162965B (en) | Log data processing method, device, computer equipment and storage medium | |
| CN109086382B (en) | Data synchronization method, device, equipment and storage medium | |
| CN111046041B (en) | Data processing method and device, storage medium and processor | |
| CN112154420A (en) | Automatic intelligent cloud service testing tool | |
| CN111651296A (en) | Interception backup method, device, equipment and storage medium for data deletion operation | |
| CN108415998B (en) | Application dependency relationship updating method, terminal, device and storage medium | |
| CN110650164B (en) | File uploading method and device, terminal and computer storage medium | |
| CN109842621A (en) | A kind of method and terminal reducing token storage quantity | |
| CN115455058A (en) | Cache data processing method and device, computer equipment and storage medium | |
| WO2020211363A1 (en) | Method and apparatus for improving efficiency of program loading, computer device and storage medium | |
| CN111125170A (en) | Cross-service data acquisition method and device of micro-service and micro-service equipment | |
| CN112671878B (en) | Block chain information subscription method, device, server and storage medium | |
| CN115062084B (en) | Method and device for constructing API (application programming interface) based on database metadata | |
| CN113986942B (en) | Message queue management method and device based on man-machine conversation | |
| EP4280565A1 (en) | Sample message processing method and apparatus | |
| CN114528049A (en) | Method and system for realizing API call information statistics based on InfluxDB | |
| CN110688201B (en) | Log management method and related equipment | |
| CN114281817A (en) | Data cleaning method and device, computer equipment and storage medium | |
| CN108491448B (en) | Data pushing method and device | |
| CN113377791A (en) | Data processing method, system and computing equipment | |
| CN111538651A (en) | Interface testing method, device, server and storage medium | |
| CN113608817B (en) | Method and system for processing bullet frame | |
| CN112702270B (en) | Node calling method, system and storage medium based on event distribution mechanism | |
| CN113268487B (en) | Data statistical method, device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |