CN112182004B - Method, device, computer equipment and storage medium for checking data in real time - Google Patents
Method, device, computer equipment and storage medium for checking data in real time Download PDFInfo
- Publication number
- CN112182004B CN112182004B CN202011052057.8A CN202011052057A CN112182004B CN 112182004 B CN112182004 B CN 112182004B CN 202011052057 A CN202011052057 A CN 202011052057A CN 112182004 B CN112182004 B CN 112182004B
- Authority
- CN
- China
- Prior art keywords
- data
- preset
- static
- dynamic
- real time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 230000003068 static effect Effects 0.000 claims abstract description 121
- 238000012545 processing Methods 0.000 claims abstract description 66
- 238000012216 screening Methods 0.000 claims description 34
- 230000008569 process Effects 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000005111 flow chemistry technique Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 description 21
- 230000008859 change Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003203 everyday effect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
Abstract
The embodiment of the application belongs to the technical field of big data, is applied to the field of intelligent logistics, and relates to a method for checking data in real time, which comprises the steps of acquiring an original data set in a database in real time, wherein the original data set comprises static data and dynamic data; storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval; processing the dynamic data based on preset stream processing in real time to obtain intermediate data; static data in a preset data cache are obtained in real time, the static data and the intermediate data are assembled, display data are obtained, and the display data are displayed in a data billboard of a front-end page. The application also provides a real-time data viewing device, computer equipment and a storage medium. Wherein the presentation data may be stored in a blockchain. The method and the device enhance the real-time performance of the data and improve the processing efficiency of the computer.
Description
Technical Field
The present disclosure relates to the field of big data technologies, and in particular, to a method, an apparatus, a computer device, and a storage medium for viewing data in real time.
Background
With the continuous high-speed development of computer technology, computers have been applied to various industries and have an irreplaceable role in life and work. The data generated by the computer can be stored in the database, and the computer can perform operations such as adding, deleting, checking, modifying and the like on the data in the database.
At present, row lock and table lock of a database often occur, and contents in the database need to be continuously read, so that real-time viewing of data is realized. When the data volume in the database is large, the computer cannot display the data content in time, so that the problem of poor user experience effect is caused.
Disclosure of Invention
An aim of the embodiment of the application is to provide a method, a device, computer equipment and a storage medium for checking data in real time, so that the real-time performance of the data is enhanced, and the processing efficiency of a computer is improved.
In order to solve the above technical problems, the embodiment of the present application provides a method for viewing data in real time, which adopts the following technical scheme:
a method of viewing data in real time, comprising the steps of:
acquiring an original data set in a database in real time, wherein the original data set comprises static data and dynamic data;
storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval;
processing the dynamic data based on preset stream processing in real time to obtain intermediate data;
static data in a preset data cache are obtained in real time, the static data and the intermediate data are assembled, display data are obtained, and the display data are displayed in a data billboard of a front-end page.
Further, the step of storing the static data in a preset data buffer, and updating the static data stored in the data buffer based on a preset time interval includes:
identifying static data in the original data set based on a preset static data lookup table;
and storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval.
Further, the step of storing the static data in a preset data buffer, and updating the static data stored in the data buffer based on a preset time interval includes:
storing the static data into a preset data cache;
acquiring static data in the original data set at the current time based on a preset time interval;
identifying whether static data is stored in the data cache;
when the static data is stored in the data cache, deleting the static data in the data cache, and storing the static data acquired at the current time into the data cache.
Further, the step of processing the dynamic data based on the preset streaming process in real time to obtain intermediate data includes:
combining the dynamic data into a dynamic data set;
calling a stream conversion function in the stream processing to convert a dynamic data set into a set stream;
acquiring screening conditions input or selected by a user on a front-end page;
adding the screening conditions into the screening function of the flow processing, and screening the aggregate flow based on the screening function added with the screening conditions;
and encapsulating the filtered aggregate flow through a collection function in the flow processing to generate the intermediate data.
Further, the step of processing the dynamic data based on the preset streaming process in real time to obtain intermediate data includes:
combining the dynamic data into a dynamic data set;
identifying a data volume of the dynamic data set;
when the data volume of the dynamic data set is larger than a preset threshold, splitting the dynamic data set to obtain at least two different data sets;
and respectively carrying out streaming processing on the data sets through different preset nodes, and summarizing the processed data sets to obtain intermediate data.
Further, the step of splitting the dynamic data set to obtain at least two different data sets includes:
identifying a source attribute of each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on the source attribute to obtain at least two different data sets.
Further, the step of splitting the dynamic data set to obtain at least two different data sets includes:
identifying a time stamp carried by each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on a preset time period and a time stamp to obtain at least two different data sets.
In order to solve the above technical problems, the embodiment of the present application further provides a real-time data viewing device, which adopts the following technical scheme:
a real-time viewing data device, comprising:
the acquisition module is used for acquiring an original data set in the database in real time, wherein the original data set comprises static data and dynamic data;
the updating module is used for storing the static data into a preset data cache and updating the static data stored in the data cache based on a preset time interval;
the processing module is used for processing the dynamic data based on preset stream processing in real time to obtain intermediate data; and
the assembly module is used for acquiring static data in a preset data cache in real time, assembling the static data and the intermediate data to obtain display data, and displaying the display data to a data billboard of a front page.
In order to solve the above technical problems, the embodiments of the present application further provide a computer device, which adopts the following technical schemes:
a computer device comprising a memory having stored therein computer readable instructions which when executed by a processor implement the steps of the method for viewing data in real time described above.
In order to solve the above technical problems, embodiments of the present application further provide a computer readable storage medium, which adopts the following technical solutions:
a computer readable storage medium having stored thereon computer readable instructions which when executed by a processor perform the steps of the method of viewing data in real time described above.
Compared with the prior art, the embodiment of the application has the following main beneficial effects:
according to the method and the device, the data is split into the static data and the dynamic data, the dynamic data is processed through streaming processing, so that the calculation efficiency is improved, the real-time performance of the data is enhanced, the calculation efficiency is improved by using the streaming processing and the static data caching, and the mass data calculation is facilitated. And displaying the display data to a data billboard of the front-end page, so that related users such as business personnel can see the data in the database in real time in the billboard, and the user experience is enhanced.
Drawings
For a clearer description of the solution in the present application, a brief description will be given below of the drawings that are needed in the description of the embodiments of the present application, it being obvious that the drawings in the following description are some embodiments of the present application, and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art.
FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;
FIG. 2 is a flow chart of one embodiment of a method of viewing data in real time according to the present application;
FIG. 3 is a schematic diagram of one embodiment of a real-time viewing data apparatus according to the present application;
FIG. 4 is a schematic structural diagram of one embodiment of a computer device according to the present application.
Reference numerals: 200. a computer device; 201. a memory; 202. a processor; 203. a network interface; 300. viewing the data device in real time; 301. an acquisition module; 302. updating a module; 303. a processing module; 304. and assembling the module.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the applications herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having" and any variations thereof in the description and claims of the present application and in the description of the figures above are intended to cover non-exclusive inclusions. The terms first, second and the like in the description and in the claims or in the above-described figures, are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
In order to better understand the technical solutions of the present application, the following description will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings.
As shown in fig. 1, a system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as a web browser application, a shopping class application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc., may be installed on the terminal devices 101, 102, 103.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablet computers, electronic book readers, MP3 players (Moving Picture Experts Group Audio Layer III, dynamic video expert compression standard audio plane 3), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic video expert compression standard audio plane 4) players, laptop and desktop computers, and the like.
The server 105 may be a server providing various services, such as a background server providing support for pages displayed on the terminal devices 101, 102, 103.
It should be noted that, the method for viewing data in real time provided in the embodiments of the present application is generally executed by a server/terminal device, and accordingly, the device for viewing data in real time is generally disposed in the server/terminal device.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to FIG. 2, a flow chart of one embodiment of a method of viewing data in real time according to the present application is shown. The method for viewing the data in real time comprises the following steps:
s1: and acquiring an original data set in a database in real time, wherein the original data set comprises static data and dynamic data.
In this embodiment, the data in the database is updated by random variation, and it is important for the user to be able to acquire the data in the database in real time. The original data set in the database comprises various data tables, wherein the data tables of different types have association relations, and the association relations are established by a computer according to specific keywords among the data tables. The specific keywords may be found from a preset keyword table. For example, financial statement and attendance table belong to different types of data tables, while financial statement 1 and financial statement 2 belong to the same type but different data tables. Financial statement 1, financial statement 2, and attendance list are passed through specific keywords such as: and (5) establishing an association relationship by Shanghai son company. Any data table comprises static data and dynamic data. Such as: static data in the financial statement 1 are personnel fields, institution name fields and the like, and dynamic data are expenditure amounts, total balances and the like. For insurance companies, the statistical table of the quantity of the insurance policy is changed in real time, and any delayed message may cause a series of adverse effects. In the data table, both static data and dynamic data have their locations fixed.
In this embodiment, the electronic device (e.g., the server/terminal device shown in fig. 1) on which the real-time data viewing method operates may acquire the original data set in the database through a wired connection manner or a wireless connection manner. It should be noted that the wireless connection may include, but is not limited to, 3G/4G connections, wiFi connections, bluetooth connections, wiMAX connections, zigbee connections, UWB (ultra wideband) connections, and other now known or later developed wireless connection means.
S2: and storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval.
In this embodiment, static data is determined according to the position of data in the data table, and generally, the data in the left column of the data table is static data, and the data in the right column is dynamic data. Of course, which type of data in the data table is static data can be correspondingly determined according to the static data lookup table. After the static data is determined, the static data is stored in a preset data cache. Because the static data cannot change in real time, the static data is not required to be processed in real time, the static data is stored and called independently, the data volume of the data processed by the computer later is reduced, the processing speed of the computer is effectively improved, and further the data is processed and displayed later in real time. And after a preset time interval, the static data in the original data set acquired in real time are stored in the cache again so as to update the static data in the cache.
Specifically, the step of storing the static data in a preset data buffer, and updating the static data stored in the data buffer based on a preset time interval includes:
identifying static data in the original data set based on a preset static data lookup table;
and storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval.
In this embodiment, the data cache in this application refers to a redis cache. Static data may be cached data, such as secondary organization information, staff information; for different business scenarios, the organization data will not change every day, and the personnel data may change every day, but for the whole business, if the latest data is refreshed every day to the cache, it is acceptable at the business level, and data like this can be prestored to redis for caching, so that quick reading can be realized when the system is used in a signboard.
The step of storing the static data in a preset data cache and updating the static data stored in the data cache based on a preset time interval comprises the following steps:
storing the static data into a preset data cache;
acquiring static data in the original data set at the current time based on a preset time interval;
identifying whether static data is stored in the data cache;
when the static data is stored in the data cache, deleting the static data in the data cache, and storing the static data acquired at the current time into the data cache.
In this embodiment, when static data is not stored in the data cache, the static data acquired at the current time may be directly stored in the data cache. And continuously updating the static data in the data cache through a preset time interval. If static data is stored in the data cache, deleting is carried out, so that subsequent data redundancy and confusion are avoided.
S3: and processing the dynamic data based on preset stream processing in real time to obtain intermediate data.
In this embodiment, the service data required by the current billboard is collected through stream processing. The streaming process simplifies the operation on the collection, and includes three parts of conversion into streams, intermediate operations, and terminal operations. Stream processing directly converts a data source into streams, including aggregate streams, array streams, file streams, and the like; after being converted into a stream, intermediate operations are carried out, wherein the intermediate operations comprise operations such as filter, limit, distict and the like; after the intermediate operation is processed, terminal operation is finally carried out, wherein the terminal operation comprises allMatvh, anyMatch, collet and the like, and the final result set is summarized and returned according to preset requirements.
In this application, for a service with a huge data volume, spark may be introduced into stream processing to perform calculation, for example, a Spark cluster may be deployed separately, and data used by the service of the billboard may be calculated specifically. The Spark calculation has the advantages that: 1) Memory-based computation; 2) Can be allowed to run directly on HDFS (easily scalable distributed file system); 3) Data can be stored in both disk and memory.
Specifically, the step of processing the dynamic data based on the preset stream processing in real time to obtain intermediate data includes:
combining the dynamic data into a dynamic data set;
invoking a stream conversion function (stream) in the stream processing to convert a dynamic data set into a set stream;
acquiring screening conditions input or selected by a user on a front-end page;
adding the screening conditions to a screening function (filter) of the streaming process, and screening the aggregate flow based on the filter added with the screening conditions;
and encapsulating the filtered aggregate flow through a collection function (collection) in the streaming process to generate a list set as the intermediate data.
In the present embodiment, after the data is captured, the data is converted into an aggregate stream by the stream method of stream processing. In the screening process, as long as the dynamic data corresponding to one data table accords with the screening condition, the dynamic data corresponding to the data table with the association relation also passes through the screening and enters the subsequent steps, and the following two examples are described:
example one: various data tables of different types form an original data set, and for a financial statement and an attendance table with an association relationship, as long as dynamic data corresponding to the financial statement accords with screening conditions, the dynamic data corresponding to the attendance table with the association relationship also passes through screening, and then the subsequent steps are carried out.
Example two: the data source of the application may be original unconditional policy data. Different data are scattered in different insurance data tables, and the different insurance data tables form an original data set. And screening the original data set according to preset screening conditions, wherein different insurance data tables have association relations, and if the dynamic data corresponding to one insurance data table accords with the screening conditions, the dynamic data corresponding to the associated insurance data table also passes through the screening, and the subsequent steps are carried out.
The efficiency of the streaming processing is doubled compared with that of the traditional iterative mode, the computing time is reduced in multiple, if the machine has a multi-core CPU, the streaming processing can be processed in parallel, the computing efficiency is further improved, and finally the effect of displaying service data in real time can be achieved; compared with the traditional oracle database calculation task, the flexibility and the calculation efficiency are greatly improved.
After the data is filtered according to the service rule, the data is packaged into a collection and a list set is appointed to be returned, namely, the data is assembled into a list set form. Specific filtration conditions are as follows: the screening condition is that the policy is handed over more than 4 times (i.e. the number of times of paying the insurance fee), the service personnel of the policy is a district topology personnel, and the state of the policy is an orphan, wherein the long-term policy of the off-office attendant is called the orphan. Filter screening conditions are added in the middle step of the streaming process.
In addition, the step of processing the dynamic data based on the preset stream processing in real time to obtain intermediate data includes:
combining the dynamic data into a dynamic data set;
identifying a data volume of the dynamic data set;
when the data volume of the dynamic data set is larger than a preset threshold, splitting the dynamic data set to obtain at least two different data sets;
and respectively carrying out streaming processing on the data sets through different preset nodes, and summarizing the processed data sets to obtain intermediate data.
In this embodiment, when the data amount of the dynamic data set is greater than a preset threshold, the above-mentioned streaming processing procedure is directly performed on the dynamic data set without splitting. And generating a dynamic data set from the dynamic data, so that the subsequent streaming processing of the dynamic data and the management of the dynamic data are facilitated. And when the data volume is large, the method automatically switches to a plurality of nodes for permission, firstly, the original data is split, the distributed nodes are utilized to process the service data in a scattered manner, and then the service data are summarized after the split processing, so that the processing speed of the data is improved, and the situations of downtime or processing blocking and the like caused by the overlarge data volume are avoided.
Wherein the step of splitting the dynamic data set to obtain at least two different data sets comprises:
identifying a source attribute of each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on the source attribute to obtain at least two different data sets.
In this embodiment, the splitting principle may be splitting by region. Data having regions as source attributes of dynamic data sets, such as Guangzhou base, is allowed on node 1; the Beijing library data is allowed on node 2. After the data is calculated by the node 1 and the node 2, the data is summarized and displayed on a data signboard in real time.
Of course, the step of splitting the dynamic data set to obtain at least two different data sets includes:
identifying a time stamp carried by each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on a preset time period and a time stamp to obtain at least two different data sets.
In this embodiment, the splitting principle may also be split according to the latitude and the longitude. The dynamic data set is derived from an original data set, wherein the original data set is obtained from a database. The original data set carries a time stamp, and each piece of dynamic data in the dynamic data set also carries a time stamp, so that the dynamic data set can be split according to different time periods, and the dynamic data set can be split based on time latitude.
S4: static data in a preset data cache are obtained in real time, the static data and the intermediate data are assembled, display data are obtained, and the display data are displayed in a data billboard of a front-end page.
In this embodiment, static data in a cache is obtained, intermediate data and the static data are placed according to a preset fixed position, whether the position of dynamic data corresponding to the static data is empty or not is identified after placement, if so, the static data is deleted, data assembly is finally completed, display data is formed, and the display data is displayed in a billboard of a front-end page, so that a user can conveniently view the data in real time.
According to the method and the device, the data is split into the static data and the dynamic data, the dynamic data is processed through streaming processing, so that the calculation efficiency is improved, the real-time performance of the data is enhanced, the calculation efficiency is improved by using the streaming processing and the static data caching, and the mass data calculation is facilitated. And displaying the display data to a data billboard of the front-end page, so that related users such as business personnel can see the data in the database in real time in the billboard, and the user experience is enhanced.
It should be emphasized that, to further ensure the privacy and security of the presentation data, the presentation data may also be stored in a blockchain node.
The blockchain referred to in the application is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm and the like. The Blockchain (Blockchain), which is essentially a decentralised database, is a string of data blocks that are generated by cryptographic means in association, each data block containing a batch of information of network transactions for verifying the validity of the information (anti-counterfeiting) and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, an application services layer, and the like.
The application can be applied to the wisdom commodity circulation field to promote the construction in wisdom city, because there are more relativity tables in commodity circulation field, and most need look over in real time, confirm information such as order state, the change condition of amount, application this application can effectively improve logistics staff's work efficiency.
Those skilled in the art will appreciate that implementing all or part of the processes of the methods of the embodiments described above may be accomplished by way of computer readable instructions, stored on a computer readable storage medium, which when executed may comprise processes of embodiments of the methods described above. The storage medium may be a nonvolatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a random access Memory (Random Access Memory, RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.
With further reference to fig. 3, as an implementation of the method shown in fig. 2, the present application provides an embodiment of a device for viewing data in real time, where the embodiment of the device corresponds to the embodiment of the method shown in fig. 2, and the device may be applied to various electronic devices.
As shown in fig. 3, the real-time viewing data apparatus 300 according to the present embodiment includes: an acquisition module 301, an update module 302, a processing module 303, and an assembly module 304. The acquiring module 301 is configured to acquire an original data set in a database in real time, where the original data set includes static data and dynamic data; the updating module 302 is configured to store the static data into a preset data cache, and update the static data stored in the data cache based on a preset time interval; the processing module 303 is configured to process the dynamic data in real time based on a preset streaming process, so as to obtain intermediate data; and an assembling module 304, configured to obtain static data in a preset data cache in real time, assemble the static data with the intermediate data, obtain display data, and display the display data to a data board of a front page.
In the embodiment, the data is split into the static data and the dynamic data, the dynamic data is processed through stream processing, so that the calculation efficiency is improved, the real-time performance of the data is enhanced, the calculation efficiency is improved by using the stream processing and the static data caching mode, and the mass data calculation is facilitated. And displaying the display data to a data billboard of the front-end page, so that related users such as business personnel can see the data in the database in real time in the billboard, and the user experience is enhanced.
The update module 302 includes an identification sub-module and an update sub-module. The identification sub-module is used for identifying static data in the original data set based on a preset static data lookup table; the updating sub-module is used for storing the static data into a preset data cache and updating the static data stored in the data cache based on a preset time interval.
The updating sub-module comprises a storage unit, an acquisition unit, an identification unit and a deletion unit. The storage unit is used for storing the static data into a preset data cache; the acquisition unit is used for acquiring static data in the original data set at the current time based on a preset time interval; the identification unit is used for identifying whether static data are stored in the data cache; the deleting unit is used for deleting the static data in the data cache when the static data is stored in the data cache, and storing the static data acquired at the current time into the data cache.
The processing module 303 includes a generating sub-module, an obtaining sub-module, a screening sub-module, and a packaging sub-module. The generation sub-module is used for combining the dynamic data into a dynamic data set; the acquisition submodule is used for calling the stream in the streaming processing and converting the dynamic data set into a set stream; the screening sub-module is used for acquiring screening conditions input or selected by a user on a front-end page; the packaging submodule is used for adding the screening conditions into the filters of the stream processing, and screening the aggregate stream based on the filters added with the screening conditions; the encapsulation submodule is used for encapsulating the filtered aggregate flow through the collection in the streaming process to generate the intermediate data.
The processing module 303 further includes a collection sub-module, a number sub-module, a split sub-module, and a summary sub-module. The collection sub-module is used for combining the dynamic data into a dynamic data collection; the number sub-module is used for identifying the data volume of the dynamic data set; the splitting submodule is used for splitting the dynamic data set to obtain at least two different data sets when the data quantity of the dynamic data set is larger than a preset threshold value; and the summarizing submodule is used for respectively carrying out streaming processing on the data sets through different preset nodes and summarizing the processed data sets to obtain intermediate data.
The splitting submodule comprises an attribute unit and a first splitting unit. The attribute unit is used for identifying the source attribute of each piece of dynamic data in the dynamic data set; the first splitting unit splits the dynamic data set based on the source attribute to obtain at least two different data sets.
The splitting submodule further comprises a time unit and a second splitting unit. The time unit is used for identifying a time stamp carried by each piece of dynamic data in the dynamic data set; the second splitting unit is used for splitting the dynamic data set based on a preset time period and a time stamp to obtain at least two different data sets.
According to the method and the device, the data is split into the static data and the dynamic data, the dynamic data is processed through streaming processing, so that the calculation efficiency is improved, the real-time performance of the data is enhanced, the calculation efficiency is improved by using the streaming processing and the static data caching, and the mass data calculation is facilitated. And displaying the display data to a data billboard of the front-end page, so that related users such as business personnel can see the data in the database in real time in the billboard, and the user experience is enhanced.
In order to solve the technical problems, the embodiment of the application also provides computer equipment. Referring specifically to fig. 4, fig. 4 is a basic structural block diagram of a computer device according to the present embodiment.
The computer device 200 includes a memory 201, a processor 202, and a network interface 203 communicatively coupled to each other via a system bus. It should be noted that only computer device 200 having components 201-203 is shown in the figures, but it should be understood that not all of the illustrated components are required to be implemented and that more or fewer components may be implemented instead. It will be appreciated by those skilled in the art that the computer device herein is a device capable of automatically performing numerical calculations and/or information processing in accordance with predetermined or stored instructions, the hardware of which includes, but is not limited to, microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASICs), programmable gate arrays (fields-Programmable Gate Array, FPGAs), digital processors (Digital Signal Processor, DSPs), embedded devices, etc.
The computer equipment can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing equipment. The computer equipment can perform man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch pad or voice control equipment and the like.
The memory 201 includes at least one type of readable storage medium including flash memory, hard disk, multimedia card, card memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), programmable Read Only Memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the storage 201 may be an internal storage unit of the computer device 200, such as a hard disk or a memory of the computer device 200. In other embodiments, the memory 201 may also be an external storage device of the computer device 200, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the computer device 200. Of course, the memory 201 may also include both internal storage units of the computer device 200 and external storage devices. In this embodiment, the memory 201 is typically used to store an operating system and various application software installed on the computer device 200, such as computer readable instructions for a method for viewing data in real time. In addition, the memory 201 may be used to temporarily store various types of data that have been output or are to be output.
The processor 202 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 202 is generally used to control the overall operation of the computer device 200. In this embodiment, the processor 202 is configured to execute computer readable instructions stored in the memory 201 or process data, for example, execute computer readable instructions of the method for viewing data in real time.
The network interface 203 may comprise a wireless network interface or a wired network interface, which network interface 203 is typically used to establish communication connections between the computer device 200 and other electronic devices.
In the embodiment, the real-time performance of the data is enhanced, the calculation efficiency is improved by using the streaming processing and static data caching modes, and the mass data calculation is facilitated.
The present application also provides another embodiment, namely, a computer-readable storage medium storing computer-readable instructions executable by at least one processor to cause the at least one processor to perform the steps of a method for viewing data in real time as described above.
In the embodiment, the real-time performance of the data is enhanced, the calculation efficiency is improved by using the streaming processing and static data caching modes, and the mass data calculation is facilitated.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method described in the embodiments of the present application.
It is apparent that the embodiments described above are only some embodiments of the present application, but not all embodiments, the preferred embodiments of the present application are given in the drawings, but not limiting the patent scope of the present application. This application may be embodied in many different forms, but rather, embodiments are provided in order to provide a more thorough understanding of the present disclosure. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described in the foregoing, or equivalents may be substituted for elements thereof. All equivalent structures made by the specification and the drawings of the application are directly or indirectly applied to other related technical fields, and are also within the protection scope of the application.
Claims (9)
1. A method for viewing data in real time, comprising the steps of:
acquiring an original data set in a database in real time, wherein the original data set comprises static data and dynamic data;
storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval;
processing the dynamic data based on preset stream processing in real time to obtain intermediate data;
static data in a preset data cache are obtained in real time, the static data and the intermediate data are assembled, display data are obtained, and the display data are displayed in a data billboard of a front-end page;
the step of processing the dynamic data based on preset stream processing in real time to obtain intermediate data comprises the following steps:
combining the dynamic data into a dynamic data set;
identifying a data volume of the dynamic data set;
when the data volume of the dynamic data set is larger than a preset threshold, splitting the dynamic data set to obtain at least two different data sets;
and respectively carrying out streaming processing on the data sets through different preset nodes, and summarizing the processed data sets to obtain intermediate data.
2. The method of viewing data in real time according to claim 1, wherein the step of storing the static data in a preset data buffer and updating the static data stored in the data buffer based on a preset time interval comprises:
identifying static data in the original data set based on a preset static data lookup table;
and storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval.
3. The method of viewing data in real time according to claim 2, wherein the step of storing the static data in a preset data buffer and updating the static data stored in the data buffer based on a preset time interval comprises:
storing the static data into a preset data cache;
acquiring static data in the original data set at the current time based on a preset time interval;
identifying whether static data is stored in the data cache;
when the static data is stored in the data cache, deleting the static data in the data cache, and storing the static data acquired at the current time into the data cache.
4. The method for viewing data in real time according to claim 1, wherein the step of processing the dynamic data based on a preset streaming process in real time to obtain intermediate data comprises:
combining the dynamic data into a dynamic data set;
calling a stream conversion function in the stream processing to convert a dynamic data set into a set stream;
acquiring screening conditions input or selected by a user on a front-end page;
adding the screening conditions into the screening function of the flow processing, and screening the aggregate flow based on the screening function added with the screening conditions;
and encapsulating the filtered aggregate flow through a collection function in the flow processing to generate the intermediate data.
5. The method of viewing data in real time according to claim 1, wherein the step of splitting the dynamic data set to obtain at least two different data sets comprises:
identifying a source attribute of each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on the source attribute to obtain at least two different data sets.
6. The method of viewing data in real time according to claim 1, wherein the step of splitting the dynamic data set to obtain at least two different data sets comprises:
identifying a time stamp carried by each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on a preset time period and a time stamp to obtain at least two different data sets.
7. A real-time viewing data device, comprising:
the acquisition module is used for acquiring an original data set in the database in real time, wherein the original data set comprises static data and dynamic data;
the updating module is used for storing the static data into a preset data cache and updating the static data stored in the data cache based on a preset time interval;
the processing module is used for processing the dynamic data based on preset stream processing in real time to obtain intermediate data; and
the assembly module is used for acquiring static data in a preset data cache in real time, assembling the static data and the intermediate data to obtain display data, and displaying the display data to a data billboard of a front page;
the processing module is further configured to combine the dynamic data into a dynamic data set; identifying a data volume of the dynamic data set; when the data volume of the dynamic data set is larger than a preset threshold, splitting the dynamic data set to obtain at least two different data sets; and respectively carrying out streaming processing on the data sets through different preset nodes, and summarizing the processed data sets to obtain intermediate data.
8. A computer device comprising a memory having stored therein computer readable instructions which when executed implement the steps of the method of viewing data in real time as claimed in any one of claims 1 to 6.
9. A computer readable storage medium having stored thereon computer readable instructions which when executed by a processor implement the steps of the method of viewing data in real time as claimed in any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011052057.8A CN112182004B (en) | 2020-09-29 | 2020-09-29 | Method, device, computer equipment and storage medium for checking data in real time |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011052057.8A CN112182004B (en) | 2020-09-29 | 2020-09-29 | Method, device, computer equipment and storage medium for checking data in real time |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112182004A CN112182004A (en) | 2021-01-05 |
| CN112182004B true CN112182004B (en) | 2024-02-02 |
Family
ID=73947267
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011052057.8A Active CN112182004B (en) | 2020-09-29 | 2020-09-29 | Method, device, computer equipment and storage medium for checking data in real time |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112182004B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113051448A (en) * | 2021-03-17 | 2021-06-29 | 平安普惠企业管理有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113746912B (en) * | 2021-08-30 | 2023-12-01 | 浙江中控技术股份有限公司 | DCS monitoring system and method for acquiring DCS control data |
| CN114780584B (en) * | 2022-06-22 | 2022-09-02 | 云账户技术(天津)有限公司 | Multi-scene streaming data processing method, system, network equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107609156A (en) * | 2017-09-26 | 2018-01-19 | 微梦创科网络科技（中国）有限公司 | The method and device that a kind of page is built |
| CN109241477A (en) * | 2018-07-27 | 2019-01-18 | 沈文策 | A kind of Website page loading method, device, electronic equipment and storage medium |
| CN110851477A (en) * | 2019-10-16 | 2020-02-28 | 浙江大搜车软件技术有限公司 | Stream data processing method, stream data processing device, computer equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE102015218346A1 (en) * | 2015-09-24 | 2017-03-30 | Siemens Aktiengesellschaft | Method, computer program and system for visualizing data |
-
2020
- 2020-09-29 CN CN202011052057.8A patent/CN112182004B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107609156A (en) * | 2017-09-26 | 2018-01-19 | 微梦创科网络科技（中国）有限公司 | The method and device that a kind of page is built |
| CN109241477A (en) * | 2018-07-27 | 2019-01-18 | 沈文策 | A kind of Website page loading method, device, electronic equipment and storage medium |
| CN110851477A (en) * | 2019-10-16 | 2020-02-28 | 浙江大搜车软件技术有限公司 | Stream data processing method, stream data processing device, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 一种基于元数据静动态数据联合查询方法的研究与实现;曾艳梅;成长生;陆忠良;苏厚勤;;计算机应用与软件(第01期);全文 * |
| 基于Storm的车辆实时展示系统;张博;;电脑知识与技术(第31期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112182004A (en) | 2021-01-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10521404B2 (en) | Data transformations with metadata | |
| CN109997126B (en) | Event driven extraction, transformation, and loading (ETL) processing | |
| CN112182004B (en) | Method, device, computer equipment and storage medium for checking data in real time | |
| Das et al. | Big data analytics: A framework for unstructured data analysis | |
| CN110569657B (en) | Data access method, device, equipment and storage medium | |
| CN111666490A (en) | Information pushing method, device, equipment and storage medium based on kafka | |
| CN112199442B (en) | Method, device, computer equipment and storage medium for distributed batch downloading files | |
| US20210357461A1 (en) | Method, apparatus and storage medium for searching blockchain data | |
| CN109522751B (en) | Access right control method and device, electronic equipment and computer readable medium | |
| CN111046237A (en) | User behavior data processing method and device, electronic equipment and readable medium | |
| WO2022134776A1 (en) | Label-based anti-crawler method and apparatus, computer device, and storage medium | |
| CN113254445B (en) | Real-time data storage method, device, computer equipment and storage medium | |
| CN113010542B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN116450723A (en) | Data extraction method, device, computer equipment and storage medium | |
| CN116842012A (en) | Method, device, equipment and storage medium for storing Redis cluster in fragments | |
| CN115033574A (en) | Information generation method, information generation device, electronic device, and storage medium | |
| CN112270594B (en) | Salary data display method, device, computer equipment and storage medium | |
| CN114357280A (en) | Information pushing method and device, electronic equipment and computer readable medium | |
| CN114253922A (en) | Resource directory management method, resource management method, device, equipment and medium | |
| CN111143328A (en) | Agile business intelligent data construction method, system, equipment and storage medium | |
| CN113918996B (en) | Distributed data processing method, device, computer equipment and storage medium | |
| CN116775649A (en) | Data classified storage method and device, computer equipment and storage medium | |
| US10896193B2 (en) | Cache fetching of OLAP based data using client to client relationships and data encoding | |
| CN116932486A (en) | File generation method, device, computer equipment and storage medium | |
| CN115345667A (en) | Big data based card and ticket issuing and intercepting method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |