CN112182004A - Method and device for viewing data in real time, computer equipment and storage medium - Google Patents
Method and device for viewing data in real time, computer equipment and storage medium Download PDFInfo
- Publication number
- CN112182004A CN112182004A CN202011052057.8A CN202011052057A CN112182004A CN 112182004 A CN112182004 A CN 112182004A CN 202011052057 A CN202011052057 A CN 202011052057A CN 112182004 A CN112182004 A CN 112182004A
- Authority
- CN
- China
- Prior art keywords
- data
- preset
- static
- real time
- dynamic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000003068 static effect Effects 0.000 claims abstract description 122
- 238000012545 processing Methods 0.000 claims abstract description 77
- 238000012216 screening Methods 0.000 claims description 31
- 230000006870 function Effects 0.000 claims description 11
- 238000004806 packaging method and process Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000005111 flow chemistry technique Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 description 21
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003203 everyday effect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008520 organization Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000005538 encapsulation Methods 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the application belongs to the technical field of big data, is applied to the field of intelligent logistics, and relates to a method for viewing data in real time, which comprises the steps of acquiring an original data set in a database in real time, wherein the original data set comprises static data and dynamic data; storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval; processing the dynamic data based on preset streaming processing in real time to obtain intermediate data; and acquiring static data in a preset data cache in real time, assembling the static data and the intermediate data to obtain display data, and displaying the display data to a data billboard of a front-end page. The application also provides a device for checking the data in real time, computer equipment and a storage medium. Therein, the display data may be stored in a block chain. The application enhances the real-time performance of data and improves the processing efficiency of a computer.
Description
Technical Field
The present application relates to the field of big data technologies, and in particular, to a method and an apparatus for viewing data in real time, a computer device, and a storage medium.
Background
With the continuous and rapid development of computer technology, computers have been applied to various industries and play an irreplaceable role in life and work. The data generated by the computer can be stored in the database, and the computer can perform operations such as increasing, deleting, checking, modifying and the like on the data in the database.
At present, the database is frequently locked and table-locked, and the content in the database needs to be continuously read, so that the data can be checked in real time. When the data volume in the database is large, the computer cannot display the data content in time, so that the problem of poor user experience effect is caused.
Disclosure of Invention
The embodiment of the application aims to provide a method and a device for viewing data in real time, computer equipment and a storage medium, so that the real-time performance of the data is enhanced, and the processing efficiency of a computer is improved.
In order to solve the above technical problem, an embodiment of the present application provides a method for viewing data in real time, which adopts the following technical scheme:
a method for viewing data in real time comprises the following steps:
acquiring an original data set in a database in real time, wherein the original data set comprises static data and dynamic data;
storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval;
processing the dynamic data based on preset streaming processing in real time to obtain intermediate data;
and acquiring static data in a preset data cache in real time, assembling the static data and the intermediate data to obtain display data, and displaying the display data to a data billboard of a front-end page.
Further, the step of storing the static data in a preset data cache and updating the static data stored in the data cache based on a preset time interval includes:
identifying static data in the original data set based on a preset static data lookup table;
and storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval.
Further, the step of storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval includes:
storing the static data into a preset data cache;
acquiring static data in the original data set at the current time based on a preset time interval;
identifying whether the data cache stores static data or not;
and when the data cache stores the static data, deleting the static data in the data cache, and storing the static data acquired at the current time into the data cache.
Further, the step of processing the dynamic data in real time based on preset streaming processing to obtain intermediate data includes:
combining the dynamic data into a dynamic data set;
calling a stream conversion function in the stream processing to convert the dynamic data set into an aggregate stream;
acquiring a screening condition input or selected by a user on a front page;
adding the screening conditions to the screening function of the streaming processing, and screening the aggregate stream based on the screening function added with the screening conditions;
and packaging the screened aggregate flow through a collection function in the flow processing to generate the intermediate data.
Further, the step of processing the dynamic data in real time based on preset streaming processing to obtain intermediate data includes:
combining the dynamic data into a dynamic data set;
identifying a data volume of the dynamic data set;
when the data volume of the dynamic data set is larger than a preset threshold value, splitting the dynamic data set to obtain at least two different data sets;
and respectively carrying out streaming processing on the data sets through different preset nodes, and summarizing the processed data sets to obtain intermediate data.
Further, the step of splitting the dynamic data set to obtain at least two different data sets includes:
identifying a source attribute of each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on the source attribute to obtain at least two different data sets.
Further, the step of splitting the dynamic data set to obtain at least two different data sets includes:
identifying a timestamp carried by each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on a preset time period and a timestamp to obtain at least two different data sets.
In order to solve the above technical problem, an embodiment of the present application further provides a device for viewing data in real time, which adopts the following technical scheme:
a real-time viewing data device, comprising:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring an original data set in a database in real time, and the original data set comprises static data and dynamic data;
the updating module is used for storing the static data into a preset data cache and updating the static data stored in the data cache based on a preset time interval;
the processing module is used for processing the dynamic data based on preset streaming processing in real time to obtain intermediate data; and
and the assembling module is used for acquiring static data in a preset data cache in real time, assembling the static data and the intermediate data to obtain display data, and displaying the display data to a data billboard of a front-end page.
In order to solve the above technical problem, an embodiment of the present application further provides a computer device, which adopts the following technical solutions:
a computer device comprising a memory having computer readable instructions stored therein and a processor that when executed implements the steps of the method for viewing data in real time as described above.
In order to solve the above technical problem, an embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solutions:
a computer readable storage medium having computer readable instructions stored thereon which, when executed by a processor, implement the steps of the method for viewing data in real time as described above.
Compared with the prior art, the embodiment of the application mainly has the following beneficial effects:
according to the method and the device, the data are split into the static data and the dynamic data, the dynamic data are processed through stream processing, the calculation efficiency is improved, the real-time performance of the data is enhanced, the calculation efficiency is improved by using the stream processing and static data caching modes, and the calculation of mass data is facilitated. The display data is displayed in a data billboard of a front-end page, and related users such as business personnel can see the data in the database in the billboard in real time, so that the user experience is enhanced.
Drawings
In order to more clearly illustrate the solution of the present application, the drawings needed for describing the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;
FIG. 2 is a flow diagram of one embodiment of a method of viewing data in real time according to the present application;
FIG. 3 is a schematic block diagram of one embodiment of a real-time viewing data device according to the present application;
FIG. 4 is a schematic block diagram of one embodiment of a computer device according to the present application.
Reference numerals: 200. a computer device; 201. a memory; 202. a processor; 203. a network interface; 300. viewing the data device in real time; 301. an acquisition module; 302. an update module; 303. a processing module; 304. and assembling the module.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the above-described drawings are used for distinguishing between different objects and not for describing a particular order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have various communication client applications installed thereon, such as a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, mpeg compression standard Audio Layer 3), MP4 players (Moving Picture Experts Group Audio Layer IV, mpeg compression standard Audio Layer 4), laptop portable computers, desktop computers, and the like.
The server 105 may be a server providing various services, such as a background server providing support for pages displayed on the terminal devices 101, 102, 103.
It should be noted that, the method for viewing data in real time provided by the embodiment of the present application is generally executed by a server/terminal device, and accordingly, the apparatus for viewing data in real time is generally disposed in the server/terminal device.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to FIG. 2, a flow diagram of one embodiment of a method of viewing data in real time according to the present application is shown. The method for viewing data in real time comprises the following steps:
s1: the method comprises the steps of obtaining a raw data set in a database in real time, wherein the raw data set comprises static data and dynamic data.
In this embodiment, the data in the database is constantly updated randomly, and it is particularly important for the user to be able to obtain the data in the database in real time. The original data set in the database comprises various data tables, wherein the data tables of different types have an association relationship, and the association relationship is established by a computer according to specific keywords between the data tables. The specific keyword may be looked up from a preset keyword table. For example, financial statements and timetables belong to different types of data tables, while financial statement 1 and financial statement 2 belong to the same type but different data tables. Financial statement 1, financial statement 2 and attendance table pass through specific keywords, such as: and establishing an association relationship by Shanghai subsidiary company. In any data table, static data and dynamic data are included. Such as: the static data in the financial statement 1 are a personnel field, an organization name field and the like, and the dynamic data are expenditure amount, total balance and the like. For insurance companies, the statistics of the number of policies change in real time, and any delayed message may cause a series of adverse effects. In the data table, both static data and dynamic data have their fixed locations.
In this embodiment, the electronic device (for example, the server/terminal device shown in fig. 1) on which the real-time data viewing method operates may obtain the original data set in the database through a wired connection manner or a wireless connection manner. It should be noted that the wireless connection means may include, but is not limited to, a 3G/4G connection, a WiFi connection, a bluetooth connection, a WiMAX connection, a Zigbee connection, a uwb (ultra wideband) connection, and other wireless connection means now known or developed in the future.
S2: and storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval.
In this embodiment, the static data is determined according to the position of the data in the data table, and generally, the data in the left column in the data table is the static data, and the data in the right column in the data table is the dynamic data. Of course, which type of data in the data table is static data may be correspondingly determined according to the static data lookup table. And after the static data are determined, storing the static data into a preset data cache. Because the static data can not be changed in real time, the static data is not required to be processed in real time, and the static data is independently stored and called, so that the data volume of the subsequent processing data of the computer is reduced, the processing speed of the computer is effectively improved, and the subsequent real-time processing and data display are realized. And after a preset time interval, storing the static data in the original data set acquired in real time into the cache again to update the static data in the cache.
Specifically, the step of storing the static data in a preset data cache and updating the static data stored in the data cache based on a preset time interval includes:
identifying static data in the original data set based on a preset static data lookup table;
and storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval.
In this embodiment, the data cache in the present application refers to a redis cache. Static data can be used as cache data, such as second-level organization information and staff information; for different business scenes, the organization data cannot change every day, and the personnel data may change every day, but for the whole business, if the latest data is refreshed to the cache every day, the data is acceptable at the business level, and the data like the data can be stored to the redis in advance for caching, so that the data can be read quickly when being used in the billboard.
The step of storing the static data into a preset data cache and updating the static data stored in the data cache based on a preset time interval includes:
storing the static data into a preset data cache;
acquiring static data in the original data set at the current time based on a preset time interval;
identifying whether the data cache stores static data or not;
and when the data cache stores the static data, deleting the static data in the data cache, and storing the static data acquired at the current time into the data cache.
In this embodiment, when the data cache does not store the static data, the static data acquired at the current time may be directly stored in the data cache. And continuously updating the static data in the data cache through a preset time interval. If the data cache stores static data, the static data is deleted, so that subsequent data redundancy and confusion are avoided.
S3: and processing the dynamic data based on preset streaming processing in real time to obtain intermediate data.
In this embodiment, the traffic data required by the current kanban is collected through streaming processing. The streaming processing simplifies the operation on the collection, and comprises three parts of conversion into streaming, intermediate operation and terminal operation. The stream processing directly converts the data source into stream, including aggregate stream, array stream, file stream, etc.; after the stream is converted, performing intermediate operation, wherein the intermediate operation comprises operations such as filter, limit, distict and the like; and after the intermediate operation is processed, finally performing terminal operation, wherein the terminal operation comprises operations such as allMatvh, anyMatch, Collect and the like, and summarizing and returning the final result set according to preset requirements.
It should be noted that, in the present application, for a service with a huge data volume, Spark may also be introduced in the streaming processing to perform calculation, for example, a Spark cluster may be separately deployed, and data used by the service is specially calculated and watched. The advantage of Spark calculation is: 1) memory-based computation; 2) can be directly allowed to run on an HDFS (easy-to-extend distributed file system); 3) data can be stored in both disk and memory.
Specifically, the step of processing the dynamic data in real time based on preset streaming processing to obtain intermediate data includes:
combining the dynamic data into a dynamic data set;
calling a stream conversion function (stream) in the streaming processing, and converting a dynamic data set into an aggregate stream;
acquiring a screening condition input or selected by a user on a front page;
adding the screening conditions into a screening function (filter) of the streaming processing, and screening the aggregate stream based on the filter added with the screening conditions;
and packaging the filtered set stream through a collection function (collect) in the stream processing to generate a list set as the intermediate data.
In this embodiment, after data is captured, data is converted into an aggregate stream by a stream method of streaming processing. In the screening process, as for the data tables having the association relationship, as long as the dynamic data corresponding to one of the data tables meets the screening condition, the dynamic data corresponding to the data table having the association relationship is also screened, and the following two examples are provided for illustrating:
the first embodiment is as follows: and for the financial statement and the attendance table with the association relation, the dynamic data corresponding to the attendance table with the association relation is also screened and enters the subsequent steps as long as the dynamic data corresponding to the financial statement and the attendance table meet the screening condition.
Example two: the data source of the present application may be raw unconditional warranty data. Different data are dispersed in different insurance data tables, and the different insurance data tables form an original data set. And screening the original data set according to preset screening conditions, wherein different insurance data tables have an incidence relation, and if the dynamic data corresponding to one insurance data table meets the screening conditions, the dynamic data corresponding to the insurance data table associated with the insurance data table is screened as well, and the subsequent steps are carried out.
The flow type processing efficiency is doubled compared with the traditional iteration mode when the data volume is more than ten thousand, the calculation time is shortened by multiple, if the machine has a multi-core CPU, the flow type processing can be processed in parallel, the calculation efficiency is further improved, and finally the effect of displaying the service data in real time can be achieved; compared with the traditional oracle database computing task, the flexibility and the computing efficiency are greatly improved.
After the data are filtered according to the business rules, the data are packaged into a collection and the returned list set is appointed, namely the data are assembled into the list set form. The specific filtration conditions are as follows: the screening condition is that the insurance policy is handed (namely the insurance premium is handed) for more than 4 times, the service personnel of the insurance policy is regional extension personnel, and the state of the insurance policy is an orphan, wherein the long-term insurance policy of the off-duty businessman is called the orphan. The filter screening conditions are added in the intermediate step of the streaming process.
In addition, the step of processing the dynamic data based on preset streaming processing in real time to obtain intermediate data includes:
combining the dynamic data into a dynamic data set;
identifying a data volume of the dynamic data set;
when the data volume of the dynamic data set is larger than a preset threshold value, splitting the dynamic data set to obtain at least two different data sets;
and respectively carrying out streaming processing on the data sets through different preset nodes, and summarizing the processed data sets to obtain intermediate data.
In this embodiment, when the data amount of the dynamic data set is greater than the preset threshold, the dynamic data set is directly subjected to the streaming processing process without splitting. And the dynamic data is generated into a dynamic data set, so that the subsequent streaming processing of the dynamic data and the management of the dynamic data are facilitated. The method has the advantages that the streaming processing is adopted on a single node, the operation is automatically switched into a plurality of nodes to allow when the data volume is large, the original data is split firstly, the distributed nodes are used for processing the service data in a dispersing mode, the split data is summarized after being split, the processing speed of the data is improved, and the situations that the computer is down or is blocked in processing and the like due to the fact that the data volume is too large are avoided.
Wherein the step of splitting the dynamic data set to obtain at least two different data sets comprises:
identifying a source attribute of each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on the source attribute to obtain at least two different data sets.
In this embodiment, the splitting principle may be splitting according to regions. The region is used as a source attribute of the dynamic data set, such as the data of the Guangzhou library is allowed on the node 1; the data of the Beijing library is allowed on node 2. And after the data are calculated at the nodes 1 and 2, data summarization is carried out, and the data are displayed on a data billboard in real time.
Of course, the step of splitting the dynamic data set to obtain at least two different data sets may also include:
identifying a timestamp carried by each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on a preset time period and a timestamp to obtain at least two different data sets.
In this embodiment, the splitting principle may also be split according to the time latitude. The dynamic data set is derived from a raw data set, wherein the raw data set is obtained from a database. The original data set carries the time stamp, each piece of dynamic data in the dynamic data set also carries the time stamp, and the dynamic data set can be split according to different time periods, so that the dynamic data set can be split based on the time latitude.
S4: and acquiring static data in a preset data cache in real time, assembling the static data and the intermediate data to obtain display data, and displaying the display data to a data billboard of a front-end page.
In this embodiment, the static data in the cache is obtained, the intermediate data and the static data are placed according to a preset fixed position, after the placement, whether the position of the dynamic data corresponding to the static data is empty is identified, if the position of the dynamic data is empty, the static data is deleted, data assembly is finally completed, display data is formed, the display data is displayed in a billboard of a front-end page, and a user can conveniently check the data in real time.
According to the method and the device, the data are split into the static data and the dynamic data, the dynamic data are processed through stream processing, the calculation efficiency is improved, the real-time performance of the data is enhanced, the calculation efficiency is improved by using the stream processing and static data caching modes, and the calculation of mass data is facilitated. The display data is displayed in a data billboard of a front-end page, and related users such as business personnel can see the data in the database in the billboard in real time, so that the user experience is enhanced.
It is emphasized that the presentation data may also be stored in a node of a blockchain in order to further ensure privacy and security of the presentation data.
The block chain referred by the application is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
This application can be applied to in the wisdom commodity circulation field to promote the construction in wisdom city, because the commodity circulation field has more relevance tables, and mostly need look over in real time, confirms information such as order state, amount of money change condition, application this application can effectively improve commodity circulation staff's work efficiency.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware associated with computer readable instructions, which can be stored in a computer readable storage medium, and when executed, can include processes of the embodiments of the methods described above. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
With further reference to fig. 3, as an implementation of the method shown in fig. 2, the present application provides an embodiment of a device for viewing data in real time, where the embodiment of the device corresponds to the embodiment of the method shown in fig. 2, and the device may be applied to various electronic devices.
As shown in fig. 3, the real-time viewing data apparatus 300 according to the present embodiment includes: an acquisition module 301, an update module 302, a processing module 303, and an assembly module 304. The acquiring module 301 is configured to acquire an original data set in a database in real time, where the original data set includes static data and dynamic data; an updating module 302, configured to store the static data in a preset data cache, and update the static data stored in the data cache based on a preset time interval; a processing module 303, configured to process the dynamic data in real time based on preset streaming processing to obtain intermediate data; and an assembling module 304, configured to obtain static data in a preset data cache in real time, assemble the static data and the intermediate data to obtain display data, and display the display data in a data billboard of a front-end page.
In this embodiment, the data is split into static data and dynamic data, and the dynamic data is processed by streaming processing to improve the calculation efficiency, so that the real-time performance of the data is enhanced, the calculation efficiency is improved by using the streaming processing and the static data caching, and the calculation of mass data is facilitated. The display data is displayed in a data billboard of a front-end page, and related users such as business personnel can see the data in the database in the billboard in real time, so that the user experience is enhanced.
The update module 302 includes an identification submodule and an update submodule. The identification submodule is used for identifying static data in the original data set based on a preset static data lookup table; the updating submodule is used for storing the static data into a preset data cache and updating the static data stored in the data cache based on a preset time interval.
The updating submodule comprises a storage unit, an acquisition unit, an identification unit and a deletion unit. The storage unit is used for storing the static data into a preset data cache; the acquisition unit is used for acquiring static data in the original data set at the current time based on a preset time interval; the identification unit is used for identifying whether the data cache stores static data or not; the deleting unit is used for deleting the static data in the data cache when the static data is stored in the data cache, and storing the static data acquired at the current time into the data cache.
The processing module 303 includes a generation sub-module, an acquisition sub-module, a screening sub-module, and an encapsulation sub-module. The generation submodule is used for combining the dynamic data into a dynamic data set; the acquisition sub-module is used for calling the stream in the stream processing and converting the dynamic data set into an aggregate stream; the screening submodule is used for acquiring screening conditions input or selected by a user on a front-end page; the encapsulation submodule is used for adding the screening condition into the filter for the streaming processing and screening the aggregate flow based on the filter added with the screening condition; and the packaging submodule is used for packaging the filtered aggregate flow through the collection in the flow processing to generate the intermediate data.
The processing module 303 further includes a collection submodule, a quantity submodule, a splitting submodule, and a summarization submodule. The collection submodule is used for combining the dynamic data into a dynamic data collection; the quantity submodule is used for identifying the data quantity of the dynamic data set; the splitting submodule is used for splitting the dynamic data set when the data volume of the dynamic data set is larger than a preset threshold value to obtain at least two different data sets; and the summarizing submodule is used for respectively carrying out streaming processing on the data set through different preset nodes and summarizing the processed data set to obtain intermediate data.
The splitting submodule comprises an attribute unit and a first splitting unit. The attribute unit is used for identifying the source attribute of each piece of dynamic data in the dynamic data set; the first splitting unit splits the dynamic data set based on the source attribute to obtain at least two different data sets.
The splitting submodule further comprises a time unit and a second splitting unit. The time unit is used for identifying a timestamp carried by each piece of dynamic data in the dynamic data set; the second splitting unit is used for splitting the dynamic data set based on a preset time period and a timestamp to obtain at least two different data sets.
According to the method and the device, the data are split into the static data and the dynamic data, the dynamic data are processed through stream processing, the calculation efficiency is improved, the real-time performance of the data is enhanced, the calculation efficiency is improved by using the stream processing and static data caching modes, and the calculation of mass data is facilitated. The display data is displayed in a data billboard of a front-end page, and related users such as business personnel can see the data in the database in the billboard in real time, so that the user experience is enhanced.
In order to solve the technical problem, an embodiment of the present application further provides a computer device. Referring to fig. 4, fig. 4 is a block diagram of a basic structure of a computer device according to the present embodiment.
The computer device 200 comprises a memory 201, a processor 202, a network interface 203 communicatively connected to each other via a system bus. It is noted that only computer device 200 having components 201 and 203 is shown, but it is understood that not all of the illustrated components are required and that more or fewer components may alternatively be implemented. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.
The memory 201 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 201 may be an internal storage unit of the computer device 200, such as a hard disk or a memory of the computer device 200. In other embodiments, the memory 201 may also be an external storage device of the computer device 200, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the computer device 200. Of course, the memory 201 may also include both internal and external storage devices of the computer device 200. In this embodiment, the memory 201 is generally used for storing an operating system and various application software installed on the computer device 200, such as computer readable instructions of a real-time data viewing method. Further, the memory 201 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 202 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 202 is generally operative to control overall operation of the computer device 200. In this embodiment, the processor 202 is configured to execute the computer readable instructions stored in the memory 201 or process data, for example, execute the computer readable instructions of the real-time data viewing method.
The network interface 203 may comprise a wireless network interface or a wired network interface, and the network interface 203 is generally used for establishing communication connection between the computer device 200 and other electronic devices.
In this embodiment, the real-time performance of the data is enhanced, the calculation efficiency is improved by using a streaming processing and static data caching mode, and the calculation of mass data is facilitated.
The present application further provides another embodiment, which is to provide a computer-readable storage medium storing computer-readable instructions executable by at least one processor to cause the at least one processor to perform the steps of the method for viewing data in real time as described above.
In this embodiment, the real-time performance of the data is enhanced, the calculation efficiency is improved by using a streaming processing and static data caching mode, and the calculation of mass data is facilitated.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.
It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields and are within the protection scope of the present application.
Claims (10)
1. A method for viewing data in real time is characterized by comprising the following steps:
acquiring an original data set in a database in real time, wherein the original data set comprises static data and dynamic data;
storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval;
processing the dynamic data based on preset streaming processing in real time to obtain intermediate data;
and acquiring static data in a preset data cache in real time, assembling the static data and the intermediate data to obtain display data, and displaying the display data to a data billboard of a front-end page.
2. A method for viewing data in real time as claimed in claim 1, wherein the step of storing the static data in a preset data buffer and updating the static data stored in the data buffer based on a preset time interval comprises:
identifying static data in the original data set based on a preset static data lookup table;
and storing the static data into a preset data cache, and updating the static data stored in the data cache based on a preset time interval.
3. A method for viewing data in real time as claimed in claim 2, wherein the step of storing the static data in a preset data buffer and updating the static data stored in the data buffer based on a preset time interval comprises:
storing the static data into a preset data cache;
acquiring static data in the original data set at the current time based on a preset time interval;
identifying whether the data cache stores static data or not;
and when the data cache stores the static data, deleting the static data in the data cache, and storing the static data acquired at the current time into the data cache.
4. A method for viewing data in real time as claimed in claim 1, wherein the step of processing the dynamic data in real time based on preset streaming processing to obtain intermediate data comprises:
combining the dynamic data into a dynamic data set;
calling a stream conversion function in the stream processing to convert the dynamic data set into an aggregate stream;
acquiring a screening condition input or selected by a user on a front page;
adding the screening conditions to the screening function of the streaming processing, and screening the aggregate stream based on the screening function added with the screening conditions;
and packaging the screened aggregate flow through a collection function in the flow processing to generate the intermediate data.
5. A method for viewing data in real time as claimed in claim 1, wherein the step of processing the dynamic data in real time based on preset streaming processing to obtain intermediate data comprises:
combining the dynamic data into a dynamic data set;
identifying a data volume of the dynamic data set;
when the data volume of the dynamic data set is larger than a preset threshold value, splitting the dynamic data set to obtain at least two different data sets;
and respectively carrying out streaming processing on the data sets through different preset nodes, and summarizing the processed data sets to obtain intermediate data.
6. A method for viewing data in real time as claimed in claim 5 wherein said step of splitting said dynamic data set to obtain at least two different data sets comprises:
identifying a source attribute of each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on the source attribute to obtain at least two different data sets.
7. A method for viewing data in real time as claimed in claim 5 wherein said step of splitting said dynamic data set to obtain at least two different data sets comprises:
identifying a timestamp carried by each piece of dynamic data in the dynamic data set;
and splitting the dynamic data set based on a preset time period and a timestamp to obtain at least two different data sets.
8. An apparatus for viewing data in real time, comprising:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring an original data set in a database in real time, and the original data set comprises static data and dynamic data;
the updating module is used for storing the static data into a preset data cache and updating the static data stored in the data cache based on a preset time interval;
the processing module is used for processing the dynamic data based on preset streaming processing in real time to obtain intermediate data; and
and the assembling module is used for acquiring static data in a preset data cache in real time, assembling the static data and the intermediate data to obtain display data, and displaying the display data to a data billboard of a front-end page.
9. A computer device comprising a memory having computer readable instructions stored therein and a processor that when executed implements the steps of the method of viewing data in real time as claimed in any one of claims 1 to 7.
10. A computer readable storage medium having computer readable instructions stored thereon which, when executed by a processor, implement the steps of the method of viewing data in real time as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011052057.8A CN112182004B (en) | 2020-09-29 | 2020-09-29 | Method, device, computer equipment and storage medium for checking data in real time |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011052057.8A CN112182004B (en) | 2020-09-29 | 2020-09-29 | Method, device, computer equipment and storage medium for checking data in real time |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112182004A true CN112182004A (en) | 2021-01-05 |
| CN112182004B CN112182004B (en) | 2024-02-02 |
Family
ID=73947267
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011052057.8A Active CN112182004B (en) | 2020-09-29 | 2020-09-29 | Method, device, computer equipment and storage medium for checking data in real time |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112182004B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113051448A (en) * | 2021-03-17 | 2021-06-29 | 平安普惠企业管理有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114780584A (en) * | 2022-06-22 | 2022-07-22 | 云账户技术(天津)有限公司 | Multi-scene streaming data processing method, system, network equipment and storage medium |
| CN113746912B (en) * | 2021-08-30 | 2023-12-01 | 浙江中控技术股份有限公司 | DCS monitoring system and method for acquiring DCS control data |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107609156A (en) * | 2017-09-26 | 2018-01-19 | 微梦创科网络科技（中国）有限公司 | The method and device that a kind of page is built |
| US20180288131A1 (en) * | 2015-09-24 | 2018-10-04 | Siemens Aktiengesellschaft | Method, Computer Program and System for Transmitting Data in Order to Produce an Interactive Image |
| CN109241477A (en) * | 2018-07-27 | 2019-01-18 | 沈文策 | A kind of Website page loading method, device, electronic equipment and storage medium |
| CN110851477A (en) * | 2019-10-16 | 2020-02-28 | 浙江大搜车软件技术有限公司 | Stream data processing method, stream data processing device, computer equipment and storage medium |
-
2020
- 2020-09-29 CN CN202011052057.8A patent/CN112182004B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180288131A1 (en) * | 2015-09-24 | 2018-10-04 | Siemens Aktiengesellschaft | Method, Computer Program and System for Transmitting Data in Order to Produce an Interactive Image |
| CN107609156A (en) * | 2017-09-26 | 2018-01-19 | 微梦创科网络科技（中国）有限公司 | The method and device that a kind of page is built |
| CN109241477A (en) * | 2018-07-27 | 2019-01-18 | 沈文策 | A kind of Website page loading method, device, electronic equipment and storage medium |
| CN110851477A (en) * | 2019-10-16 | 2020-02-28 | 浙江大搜车软件技术有限公司 | Stream data processing method, stream data processing device, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 张博;: "基于Storm的车辆实时展示系统", 电脑知识与技术, no. 31 * |
| 曾艳梅;成长生;陆忠良;苏厚勤;: "一种基于元数据静动态数据联合查询方法的研究与实现", 计算机应用与软件, no. 01 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113051448A (en) * | 2021-03-17 | 2021-06-29 | 平安普惠企业管理有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113746912B (en) * | 2021-08-30 | 2023-12-01 | 浙江中控技术股份有限公司 | DCS monitoring system and method for acquiring DCS control data |
| CN114780584A (en) * | 2022-06-22 | 2022-07-22 | 云账户技术(天津)有限公司 | Multi-scene streaming data processing method, system, network equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112182004B (en) | 2024-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Das et al. | Big data analytics: A framework for unstructured data analysis | |
| CN112182004B (en) | Method, device, computer equipment and storage medium for checking data in real time | |
| CN110225095B (en) | Data processing method, device and system | |
| CN111666490A (en) | Information pushing method, device, equipment and storage medium based on kafka | |
| CN111046237B (en) | User behavior data processing method and device, electronic equipment and readable medium | |
| CN109299174B (en) | Multi-source information data aggregation processing method and device | |
| CN110020273B (en) | Method, device and system for generating thermodynamic diagram | |
| CN112199442B (en) | Method, device, computer equipment and storage medium for distributed batch downloading files | |
| CN113254445B (en) | Real-time data storage method, device, computer equipment and storage medium | |
| CN103365971A (en) | Mass data access processing system based on cloud computing | |
| CN113282611B (en) | Method, device, computer equipment and storage medium for synchronizing stream data | |
| CN111680477A (en) | Method and device for exporting spreadsheet file, computer equipment and storage medium | |
| CN112184169A (en) | Dynamic planning method, device, equipment and storage medium for user backlogs | |
| WO2022134776A1 (en) | Label-based anti-crawler method and apparatus, computer device, and storage medium | |
| CN113010542B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN112270594B (en) | Salary data display method, device, computer equipment and storage medium | |
| CN116842012A (en) | Method, device, equipment and storage medium for storing Redis cluster in fragments | |
| CN111143328A (en) | Agile business intelligent data construction method, system, equipment and storage medium | |
| CN113283759A (en) | Account risk portrait updating method, device, equipment and storage medium | |
| CN113626438A (en) | Data table management method and device, computer equipment and storage medium | |
| Shao et al. | Optimization research of information management system based on big data technology | |
| CN111752944B (en) | Data allocation method, device, computer equipment and storage medium | |
| CN112084281B (en) | Method, device, computer equipment and storage medium for drawing survey grid | |
| CN114448976B (en) | Method, device, equipment, medium and program product for assembling network message | |
| CN116992462A (en) | Plaintext information offline method based on big data transmission scene and related equipment thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |