CN113157747A - Data service method and device - Google Patents
Data service method and device Download PDFInfo
- Publication number
- CN113157747A CN113157747A CN202110480402.6A CN202110480402A CN113157747A CN 113157747 A CN113157747 A CN 113157747A CN 202110480402 A CN202110480402 A CN 202110480402A CN 113157747 A CN113157747 A CN 113157747A
- Authority
- CN
- China
- Prior art keywords
- data
- streaming
- requirement
- filtering
- requirements
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000001914 filtration Methods 0.000 claims description 35
- 238000004590 computer program Methods 0.000 claims description 17
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 3
- 238000004458 analytical method Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a data service method and a device, which can be used in the field of big data, and the method comprises the following steps: acquiring the data requirement of a target object; reading streaming data meeting the data requirement from the kafka; processing the streaming data according to data requirements; pushing the processed streaming data to the front end in a websocket mode; offline data meeting the data requirements are inquired from a database and cached to the front end; integrating the processed streaming data and the inquired offline data at the front end to obtain integrated data; and displaying the integrated data at the front end. The invention can provide data service to the front end in real time and has high efficiency.
Description
Technical Field
The invention relates to the field of big data, in particular to a data service method and device.
Background
The traditional data service mode is to inquire from a database and then display the data through a front-end page, but part of data may not be stored in the database in time of inquiry, and the result displayed on the page does not contain the latest data and does not meet the requirement of displaying the data change condition in real time at the front end. On the other hand, the data volume after the streaming processing is large, the data quality cannot meet the requirement of direct use of the front end, further preprocessing is carried out before the data is transmitted to the front end according to the requirement, the data volume is reduced, a small amount of data required by the front end is screened out, the streaming data is not sequenced according to the time sequence, and the stability and the accuracy are not better than those of the offline data during display. Therefore, there is a need for a service that can provide data to a head-end in real-time.
Disclosure of Invention
The embodiment of the invention provides a data service method which is used for providing data service to a front end in real time and has high efficiency, and the method comprises the following steps:
acquiring the data requirement of a target object;
reading streaming data meeting the data requirement from the kafka;
processing the streaming data according to data requirements;
pushing the processed streaming data to the front end in a websocket mode;
offline data meeting the data requirements are inquired from a database and cached to the front end;
integrating the processed streaming data and the inquired offline data at the front end to obtain integrated data;
and displaying the integrated data at the front end.
The embodiment of the invention provides a data service device, which is used for providing data service to a front end in real time and has high efficiency, and the device comprises:
the data requirement acquisition module is used for acquiring the data requirement of the target object;
the streaming data reading module is used for reading streaming data meeting the data requirement from the kafka;
the data processing module is used for processing the streaming data according to data requirements;
the data pushing module is used for pushing the processed streaming data to the front end in a websocket mode;
the data query module is used for querying offline data meeting data requirements from a database and caching the offline data to the front end;
the integration module is used for integrating the processed streaming data and the inquired offline data at the front end to obtain integrated data;
and the display module is used for displaying the integrated data at the front end.
The embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the computer program, the data service method is implemented.
An embodiment of the present invention further provides a computer-readable storage medium, where a computer program for executing the data service method is stored in the computer-readable storage medium.
In the embodiment of the invention, the data requirement of the target object is acquired; reading streaming data meeting the data requirement from the kafka; processing the streaming data according to data requirements; pushing the processed streaming data to the front end in a websocket mode; offline data meeting the data requirements are inquired from a database and cached to the front end; integrating the processed streaming data and the inquired offline data at the front end to obtain integrated data; and displaying the integrated data at the front end. In the process, the streaming data are processed before front-end display, so that data flow is reduced, and pushing is more efficient and accurate. The overall efficiency is improved. Meanwhile, by inquiring the offline data, the requirement on real-time performance is met, and the requirement on checking the offline data in the database is also met.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
FIG. 1 is a flow chart of a data service method in an embodiment of the present invention;
FIG. 2 is a detailed flowchart of a data service method according to an embodiment of the present invention;
FIG. 3 is a diagram of a data service device according to an embodiment of the present invention;
FIG. 4 is a diagram of a data processing module according to an embodiment of the present invention;
FIG. 5 is a diagram of a computer device in an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
In the description of the present specification, the terms "comprising," "including," "having," "containing," and the like are used in an open-ended fashion, i.e., to mean including, but not limited to. Reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," etc., means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The sequence of steps involved in the embodiments is for illustrative purposes to illustrate the implementation of the present application, and the sequence of steps is not limited and can be adjusted as needed.
Fig. 1 is a flowchart of a data service method in an embodiment of the present invention, and as shown in fig. 1, the method includes:
103, processing the streaming data according to data requirements;
104, pushing the processed streaming data to the front end in a websocket mode;
and step 107, displaying the integrated data at the front end.
In the embodiment of the invention, the streaming data is processed before the front-end display, so that the data stream is reduced, and the pushing is more efficient and accurate. The overall efficiency is improved. Meanwhile, by inquiring the offline data, the requirement on real-time performance is met, and the requirement on checking the offline data in the database is also met.
In one embodiment, acquiring data requirements includes:
reading subscription configuration information of a target object in a database;
and obtaining the data requirement of the target object from the subscription configuration information.
In the above embodiment, the target object may be any object to be exposed at the front end, and before reading the subscription configuration information, it is required to initialize the WebSocket service in the Spring starting process, where the subscription configuration information is configured according to Topic based on the concept of kafka, and the subscription configuration information of different target objects is generally configured by the user at the front end (browser), and then, the user id is used as the target object, and the subscription configuration information is stored in the database.
In one embodiment, reading streaming data from kafka that meets data requirements includes:
streaming data satisfying the data requirements is read from kafka in a round-robin fashion.
In the above embodiment, the cycle period may be preset or default, and the cycle reading of the streaming data ensures real-time acquisition of the data.
In an embodiment, processing the streaming data according to a data requirement in a data requirement includes:
determining a data filtering strategy and a data combination method according to the data requirement in the data requirement, wherein the data filtering strategy comprises one or any combination of filtering according to keywords, filtering according to whether the calculation time is latest or not and filtering according to the maximum value;
filtering the streaming data according to a data filtering strategy;
and combining the filtered streaming data according to a data combination method.
In the above embodiment, the data filtering is to filter out unnecessary data in the data requirement, reduce the data pushing amount, improve the efficiency, and the data filtering policy may be preconfigured. The combination method comprises the combination processing of addition, subtraction and the like of two data, and generally is a data requirement analyzed according to the requirement to be displayed at the front end, so that the data combination method is determined.
The processed streaming data is pushed to the front end in a websocket mode, so that the server does not passively return the data after receiving the request of the front end, but actively pushes the data to the front end when new data exists.
When offline data meeting the data requirements are queried from a database, the query can be initiated in the form of a call interface.
Based on the above embodiments, fig. 2 is a detailed flowchart of a data service method in the embodiment of the present invention, which includes:
and step 210, displaying the integrated data at the front end.
In summary, in the method provided in the embodiment of the present invention, the data requirement of the target object is obtained; reading streaming data meeting the data requirement from the kafka; processing the streaming data according to data requirements; pushing the processed streaming data to the front end in a websocket mode; offline data meeting the data requirements are inquired from a database and cached to the front end; integrating the processed streaming data and the inquired offline data at the front end to obtain integrated data; and displaying the integrated data at the front end. In the process, the streaming data are processed before front-end display, so that data flow is reduced, and pushing is more efficient and accurate. The overall efficiency is improved. Meanwhile, by inquiring the offline data, the requirement on real-time performance is met, and the requirement on checking the offline data in the database is also met.
The embodiment of the invention also provides a data service device, the principle of which is similar to that of the data service method, and the description is omitted here.
Fig. 3 is a schematic diagram of a data service apparatus according to an embodiment of the present invention, as shown in fig. 3, the apparatus includes:
a data requirement obtaining module 301, configured to obtain a data requirement of a target object;
a streaming data reading module 302, configured to read streaming data that meets the data requirement from kafka;
a data processing module 303, configured to process the streaming data according to a data requirement;
the data pushing module 304 is configured to push the processed streaming data to the front end in a websocket manner;
a data caching module 305, configured to query offline data meeting data requirements from a database, and cache the offline data to a front end;
an integration module 306, configured to integrate the processed streaming data and the queried offline data at the front end to obtain integrated data;
and a display module 307, configured to display the integrated data at the front end.
In an embodiment, the data requirement acquisition module is specifically configured to:
reading subscription configuration information of a target object in a database;
and obtaining the data requirement of the target object from the subscription configuration information.
In an embodiment, the streaming data reading module is specifically configured to:
streaming data satisfying the data requirements is read from kafka in a round-robin fashion.
Fig. 4 is a schematic diagram of a data processing module according to an embodiment of the present invention, as shown in fig. 4, in an embodiment, the data processing module includes:
the data requirement analysis module 401 is configured to determine a data filtering policy and a data combination method according to a data requirement in a data requirement, where the data filtering policy includes one or any combination of filtering according to a keyword, filtering according to whether the calculation time is the latest, and filtering according to a maximum value;
a filtering processing module 402, configured to perform filtering processing on the streaming data according to a data filtering policy;
a combining processing module 403, configured to combine the filtered streaming data according to a data combining method.
In summary, in the apparatus provided in the embodiment of the present invention, the data requirement of the target object is obtained; reading streaming data meeting the data requirement from the kafka; processing the streaming data according to data requirements; pushing the processed streaming data to the front end in a websocket mode; offline data meeting the data requirements are inquired from a database and cached to the front end; integrating the processed streaming data and the inquired offline data at the front end to obtain integrated data; and displaying the integrated data at the front end. In the process, the streaming data are processed before front-end display, so that data flow is reduced, and pushing is more efficient and accurate. The overall efficiency is improved. Meanwhile, by inquiring the offline data, the requirement on real-time performance is met, and the requirement on checking the offline data in the database is also met.
An embodiment of the present invention further provides a computer device, and fig. 5 is a schematic diagram of a computer device in an embodiment of the present invention, where the computer device is capable of implementing all steps in the data service method in the foregoing embodiment, and the computer device specifically includes the following contents:
a processor (processor)501, a memory (memory)502, a communication Interface (Communications Interface)503, and a communication bus 504;
the processor 501, the memory 502 and the communication interface 503 complete mutual communication through the communication bus 504; the communication interface 503 is used for implementing information transmission between related devices such as server-side devices, detection devices, and user-side devices;
the processor 501 is used to call the computer program in the memory 502, and when the processor executes the computer program, the processor implements all the steps of the data service method in the above embodiments.
An embodiment of the present invention further provides a computer-readable storage medium, which can implement all the steps in the data service method in the above embodiment, wherein the computer-readable storage medium stores a computer program, and the computer program, when executed by a processor, implements all the steps of the data service method in the above embodiment.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A data service method, comprising:
acquiring the data requirement of a target object;
reading streaming data meeting the data requirement from the kafka;
processing the streaming data according to data requirements;
pushing the processed streaming data to the front end in a websocket mode;
offline data meeting the data requirements are inquired from a database and cached to the front end;
integrating the processed streaming data and the inquired offline data at the front end to obtain integrated data;
and displaying the integrated data at the front end.
2. The data service method of claim 1, wherein obtaining data requirements comprises:
reading subscription configuration information of a target object in a database;
and obtaining the data requirement of the target object from the subscription configuration information.
3. The data service method of claim 1, wherein reading streaming data that meets data requirements from kafka comprises:
streaming data satisfying the data requirements is read from kafka in a round-robin fashion.
4. The data service method of claim 1, wherein processing the streaming data according to the data requirements in the data requirements comprises:
determining a data filtering strategy and a data combination method according to the data requirement in the data requirement, wherein the data filtering strategy comprises one or any combination of filtering according to keywords, filtering according to whether the calculation time is latest or not and filtering according to the maximum value;
filtering the streaming data according to a data filtering strategy;
and combining the filtered streaming data according to a data combination method.
5. A data service apparatus, comprising:
the data requirement acquisition module is used for acquiring the data requirement of the target object;
the streaming data reading module is used for reading streaming data meeting the data requirement from the kafka;
the data processing module is used for processing the streaming data according to data requirements;
the data pushing module is used for pushing the processed streaming data to the front end in a websocket mode;
the data query module is used for querying offline data meeting data requirements from a database and caching the offline data to the front end;
the integration module is used for integrating the processed streaming data and the inquired offline data at the front end to obtain integrated data;
and the display module is used for displaying the integrated data at the front end.
6. The data service device of claim 5, wherein the data requirement acquisition module is specifically configured to:
reading subscription configuration information of a target object in a database;
and obtaining the data requirement of the target object from the subscription configuration information.
7. The data service apparatus of claim 5, wherein the streaming data reading module is specifically configured to:
streaming data satisfying the data requirements is read from kafka in a round-robin fashion.
8. The data service apparatus of claim 5, wherein the data processing module comprises:
the data requirement analysis module is used for determining a data filtering strategy and a data combination method according to the data requirement in the data requirement, wherein the data filtering strategy comprises one or any combination of filtering according to keywords, filtering according to whether the calculation time is latest or not and filtering according to the maximum value;
the filtering processing module is used for filtering the streaming data according to a data filtering strategy;
and the combination processing module is used for combining the filtered streaming data according to a data combination method.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 4 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110480402.6A CN113157747A (en) | 2021-04-30 | 2021-04-30 | Data service method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110480402.6A CN113157747A (en) | 2021-04-30 | 2021-04-30 | Data service method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113157747A true CN113157747A (en) | 2021-07-23 |
Family
ID=76872623
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110480402.6A Pending CN113157747A (en) | 2021-04-30 | 2021-04-30 | Data service method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113157747A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120330915A1 (en) * | 2011-06-21 | 2012-12-27 | Salesforce.Com, Inc. | Streaming transaction notifications |
| CN107748752A (en) * | 2017-09-05 | 2018-03-02 | 新智云数据服务有限公司 | A kind of data processing method and device |
| CN107959699A (en) * | 2016-10-18 | 2018-04-24 | 中车株洲电力机车研究所有限公司 | A kind of WebSocket data delivery systems and method based on Stream Processing |
| CN108885627A (en) * | 2016-01-11 | 2018-11-23 | 甲骨文美国公司 | Inquiry, that is, service system of query result data is provided to Terminal Server Client |
| CN112000636A (en) * | 2020-08-31 | 2020-11-27 | 民生科技有限责任公司 | User behavior statistical analysis method based on Flink streaming processing |
| CN112148578A (en) * | 2020-10-12 | 2020-12-29 | 贵州电网有限责任公司 | IT fault defect prediction method based on machine learning |
-
2021
- 2021-04-30 CN CN202110480402.6A patent/CN113157747A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120330915A1 (en) * | 2011-06-21 | 2012-12-27 | Salesforce.Com, Inc. | Streaming transaction notifications |
| CN108885627A (en) * | 2016-01-11 | 2018-11-23 | 甲骨文美国公司 | Inquiry, that is, service system of query result data is provided to Terminal Server Client |
| CN107959699A (en) * | 2016-10-18 | 2018-04-24 | 中车株洲电力机车研究所有限公司 | A kind of WebSocket data delivery systems and method based on Stream Processing |
| CN107748752A (en) * | 2017-09-05 | 2018-03-02 | 新智云数据服务有限公司 | A kind of data processing method and device |
| CN112000636A (en) * | 2020-08-31 | 2020-11-27 | 民生科技有限责任公司 | User behavior statistical analysis method based on Flink streaming processing |
| CN112148578A (en) * | 2020-10-12 | 2020-12-29 | 贵州电网有限责任公司 | IT fault defect prediction method based on machine learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8819038B1 (en) | System and method for performing set operations with defined sketch accuracy distribution | |
| US20160162793A1 (en) | Method and apparatus for decision tree based search result ranking | |
| CN105512199B (en) | Search method, search device and search server | |
| US20090234849A1 (en) | Streaming Faceted Search | |
| US11188443B2 (en) | Method, apparatus and system for processing log data | |
| CN113360554A (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN111723112B (en) | Data task execution method and device, electronic equipment and storage medium | |
| CN111258978A (en) | Data storage method | |
| CN110222124A (en) | Multidimensional data processing method and system based on OLAP | |
| CN104426838A (en) | Internet cache scheduling method and system | |
| CN111723161A (en) | Data processing method, device and equipment | |
| CN113836163A (en) | Data association query method, device, equipment and storage medium | |
| CN108154024B (en) | Data retrieval method and device and electronic equipment | |
| CN113342876B (en) | Data fuzzy query method and device of multi-tenant CRM system in SaaS environment | |
| CN109117426B (en) | Distributed database query method, device, equipment and storage medium | |
| CN113986947A (en) | Data flow display method, device, equipment and readable storage medium | |
| CN111797095B (en) | Index construction method and JSON data query method | |
| CN110019077A (en) | Log inquiring method, device, equipment and computer readable storage medium | |
| US9092338B1 (en) | Multi-level caching event lookup | |
| CN113157747A (en) | Data service method and device | |
| CN113220684A (en) | Data packet storage and query method, device, system and storage medium | |
| CN111400266B (en) | Data processing method and system, and diagnosis processing method and device for operation event | |
| CN110879819A (en) | Method, device, server and storage medium for quickly and accurately identifying routing information | |
| CN111209263A (en) | Data storage method, device, equipment and storage medium | |
| CN110928902A (en) | Query method and system for acquiring cloud platform terminal data aiming at paging |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |