CN109189815B - Second-level cache data processing method and system based on NiFi - Google Patents
Second-level cache data processing method and system based on NiFi Download PDFInfo
- Publication number
- CN109189815B CN109189815B CN201811075197.XA CN201811075197A CN109189815B CN 109189815 B CN109189815 B CN 109189815B CN 201811075197 A CN201811075197 A CN 201811075197A CN 109189815 B CN109189815 B CN 109189815B
- Authority
- CN
- China
- Prior art keywords
- cache
- level cache
- level
- server
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0893—Caches characterised by their organisation or structure
- G06F12/0897—Caches characterised by their organisation or structure with two or more cache hierarchy levels
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The invention belongs to the technical field of data management and discloses a method and a system for processing secondary cache data based on NiFi.A secondary cache server is started to monitor data signals; receiving a user operation request, converting the user operation request into a signal and sending the signal to a server; sending the request to a second-level cache; sending a Secondarycache signal to a secondary cache server; storing the cache data into a SecondaryCache; the second-level cache client service sends an applySecondardycache signal to the second-level cache server; the cache data of the SecondaryCache is copied to a new cache object, and the new cache object pointer is pointed to the cache for querying. The invention achieves the aim of safely deleting the invalid cache by adding a second-level cache for temporarily storing the cache data and having atomicity on the operation of the original cache.
Description
Technical Field
The invention belongs to the technical field of data management, and particularly relates to a second-level cache data processing method and system based on NiFi.
Background
Currently, the current state of the art commonly used in the industry is such that:
with the componentization of big data ETL, many ETL functions have been implemented to let users connect and configure ETL procedures using drag and pull on the interface. The NiFi is such a powerful ETL engine, and a lot of ETL component requirements have been realized, which greatly reduces the threshold of ETL processing data by users. As a cache component frequently used in a NiFi engine, most requirements cannot be met in an actual business scenario, which is because deleted data of a database cannot be captured when a result of a database query is cached, so that discarded data is retained, and when cache query matching is performed, an outdated cache is matched, so that quality of business data is affected.
In summary, the problems of the prior art are as follows:
(1) in the prior art, when the query result of the database is cached, deleted data of the database cannot be captured, so that the discarded data is retained, and when cache query matching is performed, an outdated cache is matched, so that the quality of service data is influenced.
(2) In the prior art, when a cache component is used, a failure cache cannot be removed in time, and the use of other service components is influenced.
The difficulty and significance for solving the technical problems are as follows:
the difficulty is as follows: in the actual project, when dictionary data of the database is inquired, deleted data cannot be captured;
the significance brought by solving the difficulty of the prior art is as follows: the invention solves the problem that when the dictionary data of the database is used as the cache data, the old cache can be reset based on single data query operation, and the invalid dictionary cache data can be effectively cleared and updated within acceptable time delay.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a method and a system for processing second-level cache data based on NiFi,
the invention is realized in this way, a second-level cache data processing method based on NiFi, comprising:
and starting a second-level cache server, which comprises a Socket pipeline monitoring service and a cache service based on a memory and a disk, and is used for monitoring a specified Socket port and processing a corresponding signal to apply to the cache service. The data signals that are listened to include: a request for resetting the second-level cache putSecondaryCache, resetSecondaryCache and all second-level caches applying the second-level cache applySecondaryCache;
and starting a second-level cache client, which comprises a second-level cache data signal interface for requesting the server to operate when all the second-level cache client communicates with the Socket pipeline service of the second-level cache server, and is used for sending the data signal of the cache operation of the server by the application end to the server for processing.
And the application terminal operates the cache service of the second-level cache server terminal through a second-level cache client terminal interface, and the operations comprise storing in a second-level cache, resetting the second-level cache and applying the second-level cache.
Further, the method for processing the second-level cache data based on the NiFi specifically comprises the following steps:
the method comprises the following steps: starting a secondary cache server, and monitoring data signals stored in a secondary cache putSecondarycache, a secondary cache resetSecondarycache and an application secondary cache applySecondarycache;
step two: starting a second-level cache client service, receiving a user operation request, converting the request into a corresponding signal and sending the signal to a server;
step three: sending cache data to a secondary cache Secondarycache through a client service request;
step four: the secondary cache client service sends a Secondarycache signal to the secondary cache server;
step five: the second-level cache server receives the Secondarycache signal and stores the cache data into the Secondarycache;
step six: the second-level cache client service sends an applySecondardycache signal to the second-level cache server;
step seven: the second-level Cache server receives the applySecondarycache signal, copies the Cache data of the Secondarycache into a new Cache object, and points the pointer of the new Cache object to the Cache for query;
step eight: the second-level cache client service sends a resetcondardarycache signal to the second-level cache server;
step nine: and the second-level cache server receives the resetseconcondary cache signal and clears the cache data of the second-level cache seconcondary cache object.
Another object of the present invention is to provide a computer program for implementing the method for processing NiFi-based second-level cache data according to any of claims 1 to 2.
Another object of the present invention is to provide an information data processing terminal for implementing the method for processing data of a second level cache based on NiFi.
Another object of the present invention is to provide a computer-readable storage medium, comprising instructions, which when run on a computer, cause the computer to perform the NiFi-based second level cache data processing method.
Another object of the present invention is to provide a second level cache data processing system based on NiFi for implementing the method for processing second level cache data based on NiFi, wherein the second level cache data processing system based on NiFi comprises:
the server side: the cache management system is used for monitoring all cache requests in a monitoring mode; the cache is also used for storing the second-level cache, resetting the second-level cache and applying the second-level cache;
a client: the server is used for operating the stored second-level cache, resetting the second-level cache and applying the second-level cache data signals, converting the received user operation request into corresponding signals and sending the corresponding signals to the server.
Another object of the present invention is to provide a cache data engine platform, which at least mounts the NiFi-based second-level cache data processing system.
The server side comprises three operations for the second-level cache, namely a putSecondarycache, a resetSecondarycache and an applySecondarycache, and the operations are respectively used for storing the second-level cache, resetting the second-level cache and applying the second-level cache to a formal cache;
the client interface comprises three corresponding methods for operating the second-level cache, namely putSeconderyCache, resetSeconderyCache and applySeconderyCache.
In summary, the advantages and positive effects of the invention are:
the invention provides a practical and convenient method for solving the defects of the cache component of the NiFi, so that the invalid cache can be removed in time when the cache component is used, and the use of other service components can not be influenced.
The invention achieves the aim of safely deleting the invalid cache by adding a second-level cache for temporarily storing the cache data and having atomicity on the operation of the original cache.
Drawings
Fig. 1 is a flowchart of a method for processing NiFi-based secondary cache data according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a method for processing data of a second level cache based on NiFi according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides a practical and convenient method for solving the defects of the cache component of the NiFi, so that the invalid cache can be removed in time when the cache component is used, and the use of other service components can not be influenced.
The invention is explained in further detail below with reference to the drawings;
as shown in fig. 1, a method for processing data of a second level cache based on NiFi according to an embodiment of the present invention includes:
s101: starting a second-level cache server, and monitoring data signals of a ' putSecondarycache ', ' resetSecondarycache ' and an ' application second-level cache ' (applySecondarycache) ';
s102: starting a second-level cache client service, receiving a user operation request, converting the request into a corresponding signal and sending the signal to a server;
s103: sending the cache data to a second-level cache (Secondarycache) through a client service request;
s104: the secondary cache client service sends a 'Secondarycache' signal to the secondary cache server;
s105: the second-level cache server receives the 'second cache' signal and stores the cache data into a second-level cache (second cache);
s106: the second-level cache client service sends an 'applySecondardycache' signal to the second-level cache server;
s107: the second-level Cache server receives an 'applySecondarycache' signal, copies the Cache data of the Secondarycache into a new Cache object, and points the pointer of the new Cache object to the Cache for query;
s108: the second-level cache client service sends a 'resetcondardyrcache' signal to the second-level cache server;
s109: the second-level cache server receives the "resetsecondary cache" signal, and clears the cache data of the second-level cache (secondary cache) object.
The second-level Cache server side provided by the embodiment of the invention comprises the Cache for inquiring the Cache and the second-level Cache Secondarycache for temporarily storing the Cache.
The embodiment of the invention provides a second-level cache data processing system based on NiFi, which comprises:
the method comprises the following steps: the server side: monitoring all cache requests in a monitoring mode; respectively used for storing the second-level cache, resetting the second-level cache and applying the second-level cache to the formal cache;
a client: and operating the second-level cache, and converting the received user operation request into a corresponding signal and sending the signal to the server.
The server provided by the embodiment of the invention comprises three operations for the secondary cache, namely a putSecondarycache, a resetSecondarycache and an applySecondarycache, which are respectively used for storing the secondary cache, resetting the secondary cache and applying the secondary cache to a formal cache;
the client interface provided by the embodiment of the invention comprises three corresponding methods for operating the second-level cache, namely putSecondaryCache, resetSecondaryCache and applySecondaryCache.
The invention is further explained in detail below with reference to the working principle;
as shown in fig. 2, a method for processing data of a second level cache based on NiFi according to an embodiment of the present invention includes:
a processor PutDistributedMapSecondaryCache is newly added. Comparing the currently inflowing FlowFile file name (currentFilename) with the last successfully stored FlowFile file name (oldFilename), and judging whether the currently inflowing FlowFile file name is a new database query result:
1) when the old file name is inconsistent with the current file name (which indicates that the current inflow is a new query result), whether the current flow is the previous batch of query results stored in the FlowFile which fails in the cache is continuously judged through the 'isputFail' attribute of the FlowFile (the previous batch of data which fails in the cache is stored in a failure queue, the situation that the previous batch of data is tried to be inserted into the cache simultaneously with the new batch of query results occurs, and the 'isputFail' attribute is added and marked for the FlowFile which fails to be stored). If yes, the query indicating the new batch is inserted, but the FlowFile with the result of the query of the last batch failed to be inserted is still trying to be stored, and the query is routed to the 'outdated' relationship. If not, the query result is indicated to be the query result of the new batch, the second-level cache can be directly reset, and the storing operation is carried out.
2) The old file name is consistent with the current file name (which indicates that the old file name is the query result of the same batch and is continuously stored in the cache).
And judging whether the new query result is stored in the second-level cache or not. And applying the secondary cache to the formal cache by judging whether the fragment.count attribute of the FlowFile is equal to the successfully stored counter handled or not and then calling a related method.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When used in whole or in part, can be implemented in a computer program product that includes one or more computer instructions. When loaded or executed on a computer, cause the flow or functions according to embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL), or wireless (e.g., infrared, wireless, microwave, etc.)). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (7)
1. A second level cache data processing method based on NiFi is characterized in that the second level cache data processing method based on NiFi comprises the following steps:
starting a second-level cache server, monitoring a specified socket port and processing a corresponding signal to apply to cache service;
starting a second-level cache client, and when all communication is carried out with Socket pipeline service of a second-level cache server, requesting a second-level cache data signal interface operated by the server to send a data signal of cache operation of the server from an application terminal to the server for processing;
and the application terminal operates the cache service of the second-level cache server terminal through the second-level cache client terminal interface.
2. The NiFi-based secondary cache data processing method of claim 1, wherein starting the secondary cache server comprises starting Socket pipe snooping service and cache service based on memory and disk;
the data signals that are listened to include: a request to store the secondary cache putSecondaryCache, reset the secondary cache resetSecondaryCache, and apply all secondary caches of the secondary cache applySecondaryCache;
the operation of the application terminal on the cache service of the second-level cache service terminal comprises the operations of storing in the second-level cache, resetting the second-level cache and applying the second-level cache.
3. The method for processing data of the second level cache based on the NiFi according to claim 1, wherein the method for processing data of the second level cache based on the NiFi specifically comprises:
the method comprises the following steps: starting a secondary cache server, and monitoring data signals stored in a secondary cache putSecondarycache, a secondary cache resetSecondarycache and an application secondary cache applySecondarycache;
step two: starting a second-level cache client service, receiving a user operation request, converting the request into a corresponding signal and sending the signal to a server;
step three: sending cache data to a putSecondardrycache stored in a second-level cache through a client service request;
step four: the secondary cache client service sends a Secondarycache signal to the secondary cache server;
step five: the second-level cache server receives the Secondarycache signal and stores the cache data into the Secondarycache;
step six: the second-level cache client service sends an applySecondardycache signal to the second-level cache server;
step seven: the second-level Cache server receives the applySecondarycache signal, copies the Cache data of the Secondarycache into a new Cache object, and points the pointer of the new Cache object to the Cache for query;
step eight: the second-level cache client service sends a resetcondardarycache signal to the second-level cache server;
step nine: and the second-level cache server receives the resetseconcondary cache signal and clears the cache data of the second-level cache seconcondary cache object.
4. An information data processing terminal for implementing the NiFi-based secondary cache data processing method according to any one of claims 1-3.
5. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the NiFi-based level two cache data processing method of any of claims 1-3.
6. A second level cache data processing system based on NiFi for implementing the method of claim 1, wherein the second level cache data processing system based on NiFi comprises:
the server side: the cache management system is used for monitoring all cache requests in a monitoring mode; the cache is also used for storing the second-level cache, resetting the second-level cache and applying the second-level cache;
a client: the server is used for operating the stored second-level cache, resetting the second-level cache and applying the second-level cache data signals, converting the received user operation request into corresponding signals and sending the corresponding signals to the server.
7. A cache data engine apparatus, characterized in that the cache data engine apparatus mounts the NiFi based level two cache data processing system of claim 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811075197.XA CN109189815B (en) | 2018-09-14 | 2018-09-14 | Second-level cache data processing method and system based on NiFi |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811075197.XA CN109189815B (en) | 2018-09-14 | 2018-09-14 | Second-level cache data processing method and system based on NiFi |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109189815A CN109189815A (en) | 2019-01-11 |
| CN109189815B true CN109189815B (en) | 2022-03-04 |
Family
ID=64911238
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811075197.XA Active CN109189815B (en) | 2018-09-14 | 2018-09-14 | Second-level cache data processing method and system based on NiFi |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109189815B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102137165A (en) * | 2011-04-22 | 2011-07-27 | 南京邮电大学 | Method for realizing ARES indexing system based on peer-to-peer (P2P) network |
| CN102404212A (en) * | 2011-11-17 | 2012-04-04 | 曙光信息产业(北京)有限公司 | Cross-platform RDMA (Remote Direct Memory Access) communication method based on InfiniBand |
| CN103259683A (en) * | 2013-05-16 | 2013-08-21 | 烽火通信科技股份有限公司 | Web network management system second level cache pushing method based on HTML5 |
| US9697239B1 (en) * | 2016-04-15 | 2017-07-04 | Lars Dierk Buchholz | Token-based database system and method of interfacing with the token-based database system |
| CN107330028A (en) * | 2017-06-23 | 2017-11-07 | 北京北信源软件股份有限公司 | Expansion application methods and system of a kind of Apache NiFi in terms of source data input database |
-
2018
- 2018-09-14 CN CN201811075197.XA patent/CN109189815B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102137165A (en) * | 2011-04-22 | 2011-07-27 | 南京邮电大学 | Method for realizing ARES indexing system based on peer-to-peer (P2P) network |
| CN102404212A (en) * | 2011-11-17 | 2012-04-04 | 曙光信息产业(北京)有限公司 | Cross-platform RDMA (Remote Direct Memory Access) communication method based on InfiniBand |
| CN103259683A (en) * | 2013-05-16 | 2013-08-21 | 烽火通信科技股份有限公司 | Web network management system second level cache pushing method based on HTML5 |
| US9697239B1 (en) * | 2016-04-15 | 2017-07-04 | Lars Dierk Buchholz | Token-based database system and method of interfacing with the token-based database system |
| CN107330028A (en) * | 2017-06-23 | 2017-11-07 | 北京北信源软件股份有限公司 | Expansion application methods and system of a kind of Apache NiFi in terms of source data input database |
Non-Patent Citations (1)
| Title |
|---|
| 一种面向应用服务器的分布式缓存机制;郭唐宝等;《科学技术与工程》;20111231;第11卷(第36期);第8988-8992页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109189815A (en) | 2019-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110598138A (en) | Cache-based processing method and device | |
| CN109213604B (en) | Data source management method and device | |
| WO2019062572A1 (en) | Data processing method, device and system | |
| WO2020019724A1 (en) | Method for acquiring sensor data in server, acquisition system, and related apparatus | |
| US20130311425A1 (en) | Database update notification method | |
| CN110311975B (en) | Data request processing method and device | |
| CN113271359A (en) | Method and device for refreshing cache data, electronic equipment and storage medium | |
| US11218542B2 (en) | Method and system for selecting a transport mechanism and a storage process | |
| WO2020232892A1 (en) | Method and apparatus for key management for blockchain, device, and storage medium | |
| CN113364887B (en) | File downloading method based on FTP, proxy server and system | |
| CN109189815B (en) | Second-level cache data processing method and system based on NiFi | |
| CN113836180A (en) | Second-level cache data processing method, processor, storage medium and system | |
| KR102665749B1 (en) | Method and apparatus for ensuring continued device operational reliability in cloud-degraded mode | |
| CN113726885B (en) | Flow quota adjusting method and device | |
| US11057470B2 (en) | Communication device and communication method for processing meta data | |
| CN113242245A (en) | TCP message flow limiting method, device, server and storage medium | |
| CN109120692B (en) | Method and apparatus for processing requests | |
| US9158817B2 (en) | Enhancing online querying service using client memorization | |
| CN107423375B (en) | Application program searching method and device | |
| CN113672248A (en) | Patch acquisition method, device, server and storage medium | |
| CN112711572A (en) | Online capacity expansion method and device suitable for sub-warehouse and sub-meter | |
| KR102460910B1 (en) | Data storage method for preventing data duplication and data platform applying the same | |
| CN113407493B (en) | Operation method, data reading and writing method, device, electronic equipment and medium | |
| US20240089339A1 (en) | Caching across multiple cloud environments | |
| CN116170432A (en) | File processing method and device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |