CN110244912B - Caching method of data distribution system - Google Patents
Caching method of data distribution system Download PDFInfo
- Publication number
- CN110244912B CN110244912B CN201910537303.XA CN201910537303A CN110244912B CN 110244912 B CN110244912 B CN 110244912B CN 201910537303 A CN201910537303 A CN 201910537303A CN 110244912 B CN110244912 B CN 110244912B
- Authority
- CN
- China
- Prior art keywords
- main body
- supplier
- demander
- identifier
- end processor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0613—Improving I/O performance in relation to throughput
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Accounting & Taxation (AREA)
- Human Computer Interaction (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a cache method of a data distribution system, wherein the data distribution system comprises a supplier and a demander; the caching method comprises the following steps: the supplier distributes a first cache region in the supplier terminal equipment for distribution, the first cache region caches the main body identification or the main body identification and the corresponding elements thereof, the demander distributes a second cache region in the demander terminal equipment for distribution, and the second cache region caches the main body identification or the main body identification and the corresponding elements thereof; and the demander data management platform acquires the elements corresponding to the main body identifier according to the main body identifier, matches the elements corresponding to the main body identifier by the demander terminal equipment, the supplier terminal equipment or the supplier data management platform according to the main body identifier, and distributes the main body identifier and the elements corresponding to the main body identifier through the demander terminal equipment and the supplier terminal equipment. According to the invention, by setting the cache region, a large amount of disk IO operations in the distribution system are avoided, the requirements of millisecond-level low delay and high throughput of data distribution are met, and the cache can be updated in real time.
Description
Technical Field
The invention relates to the technical field of data transaction, in particular to a caching method of a data distribution system.
Background
At present, distribution systems all need to meet the requirements of high concurrency, high throughput, low delay and data real-time performance, and the storage of a large amount of traditional databases or files depends on disk IO operation, so that the efficiency of reading data is very low for the distribution system with mass data.
Particularly in the case of high concurrency, the distribution system having mass data has the following problems: (1) the traditional database and file storage can generate a large amount of disk IO operations, and cannot meet the requirements of millisecond-level low delay and high throughput of data distribution; (2) the traditional database, file and common cache system are difficult to realize horizontal expansion along with service development; (3) data are loaded into a cache, and data expansion often exists, so that a large amount of memory is occupied; (4) under the premise of limited hardware resources and massive data, the data cache is difficult to keep real-time update with a data source, and the stable external service capability is kept in the data updating period.
Therefore, it is necessary to provide a cache method for a data distribution system, which achieves the purposes of low latency, high concurrency and real-time update of the cache, which can be dynamically expanded.
Disclosure of Invention
The invention aims to provide a cache method of a data distribution system, which can realize the purposes of low delay, high concurrency and real-time cache updating which can be dynamically expanded.
In order to solve the problems in the prior art, the invention provides a cache method of a data distribution system, wherein the data distribution system comprises a supplier and a demander;
the caching method comprises the following steps: the supplier allocates a first cache region in the supplier terminal equipment for distribution, the first cache region caches the main body identifier or the main body identifier and the element corresponding to the main body identifier, the demander allocates a second cache region in the demander terminal equipment for distribution, and the second cache region caches the element corresponding to the main body identifier or the main body identifier and the main body identifier;
an demander data management platform in the data distribution system acquires elements corresponding to a main body identifier according to the main body identifier, a first cache region or a supplier data management platform matches the elements corresponding to the main body identifier according to the main body identifier, and the main body identifier and the elements corresponding to the main body identifier are distributed to the demander data management platform through a supplier terminal device and a demander terminal device; and the second cache region matches the elements corresponding to the subject identification according to the subject identification, and distributes the subject identification and the elements corresponding to the subject identification to the demander data management platform through the demander terminal equipment.
Optionally, in the caching method of the data distribution system,
the supplier data management platform corresponds to at least one supplier terminal device, and the supplier terminal device is a supplier front-end processor;
the demander data management platform corresponds to at least one demander terminal device, and the demander terminal device is a demander front-end processor.
Optionally, in the cache method of the data distribution system, the second cache region caches the subject identifier and its corresponding element, and the acquirer data management platform acquires the data includes the following steps:
the demander data management platform initiates a request according to a main body identifier, wherein the request comprises the main body identifier so as to obtain an element corresponding to the main body identifier;
the acquirer front-end processor receiving the request;
matching the main body identifier in the request with the main body identifier in the second cache region; if the second cache region has a subject identifier successfully matched with the subject identifier in the request and also has an element corresponding to the successfully matched subject identifier, feeding back the successfully matched subject identifier and the element corresponding to the successfully matched subject identifier to the demander data management platform; if the matching is not successful or the matching is successful but no corresponding element exists, the request is initiated to the supplier front-end processor;
the supplier front-end processor acquires the request, matches elements according with the request in a collision way at the supplier according to the request, and distributes the successfully matched subject identification and the corresponding elements to the supplier data management platform through the supplier front-end processor and the supplier front-end processor; and for the subject identification which is not matched successfully, the feedback query fails.
Optionally, in the caching method of the data distribution system, the second cache region does not have a cache main body identifier and an element corresponding to the cache main body identifier, and the obtaining of the data by the requester includes the following steps:
the demander data management platform initiates a request according to a main body identifier, wherein the request comprises the main body identifier so as to obtain an element corresponding to the main body identifier;
the requiring side front-end processor receives the request and sends the request to the supplying side front-end processor;
the supplier front-end processor acquires the request, matches elements according with the request in a collision way at the supplier according to the request, and distributes the successfully matched subject identification and the corresponding elements to the supplier data management platform through the supplier front-end processor and the supplier front-end processor; and for the subject identification which is not matched successfully, the feedback query fails.
Optionally, in the caching method of the data distribution system, the supplier collision matching includes the following steps:
matching the main body identifier in the request with the main body identifier in the first cache region;
when the main body identification and the corresponding element thereof are cached in the first cache region, the successfully matched main body identification and the corresponding element thereof are fed back to the demander front-end processor;
when the first cache region only caches the subject identification, the supplier front-end processor sends the successfully matched subject identification to the supplier data management platform to call the corresponding element, and feeds back the successfully matched subject identification and the corresponding element to the demander front-end processor;
and for the subject identification which is unsuccessfully matched, feeding back the query failure or sending the subject identification which is unsuccessfully matched to the supplier data management platform for re-matching, distributing the subject identification which is successfully re-matched and the corresponding elements thereof to the supplier data management platform through the supplier front-end processor and the demander front-end processor, and feeding back the query failure for the subject identification which is unsuccessfully re-matched.
Optionally, in the cache method of the data distribution system, the acquiring data by the acquirer further includes the following steps:
the demander data management platform initiates a request according to a main body identifier, wherein the request comprises the main body identifier so as to obtain an element corresponding to the main body identifier;
the demander front-end processor receives the request and buffers the request in the second buffer area; the requiring side front-end processor sends a collision invitation to the supplying side front-end processor; and the supplier front-end processor receives the collision invitation, and transmits the main body identifier or the main body identifier cached in the first cache region and the corresponding element thereof to the second cache region to perform collision matching with the main body identifier in the request.
Optionally, in the cache method of the data distribution system, the collision matching includes the following cases:
when the first cache region caches the subject identifier, the supplier front-end processor transmits the subject identifier cached in the first cache region to the second cache region for collision matching after receiving a collision invitation, the demander front-end processor transmits the successfully matched subject identifier to the supplier data management platform through the supplier front-end processor to call the corresponding element, and distributes the successfully matched subject identifier and the corresponding element to the demander data management platform;
when the first cache region caches the subject identifier and the corresponding element thereof, the supplier front-end processor transmits the subject identifier cached in the first cache region and the corresponding element thereof to the second cache region for collision matching after receiving a collision invitation, and the demander front-end processor distributes the successfully matched subject identifier and the corresponding element thereof to the demander data management platform;
and for the subject identification which is unsuccessfully matched, feeding back the query failure or sending the subject identification which is unsuccessfully matched to the supplier data management platform for re-matching, distributing the subject identification which is successfully re-matched and the corresponding elements thereof to the supplier data management platform through the supplier front-end processor and the demander front-end processor, and feeding back the query failure for the subject identification which is unsuccessfully re-matched.
Optionally, in the caching method of the data distribution system, if the first cache area of more than one supplier or the supplier data management platform has a principal identifier successfully matched with the principal identifier in the request, the demander selects one of the supplier providing elements according to the requirement.
Optionally, in the cache method of the data distribution system, the data distributed by the data distribution system includes a body identifier in the request, an element conforming to the request, and distributed interface information.
Optionally, in the caching method of the data distribution system,
the cache contents in the first cache region and the second cache region are updated according to appointed time;
and/or the data distributed by the data distribution system is created according to a specified format;
and/or the body identification comprises an encrypted body identification or an unencrypted body identification.
According to the caching method of the data distribution system, provided by the invention, a large number of disk IO operations in the distribution system are avoided by setting the cache region, so that the requirements of millisecond-level low delay and high throughput of data distribution are met, and the problem that the distribution system is difficult to realize transverse expansion along with service development in the prior art is solved; the invention enables the distribution system to keep stable external service capability in the cache updating period on the premise of limited hardware resources and mass data by setting the independent cache region.
Drawings
Fig. 1a-1c are block diagrams of data distribution using cache according to an embodiment of the present invention.
Detailed Description
The following describes in more detail embodiments of the present invention with reference to the schematic drawings. The advantages and features of the present invention will become more apparent from the following description. It is to be noted that the drawings are in a very simplified form and are not to precise scale, which is merely for the purpose of facilitating and distinctly claiming the embodiments of the present invention.
In the following, the terms "first," "second," and the like are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances. Similarly, if the method described herein comprises a series of steps, the order in which these steps are presented herein is not necessarily the only order in which these steps may be performed, and some of the described steps may be omitted and/or some other steps not described herein may be added to the method.
The present invention provides a cache method of a data distribution system, as shown in fig. 1a to 1c, fig. 1a to 1c are block diagrams of data distribution using a cache according to an embodiment of the present invention, where the data distribution system includes a supplier and a demander;
the caching method comprises the following steps: the supplier allocates a first cache region in the supplier terminal equipment for distribution, the first cache region caches the main body identifier or the main body identifier and the element corresponding to the main body identifier, the demander allocates a second cache region in the demander terminal equipment for distribution, and the second cache region caches the element corresponding to the main body identifier or the main body identifier and the main body identifier;
an demander data management platform in the data distribution system acquires elements corresponding to the main body identification according to the main body identification, a first cache region or a supplier data management platform matches the elements corresponding to the main body identification according to the main body identification, and the main body identification and the elements corresponding to the main body identification are distributed to the demander data management platform through a supplier terminal device and a demander terminal device; and the second cache region matches the elements corresponding to the subject identification according to the subject identification, and distributes the subject identification and the elements corresponding to the subject identification to the demander data management platform through the demander terminal equipment.
According to the invention, by setting the cache region, a large number of disk IO operations in the distribution system are avoided, so that the requirements of millisecond-level low delay and high throughput of data distribution are met, and the problem that the distribution system is difficult to realize transverse expansion along with service development in the prior art is solved; the invention enables the distribution system to keep stable external service capability in the cache updating period on the premise of limited hardware resources and mass data by setting the independent cache region.
Further, the supplier comprises a supplier data management platform, the supplier data management platform corresponds to at least one supplier terminal device, and the supplier terminal device is a supplier front-end processor; the demander comprises the demander data management platform, the demander data management platform corresponds to at least one demander terminal device, and the demander terminal device is a demander front-end processor. The supplier data management platform and the demander data management platform are business systems of a supplier and a demander for data circulation in the distribution process, and the data management platform can be referred to as a DMP for short.
The cache collision method in the invention can include the following situations that collision matching can be carried out in both the first cache region and the second cache region, or collision matching can be carried out in the first cache region alone, or collision matching can be carried out in the second cache region alone. Preferably, in the case that the collision matching is performed in both the first buffer area and the second buffer area, which is applicable to the case that both the supplier and the demander have the body id and its corresponding elements buffered, the data distribution method adopted by the collision matching may refer to the first data distribution embodiment. The single collision matching in the first buffer area may be applicable to the case where there are more subject identifiers and their corresponding elements in the suppliers, for example, a certain demander has 100 subject identifiers that need to search for elements, and a certain supplier has 1000 subject identifiers and their corresponding elements, and at this time, the collision matching may be performed in the first buffer area separately, and the data distribution method adopted by the collision matching may refer to the second data distribution embodiment. The single collision matching in the second buffer area may be applicable to a situation where there are more subject identifiers that need to obtain elements in the acquirer, for example, a certain acquirer has 1000 subject identifiers that need to obtain elements, and a certain supplier only has 100 subject identifiers and their corresponding elements, and at this time, the collision matching may be performed in the second buffer area separately, and the data distribution method adopted by the collision matching may refer to the third data distribution embodiment. It is understood that the above numbers 1000 and 100 are only used for comparison, and that 1000 body marks or body marks and their corresponding elements belong to the case of more actual data, and 100 body marks or body marks and their corresponding elements belong to the case of less actual data. In addition, the collision is only aimed at the cache region, and if the collision matching of the first cache region and/or the second cache region fails, the collision matching can be continued on the supplier data management platform.
In the caching method provided by the present invention, the subject identifier may be abbreviated as ID, the data distributed by the data distribution system includes the subject identifier in the request, the elements conforming to the request, and the distributed interface information, and the elements corresponding to the subject identifier include KV values in a KV library. In one embodiment, the caching method may cache in the form of cache loading: firstly, caching and querying in a transaction interface are realized by using Redis; then loading the converted ID list by using Redis; and finally, loading an ID + KV value by using Redis, and updating the cache contents (including a subject Identifier (ID) and an element (KV value)) in the first cache region and the second cache region according to an appointed time, wherein the appointed time can be 1 hour or 1 day and the like. The invention can greatly reduce the expansion of the cache by adopting the cache key value design, thereby reducing the occupation of the memory.
Further, the functional modules in the first cache region and the second cache region may include: the system comprises a text import module, a processing cache module and a service access module, wherein the introduction and improvement of each module are shown in the following table:
further, in the data preparation phase, the body identifier may be an encrypted body identifier or an unencrypted body identifier, and the data distributed by the data distribution system needs to be created according to a specified format. The creation mode of the subject identifier and the element may include the following steps: the first step is text import, and the supplier data management platform regularly performs algorithm running number processing to form a file comprising ID and/or ID + KV. And secondly, processing a cache, wherein the first cache region acquires a file comprising ID and/or ID + KV, processes the file to form an index and loads the index into a relational database, and the main body identification and the elements are created according to a specified format.
For example, the prescribed format may be as follows: the file name format is: idtype _ timestamp.SOURCE (eg: ID010201_ YYYYYMMDD000001. SOURCE)
The format of the file header is as follows: exid | @ | connObjId | @ | connObjId | @ | connObjId (eg: exid | @ | DON 000010100000000000051 | @ | DON00001010000001196| @ | DON 00001010000001208).
The format of the elements in the file is as follows:
exid|@|1|@|222|@|4444
(eg：5d6a7a48dd27fdc49a82c210e5880a30|@|32|@|22|@|11)。
further, the first cache region caching step may be as follows:
the first step is as follows: downloading a file;
and scheduling a timing task, and performing inquiry downloading to a supplier transmission (SFTP) server according to the user and the password provided by the supplier. If the downloading is not generated during the processing of the timing task, the task can be acquired by manual calling afterwards.
The second step is that: downloading a KV library:
scheduling a timed task, namely scheduling according to an API (application programming interface) provided by a supplier to obtain elements, wherein the referenced API interaction flow is as follows:
1) token is obtained according to APIKey, UserName, Passsword.
http://{host}:{port}/telecom-dmp/getTokenapiKey＝{apiKey}&sign＝{apiKey}；
2) Acquiring KV value in corresponding KV table according to Token
http://{host}:{port}/telecom-dmp/kv/getValueByKeytoken＝{token}&table＝{ta ble}&key＝{key}；
2.1) acquiring a main key, a name fixed every hour, keymap, { YYYYYYmmdd }. map, and acquiring a secondary key storing the specific key;
2.2) requesting a secondary key to obtain a specific key list, wherein the list is a key converted by base64 for a relevant imei number;
2.3) request a specific key to obtain the corresponding value.
Further, the cache method is applied to a data distribution system, and the data distribution system can prepare as follows:
for the supplier, (1) the supplier is responsible for label generation processing, and the label format is as "ID + Value". (2) The supplier is responsible for loading the tags into the KV database. (3) The supplier can provide the main body mark and the elements to the supplier front-end processor according to the specified format. (4) The supplier is responsible for matching the butt joint of the supplier front-end processor to the access interface of the KV database. (5) The supplier front-end processor cleans up the expiration elements and indices at a contracted time (e.g., 1 time every 24 hours).
For the demander, (1) the demander is responsible for providing the principal identifier to the demander front-end in a prescribed format. (2) The demander is matched with the butt joint of a data query/result return interface between the demander front-end processor and the demander data management platform. (3) When data is distributed, a real-time query request (which can be in a multi-page query mode) can be initiated by the data management platform of the demander.
For the data distribution system, (1) the data distribution system is responsible for realizing static storage, processing and transaction collision matching processing on a front-end processor of a supplier, and simultaneously provides a transaction path to enable the supplier to inquire in a single real-time manner and provide charging settlement service. (2) The data distribution system is responsible for deploying a distribution module and an encryption subject identification translation module on the supply and demand side front-end processor. (3) And the supply side front-end processor retrieves the intra-frequency increment label from the KV library every hour, and establishes an ID collision matching index in the front-end processor. (4) The demander can send a transaction data query request to the supplier front-end processor in real time through the demander front-end processor, the supplier front-end processor can return a query tag result to the demander front-end processor through index generation, and the data distribution system generates a distribution log after data distribution is completed. (5) And the data distribution system generates a settlement list according to the distribution log and the tariff mode.
In a first data distribution embodiment, as shown in fig. 1a to 1b, if a body identifier or a body identifier and its corresponding element are cached in both the first cache region and the second cache region, then collision matching may be performed in both the first cache region and the second cache region, where the body identifier and its corresponding element cached in the second cache region may be data already acquired by an acquirer and is stored in the second cache region. The method for acquiring data by the demander data management platform comprises the following steps:
the demander data management platform initiates a request according to a main body identifier, wherein the request comprises the main body identifier and can also comprise a Key value (Key), the Key value belongs to one element corresponding to the main body identifier, and the request is used for acquiring a specific assignment of the Key value corresponding to the main body identifier, and the specific assignment refers to another element corresponding to the main body identifier;
the acquirer front-end processor receiving the request;
referring to fig. 1a, matching the body identifier in the request with the body identifier in the second cache region; if the second cache region has a subject identifier successfully matched with the subject identifier in the request and also has an element corresponding to the successfully matched subject identifier, feeding back the successfully matched subject identifier and the element corresponding to the successfully matched subject identifier to the demander data management platform; if the successfully matched main body identification does not exist or the successfully matched main body identification does not exist but the corresponding element, initiating the request to a supplier front-end processor, referring to fig. 1b, wherein the supplier front-end processor acquires the request, and according to the request, the supplier front-end processor collides and matches the element which accords with the request at the supplier, and distributes the successfully matched main body identification and the corresponding element to the demander data management platform through the supplier front-end processor and the demander front-end processor; and for the subject identification which is not matched successfully, the feedback query fails.
Further, the supplier collision matching may include the steps of:
matching the main body identifier in the request with the main body identifier in the first cache region;
when the main body identifier and the corresponding element thereof are cached in the first cache region, the supplier front-end processor feeds back the successfully matched main body identifier and the corresponding element thereof to the supplier front-end processor;
when the first cache region only caches the subject identifier, the supplier front-end processor needs to send the successfully matched subject identifier to the supplier data management platform to call the corresponding element, and feeds back the successfully called subject identifier and the corresponding element to the supplier front-end processor;
and feeding back the failure of query to the subject identification which is not successfully matched.
In other embodiments, the unmatched body identifier may be further processed, for example, the unmatched body identifier is sent to the supplier data management platform for being matched again, the matched body identifier and the corresponding elements after being matched again are distributed to the data management platform of the demander through the supplier front-end processor and the demander front-end processor, and the unmatched body identifier after being matched again is failed in feedback query.
In this embodiment, the subject identifier and the corresponding element cached in the second cache of the demander front-end processor may be a summary of requests sent by the demander before, so that the demander can perform collision in the demander front-end processor before sending a new request, and a situation of repeated requests can be eliminated, thereby saving cost for the demander and reducing distribution pressure.
In a second data distribution embodiment, as shown in fig. 1b, a main body identifier or a main body identifier and its corresponding elements are cached in a first cache region, and a main body identifier and its corresponding elements are not cached in a second cache region, in this embodiment, collision matching needs to be performed in the first cache region, and the requiring party acquires data includes the following steps:
the demander data management platform initiates a request according to a main body identifier, wherein the request comprises the main body identifier and can also comprise a Key value (Key), the Key value belongs to one element corresponding to the main body identifier, and the request is used for acquiring a specific assignment of the Key value corresponding to the main body identifier, and the specific assignment refers to another element corresponding to the main body identifier;
the requiring side front-end processor receives the request and sends the request to the supplying side front-end processor;
the supplier front-end processor acquires the request, matches elements according with the request in a collision way at the supplier according to the request, and distributes the successfully matched subject identification and the corresponding elements to the supplier data management platform through the supplier front-end processor and the supplier front-end processor; and for the subject identification which is not matched successfully, the feedback query fails. In this embodiment, the supplier collision matching method may be the same as the data distribution embodiment.
In a third data distribution embodiment, as shown in fig. 1c, a main body identifier or a main body identifier and its corresponding elements are cached in a first cache region, and a second cache region is used for caching the request during distribution, in this embodiment, collision matching is mainly performed in the second cache region, if the number of main body identifiers in the request is greater than a first preset value or the number of main body identifiers in the request is greater than the number of main body identifiers in the first cache region of the supplier, and the difference between the number of the subject identifiers in the request and the number of the subject identifiers in the first cache region exceeds a second preset value, then collision matching can be performed in the second cache region, the first preset value and the second preset value can be set by the data distribution system according to the transmission rate of each supplier front-end processor and each demander front-end processor or other service requirements (such as individualized requirements of the supplier and the demander or summarized according to multiple distribution experiences).
The acquirer acquiring data includes the steps of:
the demander data management platform initiates a request according to a main body identifier, wherein the request comprises the main body identifier and can also comprise a Key value (Key), the Key value belongs to one element corresponding to the main body identifier, and the request is used for acquiring a specific assignment of the Key value corresponding to the main body identifier, and the specific assignment refers to another element corresponding to the main body identifier;
the demander front-end processor receives the request and buffers the request in the second buffer area; the requiring side front-end processor sends a collision invitation to the supplying side front-end processor; the supplier front-end processor receives the collision invitation, and transmits the main body identifier or the main body identifier cached in the first cache region and the corresponding element thereof to the second cache region to perform collision matching with the main body identifier in the request;
the successfully matched main body identification and the corresponding elements thereof are distributed to the demander data management platform through the demander front-end processor; and for the subject identification which is unsuccessfully matched, feeding back the query failure or sending the subject identification which is unsuccessfully matched to the supplier data management platform for re-matching, distributing the subject identification which is successfully re-matched and the corresponding elements thereof to the supplier data management platform through the supplier front-end processor and the demander front-end processor, and feeding back the query failure for the subject identification which is unsuccessfully re-matched.
Further, in the third data distribution embodiment, collision matching includes the following cases:
when the first cache region caches the subject identifier, the supplier front-end processor transmits the subject identifier cached in the first cache region to the second cache region for collision matching after receiving a collision invitation, and the demander front-end processor transmits the successfully matched subject identifier to the supplier data management platform through the supplier front-end processor to invoke the corresponding element, and distributes the successfully matched subject identifier and the corresponding element to the demander data management platform.
When the first cache region caches the subject identifier and the corresponding element thereof, the supplier front-end processor transmits the subject identifier cached in the first cache region and the corresponding element thereof to the second cache region for collision matching after receiving the collision invitation, and the demander front-end processor distributes the successfully matched subject identifier and the corresponding element thereof to the demander data management platform.
And for the subject identification which is unsuccessfully matched, feeding back query failure or sending the subject identification which is unsuccessfully matched to the supplier data management platform for re-matching, distributing the subject identification which is successfully re-matched and the corresponding elements thereof to the supplier data management platform through the supplier front-end processor and the demander front-end processor, and feeding back the query failure for the subject identification which is unsuccessfully re-matched.
Further, in the above three data distribution embodiments, if there is a principal identifier successfully matched with the principal identifier in the request in the first cache area of more than one supplier or the supplier data management platform, the demander selects one of the supplier providing elements according to the requirement.
In one embodiment, the header format of the data creation of the feedback may be:
the method comprises the following steps: exid | @ | connObjId- {1,2} | @ | connObjId- {3} | @ | connObjId
(eg:exid|@|DON00001010000000051|@|DON00001010000001196|@|DON00001010000001208)；
Data is extracted according to parameters, and the format of the data in the file can be as follows:
exid|@|111|@|222|@|4444；
(eg：5d6a7a48dd27fdc49a82c210e5880a30|@|32|@|2|@|4444)。
in the data distribution system provided by the present invention, the following maintenance method can be adopted in the data maintenance stage:
(1) the donor front-end processor maintains a system parameter < timeout >, which may specify a data retention time range. For example, the system parameter < timeout > may be:
[cache]；
redis _ host is the IP address of the front-end processor;
redis _ port ═ redis port number;
redis_db＝6；
cashe_time_limit＝40；
timeout＝900。
(2) the acquirer front-end processor records a transaction log as a transaction certificate;
(3) setting data Range (common order setting, and crowd packet filtering does not need to set the item), and the keyword sequence of the data Range field in the configuration file corresponds to the sequence of the file header connobj id of the file, for example, the data Range field may be as follows:
data_range＝DON00001010000000051,DON00001010000001196,DON00001010000001208。
in summary, in the caching method of the data distribution system provided by the invention, by setting the cache region, a large number of disk IO operations in the distribution system are avoided, so that requirements of millisecond-level low delay and high throughput of data distribution are met, and the problem that the distribution system is difficult to realize lateral expansion along with service development in the prior art is solved; the invention enables the distribution system to keep stable external service capability in the cache updating period on the premise of limited hardware resources and mass data by setting the independent cache region.
The above description is only a preferred embodiment of the present invention, and does not limit the present invention in any way. It will be understood by those skilled in the art that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. The cache method of the data distribution system is characterized in that the data distribution system comprises a supplier and a demander;
the caching method comprises the following steps: the supplier allocates a first cache region in the supplier terminal equipment for distribution, the first cache region caches the main body identifier or the main body identifier and the element corresponding to the main body identifier, the demander allocates a second cache region in the demander terminal equipment for distribution, and the second cache region caches the element corresponding to the main body identifier or the main body identifier and the main body identifier;
an demander data management platform in the data distribution system acquires elements corresponding to the main body identification according to the main body identification, a first cache region or a supplier data management platform matches the elements corresponding to the main body identification according to the main body identification, and the main body identification and the elements corresponding to the main body identification are distributed to the demander data management platform through a supplier terminal device and a demander terminal device; and the second cache region matches the elements corresponding to the subject identification according to the subject identification, and distributes the subject identification and the elements corresponding to the subject identification to the demander data management platform through the demander terminal equipment.
2. The caching method of a data distribution system of claim 1,
the supplier data management platform corresponds to at least one supplier terminal device, and the supplier terminal device is a supplier front-end processor;
the demander data management platform corresponds to at least one demander terminal device, and the demander terminal device is a demander front-end processor.
3. The caching method for the data distribution system according to claim 2, wherein the second cache region is cached with a principal identifier and its corresponding elements, and the acquirer data management platform acquires data includes the following steps:
the demander data management platform initiates a request according to a main body identifier, wherein the request comprises the main body identifier so as to obtain elements corresponding to the main body identifier;
the acquirer front-end processor receiving the request;
matching the main body identifier in the request with the main body identifier in the second cache region; if the second cache region has a subject identifier successfully matched with the subject identifier in the request and also has an element corresponding to the successfully matched subject identifier, feeding back the successfully matched subject identifier and the element corresponding to the successfully matched subject identifier to the demander data management platform; if the successfully matched main body identification does not exist or the successfully matched main body identification does not exist but the corresponding element exists, the request is sent to the front-end processor of the supplier;
the supplier front-end processor acquires the request, matches elements meeting the request in a collision manner at the supplier according to the request, and distributes the successfully matched subject identification and the corresponding elements to the demander data management platform through the supplier front-end processor and the demander front-end processor; and for the subject identification which is not matched successfully, the feedback query fails.
4. The caching method for the data distribution system according to claim 2, wherein the second cache area does not have a cache body id and its corresponding elements, and the acquirer acquiring the data comprises the steps of:
the demander data management platform initiates a request according to a main body identifier, wherein the request comprises the main body identifier so as to obtain an element corresponding to the main body identifier;
the requiring side front-end processor receives the request and sends the request to the supplying side front-end processor;
the supplier front-end processor acquires the request, matches elements according with the request in a collision way at the supplier according to the request, and distributes the successfully matched subject identification and the corresponding elements to the supplier data management platform through the supplier front-end processor and the supplier front-end processor; and for the subject identification which is not matched successfully, the feedback query fails.
5. The caching method of a data distribution system according to claim 3 or 4, wherein the supplier collision matching comprises the steps of:
matching the main body identifier in the request with the main body identifier in the first cache region;
when the main body identification and the corresponding element thereof are cached in the first cache region, the successfully matched main body identification and the corresponding element thereof are fed back to the demander front-end processor;
when the first cache region only caches the subject identification, the supplier front-end processor sends the successfully matched subject identification to the supplier data management platform to call the corresponding element, and feeds back the successfully matched subject identification and the corresponding element to the demander front-end processor;
and for the subject identification which is unsuccessfully matched, feeding back the query failure or sending the subject identification which is unsuccessfully matched to the supplier data management platform for re-matching, distributing the subject identification which is successfully re-matched and the corresponding elements thereof to the supplier data management platform through the supplier front-end processor and the demander front-end processor, and feeding back the query failure for the subject identification which is unsuccessfully re-matched.
6. The caching method for a data distribution system of claim 2, wherein said acquirer acquiring data further comprises the steps of:
the demander data management platform initiates a request according to a main body identifier, wherein the request comprises the main body identifier so as to obtain an element corresponding to the main body identifier;
the demander front-end processor receives the request and buffers the request in the second buffer area; the requiring side front-end processor sends a collision invitation to the supplying side front-end processor; and the supplier front-end processor receives the collision invitation, and transmits the main body identifier or the main body identifier cached in the first cache region and the corresponding element thereof to the second cache region to perform collision matching with the main body identifier in the request.
7. The data distribution system caching method of claim 6, wherein the collision match comprises:
when the main body identifier is cached in the first cache region, the supplier front-end processor transmits the main body identifier cached in the first cache region to the second cache region for collision matching after receiving a collision invitation, the demander front-end processor transmits the successfully matched main body identifier to the supplier data management platform through the supplier front-end processor to call corresponding elements, and distributes the successfully matched main body identifier and the corresponding elements to the demander data management platform;
when the first cache region caches the subject identifier and the corresponding element thereof, the supplier front-end processor transmits the subject identifier cached in the first cache region and the corresponding element thereof to the second cache region for collision matching after receiving a collision invitation, and the demander front-end processor distributes the successfully matched subject identifier and the corresponding element thereof to the demander data management platform;
and for the subject identification which is unsuccessfully matched, feeding back the query failure or sending the subject identification which is unsuccessfully matched to the supplier data management platform for re-matching, distributing the subject identification which is successfully re-matched and the corresponding elements thereof to the supplier data management platform through the supplier front-end processor and the demander front-end processor, and feeding back the query failure for the subject identification which is unsuccessfully re-matched.
8. The caching method of the data distribution system according to claim 3, wherein if the first cache area of more than one supplier or the supplier data management platform has a principal identifier successfully matched with the principal identifier in the request, the demander selects one of the supplier providing elements according to the request.
9. The data distribution system caching method of claim 3, wherein the data distributed by the data distribution system comprises a subject identifier in the request, elements that conform to the request, and interface information for the distribution.
10. The caching method of a data distribution system of claim 1,
the cache contents in the first cache region and the second cache region are updated according to appointed time;
and/or the data distributed by the data distribution system is created according to a specified format;
and/or the body identification comprises an encrypted body identification or an unencrypted body identification.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910537303.XA CN110244912B (en) | 2019-06-20 | 2019-06-20 | Caching method of data distribution system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910537303.XA CN110244912B (en) | 2019-06-20 | 2019-06-20 | Caching method of data distribution system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110244912A CN110244912A (en) | 2019-09-17 |
| CN110244912B true CN110244912B (en) | 2022-06-21 |
Family
ID=67888519
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910537303.XA Active CN110244912B (en) | 2019-06-20 | 2019-06-20 | Caching method of data distribution system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110244912B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103493029A (en) * | 2011-02-28 | 2014-01-01 | 甲骨文国际公司 | Universal cache management system |
| CN105512129A (en) * | 2014-09-24 | 2016-04-20 | 中国移动通信集团江苏有限公司 | Method and device for mass data retrieval, and method and device for storing mass data |
| US9582559B1 (en) * | 2012-06-29 | 2017-02-28 | EMC IP Holding Company LLC | Multi-site storage system with replicated file system synchronization utilizing virtual block storage appliances |
| CN106934044A (en) * | 2017-03-16 | 2017-07-07 | 北京深思数盾科技股份有限公司 | A kind of data processing method and device |
| CN107025591A (en) * | 2017-03-21 | 2017-08-08 | 天津数集科技有限公司 | Data trade platform |
| CN107153644A (en) * | 2016-03-02 | 2017-09-12 | 阿里巴巴集团控股有限公司 | A kind of method of data synchronization and device |
| CN107317855A (en) * | 2017-06-21 | 2017-11-03 | 努比亚技术有限公司 | A kind of data cache method, data request method and server |
| CN108153825A (en) * | 2017-12-07 | 2018-06-12 | 石化盈科信息技术有限责任公司 | Data access method and device |
-
2019
- 2019-06-20 CN CN201910537303.XA patent/CN110244912B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103493029A (en) * | 2011-02-28 | 2014-01-01 | 甲骨文国际公司 | Universal cache management system |
| US9582559B1 (en) * | 2012-06-29 | 2017-02-28 | EMC IP Holding Company LLC | Multi-site storage system with replicated file system synchronization utilizing virtual block storage appliances |
| CN105512129A (en) * | 2014-09-24 | 2016-04-20 | 中国移动通信集团江苏有限公司 | Method and device for mass data retrieval, and method and device for storing mass data |
| CN107153644A (en) * | 2016-03-02 | 2017-09-12 | 阿里巴巴集团控股有限公司 | A kind of method of data synchronization and device |
| CN106934044A (en) * | 2017-03-16 | 2017-07-07 | 北京深思数盾科技股份有限公司 | A kind of data processing method and device |
| CN107025591A (en) * | 2017-03-21 | 2017-08-08 | 天津数集科技有限公司 | Data trade platform |
| CN107317855A (en) * | 2017-06-21 | 2017-11-03 | 努比亚技术有限公司 | A kind of data cache method, data request method and server |
| CN108153825A (en) * | 2017-12-07 | 2018-06-12 | 石化盈科信息技术有限责任公司 | Data access method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110244912A (en) | 2019-09-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200327145A1 (en) | Cooperative naming for configuration items in a distributed configuration management database environment | |
| US6226650B1 (en) | Database synchronization and organization system and method | |
| CN110765484B (en) | Credit data processing method and electronic equipment | |
| US9135454B2 (en) | Systems and methods for enabling searchable encryption | |
| US20060173850A1 (en) | Method and apparatus for collision resolution in an asynchronous database system | |
| CN106886371B (en) | Caching data processing method and device | |
| KR20020090520A (en) | Parallel logging method of transaction processing system | |
| CN105516284B (en) | A kind of method and apparatus of Cluster Database distributed storage | |
| US11226982B2 (en) | Synchronization of offline instances | |
| CN106997557B (en) | Order information acquisition method and device | |
| US20070282878A1 (en) | System and method for online reorganization of a database using flash image copies | |
| CN101442558A (en) | Method and system for providing index service for P2SP network | |
| CN112835977A (en) | Database management method and system based on block chain | |
| CN109088858B (en) | Medical system and method based on authority management | |
| US20090100082A1 (en) | Replication and mapping mechanism for recreating memory durations | |
| CN110099128B (en) | Authentication data synchronization method and device | |
| CN116701330A (en) | Logistics information sharing method, device, equipment and storage medium | |
| CN101778131A (en) | Data synchronization system | |
| CN107277188A (en) | A kind of method, client, server and operation system for determining IP address attaching information | |
| CN103440302A (en) | Real-time data exchange method and system | |
| US10185735B2 (en) | Distributed database system and a non-transitory computer readable medium | |
| US11797556B2 (en) | Database management service provision system | |
| CN101789963A (en) | Data synchronization system | |
| CN110244912B (en) | Caching method of data distribution system | |
| CN116366353A (en) | Service login method, device, medium and platform based on X86 platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |