CN110781238B - Client view caching method and device based on combination of Redis and Hbase - Google Patents
Client view caching method and device based on combination of Redis and Hbase Download PDFInfo
- Publication number
- CN110781238B CN110781238B CN201910948964.1A CN201910948964A CN110781238B CN 110781238 B CN110781238 B CN 110781238B CN 201910948964 A CN201910948964 A CN 201910948964A CN 110781238 B CN110781238 B CN 110781238B
- Authority
- CN
- China
- Prior art keywords
- client view
- redis
- hbase
- client
- batch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Abstract
The invention discloses a client view caching method and device based on combination of Redis and Hbase, and relates to the technical field of computers. One embodiment of the method includes creating an Hbase based Hbase client view table, and a Redis based Redis client view table corresponding to a first client view; generating attribute values of all attributes in the real-time tag according to the client data so as to update the real-time tag in the Hbase client view table and the Redis client view table; generating attribute values of all attributes in the batch tags according to the client data so as to update the batch tags in the Hbase client view chart; after receiving the query client view request, writing the batch tag in the Hbase client view table to the batch tag in the Redis client view table according to the data structure of the Redis client view table. The implementation mode provides that a unique, unified and integrated customer view can be maintained, batch and real-time updating are both considered, and high-performance concurrent access is supported.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a client view caching method and device based on combination of Redis and Hbase.
Background
The client view is an information base containing characteristics of client natural attributes, client social attributes and the like. The financial industry, particularly the banking industry, is a representative and large-scale internet enterprises, the business category complexity is high, the customer group is large, the system access and the transaction amount are large, and the high-capacity sparse matrix such as the customer view needs to support the business scene with the characteristics of high-performance access, high response speed such as flow calculation application and large access amount, so that the design of the customer view needs to meet the requirements of batch and real-time updating and support high-performance concurrent access.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
the existing industry systems are generally designed for batch personalized recommendation and real-time personalized recommendation instead of taking a customer view as a core, so that a unique, unified and integrated customer view does not need to be maintained; in addition, the design of the existing system in the industry is usually separated from batch recommendation and real-time recommendation, and the problems that high-performance concurrent access cannot be supported, the combination of batch attributes and real-time attributes cannot be realized, and the like exist.
Disclosure of Invention
In view of this, embodiments of the present invention provide a client view caching method and apparatus based on a combination of Redis and Hbase, which can maintain a unique, unified, and integrated client view, satisfy batch and real-time update considerations, and support high-performance concurrent access.
To achieve the above object, according to an aspect of the embodiments of the present invention, there is provided a client view caching method based on combination of Redis and Hbase, including:
creating an Hbase based Hbase client view table and a Redis based Redis client view table corresponding to the first client view; the Hbase client view table and the Redis client view table include: static tags, batch tags, and real-time tags;
generating attribute values of all attributes in the real-time tags according to the client data so as to update the real-time tags in the Hbase client view table and the Redis client view table;
generating attribute values of all attributes in the batch tags according to the client data so as to update the batch tags in the Hbase client view chart;
after receiving the request for inquiring the client view, writing the batch label in the Hbase client view table into the batch label in the Redis client view table according to the data structure of the Redis client view table.
Optionally, after creating the Hbase client chart based on Hbase, further comprising: and setting the validity periods of the batch label and the real-time label in the Hbase client view chart.
Optionally, after creating a Redis client view chart based on the Redis corresponding to the first client view, the method further includes: the validity period of the Redis client view chart is set.
Optionally, the Redis client view graph employs a HashSet data structure.
According to a second aspect of the embodiments of the present invention, there is provided a client view caching apparatus based on combination of Redis and Hbase, including:
a creation module that creates an Hbase-based Hbase client view table and a Redis-based Redis client view table corresponding to a first client view; the Hbase client view table and the Redis client view table include: static tags, batch tags, and real-time tags;
the real-time label module is used for generating attribute values of all attributes in the real-time labels according to the client data so as to update the real-time labels in the Hbase client view table and the Redis client view table;
the batch label module is used for generating attribute values of all attributes in the batch labels according to the client data so as to update the batch labels in the Hbase client view chart;
and the cache module writes the batch tags in the Hbase client view chart into the batch tags in the Redis client view chart according to the data structure of the Redis client view chart after receiving the request for inquiring the client view.
Optionally, the creating module is further configured to: after the Hbase-based Hbase client view is created, the validity period of the batch tag and the real-time tag in the Hbase client view is set.
Optionally, the creating module is further configured to: after the Redis-based client view chart corresponding to the first client view is created, the validity period of the Redis client view chart is set.
Optionally, the Redis client view graph employs a HashSet data structure.
According to a third aspect of the embodiments of the present invention, there is provided a client view caching electronic device based on combination of Redis and Hbase, including:
one or more processors;
a storage device to store one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method provided by the first aspect of the embodiments of the present invention.
According to a fourth aspect of embodiments of the present invention, there is provided a computer-readable medium on which there is stored a computer program which, when executed by a processor, performs the method provided by the first aspect of embodiments of the present invention.
One embodiment of the above invention has the following advantages or benefits: by creating a customer view table comprising static labels, batch labels and real-time labels, a unique, uniform and integrated customer view can be maintained, and batch and real-time updating are both satisfied; the client view table is created based on Hbase and Redis, and a cache mechanism is adopted, so that high-performance concurrent access can be supported; by setting the validity period of the label, the timeliness of the label can be guaranteed, the overdue residual label is avoided, and a special label cleaning tool is not needed.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic main flow diagram of a client view caching method based on Redis and Hbase combination according to an embodiment of the present invention;
FIG. 2 is a schematic illustration of updating a real-time tag in an alternative embodiment of the invention;
FIG. 3 is a schematic diagram of a method of accessing a cached client view in accordance with an embodiment of the present invention;
FIG. 4 is a schematic diagram of the main modules of a client view caching apparatus based on Redis and Hbase combination according to an embodiment of the present invention;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 6 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
To achieve the above object, according to an aspect of the embodiments of the present invention, a client view caching method based on combination of Redis and Hbase is provided.
Fig. 1 is a schematic main flow diagram of a client view caching method based on combination of Redis and Hbase according to an embodiment of the present invention, and as shown in fig. 1, the client view caching method based on combination of Redis and Hbase includes: step S101, step S102, step S103, and step S104.
Creating an Hbase-based Hbase client view table and a Redis-based Redis client view table corresponding to the first client view in step S101; the Hbase client view table and the Redis client view table each include: static tags, batch tags, and real-time tags.
HBase is a distributed, column-oriented open-ended database. Static labels refer to attributes whose value remains constant for a certain period of time or for a long period of time, such as user name, age, gender, identification (identification number, identity code, facial profile, fingerprint, etc.), etc. Batch tags refer to a class of attributes that do not require real-time updates, such as attributes that are updated once per day or week. The real-time tag refers to a type of attribute that needs to be updated in real time, such as the current deposit balance of the user. It should be noted that the above is only an exemplary illustration of the static tag, the batch tag and the real-time tag, and in the practical application process, a person skilled in the art may determine that each attribute is classified into the static tag, the batch tag or the real-time tag according to the practical situation. In addition, the number of the batch tags can be one or more according to different update frequencies or management requirements, for example, the attribute updated every day corresponds to one batch tag, and the attribute updated every week corresponds to another batch tag.
Further, after creating the Hbase based Hbase client chart, the method may further include: and setting the validity periods of the batch label and the real-time label in the Hbase client view chart. Therefore, the timeliness of the label can be guaranteed, overdue residue of the label is avoided, and a special label cleaning tool is not needed.
Redis is an open source log-type and Key-Value database which is written by using ANSI C language, supports network, can be based on memory and can also be persistent. The Redis client view table created based on Redis corresponds to the Hbase client view table one by one, and the data structures of the Redis client view table and the Hbase client view table are different. The data structure of the Redis client view diagram can be selectively determined according to actual conditions, and optionally, the Redis client view diagram adopts a HashSet data structure, so that the non-repetition of each element in the Redis client view diagram can be ensured.
Further, after creating a Redis client view table based on the Redis corresponding to the first client view, the method may further include: the validity period of the Redis client view chart is set. Therefore, the timeliness of the Redis client view table cached in Redis can be guaranteed, overdue survival is avoided, and a special cleaning tool is not needed.
Illustratively, the table structure of the client chart of Hbase for client CustA is designed as follows:
table name: tablename A
Columns (columniamy, one tag for each column):
Column-Family: PI # personal information, i.e. static tags
Column-Family: CN # composition information, i.e. real-time tags,
Column-Family: SL1 #: the validity period of the batch label is 1 day, and the label disappears after 1 day
Column-Family: SL3 #: the validity period of the batch label is 3 days, and the label disappears after 1 day
Column-Family: SL7 #: the validity period of the batch label is 7 days, and the label disappears after 7 days
Column-Family: SL30 #: the validity period of the batch label is 30 days, and the label disappears after 30 days
Curt _ ID # Explanation: a customer unique identifier, a customer ID, i.e. a static tag.
The Hbase client chart is shown in table 1 below:
TABLE 1 Hbase client chart
Wherein the Hbase client view table may automatically expire according to the columnfmaly setting, see table 2 below:
TABLE 2 expiry date of the respective tags
The corresponding Redis client view chart adopts a HashSet data structure and is mapped as follows:
TableName as Key Key.
Column-Famly. Column is a label
Value of Value
The Redis client View Table of the example CustA contains the information as in Table 3 below:
TABLE 3 Tab for Redis customer View Chart
In addition, the validity period of the Redis client view table may be set to 1 hour or 12 hours expired. For example, in order to reduce the system pressure, a 12-hour expiration is set. As shown in table 4 below:
TABLE 4 expiration date for Redis client View Table
According to the invention, a unique, unified and integrated customer view can be maintained by creating the customer view chart containing the static label, the batch label and the real-time label, and batch and real-time updating are both satisfied.
And step S102, generating attribute values of all attributes in the real-time tags according to the client data so as to update the real-time tags in the Hbase client view table and the Redis client view table.
Fig. 2 is a schematic diagram of updating a real-time tag in an alternative embodiment of the invention. Referring to fig. 2, streaming data such as Kafka (an open source streaming processing platform) is client data, and after attribute values of respective attributes in a real-time tag are calculated by a Topo Bolt (a node in Storm for processing data) calculated by a Storm stream (a streaming data framework for processing a large amount of data in a fault-tolerant and horizontally scalable method), the attribute values of the respective attributes in the real-time tag are pushed to Bolt _ Hbase (a node for assembling data written in Hbase) and Bolt _ Redis (a node for assembling data written in Redis). Wherein, the Bolt _ Hbase reassembles the data and writes the data into the Hbase; and (4) the Bolt _ Redis reassembles the data and writes the data into the Redis. At this time, the Bolt _ Redis can set the validity time of the key each time data is written.
And step S103, generating attribute values of all attributes in the batch labels according to the client data so as to update the batch labels in the Hbase client view chart.
The method for calculating and updating the attribute value of each attribute in the batch label can be selectively set according to the actual situation, which is not limited by the invention. Exemplarily, this can be done as follows:
after client data is loaded into a data warehouse such as a Hive (data warehouse) through an ETL (Extract-Transform-Load) process, various indexes of the client such as age and transaction times of the last 3 months are extracted and calculated through a data analysis process such as SQL (Structured Query Language), and then the data is imported into an Hbase client view chart in a BulkLoad (a data writing tool in Hbase) manner.
It should be noted that the order of steps S102 and S103 may be reversed or performed simultaneously in the present invention. In the invention, the attribute value of the real-time tag is written into the Hbase client view table and the Redis client view table in real time, but the attribute value of the batch tag is only updated into the Hbase client view table, so that the client data is cached conveniently.
And step S104, after receiving the request for inquiring the client view, writing the batch tags in the Hbase client view chart into the batch tags in the Redis client view chart according to the data structure of the Redis client view chart.
FIG. 3 is a schematic diagram of a method of accessing cached client views in accordance with an embodiment of the present invention. As shown in fig. 3, the main steps of accessing the cached client view according to the embodiment of the present invention include:
1. the access service queries Redis according to the CustID to see whether a fixed attribute PI.L001 (age) exists;
2. and if the client view table exists, returning the Redis client view table which is checked by the Redis, wherein the Redis client view table comprises batch labels and real-time labels.
3. And if not, inquiring the Hbase client view table corresponding to the fixed attribute PI.L001 from the Hbase, including the batch label and the real-time label, and then returning and writing the inquired batch label into the Redis client view table.
After receiving a request for inquiring the client view, the invention writes the batch label in the Hbase client view table into the batch label in the Redis client view table. When the view charts of the same client are inquired subsequently, the query can be firstly carried out in a Redis cache, and the Hbase is inquired when the cache is not hit, so that high-performance concurrent access can be supported.
According to a second aspect of the embodiments of the present invention, a client view caching apparatus based on combination of Redis and Hbase is provided.
Fig. 4 is a schematic diagram of main modules of a client view caching apparatus based on combination of Redis and Hbase according to an embodiment of the present invention, and as shown in fig. 4, a client view caching apparatus 400 based on combination of Redis and Hbase includes:
a creating module 401 that creates an Hbase-based Hbase client view table and a Redis-based Redis client view table corresponding to the first client view; the Hbase client view table and the Redis client view table include: static tags, batch tags, and real-time tags;
a real-time tag module 402, configured to generate attribute values of each attribute in a real-time tag according to the client data, so as to update the real-time tag in the Hbase client view table and the Redis client view table;
a batch label module 403, which generates attribute values of each attribute in a batch label according to the client data to update the batch label in the Hbase client view table;
the caching module 404, after receiving the query client view request, writes the batch tags in the Hbase client view table into the batch tags in the Redis client view table according to the data structure of the Redis client view table.
Optionally, the creating module is further configured to: after creating the Hbase client chart based on Hbase, the validity period of the batch tag and the real-time tag in the Hbase client chart is set.
Optionally, the creating module is further configured to: after creating the Redis-based Redis client view chart corresponding to the first client view, a validity period of the Redis client view chart is set.
Optionally, the Redis client view graph employs a HashSet data structure.
According to a third aspect of the embodiments of the present invention, there is provided a client view caching electronic device based on combination of Redis and Hbase, including:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method provided by the first aspect of the embodiments of the present invention.
According to a fourth aspect of embodiments of the present invention, there is provided a computer readable medium, on which a computer program is stored, which when executed by a processor, implements the method provided by the first aspect of embodiments of the present invention.
Fig. 5 illustrates an exemplary system architecture 500 to which the client view caching method based on the combination of Redis and Hbase or the client view caching apparatus based on the combination of Redis and Hbase according to an embodiment of the present invention may be applied.
As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 is the medium used to provide communication links between terminal devices 501, 502, 503 and the server 505. Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 501, 502, 503 to interact with a server 505 over a network 504 to receive or send messages or the like. The terminal devices 501, 502, 503 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 501, 502, 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 505 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the terminal devices 501, 502, 503. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.
It should be noted that the client view caching method based on the combination of the Redis and the Hbase provided by the embodiment of the present invention is generally executed by the server 505, and accordingly, the client view caching apparatus based on the combination of the Redis and the Hbase is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks, and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, a block diagram of a computer system 600 suitable for use with a terminal device implementing embodiments of the present invention is shown. The terminal device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609 and/or installed from the removable medium 611. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 601.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present invention, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor comprising: a creation module that creates an Hbase-based Hbase client view table and a Redis-based Redis client view table corresponding to a first client view; the real-time tag module is used for generating attribute values of all attributes in the real-time tag according to the client data so as to update the real-time tag in the Hbase client view table and the Redis client view table; the batch label module is used for generating attribute values of all attributes in the batch labels according to the client data so as to update the batch labels in the Hbase client view chart; and the cache module writes the batch tags in the Hbase client view chart into the batch tags in the Redis client view chart according to the data structure of the Redis client view chart after receiving the request for inquiring the client view. Where the names of these modules do not in some cases constitute a limitation of the module itself, for example, the caching module may also be described as a "module that creates an Hbase based Hbase client view table".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise:
creating an Hbase based Hbase client view table and a Redis based Redis client view table corresponding to the first client view; the Hbase client view table and the Redis client view table include: static tags, batch tags, and real-time tags;
generating attribute values of all attributes in the real-time tag according to the client data so as to update the real-time tag in the Hbase client view table and the Redis client view table;
generating attribute values of all attributes in the batch tags according to the client data so as to update the batch tags in the Hbase client view chart;
after receiving the query client view request, writing the batch tag in the Hbase client view table to the batch tag in the Redis client view table according to the data structure of the Redis client view table.
According to the technical scheme of the embodiment of the invention, a unique, unified and integrated customer view can be maintained by creating the customer view chart containing the static label, the batch label and the real-time label, so that both batch updating and real-time updating are satisfied; the client view table is created based on Hbase and Redis, and a cache mechanism is adopted, so that high-performance concurrent access can be supported; by setting the validity period of the label, the timeliness of the label can be guaranteed, the overdue residual label is avoided, and a special label cleaning tool is not needed.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A client view caching method based on combination of Redis and Hbase is characterized by comprising the following steps:
creating an Hbase based Hbase client view table and a Redis based Redis client view table corresponding to the first client view; the Hbase client view table and the Redis client view table include: static tags, batch tags, and real-time tags; the Redis client view diagram created based on Redis corresponds to the Hbase client view diagram one by one;
generating attribute values of all attributes in the real-time tags according to client data so as to update the real-time tags in an Hbase client view table and a Redis client view table;
generating attribute values of all attributes in the batch tags according to client data so as to update the batch tags in the Hbase client view chart;
after receiving a request for inquiring a client view, writing the batch label in the Hbase client view table into the batch label in the Redis client view table according to the data structure of the Redis client view table.
2. The client view caching method of claim 1, wherein after creating the Hbase based Hbase client view graph, further comprising: and setting the validity periods of the batch label and the real-time label in the Hbase client view chart.
3. The client view caching method of claim 1, wherein after creating the Redis-based Redis client view graph corresponding to the first client view, further comprising: and setting the validity period of the Redis client view table.
4. The client view caching method of claim 1, wherein the Redis client view graph employs a HashSet data structure.
5. A client view caching device based on Redis and Hbase combination is characterized by comprising:
a creation module that creates an Hbase-based Hbase client view table and a Redis-based Redis client view table corresponding to a first client view; the Hbase client view table and the Redis client view table include: static tags, batch tags, and real-time tags; the Redis client view diagram created based on Redis corresponds to the Hbase client view diagram one by one;
the real-time label module is used for generating attribute values of all attributes in the real-time labels according to client data so as to update the real-time labels in the Hbase client view table and the Redis client view table;
the batch label module is used for generating attribute values of all attributes in the batch labels according to client data so as to update the batch labels in the Hbase client view chart;
and the cache module writes the batch tags in the Hbase client view chart into the batch tags in the Redis client view chart according to the data structure of the Redis client view chart after receiving a client view query request.
6. The client view caching apparatus of claim 5, wherein the creation module is further configured to: after creating the Hbase client view chart based on the Hbase, setting the validity period of the batch label and the real-time label in the Hbase client view chart.
7. The client view caching apparatus of claim 5, wherein the creation module is further to: after a Redis-based client view chart corresponding to a first client view is created, a validity period of the Redis client view chart is set.
8. The client view caching apparatus of claim 5, wherein the Redis client view graph employs a HashSet data structure.
9. A client view caching electronic device based on Redis and Hbase combination, comprising:
one or more processors;
a storage device to store one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method recited in any of claims 1-4.
10. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910948964.1A CN110781238B (en) | 2019-10-08 | 2019-10-08 | Client view caching method and device based on combination of Redis and Hbase |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910948964.1A CN110781238B (en) | 2019-10-08 | 2019-10-08 | Client view caching method and device based on combination of Redis and Hbase |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110781238A CN110781238A (en) | 2020-02-11 |
| CN110781238B true CN110781238B (en) | 2022-09-13 |
Family
ID=69384817
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910948964.1A Active CN110781238B (en) | 2019-10-08 | 2019-10-08 | Client view caching method and device based on combination of Redis and Hbase |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110781238B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112634030A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Method for obtaining real-time deposit index and stream computing platform |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103473267A (en) * | 2013-08-09 | 2013-12-25 | 深圳市中科新业信息科技发展有限公司 | Data storage query method and system |
| CN106202503A (en) * | 2016-07-19 | 2016-12-07 | 北京百分点信息科技有限公司 | Data processing method and device |
| CN106484763A (en) * | 2015-09-02 | 2017-03-08 | 雅虎公司 | system and method for merging data |
| CN108664549A (en) * | 2018-03-29 | 2018-10-16 | 阿里巴巴集团控股有限公司 | A kind of big data processing system, method and apparatus |
| CN108932846A (en) * | 2017-05-22 | 2018-12-04 | 中兴通讯股份有限公司 | A kind of licence plate recognition method, equipment and computer readable storage medium |
| CN108959337A (en) * | 2018-03-22 | 2018-12-07 | 中国平安人寿保险股份有限公司 | Big data acquisition methods, device, equipment and storage medium |
| CN109284351A (en) * | 2018-08-14 | 2019-01-29 | 青海大学 | A kind of data query method based on HBase database |
| CN109460404A (en) * | 2018-09-03 | 2019-03-12 | 中新网络信息安全股份有限公司 | A kind of efficient Hbase paging query method based on redis |
| CN109726205A (en) * | 2018-12-26 | 2019-05-07 | 国电南瑞科技股份有限公司 | A kind of power system security stability analysis data-storage system and storage method |
| CN109871367A (en) * | 2019-02-28 | 2019-06-11 | 江苏实达迪美数据处理有限公司 | A kind of distributed cold and heat data separation method based on Redis and HBase |
| CN110069495A (en) * | 2019-03-13 | 2019-07-30 | 中科恒运股份有限公司 | Date storage method, device and terminal device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160306822A1 (en) * | 2015-04-17 | 2016-10-20 | Samsung Electronics Co., Ltd. | Load balancing of queries in replication enabled ssd storage |
-
2019
- 2019-10-08 CN CN201910948964.1A patent/CN110781238B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103473267A (en) * | 2013-08-09 | 2013-12-25 | 深圳市中科新业信息科技发展有限公司 | Data storage query method and system |
| CN106484763A (en) * | 2015-09-02 | 2017-03-08 | 雅虎公司 | system and method for merging data |
| CN106202503A (en) * | 2016-07-19 | 2016-12-07 | 北京百分点信息科技有限公司 | Data processing method and device |
| CN108932846A (en) * | 2017-05-22 | 2018-12-04 | 中兴通讯股份有限公司 | A kind of licence plate recognition method, equipment and computer readable storage medium |
| CN108959337A (en) * | 2018-03-22 | 2018-12-07 | 中国平安人寿保险股份有限公司 | Big data acquisition methods, device, equipment and storage medium |
| CN108664549A (en) * | 2018-03-29 | 2018-10-16 | 阿里巴巴集团控股有限公司 | A kind of big data processing system, method and apparatus |
| CN109284351A (en) * | 2018-08-14 | 2019-01-29 | 青海大学 | A kind of data query method based on HBase database |
| CN109460404A (en) * | 2018-09-03 | 2019-03-12 | 中新网络信息安全股份有限公司 | A kind of efficient Hbase paging query method based on redis |
| CN109726205A (en) * | 2018-12-26 | 2019-05-07 | 国电南瑞科技股份有限公司 | A kind of power system security stability analysis data-storage system and storage method |
| CN109871367A (en) * | 2019-02-28 | 2019-06-11 | 江苏实达迪美数据处理有限公司 | A kind of distributed cold and heat data separation method based on Redis and HBase |
| CN110069495A (en) * | 2019-03-13 | 2019-07-30 | 中科恒运股份有限公司 | Date storage method, device and terminal device |
Non-Patent Citations (2)
| Title |
|---|
| Storage and retrieval of massive heterogeneous IoT data based on hybrid storage;Shanshan Wu 等;《2017 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD)》;20180625;2982-2987 * |
| 基于手机位置数据的用户移动模式挖掘与分析;凌峰;《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》;20180315(第3期);I136-1028 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110781238A (en) | 2020-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11620400B2 (en) | Querying in big data storage formats | |
| US10521404B2 (en) | Data transformations with metadata | |
| CN110019350B (en) | Data query method and device based on configuration information | |
| US11681651B1 (en) | Lineage data for data records | |
| CN109034988B (en) | Accounting entry generation method and device | |
| US10956438B2 (en) | Catalog with location of variables for data | |
| US10360394B2 (en) | System and method for creating, tracking, and maintaining big data use cases | |
| CN111865970B (en) | Method and apparatus for implementing interface idempotency | |
| CN110019367B (en) | Method and device for counting data characteristics | |
| CN112835904A (en) | Data processing method and data processing device | |
| WO2022237764A1 (en) | Data processing method and system | |
| CN110781238B (en) | Client view caching method and device based on combination of Redis and Hbase | |
| CN111949678A (en) | Method and device for processing non-accumulation indexes across time windows | |
| CN113641706A (en) | Data query method and device | |
| US20170220656A1 (en) | Information Access System | |
| CN107886350B (en) | Method and device for analyzing data | |
| CN112507017A (en) | Data table conversion method and device | |
| CN113760966A (en) | Data processing method and device based on heterogeneous database system | |
| CN112817930A (en) | Data migration method and device | |
| CN116450622B (en) | Method, apparatus, device and computer readable medium for data warehouse entry | |
| CN113360494B (en) | Wide-table data generation method, updating method and related device | |
| CN109446183B (en) | Global anti-duplication method and device | |
| CN113450170A (en) | Information display method and device | |
| CN114756540A (en) | Data processing method and device, electronic equipment and computer readable medium | |
| CN116346760A (en) | Message processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220921 Address after: 25 Financial Street, Xicheng District, Beijing 100033 Patentee after: CHINA CONSTRUCTION BANK Corp. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Patentee before: CHINA CONSTRUCTION BANK Corp. Patentee before: Jianxin Financial Science and Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right |