CN112637305B - Data storage and query method, device, equipment and medium based on cache - Google Patents

Data storage and query method, device, equipment and medium based on cache Download PDF

Info

Publication number
CN112637305B
CN112637305B CN202011488712.4A CN202011488712A CN112637305B CN 112637305 B CN112637305 B CN 112637305B CN 202011488712 A CN202011488712 A CN 202011488712A CN 112637305 B CN112637305 B CN 112637305B
Authority
CN
China
Prior art keywords
message
query
user
information
cache
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011488712.4A
Other languages
Chinese (zh)
Other versions
CN112637305A (en
Inventor
谈叶勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Consumer Finance Co Ltd
Original Assignee
Ping An Consumer Finance Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Consumer Finance Co Ltd filed Critical Ping An Consumer Finance Co Ltd
Priority to CN202011488712.4A priority Critical patent/CN112637305B/en
Publication of CN112637305A publication Critical patent/CN112637305A/en
Application granted granted Critical
Publication of CN112637305B publication Critical patent/CN112637305B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/55—Push-based network services
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/248—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9538—Presentation of query results
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/06—Notations for structuring of protocol data, e.g. abstract syntax notation one [ASN.1]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computational Linguistics (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention discloses a data storage and query method, a device, a storage medium and computer equipment based on cache, when receiving push messages of a service system, reassembles the messages and stores the assembled messages to a background database respectively according to different user dimensions; when the front end triggers the inquiry, the second-level storage of the push message is carried out according to inquiry categories of different user dimensions, and the returned inquiry result of the second-level storage content is based on the cache of the Redis ZSET data structure, and the in-station message list based on time sequencing provided by ZSET is utilized, so that a large number of concurrent inquiry and sequencing operations on the database level are avoided, and the performance of the system is improved.

Description

Data storage and query method, device, equipment and medium based on cache
Technical Field
The invention belongs to the technical field of Internet, and particularly relates to a data storage and query method, a data storage and query device, computer equipment and a storage medium based on a Redis caching mechanism.
Background
The CF-OPR system carries the information (station internal information and short message) of all the gold-eliminating service systems for storage and pushing, wherein the station internal information is divided into single user station internal information (i.e. single user information) and group pushing/group pushing station internal information (i.e. group information), and the group pushing/group pushing station internal information needs to support pushing targets with different dimensions, including pushing according to groups, pushing according to labels and the like; the CF-OPR needs to provide a message query interface for real-time query of the front-end system, and the query interface needs to have higher concurrency and performance support.
Because the message volume of the CF-OPR is huge, hundreds of thousands or even millions of new messages are added every day, the service is accumulated continuously and the data volume is also increased continuously along with the continuous increase of the user volume and the service volume in the system, and at the moment, the server responds more and more slowly, and the throughput of the system suffers from bottlenecks. Therefore, how to reasonably design the storage, archiving and inquiring interface schemes of the messages, and simultaneously reduce the system cost under the condition of ensuring the interface performance and concurrency requirements is an urgent problem to be solved.
Disclosure of Invention
The invention aims to provide a data storage and query method based on cache, which aims to solve the problems existing in the prior art.
In order to achieve the above object, the present invention provides a data storage and query method based on cache, comprising the steps of:
s1, receiving push information of a service system;
s2, after analyzing and verifying the format of the push message, the message is reassembled according to a preset message format; s3, storing the assembled information to a background database according to different user dimensions; the user dimension comprises a single user message, a group push message and a short message;
s4, when the front end triggers the query, the push message is stored in a second level according to the query types of different user dimensions, and a query result is returned;
S41, inquiring a Redis cache and returning the cached information in the Redis cache as an inquiry result if the inquiry category is judged to be paging inquiry;
and S42, if the query class is judged to be the forced pull latest message, carrying out secondary storage of message query and push message according to the user dimension, and feeding back a query result, wherein the user dimension comprises a single user message and a group push message.
Preferably, if the user dimension is a single user message query, checking whether zset of a specified message type of a specified user exists in the Redis cache;
if not, inquiring all unexpired messages which meet the appointed inquiry conditions in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages as inquiry results;
if the information is stored, comparing the latest information date T1 in the database with the maximum date T2 of the Redis cached information, if T1 is more than T2, inquiring information records with the information date more than T2 in the database, adding the information records into the Redis cache, and returning updated Redis cache information as an inquiry result; if T1 is less than T2, the Redis cached information is returned as a query result.
Further, the latest message date T1 in the database is born time (occurrence time) in the single-user basic information table, the maximum date T2 of the rediscached message is version value VA corresponding to the user in the single-user information table, a message record with a message date greater than T2 is queried from the database and added to the Redis cache, and the updated Redis cache message is returned as a query result, including:
A. Combine all messages of the bornTime > VA in the inquiry single user basic information table, and record the inquiry time T;
B. writing the message obtained in the step (A) into a Redis cache;
C. updating a version value of a current user in the single user information table to be T;
D. and taking out the latest count record from the cache according to the input count value, and assembling and returning a result after reverse serialization.
Preferably, if the user dimension is a group push message query, checking whether zset of a specified message type of a specified user exists in the dis cache:
if not, inquiring all unexpired messages which meet the appointed inquiry conditions in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages as inquiry results;
if so, obtaining and storing the mapping relation of the current user and the group information indicated in the query condition from the query request, inquiring the corresponding information from the database according to the mapping relation of the group information, filling the information into a Redis cache, and returning the updated Redis cache information as a query result.
Further, obtaining and saving the mapping relation of the current user and the group information indicated in the query condition from the query request, querying the corresponding message from the database according to the mapping relation of the group information, filling the message into the Redis cache, and returning the updated Redis cache message as a query result, wherein the method comprises the following steps:
a. Inquiring a station internal message ID which is in a group message basic information table, is in a born time > VA and is matched with the owner information, and recording the inquiring time T; VA is a version value corresponding to a user in a group user information table, and the born time is obtained from a group basic information table;
b. newly adding a group push message and the association record of the current user for each acquired ID, and concatenating as unread;
c. writing the intra-station information obtained in the step (a) into a Redis cache;
d. updating the version value of the current user in the group user information table to be T;
e. and taking out the latest count record from the cache according to the input count value, and assembling and returning a result after reverse serialization.
Preferably, in the step S4, the query type is determined according to a value of maxTime, if maxtime=null, it is determined that the pull is forced, and if maxTime is the minimum value of born time in the last returned message, it is determined that the page is queried.
Preferably, in the step S42, after determining the user dimension, it is further determined whether the corresponding Redis cache is available, if so, the processing is shifted to the step S221 or the step S222 according to the user dimension type, and if not, the record meeting the condition in the database is directly queried and returned to the caller.
Preferably, in the step S2, the preset message body format includes: a topic field, an upstream system identification tag field, an upstream intra-system message unique identification key field, and a message body content field.
Preferably, in step S3, storing the data in the background database according to the user dimensions includes:
s31, if the push message is a group message request, storing the message and group owner information of the message into a database, wherein one group message corresponds to one storage record in the database;
s32, if the push message is a single-user message, storing the message itself into a database;
and S33, if the push message is a short message, storing the message into a database.
The invention also provides a data storage and query device based on cache, which comprises:
the push message receiving module is used for receiving push messages of the service system;
the message processing module is used for analyzing and verifying the format of the push message and then re-assembling the message according to the preset message body format;
the primary storage module is used for respectively storing the assembled messages into a background database according to different user dimensions, wherein the user dimensions comprise single user messages, group push messages and short messages;
The second-level storage and query return module is used for carrying out the second-level storage of the push message according to the query types of different user dimensions when the front end triggers the query, and returning a query result according to the second-level storage content, and comprises the following steps:
the paging inquiry sub-module is used for inquiring the Redis cache and returning the cached information in the Redis cache as an inquiry result if the inquiry category is judged to be paging inquiry;
and the forced pulling query sub-module is used for carrying out secondary storage of message query and push message according to the user dimension and feeding back a query result if the query class is judged to be the forced pulling latest message, wherein the user dimension comprises single user message and group push message.
As a preferred solution, the single-user message query unit is configured to check whether zset of a specified message type of a specified user in the dis cache exists if the user dimension is a single-user message query; if not, inquiring all unexpired messages which meet the appointed inquiry conditions in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages as inquiry results; if the information is stored, comparing the latest information date T1 in the database with the maximum date T2 of the Redis cached information, if T1 is more than T2, inquiring information records with the information date more than T2 in the database, adding the information records into the Redis cache, and returning updated Redis cache information as an inquiry result; if T1 is less than T2, the Redis cached information is returned as a query result.
Further, in the single-user message query unit, the latest message date T1 in the database is born time in the single-user basic information table, the maximum date T2 of the Redis cached message is version value VA corresponding to the user in the single-user information table, the message record with the message date greater than T2 is queried from the database and added to the Redis cache, and the updated Redis cache message is returned as a query result, including:
A. combine all messages of the bornTime > VA in the inquiry single user basic information table, and record the inquiry time T;
B. writing the message obtained in the step (A) into a Redis cache;
C. updating the version value of the current user in the single user information table to be T;
D. and taking out the latest count record from the cache according to the input count value, and assembling and returning a result after reverse serialization.
As a preferred solution, the group push query unit is configured to check whether zset of a specified message type of a specified user in the Redis cache exists if the user dimension is a group push message query: if not, inquiring all unexpired messages which meet the appointed inquiry conditions in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages as inquiry results; if so, obtaining and storing the mapping relation of the current user and the group information indicated in the query condition from the query request, inquiring the corresponding information from the database according to the mapping relation of the group information, filling the information into a Redis cache, and returning the updated Redis cache information as a query result.
Further, in the group push query unit, a mapping relationship between a current user and group information indicated in a query condition is obtained and stored from the query request, a corresponding message is queried from the database according to the mapping relationship of the group information, and then the message is filled into a dis cache, and the updating dis cache message is returned as a query result, which includes:
a. inquiring a station internal message ID which is in a group message basic information table, is in a born time > VA and is matched with the owner information, and recording the inquiring time T; VA is a version value corresponding to a user in a group user information table, and the born time is obtained from a group basic information table;
b. newly adding a group push message and the association record of the current user for each acquired ID, and concatenating as unread;
c. writing the intra-station information obtained in the step (a) into a Redis cache;
d. updating the version value of the current user in the group user information table to be T;
e. and taking out the latest count record from the cache according to the input count value, and assembling and returning a result after reverse serialization.
As a preferred solution, the secondary storage and query return module further includes a query class determination module, configured to determine the query class according to a value of maxTime, determine that pulling is forced if maxtime=null, and determine that paging query is performed if maxTime is a minimum value of born time in the last returned message.
As a preferred solution, the forced pull query sub-module further includes a dis cache judging sub-module, configured to further judge whether a corresponding dis cache is available after determining the user dimension, and if so, switch to step S221 or step S222 for processing according to the user dimension category, and if not, directly query a record in the database that satisfies a condition and return to the caller.
As a preferred solution, in the message processing module, the preset message body format includes: a topic field, an upstream system identification tag field, an upstream intra-system message unique identification key field, and a message body content field.
As a preferred solution, the primary storage module is configured to store the primary storage modules to a background database according to different user dimensions, and includes:
the group message request storage submodule is used for storing the message and the group owner information of the message into a database when the push message is a group message request, wherein one group message corresponds to one storage record in the database;
the single-user message storage sub-module is used for storing the message itself into a database if the push message is a single-user message;
And the short message storage sub-module is used for storing the message itself into a database if the push message is a short message.
The invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of any of the methods described above when the computer program is executed.
The invention also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of any of the methods described above.
According to the data storage and query method, device, computer equipment and storage medium based on cache, the cache of the Redis ZSET data structure is based in the whole scheme design, a sequencing tool (such as a RangeByScore API) provided by ZSET is utilized to provide an intra-station message list based on time sequencing, so that a large number of concurrent query and sequencing operations at a database level are avoided, and the performance of a system is improved.
The CF-OPR is used as an independent message processing system aiming at the group push/group push message, and does not maintain the relationship information of users and groups in the service system; when the CF-OPR receives a group message request pushed by a service system, only the message and the group owner information of the message are stored, and one group message corresponds to one storage record in a database; when the front end triggers the inquiry of the group push/group push station internal message, the CF-OPR obtains and stores the mapping relation between the current user and the group information indicated in the inquiry condition from the inquiry request, and then inquires out the corresponding information from the background storage and returns the information to the calling party.
In addition, a unified and extensible message body format is designed for different types of service messages of the gold eliminating service system, and is compatible with single user/group messages.
Drawings
FIG. 1 is a schematic workflow diagram of an embodiment of a cache-based data storage and query method according to the present invention;
FIG. 2 is a schematic diagram illustrating an embodiment of a data storage and query method based on cache according to the present invention;
FIG. 3 is a schematic workflow diagram of an embodiment of a cache-based data storage and query method of the present invention;
FIG. 4 is a schematic diagram of a cache-based data storage and query device according to the present invention;
fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the scope of the invention in any way.
Like reference numerals refer to like elements throughout the specification. The expression "and/or" includes any and all combinations of one or more of the associated listed items. In the drawings, the thickness, size and shape of the object have been slightly exaggerated for convenience of explanation. The figures are merely examples and are not drawn to scale.
It will be further understood that the terms "comprises," "comprising," "includes," "including," "having," "containing," and/or "including," when used in this specification, specify the presence of stated features, steps, integers, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, integers, operations, elements, components, and/or groups thereof.
The terms "substantially," "about," and the like as used in the specification are used as terms of approximation, not as terms of degree, and are intended to account for inherent deviations in measured or calculated values that will be recognized by one of ordinary skill in the art.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
Example 1
As shown in fig. 1 and 2, the invention discloses a data storage and query method based on cache, which comprises the following steps:
s1, receiving push information of a service system;
s2, after analyzing and verifying the format of the push message, the message is reassembled according to a preset message format; s3, respectively storing the assembled messages to a background database (DB store) according to different user dimensions; the user dimension comprises a single user message, a group push message and a short message;
s4, when the front end triggers the query, the push message is stored in a second level according to the query types of different user dimensions, and a query result is returned;
s41, inquiring a Redis cache and returning the cached information in the Redis cache as an inquiry result if the inquiry category is judged to be paging inquiry;
and S42, if the query class is judged to be the forced pull latest message, carrying out secondary storage of message query and push message according to the user dimension, and feeding back a query result, wherein the user dimension comprises a single user message and a group push message.
The invention discloses a data storage and query method based on cache, which is characterized in that when the query interface is designed, the cache based on Redis ZSET (ordered set) data structure is used, a JSET provided RangeByScore API (a sort tool in ZSET) is used for providing a time-sorting-based in-station message list, so that a large number of concurrent query and sort operations on a database layer are avoided, and the performance of a system is improved.
Example two
As shown in fig. 3, in this embodiment, the data storage and query method based on cache according to the present invention specifically includes the following steps:
s1, receiving push information of a service system;
in this step, the CF-OPR system receives a message pushed by an upstream system through the MQ, the upstream system sends an asynchronous push message to the CF-OPR system through the MQ, the CF-OPR asynchronously consumes the MQ message and sends a processing result to the MQ after processing, and the upstream system monitors the MQ to obtain a processing result of the original request.
S2, after analyzing and verifying the format of the push message, the message is reassembled according to the preset message format.
After the CF-OPR system receives the information of the upstream system, the unified information body is analyzed, and then template assembly and storage are carried out according to the preset information body format, as a preferred scheme, the query method based on the data cache, which is shown in the invention, designs a unified and extensible preset information body format for different types of service information of the gold eliminating service system, and is compatible with single user/group information, wherein the preset information body format sequentially comprises the following steps: an msgId field for describing a unique identification field of a system message, a topic field for describing a message to MQ, a tag field for describing an upstream system identification, a key field for describing a unique identification of a message in an upstream system, an orderKey field for describing a sort key of a sequential message, and a content field for describing a message body. The specific composition of the preset message body format and the functions of the fields are shown in the following table:
Figure BDA0002840104910000111
Taking the example of respectively buffering single user messages and group messages, the specific description of the preset message body format is as follows: a score is generated in each zset according to the generation time of the message; the key composition of zset is as follows:
single user message: pea_ INSITEMSG _my_ { $app_name } _ { $identity_type } _ { identity };
group message: pea_ INSITEMSG _sys_ { $app_name } _ { $identity_type } _ { identity };
for example, user a logs in the universal App with the handset number 13809467826, when the OPR system caches the personal message zset key=pea_ INSITEMSG _my_puapp_mobile_13809467826 for the user, and when the user has a new message (the generation time is T1, and the serialized message body is "json 1"), the user writes the new message into the cache through add pea_ INSITEMSG _my_puapp_mobile_13809467826 T1"json1".
S3, according to different user dimensions, the assembled messages are respectively stored in a background database, wherein the user dimensions comprise single user messages, group messages and short messages.
In this embodiment, different storage modes are adopted for the pushed messages according to different user dimensions. When storing the messages, the user dimension includes short messages besides the single user messages and the group messages, and the assembled messages are respectively stored in a background database, which specifically includes the following steps:
And S31, if the push message is judged to be a group message request, storing the message and the group owner information of the message into a database, wherein one group message corresponds to one storage record in the database.
In step S31, the parameters in the message body field are analyzed to identify that the push message is a group message request, then a message template is assembled, the placeholder in the template is replaced with the specified value in the message body, a group message corresponds to a storage record in the database, and finally the assembled message is stored. In this way, for the service requirement of the group push/group push station internal letter, the CF-OPR stores the message itself and the group owner information of the message when the group push station internal letter is sent to the upstream system, and does not store the corresponding relation between the user and the group in advance (storing each corresponding relation can multiply the total record number of the background database, and most mapping relations cannot be used, but when the front end triggers the inquiry of the group push/group push station internal letter, the system stores the corresponding relation between the current login user and the group information indicated in the current inquiry condition, and returns to the caller after inquiring the message meeting the condition, so that the CF-OPR only serves as an independent message processing system without maintaining the user and group relation information in the service system.
And S32, if the push message is judged to be a single-user message or a short message, storing the message into a database. Wherein, the single user message only stores the message itself, the short message is the same as the single user message, also stores the message itself,
and S4, when the front end triggers the query, carrying out secondary storage of the message according to the query types of different user dimensions and returning a query result according to the query request.
In step S4, the query class includes forced pull query and paging query; as a preferred scheme, the specific query category is judged by reading the maxtime value. If maxtime=null, it is determined to be forced pull, and if maxTime is the minimum value of born time in the last returned message, it is determined to be paging inquiry.
S41, if the query category is the paging query, querying a Redis cache and returning the cached information in the Redis cache as a query result.
In this embodiment, when maxTime is the minimum value of borne time in the last returned message, it is determined that the paging query message of a single user or a group user is a paging query message, and at this time, the OPR directly queries the redis cache through zrangebyscore, and returns the result.
And S42, if the query class is judged to be the forced pull latest message, carrying out secondary storage of message query and push message according to the user dimension, and feeding back a query result, wherein the user dimension comprises a single user message and a group push message.
When the front-end trigger query is a single-user message query and the query class is forced pull latest message, the system checks whether zset of the designated message type (multi-user) of the designated user (identity type, identity mark) exists in the dis cache. The single user message only stores the message itself, when the inquiry is triggered, if the message of the user is found not cached before, the inquiry is carried out from the database, and then the message is cached again, so as to carry out the secondary storage of the push message.
If zset does not exist, S4211 queries all unexpired messages in the database meeting the specified query conditions and fills in the Redis cache.
S4212, if zset exists, comparing the latest message date T1 in the database with the maximum date T2 of the Redis cached message, if T1> T2, inquiring the message record with the message date larger than T2 in the database, adding the message record to the Redis cache, and returning the updated Redis cache message as an inquiry result.
Further, in step S4212, querying the database for a message record with a message date greater than T2 and adding the message record to the Redis cache comprises:
A. combine all messages of the bornTime > VA in the inquiry single user basic information table, and record the inquiry time T;
B. writing the message obtained in the step (A) into a Redis cache;
C. Updating the version value of the current user in the single user information table to be T;
D. and taking out the latest count record from the cache according to the input count value, and assembling and returning a result after reverse serialization.
As shown in fig. 3, the latest message date T1 in the database is borne time in the single-user basic information table, the maximum date T2 of the rediscached message is a version value VA corresponding to the user in the single-user information table, the single-user basic information table is an opr_instete_msg table and an opr_instete_msg_detail table, and the single-user information table is an opr_instete_msg_user_version table.
And/or, if the user dimension is the group push message query and the query class is the forced pull latest message, S422 checks whether zset exists for the specified message type of the specified user in the Redis cache.
The group message is a storage message and an owner, when the inquiry is triggered, a mapping relation is acquired and stored in the dbstore, corresponding data is searched from the dbstore according to the mapping relation and returned to perform secondary storage of the push message, and when the user dimension is judged to be the inquiry of the group push message and the inquiry category is forced pulling of a new message, the system checks whether zset of a designated message type (multi-user) of a designated user (identity type and identity identification) in the Redis cache exists or not.
If zset does not exist, S4221 queries all unexpired messages in the database meeting the specified query conditions and fills in the Redis cache.
If zset exists, S4222 obtains and stores the mapping relationship between the current user and the group information indicated in the query condition from the query request, and then queries the corresponding message from the database, and returns the corresponding message to the caller, which includes: inquiring the born time > VA, putting the information matched with the owner information into a Redis cache, and returning an inquiring result.
Further, in step S4222, the message for querying the born time > VA and the owner information is matched is put into the Redis cache, and the returning of the query result includes:
a. inquiring a station internal message ID which is in a group message basic information table, is in a born time > VA and is matched with the owner information, and recording the inquiring time T; and VA is a version value corresponding to a user in the group user information table, and the born time is obtained from the group basic information table.
b. And newly adding a group of push messages and the association record of the current user for each acquired ID, and collocating as unread.
c. Writing the intra-station information acquired in the step (a) into a Redis cache.
d. And updating the version value of the current user in the group user information table to be T.
e. And taking out the latest count record from the cache according to the input count value, and assembling and returning a result after reverse serialization.
As shown in fig. 3, VA is a version value corresponding to a user in a group user information table, the borne time is obtained from a group basic information table, the group basic information table is an opr_inste_msg and opr_inste_msg_audios table, and the group user information table is an opr_inste_msg_user_version table. According to the data storage and query method based on the cache, when the front end triggers the query of the group push/group push station internal messages, the mapping relation of the current user and the group information indicated in the query condition is obtained and stored from the query request, and the corresponding message is queried from the database and returned to the calling party, so that the time sequencing-based internal message list (such as RangeByScore API) provided by ZSET is utilized, a large number of concurrent query and sequencing operations on the database level are avoided, and the performance of the system is improved.
Example III
As shown in fig. 4, the present invention further provides a data storage and query device 10 based on cache, including:
a push message receiving module 11, configured to receive a push message of a service system;
the message processing module 12 is configured to parse and verify the format of the push message, and reassemble the message according to a preset message format;
The primary storage module 13 is used for respectively storing the assembled messages into a background database according to different user dimensions, wherein the user dimensions comprise single user messages, group push messages and short messages;
the secondary storage and query return module 14 is configured to perform secondary storage of the push message according to query types of different user dimensions when the front end triggers a query, and return a query result according to the secondary storage content, where the secondary storage and query return module includes:
the paging inquiry sub-module is used for inquiring the Redis cache and returning the cached information in the Redis cache as an inquiry result if the inquiry category is judged to be paging inquiry;
and the forced pulling query sub-module is used for carrying out secondary storage of message query and push message according to the user dimension and feeding back a query result if the query class is judged to be the forced pulling latest message, wherein the user dimension comprises single user message and group push message.
The data storage and query device 10 based on the cache, which is shown in the invention, has the advantages that the data volume of the message table in the station is large and a large number of new records are added every day due to the large concurrency of the message query interface in the station, the cache based on the Redis ZSET data structure is designed when the query interface is designed, the ZSET is utilized to provide the message list (such as RangeByScore API) based on time ordering, a large number of concurrency query and ordering operations on the database level are avoided, and the performance of the system is improved.
As a preferred solution, the forced pulling query sub-module includes a single-user message query unit, configured to check whether zset of a specified message type of a specified user in the dis cache exists if the user dimension is a single-user message query; if not, inquiring all unexpired messages which meet the appointed inquiry conditions in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages as inquiry results; if the information is stored, comparing the latest information date T1 in the database with the maximum date T2 of the Redis cached information, if T1 is more than T2, inquiring information records with the information date more than T2 in the database, adding the information records into the Redis cache, and returning updated Redis cache information as an inquiry result; if T1 is less than T2, the Redis cached information is returned as a query result.
Further, in the single-user message query unit, the latest message date T1 in the database is born time in the single-user basic information table, the maximum date T2 of the Redis cached message is version value VA corresponding to the user in the single-user information table, the message record with the message date greater than T2 is queried from the database and added to the Redis cache, and the updated Redis cache message is returned as a query result, including:
A. Combine all messages of the bornTime > VA in the inquiry single user basic information table, and record the inquiry time T;
B. writing the message obtained in the step (A) into a Redis cache;
C. updating the version value of the current user in the single user information table to be T;
D. and taking out the latest count record from the cache according to the input count value, and assembling and returning a result after reverse serialization.
As a preferred solution, the forced pull query sub-module further includes a group push query unit, configured to check whether zset of a specified message type of a specified user in the dis cache exists if the user dimension is a group push message query: if not, inquiring all unexpired messages which meet the appointed inquiry conditions in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages as inquiry results; if so, obtaining and storing the mapping relation of the current user and the group information indicated in the query condition from the query request, inquiring the corresponding information from the database according to the mapping relation of the group information, filling the information into a Redis cache, and returning the updated Redis cache information as a query result.
Further, in the group push query unit, a mapping relationship between a current user and group information indicated in a query condition is obtained and stored from the query request, a corresponding message is queried from the database according to the mapping relationship of the group information, and then the message is filled into a dis cache, and the updating dis cache message is returned as a query result, which includes:
a. Inquiring a station internal message ID which is in a group message basic information table, is in a born time > VA and is matched with the owner information, and recording the inquiring time T; VA is a version value corresponding to a user in a group user information table, and the born time is obtained from a group basic information table;
b. newly adding a group push message and the association record of the current user for each acquired ID, and concatenating as unread;
c. writing the intra-station information obtained in the step (a) into a Redis cache;
d. updating the version value of the current user in the group user information table to be T;
e. and taking out the latest count record from the cache according to the input count value, and assembling and returning a result after reverse serialization.
As a preferred solution, the secondary storage and query return module further includes a query class determination module, configured to determine the query class according to a value of maxTime, determine that pulling is forced if maxtime=null, and determine that paging query is performed if maxTime is a minimum value of born time in the last returned message.
As a preferred solution, the forced pull query sub-module further includes a dis cache judging sub-module, configured to further judge whether a corresponding dis cache is available after determining the user dimension, if so, transfer to a single user message query unit or a group push query unit for processing according to the user dimension category, and if not, directly query a record meeting a condition in a database and return to a caller.
As a preferred solution, the primary storage module 13 is configured to store the primary storage data to the background database according to different user dimensions, and includes:
the group message request storage submodule is used for storing the message and the group owner information of the message into a database when the push message is a group message request, wherein one group message corresponds to one storage record in the database;
the single-user message storage sub-module is used for storing the message itself into a database if the push message is a single-user message;
and the short message storage sub-module is used for storing the message itself into a database if the push message is a short message.
Example IV
Fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present invention, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack-mounted server, a blade server, a tower server, or a rack-mounted server (including an independent server or a server cluster formed by multiple servers), which may execute a program. The computer device 20 of the present embodiment includes at least, but is not limited to: a memory 21, a processor 22, which may be communicatively coupled to each other via a system bus, as shown in fig. 5. It should be noted that fig. 5 only shows a computer device 20 having components 21-22, but it should be understood that not all of the illustrated components are required to be implemented, and that more or fewer components may be implemented instead.
In this embodiment, the memory 21 (i.e., readable storage medium) includes a Flash memory, a hard disk, a multimedia Card, a Card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), and the like, and the memory 21 may also be an external storage device of the computer device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card), etc. which are provided on the computer device 20. Of course, the memory 21 may also include both internal storage units of the computer device 20 and external storage devices. In this embodiment, the memory 21 is typically used to store an operating system and various application software installed on the computer device 20, such as program codes of a cache-based data storage and query device in the method embodiment. Further, the memory 21 may be used to temporarily store various types of data that have been output or are to be output.
The processor 22 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 22 is generally used to control the overall operation of the computer device 20. In this embodiment, the processor 22 is configured to execute the program code stored in the memory 21 or process data, for example, execute the cache-based data storage and query device 10, to implement the cache-based data storage and query method in the method embodiment.
Example five
The present application also provides a computer readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., having stored thereon a computer program that when executed by a processor performs a corresponding function. The computer readable storage medium of the present embodiment is used for storing the data storing and querying device 10 based on the cache, and when executed by a processor, implements the data storing and querying method based on the cache of the method embodiment.
It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.

Claims (10)

1. A data storage and query method based on cache is characterized in that: the method comprises the following steps:
s1, receiving push information of a service system;
s2, after analyzing and verifying the format of the push message, the message is reassembled according to a preset message format;
s3, storing the assembled information to a background database according to different user dimensions;
s4, when the front end triggers the query, the push message is stored in a second level according to the query types of different user dimensions, and a query result is returned;
s41, inquiring a Redis cache and returning the cached information in the Redis cache as an inquiry result if the inquiry category is judged to be paging inquiry;
and S42, if the query class is judged to be the forced pull latest message, carrying out secondary storage of message query and push message according to the user dimension, and feeding back a query result, wherein the user dimension comprises a single user message and a group push message.
2. A cache-based data storage and querying method according to claim 1, wherein: if the user dimension is single-user message query, checking whether zset of the designated message type of the designated user exists in the Redis cache;
If not, inquiring all unexpired messages which meet the appointed inquiry conditions in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages as inquiry results;
if the information is stored, comparing the latest information date T1 in the database with the maximum date T2 of the Redis cached information, if T1 is more than T2, inquiring information records with the information date more than T2 in the database, adding the information records into the Redis cache, and returning updated Redis cache information as an inquiry result; if T1 is less than T2, the Redis cached information is returned as a query result.
3. A cache-based data storage and querying method according to claim 2, wherein: the latest information date T1 in the database is born Time in the single-user basic information table, the maximum date T2 of the Redis cached information is version value VA corresponding to the user in the single-user information table, the information record with the information date larger than T2 is queried from the database and added into the Redis cache, and the updated Redis cache information is returned as a query result and comprises the following steps:
A. combine all messages of the bornTime > VA in the inquiry single user basic information table, and record the inquiry time T;
B. writing the information obtained in the step A into a Redis cache;
C. Updating the version value of the current user in the single user information table to be T;
D. and taking out the latest count record from the cache according to the input count value, and assembling a return result after reverse serialization, wherein the count is the record value of each query.
4. A cache-based data storage and querying method according to claim 1, wherein: if the user dimension is the group push message query, checking whether zset of the designated message type of the designated user exists in the Redis cache:
if not, inquiring all unexpired messages which meet the appointed inquiry conditions in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages as inquiry results;
if so, obtaining and storing the mapping relation between the current user and the group information indicated in the query condition from the query request, inquiring the corresponding information from the database according to the mapping relation of the group information, filling the information into the Redis cache, and returning the updated Redis cache information as a query result.
5. The cache-based data storage and querying method as claimed in claim 4, wherein: obtaining and storing the mapping relation of the current user and the group information indicated in the query condition from the query request, inquiring the corresponding information from the database according to the mapping relation of the group information, filling the information into a Redis cache, and returning the updated Redis cache information as a query result, wherein the method comprises the following steps:
a. Inquiring a station internal message ID which is in a group message basic information table, is in a born time > VA and is matched with the owner information, and recording the inquiring time T; VA is a version value corresponding to a user in a group user information table, and the born time is obtained from a group basic information table;
b. newly adding a group push message and the association record of the current user for each acquired ID, and concatenating as unread;
c. writing the intra-station information obtained in the step a into a Redis cache;
d. updating the version value of the current user in the group user information table to be T;
e. and taking out the latest count record from the cache according to the input count value, and assembling a return result after reverse serialization, wherein the count is the record value of each query.
6. A cache-based data storage and querying method according to claim 1, wherein: in the step S4, the query class is determined according to the value of maxTime, if maxtime=null, it is determined that pulling is forced, and if maxTime is the minimum value of born time in the last returned message, it is determined that paging query is performed.
7. A cache-based data storage and querying method according to claim 1, wherein: the method is characterized in that: in the step S3, the user dimension includes a single user message, a group push message and a short message; storing the data to a background database according to different user dimensions comprises the following steps:
S31, if the push message is a group message request, storing the message and group owner information of the message into a database, wherein one group message corresponds to one storage record in the database;
s32, if the push message is a single-user message, storing the message itself into a database;
and S33, if the push message is a short message, storing the message into a database.
8. A data storage and query device based on cache, characterized in that: comprising the following steps:
the push message receiving module is used for receiving push messages of the service system;
the message processing module is used for analyzing and verifying the format of the push message and then re-assembling the message according to the preset message body format;
the primary storage module is used for respectively storing the assembled messages into a background database according to different user dimensions, wherein the user dimensions comprise single user messages, group push messages and short messages;
the second-level storage and query return module is used for carrying out the second-level storage of the push message according to the query types of different user dimensions when the front end triggers the query, and returning a query result according to the second-level storage content, and comprises the following steps:
The paging inquiry sub-module is used for inquiring the Redis cache and returning the cached information in the Redis cache as an inquiry result if the inquiry category is judged to be paging inquiry;
and the forced pulling query sub-module is used for carrying out secondary storage of message query and push message according to the user dimension and feeding back a query result if the query class is judged to be the forced pulling latest message, wherein the user dimension comprises single user message and group push message.
9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized by: the processor, when executing the computer program, implements the steps of the method of any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon a computer program, characterized by: the computer program implementing the steps of the method of any of claims 1 to 7 when executed by a processor.
CN202011488712.4A 2020-12-16 2020-12-16 Data storage and query method, device, equipment and medium based on cache Active CN112637305B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011488712.4A CN112637305B (en) 2020-12-16 2020-12-16 Data storage and query method, device, equipment and medium based on cache

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011488712.4A CN112637305B (en) 2020-12-16 2020-12-16 Data storage and query method, device, equipment and medium based on cache

Publications (2)

Publication Number Publication Date
CN112637305A CN112637305A (en) 2021-04-09
CN112637305B true CN112637305B (en) 2023-05-02

Family

ID=75313940

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011488712.4A Active CN112637305B (en) 2020-12-16 2020-12-16 Data storage and query method, device, equipment and medium based on cache

Country Status (1)

Country Link
CN (1) CN112637305B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113486099A (en) * 2021-06-23 2021-10-08 华泰证券股份有限公司 Memory computing service framework and implementation system
CN113343048A (en) * 2021-06-30 2021-09-03 中国农业银行股份有限公司 Data query method and device and readable storage medium
CN113626483B (en) * 2021-08-18 2022-04-26 重庆允成互联网科技有限公司 Front-end caching method, system, equipment and storage medium for filling forms
CN114422593B (en) * 2021-12-30 2024-03-15 杭州数梦工场科技有限公司 Data distribution method, device, electronic equipment and storage medium
CN115314423A (en) * 2022-07-19 2022-11-08 杭州安恒信息技术股份有限公司 Traffic data statistical method, device and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109710639A (en) * 2018-11-26 2019-05-03 厦门市美亚柏科信息股份有限公司 A kind of search method based on pair buffers, device and storage medium
CN110213380A (en) * 2019-06-11 2019-09-06 深圳市思迪信息技术股份有限公司 A kind of information push method, device, computer equipment and storage medium
CN110502327A (en) * 2019-08-28 2019-11-26 四川长虹电器股份有限公司 Method based on the processing of Redis high concurrent delayed tasks
CN110674432A (en) * 2019-09-09 2020-01-10 中国平安财产保险股份有限公司 Second-level caching method and device and computer readable storage medium
CN110909025A (en) * 2018-09-17 2020-03-24 深圳市优必选科技有限公司 Database query method, query device and terminal

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10187445B2 (en) * 2012-12-17 2019-01-22 Beijing Qihoo Technology Company Limited System, method and browser client for enabling browser data synchronization

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110909025A (en) * 2018-09-17 2020-03-24 深圳市优必选科技有限公司 Database query method, query device and terminal
CN109710639A (en) * 2018-11-26 2019-05-03 厦门市美亚柏科信息股份有限公司 A kind of search method based on pair buffers, device and storage medium
CN110213380A (en) * 2019-06-11 2019-09-06 深圳市思迪信息技术股份有限公司 A kind of information push method, device, computer equipment and storage medium
CN110502327A (en) * 2019-08-28 2019-11-26 四川长虹电器股份有限公司 Method based on the processing of Redis high concurrent delayed tasks
CN110674432A (en) * 2019-09-09 2020-01-10 中国平安财产保险股份有限公司 Second-level caching method and device and computer readable storage medium

Also Published As

Publication number Publication date
CN112637305A (en) 2021-04-09

Similar Documents

Publication Publication Date Title
CN112637305B (en) Data storage and query method, device, equipment and medium based on cache
CN110674432B (en) Second-level caching method, device and computer readable storage medium
CN109344197B (en) Paging downloading method and device based on big data
CN112162965B (en) Log data processing method, device, computer equipment and storage medium
CN109388657B (en) Data processing method, device, computer equipment and storage medium
US20100057460A1 (en) Verbal labels for electronic messages
CN110457346B (en) Data query method, device and computer readable storage medium
US8805848B2 (en) Systems, methods and computer program products for fast and scalable proximal search for search queries
CN110659971B (en) Transaction data processing method and device
CN104199978A (en) System and method for realizing metadata cache and analysis based on NoSQL and method
CN114547095A (en) Data rapid query method and device, electronic equipment and storage medium
CN113010542A (en) Service data processing method and device, computer equipment and storage medium
US6816879B1 (en) System and method for serving pre-generated presentation views of requested data
CN115455058A (en) Cache data processing method and device, computer equipment and storage medium
CN116628018A (en) Data query method, device, server and storage medium
CN110032578B (en) Mass data query caching method and device
US11321375B2 (en) Text object management system
CN111858581B (en) Paging query method and device, storage medium and electronic equipment
CN113641769B (en) Data processing method and device
CN114064729B (en) Data retrieval method, device, equipment and storage medium
CN111221817B (en) Service information data storage method, device, computer equipment and storage medium
US11847149B1 (en) Message processing for job title extraction
CN113535770B (en) Data query method and device
CN112286970B (en) Information processing method, system, computer device and computer readable storage medium
CN111079391B (en) Report generation method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant