CN112637305A - Data storage and query method, device, equipment and medium based on cache - Google Patents

Data storage and query method, device, equipment and medium based on cache Download PDF

Info

Publication number
CN112637305A
CN112637305A CN202011488712.4A CN202011488712A CN112637305A CN 112637305 A CN112637305 A CN 112637305A CN 202011488712 A CN202011488712 A CN 202011488712A CN 112637305 A CN112637305 A CN 112637305A
Authority
CN
China
Prior art keywords
message
query
user
cache
group
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011488712.4A
Other languages
Chinese (zh)
Other versions
CN112637305B (en
Inventor
谈叶勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Consumer Finance Co Ltd
Original Assignee
Ping An Consumer Finance Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Consumer Finance Co Ltd filed Critical Ping An Consumer Finance Co Ltd
Priority to CN202011488712.4A priority Critical patent/CN112637305B/en
Publication of CN112637305A publication Critical patent/CN112637305A/en
Application granted granted Critical
Publication of CN112637305B publication Critical patent/CN112637305B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/55Push-based network services
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24552Database cache management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/248Presentation of query results
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/953Querying, e.g. by the use of web search engines
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/953Querying, e.g. by the use of web search engines
    • G06F16/9538Presentation of query results
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • H04L67/568Storing data temporarily at an intermediate stage, e.g. caching
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/06Notations for structuring of protocol data, e.g. abstract syntax notation one [ASN.1]
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computational Linguistics (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention discloses a data storage and query method, a device, a storage medium and computer equipment based on cache.A message is reassembled and the reassembled message is respectively stored in a background database according to different user dimensions after a push message of a service system is received; when the front end triggers query, secondary storage of the push message is carried out according to query types of different user dimensions, a query result is returned according to secondary storage contents, caching is carried out on the basis of a Redis ZSE data structure, and a station mail list based on time sequencing provided by ZSE is utilized, so that a large amount of concurrent query and sequencing operations of a database layer are avoided, and the performance of the system is improved.

Description

Data storage and query method, device, equipment and medium based on cache
Technical Field
The invention belongs to the technical field of internet, and particularly relates to a data storage and query method and device based on a Redis cache mechanism, computer equipment and a storage medium.
Background
The CF-OPR system carries information (in-station information and short messages) storage and pushing of all the delinquent service systems, wherein the in-station information is divided into single-user station information (namely single-user information) and group pushing/group pushing station information (namely group information), and the group pushing/group pushing station information needs to support pushing targets with different dimensions, including pushing according to groups, pushing according to labels and the like; the CF-OPR also needs to provide a message query interface for the real-time query of a front-end system, and the query interface also needs to have higher concurrency and performance support.
Due to the huge message volume of the CF-OPR, hundreds of thousands or even millions of new messages are generated every day, with the continuous increase of the user volume and the service volume in the system, the data volume is also increased while the service is continuously accumulated, at the moment, the server response is slower and slower, and the throughput of the system is subjected to bottleneck. Therefore, how to reasonably design the interface scheme for storing, archiving and inquiring messages is an urgent problem to be solved while reducing the system cost under the condition of ensuring the interface performance and the concurrent requirements.
Disclosure of Invention
The invention aims to provide a data storage and query method based on cache to solve the problems in the prior art.
In order to achieve the above object, the present invention provides a data storage and query method based on cache, which comprises the following steps:
s1, receiving a push message of a service system;
s2, after analyzing and verifying the format of the push message, reassembling the message according to the preset message format; s3, storing the assembled messages into a background database according to different user dimensions; the user dimension comprises single user messages, group push messages and short messages;
s4, when the front end triggers the query, according to the query types of different user dimensions, performing secondary storage of the push message, and returning a query result;
s41, if the query type is judged to be paging query, querying a Redis cache and returning a message cached in the Redis cache as a query result;
and S42, if the query type is judged to be the forced pull of the latest message, performing secondary storage of message query and the push message according to the user dimension, and feeding back a query result, wherein the user dimension comprises a single-user message and a group push message.
Preferably, if the user dimension is a single-user message query, checking whether zset of a specified message type of a specified user in the Redis cache exists;
if not, inquiring all unexpired messages which accord with the specified inquiry condition in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages serving as inquiry results;
if the message exists, comparing the latest message date T1 in the database with the maximum date T2 of the Redis cached message, if T1> T2, inquiring the message record with the message date larger than T2 from the database, adding the message record to the Redis cache, and returning the updated Redis cache message as an inquiry result; if T1 < T2, the message that Redis has cached is returned as the query result.
Further, the latest message date T1 in the database is born time (occurrence time) in the single-user basic information table, the maximum date T2 of the Redis cached message is a version value VA corresponding to the user in the single-user information table, a message record with a message date greater than T2 is queried from the database and added to the Redis cache, and returning the updated Redis cache message as a query result includes:
A. jointly inquiring all messages of born time & gt VA in the single-user basic information table, and recording the inquiry time T;
B. writing the message acquired in the step (A) into a Redis cache;
C. updating the version value of the current user in the single-user information table to be T;
D. and taking out the latest count record from the cache according to the transmitted count value, and assembling and returning a result after deserialization.
Preferably, if the user dimension is a group push message query, checking whether zset of a specified message type of a specified user in the Redis cache exists:
if not, inquiring all unexpired messages which accord with the specified inquiry condition in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages serving as inquiry results;
if the query request exists, acquiring and storing the mapping relation between the current user and the group information indicated in the query condition from the query request, querying a corresponding message from the database according to the mapping relation of the group information, filling the message into a Redis cache, and returning the message for updating the Redis cache as a query result.
Further, obtaining and storing a mapping relationship between the current user and the group information indicated in the query condition from the query request, querying a corresponding message from the database according to the mapping relationship of the group information, filling the message into a Redis cache, and returning an updated Redis cache message as a query result, including:
a. inquiring the borne time & ltVA & gt in the group message basic information table and the station information ID matched with the owner information, and recording the inquiry time T; the VA is a version value corresponding to a user in the group user information table, and the born time is acquired from the group basic information table;
b. adding a group push message and a current user association record to each obtained ID, and juxtaposing the group push message and the current user association record as unread;
c. writing the in-station message acquired in the step (a) into a Redis cache;
d. updating the version value of the current user in the group user information table to be T;
e. and taking out the latest count record from the cache according to the transmitted count value, and assembling and returning a result after deserialization.
Preferably, in step S4, the query type is determined according to a value of maxTime, and if maxTime is null, it is determined as forced pull, and if maxTime is a minimum value of bornTime in a message returned last time, it is determined as a paging query.
Preferably, in step S42, after the user dimension is determined, it is further determined whether a corresponding Redis cache is available, if so, the process is switched to step S221 or step S222 according to the user dimension type, and if not, records satisfying the condition in the database are directly queried and returned to the caller.
Preferably, in step S2, the preset message format includes: a topic field, an upstream system identification tag field, an upstream system message unique identification key field and a message body content field.
Preferably, in step S3, the storing into the background database according to the user dimension includes:
s31, if the push message is a group message request, storing the message itself and the group owner information of the message in a database, wherein one group message corresponds to one storage record in the database;
s32, if the push message is a single-user message, storing the message into a database;
s33, if the push message is a short message, storing the message itself in a database.
The invention also provides a data storage and query device based on cache, which comprises:
the push message receiving module is used for receiving a push message of a service system;
the message processing module is used for analyzing and verifying the format of the push message and then reassembling the message according to the preset message format;
the primary storage module is used for respectively storing the assembled messages into a background database according to different user dimensions, wherein the user dimensions comprise single-user messages, group push messages and short messages;
the secondary storage and query returning module is used for performing secondary storage on the push message according to query categories of different user dimensions when the front end triggers query, and returning a query result according to secondary storage contents, and comprises:
the paging query submodule is used for querying a Redis cache and returning a cached message in the Redis cache as a query result if the query type is judged to be paging query;
and the forced pull query submodule is used for performing secondary storage of message query and the push message according to the user dimension and feeding back a query result if the query type is judged to be the forced pull latest message, wherein the user dimension comprises a single-user message and a group push message.
As a preferred scheme, the single-user message querying unit is configured to check whether zset of a specified message type of a specified user in the Redis cache exists, if the user dimension is a single-user message query; if not, inquiring all unexpired messages which accord with the specified inquiry condition in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages serving as inquiry results; if the message exists, comparing the latest message date T1 in the database with the maximum date T2 of the Redis cached message, if T1> T2, inquiring the message record with the message date larger than T2 from the database, adding the message record to the Redis cache, and returning the updated Redis cache message as an inquiry result; if T1 < T2, the message that Redis has cached is returned as the query result.
Further, in the single-user message query unit, a latest message date T1 in the database is a borne time in the single-user basic information table, a maximum date T2 of a Redis cached message is a version value VA corresponding to the user in the single-user information table, a message record with a message date greater than T2 is queried from the database and added to the Redis cache, and returning an updated Redis cache message as a query result includes:
A. jointly inquiring all messages of born time & gt VA in the single-user basic information table, and recording the inquiry time T;
B. writing the message acquired in the step (A) into a Redis cache;
C. updating the version value of the current user in the single-user information table to be T;
D. and taking out the latest count record from the cache according to the transmitted count value, and assembling and returning a result after deserialization.
As a preferred scheme, the group push querying unit is configured to, if the user dimension is a group push message query, check whether zset of a specified message type of a specified user in the Redis cache exists: if not, inquiring all unexpired messages which accord with the specified inquiry condition in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages serving as inquiry results; if the query request exists, acquiring and storing the mapping relation between the current user and the group information indicated in the query condition from the query request, querying a corresponding message from the database according to the mapping relation of the group information, filling the message into a Redis cache, and returning the message for updating the Redis cache as a query result.
Further, in the group push query unit, obtaining and storing a mapping relationship between the current user and the group information indicated in the query condition from the query request, querying a corresponding message from the database according to the mapping relationship of the group information, and then filling the message into the Redis cache, and returning an updated Redis cache message as a query result includes:
a. inquiring the borne time & ltVA & gt in the group message basic information table and the station information ID matched with the owner information, and recording the inquiry time T; the VA is a version value corresponding to a user in the group user information table, and the born time is acquired from the group basic information table;
b. adding a group push message and a current user association record to each obtained ID, and juxtaposing the group push message and the current user association record as unread;
c. writing the in-station message acquired in the step (a) into a Redis cache;
d. updating the version value of the current user in the group user information table to be T;
e. and taking out the latest count record from the cache according to the transmitted count value, and assembling and returning a result after deserialization.
As a preferred scheme, the secondary storage and query returning module further includes a query type determining module, configured to determine the query type according to a value of maxTime, and if maxTime is null, it is determined as forced pull, and if maxttime is a minimum value of born time in a last returned message, it is determined as paging query.
As a preferred scheme, the forced pull query submodule further includes a Redis cache determination submodule, configured to further determine whether a corresponding Redis cache is available after the user dimension is determined, if the corresponding Redis cache is available, the step S221 or the step S222 is performed according to the user dimension type, and if the corresponding Redis cache is unavailable, the record meeting the condition in the database is directly queried and returned to the caller.
As a preferred scheme, in the message processing module, the preset message format includes: a topic field, an upstream system identification tag field, an upstream system message unique identification key field and a message body content field.
As a preferred scheme, the primary storage module is configured to store to a background database according to different user dimensions, and includes:
a group message request storage submodule, configured to store the message itself and group owner information of the message in a database when the push message is a group message request, where one group message corresponds to one storage record in the database;
the single-user message storage submodule is used for storing the message into a database if the push message is a single-user message;
and the short message storage sub-module is used for storing the message to a database if the push message is a short message.
The invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of any of the methods described above when executing the computer program.
The invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of any of the methods described above.
According to the data storage and query method, device, computer equipment and storage medium based on the cache, cache of a Redis ZSE data structure is used in the whole scheme design, a sequencing tool (such as a RangeByScore API) provided by ZSE is used for providing the in-station credit list based on time sequencing, a large amount of concurrent query and sequencing operations of a database layer are avoided, and the performance of the system is improved.
The CF-OPR is used as an independent message processing system for group push/group push messages, and does not maintain the information of the relation between users and groups in a service system; when the CF-OPR receives a group message request pushed by a service system, only the message and the group owner information of the message are stored, and one group message corresponds to one storage record in a database; when the front end triggers the inquiry of the group push/group push station internal information, the CF-OPR obtains and stores the mapping relation between the current user and the group information indicated in the inquiry condition from the inquiry request, and then inquires out the corresponding information from the background storage and returns the corresponding information to the calling party.
In addition, aiming at different types of service messages of the fund-removing service system, a uniform and extensible message body format is designed, and the message is compatible with single user/group messages.
Drawings
FIG. 1 is a schematic diagram of a workflow of an embodiment of a cache-based data storage and query method according to the present invention;
FIG. 2 is a schematic diagram illustrating an operating state of an embodiment of a cache-based data storage and query method according to the present invention;
FIG. 3 is a schematic view of a working flow of an embodiment of a cache-based data storage and query method according to the present invention;
FIG. 4 is a schematic structural diagram of a cache-based data storage and query apparatus according to the present invention;
fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of the invention in any way.
Like reference numerals refer to like elements throughout the specification. The expression "and/or" includes any and all combinations of one or more of the associated listed items. In the drawings, the thickness, size, and shape of an object have been slightly exaggerated for convenience of explanation. The figures are purely diagrammatic and not drawn to scale.
It will be further understood that the terms "comprises," "comprising," "includes," "including," "has," "includes" and/or "including," when used in this specification, specify the presence of stated features, steps, integers, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, integers, operations, elements, components, and/or groups thereof.
The terms "substantially", "about" and the like as used in the specification are used as terms of approximation and not as terms of degree, and are intended to account for inherent deviations in measured or calculated values that would be recognized by one of ordinary skill in the art.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
Example one
As shown in fig. 1 and fig. 2, the present invention discloses a data storage and query method based on cache, which comprises the following steps:
s1, receiving a push message of a service system;
s2, after analyzing and verifying the format of the push message, reassembling the message according to the preset message format; s3, storing the assembled messages into a background database (DB store) according to different user dimensions; the user dimension comprises single user messages, group push messages and short messages;
s4, when the front end triggers the query, according to the query types of different user dimensions, performing secondary storage of the push message, and returning a query result;
s41, if the query type is judged to be paging query, querying a Redis cache and returning a message cached in the Redis cache as a query result;
and S42, if the query type is judged to be the forced pull of the latest message, performing secondary storage of message query and the push message according to the user dimension, and feeding back a query result, wherein the user dimension comprises a single-user message and a group push message.
Because the concurrency of the in-station message query interface is large, the data size of the in-station message table is large, and a large number of newly added records exist every day, the cache-based data storage and query method disclosed by the invention has the advantages that when the query interface is designed, the in-station message list based on the time sequencing is provided by using a RangeByScore API (a sequencing tool in ZSET) provided by ZSET based on the cache of a Redis ZSET (ordered set) data structure, so that a large number of concurrent query and sequencing operations of a database layer are avoided, and the performance of the system is improved.
Example two
As shown in fig. 3, in this embodiment, the method for storing and querying data based on cache according to the present invention specifically includes the following steps:
s1, receiving a push message of a service system;
in this step, the CF-OPR system receives the message pushed by the upstream system through the MQ, the upstream system sends the asynchronous push message to the CF-OPR system through the MQ, the CF-OPR asynchronously consumes the MQ message and sends the processing result to the MQ after processing, and the upstream system listens to the MQ to obtain the processing result of the original request.
And S2, after the format of the push message is analyzed and verified, the message is reassembled according to the preset message format.
After receiving the message of the upstream system, the CF-OPR system analyzes the uniform message body, and then performs template assembly and storage according to a preset message body format, as a preferred scheme, the query method based on data cache disclosed by the invention designs a uniform and expandable preset message body format for different types of service messages of the delinquent service system, and is compatible with single user/group messages, wherein the preset message body format sequentially comprises the following steps: an msgId field to describe the unique identification field of a system message, a topic field to describe the identity of a message to an MQ, a tag field to describe the identification of an upstream system, a key field to describe the unique identification of a message within an upstream system, an orderKey field to describe the sort key of a sequential message, and a content field to describe the body of a message. The specific composition of the format of the preset message body and the functions of each field are shown in the following table:
Figure BDA0002840104910000111
the specific description of the preset message format takes the respective caching of the single-user message and the group message as an example: the generation time of the message is score in each zset; the key composition of zset is as follows:
single user message: PEA _ INSITEMSG _ MY _ { $ app _ name } ____ { $ identity _ type } _{ identity };
group information: PEA _ INSITEMSG _ SYS _ { $ app _ name } ___ { $ identity _ type } _{ identity };
for example, the user a logs in the App with the mobile phone number 13809467826, and when the personal message zset key cached by the OPR system for the user is PEA _ INSITEMSG _ MY _ puapp _ mobile _13809467826, when the user has a new message (the generation time is T1, and the serialized message body is "json1"), the user writes the new message into the cache through add PEA _ INSITEMSG _ MY _ puapp _ mobile _ 13809467826T 1"json 1".
S3, storing the assembled messages into background databases respectively according to different user dimensions, wherein the user dimensions comprise single-user messages, group messages and short messages.
In this embodiment, different storage modes are adopted for the pushed messages according to different user dimensions. When storing messages, the user dimension includes short messages in addition to the single-user messages and the group messages, and the step of storing the assembled messages into the background database respectively specifically includes the following steps:
s31 stores the message itself and the group owner information of the message in the database if it is determined that the push message is the group message request, wherein one group message corresponds to one storage record in the database.
In step S31, the parameters in the message body field are analyzed to identify that the push message is a group message request, then a message template is assembled, the placeholder in the template is replaced with the specified value in the message body, a group message corresponds to a storage record in the database, and finally the assembled message is stored. According to the service requirement of the group push/group push station internal mail, when the CF-OPR pushes the station internal mail in an upstream system, the CF-OPR firstly stores the message and the group owner information of the message, does not pre-store the corresponding relation between the user and the group (the stored corresponding relation can multiply the total record number of a background database, most mapping relations cannot be used and cannot be compensated), but when the front end triggers the query of the group push/group push station internal mail, the system stores the corresponding relation between the current login user and the group information indicated in the current query condition, and then queries the message meeting the condition and returns the message to the calling party, so that the CF-OPR only serves as an independent message processing system without maintaining the user and group relation information in the service system.
S32, if the push message is judged to be a single-user message or a short message, storing the message into a database. Wherein, the single user message only stores the message itself, the short message is the same as the single user message, and only stores the message itself,
and S4, when the front end triggers the query, according to the query types of different user dimensions, performing secondary storage of the message and returning a query result according to the query request.
In step S4, the query categories include forced pull queries and paging queries; as a preferred scheme, the concrete query type is judged by reading the maxtime value. And if maxTime is the minimum value of born time in the last returned message, determining the message is a paging query.
And S41, if the query type is judged to be paging query, querying a Redis cache and returning a message cached in the Redis cache as a query result.
In this embodiment, when maxTime is the minimum value of born time in the last returned message, it is determined as a paging query message of a single user or a group user, and at this time, the OPR directly queries a redis cache through zrangebyscore, and returns a result.
And S42, if the query type is judged to be the forced pull of the latest message, performing secondary storage of message query and the push message according to the user dimension, and feeding back a query result, wherein the user dimension comprises single-user messages and group push messages.
S421, when the front-end trigger query is a single-user message query and the query type is a forced pull latest message, the system checks whether zset of the specified message type (multi-user) of the specified user (identity type, identity) exists in the Redis cache. The single-user message only stores the message, and when the query is triggered, if the message of the user is not cached before, the message is queried from the database and then cached back to the seed so as to perform secondary storage of the push message.
S4211, if zset does not exist, querying all unexpired messages meeting the specified query conditions in the database and filling the unexpired messages into a Redis cache.
S4212, if zset exists, comparing the latest message date T1 in the database with the maximum date T2 of the Redis cached messages, if T1> T2, querying message records with message dates larger than T2 from the database, adding the message records to the Redis cache, and returning the updated Redis cache messages as query results.
Further, in step S4212, the querying the database for the message record with the message date greater than T2 and adding the message record to the Redis cache includes:
A. jointly inquiring all messages of born time & gt VA in the single-user basic information table, and recording the inquiry time T;
B. writing the message acquired in the step (A) into a Redis cache;
C. updating the version value of the current user in the single-user information table to be T;
D. and taking out the latest count record from the cache according to the transmitted count value, and assembling and returning a result after deserialization.
As shown in fig. 3, a latest message date T1 in the database is born time in the single-user basic information table, a maximum date T2 of the Redis cached message is a version value VA corresponding to the user in the single-user information table, the single-user basic information tables are an opr _ inite _ msg table and an opr _ inite _ msg _ detail table, and the single-user information table is an opr _ inite _ msg _ user _ version table.
And/or S422, if the user dimension is a group push message query and the query type is a forced pull latest message, checking whether zset of the specified message type of the specified user in the Redis cache exists.
The method comprises the steps that a group message is a storage message and an owner, when query is triggered, a mapping relation is obtained and stored in dbstore, corresponding data is searched from the dbstore according to the mapping relation and returned to carry out secondary storage of the push message, and when the user dimension is judged to be the group push message query and the query type is the forced pull new message, a system checks whether zset of the designated message type (multiple users) of the designated user (identity type and identity identification) exists in a Redis cache.
S4221, if zset does not exist, querying all unexpired messages meeting the specified query conditions in the database and filling the unexpired messages into a Redis cache.
S4222, if there is zset, obtaining and storing the mapping relationship between the current user and the group information indicated in the query condition from the query request, querying a corresponding message from the database, and returning the message to the caller, including: and inquiring the born time > VA, putting the message matched with the owner information into a Redis cache, and returning an inquiry result.
Further, in step S4222, the querying that bornTime > VA and the message matching the owner information are put into a Redis cache, and returning the query result includes:
a. inquiring the borne time & ltVA & gt in the group message basic information table and the station information ID matched with the owner information, and recording the inquiry time T; and the VA is a version value corresponding to the user in the group user information table, and the born time is acquired from the group basic information table.
b. And adding a group push message and the associated record of the current user to each obtained ID, and juxtaposing the group push message and the associated record as unread.
c. Writing the in-station message acquired in the step (a) into a Redis cache.
d. And updating the version value of the current user in the group user information table to be T.
e. And taking out the latest count record from the cache according to the transmitted count value, and assembling and returning a result after deserialization.
As shown in fig. 3, VA is a version value corresponding to a user in the group user information table, the born time is obtained from the group basic information table, the group basic information table is an opr _ instance _ msg and opr _ instance _ msg _ audios table, and the group user information table is an opr _ instance _ msg _ user _ version table. According to the data storage and query method based on the cache, when the front end triggers the query of the group push/group push station mail, the mapping relation of the group information indicated in the current user and the query condition is obtained and stored from the query request, and then the corresponding message is queried from the database and returned to the calling party, so that the station mail list (such as RangeByScore API) based on time sequencing provided by ZSET is utilized, a large amount of concurrent query and sequencing operations of a database layer are avoided, and the performance of the system is improved.
EXAMPLE III
As shown in fig. 4, the present invention further provides a data storage and query apparatus 10 based on cache, including:
a push message receiving module 11, configured to receive a push message of a service system;
the message processing module 12 is configured to analyze and verify the format of the push message, and then reassemble the message according to a preset message format;
the primary storage module 13 is configured to store the assembled messages into a background database according to different user dimensions, where the user dimensions include single-user messages, group push messages, and short messages;
the secondary storage and query returning module 14 is configured to perform secondary storage of the push message according to query categories of different user dimensions when the front end triggers a query, and return a query result according to secondary storage contents, and includes:
the paging query submodule is used for querying a Redis cache and returning a cached message in the Redis cache as a query result if the query type is judged to be paging query;
and the forced pull query submodule is used for performing secondary storage of message query and the push message according to the user dimension and feeding back a query result if the query type is judged to be the forced pull latest message, wherein the user dimension comprises a single-user message and a group push message.
According to the data storage and query device 10 based on the cache, due to the fact that the concurrency of the in-station message query interface is large, the data size of the in-station message table is large, and a large number of newly added records exist every day, the cache based on the Redis ZSET data structure is designed when the query interface is designed, the in-station message list based on time sequencing (such as RangeByScore API) is provided through ZSET, a large number of concurrent query and sequencing operations of a database layer are avoided, and the performance of the system is improved.
As a preferred scheme, the forced pull query submodule includes a single-user message query unit, configured to check whether zset of the specified message type of the specified user in the Redis cache exists, if the user dimension is a single-user message query; if not, inquiring all unexpired messages which accord with the specified inquiry condition in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages serving as inquiry results; if the message exists, comparing the latest message date T1 in the database with the maximum date T2 of the Redis cached message, if T1> T2, inquiring the message record with the message date larger than T2 from the database, adding the message record to the Redis cache, and returning the updated Redis cache message as an inquiry result; if T1 < T2, the message that Redis has cached is returned as the query result.
Further, in the single-user message query unit, a latest message date T1 in the database is a borne time in the single-user basic information table, a maximum date T2 of a Redis cached message is a version value VA corresponding to the user in the single-user information table, a message record with a message date greater than T2 is queried from the database and added to the Redis cache, and returning an updated Redis cache message as a query result includes:
A. jointly inquiring all messages of born time & gt VA in the single-user basic information table, and recording the inquiry time T;
B. writing the message acquired in the step (A) into a Redis cache;
C. updating the version value of the current user in the single-user information table to be T;
D. and taking out the latest count record from the cache according to the transmitted count value, and assembling and returning a result after deserialization.
As a preferred scheme, the forced pull query submodule further includes a group push query unit, configured to check whether zset of the specified message type of the specified user in the Redis cache exists, if the user dimension is a group push message query: if not, inquiring all unexpired messages which accord with the specified inquiry condition in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages serving as inquiry results; if the query request exists, acquiring and storing the mapping relation between the current user and the group information indicated in the query condition from the query request, querying a corresponding message from the database according to the mapping relation of the group information, filling the message into a Redis cache, and returning the message for updating the Redis cache as a query result.
Further, in the group push query unit, obtaining and storing a mapping relationship between the current user and the group information indicated in the query condition from the query request, querying a corresponding message from the database according to the mapping relationship of the group information, and then filling the message into the Redis cache, and returning an updated Redis cache message as a query result includes:
a. inquiring the borne time & ltVA & gt in the group message basic information table and the station information ID matched with the owner information, and recording the inquiry time T; the VA is a version value corresponding to a user in the group user information table, and the born time is acquired from the group basic information table;
b. adding a group push message and a current user association record to each obtained ID, and juxtaposing the group push message and the current user association record as unread;
c. writing the in-station message acquired in the step (a) into a Redis cache;
d. updating the version value of the current user in the group user information table to be T;
e. and taking out the latest count record from the cache according to the transmitted count value, and assembling and returning a result after deserialization.
As a preferred scheme, the secondary storage and query returning module further includes a query type determining module, configured to determine the query type according to a value of maxTime, and if maxTime is null, it is determined as forced pull, and if maxttime is a minimum value of born time in a last returned message, it is determined as paging query.
As a preferred scheme, the forced pull query submodule further includes a Redis cache determination submodule, configured to further determine whether a corresponding Redis cache is available after the user dimension is determined, if the corresponding Redis cache is available, the corresponding Redis cache is switched to a single-user message query unit or a group push query unit for processing according to the user dimension category, and if the corresponding Redis cache is unavailable, records satisfying conditions in a database are directly queried and returned to a calling party.
As a preferred scheme, the primary storage module 13 is configured to store to a background database according to different user dimensions, and includes:
a group message request storage submodule, configured to store the message itself and group owner information of the message in a database when the push message is a group message request, where one group message corresponds to one storage record in the database;
the single-user message storage submodule is used for storing the message into a database if the push message is a single-user message;
and the short message storage sub-module is used for storing the message to a database if the push message is a short message.
Example four
Fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present invention, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack-mounted server, a blade server, a tower server, or a rack-mounted server (including an independent server or a server cluster formed by multiple servers) that can execute programs. The computer device 20 of the present embodiment includes at least, but is not limited to: a memory 21, a processor 22, which may be communicatively coupled to each other via a system bus, as shown in FIG. 5. It is noted that fig. 5 only shows a computer device 20 with components 21-22, but it is to be understood that not all shown components are required to be implemented, and that more or fewer components may be implemented instead.
In this embodiment, the memory 21 (i.e., the readable storage medium) includes a Flash memory, a hard disk, a multimedia Card, a Card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), and a Programmable Read Only Memory (PROM) memory 21, and the memory 21 may also be an external storage device of the computer device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card), and the like, which are provided on the computer device 20. Of course, the memory 21 may also include both internal and external storage devices of the computer device 20. In this embodiment, the memory 21 is generally used for storing an operating system and various application software installed in the computer device 20, such as program codes of the cache-based data storage and query apparatus in the method embodiment. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 22 is typically used to control the overall operation of the computer device 20. In this embodiment, the processor 22 is configured to execute the program code stored in the memory 21 or process data, for example, execute the cache-based data storage and query apparatus 10, so as to implement the cache-based data storage and query method in the method embodiment.
EXAMPLE five
The present application also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, etc., on which a computer program is stored, which when executed by a processor implements corresponding functions. The computer-readable storage medium of the embodiment is used for storing the cache-based data storage and query apparatus 10, and when executed by a processor, the cache-based data storage and query method of the embodiment is implemented.
It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims (10)

1. A data storage and query method based on cache is characterized in that: the method comprises the following steps:
s1, receiving a push message of a service system;
s2, after analyzing and verifying the format of the push message, reassembling the message according to the preset message format;
s3, storing the assembled messages into a background database according to different user dimensions;
s4, when the front end triggers the query, according to the query types of different user dimensions, performing secondary storage of the push message, and returning a query result;
s41, if the query type is judged to be paging query, querying a Redis cache and returning a message cached in the Redis cache as a query result;
and S42, if the query type is judged to be the forced pull of the latest message, performing secondary storage of message query and the push message according to the user dimension, and feeding back a query result, wherein the user dimension comprises a single-user message and a group push message.
2. The cache-based data storage and query method of claim 1, wherein: if the user dimension is the single-user message query, checking whether zset of the specified message type of the specified user in the Redis cache exists;
if not, inquiring all unexpired messages which accord with the specified inquiry condition in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages serving as inquiry results;
if the message exists, comparing the latest message date T1 in the database with the maximum date T2 of the Redis cached message, if T1> T2, inquiring the message record with the message date larger than T2 from the database, adding the message record to the Redis cache, and returning the updated Redis cache message as an inquiry result; if T1 < T2, the message that Redis has cached is returned as the query result.
3. The cache-based data storage and query method of claim 2, wherein: the latest message date T1 in the database is born time in a single-user basic information table, the maximum date T2 of the Redis cached messages is a version value VA corresponding to the user in the single-user information table, a message record with a message date greater than T2 is queried from the database and added to the Redis cache, and returning the updated Redis cache message as a query result includes:
A. jointly inquiring all messages of born time & gt VA in the single-user basic information table, and recording the inquiry time T;
B. writing the message acquired in the step (A) into a Redis cache;
C. updating the version value of the current user in the single-user information table to be T;
D. and taking out the latest count records from the cache according to the transmitted count value, and assembling a return result after deserialization, wherein the count is a record numerical value of each query.
4. The cache-based data storage and query method of claim 1, wherein: if the user dimension is a group push message query, checking whether zset of the specified message type of the specified user in the Redis cache exists:
if not, inquiring all unexpired messages which accord with the specified inquiry condition in the database, filling the unexpired messages into a Redis cache, and returning the updated Redis cache messages serving as inquiry results;
if the query request exists, acquiring and storing the mapping relation between the current user and the group information indicated in the query condition from the query request, querying a corresponding message from the database according to the mapping relation of the group information, filling the message into a Redis cache, and returning the message for updating the Redis cache as a query result.
5. The cache-based data storage and query method of claim 4, wherein: obtaining and storing the mapping relationship between the current user and the group information indicated in the query condition from the query request, filling the corresponding information into a Redis cache after querying the corresponding information from the database according to the mapping relationship of the group information, and returning the updated Redis cache information as a query result, wherein the step of returning comprises the following steps:
a. inquiring the borne time & ltVA & gt in the group message basic information table and the station information ID matched with the owner information, and recording the inquiry time T; the VA is a version value corresponding to a user in the group user information table, and the born time is acquired from the group basic information table;
b. adding a group push message and a current user association record to each obtained ID, and juxtaposing the group push message and the current user association record as unread;
c. writing the in-station message acquired in the step (a) into a Redis cache;
d. updating the version value of the current user in the group user information table to be T;
e. and taking out the latest count records from the cache according to the transmitted count value, and assembling a return result after deserialization, wherein the count is a record numerical value of each query.
6. The cache-based data storage and query method of claim 1, wherein: in step S4, the query type is determined according to the value of maxTime, and if maxTime is null, it is determined as forced pull, and if maxTime is the minimum value of born time in the last returned message, it is determined as page query.
7. The cache-based data storage and query method of claim 1, wherein: the method is characterized in that: in step S3, the user dimension includes a single user message, a group push message, and a short message; the step of respectively storing the data into the background database according to different user dimensions comprises the following steps:
s31, if the push message is a group message request, storing the message itself and the group owner information of the message in a database, wherein one group message corresponds to one storage record in the database;
s32, if the push message is a single-user message, storing the message into a database;
s33, if the push message is a short message, storing the message itself in a database.
8. A data storage and inquiry device based on cache is characterized in that: the method comprises the following steps:
the push message receiving module is used for receiving a push message of a service system;
the message processing module is used for analyzing and verifying the format of the push message and then reassembling the message according to the preset message format;
the primary storage module is used for respectively storing the assembled messages into a background database according to different user dimensions, wherein the user dimensions comprise single-user messages, group push messages and short messages;
the secondary storage and query returning module is used for performing secondary storage on the push message according to query categories of different user dimensions when the front end triggers query, and returning a query result according to secondary storage contents, and comprises:
the paging query submodule is used for querying a Redis cache and returning a cached message in the Redis cache as a query result if the query type is judged to be paging query;
and the forced pull query submodule is used for performing secondary storage of message query and the push message according to the user dimension and feeding back a query result if the query type is judged to be the forced pull latest message, wherein the user dimension comprises a single-user message and a group push message.
9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein: the processor, when executing the computer program, realizes the steps of the method of any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program when executed by a processor implements the steps of the method of any one of claims 1 to 7.
CN202011488712.4A 2020-12-16 2020-12-16 Data storage and query method, device, equipment and medium based on cache Active CN112637305B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011488712.4A CN112637305B (en) 2020-12-16 2020-12-16 Data storage and query method, device, equipment and medium based on cache

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011488712.4A CN112637305B (en) 2020-12-16 2020-12-16 Data storage and query method, device, equipment and medium based on cache

Publications (2)

Publication Number Publication Date
CN112637305A true CN112637305A (en) 2021-04-09
CN112637305B CN112637305B (en) 2023-05-02

Family

ID=75313940

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011488712.4A Active CN112637305B (en) 2020-12-16 2020-12-16 Data storage and query method, device, equipment and medium based on cache

Country Status (1)

Country Link
CN (1) CN112637305B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113343048A (en) * 2021-06-30 2021-09-03 中国农业银行股份有限公司 Data query method and device and readable storage medium
CN113486099A (en) * 2021-06-23 2021-10-08 华泰证券股份有限公司 Memory computing service framework and implementation system
CN113626483A (en) * 2021-08-18 2021-11-09 重庆允成互联网科技有限公司 Front-end caching method, system, equipment and storage medium for filling forms
CN114422593A (en) * 2021-12-30 2022-04-29 杭州数梦工场科技有限公司 Data distribution method and device, electronic equipment and storage medium
CN115314423A (en) * 2022-07-19 2022-11-08 杭州安恒信息技术股份有限公司 Traffic data statistical method, device and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150334182A1 (en) * 2012-12-17 2015-11-19 Beijing Qihoo Technology Limited System, Method and Browser Client for Enabling Browser Data Synchronization
CN109710639A (en) * 2018-11-26 2019-05-03 厦门市美亚柏科信息股份有限公司 A kind of search method based on pair buffers, device and storage medium
CN110213380A (en) * 2019-06-11 2019-09-06 深圳市思迪信息技术股份有限公司 A kind of information push method, device, computer equipment and storage medium
CN110502327A (en) * 2019-08-28 2019-11-26 四川长虹电器股份有限公司 Method based on the processing of Redis high concurrent delayed tasks
CN110674432A (en) * 2019-09-09 2020-01-10 中国平安财产保险股份有限公司 Second-level caching method and device and computer readable storage medium
CN110909025A (en) * 2018-09-17 2020-03-24 深圳市优必选科技有限公司 Database query method, query device and terminal

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150334182A1 (en) * 2012-12-17 2015-11-19 Beijing Qihoo Technology Limited System, Method and Browser Client for Enabling Browser Data Synchronization
CN110909025A (en) * 2018-09-17 2020-03-24 深圳市优必选科技有限公司 Database query method, query device and terminal
CN109710639A (en) * 2018-11-26 2019-05-03 厦门市美亚柏科信息股份有限公司 A kind of search method based on pair buffers, device and storage medium
CN110213380A (en) * 2019-06-11 2019-09-06 深圳市思迪信息技术股份有限公司 A kind of information push method, device, computer equipment and storage medium
CN110502327A (en) * 2019-08-28 2019-11-26 四川长虹电器股份有限公司 Method based on the processing of Redis high concurrent delayed tasks
CN110674432A (en) * 2019-09-09 2020-01-10 中国平安财产保险股份有限公司 Second-level caching method and device and computer readable storage medium

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113486099A (en) * 2021-06-23 2021-10-08 华泰证券股份有限公司 Memory computing service framework and implementation system
CN113343048A (en) * 2021-06-30 2021-09-03 中国农业银行股份有限公司 Data query method and device and readable storage medium
CN113626483A (en) * 2021-08-18 2021-11-09 重庆允成互联网科技有限公司 Front-end caching method, system, equipment and storage medium for filling forms
CN114422593A (en) * 2021-12-30 2022-04-29 杭州数梦工场科技有限公司 Data distribution method and device, electronic equipment and storage medium
CN114422593B (en) * 2021-12-30 2024-03-15 杭州数梦工场科技有限公司 Data distribution method, device, electronic equipment and storage medium
CN115314423A (en) * 2022-07-19 2022-11-08 杭州安恒信息技术股份有限公司 Traffic data statistical method, device and storage medium

Also Published As

Publication number Publication date
CN112637305B (en) 2023-05-02

Similar Documents

Publication Publication Date Title
CN112637305B (en) Data storage and query method, device, equipment and medium based on cache
CN109344197B (en) Paging downloading method and device based on big data
CN110597859B (en) Method and device for querying data in pages
CN111506559B (en) Data storage method, device, electronic equipment and storage medium
US11567681B2 (en) Method and system for synchronizing requests related to key-value storage having different portions
CN110457346B (en) Data query method, device and computer readable storage medium
CN110659971B (en) Transaction data processing method and device
US11748357B2 (en) Method and system for searching a key-value storage
CN111414362A (en) Data reading method, device, equipment and storage medium
CN110032578B (en) Mass data query caching method and device
CN115455058A (en) Cache data processing method and device, computer equipment and storage medium
CN113157777B (en) Distributed real-time data query method, cluster, system and storage medium
CN116783588A (en) Column technique for large metadata management
CN107609870B (en) Multi-application key management method and system for POS (point of sale) and POS terminal
CN111143365B (en) Data sub-table method, device, computer equipment and storage medium
US11620284B2 (en) Backend data aggregation system and method
CN112434037A (en) Data processing method, processing device, data processing apparatus, and storage medium
CN112015766A (en) Data processing method and device based on pipelining and data processing system
CN114840608B (en) Distributed data access method, device and storage medium
US11321375B2 (en) Text object management system
CN107896246A (en) Message dissemination system based on XBRL
CN112632115A (en) BI-based data query method and system
CN113535770B (en) Data query method and device
CN118261690A (en) Credit investigation credit main body data searching method, device, computer equipment and medium
CN117171404A (en) Document data query method, query system and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant