WO2023056946A1 - Procédé et appareil de mise en cache de données et dispositif électronique - Google Patents

Procédé et appareil de mise en cache de données et dispositif électronique Download PDF

Info

Publication number
WO2023056946A1
WO2023056946A1 PCT/CN2022/123867 CN2022123867W WO2023056946A1 WO 2023056946 A1 WO2023056946 A1 WO 2023056946A1 CN 2022123867 W CN2022123867 W CN 2022123867W WO 2023056946 A1 WO2023056946 A1 WO 2023056946A1
Authority
WO
WIPO (PCT)
Prior art keywords
data table
cache
source data
matching
field name
Prior art date
Application number
PCT/CN2022/123867
Other languages
English (en)
Chinese (zh)
Inventor
凌鹏
宋荣鑫
黄志文
黄师亮
邢吕亮
胡俊松
Original Assignee
上海淇馥信息技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海淇馥信息技术有限公司 filed Critical 上海淇馥信息技术有限公司
Publication of WO2023056946A1 publication Critical patent/WO2023056946A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases

Definitions

  • the present application relates to the computer field, in particular to a data caching method, device and electronic equipment.
  • a relational database refers to a database that uses a relational model to organize data, and stores data in the form of rows and columns.
  • Key-value database A key-value database is a non-relational database that stores data as a set of key-value pairs, and the key name is used as a unique identifier. Because a key-value database uses a simple key-value pair method to store data, its data storage capacity is relatively large. Poor, rarely used in databases.
  • the embodiments of this specification provide a data caching method, device and electronic equipment, so as to improve the convenience of data caching.
  • the embodiment of this specification provides a data caching method, including:
  • the field name query is performed on the source data table by using the table structure information in the matching rule, and the queried target field name is read.
  • the table structure information in the matching rule is a table structure function with time as an independent variable
  • the value of the table structure function is a row number or a column number.
  • Configure data table monitoring rules for the cache model to monitor update events of the source data table, and if the source data table is detected, perform data synchronous reading to update the cache.
  • the matching rules include preset field names and similarity calculation rules
  • variable-based construction of the cache model includes:
  • the matching rules also include source data table matching rules
  • the acquisition source data table includes:
  • the source data table to be cached is determined and acquired from the database according to the source data table matching rule.
  • the embodiment of this specification also provides a data cache device, including:
  • the model construction module constructs a cache model based on variables, and configures annotation information with matching rules for the variables in the cache model;
  • the matching assignment module calls the cache model to read the annotation information in response to the cache trigger event, obtains the source data table, and determines the target field name in the source data table that matches the matching rule, based on the target
  • the field name is the variable assignment in the cache model
  • the cache execution module uses the assigned cache model to read the field value corresponding to the target field name in the source data table, and generates key-value pair information to store in the cache space.
  • the embodiment of this specification also provides an electronic device, wherein the electronic device includes:
  • a memory storing a computer-executable program which, when executed, causes the processor to perform any of the methods described above.
  • the embodiment of this specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, any one of the above-mentioned methods can be implemented .
  • the various technical solutions provided by the embodiments of this specification construct a cache model based on variables, and configure annotation information with matching rules for the variables in the cache model, and call the cache model to read the annotation information in response to a cache trigger event , obtain the source data table, and determine the name of the target field in the source data table that matches the matching rule, assign the value of the variable in the cache model based on the name of the target field, and use the cache model after the assignment to read
  • the field value corresponding to the target field name in the source data table generates key-value pair information and stores it in the cache space.
  • the cache model can be independent of the data table, so it can be applied to different data tables, and can automatically read the field names and corresponding field values in the source data table, and cache them in advance to avoid temporarily reading data from relational data tables.
  • the inefficiency of the data table is eliminated, and the convenience is improved by eliminating the need to develop code for each data table separately.
  • FIG. 1 is a schematic diagram of the principle of a data caching method provided by an embodiment of this specification
  • FIG. 2 is a schematic structural diagram of a data cache device provided by an embodiment of this specification
  • FIG. 3 is a schematic structural diagram of an electronic device provided by an embodiment of this specification.
  • Fig. 4 is a schematic diagram of the principle of a computer-readable medium provided by the embodiment of this specification.
  • Figure 1 is a schematic diagram of the principle of a data caching method provided by an embodiment of this specification, the method may include:
  • S101 Construct a cache model based on variables, and configure annotation information with matching rules for the variables in the cache model.
  • the performance of memory in terms of reading and writing is generally several orders of magnitude higher than that of disk.
  • variable-based construction of the cache model may include:
  • attributes of variables such as line numbers, can be configured in annotations.
  • the cache model corresponds to the table structure in the relational database, or corresponds to the data object that needs to be cached.
  • the customer table contains the following fields: customer number, name, gender, ID number, and the business primary key can be: customer number, ID number.
  • the configuration cache model may include:
  • the data table structure information includes field attributes corresponding to the data to be cached.
  • the structural feature of the data table may be the number of rows and columns where the target field is located.
  • the cache model can be adapted to multiple data tables.
  • the relational database can use mysql, oracle. Redis, memecache, etc. can be used in non-relational databases. In terms of caching, redis is usually used.
  • the embodiment of this specification can implement caching based on redis.
  • the API in the redis client may be difficult to meet more complex requirements, for example, when implementing some specific functions, it is likely to require multiple commands or interactions of multiple data types to complete, then such multiple network requests It has a relatively large impact on performance, so we can provide a mechanism to write some commands with business logic in order to reduce network requests.
  • support for the Lua environment is embedded in Redis, allowing developers to use Lua language to write scripts and transmit them to Redis for execution.
  • Redis clients can use Lua scripts to directly execute multiple Redis commands atomically on the server side.
  • the customer table contains the following fields: customer number, name, gender, and ID number.
  • the operations on the customer table can include: query (such as query by customer number, or query by ID number) , Add (add a new customer), modify (modify the customer's name through the customer number, modify the customer's ID number through the customer number), delete (delete a customer through the customer number).
  • the cache model is the CustEntity entity model
  • the configuring the cache model may further include: configuring and initializing the cache of the CustEntity entity model based on xml.
  • it can include: the system name for initializing the CustEntity entity model, the model type, the main cache switch, the maximum number of default caches, the default cache time, and the dynamic configuration environment information (based on the hot cache configuration implemented by the apollo configuration center: the maximum number of caches, the cache time wait).
  • S102 Call the cache model to read the annotation information in response to the cache trigger event, obtain the source data table, and determine the target field name in the source data table that matches the matching rule, based on the target field name Assign values to variables in the cached model.
  • the caching trigger event may be that a data table to be cached in the database is updated, or a data table is newly created.
  • the matching rules also include source data table matching rules
  • the acquisition source data table may include:
  • the source data table to be cached is determined and acquired from the database according to the source data table matching rule.
  • the table structure information of the target field name is included in the matching rule
  • the field name query is performed on the source data table by using the table structure information in the matching rule, and the queried target field name is read.
  • the table structure information in the matching rule is a table structure function with time as an argument
  • the value of the table structure function is a row number or a column number.
  • the matching rules have preset field names and similarity calculation rules
  • the similarity may be semantic similarity.
  • the semantics here can be equivalent semantics, or upper and lower semantics.
  • the matched field name can be: “Chinese Score”, “Mathematics Score” or "English Score”.
  • "a” can be constructed as a variable in the cache instruction, and its attribute "column number” and the matching rule for matching the column number of a can be added to the annotation.
  • the field is matched according to the matching rule If the name is "mathematics score”, assign "mathematics score” to a, and the cache model is called the code that can perform the cache function, and then read the data for caching.
  • S103 Use the assigned cache model to read the field value corresponding to the target field name in the source data table, generate key-value pair information and store it in the cache space.
  • the target field name in the data table that matches the matching rule is based on the variable assignment in the cache model of the target field name, and the value corresponding to the target field name in the source data table is read using the cache model after the assignment.
  • Field value generate key-value pair information and store it in the cache space.
  • the cache model can be independent of the data table, so it can be applied to different data tables, and can automatically read the field names and corresponding field values in the source data table, and cache them in advance to avoid temporarily reading data from relational data tables.
  • the inefficiency of the data table is eliminated, and the convenience is improved by eliminating the need to develop code for each data table separately.
  • the cache model When reading data, the cache model finds the column "mathematics score”, reads the field value under it, that is, the specific score value (such as 90), constructs a key-value pair "mathematics score: 90", and stores it in the memory middle.
  • the specific score value such as 90
  • the method may also include:
  • Configure data table monitoring rules for the cache model to monitor update events of the source data table, and if the source data table is detected, perform data synchronous reading to update the cache.
  • the apolloDefaultEnv environment configuration is initialized, and the configuration can be loaded from Apollo in real time;
  • An AOP interceptor can perform the following steps:
  • Step 1 When executing the method (selectCustByCustNo), it will first obtain the annotation on the method header, which is @CacheByFirstKeyAnnotation, then obtain the LUA expression, and obtain the value of the method input parameter custNo through the EL expression. Then use this value to execute the specific LUA script.
  • REDIS_VALUE is the specified string "LOCK", it means that the value is locked, and query the database again;
  • Step 2 When the getCustByIdNo method is executed, the annotation on the method header will be obtained first. This annotation is @CacheBySecondKeyAnnotation, the LUA expression is obtained, and the value of the method input parameter idNo is obtained through the EL expression. Then use this value to execute the specific LUA script.
  • Execute specific LUA scripts including:
  • Step 3 When the updateCustNameByCustNo method is executed, the EvictCacheByFirstKeyAnnotation annotation on the method header will be obtained first, the LUA expression will be obtained, and the value of the method input parameter custNo will be obtained through the EL expression.
  • Execute specific LUA scripts including:
  • Fig. 2 is a schematic structural diagram of a data caching device provided by an embodiment of this specification, and the device may include:
  • the model construction module 201 constructs a cache model based on variables, and configures annotation information with matching rules for the variables in the cache model;
  • the matching assignment module 202 calls the cache model to read the annotation information in response to the cache trigger event, obtains the source data table, and determines the name of the target field in the source data table that matches the matching rule, based on the The target field name is the variable assignment in the cache model;
  • the cache execution module 203 uses the assigned cache model to read the field value corresponding to the target field name in the source data table, and generates key-value pair information to store in the cache space.
  • the device constructs a cache model based on variables, configures annotation information with matching rules for the variables in the cache model, calls the cache model to read the annotation information in response to a cache trigger event, obtains a source data table, and determines the The target field name matching the matching rule in the source data table, based on the variable assignment in the cache model of the target field name, use the cache model after the assignment to read the target field name in the source data table Corresponding field values, generate key-value pair information and store them in the cache space.
  • the cache model can be independent of the data table, so it can be applied to different data tables, and can automatically read the field names and corresponding field values in the source data table, and cache them in advance to avoid temporarily reading data from relational data tables. The inefficiency of the data table is eliminated, and the convenience is improved by eliminating the need to develop code for each data table separately.
  • the embodiment of this specification also provides an electronic device.
  • FIG. 3 is a schematic structural diagram of an electronic device provided by an embodiment of this specification. An electronic device 300 according to this embodiment of the present invention is described below with reference to FIG. 3 .
  • the electronic device 300 shown in FIG. 3 is only an example, and should not limit the functions and application scope of this embodiment of the present invention.
  • electronic device 300 takes the form of a general-purpose computing device.
  • Components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different system components (including the storage unit 320 and the processing unit 310), a display unit 340, and the like.
  • the storage unit stores program codes, and the program codes can be executed by the processing unit 310, so that the processing unit 310 executes the processing methods according to various exemplary embodiments of the present invention described in the above processing method section of this specification. step.
  • the processing unit 310 may execute the steps shown in FIG. 1 .
  • the storage unit 320 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 3201 and/or a cache storage unit 3202 , and may further include a read-only storage unit (ROM) 3203 .
  • RAM random access storage unit
  • ROM read-only storage unit
  • the storage unit 320 may also include a program/utility 3204 having a set (at least one) of program modules 3205, such program modules 3205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include the implementation of the network environment.
  • program modules 3205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include the implementation of the network environment.
  • Bus 330 may represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local area using any of a variety of bus structures. bus.
  • the electronic device 300 can also communicate with one or more external devices 400 (such as keyboards, pointing devices, Bluetooth devices, etc.), and can also communicate with one or more devices that enable the user to interact with the electronic device 300, and/or communicate with Any device (eg, router, modem, etc.) that enables the electronic device 300 to communicate with one or more other computing devices. Such communication may occur through input/output (I/O) interface 350 .
  • the electronic device 300 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and/or a public network such as the Internet) through the network adapter 360.
  • the network adapter 360 can communicate with other modules of the electronic device 300 through the bus 330 .
  • other hardware and/or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape Drives and data backup storage systems, etc.
  • the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, and the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or on a network , including several instructions to make a computing device (which may be a personal computer, a server, or a network device, etc.) execute the above-mentioned method according to the present invention.
  • a computer-readable storage medium which can be a CD-ROM, U disk, mobile hard disk, etc.
  • the computer-readable medium can realize the above-mentioned method of the present invention, that is, the method as shown in FIG. 1 .
  • Fig. 4 is a schematic diagram of the principle of a computer-readable medium provided by the embodiment of this specification.
  • the computer program implementing the method shown in Figure 1 can be stored on one or more computer readable media.
  • the computer readable medium may be a readable signal medium or a readable storage medium.
  • the readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
  • the computer readable storage medium may include a data signal carrying readable program code in baseband or as part of a carrier wave traveling as a data signal. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • a readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate or transport a program for use by or in conjunction with an instruction execution system, apparatus or device.
  • Program code contained on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, cable, optical cable, RF, etc., or any suitable combination of the above.
  • Program code for carrying out the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural programming languages. Programming language - such as "C" or a similar programming language.
  • the program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server to execute.
  • the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider). business to connect via the Internet).
  • LAN local area network
  • WAN wide area network
  • Internet service provider an Internet service provider
  • the present invention can be realized by hardware, or by software modules running on one or more processors, or by a combination thereof.
  • general data processing devices such as microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all functions of some or all components in the embodiments of the present invention.
  • DSPs digital signal processors
  • the present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein.
  • Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals.
  • Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

Abstract

Selon des modes de réalisation, la présente invention concerne un procédé de mise en cache de données. Le procédé consiste à : construire un modèle de mémoire cache sur la base d'une variable, et configurer, pour la variable dans le modèle de mémoire cache, des informations d'annotation qui ont une règle de mise en correspondance ; en réponse à un événement déclencheur de mémoire cache, appeler le modèle de mémoire cache pour lire les informations d'annotation à l'intérieur de celui-ci, acquérir une table de données source, déterminer un nom de champ cible, qui correspond à la règle de correspondance, dans la table de données source, et attribuer une valeur à la variable dans le modèle de mémoire cache sur la base du nom de champ cible ; et lire, à l'aide du modèle de mémoire cache soumis à une attribution de valeur, une valeur de champ correspondant au nom de champ cible dans la table de données de source, et générer des informations de paire clé-valeur et stocker celles-ci dans un espace de mémoire cache. Comme un modèle de mémoire cache peut être indépendant d'une table de données, le modèle de mémoire cache peut être appliqué à différentes tables de données, et un nom de champ dans une table de données source et une valeur de champ correspondante peuvent être automatiquement lus et mis en cache à l'avance, de telle sorte que la faible efficacité provoquée par la lecture temporaire de données à partir d'une table de données relationnelle est évitée, et il n'est pas nécessaire de développer individuellement un code pour chaque table de données, ce qui permet d'améliorer la commodité.
PCT/CN2022/123867 2021-10-09 2022-10-08 Procédé et appareil de mise en cache de données et dispositif électronique WO2023056946A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111176782.0A CN113918602A (zh) 2021-10-09 2021-10-09 一种数据缓存方法、装置和电子设备
CN202111176782.0 2021-10-09

Publications (1)

Publication Number Publication Date
WO2023056946A1 true WO2023056946A1 (fr) 2023-04-13

Family

ID=79238764

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/123867 WO2023056946A1 (fr) 2021-10-09 2022-10-08 Procédé et appareil de mise en cache de données et dispositif électronique

Country Status (2)

Country Link
CN (1) CN113918602A (fr)
WO (1) WO2023056946A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117033318A (zh) * 2023-10-08 2023-11-10 深圳安天网络安全技术有限公司 一种待测数据的生成方法、装置、存储介质及电子设备

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113918602A (zh) * 2021-10-09 2022-01-11 上海淇馥信息技术有限公司 一种数据缓存方法、装置和电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110490724A (zh) * 2019-07-03 2019-11-22 平安证券股份有限公司 账户数据的存储方法和装置
CN113434091A (zh) * 2021-07-07 2021-09-24 中国人民解放军国防科技大学 一种基于混合dram-nvm内存的冷热键值识别方法
CN113836162A (zh) * 2021-11-16 2021-12-24 湖北省楚建易网络科技有限公司 一种业务解耦并实现多级缓存的自动化更新的方法及装置
CN113918602A (zh) * 2021-10-09 2022-01-11 上海淇馥信息技术有限公司 一种数据缓存方法、装置和电子设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110490724A (zh) * 2019-07-03 2019-11-22 平安证券股份有限公司 账户数据的存储方法和装置
CN113434091A (zh) * 2021-07-07 2021-09-24 中国人民解放军国防科技大学 一种基于混合dram-nvm内存的冷热键值识别方法
CN113918602A (zh) * 2021-10-09 2022-01-11 上海淇馥信息技术有限公司 一种数据缓存方法、装置和电子设备
CN113836162A (zh) * 2021-11-16 2021-12-24 湖北省楚建易网络科技有限公司 一种业务解耦并实现多级缓存的自动化更新的方法及装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117033318A (zh) * 2023-10-08 2023-11-10 深圳安天网络安全技术有限公司 一种待测数据的生成方法、装置、存储介质及电子设备
CN117033318B (zh) * 2023-10-08 2023-12-08 深圳安天网络安全技术有限公司 一种待测数据的生成方法、装置、存储介质及电子设备

Also Published As

Publication number Publication date
CN113918602A (zh) 2022-01-11

Similar Documents

Publication Publication Date Title
US11163739B2 (en) Database table format conversion based on user data access patterns in a networked computing environment
US10635675B2 (en) Supporting pluggable databases with heterogeneous database character sets in a container database
WO2023056946A1 (fr) Procédé et appareil de mise en cache de données et dispositif électronique
CN111949693B (zh) 一种数据处理装置、数据处理方法、存储介质及电子设备
CN112513835A (zh) 用存储器内列式查询处理启用和集成存储器内半结构化数据和文本文档搜索的技术
US11200231B2 (en) Remote query optimization in multi data sources
US10248668B2 (en) Mapping database structure to software
US11074260B2 (en) Space-efficient methodology for representing label information in large graph data for fast distributed graph query
US20230393984A1 (en) In-memory normalization of cached objects to reduce cache memory footprint
CN109947768B (zh) 用于数据库对象的局部标识符
WO2024016594A1 (fr) Procédé et appareil de mise en œuvre de pseudo-colonne, dispositif électronique et support d'enregistrement
US9201937B2 (en) Rapid provisioning of information for business analytics
JP7291759B2 (ja) イーサリアム仮想マシンのトランザクション処理方法、装置、機器、プログラムおよび媒体
AU2019425532B2 (en) System and methods for loading objects from hash chains
US10114864B1 (en) List element query support and processing
AU2019386603B2 (en) Systems and methods for storing object state on hash chains
US20240126727A1 (en) Techniques for comprehensively supporting json schema in a rdbms
US20210357453A1 (en) Query usage based organization for very large databases
US20240126726A1 (en) Techniques for comprehensively supporting json schema in a rdbms
US11914610B1 (en) Multi-way data store conversion
CN117112189A (zh) 数据处理方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22877961

Country of ref document: EP

Kind code of ref document: A1