CN109408507B - Multi-attribute data processing method, device, equipment and readable storage medium - Google Patents
Multi-attribute data processing method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN109408507B CN109408507B CN201811130355.7A CN201811130355A CN109408507B CN 109408507 B CN109408507 B CN 109408507B CN 201811130355 A CN201811130355 A CN 201811130355A CN 109408507 B CN109408507 B CN 109408507B
- Authority
- CN
- China
- Prior art keywords
- data
- attribute
- key
- primary key
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims description 15
- 238000000034 method Methods 0.000 claims abstract description 36
- 238000000354 decomposition reaction Methods 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 11
- 238000010586 diagram Methods 0.000 description 11
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method, a device and equipment for processing multi-attribute data and a readable storage medium. The method comprises the following steps: acquiring primary key data, attribute data and attribute value data of data to be stored; if the primary key data exists in the Redis database, storing the attribute data and the attribute value data into the value of the key value pair corresponding to the primary key data; if the primary key data does not exist in the Redis database, a key value pair of the data to be stored is created, the primary key data is stored in a primary key of the key value pair, the attribute data and the attribute value data are stored in a value of the key value pair, and the attribute data is not stored in the primary key in the prior art, so that the number of the primary keys is extremely large.
Description
Technical Field
The embodiment of the invention relates to the technical field of databases, in particular to a method, a device and equipment for processing multi-attribute data and a readable storage medium.
Background
In the customer-oriented industries of telecommunication, finance, commerce and the like, the processed data has two major characteristics, namely, the data volume is large, the total amount can reach hundreds of millions or even billions, and each data has various data, namely, each customer data comprises various attribute categories. Therefore, for the service industries such as telecommunication, finance, business and the like, the data processing system faces the service industries such as telecommunication, finance, business and the like, and has great data volume and multiple attributes. When a large amount of multi-attribute data is stored, a Remote Dictionary Server (hereinafter, referred to as a Remote Dictionary Server) is generally used for storing. Redis is an open source Key-Value type memory database written using ANSI C language, storing data in a memory Key-Value dictionary structure and allowing other applications to read the contents in the dictionary via TCP protocol.
The Redis database supports five native Key-Value data types. Respectively as follows: string type, hash type, list type, collection type, and ordered collection type. In the case of multi-attribute data, the list type, the set type and the ordered set type do not satisfy the storage and use of the multi-attribute data. The multi-attribute data is stored using a string type and a hash type.
However, when the existing Redis database stores multi-attribute data by adopting a character string type and a hash type, the attribute of the data is stored by adopting a main key, so that the total amount of the main key data in the Redis database is extremely large, and further the operating efficiency of the Redis database is reduced.
Disclosure of Invention
Embodiments of the present invention provide a method, an apparatus, a device, and a readable storage medium for processing multi-attribute data, which solve the technical problem that when a Redis database in the prior art stores multi-attribute data by using a string type and a hash type, the total amount of primary key data in the Redis database is very large because the primary key is used to store the attribute of the data, and thus the total amount of the primary key data in the Redis database cannot be compressed, thereby reducing the operating efficiency of the Redis database.
In a first aspect, an embodiment of the present invention provides a method for processing multi-attribute data, including:
acquiring primary key data, attribute data and attribute value data of data to be stored;
if the primary key data exists in a Redis database, storing the attribute data and the attribute value data into the value of the key value pair corresponding to the primary key data;
if the primary key data does not exist in the Redis database, creating a key value pair of the data to be stored, storing the primary key data into a primary key in the key value pair, and storing the attribute data and the attribute value data into a value in the key value pair.
Further, the method for storing the attribute data and the attribute value data in the values in the key value pair specifically includes:
storing the attribute data in an attribute offset table in the value of the key-value pair;
storing the offset association of the attribute data into the attribute offset table;
storing the attribute value data at a corresponding offset position in the values of the key-value pairs.
Further, the method, after acquiring the primary key data, the attribute data and the attribute value data of the data to be stored, further includes:
determining the storage type of the data to be stored;
the storage type of the data to be stored is a character string type or a hash type.
Further, according to the method, if the storage type of the data to be stored is a hash type, the storing the primary key data into the primary key in the key value pair specifically includes:
acquiring a primary key data decomposition configuration strategy;
storing first primary Key data into a Key in the Key value pair according to the primary Key data decomposition configuration strategy, and storing second primary Key data into a Field Key in the Key value pair;
wherein the primary key data includes: the first primary key data and the second primary key data.
Further, the method as described above, further comprising:
receiving a data reading request, wherein the data reading request carries a main key identifier and a data attribute identifier;
determining a corresponding key value pair according to the primary key identification;
determining the offset of the corresponding attribute data in the key value pair according to the data attribute identification;
and acquiring corresponding attribute value data according to the offset of the attribute data.
In a second aspect, an embodiment of the present invention provides a device for processing multi-attribute data, including:
the acquisition module is used for acquiring the primary key data, the attribute data and the attribute value data of the data to be stored;
the storage module is used for storing the attribute data and the attribute value data into the value of the key value pair corresponding to the primary key data if the primary key data exists in a Redis database;
the creation module is used for creating the key value pair of the data to be stored if the primary key data does not exist in a Redis database;
the storage module is further configured to store the primary key data into a primary key in the key value pair, and store the attribute data and the attribute value data into a value in the key value pair.
Further, the apparatus as described above, the storage module comprising a first storage submodule for storing the attribute data into an attribute offset table in the values of the key-value pairs; storing the offset association of the attribute data into the attribute offset table; storing the attribute value data at a corresponding offset position in the values of the key-value pairs.
Further, the apparatus as described above, further comprising:
the first determining module is used for determining the storage type of the data to be stored; the storage type of the data to be stored is a character string type or a hash type.
Further, the apparatus as described above, the storage module, further comprising: a second storage submodule, configured to, if the storage type of the data to be stored is a hash type, specifically:
acquiring a primary key data decomposition configuration strategy; storing first primary Key data into a Key in the Key value pair according to the primary Key data decomposition configuration strategy, and storing second primary Key data into a Field Key in the Key value pair; wherein the primary key data includes: the first primary key data and the second primary key data.
Further, the apparatus as described above, further comprising: a receiving module and a second determining module;
the receiving module is used for receiving a data reading request, and the data reading request carries a main key identifier and a data attribute identifier;
the second determining module is used for determining the corresponding key value pair according to the primary key identification;
the second determining module is further configured to determine, according to the data attribute identifier, an offset of attribute data corresponding to the key value pair;
the obtaining module is further configured to obtain corresponding attribute value data according to the offset of the attribute data.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
a memory, a processor, and a computer program;
wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any of the first aspects.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, the computer program being executed by a processor to implement the method according to any one of the first aspect.
The embodiment of the invention provides a method, a device and equipment for processing multi-attribute data and a readable storage medium, which are characterized in that main key data, attribute data and attribute value data of data to be stored are obtained; if the primary key data exists in the Redis database, storing the attribute data and the attribute value data into the value of the key value pair corresponding to the primary key data; if the primary key data does not exist in the Redis database, a key value pair of the data to be stored is created, the primary key data is stored in a primary key of the key value pair, the attribute data and the attribute value data are stored in a value of the key value pair, and the attribute data is not stored in the primary key in the prior art, so that the number of the primary keys is extremely large.
It should be understood that what is described in the summary above is not intended to limit key or critical features of embodiments of the invention, nor is it intended to limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a flowchart of a method for processing multi-attribute data according to an embodiment of the present invention;
fig. 2 is a flowchart of a processing method of multi-attribute data according to a second embodiment of the present invention;
fig. 3 is a schematic diagram illustrating storage of character string types in the processing method of multi-attribute data according to the second embodiment of the present invention;
fig. 4 is a schematic diagram illustrating hash type storage in a processing method of multi-attribute data according to a second embodiment of the present invention;
fig. 5 is a schematic structural diagram of a device for processing multi-attribute data according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of a device for processing multi-attribute data according to a fourth embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and the embodiments of the present invention are illustrative only and are not intended to limit the scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, and in the above-described drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Fig. 1 is a flowchart of a processing method of multi-attribute data according to an embodiment of the present invention, and as shown in fig. 1, an execution main body of the embodiment is a processing apparatus of multi-attribute data, the processing apparatus of multi-attribute data may be integrated on an electronic device, and the electronic device may be a computer, a notebook computer, a tablet computer, a server, and the like. The processing method of the multi-attribute data provided by the embodiment includes the following steps.
Step 101, acquiring primary key data, attribute data and attribute value data of data to be stored.
In this embodiment, the data to be stored is multi-attribute data that needs to be stored in the Redis database.
Specifically, in this embodiment, the data to be stored is acquired, the data to be stored is analyzed, and the primary key data, the attribute data, and the attribute value data in the data to be stored are acquired.
The method for acquiring the data to be stored may be to receive a storage request sent by another device, where the storage request carries the data to be stored. Or pre-store the data to be stored in the fixed area, acquire the data to be stored from the fixed area, or acquire the data to be stored in other manners, which is not limited in this embodiment.
In the Redis database, data to be stored is stored in a Key-Value Key Value pair mode. The storage mode of the multi-attribute data can be a character string type or a hash type. The main Key data is data which needs to be stored in a Key or a Key and a Field Key, and the attribute data and the attribute Value data are data which needs to be stored in Value.
And 102, if the primary key data exists in the Redis database, storing the attribute data and the attribute value data into the value of the key value pair corresponding to the primary key data.
Specifically, in this embodiment, it is determined whether the primary key data of the data to be stored exists in the Redis database, and the determination may be performed in a manner of comparing the primary key data of the data to be stored with the primary key data in the Redis database. If the primary Key data exists in the Redis database, the attribute data and the attribute Value data are stored in the Value of the Key Value pair corresponding to the primary Key data, that is, the attribute data and the attribute Value data are stored in the Value of the Key-Value corresponding to the primary Key data, and the storage manner in the Value is not limited in this embodiment.
And 103, if the primary key data does not exist in the Redis database, creating a key value pair of the data to be stored, storing the primary key data into a primary key in the key value pair, and storing the attribute data and the attribute value data into a value in the key value pair.
Specifically, in this embodiment, if the primary Key data does not exist in the Redis database, a Key Value pair, i.e., Key-Value, of the data to be stored is created in the Redis database, the primary Key data is stored in the primary Key in the Key Value pair, and if the storage type is the character string type, the primary Key data is stored in the Key. And if the storage type is the hash type, storing the primary Key into the Key Key and the Field Key. Storing the attribute data and the attribute value data into values in the key value pair.
In the processing method of multi-attribute data provided by this embodiment, the primary key data, the attribute data and the attribute value data of the data to be stored are acquired; if the primary key data exists in the Redis database, storing the attribute data and the attribute value data into the value of the key value pair corresponding to the primary key data; if the primary key data does not exist in the Redis database, a key value pair of the data to be stored is created, the primary key data is stored in a primary key of the key value pair, the attribute data and the attribute value data are stored in a value of the key value pair, and the attribute data is not stored in the primary key in the prior art, so that the number of the primary keys is extremely large.
Example two
Fig. 2 is a flowchart of a processing method of multi-attribute data according to a second embodiment of the present invention, and as shown in fig. 2, the processing method of multi-attribute data according to the present embodiment is further detailed in steps 102 to 103 on the basis of the processing method of multi-attribute data according to the first embodiment of the present invention, and further includes a step of reading data in a Redis database, so that the processing method of multi-attribute data according to the present embodiment includes the following steps.
Step 201, obtaining the primary key data, attribute data and attribute value data of the data to be stored.
In this embodiment, an implementation manner of step 201 is the same as that of step 101 of the method for processing multi-attribute data provided in the first embodiment of the present invention, and details are not repeated here.
The storage type of the data to be stored is a character string type or a hash type.
Step 202, judging whether the primary key data exists in the Redis database, if so, executing step 203, otherwise, executing step 204.
Step 203, storing the attribute data and the attribute value data in the value of the key value pair corresponding to the primary key data.
Further, in this embodiment, storing the attribute data and the attribute value data in the value of the key-value pair corresponding to the primary key data specifically includes:
first, attribute data is stored into an attribute offset table in the values of key-value pairs.
Secondly, the offset association of the attribute data is stored in an attribute offset table.
Finally, the attribute value data is stored at the corresponding offset position in the value of the key-value pair.
Specifically, in the present embodiment, the key value pair of the present embodiment includes a primary key and a value. Each attribute data and the corresponding attribute value data are included in the value. If the primary key data exists in the Redis database, when storing the attribute data and the attribute value data in the value of the key-value pair corresponding to the primary key data, the value has two parts, one part is the attribute offset table, and the other part is the attribute value data. In the attribute offset table, each attribute data and the offset amount corresponding to the attribute data are stored in association with each other. In the attribute value data, the attribute value data is stored in accordance with the position of the offset.
The offset corresponding to the attribute data may be a starting position of the offset.
Fig. 3 is a schematic diagram of storing character string types in the processing method of multi-attribute data according to the second embodiment of the present invention, and fig. 4 is a schematic diagram of storing hash types in the processing method of multi-attribute data according to the second embodiment of the present invention. As shown in fig. 3 and 4, in the values of the key value pairs, the attribute data and the attribute value data are both stored in the values, so that the total amount of the primary key data in the Redis database can be greatly reduced, and the operating efficiency of the Redis database is further improved.
Further, in this embodiment, after acquiring the primary key data, the attribute data, and the attribute value data of the data to be stored, the method further includes:
determining the storage type of the data to be stored.
The storage type of the data to be stored is a character string type or a hash type.
Further, in this embodiment, the manner of determining the storage type of the data to be stored may be: and determining the storage type of the data to be stored according to a preset strategy, wherein the preset strategy can be that if the primary key data exists in the Redis database, the primary key data is stored according to the original storage type in the Redis database. If the primary key data does not exist in the Redis database, the storage type may be determined according to the data size of the data to be stored or according to another preset policy, which is not limited in this embodiment.
And 204, creating a key value pair of the data to be stored, storing the primary key data into the primary key in the key value pair, and storing the attribute data and the attribute value data into the value in the key value pair.
Further, if the storage type of the data to be stored is a hash type, storing the primary key data into the primary key in the key value pair, specifically including:
first, a primary key data decomposition configuration policy is acquired.
Specifically, in this embodiment, a hash-type primary key data decomposition configuration policy is stored in advance. The primary Key data decomposition configuration strategy comprises a configuration strategy of first primary Key data and second primary Key data, wherein the configuration strategy is used for storing the first primary Key data into a Key in a Key value pair in a configuration mode and storing the second primary Key data into a Field Key in the Key value pair in a configuration mode.
And secondly, storing the first primary Key data into a Key in the Key value pair according to a primary Key data decomposition configuration strategy, and storing the second primary Key data into a Field Key in the Key value pair.
Specifically, in this embodiment, the primary key data includes: first primary key data and second primary key data. And acquiring first primary Key data and second primary Key data from the primary Key data, storing the first primary Key data into a Key in a Key value pair according to a primary Key data decomposition configuration strategy, and storing the second primary Key data into a Field Key in the Key value pair.
In this embodiment, for obtaining the primary Key data decomposition configuration policy, the first primary Key data is stored in a Key of the Key value pair according to the primary Key data decomposition configuration policy, and the storage of the second primary Key data in a Field Key of the Key value pair is exemplarily described as follows: the primary Key data is a mobile phone number, the primary Key data decomposition configuration strategy is to determine the first 7 bits of the mobile phone number as first primary Key data, determine the last 4 bits of the mobile phone number as second primary Key data, store the first 7 bits of the mobile phone number in a Key of a Key value pair, and store the last four bits of data in a Field Key of the Key value pair.
In this embodiment, if the storage type of the data to be stored is a hash type, a primary key data decomposition configuration policy is acquired; the method comprises the steps of storing first main Key data into a Key in a Key value pair according to a main Key data decomposition configuration strategy, storing second main Key data into a Field Key in the Key value pair, releasing the Field Key from the original function of storing attribute data, enabling the application range of the Field Key to be wider, classifying the main Key data, storing second main Key data representing secondary characteristics in the Field Key, and effectively reducing the total amount of the Key, thereby improving the operating efficiency of a Redis database.
In this embodiment, if the storage type of the data to be stored is a character string type, the primary key data is stored in the primary key in the key value pair, which specifically includes: and storing all primary Key data into the Key in the Key value pair.
It should be noted that, in this embodiment, in step 205, the manner of storing the attribute data and the attribute value data into the value in the key value pair is the same as the implementation manner in step 204 in the second embodiment of the present invention, and details are not repeated here.
The primary key identifier may be a serial number, etc. of the primary key data, which is not limited in this embodiment. Similarly, the data attribute identifier may be a serial number, or the like of the data attribute, which is not limited in this embodiment.
Specifically, in this embodiment, a data reading request sent by a user through an electronic device may be received, and the electronic device may be a computer, a notebook computer, a tablet computer, a server, and the like, which is not limited in this embodiment.
And step 206, determining a corresponding key value pair according to the primary key identification.
Specifically, in this embodiment, the storage type of the read data may be determined according to the primary Key identifier, and if the storage type is the hash type, it may be determined whether the primary Key identifier exists in the hash type data according to the Key and the Field Key, and the primary Key identifier is matched with the Key and the Field Key to determine whether a Key value pair corresponding to the primary Key identifier exists. If the storage type is the character string type, whether the primary Key identification exists in the data of the character string type can be determined according to the Key, the primary Key identification is matched with the Key, and whether a Key value pair corresponding to the primary Key identification exists is judged.
Step 207, determining the offset of the corresponding attribute data in the key value pair according to the data attribute identifier.
Further, after determining that the main key identifier determines the corresponding key-value pair, determining the offset of the attribute data corresponding to the data attribute identifier in the values of the key-value pair according to the data attribute identifier.
The offset of the attribute data may be an offset start position.
And step 208, acquiring corresponding attribute value data according to the offset of the attribute data.
Further, since the offset amount of the attribute data is the storage position of the attribute value data corresponding to the attribute data, the corresponding attribute value data is acquired in accordance with the offset amount of the attribute data.
It will be appreciated that if it is determined from the primary key identification that the corresponding key-value pair is not included in the Redis database, then a null value is returned.
The method for processing multi-attribute data provided in this embodiment determines whether primary key data exists in a Redis database by obtaining primary key data, attribute data, and attribute value data of data to be stored, stores the attribute data and the attribute value data into values of key value pairs corresponding to the primary key data if the primary key data exists in the Redis database, otherwise creates key value pairs of the data to be stored, stores the primary key data into primary keys in the key value pairs, stores the attribute data and the attribute value data into the values in the key value pairs, receives a data reading request, where the data reading request carries primary key identifiers and data attribute identifiers, determines corresponding key value pairs according to the primary key identifiers, determines offsets of the attribute data corresponding to the key value pairs according to the data attribute identifiers, and obtains corresponding attribute value data according to the offsets of the attribute data. For multi-attribute data with the storage types of character strings and hashes, the compression of the total amount of the primary key data is realized, the total amount of the primary key data in the Redis database is greatly reduced, the storage efficiency of the Redis database is improved, and the reading efficiency of the Redis database is improved.
EXAMPLE III
FIG. 5 is a schematic structural diagram of a device for processing multi-attribute data according to a third embodiment of the present invention, such as
As shown in fig. 5, the apparatus 50 for processing multi-attribute data according to the present embodiment includes: an acquisition module 51, a storage module 52 and a creation module 53.
The obtaining module 51 is configured to obtain primary key data, attribute data, and attribute value data of data to be stored. And a storage module 52, configured to store the attribute data and the attribute value data in the value of the key-value pair corresponding to the primary key data if the primary key data exists in the Redis database. The creating module 53 creates a key-value pair of the data to be stored if the primary key data does not exist in the Redis database. The storage module 52 is further configured to store the primary key data into the primary key in the key value pair, and store the attribute data and the attribute value data into the value in the key value pair.
The processing apparatus for multi-attribute data provided in this embodiment may execute the technical solution of the method embodiment shown in fig. 1, and the implementation principle and the technical effect are similar, which are not described herein again.
Example four
Fig. 6 is a schematic structural diagram of a processing apparatus for multi-attribute data according to a fourth embodiment of the present invention, and as shown in fig. 6, the processing apparatus for multi-attribute data according to the present embodiment further includes, on the basis of the processing apparatus for multi-attribute data according to the first embodiment of the present invention: a first determining module 61, a receiving module 62 and a second determining module 63.
Further, the storage module 52 includes a first storage submodule 521 for storing the attribute data into the attribute offset table in the value of the key-value pair; storing the offset of the attribute data into an attribute offset table in an associated manner; the attribute value data is stored at the corresponding offset position in the value of the key-value pair.
Further, the first determining module 61 is configured to determine a storage type of the data to be stored; the storage type of the data to be stored is a character string type or a hash type.
Further, the storage module 52 further includes: the second storage submodule 521 is configured to, if the storage type of the data to be stored is a hash type, specifically: acquiring a primary key data decomposition configuration strategy; storing first primary Key data into a Key in a Key value pair according to a primary Key data decomposition configuration strategy, and storing second primary Key data into a Field Key in the Key value pair; wherein the primary key data includes: first primary key data and second primary key data.
Further, the receiving module 62 is configured to receive a data reading request, where the data reading request carries a primary key identifier and a data attribute identifier. And a second determining module 63, configured to determine a corresponding key value pair according to the primary key identifier. The second determining module 63 is further configured to determine, according to the data attribute identifier, an offset of the attribute data corresponding to the key value pair. The obtaining module 51 is further configured to obtain corresponding attribute value data according to the offset of the attribute data.
The processing apparatus for multi-attribute data provided in this embodiment may execute the technical solution of the method embodiment shown in fig. 2, and the implementation principle and the technical effect are similar, which are not described herein again.
EXAMPLE five
Fig. 7 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention, and as shown in fig. 7, the electronic device according to the fifth embodiment of the present invention includes: a memory 71, a processor 72 and a computer program.
Among them, the computer program is stored in the memory 71 and configured to be executed by the processor 72 to implement the method of processing the multi-attribute data provided in the first embodiment of the present invention or the method of processing the multi-attribute data provided in the second embodiment of the present invention.
The relevant description may be understood by referring to the relevant description and effect corresponding to the steps in fig. 1 to fig. 2, and redundant description is not repeated here.
EXAMPLE six
The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the method for processing multi-attribute data provided in the first embodiment of the present invention or the method for processing multi-attribute data provided in the second embodiment of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of modules is merely a division of logical functions, and an actual implementation may have another division, for example, a plurality of modules or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
Modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware form, and can also be realized in a form of hardware and a software functional module.
Program code for implementing the methods of the present invention may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Further, while operations are depicted in a particular order, this should be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
Claims (10)
1. A processing method of multi-attribute data is characterized by comprising the following steps:
acquiring primary key data, attribute data and attribute value data of data to be stored;
if the primary key data exists in a Redis database, storing the attribute data and the attribute value data into the value of the key value pair corresponding to the primary key data;
if the primary key data does not exist in a Redis database, creating a key value pair of the data to be stored, storing the primary key data into a primary key in the key value pair, and storing the attribute data and the attribute value data into a value in the key value pair;
storing the attribute data and the attribute value data into values in the key value pair specifically includes:
storing the attribute data in an attribute offset table in the value of the key-value pair;
storing the offset association of the attribute data into the attribute offset table;
storing the attribute value data at an offset position corresponding to an offset of the attribute data in the values of the key-value pairs.
2. The method according to claim 1, wherein after acquiring the primary key data, the attribute data and the attribute value data of the data to be stored, the method further comprises:
determining the storage type of the data to be stored;
the storage type of the data to be stored is a character string type or a hash type.
3. The method according to claim 2, wherein if the storage type of the data to be stored is a hash type, the storing the primary key data into the primary key in the key value pair specifically includes:
acquiring a primary key data decomposition configuration strategy;
storing first primary Key data into a Key in the Key value pair according to the primary Key data decomposition configuration strategy, and storing second primary Key data into a Field Key in the Key value pair;
wherein the primary key data includes: the first primary key data and the second primary key data.
4. The method of claim 1, further comprising:
receiving a data reading request, wherein the data reading request carries a main key identifier and a data attribute identifier;
determining a corresponding key value pair according to the primary key identification;
determining the offset of the corresponding attribute data in the key value pair according to the data attribute identification;
and acquiring corresponding attribute value data according to the offset of the attribute data.
5. An apparatus for processing multi-attribute data, comprising:
the acquisition module is used for acquiring the primary key data, the attribute data and the attribute value data of the data to be stored;
the storage module is used for storing the attribute data and the attribute value data into the value of the key value pair corresponding to the primary key data if the primary key data exists in a Redis database;
the creation module is used for creating the key value pair of the data to be stored if the primary key data does not exist in a Redis database;
the storage module is further configured to store the primary key data into a primary key in the key value pair, and store the attribute data and the attribute value data into a value in the key value pair;
the storage module comprises a first storage submodule and a second storage submodule, wherein the first storage submodule is used for storing the attribute data into an attribute offset table in the values of the key-value pairs; storing the offset association of the attribute data into the attribute offset table; storing the attribute value data at an offset position corresponding to an offset of the attribute data in the values of the key-value pairs.
6. The apparatus of claim 5, further comprising:
the first determining module is used for determining the storage type of the data to be stored; the storage type of the data to be stored is a character string type or a hash type.
7. The apparatus of claim 6, wherein the storage module further comprises: a second storage submodule, configured to, if the storage type of the data to be stored is a hash type, specifically:
acquiring a primary key data decomposition configuration strategy; storing first primary Key data into a Key in the Key value pair according to the primary Key data decomposition configuration strategy, and storing second primary Key data into a Field Key in the Key value pair; wherein the primary key data includes: the first primary key data and the second primary key data.
8. The apparatus of claim 5, further comprising: a receiving module and a second determining module;
the receiving module is used for receiving a data reading request, and the data reading request carries a main key identifier and a data attribute identifier;
the second determining module is used for determining the corresponding key value pair according to the primary key identification;
the second determining module is further configured to determine, according to the data attribute identifier, an offset of attribute data corresponding to the key value pair;
the obtaining module is further configured to obtain corresponding attribute value data according to the offset of the attribute data.
9. An electronic device, comprising:
a memory, a processor, and a computer program;
wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-4.
10. A computer-readable storage medium, having stored thereon a computer program for execution by a processor to perform the method of any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811130355.7A CN109408507B (en) | 2018-09-27 | 2018-09-27 | Multi-attribute data processing method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811130355.7A CN109408507B (en) | 2018-09-27 | 2018-09-27 | Multi-attribute data processing method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109408507A CN109408507A (en) | 2019-03-01 |
| CN109408507B true CN109408507B (en) | 2021-07-27 |
Family
ID=65466460
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811130355.7A Active CN109408507B (en) | 2018-09-27 | 2018-09-27 | Multi-attribute data processing method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109408507B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110609766A (en) * | 2019-09-04 | 2019-12-24 | 深圳前海微众银行股份有限公司 | KV data storage method and device based on Redis protocol |
| CN110609926A (en) * | 2019-09-20 | 2019-12-24 | 中国银行股份有限公司 | Data tag storage management method and device |
| CN110737661A (en) * | 2019-10-08 | 2020-01-31 | 深圳市高斯通信息技术有限公司 | extremely-compact space caching technology |
| CN111274245B (en) * | 2020-01-17 | 2022-07-12 | 苏州浪潮智能科技有限公司 | Method and device for optimizing data storage |
| CN113297224B (en) * | 2021-05-31 | 2022-06-28 | 上海艾麒信息科技股份有限公司 | Mass data classification storage method and system based on Redis |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104866608A (en) * | 2015-06-05 | 2015-08-26 | 中国人民大学 | Query optimization method based on join index in data warehouse |
| CN106991149A (en) * | 2017-03-28 | 2017-07-28 | 桂林电子科技大学 | A kind of magnanimity spatial object storage method for merging coding and multi-edition data |
| CN107464133A (en) * | 2017-07-05 | 2017-12-12 | 微梦创科网络科技（中国）有限公司 | A kind of method and device for launching advertisement |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9305075B2 (en) * | 2009-05-29 | 2016-04-05 | Oracle International Corporation | Extending dynamic matrices for improved setup capability and runtime search performance of complex business rules |
-
2018
- 2018-09-27 CN CN201811130355.7A patent/CN109408507B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104866608A (en) * | 2015-06-05 | 2015-08-26 | 中国人民大学 | Query optimization method based on join index in data warehouse |
| CN106991149A (en) * | 2017-03-28 | 2017-07-28 | 桂林电子科技大学 | A kind of magnanimity spatial object storage method for merging coding and multi-edition data |
| CN107464133A (en) * | 2017-07-05 | 2017-12-12 | 微梦创科网络科技（中国）有限公司 | A kind of method and device for launching advertisement |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109408507A (en) | 2019-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109408507B (en) | Multi-attribute data processing method, device, equipment and readable storage medium | |
| CN107832406B (en) | Method, device, equipment and storage medium for removing duplicate entries of mass log data | |
| CN108536745B (en) | Shell-based data table extraction method, terminal, equipment and storage medium | |
| CN108415998B (en) | Application dependency relationship updating method, terminal, device and storage medium | |
| CN112052138A (en) | Service data quality detection method and device, computer equipment and storage medium | |
| CN113127125B (en) | Page automatic adaptation method, device, equipment and storage medium | |
| US20190065518A1 (en) | Context aware delta algorithm for genomic files | |
| CN112214505B (en) | Data synchronization method, device, computer readable storage medium and electronic equipment | |
| CN111078672A (en) | Data comparison method and device for database | |
| CN110688305B (en) | Test environment synchronization method, device, medium and electronic equipment | |
| CN111445319A (en) | Voucher generation method and device, computer equipment and storage medium | |
| CN111858581B (en) | Paging query method and device, storage medium and electronic equipment | |
| CN110647753B (en) | Method, device and equipment for acquiring kernel file and storage medium | |
| CN112434037A (en) | Data processing method, processing device, data processing apparatus, and storage medium | |
| CN116010461A (en) | Data blood relationship analysis method and device, storage medium and electronic equipment | |
| CN116010345A (en) | Method, device and equipment for realizing table service scheme of flow batch integrated data lake | |
| CN112579357B (en) | Snapshot difference obtaining method, device, equipment and storage medium | |
| CN110471708B (en) | Method and device for acquiring configuration items based on reusable components | |
| CN112417012A (en) | Data processing method and system | |
| CN112612817A (en) | Data processing method and device, terminal equipment and computer readable storage medium | |
| CN112182085A (en) | Data export method, device, equipment and storage medium | |
| CN110580243A (en) | file comparison method and device, electronic equipment and storage medium | |
| CN114490095B (en) | Request result determination method and device, storage medium and electronic device | |
| CN113468125B (en) | File service method, system, equipment and computer readable storage medium | |
| CN114328529A (en) | Method and system for sub-table query of large data table by mybatis interceptor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |