CN110766501A - Data nesting storage method and device, electronic equipment and computer readable medium - Google Patents
Data nesting storage method and device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN110766501A CN110766501A CN201810847055.4A CN201810847055A CN110766501A CN 110766501 A CN110766501 A CN 110766501A CN 201810847055 A CN201810847055 A CN 201810847055A CN 110766501 A CN110766501 A CN 110766501A
- Authority
- CN
- China
- Prior art keywords
- data
- granularity
- processed
- filling
- nesting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000013500 data storage Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 abstract description 13
- 235000019580 granularity Nutrition 0.000 description 134
- 238000010586 diagram Methods 0.000 description 13
- 238000012545 processing Methods 0.000 description 10
- 238000004590 computer program Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 4
- 238000013499 data model Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 239000000945 filler Substances 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013481 data capture Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/06—Buying, selling or leasing transactions
- G06Q30/0601—Electronic shopping [e-shopping]
- G06Q30/0633—Lists, e.g. purchase orders, compilation or processing
- G06Q30/0635—Processing of requisition or of purchase orders
Landscapes
- Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure provides a data nesting storage method and device, electronic equipment and a computer readable medium, and belongs to the technical field of internet. The method comprises the following steps: judging the granularity of received data to be processed to obtain granularity levels, wherein the data to be processed at least comprises two granularity levels; constructing a nested data structure for the data to be processed according to the granularity level, wherein the nested data structure comprises a data primary key corresponding to the data to be processed; filling the data to be processed according to the data primary key to obtain filling data, wherein the data primary key is marked by the granularity level with the highest level; and storing the filling data into a database according to the data main key memory. The nested data structures are respectively constructed for the data with different granularity levels, so that cache waiting is not needed when the data are stored in a warehouse, the data with any granularity can be stored in the warehouse in real time, the data query performance is improved, and the multi-data warehousing process is simplified.
Description
Technical Field
The present disclosure relates generally to the field of internet technologies, and in particular, to a data nesting storage method and apparatus, an electronic device, and a computer-readable medium.
Background
The goods sold by the e-commerce platform to the platform user (i.e., the buyer) are the goods sold by the third party merchant to the user based on the e-commerce platform in addition to the self-owned goods. The e-commerce platform needs to capture order data of a third-party merchant, and the order data is divided into three kinds of data with different granularities in an order settlement model according to the different granularities of the data, namely coarse-grained data, medium-grained data and fine-grained data. The coarse-grained data refers to order data and records basic information of the order; the medium granularity data refers to orders and settlement subject data, and describes the settlement condition of each order corresponding to each merchant, such as settlement progress, settlement account information of the merchant and the like; the fine-grained data refers to order settlement detail data, and records settlement expense detail information of each order to each merchant.
In the prior art, the distributed real-time computing platform needs to merge the three kinds of data with different granularities, and then store the merged data into the distributed database according to the finest granularity (i.e. order settlement detail data). Because data storage is according to the finest granularity (order settlement detail data), the fine-granularity data must be put in storage simultaneously with the coarse-granularity data or in advance, but the real-time acquisition time sequence of three data sources corresponding to the same order cannot be guaranteed, and therefore before storage, caching waiting logic (all fine-granularity data corresponding to the same coarse-granularity data are acquired) is needed for storage. In this process, both the cache and the data stored in the database are maintained, and when frequently updated, the maintenance is difficult. And when the fine-grained data of the same type cannot be judged whether to be acquired, the data cannot be put in storage in real time.
Therefore, there is still a need for improvement in the prior art solutions.
The above information disclosed in this background section is only for enhancement of understanding of the background of the disclosure and therefore it may contain information that does not constitute prior art that is already known to a person of ordinary skill in the art.
Disclosure of Invention
The present disclosure provides a data nesting storage method, device, electronic device and computer readable medium, which solve the above technical problems.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.
According to an aspect of the present disclosure, there is provided a data nesting storage method, including:
judging the granularity of received data to be processed to obtain granularity levels, wherein the data to be processed at least comprises two granularity levels;
constructing a nested data structure for the data to be processed according to the granularity level, wherein the nested data structure comprises a data primary key corresponding to the data to be processed;
filling the data to be processed according to the data main key to obtain filling data;
and storing the filling data into a database according to the data main key memory.
In an embodiment of the present disclosure, before determining the granularity of the received to-be-processed data, the method further includes:
distributing the data to be processed with different granularity levels according to a routing strategy.
In one embodiment of the present disclosure, the data to be processed includes three granularity levels, i.e., coarse granularity, medium granularity, and fine granularity, and the primary key of the data is marked with a coarse-granularity identification number.
In an embodiment of the present disclosure, the padding the to-be-processed data according to the data master key, and obtaining padding data includes:
and when the granularity level of the data to be processed is coarse granularity, filling the field of the coarse granularity field of the data to be processed according to the data primary key.
In an embodiment of the present disclosure, the padding the to-be-processed data according to the data master key, and obtaining padding data includes:
and when the granularity level of the data to be processed is the medium granularity, filling the data of the coarse granularity field and the medium granularity field of the data to be processed according to the data main key.
In an embodiment of the present disclosure, the padding the to-be-processed data according to the data master key, and obtaining padding data includes:
and when the granularity level of the data to be processed is fine granularity, filling the data of the coarse granularity field, the medium granularity field and the fine granularity field of the data to be processed according to the data main key.
In one embodiment of the present disclosure, storing the filler data into the database according to the data primary key includes:
and inserting or updating the filling data into a data storage model according to the data primary key, wherein the data storage model is an Elasticissearch storage model.
According to still another aspect of the present disclosure, there is provided a data nesting storage apparatus including:
the granularity judging module is configured to judge the granularity of the received data to be processed to obtain granularity levels, wherein the data to be processed at least comprises two granularity levels;
the nesting module is configured to construct a nesting data structure for the data to be processed according to the granularity level, and the nesting data structure contains a data primary key corresponding to the data to be processed;
the filling module is configured to fill the data to be processed according to the data main key to obtain filling data;
and the storage module is configured to store the filling data into a database according to the data primary key storage.
According to yet another aspect of the present disclosure, there is provided an electronic device comprising a processor; a memory storing instructions for the processor to control the method steps as described above.
According to another aspect of the present disclosure, there is provided a computer-readable medium having stored thereon computer-executable instructions that, when executed by a processor, implement the method steps as described above.
According to the data nesting storage method, the data nesting storage device, the electronic equipment and the computer readable medium, on one hand, the nesting data structures are respectively constructed for data of different granularity levels, so that caching waiting is not needed when the data are put in storage, the data of any granularity can be put in storage in real time, the data query performance is improved, and the multi-data storage process is simplified. On the other hand, since the statistics is performed using the coarse granularity having the highest granularity level as an index, deduplication is not required, and the indexes to be summed up by the coarse granularity statistics value can be directly summed up, and there is no problem of duplicate summing up in the summation result.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.
Fig. 1 shows a flowchart of a data nesting storage method provided in an embodiment of the present disclosure.
FIG. 2 shows a model diagram of a nested data structure in an embodiment of the present disclosure.
Fig. 3 shows a flowchart of a data processing method provided in an embodiment of the present disclosure.
Fig. 4 shows a schematic diagram of a data nesting storage device provided in another embodiment of the present disclosure.
FIG. 5 is a diagram showing an overall architecture of a data processing system in another embodiment of the present disclosure.
Fig. 6 shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present application, provided by an embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and the like. In other instances, well-known structures, methods, devices, implementations, materials, or operations are not shown or described in detail to avoid obscuring aspects of the disclosure.
Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to specific embodiments and the accompanying drawings.
In the related embodiment of the disclosure, in the order settlement model for the third-party merchant, three kinds of data with different granularities are generated according to the granularity of the data, and the e-commerce platform needs to perform some real-time statistical analysis based on the three kinds of data in order to perform real-time monitoring analysis on the whole process of order settlement.
Through the distributed real-time computing platform, the e-commerce platform merges the three kinds of data and stores the merged data into the distributed database Elasticsearch according to the finest granularity (settlement granularity of the order), but the following problems can occur:
firstly, if the indexes (such as order quantity) are calculated according to the coarse-grained statistics, the duplication removal is needed, and the performance of the database is very consumed by the duplication removal operation; secondly, according to the indexes (such as the sum of the order amount) summed by the coarse granularity statistics value, the direct summation has the problem of repeated summation, so that the data statistics is wrong; finally, since data storage is according to the finest granularity (such as order settlement details), fine-grained data must be put in storage simultaneously with coarse-grained data or in advance, but since three data sources corresponding to the same order are obtained in real time, the time sequence cannot be guaranteed, and before putting in storage, caching wait logic (all fine-grained data corresponding to the same coarse-grained data are obtained) is needed to put in storage. In the process, both the cache and the data stored in the database need to be maintained, and the maintenance is difficult when the data are frequently updated. When the situation that whether the fine-grained data of the same type are acquired cannot be judged, the scheme has the defect that real-time storage cannot be achieved.
Based on the above problems, the present disclosure provides a new data nesting storage method, apparatus, electronic device and computer readable medium.
Fig. 1 shows a flowchart of a data nesting storage method provided in an embodiment of the present disclosure, which includes the following steps:
as shown in fig. 1, in step S110, the granularity of the received to-be-processed data is determined, and granularity levels are obtained, where the to-be-processed data at least includes two granularity levels.
As shown in fig. 1, in step S120, a nested data structure is constructed for the to-be-processed data according to the granularity level, where the nested data structure includes a data primary key corresponding to the to-be-processed data.
As shown in fig. 1, in step S130, the data to be processed is filled according to the data master key, so as to obtain filled data.
As shown in fig. 1, in step S140, the filling data is stored in the database according to the data primary key.
According to the data nesting storage method provided by the embodiment of the disclosure, on one hand, nesting data structures are respectively constructed for data of different granularity levels, so that caching waiting is not needed when the data are put in storage, the data of any granularity can be put in storage in real time, the data query performance is improved, and the multi-data storage process is simplified. On the other hand, since the statistics is performed using the coarse granularity having the highest granularity level as an index, deduplication is not required, and the indexes to be summed up by the coarse granularity statistics value can be directly summed up, and there is no problem of duplicate summing up in the summation result.
The following describes in detail a data nesting storage method provided by the present disclosure with reference to a flowchart shown in fig. 1, specifically as follows:
in step S110, the granularity of the received to-be-processed data is determined to obtain granularity levels, where the to-be-processed data includes at least two granularity levels.
In one embodiment of the present disclosure, the data granularity refers to the refinement and integration degree of data in the data warehouse, and according to the data granularity refinement criterion: the higher the refinement degree is, the smaller the granularity is; the lower the degree of refinement, the larger the particle size. In this embodiment, data to be processed is described as including three granularity levels, that is, the data to be processed includes three granularity levels, namely, coarse granularity, medium granularity, and fine granularity.
The data primary key is marked by a coarse-grained identification number (namely coarse-grained ID), and the data primary key is a short name of the database primary key and is a unique identifier capable of determining one record in the database. For example, student tables (school number, name, gender, class) in which the number of each student is unique, the school number being a primary key. In this embodiment, for the order settlement data model, the data primary key may be an order number, which is a unique label for a record.
In an embodiment of the present disclosure, before determining the granularity of the received to-be-processed data, the method further includes:
distributing the data to be processed with different granularity levels according to a routing strategy.
Routing policies are techniques for modifying routing information in order to change the way network traffic travels, and are implemented primarily by changing routing attributes, including reachability. The routing strategy is a more flexible packet routing and forwarding mechanism than routing based on a target network. The router determines how to process the data packet to be routed through a routing graph, which determines the router for next hop of a data packet.
Taking three granularity levels as an example, three kinds of granularity data (namely, coarse granularity data, medium granularity data and fine granularity data) under the same coarse granularity ID (namely, primary key ID) are distributed to the same downstream node through a routing strategy to be put into storage.
It should be noted that, in a data model including different levels of granularity, a coarse-grained (or highest-level-granularity) ID is usually selected as the data primary key ID.
In step S120, a nested data structure is constructed for the data to be processed according to the granularity level.
In an embodiment of the present disclosure, still taking three granularity levels as an example, fig. 2 is a model diagram illustrating a nested data structure, as shown in fig. 2, the nested data structure includes three levels according to the three granularity levels, a first level is coarse-granularity data, each piece of coarse-granularity data, in addition to basic information of coarse granularity (i.e., basic information of an order), may contain a plurality of pieces of medium-granularity data, such as medium-granularity data 1, medium-granularity data 2 … … medium-granularity data N; the second level is medium-granularity data, each of which may contain a plurality of fine-granularity data in addition to medium-granularity basic information (i.e., order-settlement subject information), such as fine-granularity data 1 and fine-granularity data 2 … … under medium-granularity data 1, and fine-granularity data m under other medium-granularity data, similarly, and each of which includes order-settlement detail data.
Based on the above-mentioned nesting mode in terms of coarse granularity, medium granularity and fine granularity step by step, the nested data structure shown in fig. 2 is formed, and accordingly, if the data model in other embodiments of the present disclosure includes more granularity levels, the formed nested data structure is also in a plurality of hierarchical relationships and is nested step by step from coarse granularity to fine granularity.
In step S130, the data to be processed is filled according to the data primary key, so as to obtain filled data.
In an embodiment of the present disclosure, taking three granularity levels as an example, the obtaining of the filling data by filling the to-be-processed data according to the data primary key includes:
(1) when the granularity level of the data to be processed is coarse granularity, filling fields of coarse granularity fields of the data to be processed according to the data main key;
(2) when the granularity level of the data to be processed is medium granularity, filling data of a coarse granularity field and a medium granularity field of the data to be processed according to the data main key;
(3) and when the granularity level of the data to be processed is fine granularity, filling the data of the coarse granularity field, the medium granularity field and the fine granularity field of the data to be processed according to the data main key.
Based on the above steps, on the basis of the nested data structure shown in fig. 2, data of three granularity levels, namely coarse granularity, medium granularity and fine granularity, are correspondingly filled according to the data primary key (i.e. order number).
In step S140, the filling data is stored in a database according to the data primary key storage.
In one embodiment of the present disclosure, storing the filler data into the database according to the data primary key includes:
and inserting or updating the filling data into a data storage model according to the data primary key.
The data storage model which can be selected in the embodiment of the disclosure is an Elasticsearch storage model, and a mongoDB storage model can be selected to design a data nesting structure, the difference is that the mongoDB is a universal database, the Elasticsearch is a distributed, expandable and real-time search and data analysis engine supported by Lucene, the data storage, aggregation, full-text retrieval and the like can be realized, and the performance of the Elasticsearch is superior to that of the mongoDB. Based on the above, by performing nested storage on data according to different granularity levels and storing the data by using an elastic search storage model, query statistical analysis such as filtering, aggregation and the like can be flexibly performed in real time, and the effect of real-time monitoring is achieved.
The following describes in detail the upstream and downstream data processing of data nesting storage in the embodiment of the present disclosure with reference to a flowchart of specific processing steps of the data processing method shown in fig. 3:
as shown in fig. 3, in step S301, data is captured.
The step is mainly that data with various granularities are respectively consumed from respective message queues Kafka by distributed real-time computing tasks based on Storm, for example, the distributed real-time computing tasks comprise Kafka message queue 1, Kafka message queue 2 and Kafka message queue 3 … …, wherein the Storm is a distributed real-time streaming computing system, can be horizontally expanded, is automatically fault-tolerant, and can delay the millisecond level. Common and real-time analysis, online machine learning, etc.
As shown in fig. 3, in step S302, data preprocessing is performed.
The step is mainly that in the Storm real-time calculation task, three kinds of granularity data with the same coarse granularity ID are distributed to the same downstream node through a routing strategy to be put into storage. For example, order data (belonging to coarse-grained data) with the same order number (i.e. primary key ID), order and settlement subject data (belonging to medium-grained data) with the same order number, and order settlement detail data (belonging to fine-grained data) with the same order number are distributed to the same downstream node through a routing policy.
As shown in fig. 3, in step S303, data is stored.
The step is mainly to receive the data processed by the front end of the data in the previous step S302, and store the data in the nested data model of the Elasticsearch. Taking an order calculation model as an example, when data is put into a storage model of an Elasticsearch, the stored primary key ID is a coarse-grained ID (such as an order number); when the received data is coarse-grained data (namely order information), constructing a nested data structure, filling coarse-grained field data, and inserting or updating the data into an Elasticissearch storage model according to a primary key ID (namely the coarse-grained ID); when the received data is medium-granularity data (namely an order-settlement main body), constructing a nested data structure, filling a coarse-granularity ID field and the medium-granularity field data (the medium-granularity data is stored in a list form), and inserting or updating the medium-granularity field data into an Elasticisarch storage model according to a primary key ID (namely the coarse-granularity ID); when the received data is fine-grained data, a nested data structure is constructed, a coarse-grained ID field, a medium-grained ID field (the medium-grained ID is stored in a list form) and fine-grained field data (the fine-grained data is stored in a list form) are filled, and the fine-grained data are inserted or updated into an Elasticissearch storage model according to the primary key ID (the coarse-grained ID).
In addition, in the operation process of the Elasticissearch storage model, when no record exists in the inquiry of the ID of the primary key, the unfilled field in the constructed storage structure is automatically filled with a default value; when the inquiry record exists by pressing the primary key ID, only the received field is updated.
As shown in fig. 3, in step S304, data is queried.
The step is mainly from the elastic search storage model, and the fields with various granularities can be filtered, aggregated and the like. In this embodiment, the number of records of the query is an index of the coarse-grained statistic, and the sum of the coarse-grained fields is an aggregate index of the field data without deduplication. For example, the order quantity, the order amount and the settlement amount of each link of the order settlement can be summarized, and the performance and the data backlog condition of each link can be counted. Not only can be inquired according to order granularity (coarse granularity), but also can be inquired according to order-settlement main granularity (medium granularity) and order settlement detail (fine granularity).
Based on the steps, a nested data structure is designed based on a distributed database elastic search, and the whole design schemes such as warehousing and query based on the nested data structure can improve the data query performance and simplify the multi-data source warehousing process.
To sum up, according to the data nesting storage method provided by the embodiment of the present disclosure, on one hand, nested data structures are respectively constructed for data of different granularity levels, so that caching waiting is not needed when the data is put into a warehouse, and no matter which granularity of the data is obtained, the data can be put into the warehouse in real time, thereby improving data query performance and simplifying a multi-data warehousing process. On the other hand, since the statistics is performed using the coarse granularity having the highest granularity level as an index, deduplication is not required, and the indexes to be summed up by the coarse granularity statistics value can be directly summed up, and there is no problem of duplicate summing up in the summation result.
Fig. 4 is a schematic diagram of a data nesting storage device provided in another embodiment of the present disclosure, as shown in fig. 4, the device 400 includes: a granularity judging module 410, a nesting module 420, a filling module 430 and a storing module 440.
The granularity judging module 410 is configured to judge the granularity of the received data to be processed, so as to obtain granularity levels, where the data to be processed at least includes two granularity levels; the nesting module 420 is configured to construct a nesting data structure for the data to be processed according to the granularity level, wherein the nesting data structure contains a data primary key corresponding to the data to be processed; the filling module 430 is configured to fill the data to be processed according to the data primary key, so as to obtain filled data; the storage module 440 is configured to store the filler data in the database according to the data primary key memory.
The functions of each module in the apparatus are described in the above method embodiments, and are not described again here.
FIG. 5, below, illustrates an overall architecture diagram of the data processing system, as shown in FIG. 5, including a data fetch module 510, a data processing module 520, a data storage module 530, and a data query module 540. The data capture module 510, the data processing module 520, the data storage module 530, and the data query module 540 respectively correspond to steps S301 to S304 shown in fig. 3 in the above embodiment, and are not described herein again.
In summary, the data nesting storage device provided in the embodiment of the present disclosure, on one hand, the nesting data structures are respectively constructed for data of different granularity levels, so that no cache wait is needed when the data is put into a storage, and no matter which granularity of the obtained data can be put into the storage in real time, thereby improving the data query performance and simplifying the multi-data storage process. On the other hand, since the statistics is performed using the coarse granularity having the highest granularity level as an index, deduplication is not required, and the indexes to be summed up by the coarse granularity statistics value can be directly summed up, and there is no problem of duplicate summing up in the summation result.
In another aspect, the present disclosure also provides an electronic device, including a processor and a memory, where the memory stores operating instructions for the processor to control the following method:
judging the granularity of received data to be processed to obtain granularity levels, wherein the data to be processed at least comprises two granularity levels; constructing a nested data structure for the data to be processed according to the granularity level, wherein the nested data structure comprises a data primary key corresponding to the data to be processed; filling the data to be processed according to the data main key to obtain filling data; and storing the filling data into a database according to the data main key memory.
Referring now to FIG. 6, shown is a block diagram of a computer system 600 suitable for use in implementing the electronic device of an embodiment of the present application. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage portion 607 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present application are executed when the computer program is executed by the Central Processing Unit (CPU) 601.
It should be noted that the computer readable medium shown in the present application may be a computer readable signal medium or a computer readable medium or any combination of the two. A computer readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present application may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a transmitting unit, an obtaining unit, a determining unit, and a first processing unit. The names of these units do not in some cases constitute a limitation to the unit itself, and for example, the sending unit may also be described as a "unit sending a picture acquisition request to a connected server".
In another aspect, the present disclosure also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to include the method steps of:
judging the granularity of received data to be processed to obtain granularity levels, wherein the data to be processed at least comprises two granularity levels; constructing a nested data structure for the data to be processed according to the granularity level, wherein the nested data structure comprises a data primary key corresponding to the data to be processed; filling the data to be processed according to the data main key to obtain filling data; and storing the filling data into a database according to the data main key memory.
It should be clearly understood that this disclosure describes how to make and use particular examples, but the principles of this disclosure are not limited to any details of these examples. Rather, these principles can be applied to many other embodiments based on the teachings of the present disclosure.
Exemplary embodiments of the present disclosure are specifically illustrated and described above. It is to be understood that the present disclosure is not limited to the precise arrangements, instrumentalities, or instrumentalities described herein; on the contrary, the disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (10)
1. A data nesting storage method is characterized by comprising the following steps:
judging the granularity of received data to be processed to obtain granularity levels, wherein the data to be processed at least comprises two granularity levels;
constructing a nested data structure for the data to be processed according to the granularity level, wherein the nested data structure comprises a data primary key corresponding to the data to be processed;
filling the data to be processed according to the data main key to obtain filling data;
and storing the filling data into a database according to the data main key memory.
2. The data nesting storage method according to claim 1, before determining the granularity of the received data to be processed, further comprising:
distributing the data to be processed with different granularity levels according to a routing strategy.
3. The data nesting storage method according to claim 1, wherein the data to be processed comprises three granularity levels of coarse granularity, medium granularity and fine granularity, and the data primary key is marked by a coarse-granularity identification number.
4. The data nesting storage method according to claim 3, wherein the padding of the data to be processed according to the data master key to obtain padding data comprises:
and when the granularity level of the data to be processed is coarse granularity, filling the field of the coarse granularity field of the data to be processed according to the data primary key.
5. The data nesting storage method according to claim 3, wherein the padding of the data to be processed according to the data master key to obtain padding data comprises:
and when the granularity level of the data to be processed is the medium granularity, filling the data of the coarse granularity field and the medium granularity field of the data to be processed according to the data main key.
6. The data nesting storage method according to claim 3, wherein the padding of the data to be processed according to the data master key to obtain padding data comprises:
and when the granularity level of the data to be processed is fine granularity, filling the data of the coarse granularity field, the medium granularity field and the fine granularity field of the data to be processed according to the data main key.
7. The data nesting storage method according to any one of claims 4-6, wherein storing the filling data into a database according to the data primary key comprises:
and inserting or updating the filling data into a data storage model according to the data primary key, wherein the data storage model is an Elasticissearch storage model.
8. A data nesting storage device, comprising:
the granularity judging module is configured to judge the granularity of the received data to be processed to obtain granularity levels, wherein the data to be processed at least comprises two granularity levels;
the nesting module is configured to construct a nesting data structure for the data to be processed according to the granularity level, and the nesting data structure contains a data primary key corresponding to the data to be processed;
the filling module is configured to fill the data to be processed according to the data main key to obtain filling data;
and the storage module is configured to store the filling data into a database according to the data primary key storage.
9. An electronic device, comprising:
a processor;
memory storing instructions for the processor to control the method steps according to any one of claims 1-7.
10. A computer-readable medium having stored thereon computer-executable instructions, which when executed by a processor, perform the method steps of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810847055.4A CN110766501B (en) | 2018-07-27 | 2018-07-27 | Data nested storage method, device, electronic equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810847055.4A CN110766501B (en) | 2018-07-27 | 2018-07-27 | Data nested storage method, device, electronic equipment and computer readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110766501A true CN110766501A (en) | 2020-02-07 |
| CN110766501B CN110766501B (en) | 2024-08-20 |
Family
ID=69328324
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810847055.4A Active CN110766501B (en) | 2018-07-27 | 2018-07-27 | Data nested storage method, device, electronic equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110766501B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112364016A (en) * | 2020-10-27 | 2021-02-12 | 中国地震局地质研究所 | Construction method of time nested cache model of pilot frequency data object |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101009726A (en) * | 2006-12-28 | 2007-08-01 | 上海柯斯软件有限公司 | The method of the cooperative use of the data structure of the phone directory in the large-capacity SIM card |
| CN101183986A (en) * | 2007-11-26 | 2008-05-21 | 河北全通通信有限公司 | Method for economized storage, construction ''green'' database |
| CN102291269A (en) * | 2011-09-29 | 2011-12-21 | 迈普通信技术股份有限公司 | Data merging processing method |
| CN102541759A (en) * | 2011-12-28 | 2012-07-04 | 畅捷通信息技术股份有限公司 | Cache control device and cache control method |
| CN103092869A (en) * | 2011-11-01 | 2013-05-08 | 镇江华扬信息科技有限公司 | Mapping transformation method of Enable extensive makeup language (XML) Database |
| CN103745130A (en) * | 2014-01-27 | 2014-04-23 | 东北大学 | Predicting method for multi-attribute event under environment of wireless sensor |
| JP2016151793A (en) * | 2015-02-16 | 2016-08-22 | 日本電信電話株式会社 | Database and database search method |
| CN106557531A (en) * | 2015-09-30 | 2017-04-05 | 伊姆西公司 | Labyrinth object is converted into planarizing data |
| CN107729413A (en) * | 2017-09-25 | 2018-02-23 | 安徽畅通行交通信息服务有限公司 | Regional traffic intelligent management system based on big data |
-
2018
- 2018-07-27 CN CN201810847055.4A patent/CN110766501B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101009726A (en) * | 2006-12-28 | 2007-08-01 | 上海柯斯软件有限公司 | The method of the cooperative use of the data structure of the phone directory in the large-capacity SIM card |
| CN101183986A (en) * | 2007-11-26 | 2008-05-21 | 河北全通通信有限公司 | Method for economized storage, construction ''green'' database |
| CN102291269A (en) * | 2011-09-29 | 2011-12-21 | 迈普通信技术股份有限公司 | Data merging processing method |
| CN103092869A (en) * | 2011-11-01 | 2013-05-08 | 镇江华扬信息科技有限公司 | Mapping transformation method of Enable extensive makeup language (XML) Database |
| CN102541759A (en) * | 2011-12-28 | 2012-07-04 | 畅捷通信息技术股份有限公司 | Cache control device and cache control method |
| CN103745130A (en) * | 2014-01-27 | 2014-04-23 | 东北大学 | Predicting method for multi-attribute event under environment of wireless sensor |
| JP2016151793A (en) * | 2015-02-16 | 2016-08-22 | 日本電信電話株式会社 | Database and database search method |
| CN106557531A (en) * | 2015-09-30 | 2017-04-05 | 伊姆西公司 | Labyrinth object is converted into planarizing data |
| CN107729413A (en) * | 2017-09-25 | 2018-02-23 | 安徽畅通行交通信息服务有限公司 | Regional traffic intelligent management system based on big data |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112364016A (en) * | 2020-10-27 | 2021-02-12 | 中国地震局地质研究所 | Construction method of time nested cache model of pilot frequency data object |
| CN112364016B (en) * | 2020-10-27 | 2021-08-31 | 中国地震局地质研究所 | Construction method of time nested cache model of pilot frequency data object |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110766501B (en) | 2024-08-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109299348B (en) | Data query method and device, electronic equipment and storage medium | |
| CN104731690B (en) | Adaptive metrology collection, storage and warning threshold | |
| EP3955132A1 (en) | Graph computing method and device | |
| CN105530272A (en) | Method and device for application data synchronization | |
| CN109597810B (en) | Task segmentation method, device, medium and electronic equipment | |
| CN110245684B (en) | Data processing method, electronic device, and medium | |
| CN113011672B (en) | Logistics aging prediction method and device, electronic equipment and storage medium | |
| CN104580428A (en) | Data routing method, data management device and distributed storage system | |
| CN110766501B (en) | Data nested storage method, device, electronic equipment and computer readable medium | |
| CN109344152A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN111897822A (en) | Account state information processing method and device, electronic equipment and storage medium | |
| US10666557B2 (en) | Method to manage data flow for edge-core distributed analytics systems | |
| JP4718939B2 (en) | Object categorization | |
| US20230188619A1 (en) | Methods and systems for event management | |
| CN109246035A (en) | A kind of method and device of data transfer management | |
| CN106202503B (en) | Data processing method and device | |
| CN114820132A (en) | Order distribution method and device, electronic equipment and storage medium | |
| US11128547B2 (en) | Value optimization with intelligent service enablements | |
| US11392611B2 (en) | Blockchain based IOT data management | |
| CN113485890A (en) | Flight inquiry system service monitoring method, device, equipment and storage medium | |
| CN113568803A (en) | Method, electronic device and computer program product for monitoring a storage system | |
| CN111210616A (en) | Vehicle flow prediction method, system, terminal and medium based on multi-source data | |
| CN109727144A (en) | Insure Incidence calculus method, apparatus, equipment and computer readable storage medium | |
| US11341137B1 (en) | Processing search requests | |
| US12105705B2 (en) | Database query processing with database clients |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |