CN113947498A - Data storage and retrieval method for converged terminal data center - Google Patents
Data storage and retrieval method for converged terminal data center Download PDFInfo
- Publication number
- CN113947498A CN113947498A CN202111080328.5A CN202111080328A CN113947498A CN 113947498 A CN113947498 A CN 113947498A CN 202111080328 A CN202111080328 A CN 202111080328A CN 113947498 A CN113947498 A CN 113947498A
- Authority
- CN
- China
- Prior art keywords
- data
- app
- storage
- service
- management
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 19
- 238000013500 data storage Methods 0.000 title claims abstract description 17
- 238000007726 management method Methods 0.000 claims abstract description 31
- 238000013461 design Methods 0.000 claims abstract description 10
- 238000005516 engineering process Methods 0.000 claims abstract description 6
- 238000004891 communication Methods 0.000 claims description 11
- 238000004458 analytical method Methods 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000011664 signaling Effects 0.000 claims description 3
- 238000003032 molecular docking Methods 0.000 claims 1
- 230000004927 fusion Effects 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/06—Addressing a physical block of locations, e.g. base addressing, module addressing, memory dedication
- G06F12/0646—Configuration or reconfiguration
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- General Physics & Mathematics (AREA)
- Public Health (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Human Resources & Organizations (AREA)
- General Business, Economics & Management (AREA)
- General Health & Medical Sciences (AREA)
- Water Supply & Treatment (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage and retrieval method of a fusion terminal data center, which comprises the following steps: integrating the design of the whole framework of the terminal software and designing an App framework according to business requirements; calculating the hash value of each data according to the relevance of the data under different services; storing the data in corresponding address blocks in a separated storage mode and stamping time stamps according to different hash values; calculating a hash value according to the correlation characteristics of the fields, roughly positioning a storage address block of the data, then reducing the range of a storage sector in the address block by utilizing a timestamp, and accurately retrieving in a target sector; the data center App encapsulates a storage interface based on sqlite3, forms a storage service and a management function, provides a service for external messages, and achieves unified management of data and data sharing among different Apps. The invention deploys each App by using a containerization technology and establishes a data grading protection mechanism, and different service data are graded and partitioned for storage and retrieval, thereby improving the storage efficiency and the retrieval speed.
Description
Technical Field
The invention relates to the technical field of distribution network automation, in particular to a data storage and retrieval method of a converged terminal data center.
Background
With the continuous development of the power industry and the intelligent power grid technology, a large amount of intelligent power distribution and utilization data are formed in the power grid industry, power information data among different services are not obviously divided, and certain defects still exist in the aspects of safety and convenience of data management. Based on the design structure and the management mode of the traditional data center, the requirements of a fusion type terminal system under an intelligent power distribution and utilization network cannot be met, and a scientific, reasonable and effective design scheme is lacked in the design scheme of the data center of the fusion type terminal equipment aiming at marketing and power distribution services in the existing power information acquisition system.
The data center of the current intelligent terminal mainly uses an index mode in the aspects of storage and retrieval, although the retrieval speed is improved to a certain extent, more time delay is brought in the process of establishing index-induced storage, more storage space is occupied, the phenomenon of App card death occasionally occurs in the retrieval process, the balance point of whether to establish the index is difficult to find due to the diversification of data characteristics, the data storage and retrieval in the traditional mode cannot meet the data service requirement of a fusion type terminal, and an efficient and quick storage and retrieval scheme is still lacked in the aspects of storage and retrieval of large data volume.
Disclosure of Invention
In order to solve the technical problems, the invention provides a data storage and retrieval method of a fusion terminal data center, so as to achieve the purposes of improving the storage efficiency and the retrieval speed and ensuring the data security.
In order to achieve the purpose, the technical scheme of the invention is as follows:
a data storage and retrieval method of a converged terminal data center comprises the following steps:
step 1: integrating the design of the terminal software overall framework, designing App frameworks according to business requirements, deploying each App by utilizing a containerization technology, and respectively accessing corresponding hardware resources;
step 2: calculating the hash value of each data by using a hash bucket-dividing algorithm based on data correlation by using a correlation field with a unique identifier according to the correlation of the data under different services;
and step 3: storing the data in corresponding address blocks in a separated storage mode and stamping time stamps according to different hash values in the associated data records;
and 4, step 4: when data is retrieved, a hash value is calculated according to the associated characteristics of the fields, a storage address block of the data is roughly located, then the range of a storage sector is narrowed in the storage address block by utilizing a timestamp, a target sector is found, and finally accurate retrieval is carried out on the target sector;
and 5: the data center App encapsulates a storage interface based on sqlite3, forms a storage service and a management function, provides a service for external messages, and achieves unified management of data and data sharing among different Apps.
In the scheme, the apps in the step 1 are divided into a basic App and a service App, the basic App is responsible for managing shared data in different services, and the service App divides different services according to the trend of data streams and comprises a collection service App which is responsible for collecting downlink equipment data of a monitoring terminal, a master station service App which is responsible for butting uplink master station services and an analysis service App which is responsible for analyzing the collected data through edge calculation.
In a further technical scheme, the data center App belongs to a basic App, and the basic App further comprises a local communication management App, a wireless dialing management App, an expansion module management App, a serial port management App, a Bluetooth management App, a delivery and acquisition metering App, a remote signaling pulse sampling App, a safety agent App and a safety management App.
In the above solution, the uniquely identified association field in step 2 includes an object attribute descriptor, a communication address, and a device class.
In the foregoing solution, the hash value in step 3 is calculated in the following manner:
HashValue＝f(key)
wherein, f is hash function, which is a mapping relation, dividing address blocks according to different services and corresponding to different hash values, and mapping the address blocks to different service data areas and shared data areas for storing corresponding data.
In the above scheme, in step 5, the MsgServer module of the data center App is responsible for receiving message requests of other apps, generating a task after parsing and sending the task to the DataHandle module for processing, and finally sending the task to other apps in the form of an event message, the MsgServer module supports UDP/TCP/IPC/MQTT communication service in design, and the DataHandle module is responsible for calling an API interface according to the task to access a database and returning a request result to the MsgServer module.
Through the technical scheme, the data storage and retrieval method of the fusion terminal data center provided by the invention has the following beneficial effects:
according to the method, the software framework of the fusion terminal App is designed, the App is classified according to different services, and a containerization technology is used for deployment, so that the independence among the App is improved, and the data safety is guaranteed; in the aspect of data storage and retrieval, a data grading protection mechanism is established, different service data are stored in a grading mode, a Hash barrel storage optimization algorithm based on an associated field is designed, the storage efficiency and the retrieval speed of data center apps are improved through service blocking and timestamp retrieval modes, scientific classification from services to data among the apps is achieved, and the data safety of a fusion terminal system is guaranteed.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
Fig. 1 is a flowchart of a data storage and retrieval method of the converged terminal data center according to the present invention.
FIG. 2 is a software framework diagram of the present invention.
Fig. 3 is a schematic diagram of the App architecture of the present invention.
FIG. 4 is a flow chart of the present invention for accurate retrieval based on time stamps.
Fig. 5 is a flow chart of the MsgServer module of the data center App in the present invention.
Fig. 6 is a flow chart of the data handle module of the data center App according to the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
The invention provides a data storage and retrieval method of a converged terminal data center, which comprises the following steps as shown in figure 1:
step 1: the design of the integrated terminal software framework is shown in fig. 2, wherein the software framework includes two layers, namely an operating system and application software, and the operating system layer includes a kernel, a driver, a system component and the like. The application software comprises a basic App and a business App, and the system component, the basic App and the business App are interacted through a message bus.
Designing an App architecture according to business requirements, wherein each App is deployed by using a containerization technology and accesses hardware resources corresponding to each App as shown in fig. 3; the basic App is responsible for management of shared data in different services, and comprises a data center App, a local communication management App, a wireless dialing management App, an expansion module management App, a serial port management App, a Bluetooth management App, a collection metering App, a remote signaling pulse sampling App, a safety agent App and a safety management App.
The business App divides different businesses according to the trend of the data flow, and comprises a collection business class App which is responsible for collecting and monitoring downlink equipment data of the terminal, a master station business class App which is responsible for butting uplink master station business, and an analysis business class App which is responsible for analyzing the collected data through edge calculation.
The collected service type App comprises a power distribution type service App and a marketing type service App, wherein the power distribution type service App comprises a low-voltage intelligent switch collected App, a reactive compensation device collected App, a power distribution station room environment collected App and the like; the marketing service App comprises an electric meter acquisition task management App, a loop state monitoring App and the like.
The master station service class App comprises an IEC104 App, an IEC101 App, an MQTT-IoT App, a low-voltage meter reading App, an Internet of vehicles cloud communication App and the like.
The analysis service class App comprises a power quality analysis App, a line loss lean analysis App, a topology identification App, a load identification App, a comprehensive fault study and judgment App, a distributed energy management App, an electric vehicle ordered charging App, a district openable capacity prediction App and the like.
Step 2: calculating the hash value of each data by using a hash bucket-dividing algorithm based on data correlation by using a correlation field with a unique identifier according to the correlation of the data under different services; wherein the uniquely identified associated fields may employ object attribute descriptors, communication addresses, device classes, and the like.
The hash value is calculated in the following way:
HashValue＝f(key)
wherein, f is hash function, which is a mapping relation, dividing address blocks according to different services and corresponding to different hash values, and mapping the address blocks to different service data areas and shared data areas for storing corresponding data.
And step 3: storing the data in corresponding address blocks in a separated storage mode and stamping time stamps according to different hash values in the associated data records; the address blocks are divided according to different services and correspond to different hash values, and the hash values are mapped to different service data areas and shared data areas for storing corresponding data.
And 4, step 4: when data is retrieved, a hash value is calculated according to the associated characteristics of the fields, a storage address block of the data is roughly located, then the range of a storage sector is narrowed in the storage address block by utilizing a timestamp, a target sector is found, and finally accurate retrieval is carried out on the target sector; because the storage timestamps in the data blocks are not necessarily arranged in a strict address order, but a certain relationship exists between the storage timestamps and the data blocks, the data sectors are accurately positioned according to the relationship, as shown in fig. 4, the data center sequentially compares the data timestamps of the head addresses of two adjacent sectors in the range, and the accurate positioning of the sectors is realized according to a certain rule, and the specific method comprises the following steps:
first Mid _ Addr location data chunk timestamp TM is read, then Sta _ Addr location data chunk timestamp T0 is read, and the size of TM and T0 are compared.
If T0 is not more than TM, the sizes of Tr and T0 are continuously compared, if Tr is not more than T0, the second half sector Sear _ fan (Mid _ Addr) of the address block is searched, if Tr is more than T0, the sizes of Tr and TM are continuously compared, if Tr is not more than TM, the first half sector Sear _ fan (Sta _ Addr) of the address block is searched, otherwise, the second half sector Sear _ fan (Mid _ Addr) of the address block is searched.
If T0 > TM, continuing to compare Tr and T0, if Tr ≧ T0, searching the first half sector Sear _ fan (Sta _ Addr) of the address block; if Tr < T0, the sizes of Tr and TM are continuously compared, if Tr ≦ TM, the first half sector Sear _ fan (Sta _ Addr) of the address block is searched, otherwise, the second half sector Sear _ fan (Mid _ Addr) of the address block is searched.
Wherein Mid _ Addr refers to the address of the middle sector of an address block, Sta _ Addr refers to the address of the first sector of the address block, and Tr refers to the historical time stamp of the user-specified search.
And 5: the data center App encapsulates a storage interface based on sqlite3, forms a storage service and a management function, provides a service for external messages, and achieves unified management of data and data sharing among different Apps.
The MsgServer module of the data center App is responsible for receiving message requests of other apps, generating tasks after parsing and sending the tasks to the DataHandle module for processing, as shown in fig. 5, the MsgServer module receives message threads, checks message links, receives communication messages after checking to be normal, then analyzes the messages, checks message formats, sets a message processing callback function after checking to be normal, finally sends the message tasks to the DataHandle module, and finally sends the processing results of the DataHandle module to other apps in the form of event messages, and the MsgServer module supports UDP/TCP/IPC/MQTT communication services in design and currently uses MQTT.
As shown in fig. 6, the DataHandle module is responsible for calling an API interface to access the database according to the task, and returning the request result to the MsgServer module. Firstly, the DataHandle module carries out initialization operation, then receives a data request task and distributes threads for the data request task, and a processing function is called according to the validity of the ID. And finally, calling back the reestablishment message according to the response to complete the data request task.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (6)
1. A data storage and retrieval method of a converged terminal data center is characterized by comprising the following steps:
step 1: integrating the design of the terminal software overall framework, designing App frameworks according to business requirements, deploying each App by utilizing a containerization technology, and respectively accessing corresponding hardware resources;
step 2: calculating the hash value of each data by using a hash bucket-dividing algorithm based on data correlation by using a correlation field with a unique identifier according to the correlation of the data under different services;
and step 3: storing the data in corresponding address blocks in a separated storage mode and stamping time stamps according to different hash values in the associated data records;
and 4, step 4: when data is retrieved, a hash value is calculated according to the associated characteristics of the fields, a storage address block of the data is roughly located, then the range of a storage sector is narrowed in the storage address block by utilizing a timestamp, a target sector is found, and finally accurate retrieval is carried out on the target sector;
and 5: the data center App encapsulates a storage interface based on sqlite3, forms a storage service and a management function, provides a service for external messages, and achieves unified management of data and data sharing among different Apps.
2. The data storage and retrieval method of the converged terminal data center according to claim 1, wherein the App in the step 1 is divided into a basic App and a service App, the basic App is responsible for management of shared data in different services, and the service App divides different services according to the trend of data streams, and comprises an acquisition service class App which is responsible for acquisition and monitoring of downlink device data of a terminal, a master station service class App which is responsible for docking of uplink master station services, and an analysis service class App which is responsible for analysis of acquired data through edge calculation.
3. The data storage and retrieval method of the converged terminal data center according to claim 2, wherein the data center App belongs to a basic App, and the basic App further comprises a local communication management App, a wireless dial-up management App, an expansion module management App, a serial port management App, a Bluetooth management App, an acquisition metering App, a remote signaling pulse sampling App, a safety agent App and a safety management App.
4. The data storage and retrieval method of the converged terminal data center according to claim 1, wherein the uniquely identified association field in the step 2 comprises an object attribute descriptor, a communication address and a device class.
5. The data storage and retrieval method for the converged terminal data center according to claim 1, wherein the hash value in the step 3 is calculated in a manner that:
HashValue＝f(key)
wherein, f is hash function, which is a mapping relation, dividing address blocks according to different services and corresponding to different hash values, and mapping the address blocks to different service data areas and shared data areas for storing corresponding data.
6. The data storage and retrieval method for the converged terminal data center as claimed in claim 1, wherein in the step 5, the MsgServer module of the data center App is responsible for receiving message requests of other apps, generating tasks after parsing and sending the tasks to the DataHandle module for processing, and finally sending the tasks to other apps in the form of event messages, the MsgServer module supports UDP/TCP/IPC/MQTT communication service in design, and the DataHandle module is responsible for calling an API interface to access a database according to the tasks and sending the request results back to the MsgServer module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111080328.5A CN113947498A (en) | 2021-09-15 | 2021-09-15 | Data storage and retrieval method for converged terminal data center |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111080328.5A CN113947498A (en) | 2021-09-15 | 2021-09-15 | Data storage and retrieval method for converged terminal data center |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113947498A true CN113947498A (en) | 2022-01-18 |
Family
ID=79328566
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111080328.5A Pending CN113947498A (en) | 2021-09-15 | 2021-09-15 | Data storage and retrieval method for converged terminal data center |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113947498A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114520754A (en) * | 2022-04-21 | 2022-05-20 | 石家庄科林电气股份有限公司 | Software architecture method of power distribution network edge gateway and gateway terminal |
| CN117785073A (en) * | 2024-02-28 | 2024-03-29 | 深圳市智岩科技有限公司 | Internet of things equipment, time sequence data access method thereof, corresponding device and medium |
-
2021
- 2021-09-15 CN CN202111080328.5A patent/CN113947498A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114520754A (en) * | 2022-04-21 | 2022-05-20 | 石家庄科林电气股份有限公司 | Software architecture method of power distribution network edge gateway and gateway terminal |
| CN114520754B (en) * | 2022-04-21 | 2022-07-19 | 石家庄科林电气股份有限公司 | Software architecture method of power distribution network edge gateway and gateway terminal |
| CN117785073A (en) * | 2024-02-28 | 2024-03-29 | 深圳市智岩科技有限公司 | Internet of things equipment, time sequence data access method thereof, corresponding device and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113947498A (en) | Data storage and retrieval method for converged terminal data center | |
| CN103325074B (en) | Power system real-time data processing method | |
| CN112235250B (en) | Method and system for acquiring data of distribution room in minute-level same frequency | |
| TW201800967A (en) | Method and device for processing distributed streaming data | |
| CN102880854B (en) | Distributed processing and Hash mapping-based outdoor massive object identification method and system | |
| CN104778188A (en) | Distributed device log collection method | |
| CN114281790B (en) | Multi-type load resource aggregator access system and method | |
| CN109739919A (en) | A kind of front end processor and acquisition system for electric system | |
| CN112235142B (en) | Power utilization information acquisition system capable of realizing key business disaster tolerance and operation method thereof | |
| CN109271113B (en) | Data management system and method based on cloud storage | |
| CN111368166A (en) | Resource search method, resource search apparatus, and computer-readable storage medium | |
| CN114374701B (en) | Transparent sharing device for sample model of multistage linkage artificial intelligent platform | |
| CN117194156A (en) | Unified monitoring operation and maintenance management method and system for multi-cloud platform | |
| CN116055448A (en) | Identification data management platform for electric power operation | |
| CN103530335A (en) | In-stockroom operation method and device of electric power measurement acquisition system | |
| CN110377757A (en) | A kind of real time knowledge map construction system | |
| CN110636107A (en) | Power grid data interaction system based on 5g communication and interaction method thereof | |
| Wu et al. | Storage and retrieval of massive heterogeneous iot data based on hybrid storage | |
| CN110708209B (en) | Virtual machine flow acquisition method and device, electronic equipment and storage medium | |
| CN115658742A (en) | Identification analysis method and system for field-level active identification carrier | |
| CN105677853A (en) | Data storage method and device based on big data technology framework | |
| CN115114316A (en) | Processing method, device, cluster and storage medium for high-concurrency data | |
| CN110515955A (en) | Storage, querying method, system, electronic equipment and the storage medium of data | |
| CN111835791A (en) | BGP security event rapid detection system | |
| Wen et al. | Revolutionizing Network Performance: The Active and Passive Service Path Performance Monitoring Analysis Method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |