CN109508317A - A kind of Large Volume Data and service management system - Google Patents
A kind of Large Volume Data and service management system Download PDFInfo
- Publication number
- CN109508317A CN109508317A CN201811283383.2A CN201811283383A CN109508317A CN 109508317 A CN109508317 A CN 109508317A CN 201811283383 A CN201811283383 A CN 201811283383A CN 109508317 A CN109508317 A CN 109508317A
- Authority
- CN
- China
- Prior art keywords
- file
- data
- service
- data source
- management
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to data management service technical field more particularly to a kind of Large Volume Datas and service management system.Large Volume Data and service managing apparatus of the invention improves the efficiency of file data transfer storage, improve the resource consumption of the data manipulations such as inquiry, the interstitial content in data file can be significantly reduced, the efficiency of lifting system overall operation in Large Volume Data system, reduce the burden of system file processing, file association structure based on dependence simplifies the difficulty of positioning and the management of file, simplify the positioning operation for relying on node in the positioning of associated with and cluster process, the thread consumption for reducing file data read-write, improves read and write rate.
Description
Technical field
The invention belongs to data management service technical field more particularly to a kind of Large Volume Datas and service management system.
Background technique
With information transmission, the development of detection technique, during daily learning life and the manufacturing, using synthesis
Data processing technique is more and more extensive, by the way that a variety of data acquisition devices and transmitting device is arranged, realize it is wider or
More fully data information is collected and treatment effeciency, and in contrast, the total amount of data volume is more, the accuracy of data and complete
Face property is higher, and but then, a large amount of data source represents huge data transmission and processing pressure, is especially producing
In control, real-time acquisition applications environment, with general daily monitoring, continuous transmission is different, containing a large amount of in these application environments
The small documents data of dispersion, such as sensor monitoring data, control signal data etc., the single capacity of these data itself is small, but
Overall quantity is huge, and single space/byte length of actual file transmission and caching, storage system is generally higher than these
Data, cause in processing, storage and process of caching cannot abundant device efficiency, while the band in a large amount of such Data Concurrents
Carry out high system load burden.
Summary of the invention
The purpose of the invention is, provides a kind of Large Volume Data and service management system, can be by dispersion
Small documents data carry out integrated treatment, improve it and transmit storage efficiency, reduce file system load.
To achieve the above object, the invention adopts the following technical scheme that.
A kind of Large Volume Data and service management system, including a main control module and multiple service modules service mould
Control and management of the block by main control module；
Main control module includes monitoring transmission interface, write control assembly and merging control assembly, monitors transmission interface and receives
It includes data management element that instruction message from control terminal and service module, which writes control assembly, and data management element record is former
The mapping relations of beginning data and service module；
Merging control assembly includes merging management element, dependency graph computing element, cluster element, dependency graph integrating element control
Each memory service in service module processed carries out the resolving of dependency graph, converges and initialize, cluster element according to dependency graph into
Row hierarchical clustering and cluster scheduling, to determine each cluster merges in which memory service；Merge management element according to cluster tune
Same clustered file is transferred in the same memory service by degree result, as merging serviced component to data file included in cluster
It merges, while generating retrieval file；
Service module obtains and the write-in requirement of processing data source, and feeds back and accordingly reply message；It is connect including monitoring to send
Mouthful, for file carry out write buffer operation write serviced component, the merging serviced component for merging operation to file；
Writing serviced component includes data write-in interface, memory management element, initial data management element；Initial data management
The mapping relations of data source and object, data source and file are stored in element；Memory management element divides document memory
With management, specifically, memory management element handles file in a manner of doubly linked list, wherein data source indicator and text
Part block chained list is stored in hash table, and the number of each data source indicator is inserted in chained list by the hash method of salary distribution；It is two-way to utilize
Chained list is managed and safeguards to data source indicator and blocks of files；It includes merging element, process pipe that service, which merges serviced component,
Manage element；
Merge serviced component and the blocks of files cached into memory management element is constantly merged into the big of compressed file format
File, until the sizableness of the size of big file and file system default tile, then be written in file system；Management of process element
Including a thread pool, number of threads is the number of threads of chip, and the generation for merging, searching for, Parallel Implementation file
Ongoing operation；Wherein, poly- using the layering based on dependence when file total amount reaches certain given threshold in file system
Class clusters data source.
It further include, in service module, interface being written to above-mentioned Large Volume Data and advanced optimizing for service managing apparatus
In write thread obtain data and with file protocol specify form be sent to caching, if it is data source first read-write clothes
Business, then memory management element is inserted into data source identifier in hash table, while initializing to doubly linked list, with
In the Memory Allocation of the data source, memory block corresponding with file size is distributed for data, after the completion of data copy, memory block is released
It puts into doubly linked list, and safeguards the mapping relations of data source and file, block in initial data management element；More service modules
Load is more than that certain limits threshold value, then data source returns to overload signal, has data source to retransmit read-write requests.
Further include to above-mentioned Large Volume Data and advanced optimizing for service managing apparatus, in main control module, when a certain number
When reaching connection with Large Volume Data and service managing apparatus for the first time according to source, main control module writes control assembly according to current big
The internal storage state of capacity data and service managing apparatus returns to main storage address to data source, and establishes data source and buffer service
Mapping relations be stored in hash table, when the monitoring transmission interface of service module receives write instruction, write control assembly return
To the corresponding replying instruction of data source to start corresponding buffer service.
The beneficial effect is that:
Large Volume Data and service managing apparatus of the invention improves the efficiency of file data transfer storage, improves inquiry
The resource consumption of equal data manipulations, can significantly reduce the interstitial content in data file, be promoted in Large Volume Data system
The efficiency of system overall operation, reduces the burden of system file processing, and the file association structure based on dependence simplifies text
The difficulty of positioning and the management of part simplifies the positioning operation for relying on node in the positioning of associated with and cluster process,
The thread consumption for reducing file data read-write, improves read and write rate.
Detailed description of the invention
Fig. 1 is the structural schematic diagram of Large Volume Data and service managing apparatus；
Fig. 2 is dependency graph schematic illustration；
Fig. 3 is file method of salary distribution schematic diagram in memory management element.
Specific embodiment
It elaborates below in conjunction with specific embodiment to the invention.
As shown in Figure 1, Large Volume Data and service managing apparatus, including a main control module and multiple service modules,
Control and management of the service module by main control module；
Main control module includes monitoring transmission interface, write control assembly and merging control assembly, monitors transmission interface and receives
Instruction message from control terminal and service module, writing control assembly includes data management element, data management element record
The mapping relations of initial data and service module；
Merging control assembly includes merging management element, dependency graph computing element, cluster element, dependency graph integrating element control
Each memory service in service module processed carries out the resolving of dependency graph, converges and initialize, cluster element according to dependency graph into
Row hierarchical clustering and cluster scheduling, to determine each cluster merges in which memory service；Merge management element according to cluster tune
Same clustered file is transferred in the same memory service by degree result, as merging serviced component to data file included in cluster
It merges, while generating retrieval file；
Relevance when accessing when retrieving to file to data source is the degree of association of data source, it is general and
Speech then claims data source 1 to depend on data source 2 when the access for a certain data source 1 will necessarily cause the access to data source 2；
Dependency graph is the dependence indicated between different data sources；Dependency graph is an oriented weight map, basic system such as Fig. 2 institute
Show, wherein include host node, such as the node 1,2,3 and partial node in figure, such as 4i, 5i, 6i, 7i, 8i in figure, accordingly
With it is unidirectional while and it is two-way while, unidirectional side has partial node to be directed toward host node, and two-way side is located between two host nodes, and each side has
There is the weight for representing data similarity；Data similarityWherein OiIt is data source i
The set of middle data；
Service module obtains and the write-in requirement of processing data source, and feeds back and accordingly reply message；It is connect including monitoring to send
Mouthful, for file carry out write buffer operation write serviced component, the merging serviced component for merging operation to file；
Writing serviced component includes data write-in interface, memory management element, initial data management element；Initial data management
The mapping relations of data source and object, data source and file are stored in element；As shown in figure 3, memory management element is to file
Memory is allocated management, specifically, memory management element handles file in a manner of doubly linked list, wherein data
Source indicator and blocks of files chained list are stored in hash table, and the number of each data source indicator is inserted in chained list by the hash method of salary distribution；
Data source indicator and blocks of files are managed and be safeguarded using doubly linked list；It includes merging member that service, which merges serviced component,
Part, management of process element；
Merge serviced component and the blocks of files cached into memory management element is constantly merged into the big of compressed file format
File, until the sizableness of the size of big file and file system default tile, then be written in file system；Management of process element
Including a thread pool, number of threads is the number of threads of chip, and the generation for merging, searching for, Parallel Implementation file
Ongoing operation；Wherein, poly- using the layering based on dependence when file total amount reaches certain given threshold in file system
Class clusters data source
Wherein, in main control module, reach for the first time with Large Volume Data and service managing apparatus when a certain data source and connect
When, main control module writes control assembly according to the internal storage state of current Large Volume Data and service managing apparatus return primary storage
Location is to data source, and the mapping relations for establishing data source and buffer service are stored in hash table, when the monitoring of service module is sent
Interface writes control assembly and returns to the corresponding replying instruction of data source to start corresponding buffer service to write instruction；
In service module, the thread of writing being written in interface obtains data and is sent in the form that file protocol is specified slow
It deposits, the first read-write if it is data source services, then memory management element is inserted into data source identifier in hash table, while right
Doubly linked list initializes, corresponding with file size interior for data distribution with the Memory Allocation for the data source
Counterfoil, after the completion of data copy, memory block is discharged into doubly linked list, and in initial data management element maintenance data source with
File.The mapping relations of block；More service module loads are more than that certain limits threshold value, then data source returns to overload signal, there is number
Read-write requests are retransmitted according to source；
Finally it should be noted that above embodiments are only to illustrate the technical solution of the invention, rather than to this hair
It is bright create protection scope limitation, although being explained in detail referring to preferred embodiment to the invention, this field it is general
Lead to it will be appreciated by the skilled person that can be modified or replaced equivalently to the technical solution of the invention, without departing from this
The spirit and scope of innovation and creation technical solution.
Claims (3)
1. a kind of Large Volume Data and service management system, which is characterized in that including a main control module and multiple service moulds
Block, control and management of the service module by main control module；
Main control module includes monitoring transmission interface, write control assembly and merging control assembly, monitors transmission interface receiving and comes from
The instruction message of control terminal and service module, writing control assembly includes data management element, and data management element records original
The mapping relations of data and service module；
Merging control assembly includes merging management element, dependency graph computing element, cluster element, dependency graph integrating element control clothes
Each memory service in module of being engaged in, carries out the resolving of dependency graph, converges and initialize, and cluster element carries out layer according to dependency graph
Secondary cluster and cluster scheduling, to determine each cluster merges in which memory service；Merge management element and knot is dispatched according to cluster
Same clustered file is transferred in the same memory service by fruit, is carried out as merging serviced component to data file included in cluster
Merge, while generating retrieval file；
Service module obtains and the write-in requirement of processing data source, and feeds back and accordingly reply message；Including monitoring transmission interface, using
In to file carry out write buffer operation write serviced component, the merging serviced component for merging operation to file；
Writing serviced component includes data write-in interface, memory management element, initial data management element；Initial data manages element
In be stored with the mapping relations of data source and object, data source and file；Memory management element is allocated pipe to document memory
Reason, specifically, memory management element handles file in a manner of doubly linked list, wherein data source indicator and blocks of files
Chained list is stored in hash table, and the number of each data source indicator is inserted in chained list by the hash method of salary distribution；To utilize doubly linked list
Data source indicator and blocks of files are managed and are safeguarded；It includes merging element, management of process member that service, which merges serviced component,
Part；
Merge the big file that the blocks of files cached into memory management element is constantly merged into compressed file format by serviced component,
Until the size of big file and the sizableness of file system default tile, then be written in file system；Management of process element includes
One thread pool, number of threads is the number of threads of chip, and the generation for merging, searching for, Parallel Implementation file are held
Continuous operation；Wherein, when file total amount reaches certain given threshold in file system, the hierarchical cluster pair based on dependence is used
Data source is clustered.
2. a kind of Large Volume Data and service management system according to claim 1, which is characterized in that in service module, write
Thread of writing in incoming interface obtains data and is sent to caching in the form that file protocol is specified, if it is the first reading of data source
Write service, then memory management element is inserted into data source identifier in hash table, while initializing to doubly linked list,
With the Memory Allocation for the data source, memory block corresponding with file size is distributed for data, after the completion of data copy, memory
Block is discharged into doubly linked list, and the mapping relations of data source and file, block are safeguarded in initial data management element；More services
Module load is more than that certain limits threshold value, then data source returns to overload signal, has data source to retransmit read-write requests.
3. a kind of Large Volume Data and service management system according to claim 1, which is characterized in that in main control module, when
When a certain data source reaches connection with distributed document device for the first time, main control module writes control assembly according to current distributed
The internal storage state of file device returns to main storage address to data source, and the mapping relations for establishing data source and buffer service are stored in
In hash table, when the monitoring transmission interface of service module receives write instruction, writing control assembly, to return to data source corresponding
Replying instruction is to start corresponding buffer service.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811283383.2A CN109508317B (en) | 2018-10-31 | 2018-10-31 | High-capacity data and service management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811283383.2A CN109508317B (en) | 2018-10-31 | 2018-10-31 | High-capacity data and service management system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109508317A true CN109508317A (en) | 2019-03-22 |
| CN109508317B CN109508317B (en) | 2023-06-09 |
Family
ID=65747229
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811283383.2A Active CN109508317B (en) | 2018-10-31 | 2018-10-31 | High-capacity data and service management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109508317B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110059019A (en) * | 2019-04-17 | 2019-07-26 | 珠海金山网络游戏科技有限公司 | A kind of distribution method and device, calculating equipment and storage medium of memory address |
| CN112328550A (en) * | 2020-11-03 | 2021-02-05 | 深圳壹账通智能科技有限公司 | File management method and device under distributed file system architecture |
| CN113778949A (en) * | 2021-09-27 | 2021-12-10 | 武汉英仕达信息技术有限公司 | Data middleware system for Internet of things |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017088664A1 (en) * | 2015-11-26 | 2017-06-01 | 深圳市中博科创信息技术有限公司 | Data processing method and apparatus for cluster file system |
| CN108170770A (en) * | 2017-12-26 | 2018-06-15 | 山东联科云计算股份有限公司 | A kind of analyzing and training platform based on big data |
| CN108717457A (en) * | 2018-05-23 | 2018-10-30 | 苏州易康萌思网络科技有限公司 | A kind of e-commerce platform big data processing method and system |
-
2018
- 2018-10-31 CN CN201811283383.2A patent/CN109508317B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017088664A1 (en) * | 2015-11-26 | 2017-06-01 | 深圳市中博科创信息技术有限公司 | Data processing method and apparatus for cluster file system |
| CN108170770A (en) * | 2017-12-26 | 2018-06-15 | 山东联科云计算股份有限公司 | A kind of analyzing and training platform based on big data |
| CN108717457A (en) * | 2018-05-23 | 2018-10-30 | 苏州易康萌思网络科技有限公司 | A kind of e-commerce platform big data processing method and system |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110059019A (en) * | 2019-04-17 | 2019-07-26 | 珠海金山网络游戏科技有限公司 | A kind of distribution method and device, calculating equipment and storage medium of memory address |
| CN110059019B (en) * | 2019-04-17 | 2021-12-10 | 珠海金山网络游戏科技有限公司 | Memory address allocation method and device, computing equipment and storage medium |
| CN112328550A (en) * | 2020-11-03 | 2021-02-05 | 深圳壹账通智能科技有限公司 | File management method and device under distributed file system architecture |
| CN113778949A (en) * | 2021-09-27 | 2021-12-10 | 武汉英仕达信息技术有限公司 | Data middleware system for Internet of things |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109508317B (en) | 2023-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20180285167A1 (en) | Database management system providing local balancing within individual cluster node | |
| CN103294710B (en) | A kind of data access method and device | |
| EP3254210B1 (en) | Big data statistics at data-block level | |
| US20200117383A1 (en) | Transferring data between memories utilizing logical block addresses | |
| US8819335B1 (en) | System and method for executing map-reduce tasks in a storage device | |
| CN104820714B (en) | Magnanimity tile small documents memory management method based on hadoop | |
| CN103812939B (en) | Big data storage system | |
| CN103020257B (en) | The implementation method of data manipulation and device | |
| CN109508317A (en) | A kind of Large Volume Data and service management system | |
| CN106909651A (en) | A kind of method for being write based on HDFS small documents and being read | |
| US11709835B2 (en) | Re-ordered processing of read requests | |
| CN110727406A (en) | Data storage scheduling method and device | |
| CN105094709A (en) | Dynamic data compression method for solid-state disc storage system | |
| CN103207889A (en) | Method for retrieving massive face images based on Hadoop | |
| CN106547911A (en) | A kind of access method and system of mass small documents | |
| CN108664577B (en) | File management method and system based on FLASH idle area | |
| CN104216908A (en) | Internet data management system and reading and writing method thereof | |
| CN113486026A (en) | Data processing method, device, equipment and medium | |
| CN108647278B (en) | File management method and system | |
| US20230418827A1 (en) | Processing multi-column streams during query execution via a database system | |
| CN111427920B (en) | Data acquisition method, device, system, computer equipment and storage medium | |
| CN110311817B (en) | Container log processing system for Kubernetes cluster | |
| US20200133732A1 (en) | Coordinating main memory access of a plurality of sets of threads | |
| CN105912621A (en) | Area building energy consumption platform data storing and query method | |
| US9069821B2 (en) | Method of processing files in storage system and data server using the method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230512 Address after: Room 5-01, Floor 5, Building 6, Headquarters Economic Park, No. 1309, Shangye Road, Fengxi New Town, Xixian New District, Xianyang City, Shaanxi Province, 712000 Applicant after: SHAANXI HEYOU NETWORK TECHNOLOGY CO.,LTD. Address before: 430000 No. 04, room 01, floor 1-2, zone 3, 3S geospatial information industry base, wudayuan Road, Donghu New Technology Development Zone, Wuhan City, Hubei Province Applicant before: WUHAN OPTICS VALLEY DATA TECHNOLOGIES Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |