CN108920542B - 一种分布式内存大数据处理系统及其数据处理方法 - Google Patents
一种分布式内存大数据处理系统及其数据处理方法 Download PDFInfo
- Publication number
- CN108920542B CN108920542B CN201810605777.9A CN201810605777A CN108920542B CN 108920542 B CN108920542 B CN 108920542B CN 201810605777 A CN201810605777 A CN 201810605777A CN 108920542 B CN108920542 B CN 108920542B
- Authority
- CN
- China
- Prior art keywords
- data
- request
- processing
- aggregation
- local
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 149
- 238000003672 processing method Methods 0.000 title claims abstract description 8
- 230000002776 aggregation Effects 0.000 claims abstract description 82
- 238000004220 aggregation Methods 0.000 claims abstract description 82
- 238000012986 modification Methods 0.000 claims description 30
- 230000004048 modification Effects 0.000 claims description 30
- 238000012217 deletion Methods 0.000 claims description 28
- 230000037430 deletion Effects 0.000 claims description 28
- 238000003780 insertion Methods 0.000 claims description 11
- 230000037431 insertion Effects 0.000 claims description 11
- 230000006870 function Effects 0.000 claims description 5
- 230000006835 compression Effects 0.000 claims 1
- 238000007906 compression Methods 0.000 claims 1
- 239000000203 mixture Substances 0.000 claims 1
- 239000002994 raw material Substances 0.000 claims 1
- 230000004044 response Effects 0.000 abstract description 4
- 238000004364 calculation method Methods 0.000 description 9
- 238000000034 method Methods 0.000 description 4
- 230000000903 blocking effect Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Computer And Data Communications (AREA)
Abstract
Description
Claims (7)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810605777.9A CN108920542B (zh) | 2018-06-13 | 2018-06-13 | 一种分布式内存大数据处理系统及其数据处理方法 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810605777.9A CN108920542B (zh) | 2018-06-13 | 2018-06-13 | 一种分布式内存大数据处理系统及其数据处理方法 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN108920542A CN108920542A (zh) | 2018-11-30 |
CN108920542B true CN108920542B (zh) | 2021-07-20 |
Family
ID=64420962
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201810605777.9A Active CN108920542B (zh) | 2018-06-13 | 2018-06-13 | 一种分布式内存大数据处理系统及其数据处理方法 |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN108920542B (zh) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111092943B (zh) * | 2019-12-13 | 2022-09-20 | 中国科学院深圳先进技术研究院 | 一种树状结构的多集群遥感方法、系统及电子设备 |
CN116956164A (zh) * | 2023-09-18 | 2023-10-27 | 中国科学院精密测量科学与技术创新研究院 | 基于wasm技术的全高层大气激光雷达数据处理方法 |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN1520110A (zh) * | 2003-01-30 | 2004-08-11 | ���ǵ�����ʽ���� | 对转发信息进行动态管理的分布式路由器及其方法 |
US7325109B1 (en) * | 2003-10-24 | 2008-01-29 | Network Appliance, Inc. | Method and apparatus to mirror data at two separate sites without comparing the data at the two sites |
CN101291245A (zh) * | 2007-04-16 | 2008-10-22 | 阿里巴巴集团控股有限公司 | 一种分布式处理方法、系统及其装置 |
CN101697168A (zh) * | 2009-10-22 | 2010-04-21 | 中国科学技术大学 | 一种分布式文件系统动态元数据管理方法及系统 |
CN102123136A (zh) * | 2010-12-26 | 2011-07-13 | 广州大学 | 一种分布式DDoS攻击流识别方法 |
CN102148756A (zh) * | 2011-01-26 | 2011-08-10 | 武汉邮电科学研究院 | 一种基于6LoWPAN邻居发现的树状路由方法 |
CN105897646A (zh) * | 2014-09-25 | 2016-08-24 | 高明利 | 一种基于互联网构建泛在网的方法 |
CN105930345A (zh) * | 2016-04-05 | 2016-09-07 | 国网浙江省电力公司湖州供电公司 | 基于分布式实时数据库的层次索引方法 |
-
2018
- 2018-06-13 CN CN201810605777.9A patent/CN108920542B/zh active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN1520110A (zh) * | 2003-01-30 | 2004-08-11 | ���ǵ�����ʽ���� | 对转发信息进行动态管理的分布式路由器及其方法 |
US7325109B1 (en) * | 2003-10-24 | 2008-01-29 | Network Appliance, Inc. | Method and apparatus to mirror data at two separate sites without comparing the data at the two sites |
CN101291245A (zh) * | 2007-04-16 | 2008-10-22 | 阿里巴巴集团控股有限公司 | 一种分布式处理方法、系统及其装置 |
CN101697168A (zh) * | 2009-10-22 | 2010-04-21 | 中国科学技术大学 | 一种分布式文件系统动态元数据管理方法及系统 |
CN102123136A (zh) * | 2010-12-26 | 2011-07-13 | 广州大学 | 一种分布式DDoS攻击流识别方法 |
CN102148756A (zh) * | 2011-01-26 | 2011-08-10 | 武汉邮电科学研究院 | 一种基于6LoWPAN邻居发现的树状路由方法 |
CN105897646A (zh) * | 2014-09-25 | 2016-08-24 | 高明利 | 一种基于互联网构建泛在网的方法 |
CN105930345A (zh) * | 2016-04-05 | 2016-09-07 | 国网浙江省电力公司湖州供电公司 | 基于分布式实时数据库的层次索引方法 |
Also Published As
Publication number | Publication date |
---|---|
CN108920542A (zh) | 2018-11-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11422982B2 (en) | Scaling stateful clusters while maintaining access | |
CN108170768B (zh) | 数据库同步方法、装置及可读介质 | |
US10944807B2 (en) | Organizing present and future reads from a tiered streaming data storage layer | |
CN109783438B (zh) | 基于librados的分布式NFS系统及其构建方法 | |
US10437721B2 (en) | Efficient garbage collection for a log-structured data store | |
CN109947668B (zh) | 存储数据的方法和装置 | |
US20170116215A1 (en) | System and method for in-place data writes to reduce fragmentation in a multidimensional database environment | |
JP2018505501A (ja) | アプリケーション中心のオブジェクトストレージ | |
US20200117676A1 (en) | Method and system for executing queries on indexed views | |
CN103595797B (zh) | 一种分布式存储系统中的缓存方法 | |
CN104899156A (zh) | 一种面向大规模社交网络的图数据存储及查询方法 | |
CN113377868B (zh) | 一种基于分布式kv数据库的离线存储系统 | |
CN112506870B (zh) | 数据仓库增量更新方法、装置及计算机设备 | |
CN114385760A (zh) | 增量数据实时同步的方法、装置、计算机设备及存储介质 | |
CN111177254B (zh) | 一种异构关系型数据库之间数据同步的方法和装置 | |
CN108920542B (zh) | 一种分布式内存大数据处理系统及其数据处理方法 | |
CN103593460A (zh) | 数据分级存储系统和数据分级存储方法 | |
US20170031959A1 (en) | Scheduling database compaction in ip drives | |
CN112579550B (zh) | 一种分布式文件系统的元数据信息同步方法及系统 | |
US20240129251A1 (en) | Data processing method and apparatus, computer device, and readable storage medium | |
CN114090580A (zh) | 数据处理方法、装置、设备、存储介质及产品 | |
CN112612799A (zh) | 一种数据同步方法及终端 | |
CN112486913A (zh) | 一种基于集群环境下的日志异步存储方法和设备 | |
CN116991858A (zh) | 变更数据的采集方法、采集装置和数据采集系统 | |
CN100486177C (zh) | 一种网管对网元进行同步操作的方法及其系统 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
TR01 | Transfer of patent right | ||
TR01 | Transfer of patent right |
Effective date of registration: 20230710 Address after: 362500 42 shangletao, letao village, XunZhong Town, Dehua County, Quanzhou City, Fujian Province Patentee after: Sun Qingzhu Address before: Room 521, Yuanhe building, 959 Jiayuan Road, Yuanhe street, Xiangcheng District, Suzhou City, Jiangsu Province Patentee before: SUZHOU NIEVA INFORMATION TECHNOLOGY Co.,Ltd. |
|
TR01 | Transfer of patent right | ||
TR01 | Transfer of patent right |
Effective date of registration: 20230823 Address after: 710000, No. 5-20001207, 30 Chama Lane, Shuangzhao Village, Yongle Town, Jinghe New City, Xixian New District, Xi'an City, Shaanxi Province Patentee after: Xi'an Xuanyue Cloud Network Technology Co.,Ltd. Address before: 362500 42 shangletao, letao village, XunZhong Town, Dehua County, Quanzhou City, Fujian Province Patentee before: Sun Qingzhu |