CN101957863B - 数据并行处理方法、装置及系统 - Google Patents
数据并行处理方法、装置及系统 Download PDFInfo
- Publication number
- CN101957863B CN101957863B CN2010105125912A CN201010512591A CN101957863B CN 101957863 B CN101957863 B CN 101957863B CN 2010105125912 A CN2010105125912 A CN 2010105125912A CN 201010512591 A CN201010512591 A CN 201010512591A CN 101957863 B CN101957863 B CN 101957863B
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- partition
- data partition
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 100
- 238000005192 partition Methods 0.000 claims abstract description 91
- 238000000034 method Methods 0.000 claims abstract description 32
- 230000002776 aggregation Effects 0.000 claims abstract description 15
- 238000004220 aggregation Methods 0.000 claims abstract description 15
- 230000015572 biosynthetic process Effects 0.000 claims description 23
- 238000001514 detection method Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 9
- 238000012217 deletion Methods 0.000 claims description 5
- 230000037430 deletion Effects 0.000 claims description 5
- 238000012546 transfer Methods 0.000 claims description 4
- 238000012423 maintenance Methods 0.000 claims description 2
- 238000011084 recovery Methods 0.000 claims description 2
- 238000003860 storage Methods 0.000 abstract description 10
- 230000006870 function Effects 0.000 description 10
- 238000007726 management method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 239000012467 final product Substances 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000005111 flow chemistry technique Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000006116 polymerization reaction Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Claims (12)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN2010105125912A CN101957863B (zh) | 2010-10-14 | 2010-10-14 | 数据并行处理方法、装置及系统 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN2010105125912A CN101957863B (zh) | 2010-10-14 | 2010-10-14 | 数据并行处理方法、装置及系统 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN101957863A CN101957863A (zh) | 2011-01-26 |
CN101957863B true CN101957863B (zh) | 2012-05-09 |
Family
ID=43485192
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN2010105125912A Expired - Fee Related CN101957863B (zh) | 2010-10-14 | 2010-10-14 | 数据并行处理方法、装置及系统 |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN101957863B (zh) |
Families Citing this family (40)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102147750A (zh) * | 2011-01-27 | 2011-08-10 | 中国农业银行股份有限公司 | 作业处理方法和系统 |
CN102111301A (zh) * | 2011-03-28 | 2011-06-29 | 上海云高软件科技有限公司 | 一种通用文件传输系统及其实现方法 |
US9798831B2 (en) | 2011-04-01 | 2017-10-24 | Google Inc. | Processing data in a MapReduce framework |
US8595267B2 (en) * | 2011-06-27 | 2013-11-26 | Amazon Technologies, Inc. | System and method for implementing a scalable data storage service |
CN102332027A (zh) * | 2011-10-15 | 2012-01-25 | 西安交通大学 | 一种基于Hadoop的海量非独立小文件关联存储方法 |
WO2013078583A1 (zh) * | 2011-11-28 | 2013-06-06 | 华为技术有限公司 | 优化数据访问的方法及装置、优化数据存储的方法及装置 |
CN102638456B (zh) * | 2012-03-19 | 2015-09-23 | 杭州海康威视数字技术股份有限公司 | 基于云计算的海量实时视频码流智能分析方法及其系统 |
CN102779025A (zh) * | 2012-03-19 | 2012-11-14 | 南京大学 | 一种基于Hadoop的并行化PLSA方法 |
CN102737114B (zh) * | 2012-05-18 | 2014-08-06 | 北京大学 | 基于MapReduce的大图上距离连接查询方法 |
CN103455374B (zh) * | 2012-06-05 | 2016-10-19 | 阿里巴巴集团控股有限公司 | 一种基于MapReduce的分布式计算方法和装置 |
CN103793442B (zh) * | 2012-11-05 | 2019-05-07 | 北京超图软件股份有限公司 | 空间数据的处理方法及系统 |
CN103023995B (zh) * | 2012-11-29 | 2015-09-09 | 中国电力科学研究院 | 一种基于Hadoop的分布式云存储自动分级数据管理系统 |
CN103034698B (zh) * | 2012-12-05 | 2016-03-30 | 北京奇虎科技有限公司 | 数据存储方法及装置 |
CN104252472B (zh) * | 2013-06-27 | 2018-01-23 | 国际商业机器公司 | 用于并行化数据处理的方法和装置 |
CN103617033A (zh) * | 2013-11-22 | 2014-03-05 | 北京掌阔移动传媒科技有限公司 | 基于MapReduce的数据处理方法、客户端和系统 |
CN103646073A (zh) * | 2013-12-11 | 2014-03-19 | 浪潮电子信息产业股份有限公司 | 一种基于HBase表的条件查询优化方法 |
CN103646541B (zh) * | 2013-12-16 | 2017-05-24 | 电子科技大学 | 一种基于Hadoop的车辆拥挤度获取方法 |
CN104376029B (zh) * | 2014-04-10 | 2017-12-19 | 北京亚信时代融创咨询有限公司 | 一种数据的处理方法及系统 |
CN104199963A (zh) * | 2014-09-19 | 2014-12-10 | 浪潮(北京)电子信息产业有限公司 | HBase数据备份恢复的方法和装置 |
CN104407879B (zh) * | 2014-10-22 | 2018-02-02 | 江苏瑞中数据股份有限公司 | 一种电网时序大数据并行加载方法 |
CN104537003B (zh) * | 2014-12-16 | 2018-01-09 | 北京中交兴路车联网科技有限公司 | 一种Hbase数据库的通用高性能数据写入方法 |
CN104731921B (zh) * | 2015-03-26 | 2018-03-30 | 江苏物联网研究发展中心 | Hadoop分布式文件系统针对日志型小文件的存储和处理方法 |
CN104850591B (zh) * | 2015-04-24 | 2019-03-19 | 百度在线网络技术(北京)有限公司 | 一种数据的转换存储方法及装置 |
CN106570572B (zh) * | 2015-10-12 | 2019-12-17 | 中国石油化工股份有限公司 | 基于MapReduce的旅行时计算方法和装置 |
CN105578212B (zh) * | 2015-12-15 | 2019-02-19 | 南京邮电大学 | 一种大数据中流计算平台下的点对点流媒体实时监测方法 |
CN106648872A (zh) * | 2016-12-29 | 2017-05-10 | 深圳市优必选科技有限公司 | 用于多线程处理的方法及装置、服务器 |
CN106780154B (zh) * | 2017-01-23 | 2020-10-16 | 国网山东省电力公司电力科学研究院 | 多线程信息聚合的输变电工程建设过程环保措施监控系统及方法 |
CN107395669B (zh) * | 2017-06-01 | 2020-04-07 | 华南理工大学 | 一种基于流式实时分布式大数据的数据采集方法及系统 |
CN107391303B (zh) * | 2017-06-30 | 2021-02-23 | 北京奇虎科技有限公司 | 数据处理方法、装置、系统、服务器及计算机存储介质 |
CN108241539B (zh) * | 2018-01-03 | 2021-05-07 | 百度在线网络技术(北京)有限公司 | 基于分布式系统的交互式大数据查询方法、装置、存储介质和终端设备 |
CN108182281B (zh) * | 2018-01-26 | 2022-02-01 | 创新先进技术有限公司 | 基于流式计算的数据处理控制方法、装置、服务器及介质 |
CN108491255B (zh) * | 2018-02-08 | 2020-11-03 | 昆仑智汇数据科技(北京)有限公司 | 自助式MapReduce数据优化分配方法及系统 |
CN112335217A (zh) * | 2018-08-17 | 2021-02-05 | 西门子股份公司 | 分布式数据处理方法、装置及系统和机器可读介质 |
CN109582696B (zh) * | 2018-10-09 | 2023-07-04 | 北京奥星贝斯科技有限公司 | 扫描任务的生成方法及装置、电子设备 |
CN111259047B (zh) * | 2018-12-03 | 2024-06-14 | 顺丰科技有限公司 | 数据加载方法、装置、设备及其存储介质 |
CN109597795B (zh) * | 2018-12-06 | 2020-10-16 | 南京天辰礼达电子科技有限公司 | 一种路基压实施工数据高效处理系统 |
CN110765082B (zh) * | 2019-09-06 | 2023-11-24 | 深圳平安通信科技有限公司 | Hadoop文件处理方法、装置、存储介质及服务器 |
CN111581155B (zh) * | 2020-03-30 | 2023-07-25 | 平安科技(深圳)有限公司 | 数据入数据库的方法、装置和计算机设备 |
CN111625254B (zh) * | 2020-05-06 | 2023-09-08 | Oppo(重庆)智能科技有限公司 | 文件处理方法、装置、终端及存储介质 |
CN112347052A (zh) * | 2020-11-04 | 2021-02-09 | 深圳集智数字科技有限公司 | 一种文件匹配方法及相关装置 |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7698251B2 (en) * | 2006-04-27 | 2010-04-13 | International Business Machines Corporation | Fault tolerant facility for the aggregation of data from multiple processing units |
US7921416B2 (en) * | 2006-10-20 | 2011-04-05 | Yahoo! Inc. | Formal language and translator for parallel processing of data |
US20100162230A1 (en) * | 2008-12-24 | 2010-06-24 | Yahoo! Inc. | Distributed computing system for large-scale data handling |
CN101799809B (zh) * | 2009-02-10 | 2011-12-14 | 中国移动通信集团公司 | 数据挖掘方法和数据挖掘系统 |
-
2010
- 2010-10-14 CN CN2010105125912A patent/CN101957863B/zh not_active Expired - Fee Related
Also Published As
Publication number | Publication date |
---|---|
CN101957863A (zh) | 2011-01-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN101957863B (zh) | 数据并行处理方法、装置及系统 | |
US11558270B2 (en) | Monitoring a stale data queue for deletion events | |
KR101885688B1 (ko) | 낮은 지연속도 데이터 액세스를 위한 데이터 스트림의 분할 | |
US10990288B2 (en) | Systems and/or methods for leveraging in-memory storage in connection with the shuffle phase of MapReduce | |
CN105049268A (zh) | 分布式计算资源分配系统和任务处理方法 | |
US10419528B2 (en) | Dynamically instantiating and terminating data queues | |
US10599529B2 (en) | Instantiating data queues for management of remote data stores | |
CN103176849A (zh) | 一种基于资源分类的虚拟机集群的部署方法 | |
CN106095940A (zh) | 一种基于任务负载的数据迁移方法 | |
US11132221B2 (en) | Method, apparatus, and computer-readable medium for dynamic binding of tasks in a data exchange | |
CN102880658A (zh) | 基于地震数据处理的分布式文件管理系统 | |
CN110347651A (zh) | 基于云存储的数据同步方法、装置、设备及存储介质 | |
US9535743B2 (en) | Data processing control method, computer-readable recording medium, and data processing control device for performing a Mapreduce process | |
Ubarhande et al. | Novel data-distribution technique for Hadoop in heterogeneous cloud environments | |
US20210119854A1 (en) | Scalable statistics and analytics mechanisms in cloud networking | |
WO2018121025A1 (zh) | 比较数据表的数据的方法和系统 | |
CN110308984A (zh) | 一种用于处理地理分布式数据的跨集群计算系统 | |
GB2555682A (en) | Repartitioning data in a distributed computing system | |
Khanna et al. | A dynamic scheduling approach for coordinated wide-area data transfers using gridftp | |
CN107528871A (zh) | 存储系统中的数据分析 | |
CN107609129B (zh) | 日志实时处理系统 | |
Wang et al. | Coupling GPU and MPTCP to improve Hadoop/MapReduce performance | |
Meng et al. | A network load sensitive block placement strategy of HDFS | |
Li et al. | Self-clearing Strategy of Container Image in High Performance Job Scheduling | |
Chen et al. | Research on Data Storage and Processing Optimization Based on Federation HDFS and Spark |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
C14 | Grant of patent or utility model | ||
GR01 | Patent grant | ||
C56 | Change in the name or address of the patentee |
Owner name: CONGXING TECHNOLOGY CO., LTD. Free format text: FORMER NAME: SNRISE CORPORATION |
|
CP03 | Change of name, title or address |
Address after: 510070 one of the 83 best and 507 self compiled works in martyrs Middle Road, Yuexiu District, Guangdong, Guangzhou four, 508 Patentee after: Sunrise Technology Co., Ltd. Address before: 510300, No. 368, Guangzhou Avenue, Guangzhou, Guangdong Patentee before: Snrise Corporation |
|
ASS | Succession or assignment of patent right |
Owner name: HONGKONG SHIYE DEVELOPMENT CO., LTD. Free format text: FORMER OWNER: CONGXING TECHNOLOGY CO., LTD. Effective date: 20150805 |
|
C41 | Transfer of patent application or patent right or utility model | ||
TR01 | Transfer of patent right |
Effective date of registration: 20150805 Address after: Room 32, building 3205, Bank of America, 12 Cecil Harcourt Road, central, Hongkong, China Patentee after: Hongkong world industry development Co., Ltd. Address before: 510070 one of the 507 writers in 83 Middle Road, martyrs' road, Guangzhou, Guangdong, four, 508, edited by myself, Yuexiu District Patentee before: Sunrise Technology Co., Ltd. |
|
ASS | Succession or assignment of patent right |
Owner name: TELEFON AB L.M. ERICSSON (SE) Free format text: FORMER OWNER: HONGKONG SHIYE DEVELOPMENT CO., LTD. Effective date: 20150909 |
|
C41 | Transfer of patent application or patent right or utility model | ||
TR01 | Transfer of patent right |
Effective date of registration: 20150909 Address after: Stockholm Patentee after: Telefon AB L.M. Ericsson [SE] Address before: Room 32, building 3205, Bank of America, 12 Cecil Harcourt Road, central, Hongkong, China Patentee before: Hongkong world industry development Co., Ltd. |
|
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20120509 Termination date: 20191014 |
|
CF01 | Termination of patent right due to non-payment of annual fee |