CN110263059B - Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 - Google Patents
Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 Download PDFInfo
- Publication number
- CN110263059B CN110263059B CN201910438036.0A CN201910438036A CN110263059B CN 110263059 B CN110263059 B CN 110263059B CN 201910438036 A CN201910438036 A CN 201910438036A CN 110263059 B CN110263059 B CN 110263059B
- Authority
- CN
- China
- Prior art keywords
- elements
- frequency
- partitioning
- sampling
- intermediate data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Complex Calculations (AREA)
Abstract
Description
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910438036.0A CN110263059B (zh) | 2019-05-24 | 2019-05-24 | Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910438036.0A CN110263059B (zh) | 2019-05-24 | 2019-05-24 | Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110263059A CN110263059A (zh) | 2019-09-20 |
CN110263059B true CN110263059B (zh) | 2021-05-11 |
Family
ID=67915335
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910438036.0A Active CN110263059B (zh) | 2019-05-24 | 2019-05-24 | Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110263059B (zh) |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111258624B (zh) * | 2020-01-13 | 2023-04-28 | 上海交通大学 | 开源软件开发中Issue解决时间的预测方法及系统 |
CN112000467A (zh) * | 2020-07-24 | 2020-11-27 | 广东技术师范大学 | 一种数据倾斜处理方法、装置、终端设备及存储介质 |
CN112612614B (zh) * | 2020-12-28 | 2024-07-26 | 江苏苏宁云计算有限公司 | 一种数据排序方法、装置及系统 |
CN113626426B (zh) * | 2021-07-06 | 2022-06-14 | 佛山市禅城区政务服务数据管理局 | 一种生态网格数据的采集传输方法及系统 |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105550374A (zh) * | 2016-01-29 | 2016-05-04 | 湖南大学 | Spark云服务环境下面向大数据的随机森林并行机器学习方法 |
CN109034981A (zh) * | 2018-08-23 | 2018-12-18 | 上海海事大学 | 一种电商协同过滤推荐方法 |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9171081B2 (en) * | 2012-03-06 | 2015-10-27 | Microsoft Technology Licensing, Llc | Entity augmentation service from latent relational data |
US9697274B2 (en) * | 2013-12-27 | 2017-07-04 | International Business Machines Corporation | Stratified sampling using adaptive parallel data processing |
-
2019
- 2019-05-24 CN CN201910438036.0A patent/CN110263059B/zh active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105550374A (zh) * | 2016-01-29 | 2016-05-04 | 湖南大学 | Spark云服务环境下面向大数据的随机森林并行机器学习方法 |
CN109034981A (zh) * | 2018-08-23 | 2018-12-18 | 上海海事大学 | 一种电商协同过滤推荐方法 |
Also Published As
Publication number | Publication date |
---|---|
CN110263059A (zh) | 2019-09-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110263059B (zh) | Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 | |
CN105446979B (zh) | 数据挖掘方法和节点 | |
CN110880014B (zh) | 数据处理方法、装置、计算机设备及存储介质 | |
CN113343545B (zh) | 结构自适应优化设计方法、装置、设备和介质 | |
Awad et al. | Dynamic graphs on the GPU | |
CN105556474A (zh) | 管理数据操作的存储器和存储空间 | |
US20200387548A1 (en) | Graph data processing method and apparatus thereof | |
CN112819258A (zh) | 银行网点到店客户数量预测方法及装置 | |
CN108875914B (zh) | 对神经网络数据进行预处理和后处理的方法和装置 | |
US10621173B2 (en) | Data processing device, data processing method, and recording medium | |
CN115098600A (zh) | 用于数据仓库的有向无环图构建方法、装置和计算机设备 | |
CN114168318A (zh) | 存储释放模型的训练方法、存储释放方法及设备 | |
CN110245147A (zh) | 区块链数据处理方法、装置、可读存储介质和计算机设备 | |
CN112948504B (zh) | 数据采集方法、装置、计算机设备和存储介质 | |
US7647592B2 (en) | Methods and systems for assigning objects to processing units | |
CN113111351B (zh) | 测试方法、装置和计算机可读存储介质 | |
JP6812826B2 (ja) | 格納方法、格納装置及び格納プログラム | |
CN108073658B (zh) | 一种数据同步系统及方法 | |
CN113010310A (zh) | 作业数据的处理方法、装置和服务器 | |
CN111849544B (zh) | 加氢裂化的产品质量自控方法、装置和存储器 | |
CN109165325B (zh) | 用于切分图数据的方法、装置、设备以及计算机可读存储介质 | |
CN111414422A (zh) | 一种数据分布方法、装置、设备和存储介质 | |
CN111158732A (zh) | 访问数据处理方法、装置、计算机设备及存储介质 | |
CN113360479A (zh) | 数据迁移方法、装置、计算机设备和存储介质 | |
Chang et al. | Multiobjective optimization of the variability of the high-performance LINPACK solver |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CB03 | Change of inventor or designer information | ||
CB03 | Change of inventor or designer information |
Inventor after: Tang Zhuo Inventor after: Fu Zhongming Inventor after: Chen Cen Inventor after: Chen Jianguo Inventor after: Li Kenli Inventor after: Li Keqin Inventor before: Tang Zhuo Inventor before: Fu Zhongming Inventor before: Chen Cen Inventor before: Chen Jianguo Inventor before: Li Kenli Inventor before: Li Keqin Inventor before: Liao Xiangke |
|
CB03 | Change of inventor or designer information | ||
CB03 | Change of inventor or designer information |
Inventor after: Li Kenli Inventor after: Fu Zhongming Inventor after: Tang Zhuo Inventor after: Chen Cen Inventor after: Chen Jianguo Inventor after: Li Keqin Inventor before: Tang Zhuo Inventor before: Fu Zhongming Inventor before: Chen Cen Inventor before: Chen Jianguo Inventor before: Li Kenli Inventor before: Li Keqin |