CN107480202A - 一种用于多并行处理框架的数据处理方法及装置 - Google Patents
一种用于多并行处理框架的数据处理方法及装置 Download PDFInfo
- Publication number
- CN107480202A CN107480202A CN201710584226.4A CN201710584226A CN107480202A CN 107480202 A CN107480202 A CN 107480202A CN 201710584226 A CN201710584226 A CN 201710584226A CN 107480202 A CN107480202 A CN 107480202A
- Authority
- CN
- China
- Prior art keywords
- data
- data storage
- storage
- configuration information
- unified
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/116—Details of conversion of file system types or formats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/545—Interprogram communication where tasks reside in different layers, e.g. user- and kernel-space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/544—Remote
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Fuzzy Systems (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710584226.4A CN107480202B (zh) | 2017-07-18 | 2017-07-18 | 一种用于多并行处理框架的数据处理方法及装置 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710584226.4A CN107480202B (zh) | 2017-07-18 | 2017-07-18 | 一种用于多并行处理框架的数据处理方法及装置 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN107480202A true CN107480202A (zh) | 2017-12-15 |
CN107480202B CN107480202B (zh) | 2020-06-02 |
Family
ID=60595186
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201710584226.4A Active CN107480202B (zh) | 2017-07-18 | 2017-07-18 | 一种用于多并行处理框架的数据处理方法及装置 |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN107480202B (zh) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109271456A (zh) * | 2018-11-16 | 2019-01-25 | 中国银行股份有限公司 | 主机数据库文件导出方法及装置 |
CN110704518A (zh) * | 2019-08-26 | 2020-01-17 | 苏宁云计算有限公司 | 基于Flink引擎的业务数据处理方法及装置 |
CN111125013A (zh) * | 2019-12-26 | 2020-05-08 | 北京锐安科技有限公司 | 一种数据入库方法、装置、设备及介质 |
CN112052239A (zh) * | 2020-08-12 | 2020-12-08 | 网宿科技股份有限公司 | 数据封装方法、电子设备及存储介质 |
CN114880103A (zh) * | 2022-07-11 | 2022-08-09 | 中电云数智科技有限公司 | 一种flink任务适配hadoop生态的系统和方法 |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104573115A (zh) * | 2015-02-04 | 2015-04-29 | 新余兴邦信息产业有限公司 | 支持多类型数据库操作的集成接口的实现方法及系统 |
CN105550318A (zh) * | 2015-12-15 | 2016-05-04 | 深圳市华讯方舟软件技术有限公司 | 一种基于Spark大数据处理平台的查询方法 |
-
2017
- 2017-07-18 CN CN201710584226.4A patent/CN107480202B/zh active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104573115A (zh) * | 2015-02-04 | 2015-04-29 | 新余兴邦信息产业有限公司 | 支持多类型数据库操作的集成接口的实现方法及系统 |
CN105550318A (zh) * | 2015-12-15 | 2016-05-04 | 深圳市华讯方舟软件技术有限公司 | 一种基于Spark大数据处理平台的查询方法 |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109271456A (zh) * | 2018-11-16 | 2019-01-25 | 中国银行股份有限公司 | 主机数据库文件导出方法及装置 |
CN110704518A (zh) * | 2019-08-26 | 2020-01-17 | 苏宁云计算有限公司 | 基于Flink引擎的业务数据处理方法及装置 |
CN110704518B (zh) * | 2019-08-26 | 2022-11-08 | 苏宁云计算有限公司 | 基于Flink引擎的业务数据处理方法及装置 |
CN111125013A (zh) * | 2019-12-26 | 2020-05-08 | 北京锐安科技有限公司 | 一种数据入库方法、装置、设备及介质 |
CN111125013B (zh) * | 2019-12-26 | 2023-03-17 | 北京锐安科技有限公司 | 一种数据入库方法、装置、设备及介质 |
CN112052239A (zh) * | 2020-08-12 | 2020-12-08 | 网宿科技股份有限公司 | 数据封装方法、电子设备及存储介质 |
CN112052239B (zh) * | 2020-08-12 | 2024-02-27 | 网宿科技股份有限公司 | 数据封装方法、电子设备及存储介质 |
CN114880103A (zh) * | 2022-07-11 | 2022-08-09 | 中电云数智科技有限公司 | 一种flink任务适配hadoop生态的系统和方法 |
CN114880103B (zh) * | 2022-07-11 | 2022-09-09 | 中电云数智科技有限公司 | 一种flink任务适配hadoop生态的系统和方法 |
Also Published As
Publication number | Publication date |
---|---|
CN107480202B (zh) | 2020-06-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN107480202A (zh) | 一种用于多并行处理框架的数据处理方法及装置 | |
CN103377148B (zh) | 一种统一表架构中执行部分合并的方法及系统 | |
Borkar et al. | Inside" Big Data management" ogres, onions, or parfaits? | |
CN103377290B (zh) | 删除多级存储架构中的记录 | |
Karpathiotakis et al. | Just-in-time data virtualization: Lightweight data management with ViDa | |
CN103377289B (zh) | 统一表查询处理 | |
Atzeni et al. | Uniform access to non-relational database systems: The SOS platform | |
Karnitis et al. | Migration of relational database to document-oriented database: Structure denormalization and data transformation | |
CN103246749B (zh) | 面向分布式计算的矩阵数据库系统及其查询方法 | |
Atzeni et al. | The relational model is dead, SQL is dead, and I don't feel so good myself | |
CN103049556B (zh) | 一种海量医疗数据的快速统计查询方法 | |
CN109272155A (zh) | 一种基于大数据的企业行为分析系统 | |
US20190026336A1 (en) | Retrieving data from a data storage system | |
CN101477572B (zh) | 基于tds过渡数据存储技术的动态数据仓库的方法与系统 | |
CN107766402A (zh) | 一种楼盘字典云房源大数据平台 | |
CN111712809A (zh) | 通过示例来学习etl规则 | |
CN103678519B (zh) | 一种支持Hive DML增强的混合存储系统及其方法 | |
US20180150515A1 (en) | Query Planning and Execution With Source and Sink Operators | |
CN104504008B (zh) | 一种基于嵌套的SQL到HBase的数据迁移算法 | |
CN103646051A (zh) | 一种基于列存储的大数据并行处理系统及方法 | |
Meehan et al. | Integrating real-time and batch processing in a polystore | |
CN102479217A (zh) | 一种分布式数据仓库中实现计算均衡的方法及装置 | |
CN106453618A (zh) | 基于G‑Cloud云计算的遥感图像处理服务云平台系统 | |
Das et al. | A study on big data integration with data warehouse | |
Saddad et al. | Lake data warehouse architecture for big data solutions |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
CB03 | Change of inventor or designer information | ||
CB03 | Change of inventor or designer information |
Inventor after: Tang Zhuo Inventor after: Liu Kunkun Inventor after: Chen Du Inventor before: Liu Kunkun Inventor before: Tang Zhuo Inventor before: Chen Du |
|
GR01 | Patent grant | ||
GR01 | Patent grant | ||
EE01 | Entry into force of recordation of patent licensing contract | ||
EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20171215 Assignee: TEEWON DIGITAL MEDIA TECHNOLOGY (HUNAN) Co.,Ltd. Assignor: HUNAN University Contract record no.: X2020980006435 Denomination of invention: A data processing method and device for multi parallel processing framework Granted publication date: 20200602 License type: Common License Record date: 20200925 |