CA2930026C - Techniques d'ingestion et de persistance de flux de donnees - Google Patents
Techniques d'ingestion et de persistance de flux de donnees Download PDFInfo
- Publication number
- CA2930026C CA2930026C CA2930026A CA2930026A CA2930026C CA 2930026 C CA2930026 C CA 2930026C CA 2930026 A CA2930026 A CA 2930026A CA 2930026 A CA2930026 A CA 2930026A CA 2930026 C CA2930026 C CA 2930026C
- Authority
- CA
- Canada
- Prior art keywords
- data
- stream
- policy
- nodes
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000037406 food intake Effects 0.000 title claims abstract description 215
- 230000002688 persistence Effects 0.000 title claims abstract description 101
- 238000000034 method Methods 0.000 title claims description 103
- 230000004044 response Effects 0.000 claims abstract description 58
- 230000005540 biological transmission Effects 0.000 claims abstract description 32
- 238000005192 partition Methods 0.000 claims description 374
- 238000007726 management method Methods 0.000 claims description 102
- 238000000638 solvent extraction Methods 0.000 claims description 66
- 238000013500 data storage Methods 0.000 claims description 19
- 239000007787 solid Substances 0.000 claims description 7
- 238000012545 processing Methods 0.000 description 179
- 238000013507 mapping Methods 0.000 description 109
- 230000006870 function Effects 0.000 description 48
- 238000011084 recovery Methods 0.000 description 40
- 230000008569 process Effects 0.000 description 23
- 238000010586 diagram Methods 0.000 description 20
- 238000012544 monitoring process Methods 0.000 description 17
- 230000002085 persistent effect Effects 0.000 description 15
- 230000008859 change Effects 0.000 description 14
- 238000001514 detection method Methods 0.000 description 14
- 230000036541 health Effects 0.000 description 14
- 238000013459 approach Methods 0.000 description 13
- 230000000694 effects Effects 0.000 description 13
- 230000003993 interaction Effects 0.000 description 12
- 238000005516 engineering process Methods 0.000 description 11
- 102100033972 Amyloid protein-binding protein 2 Human genes 0.000 description 8
- 101000785279 Dictyostelium discoideum Calcium-transporting ATPase PAT1 Proteins 0.000 description 8
- 101000779309 Homo sapiens Amyloid protein-binding protein 2 Proteins 0.000 description 8
- 101000713296 Homo sapiens Proton-coupled amino acid transporter 1 Proteins 0.000 description 8
- 238000004458 analytical method Methods 0.000 description 8
- 230000008901 benefit Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 7
- 238000012217 deletion Methods 0.000 description 7
- 230000037430 deletion Effects 0.000 description 7
- 230000009471 action Effects 0.000 description 6
- 230000006855 networking Effects 0.000 description 6
- 230000000717 retained effect Effects 0.000 description 6
- 230000000977 initiatory effect Effects 0.000 description 5
- 238000002955 isolation Methods 0.000 description 5
- 239000003795 chemical substances by application Substances 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000008520 organization Effects 0.000 description 4
- 230000002093 peripheral effect Effects 0.000 description 4
- 238000009966 trimming Methods 0.000 description 4
- 238000010276 construction Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 235000019580 granularity Nutrition 0.000 description 3
- 230000003862 health status Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000010076 replication Effects 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 208000034188 Stiff person spectrum disease Diseases 0.000 description 2
- 125000002015 acyclic group Chemical group 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- ZPUCINDJVBIVPJ-LJISPDSOSA-N cocaine Chemical compound O([C@H]1C[C@@H]2CC[C@@H](N2C)[C@H]1C(=O)OC)C(=O)C1=CC=CC=C1 ZPUCINDJVBIVPJ-LJISPDSOSA-N 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 230000008030 elimination Effects 0.000 description 2
- 238000003379 elimination reaction Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000003780 insertion Methods 0.000 description 2
- 230000037431 insertion Effects 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 230000014759 maintenance of location Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 230000008093 supporting effect Effects 0.000 description 2
- 229920010524 Syndiotactic polystyrene Polymers 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000001994 activation Methods 0.000 description 1
- 239000000872 buffer Substances 0.000 description 1
- 238000001816 cooling Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013501 data transformation Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000003116 impacting effect Effects 0.000 description 1
- 230000001976 improved effect Effects 0.000 description 1
- 239000003999 initiator Substances 0.000 description 1
- 208000012112 ischiocoxopodopatellar syndrome Diseases 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003449 preventive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004043 responsiveness Effects 0.000 description 1
- 238000009938 salting Methods 0.000 description 1
- 238000002490 spark plasma sintering Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 229920000638 styrene acrylonitrile Polymers 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Selon l'invention, une interface programmatique est mise en uvre, permettant à un client d'un service de gestion de flux de sélectionner une politique d'ingestion de données pour un flux de données. Une requête de client sélectionnant une politique d'ingestion au moins une fois est reçue. Conformément à la politique d'ingestion au moins une fois, un client peut transmettre une indication d'un enregistrement de données une ou plusieurs fois au service jusqu'à ce qu'un accusé de réception positif soit reçu. En réponse à la réception d'une pluralité de transmissions indiquant un enregistrement de données particulier, des accusés de réception positifs respectifs sont envoyés au client. Sur la base d'une politique de persistance sélectionnée pour le flux, des copies de l'enregistrement de données sont mémorisées à un ou plusieurs emplacements de stockage en réponse à une transmission particulière parmi la pluralité de transmissions.
Applications Claiming Priority (5)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US14/077,171 | 2013-11-11 | ||
| US14/077,171 US9720989B2 (en) | 2013-11-11 | 2013-11-11 | Dynamic partitioning techniques for data streams |
| US14/077,162 | 2013-11-11 | ||
| US14/077,162 US9858322B2 (en) | 2013-11-11 | 2013-11-11 | Data stream ingestion and persistence techniques |
| PCT/US2014/065052 WO2015070232A1 (fr) | 2013-11-11 | 2014-11-11 | Techniques d'ingestion et de persistance de flux de données |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CA2930026A1 CA2930026A1 (fr) | 2015-05-14 |
| CA2930026C true CA2930026C (fr) | 2020-06-16 |
Family
ID=53042245
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA2930026A Active CA2930026C (fr) | 2013-11-11 | 2014-11-11 | Techniques d'ingestion et de persistance de flux de donnees |
Country Status (5)
| Country | Link |
|---|---|
| EP (1) | EP3069275A4 (fr) |
| JP (2) | JP6357243B2 (fr) |
| CN (1) | CN105765575B (fr) |
| CA (1) | CA2930026C (fr) |
| WO (1) | WO2015070232A1 (fr) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11941029B2 (en) | 2022-02-03 | 2024-03-26 | Bank Of America Corporation | Automatic extension of database partitions |
Families Citing this family (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10148719B2 (en) | 2015-06-05 | 2018-12-04 | Microsoft Technology Licensing, Llc. | Using anchors for reliable stream processing |
| US9942272B2 (en) * | 2015-06-05 | 2018-04-10 | Microsoft Technology Licensing, Llc. | Handling out of order events |
| US9880769B2 (en) | 2015-06-05 | 2018-01-30 | Microsoft Technology Licensing, Llc. | Streaming joins in constrained memory environments |
| US10868741B2 (en) | 2015-06-05 | 2020-12-15 | Microsoft Technology Licensing, Llc | Anchor shortening across streaming nodes |
| US9602455B2 (en) * | 2015-08-07 | 2017-03-21 | Machine Zone, Inc. | Scalable, real-time messaging system |
| KR102181640B1 (ko) | 2016-05-17 | 2020-11-23 | 아브 이니티오 테크놀로지 엘엘시 | 재구성가능한 분산 처리 |
| CN107783834B (zh) | 2016-08-30 | 2021-05-07 | 伊姆西公司 | 用于处理数据的方法和系统 |
| US10594760B2 (en) * | 2017-01-25 | 2020-03-17 | Futurewei Technologies, Inc. | Intelligent event streaming |
| US11314648B2 (en) * | 2017-02-08 | 2022-04-26 | Arm Limited | Data processing |
| US10719495B2 (en) * | 2017-02-09 | 2020-07-21 | Micron Technology, Inc. | Stream selection for multi-stream storage devices |
| CN107612687B (zh) * | 2017-09-25 | 2021-04-27 | 西安建筑科技大学 | 一种基于ElGamal加密的动态多副本数据持有性验证方法 |
| US11120052B1 (en) | 2018-06-28 | 2021-09-14 | Amazon Technologies, Inc. | Dynamic distributed data clustering using multi-level hash trees |
| US10884644B2 (en) * | 2018-06-28 | 2021-01-05 | Amazon Technologies, Inc. | Dynamic distributed data clustering |
| US11163737B2 (en) * | 2018-11-21 | 2021-11-02 | Google Llc | Storage and structured search of historical security data |
| DE112019005842T5 (de) * | 2018-11-23 | 2021-08-12 | Amazon Technologies, Inc. | Skalierbare architektur für eine verteilte zeitreihendatenbank |
| US11989186B2 (en) * | 2018-11-23 | 2024-05-21 | Amazon Technologies, Inc. | Scalable architecture for a distributed time-series database |
| US11934409B2 (en) | 2018-11-23 | 2024-03-19 | Amazon Technologies, Inc. | Continuous functions in a time-series database |
| US11409725B1 (en) | 2019-02-04 | 2022-08-09 | Amazon Technologies, Inc. | Multi-tenant partitioning in a time-series database |
| US11853317B1 (en) | 2019-03-18 | 2023-12-26 | Amazon Technologies, Inc. | Creating replicas using queries to a time series database |
| CN110147354B (zh) * | 2019-04-19 | 2023-06-02 | 平安科技（深圳）有限公司 | 批量数据编辑方法、装置、计算机设备及存储介质 |
| US11599516B1 (en) | 2020-06-24 | 2023-03-07 | Amazon Technologies, Inc. | Scalable metadata index for a time-series database |
| JP7564449B2 (ja) | 2021-03-29 | 2024-10-09 | 富士通株式会社 | データ処理プログラム、情報処理システム及びデータ処理方法 |
| US11941014B1 (en) | 2021-06-16 | 2024-03-26 | Amazon Technologies, Inc. | Versioned metadata management for a time-series database |
| US11461347B1 (en) | 2021-06-16 | 2022-10-04 | Amazon Technologies, Inc. | Adaptive querying of time-series data over tiered storage |
| CN114116732B (zh) * | 2022-01-24 | 2022-04-05 | 北京奥星贝斯科技有限公司 | 处理事务的方法、装置、存储装置以及服务器 |
| CN114676117B (zh) * | 2022-05-27 | 2022-08-16 | 成都明途科技有限公司 | 一种岗位数据存储方法、装置及岗位机器人 |
| US11960774B2 (en) | 2022-07-20 | 2024-04-16 | The Toronto-Dominion Bank | System, method, and device for uploading data from premises to remote computing environments |
| CN117666928A (zh) * | 2022-08-30 | 2024-03-08 | 华为云计算技术有限公司 | 一种数据访问方法和系统 |
| CN115480914B (zh) * | 2022-09-02 | 2023-07-21 | 江苏安超云软件有限公司 | 一种多租户服务的实现方法及系统 |
Family Cites Families (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5692177A (en) * | 1994-10-26 | 1997-11-25 | Microsoft Corporation | Method and system for data set storage by iteratively searching for perfect hashing functions |
| US5813009A (en) * | 1995-07-28 | 1998-09-22 | Univirtual Corp. | Computer based records management system method |
| JP4057074B2 (ja) * | 1996-04-25 | 2008-03-05 | 富士通株式会社 | ストリームデータのストライピング方法およびストリームサーバ |
| US6272598B1 (en) * | 1999-03-22 | 2001-08-07 | Hewlett-Packard Company | Web cache performance by applying different replacement policies to the web cache |
| US6505216B1 (en) * | 1999-10-01 | 2003-01-07 | Emc Corporation | Methods and apparatus for backing-up and restoring files using multiple trails |
| CA2564967C (fr) * | 2004-04-30 | 2014-09-30 | Commvault Systems, Inc. | Systemes et procedes hierarchiques permettant d'obtenir une vue unifiee d'informations de stockage |
| US7814056B2 (en) * | 2004-05-21 | 2010-10-12 | Computer Associates Think, Inc. | Method and apparatus for data backup using data blocks |
| US7606844B2 (en) * | 2005-12-19 | 2009-10-20 | Commvault Systems, Inc. | System and method for performing replication copy storage operations |
| US7921077B2 (en) * | 2006-06-29 | 2011-04-05 | Netapp, Inc. | System and method for managing data deduplication of storage systems utilizing persistent consistency point images |
| US7716186B2 (en) * | 2007-01-22 | 2010-05-11 | International Business Machines Corporation | Method and system for transparent backup to a hierarchical storage system |
| US8190960B1 (en) * | 2007-12-13 | 2012-05-29 | Force10 Networks, Inc. | Guaranteed inter-process communication |
| US8533478B2 (en) * | 2008-10-24 | 2013-09-10 | Hewlett-Packard Development Company, L. P. | System for and method of writing and reading redundant data |
| JP5396848B2 (ja) * | 2008-12-16 | 2014-01-22 | 富士通株式会社 | データ処理プログラム、サーバ装置およびデータ処理方法 |
| US20100257140A1 (en) * | 2009-03-31 | 2010-10-07 | Philip John Davis | Data archiving and retrieval system |
| US8560639B2 (en) * | 2009-04-24 | 2013-10-15 | Microsoft Corporation | Dynamic placement of replica data |
| US20100306232A1 (en) * | 2009-05-28 | 2010-12-02 | Harris Corporation | Multimedia system providing database of shared text comment data indexed to video source data and related methods |
| CN102741800B (zh) * | 2009-09-18 | 2015-11-25 | 株式会社日立制作所 | 删除复制数据的存储系统 |
| US20120079090A1 (en) * | 2010-09-17 | 2012-03-29 | Oracle International Corporation | Stateful subnet manager failover in a middleware machine environment |
| US9137304B2 (en) * | 2011-05-25 | 2015-09-15 | Alcatel Lucent | Method and apparatus for achieving data security in a distributed cloud computing environment |
| JP5544523B2 (ja) * | 2011-07-19 | 2014-07-09 | 日本電信電話株式会社 | 分散処理システム、分散処理方法、負荷分散装置、負荷分散方法、及び、負荷分散プログラム |
| JP2013058101A (ja) * | 2011-09-08 | 2013-03-28 | Interlink:Kk | クラウドコンピューティングシステム |
| US20130085995A1 (en) * | 2011-09-29 | 2013-04-04 | International Business Machines Corporation | Managing back up operations for data |
| WO2013069073A1 (fr) * | 2011-11-07 | 2013-05-16 | 株式会社日立製作所 | Système, appareil et procédé de gestion de données de séquence temporelle |
| WO2013070873A1 (fr) * | 2011-11-10 | 2013-05-16 | Treasure Data, Inc. | Système et procédé permettant d'utiliser une plate-forme de données volumineuses |
| US8886781B2 (en) * | 2011-12-13 | 2014-11-11 | Microsoft Corporation | Load balancing in cluster storage systems |
| US9098344B2 (en) * | 2011-12-27 | 2015-08-04 | Microsoft Technology Licensing, Llc | Cloud-edge topologies |
-
2014
- 2014-11-11 EP EP14860617.1A patent/EP3069275A4/fr not_active Withdrawn
- 2014-11-11 CN CN201480061590.7A patent/CN105765575B/zh active Active
- 2014-11-11 WO PCT/US2014/065052 patent/WO2015070232A1/fr active Application Filing
- 2014-11-11 JP JP2016553274A patent/JP6357243B2/ja active Active
- 2014-11-11 CA CA2930026A patent/CA2930026C/fr active Active
-
2018
- 2018-04-26 JP JP2018084896A patent/JP6510112B2/ja active Active
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11941029B2 (en) | 2022-02-03 | 2024-03-26 | Bank Of America Corporation | Automatic extension of database partitions |
Also Published As
| Publication number | Publication date |
|---|---|
| JP2017501515A (ja) | 2017-01-12 |
| EP3069275A1 (fr) | 2016-09-21 |
| JP6510112B2 (ja) | 2019-05-08 |
| JP2018133105A (ja) | 2018-08-23 |
| CA2930026A1 (fr) | 2015-05-14 |
| WO2015070232A1 (fr) | 2015-05-14 |
| EP3069275A4 (fr) | 2017-04-26 |
| JP6357243B2 (ja) | 2018-07-11 |
| CN105765575B (zh) | 2019-11-05 |
| CN105765575A (zh) | 2016-07-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10795905B2 (en) | Data stream ingestion and persistence techniques | |
| US10691716B2 (en) | Dynamic partitioning techniques for data streams | |
| CA2929776C (fr) | Options de securite configurables par le client pour des flux de donnees | |
| CA2930026C (fr) | Techniques d'ingestion et de persistance de flux de donnees | |
| CA2930101C (fr) | Cadriciel de traitement de flux de donnees base sur partitionnement | |
| CA2929777C (fr) | Service gere pour l'acquisition, la memorisation et l'utilisation de flux de donnees a grande echelle | |
| US10467105B2 (en) | Chained replication techniques for large-scale data streams | |
| US9471585B1 (en) | Decentralized de-duplication techniques for largescale data streams |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request |
Effective date: 20160506 |