EP4222617A1 - Bahnabkratzen durch verwendung von proxies und anwendungen davon - Google Patents
Bahnabkratzen durch verwendung von proxies und anwendungen davonInfo
- Publication number
- EP4222617A1 EP4222617A1 EP22740786.3A EP22740786A EP4222617A1 EP 4222617 A1 EP4222617 A1 EP 4222617A1 EP 22740786 A EP22740786 A EP 22740786A EP 4222617 A1 EP4222617 A1 EP 4222617A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- request
- web
- target website
- content
- requests
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007790 scraping Methods 0.000 title claims abstract description 481
- 238000000034 method Methods 0.000 claims description 223
- 230000004044 response Effects 0.000 claims description 175
- 230000001360 synchronised effect Effects 0.000 claims description 36
- 230000006870 function Effects 0.000 claims description 29
- 238000003860 storage Methods 0.000 claims description 23
- 230000008569 process Effects 0.000 claims description 21
- 238000012545 processing Methods 0.000 claims description 21
- 238000007906 compression Methods 0.000 claims description 20
- 230000006835 compression Effects 0.000 claims description 20
- 230000036541 health Effects 0.000 claims description 18
- 230000005540 biological transmission Effects 0.000 claims description 14
- 238000007792 addition Methods 0.000 claims description 12
- 230000000694 effects Effects 0.000 claims description 8
- 238000013500 data storage Methods 0.000 claims description 7
- 230000003278 mimic effect Effects 0.000 claims description 7
- 238000005192 partition Methods 0.000 claims description 7
- 230000004931 aggregating effect Effects 0.000 claims description 5
- 230000000873 masking effect Effects 0.000 abstract description 3
- 230000000903 blocking effect Effects 0.000 abstract description 2
- 235000014510 cooky Nutrition 0.000 description 33
- 238000010586 diagram Methods 0.000 description 18
- 238000004891 communication Methods 0.000 description 9
- 238000013475 authorization Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 4
- 238000005457 optimization Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 3
- 230000000737 periodic effect Effects 0.000 description 3
- 238000000638 solvent extraction Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000002085 persistent effect Effects 0.000 description 2
- 230000001105 regulatory effect Effects 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 241000239290 Araneae Species 0.000 description 1
- 241001522296 Erithacus rubecula Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004378 air conditioning Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000004132 cross linking Methods 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000013497 data interchange Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000003306 harvesting Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
- 230000005641 tunneling Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
- G06F16/972—Access to data in other repository systems, e.g. legacy data or dynamic Web page generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EP23171700.0A EP4227829A1 (de) | 2021-07-08 | 2022-06-24 | Bahnabkratzen durch verwendung von proxies und anwendungen davon |
| EP23171587.1A EP4227828A1 (de) | 2021-07-08 | 2022-06-24 | Bahnabkratzen durch verwendung von proxies und anwendungen davon |
Applications Claiming Priority (8)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US202163219660P | 2021-07-08 | 2021-07-08 | |
| US17/373,634 US11416564B1 (en) | 2021-07-08 | 2021-07-12 | Web scraper history management across multiple data centers |
| US17/373,312 US20230018983A1 (en) | 2021-07-08 | 2021-07-12 | Traffic counting for proxy web scraping |
| US17/373,482 US11416291B1 (en) | 2021-07-08 | 2021-07-12 | Database server management for proxy scraping jobs |
| US17/373,570 US11281730B1 (en) | 2021-07-08 | 2021-07-12 | Direct leg access for proxy web scraping |
| US17/373,608 US11204971B1 (en) | 2021-07-08 | 2021-07-12 | Token-based authentication for a proxy web scraping service |
| US17/373,287 US11372937B1 (en) | 2021-07-08 | 2021-07-12 | Throttling client requests for web scraping |
| PCT/EP2022/067331 WO2023280593A1 (en) | 2021-07-08 | 2022-06-24 | Web scraping through use of proxies, and applications thereof |
Related Child Applications (4)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP23171587.1A Division EP4227828A1 (de) | 2021-07-08 | 2022-06-24 | Bahnabkratzen durch verwendung von proxies und anwendungen davon |
| EP23171587.1A Division-Into EP4227828A1 (de) | 2021-07-08 | 2022-06-24 | Bahnabkratzen durch verwendung von proxies und anwendungen davon |
| EP23171700.0A Division EP4227829A1 (de) | 2021-07-08 | 2022-06-24 | Bahnabkratzen durch verwendung von proxies und anwendungen davon |
| EP23171700.0A Division-Into EP4227829A1 (de) | 2021-07-08 | 2022-06-24 | Bahnabkratzen durch verwendung von proxies und anwendungen davon |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| EP4222617A1 true EP4222617A1 (de) | 2023-08-09 |
Family
ID=87068173
Family Applications (3)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP22740786.3A Pending EP4222617A1 (de) | 2021-07-08 | 2022-06-24 | Bahnabkratzen durch verwendung von proxies und anwendungen davon |
| EP23171587.1A Pending EP4227828A1 (de) | 2021-07-08 | 2022-06-24 | Bahnabkratzen durch verwendung von proxies und anwendungen davon |
| EP23171700.0A Pending EP4227829A1 (de) | 2021-07-08 | 2022-06-24 | Bahnabkratzen durch verwendung von proxies und anwendungen davon |
Family Applications After (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP23171587.1A Pending EP4227828A1 (de) | 2021-07-08 | 2022-06-24 | Bahnabkratzen durch verwendung von proxies und anwendungen davon |
| EP23171700.0A Pending EP4227829A1 (de) | 2021-07-08 | 2022-06-24 | Bahnabkratzen durch verwendung von proxies und anwendungen davon |
Country Status (3)
| Country | Link |
|---|---|
| EP (3) | EP4222617A1 (de) |
| CA (1) | CA3214799A1 (de) |
| IL (1) | IL308559A (de) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| LT3780557T (lt) * | 2019-02-25 | 2023-03-10 | Bright Data Ltd. | Turinio parsisiuntimo, naudojant url bandymų mechanizmą, sistema ir būdas |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7653617B2 (en) * | 2005-08-29 | 2010-01-26 | Google Inc. | Mobile sitemaps |
| GB2507294A (en) * | 2012-10-25 | 2014-04-30 | Ibm | Server work-load management using request prioritization |
| US10965770B1 (en) * | 2020-09-11 | 2021-03-30 | Metacluster It, Uab | Dynamic optimization of request parameters for proxy server |
-
2022
- 2022-06-24 EP EP22740786.3A patent/EP4222617A1/de active Pending
- 2022-06-24 EP EP23171587.1A patent/EP4227828A1/de active Pending
- 2022-06-24 EP EP23171700.0A patent/EP4227829A1/de active Pending
- 2022-06-24 CA CA3214799A patent/CA3214799A1/en active Pending
- 2022-06-24 IL IL308559A patent/IL308559A/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| CA3214799A1 (en) | 2023-01-12 |
| EP4227829A1 (de) | 2023-08-16 |
| EP4227828A1 (de) | 2023-08-16 |
| IL308559A (en) | 2024-01-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11204971B1 (en) | Token-based authentication for a proxy web scraping service | |
| AU2018201459B2 (en) | System and method for improving access to search results | |
| US7904345B2 (en) | Providing website hosting overage protection by transference to an overflow server | |
| US9444759B2 (en) | Service provider registration by a content broker | |
| RU2549135C2 (ru) | Система и способ для обеспечения более быстрой и более эффективной передачи данных | |
| US10261938B1 (en) | Content preloading using predictive models | |
| CN109672757B (zh) | 文件访问方法及文件访问处理装置 | |
| US20080243536A1 (en) | Providing website hosting overage protection by storage on an independent data server | |
| US20160226998A1 (en) | Using resource timing data for server push | |
| WO2013143403A1 (zh) | 一种访问网站的方法和系统 | |
| JP2006526301A (ja) | ネットワークに対する知能型トラフィック管理システムおよびそれを利用した知能型トラフィック管理方法 | |
| US11936753B2 (en) | Graceful shutdown of supernodes in an internet proxy system | |
| EP4227829A1 (de) | Bahnabkratzen durch verwendung von proxies und anwendungen davon | |
| US20230018983A1 (en) | Traffic counting for proxy web scraping | |
| WO2017097092A1 (zh) | 缓存集群服务的处理方法及系统 | |
| US8745245B1 (en) | System and method for offline detection | |
| CN108664493B (zh) | 统计url是否有效的方法、装置、电子设备和存储介质 | |
| CN115883657A (zh) | 一种云盘服务加速调度的方法及系统 | |
| WO2023280593A1 (en) | Web scraping through use of proxies, and applications thereof | |
| US20240104145A1 (en) | Using a graph of redirects to identify multiple addresses representing a common web page | |
| TWI446772B (zh) | A cross - domain cookie access method, system and device | |
| CN114338686A (zh) | 一种cdn节点服务器的回源方法及装置 | |
| CN117527809A (zh) | 资源获取方法、装置、设备及存储介质 | |
| US20030177232A1 (en) | Load balancer based computer intrusion detection device | |
| LATENCY | INTERNATIONAL JOURNAL OF COMPUTER ENGINEERING & TECHNOLOGY (IJCET) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
| PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
| 17P | Request for examination filed |
Effective date: 20230502 |
|
| AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40089701 Country of ref document: HK |