TR2023019952A2

TR2023019952A2 - A SYSTEM THAT CREATES OUTPUT BY COLLECTING DATA FROM THE INTERNET ENVIRONMENT

Info

Publication number: TR2023019952A2
Application number: TR2023/019952
Authority: TR
Inventors: Ali̇han Di̇kel Umut; Yilmaz Mahmut; Cali Ir Emre; Si̇nan Ergen Mi̇that
Original assignee: Turkcell Teknoloji̇ Araştirma Ve Geli̇şti̇rme Anoni̇m Şi̇rketi̇
Filing date: 2023-12-30
Publication date: 2024-02-21

Abstract

Bu buluş, internet ortamında HTML altyapısında yer alan verilerin hızlı ve kolay bir şekilde toplanmasını sağlayarak veri çerçevesi (data frame) üzerinde kayıt altına alan ve/veya veri çerçevesi üzerinde yer alan verilerin kullanılarak istenilen formatta çıktı üretmesine olanak sağlanan bir sistem (1) ile ilgilidir.This invention relates to a system (1) that allows the data in the HTML infrastructure to be collected quickly and easily on the internet, recorded on a data frame and/or produces output in the desired format by using the data on the data frame. .

Description

TARIFNAME INTERNET ORTAMINDAN VERI TOPLAYARAK ÇIKTI OLUSTURAN BIR SISTEM Teknik Alan Bu bulus, internet ortaminda HTML altyapisinda yer alan verilerin hizli ve kolay bir sekilde toplanmasini saglayarak veri çerçevesi (data frame) üzerinde kayit altina alan ve/veya veri çerçevesi üzerinde yer alan verilerin kullanilarak istenilen formatta çikti üretmesine olanak saglanan bir sistem ile ilgilidir. Önceki Teknik Internet ortaminda bulunan yapilandirilmamis verilerin yapisal veride toplanmasi en önemli is kollarindan biridir. Teknolojinin gelismesiyle beraber dijital bilgi sayisinin giderek artmasi söz konusu yapisal verinin toplanmasini zorlastirmaktadir. Özellikle her bir internet sitesinin HTML altyapisi birbirinden farkli olmasi ve HTML altyapi içerisine yer alan farkli verilerin bulunmasi yapisal verinin toplanmasini da zorlastirmaktadir. DESCRIPTION OUTPUT BY COLLECTING DATA FROM THE INTERNET ENVIRONMENT A SYSTEM THAT CREATES Technical Field This invention enables the rapid and easy access of data contained in the HTML infrastructure on the internet. It is recorded on the data frame by ensuring that it is collected in some way. desired data using the data on the field and/or data frame. It is about a system that allows to produce output in the format. Prior Art Collecting unstructured data on the Internet into structured data It is one of the most important business lines. With the development of technology, digital information The increasing number of data requires the collection of the structural data in question. It makes it difficult. In particular, the HTML infrastructure of each website is different from each other. being different and having different data within the HTML infrastructure It also makes data collection difficult.

Bu sebeple teknigin bilinen durumunda HTML etiketlerinin YAML formatindaki bir konfigürasyon dosyasinda tutularak verilerin kolay ve hizli bir sekilde toplanmasini saglayan ve toplanan verilerin veri çerçevesi (data frame) üzerinde depolanmasini saglayarak istenilen formatta çiktinin olusturulmasini saglayan bir patent dokümaninda web arayicilarinin medya erisimini kisitlama sisteminden bahsedilmektedir. Söz konusu Birlesik Devletler patent dokümanina konu olan bulus bir medya talep mesajindan kaynak tanimlama bilgisinin elde edilmesi için programlanabilir devre, medya talep mesajiyla iliskili bir medya talep eden kaynagin kaynak tanimlama bilgisi göstergesi; kaynagi tanimlayan bilgilerin bir veri toplama tesisindeki kaynak kategorisi tanimlayicisiyla karsilastirilmasina dayali olarak medya talep eden kaynagin ortak üyeligi durumunu belirlemek; ortak üyelik durumu, medya talep eden kaynagin bir medya ile ortakligi olup olmadiginin göstergesidir saglayicinin sunucu sistemiyle iliskili izleyici ölçüm bilgilerini saglamasi; ve ortak üyeligi durumuna göre medya talep eden kaynaga bir medya yanit mesajina bir gecikme etiketi eklemek; gecikme etiketi, bir gecikme süresi periyoduna karsilik gelir. For this reason, in the state of the art, HTML tags in YAML format data can be easily and quickly stored in a configuration file. enabling the collection of data on the data frame. It is a device that enables the storage of data and the creation of output in the desired format. In the patent document, the system for restricting media access to web searchers is mentioned. is mentioned. The subject of the United States patent document in question The invention is for obtaining source identification information from a media request message. programmable circuit, a media requesting device associated with the media request message. resource cookie indicator of the resource; a piece of information identifying the source compared to the source category identifier at the data collection facility. to determine the associate membership status of the media requesting source; partner membership status, whether the source requesting media has a partnership with a media is an indicator of the provider's viewer measurement information associated with the server system. to ensure; and a media request to the resource requesting media based on its associate membership status. adding a delay tag to the response message; delay label, a delay time corresponds to the period.

Bulusun Kisa Açiklamasi Bu bulusun amaci, internet ortaminda HTML altyapisinda yer alan verilerin hizli ve kolay bir sekilde toplanmasini saglayarak veri çerçevesi (data frame) üzerinde kayit altina alan ve/veya veri çerçevesi üzerinde yer alan verilerin kullanilarak istenilen formatta çikti üretmesine olanak saglanan bir sistem gerçeklestirmektir. Brief Description of the Invention The purpose of this invention is to quickly access the data contained in the HTML infrastructure on the internet. and on the data frame by enabling easy collection. using the data recorded and/or located on the data frame The aim is to realize a system that allows it to produce output in the desired format.

Bulusun Ayrintili Açiklamasi Bu bulusun amacina ulasmak için gerçeklestirilen “Internet Ortaminda Veri Toplayarak Çikti Olusturan Bir Sistem” ekli sekilde gösterilmis olup, bu sekil; Sekil 1. Bulus konusu internet ortaminda veri toplayarak çikti olusturan bir sistemin sematik görünüsüdür. Detailed Description of the Invention "Data in the Internet Environment" was carried out to achieve the purpose of this invention. A System That Creates Output by Adding” is shown in the figure below; Figure 1. The subject of the invention is a system that creates output by collecting data on the internet. schematic view.

Sekilde yer alan parçalar tek tek numaralandirilmis olup, bu numaralarin karsiliklari asagida verilmistir: 1 . Sistem 2. Elektronik cihaz 3 . Uygulama 4. Sunucu . Veri tabani Internet ortamindaki verilerin toplanarak kayit altina alinmasini ve istenilen formatta çikti üretilmesini saglayan bulus konusu bir sistem (1); - üzerinde en azindan bir uygulama yürütmek üzere yapilandirilan en az bir elektronik cihaz (2), - elektronik cihaz (2) üzerinde yürütülmekte olup kullanicinin internet ortaminda toplanmasi istenilen veri bilgisinin, verilerin toplanmasini istedigi adres bilgisinin, etiket bilgisinin seklindeki verilerin girilerek bir konfigürasyon dosyasinin olusturulmasina olanak saglayan en az bir arayüz sunmak üzere yapilandirilan en az bir uygulama (3), - elektronik cihaz (2) ile iletisim kurarak uygulama (3) üzerinden kullanici tarafindan olusturulan konfigürasyon dosyasini almak, alinan konfigürasyon verisi üzerinde yer alan adres bilgisini kullanarak ilgili web sitelerinden kaynak dosyasini almak, konfigürasyon verisi üzerinde yer alan etiket bilgisini kullanarak kaynak dosyasinda ilgili etikete iliskin metin verilerinin belirlemek, belirlenen metin verilerini veri çerçevesi (data frame) olusturmak ve olusturulan veri çerçevesi (data frame) üzerinden istenilen formatta bir çiktinin olusturularak uygulama (3) üzerinden sunulmasini saglamak üzere yapilandirilan en az bir sunucu (4) içermektedir. The parts in the figure are numbered one by one, and the corresponding numbers are is given below: one . System 2. Electronic device 3 . APPLICATION 4. Server . Database Collecting and recording data on the Internet and An inventive system (1) that enables output to be produced in the following format; - at least one device configured to run at least one application on it electronic device (2), - It is carried out on the electronic device (2) and the user's internet access is The address where the data information desired to be collected in the environment is requested to be collected. A configuration is created by entering data in the form of information, label information. To provide at least one interface that allows the creation of at least one application configured (3), - the user via the application (3) by communicating with the electronic device (2) importing the configuration file created by, the received configuration data You can download the source file from the relevant websites using the address information on it. source, using the tag information on the configuration data. To determine the text data related to the relevant label in the file, to to create data frame (data frame) and create data frame (data frame) Application by creating an output in the desired format via frame) (3) At least one server (4) configured to provide Contains.

Bulus konusu sistemde (1) yer alan elektronik cihaz (2) teknigin bilinen durumunda yer alan herhangi bir uzak iletisim protokolü üzerinden sunucu (4) ile iletisim kuran ve kurulan iletisim üzerinden veri alisverisinin gerçeklestirilmesini saglayan cep telefonu, akilli telefon, masaüstü bilgisayar, tasinabilir bilgisayar veya tablet seklindeki bir cihazdir. Bulusun tercih edilen bir uygulamasinda elektronik cihaz (2) internet seklindeki bir veri yolu üzerinden sunucu (4) ile iletisim kurarak sunucu (4) ile veri alisverisinin gerçeklestirilmesini saglamak üzere yapilandirilmaktadir. The electronic device (2) included in the system (1) subject to the invention is in the state of the art. communicating with the server (4) via any remote communication protocol and mobile phone that enables data exchange via the established communication. phone, smartphone, desktop computer, portable computer or tablet It is a device in the form of. In a preferred embodiment of the invention, the electronic device (2) communicates with the server (4) over a data bus in the form of the internet. It is configured to enable data exchange with (4).

Bulus konusu sistemde (1) yer alan uygulama (3) elektronik cihaz (2) üzerinde yürütülmekte internet ortaminda toplanmasi istenilen veriye iliskin veri bilgisi, verinin toplanacagi internet sitesinin kaynak adresi (URL), kaynak adresinde verinin bulundugu etiket bilgisi seklindeki verilerin girilmesine olanak saglayan en az bir arayüz sunmak ve girilen bilgiler üzerinden bir konfigürasyon dosyasinin olusturulmasini saglamak üzere yapilandirilmaktadir. Bulusun tercih edilen uygulamasinda uygulama (3) konfigürasyon dosyasini YAML formatina uygun olacak sekilde olusturmakta olup konfigürasyon dosyasinin birden fazla veri bilgisi, verinin toplanacagi internet sitesinin kaynak adresi (URL), kaynak adresinde verinin bulundugu etiket bilgisi içermesini saglamak üzere yapilandirilmaktadir. The application (3) in the system (1) subject to the invention is installed on the electronic device (2). Data information regarding the data that is being carried out and is desired to be collected on the internet, The source address (URL) of the website where the data will be collected is at the source address which allows the entry of data in the form of tag information containing the data. To provide a simple interface and to create a configuration file based on the entered information. It is configured to ensure the creation of Find the preferred In the application, create the application (3) configuration file in accordance with YAML format. It is created in such a way that the configuration file has more than one data information, The source address (URL) of the website where the data will be collected is at the source address It is configured to ensure that the tag containing the data contains information.

Uygulama (3) elektronik cihaz (2) ile sunucu (4) arasinda kurulan iletisim üzerinden olusturulan konfigürasyon dosyasini sunucuya (4) göndermek üzere yapilandirilmaktadir. The application (3) through the communication established between the electronic device (2) and the server (4). to send the created configuration file to the server (4). is being configured.

Bulus konusu sistemde (1) yer alan sunucu (4) teknigin bilinen durumunda yer alan herhangi bir uzak iletisim protokolü üzerinden elektronik cihaz ile iletisim kurmak ve kurulan iletisim üzerinden uygulama (3) ile veri alisverisi gerçeklestirilmesini saglamak üzere yapilandirilmaktadir. Sunucu (4) tercih edilen bir uygulamada internet seklindeki bir veri yolu üzerinden elektronik cihaz (2) ile kurulan baglanti üzerinden veri alisverisi gerçeklestirmek üzere yapilandirilmaktadir. Sunucu (4) uygulama (3) üzerinde olusturulan konfigürasyon dosyasini almak ve konfigürasyon dosyasi içerisinde yer alan her bir internet adresine bir veri talebinin gönderilmesini saglamaktadir. Sunucu (4) olusturulan en azindan bir veri talebi üzerinden ilgili internet adresinin HTML kaynak kodunu almak ve kaydetmek üzere yapilandirilmaktadir. Sunucu (4) kaydedilen kaynak kodu üzerinde konfigürasyon dosyasinda yer alan etiket verisini kullanarak hedeflenen metin verisinin belirlenmesini saglamak üzere yapilandirilmaktadir. Sunucu (4) belirlenen metin verisini veri çerçevesi (data frame) üzerine islemek üzere yapilandirilmaktadir. Sunucu (4) veri çerçevesi (data fame) üzerinde toplanan verileri kullanarak uygulama (3) üzerinden kullanicinin talebine bagli olarak herhangi bir formatta çikti olusturmak ve uygulama (3) üzerinden sunulmasini saglamak üzere yapilandirilmaktadir. The server (4) in the system (1) that is the subject of the invention is in the known state of the art. communicating with an electronic device via any remote communication protocol and data exchange with the application (3) via the established communication. It is configured to provide Server (4) in a preferred embodiment connection established with the electronic device (2) via a data bus in the form of the internet It is configured to exchange data via . Server (4) to receive the configuration file created on the application (3) and A data request is made to each internet address in the configuration file. allows it to be sent. Server (4) created at least one data request Retrieving and saving the HTML source code of the relevant internet address via It is structured as follows. Server (4) on saved source code targeted text using label data in the configuration file It is configured to enable the determination of data. Server (4) designated To process text data onto a data frame is being configured. Collected on server (4) data frame (data fame) depending on the user's request via the application (3) using the data to create output in any format and have it presented through the application (3) It is configured to provide

Bulusun tercih edilen bir uygulamasinda sunucu (5) konfigürasyon dosyasinda yer alan verileri kullanarak konfigürasyon dosyasi içerisinde yer alan her bir internet adresine bir veri talebinin gönderilmesini, internet adresinin HTML kaynak kodunun alinmasi ve etiket verisi üzerinden kaynak kodunda yer alan metinleri belirleyerek veri çerçevesine (data frame) islenmesini daha önceden belirlenen bir periyotta tekrarlanmasini saglamak üzere yapilandirilmaktadir. In a preferred embodiment of the invention, the server (5) is included in the configuration file. each internet connection in the configuration file using field data. sending a data request to , the HTML source of the internet address Retrieving the code and extracting the texts in the source code via tag data by determining the processing into the data frame at a predetermined time. It is structured to ensure repetition in the period.

Bulusun tercih edilen bir uygulamasinda veri tabani (5) teknigin bilinen durumunda yer alan herhangi bir iletisim protokolü üzerinden sunucu (4) ile iletisim kurarak veri alisverisinin gerçeklestirilmesini saglamak ve kurulan iletisim üzerinden yönetilmek üzere yapilandirilmaktadir. Bulusun tercih edilen bir uygulamasinda veri tabani (5) internet seklinde bir veri yolu üzerinden sunucu (4) ile baglanti kurarak veri alisverisinin gerçeklestirilmesini saglamak ve kurulan baglanti üzerinden yönetilmek üzere yapilandirilmaktadir. Veri tabani (5) sunucu (4) ile kurulan iletisim üzerinden kayit altinda tutulan verilerin silinmesi, degistirilmesi ve/Veya güncellenmesine olanak saglamak üzere yapilandirilmaktadir. Veri tabani (5) sunucu (4) tarafindan olusturulan veri çerçevesinin (data frame) kayit altinda tutulmasini saglamak üzere yapilandirilmaktadir. In a preferred embodiment of the invention, the database (5) is in the state of the art. by communicating with the server (4) via any communication protocol available to ensure data exchange and through the established communication It is structured to be managed. In a preferred embodiment of the invention The database (5) is connected to the server (4) via a data bus in the form of the internet. to ensure data exchange by establishing the established connection It is configured to be managed through. Database (5) with server (4) Deleting and changing the data recorded through the established communication and/or is configured to allow it to be updated. Database (5) recording the data frame created by the server (4) It is configured to ensure retention.

Bulusun Sanayiye Uygulanmasi Bulus konusu sistem (1) ile, internet adresleri üzerinden veri toplanmasini isteyen bir kullanici tarafindan elektronik cihaz (2) üzerinde yürütülen uygulama (3) araciligiyla aramanin gerçeklestirilmesini saglayan konfigürasyon dosyasi olusturulmaktadir. Uygulama (3) konfigürasyon dosyasinin olusturulmasini saglayan toplanmasi istenilen veri bilgisi, verinin toplanacagi internet adresi bilgisi ve internet adresinde verinin toplanacagi etiket bilgisine iliskin verileri girmesine olanak saglamaktadir. Uygulama (3) olusturulan konfigürasyon dosyasini sunucuya (4) göndermektedir. Sunucu (4) uygulama (3) üzerinden aldigi konfigürasyon dosyasinda yer alan verileri kullanarak konfigürasyon dosyasi içerisinde yer alan her bir internet adresine bir veri talebinin gönderilmesini, internet adresinin HTML kaynak kodunun alinmasi ve etiket verisi üzerinden kaynak kodunda yer alan metinleri belirleyerek veri çerçevesine (data frame) islenmesini saglamaktadir. Application of the Invention to Industry With the system (1) subject to the invention, application (3) executed by a user on an electronic device (2) Configuration file that allows the search to be performed through is being created. The application (3) requires the creation of the configuration file. the data information to be collected, the internet address information where the data will be collected and enter data regarding the label information from which data will be collected on the internet address. It provides opportunities. Application (3) sends the created configuration file to the server. (4) sends. The configuration received by the server (4) via the application (3) in the configuration file using the data in the file. sending a data request to each internet address, the HTML of the internet address Retrieving the source code and using the tag data in the source code It determines the texts and enables them to be processed into the data frame.

Sunucu (4) veri çerçevesi üzerine islenen verilerin kullanicinin talebine bagli olarak herhangi bir formatta bir çikti olusturmakta ve/veya veri tabani (5) üzerinde kayit altina almaktadir. The data processed on the server (4) data frame is processed depending on the user's request. It creates an output in any format and/or records it in the database (5). takes it under.

Bu temel kavramlar etrafinda, bulus konusu “Internet Ortaminda Veri Toplayarak Çikti Olusturan Bir Sistem (1)” ile ilgili çok çesitli uygulamalarin gelistirilmesi mümkün olup, bulus burada açiklanan örneklerle sinirlandirilamaZ, esas olarak istemlerde belirtildigi gibidir. Around these basic concepts, the subject of the invention is "Collecting Data on the Internet" Development of a wide variety of applications related to “A System That Generates Output (1)” is possible, the invention cannot be limited to the examples explained here, mainly as stated in the claims.

Claims

STEMS enable collecting and recording data on the Internet and/or producing output in the desired format; - at least one electronic device (2) configured to run at least one application on it, - it is executed on the electronic device (2) and a configuration file is created by entering the data information that the user wants to collect on the internet, the address information where the data is wanted to be collected, and the data in the form of label information. - at least one application (3) configured to provide at least one interface that enables At least one server is configured to receive the source file, determine the text data related to the relevant label in the source file using the label information on the configuration data, create a data frame for the determined text data, and create an output in the desired format over the created data frame and present it through the application (3). The subject of the invention, characterized by (4), is an electronic device (a device in the form of a mobile phone, smartphone, desktop computer, portable computer or tablet) that communicates with the server (4) via any remote communication protocol and enables data exchange via the established communication. A system (1) as in Claim 1, characterized by 2). . A system (1) as in Claim 2, characterized by an electronic device (2) configured to enable data exchange with the server (4) by communicating with the server (4) over a data bus in the form of the Internet. . It is carried out on the electronic device (2) and is configured to provide at least one interface that allows the entry of data such as data information regarding the data desired to be collected on the internet, the source address of the website where the data will be collected, the tag information where the data is located at the source address, and to enable the creation of a configuration file based on the entered information. A system (1) as in any of the above claims, characterized by application (3). . It creates the configuration file in accordance with the YAML format and a system (1) as in Request 4, characterized by the application (3) configured to ensure that the configuration file contains more than one data information, the source address of the website where the data will be collected, and the tag information where the data is located at the source address. . . A system (1) as in any of the above claims, characterized by the application (3) configured to send the configuration file created through the communication established between the electronic device (2) and the server (4) to the server (4). . A system as in any of the above claims, characterized by a server (4) configured to communicate with the electronic device via any remote communication protocol and to enable data exchange with the application (3) over the established communication. A system (1) as in Request 7, characterized by the server (4) configured to exchange data over the connection established with the electronic device (2) over a data bus in the form of the Internet. A system as in any of the above claims, characterized by the server (4) configured to receive the configuration file created on the application (3) and to ensure that a data request is sent to each internet address in the configuration file. As in any of the above claims, characterized by the server (4) configured to receive and save the HTML source code. A system like (1). A system (1) as in any of the above claims, characterized by the server (4) configured to process the specified text data on the data frame. Configuration using the data contained in a Configuration file as in any of the above claims, characterized by the server (4) configured to create output in any format depending on the user's request via the application (3) using the data collected on the data frame and to ensure that it is presented via the application (3). The above claims are characterized by the server (5) configured to send a data request to each internet address in the file, to receive the HTML source code of the internet address, to determine the texts in the source code through the tag data and to process them into the data frame and to repeat them in a predetermined period. a system like any other (1). A system (1) as in any of the above claims, characterized by a database (5) configured to enable data exchange by communicating with the server (4) over any communication protocol and to be managed over the established communication. A system (1) as in Claim 15, characterized by a database (5) configured to enable data exchange by establishing a connection with the server (4) over a data bus in the form of the Internet and to be managed over the established connection. A system (1) as in Claim 15 or 16, characterized by a database (5) configured to enable the deletion, modification and/or updating of recorded data through the communication established with the server (4). A system (1) as in any of the claims 15 to 17, characterized by a database (5) configured to record the data frame created by the server (4).