WO2010118115A1 - Statut objectif et catégories de domaine - Google Patents

Statut objectif et catégories de domaine Download PDF

Info

Publication number
WO2010118115A1
WO2010118115A1 PCT/US2010/030211 US2010030211W WO2010118115A1 WO 2010118115 A1 WO2010118115 A1 WO 2010118115A1 US 2010030211 W US2010030211 W US 2010030211W WO 2010118115 A1 WO2010118115 A1 WO 2010118115A1
Authority
WO
WIPO (PCT)
Prior art keywords
page
domain
status
content
identifying
Prior art date
Application number
PCT/US2010/030211
Other languages
English (en)
Inventor
Leonard Orentas
Mark Kosters
Dave Moston
Karthik Shyamsunder
Srinivas Sunkara
Eduardo Jimenez
Phani Pattapu
Alex Holmes
Andrew Simpson
Original Assignee
Verisign, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US12/428,208 external-priority patent/US9292612B2/en
Application filed by Verisign, Inc. filed Critical Verisign, Inc.
Priority to RU2011144859/08A priority Critical patent/RU2011144859A/ru
Priority to CA2757833A priority patent/CA2757833C/fr
Priority to AU2010234488A priority patent/AU2010234488B2/en
Priority to CN201080024769.7A priority patent/CN102460417B/zh
Priority to EP10762358.9A priority patent/EP2417536A4/fr
Priority to JP2012504817A priority patent/JP2012523626A/ja
Priority to KR1020117026116A priority patent/KR101670700B1/ko
Priority to BRPI1014177A priority patent/BRPI1014177A2/pt
Publication of WO2010118115A1 publication Critical patent/WO2010118115A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/954—Navigation, e.g. using categorised browsing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
    • G06F16/9558—Details of hyperlinks; Management of linked annotations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis

Definitions

  • Search engines have made available information easier for individual users to find and access.
  • Typical search engines may include algorithms that aid in identifying relevant content based on a user's search terms. These may include, for example, taking into account an apparent popularity, based on web traffic, of a particular page with the requested content.
  • algorithms may include, for example, taking into account an apparent popularity, based on web traffic, of a particular page with the requested content.
  • DNS Domain Name System
  • IP Internet Protocol
  • TLD top-level domain
  • TLDs examples are ".com”; “.net”; “.org.” etc.
  • Each TLD supports second- level domains, listed immediately to the left of the TLD, e.g. the “example” level in 'Svww.example.com”.
  • Each second-level domain can include a number of third-level domains located immediately to the left of the second- level domain, e.g. the "www" level in "www.example.com”.
  • a single IP address e.g. a single server
  • Additional non-domain information may be included in a Uniform Resource Identifier ("URI") structure that includes the domain name.
  • URI Uniform Resource Identifier
  • a "path” part is a sequence of segments (conceptually similar to directories, though not necessarily representing them) separated by a forward slash ("/")- This information may be included immediately to the right of the domain name, such as the "blog" in "www.example.com/blog", and may be used by a server or other receiving device to identify and deliver specific content or run particular code.
  • Other examples of non-domain information may include queries and fragments, the specifics of which are understood by those of ordinary skill in the art and are not discussed in detail herein. Combinations of this information may be included in web page hyperlinks that navigate a user to another section of the same page or to another web page that may be part of the same, or a different, domain.
  • the responsibility for operating each TLD (including maintaining a registry of the second-level domains within the TLD) is delegated to a particular organization, known as a domain name registry ("registry").
  • the registry is primarily responsible for converting domain names to IP addresses (“resolving”) through DNS servers that maintain such information in large databases, and operating its top-level domain.
  • the present subject matter can provide benefits in improving the accessibility, and meaningfulness, of domain related content.
  • users may search on the attributes of a domain and, in a broader application, a profile may be constructed about a domain's web site using information that can be publically available via DNS servers and web sites.
  • domain features may include, but are not limited to, data content that is part of a web page, data content that is linked to a web page, data content that is part of the DNS resolution support architecture of the web page, etc.
  • associated information such as attributes and characteristics of the web site may be provided based on an expanded search based on the domain name.
  • the associated information may also provide improved search results based on attributes describing the domain and/or the content of the domain. From these profiles, improved analysis of subsets of the available content on the internet may be conducted that can also be used to improve the efficiency and productivity of individual users, business activities, and web content support services in general.
  • Embodiments of the present invention relate to systems and methods for analyzing a domain.
  • Embodiments may include identifying a status of a first page associated with a domain. Identifying a status may include attempting to resolve a domain name of the first page and, if the domain name fails to resolve, identifying the status of the domain as nonfunctional. If the domain name resolves, but content is not accessible, the status may be identified as non-accessible.
  • the content of the first page may then be retrieved.
  • a plurality of additional pages the domain may then be identified based on hyperlinks from the first page.
  • the status of the additional pages may then be identified, as was the status of the first page. That is, identifying a status of the additional pages may include attempting to resolve a domain name or network address of a first additional page, and, if the domain name or network address fails to resolve, the status of the first additional page may be identified as non-functional. If the domain name resolves, but content is not accessible, the status of the first additional page may be identified as non-accessible, and so on with other pages.
  • the hyperlinks may be prioritized based on the status and/or a comparison with predetermined data. For example, character strings within a URI may be predetermined as having particularly high or low significance. Thus, hyperlinks that contain such strings may be prioritized accordingly.
  • Content may be extracted from the first page and selected pages of the plurality of additional pages. Content may include the full range of domain features discussed herein. The selection of specific pages from among the additional pages may be based on their prioritization.
  • Content of the first page and additional pages may be processed through a set of predetermined or generated data to determine contextual matches among the content.
  • the predetermined or generated data will be referred to as a signature marker set.
  • a signature marker set should be understood as a table of data that correlates known elements with some other subject.
  • known terms may be associated with a frequency with which the term appears on a sampling of domain names. Comparison of a term found on a page against the term's domain frequency may be a first step in processing the term through the signature marker set. In embodiments, this can help to determine the domain profile more accurately by focusing on terms with a low domain frequency, which may be more discriminating than terms with a high domain frequency.
  • the signature marker set may also use techniques to reduce potential term ambiguity. For example, content may be compared against predetermined data that represents known associations of multiple data content. This can include known textual associations, known data-type associations, and various combinations of like data that is included in domain features. Therefore, the significance of a first object may be more accurately determined by identification of a second associated object in the domain features.
  • domain features can include data from, or linked to, the webpage itself, or DNS information such as the IP address or URI.
  • a purpose of the domain may be determined.
  • the content includes executable code embedded in the first page or the additional pages, and the signature marker set is configured to determine a contextual match within the executable code.
  • Embodiments may also include wherein the content includes image, video and audio information embedded in the first page or the additional pages, and the signature marker set is configured to determine a contextual match within the image, video and audio information.
  • the domain purpose may be used in a number of different ways, including searching, displaying, storing and/or sending the determined purpose of the domain.
  • Embodiments may include identifying and retrieving related domains based on the determined status and purpose of the domain.
  • Embodiments may include determining the purpose of the domain based on at least one of a determined domain name, a registrar reseller marker, an absence of a specified data type, an alternate publisher of the pages, a social community identifier, and a data type, in the content.
  • Embodiments may also include identifying one or more domains based on their purposes. A category of the identified domains may then be determined that is distinct from the respective purposes of the domains. Related domains may then be identified and retrieved based on the determined purpose and categories of the domain.
  • Embodiments may include following hyperlinks from the first page by traversing redirects to an https version of the web page.
  • Embodiments may also include receiving an input set of domains from a user; identifying attributes that are common amongst the input domains from among the contextually matched content; and outputting the identified attributes to the user.
  • Embodiments may include receiving an input set of purposes and/or categories from a user; identifying domains that correspond to the input set of purposes and/or categories; and outputting the identified domains to the user.
  • Embodiments may include performing the described methods in an iterative manner on a range of domains and compiling the results of a historical analysis of the relevant domain space.
  • Embodiments of the present invention can also include systems for implementing the described methods, as well as computer-readable storage medium coded with instructions for causing a computer to execute the described methods.
  • an electronic system including a processor, a memory and an electronic communication device may be configured to identify a status of a first page associated with a domain, and retrieve the first page, via the electronic communication device; identify a plurality of additional pages from the domain based on hyperlinks from the first page; identify a status of the additional pages via the electronic communication device; prioritize the hyperlinks based on a comparison with predetermined data; extract content from the first page and at least one page of the plurality of additional pages via the electronic communication device, the at least one page of the plurality of additional pages selected based on the prioritization; process the content through a signature marker set to determine a contextual match; determine a purpose of the domain according to the status of the first page, the status of the additional pages and results of the processing of the content; and at least one of display, store and send the determined purpose of the domain
  • Figure 1 depicts an exemplary system in accordance with an embodiment of the present invention
  • Figure 2 depicts an exemplary method in accordance with an embodiment of the present invention
  • Figure 3 depicts an exemplary web page, and related content, in accordance with an embodiment of the present invention.
  • Figure 4 graphically depicts an exemplary layered data resolution in accordance with an embodiment of the present invention
  • Figure 5 graphically depicts an exemplary process flow in accordance with an embodiment of the present invention.
  • Figure 6 depicts aspects of an exemplary method in accordance with an embodiment of the present invention.
  • Embodiments of the present invention can help a user, web content provider and/or registry understand how domain names are purposed.
  • a holistic view of on-line activity by industry segment can be constructed to improve understanding of the online environment and to complement other micro-analytic tools.
  • the systems and methods described herein may generate results that can provide domain site-purposed data.
  • the described systems and methods can derive attributes/characteristics about the web site accessible through the domain name.
  • the service can also allow users to search for domains based on criteria that may include several attributes describing the domain or the content on the domain.
  • Embodiments can use information contained in publicly available DNS Servers and content available in publicly available websites to build a profile about a domain's web site.
  • Embodiments can collect and provide different types of data associated with a domain in a layered fashion, for example as depicted in Figure 1.
  • Embodiments can determine a domain status by attempting to load a web page, and determining a domain purpose and domain categories by collecting textual, or other, data from a website and passing it through a signature marker set.
  • inventions of the present invention can also provide an improved directory of TLDs, such as the .com and .net TLDs, and an overview of their use for website hosting. For example, by determining domain profiles for all, or a subset, of the domains within a TLD, an improved directory can be created that classifies the relevant domains according to patterns that exist across the domain- space, rather than categories that are determined and applied according to individual web page content.
  • step SlOOO a status of a first page is determined.
  • Domain status generally relates to whether and how a domain resolves. For example, it can be determined whether there is a web server associated with the domain, and, if so, whether the web server can be contacted. Further information may include if there are any specifically identified web server errors. For example, a domain name is input and transmitted to a DNS server to attempt to resolve the domain name. If the domain name fails to resolve, the status of the domain may be determined as non-functional. If the domain name resolves, but content is not accessible, the status may be identified as non-accessible. Other status identifiers are also possible.
  • Domain Status may indicate the operational status of the associated website, e.g. active, HTTP error, etc.
  • the domain status may indicate whether a website is accessible from a particular domain, and if not, at what stage attempting to access that website fails. Access failures may include error codes assigned to the domain web site such as those indicated in Table 1.
  • step SlOlO a purpose of the domain is determined based on the determined status.
  • a status code such as a name server error, may be used to determine a general non-functional purpose, or a more specific non-functional purpose related to the error code.
  • the domain name resolves, but results in a redirection, this may be included in determining the status of the domain. For example, the status may be determined to be a "redirect" domain, without active content.
  • the web server for the domain name is successfully connected to, and does not result in being redirected, the content of a first page 500, an example of which is depicted in Fig. 3 , may then be retrieved in step S 1100.
  • step Sl 100 content from the first page, e.g. elements 502, 504, 506, and 508, is retrieved.
  • This content can be in various forms known to those of skill in the art, including, for example, text, multimedia, hyperlink, or other executable code.
  • elements 502, 504 and 506 may be web page buttons that activate hyperlinks to web pages 510, 520 and 530 respectively.
  • Element 508 may be text, image, or other multimedia data content.
  • the content may be used for at least two purposes described herein. One purpose can be to identify any other pages that are associated with the first page, for example, accessible through the first page, based on hyperlinks embedded in the first page, e.g. elements 502, 504, 506. This function may be performed in step S 1200.
  • a plurality of additional pages may be identified based on hyperlinks detected in the first page.
  • additional pages e.g. 510, 520, 530
  • exemplary pages 510, 520 and 530 share a same second level domain
  • other pages that are associated with, e.g. hyperlinked from, page 500 need not necessarily share the same domain.
  • Another purpose of the content can be to assist in determining the purpose of the domain, which is discussed further below.
  • the method continues with step S 1300.
  • step Sl 300 the status of the additional pages, e.g. 510, 520, 530, may be identified. Identifying a status of the additional pages may include attempting to resolve a domain name or network address of a first additional page, and, if the domain name or network address fails to resolve, the status of the first additional page may be identified as non-functional. If the domain name resolves, but content is not accessible, the status of the first additional page may be identified as functional, but non-accessible. Other status identifiers are also possible, as discussed above with respect to determining the status of the first page 500. The method continues with step S 1400.
  • the identified hyperlinks and associated additional web pages may be prioritized based on a comparison with predetermined data.
  • predetermined data including domain, and non-domain URI information
  • predetermined markers such as lists of significant keywords, strings or other values, that suggest desired content.
  • the structure of the hyperlink data may also be analyzed as part of the process to recognize patterns of information, e.g. particular nesting formats etc. As a result, a prioritized list of the identified hyperlinks may be produced.
  • a user may select the particular markers to use in order to customize the prioritization. This may include presenting the user with a list of markers that have been recognized in the identified hyperlink data, and allowing the user to select among these markers. To further assist the user, the user may be provided with optional additional information relevant to the recognized markers. For example, each of the recognized markers may be given a quantitative or qualitative value that assists the user in evaluating which markers are of most significance to the user or the domain in question. For example, the user may have a particular interest in certain identified markers, or a set of markers may appear to be more relevant based on a quantitative value of the presence of the marker in the hyperlink data. Thus, the user may select from the markers based on additional predetermined information relating to the particular markers that is presented to the user. This can provide benefits in identifying more pertinent webpages from which to extract content from as part of the domain profiling. The method continues with step S 1500.
  • step Sl 500 additional content may be extracted from the first page 500, such as content 508, if needed, and content may be extracted from selected pages of the plurality of additional pages.
  • the selection of specific pages from among the additional pages may be based on their prioritization. For example, if additional pages 510, 520 and 530 are prioritized such that page 530 is lowest in priority, content may only be gathered from pages 510 and 520.
  • page 530 could be designated as low priority based on the text string "mail" within the domain name. This may conserve resources and also lead to more accurate results in later steps by identifying the most relevant additional pages and content.
  • the type of content gathered may also be determined based on parameters of the signature marker set discussed further below. For example, text content may be all the data needed if the signature marker set is configured for text only. This discrimination may also provide efficiency in large-scale data gathering and analysis. If the signature marker set is configured to process multiple data types, this may improve the overall accuracy of the analysis.
  • hyperlink data from the first page may also include traversing redirects.
  • hyperlink data may not resolve directly to another web page, but instead may require at least one redirect.
  • additional pages may be indirectly linked or associated with the first page. This may also include traversing redirects to an https version of the web page. The method continues with step S 1600.
  • step S 1600 the content gathered from the first page and the additional pages may be processed through a signature marker set to determine contextual matches among the content.
  • a signature marker set may include linked data elements that provide a contextual match, or significance, for one of the data elements.
  • appropriate weight may be given to the significance of various portions of the content.
  • the term "Ford” is an automobile manufacturer, it is also a common surname, and, by itself, is of uncertain significance. This may result in improperly considering or disregarding this term as part of web page content. Term disambiguation may require "Ford” and other identified automotive terms to be within close proximity for the use of the term "Ford” to be considered related to the automotive class.
  • N-gram models are a type of probabilistic model for predicting the next item in a sequence. N-grams are used in various areas of statistical natural language processing and genetic sequence analysis, and may be applied to the present subject matter to refine the content processing described herein. For example, an n-gram model predicts x, based on x ;- i, x,. 2 ,..., x (-n . As described above with respect to the markers used for prioritizing the hyperlink data and additional pages, the method may allow the user to select the desired association from among identified associations. For example, the user may identify certain identified contextual matches as valid, and other contextual matches as invalid, or to be ignored.
  • the methods may assist the user in identifying the significance of contextual matches by providing additional quantitative or qualitative information in conjunction with the contextual matches.
  • automated methods may be used to assess the weight given to the contextual matches, as in the case of predetermined signature marker sets that are automatically applied to extracted content, or may assist the user in doing so, as in cases where the user is allowed to accept, weight or rejects identified contextual matches.
  • the particular types of data that are processed from the content are not limited to text or other specific data.
  • the content includes executable code embedded in the first page or the additional pages, and the signature marker set is configured to determine a contextual match within the executable code, or within results of the executable code.
  • the content includes image, video and/or audio information embedded in the first page or the additional pages, and the signature marker set is configured to determine a contextual match within any of the image, video and audio information.
  • there may be contextual matches amongst various audio data such as pattern recognition results, audio type, audio length, or any number of related attributes.
  • a large number of similarly patterned audio information may be a strong indicator of a particular type of web page, e.g. directed to a particular musical artist's content, and thus be of particular significance.
  • a Marker Performance Report may be included to provide improved performance.
  • a user may be presented with a qualitative assessment of the performance of individual markers.
  • the user may independently assess and rank the effectiveness of the markers. Therefore, the system may ensure that the markers used are effective by providing a report on the marker performance, and allowing for changes in the use of the markers.
  • the report may be presented to a user, who may implement changes, or the system may be automated to discard poorly performing markers such as those that fall below a certain threshold. After processing the content, the method continues with step S 1700.
  • a purpose of the domain may be determined based on the status of the first page, the statuses of any additional pages, and any results of the contextual matching.
  • Domain purpose can relate to the theme or overall meaning or intended use or uses of the content associated with the domain.
  • the purpose may reflect an apparent reason for which a domain is used. In the case of domains that fail to resolve, or return other access errors, the purpose may be that the domain is not being used, or is restricted.
  • Other identified purposes may include Pay Per Click (PPC) Parking, Purposed, Masked Redirection, Redirection, Blogging etc. as detailed in table 2 below.
  • PPC Pay Per Click
  • the results of the contextual matches can be particularly effective in determining specialized purposes of active domains.
  • the accuracy of this assessment is improved by the additional analysis of the prioritized additional pages.
  • functioning web sites may be assigned non-exclusive purpose codes such as those identified in Table 2 and any other suitable codes to designate other purposes.
  • Additional information may also be included in the purpose determination. For example, measures of domain traffic, such as statistics primarily from a DNS Traffic Processor, may be included to assess whether a web site is an authentic retail site. Domain key terms, such as heading, title and description, may also be given specified weights in addition to the contextual match weight factor described above. Domain properties, features, and/or functions indicating whether common features are present on the website and additional pages may be used in determining a purpose, such as blogging, retail, etc.
  • identifying what type of technology the websites use e.g. mail servers associated with the domain, cookies, multimedia, SSL certificates to secure data, etc.
  • can provide indications of purpose such as retail versus personal or other purposes.
  • Additional technical data beyond specific content may also be indicative, such as the average time to render the content on the relevant pages, geographical location of the web server, etc.
  • Embodiments may include determining the purpose of the domain based on at least one of a determined domain name, a registrar reseller marker, an absence of a specified data type, an alternate publisher of the pages, a social community identifier, and a data type, in the content. That is, the presence, or absence, of particular content, such as that listed above, may have independent significance in establishing an overall purpose of the domain. Examples may include indications of advertiser information, or the lack thereof, or datatypes such as image data, or the lack thereof.
  • a category of the domain may be determined that is distinct from the respective purposes of the domain.
  • a Domain Category may reflect a category associated with a business segment related to the content on the site. This may involve determining to which categories the content from the first page and the additional pages belongs. For example, the Domain Category may place the domain into a system of business classification taxonomy in conformance with the North American Industry Classification System.
  • Table 3 is a partial list of exemplary categories, which may include an assigned prioritization of the categories within the category code.
  • the information used to determine the Domain Category may include the content obtained from the first page and the additional pages, and may even be the same content used to determine the Domain Purpose. However, the significance attributed to various information may be different in each process. For example, as discussed above, the presence of "Ford" with other contextual corporate-related information may be used to determine a COMPANY/ORGANIZATION purpose of the web page. The presence of the same information "Ford” with other automotive information may also be used to determine a Domain Category in the automotive arts.
  • the Domain Classification is not limited to the content of a particular web page. Rather, the Domain Classification may be derived from the content of the first page and the prioritized additional pages, to arrive at a comprehensive classification of the domain. Embodiments may also include identifying one or more domains based on their purposes before performing the classification of the domains. This may provide advantages in accurately categorizing such domains into a classification, with or without user interaction.
  • Domain Purpose may include the purpose of the website associated with the domain, e.g., news, blog, parked, etc.
  • the Domain Purpose can be the primary purpose of a domain's website. This purpose can relate to the content of the particular website, and not necessarily the company owning it. That is, the corporate website of a given company can have a different purpose than, for example, a promotional blogging site for the company.
  • the exemplary method may proceed at step S 1900 with at least one of storing the results to a computer-readable storage medium, displaying the results to a user, or otherwise electronically communicating the results to a requestor over an electronic communication network.
  • Embodiments may further include performing targeted searches within the saved data, and/or performing the described methods in an iterative manner on a range of domains and compiling the results for historical analysis of the relevant domain space, as discussed further below.
  • Figure 4 depicts an example of a high-level process flow for a monthly iterative cycle in accordance with embodiments of the present invention. It consists generally of inputs 610, components 620, and outputs 630. Additional tools 690 may also be included. Such additional tools may assist a user in performing various selections/inputs 610, e.g. prioritizing identified hyperlinks, terms etc.
  • the inputs may include, for example, a zone file 612, a Signature Marker Set 614 and a Training Set 618.
  • a user may input, or select, items within the signature marker set 614 and/or the training set 618.
  • the Components may include a Web Crawler 622 and an Analyzer 624.
  • the Analyzer can be functionally split into to Categorization 626 and Classification 628 parts.
  • a registry 616 may maintain the various processes (inputs, components and output), and implement them on a designated portion of web space 650 in order to better understand relevant online activity by industry segment. For example, the registry may collect all the DNS traffic from a given server. Data for all sub-domain names for a TLD may be collected periodically and kept for a period of time in file storage 632. Reports 634 may also be generated based on a method including the training set 616 working in cooperation with the additional tools 690 to process information from the designated portion of web space 650. This can provide a DNS traffic value, and Domain Status, Purpose and Categories for each domain name over a period of time and provide access to information that has not been available in the past.
  • Embodiments may also include receiving as part of a query in step S2000 an input set of domains from a user for analysis.
  • the system may automatically identify attributes that are common amongst the input domains. These attributes may be from among the contextually matched content or other gathered information.
  • the results of this analysis may include outputting the identified common attributes to the user in step S2100. This capability may provide advantages such as being able to automatically identify common purpose-related attributes of domains, including the additional identified pages.
  • Embodiments may also include receiving as part of a query in step S2000 an input set of purposes and/or categories, e.g. from a user. Domains that correspond to the input set of purposes and/or categories, based on the described methods, may be identified, and the identified domains output to the user in step S2100. This capability may be advantageous in providing improved classification and identification of relevant information and/or domains that would not be possible merely from conventional methods of analyzing web page content.
  • the described methods can provide for prioritizing and parsing downloaded content, identify and collect significant attributes about the domain, including various categorizations of direct and indirect content, and can allow users or administrators to search based on the attributes of the domain.
  • the process may start at S600 and proceed to S610, where a target zone file is obtained.
  • the target zone may be a designated domain as discussed herein.
  • the method proceeds to S620.
  • the process attempts to connect to the target zone, which may be a designated domain. Different types of errors may result based on the attempt to connect to the target zone. For example, there may be a lame delegation, in which the DNS server is inoperable. If no DNS is identified, or other errors similar to those described above occur at S620, the method proceeds to S624, which is referred to as a "No DNS" error. Based on this determination, a status may be reported at S680.
  • a successful attempt may include a requested domain name being resolved by the DNS server to an IP address.
  • errors that can occur at the web server level that may prohibit obtaining content from the requested address.
  • a server error may be identified, such as the name server timing out, or otherwise indicating an error associated with the server in response to the request to connect to the IP address. If a server error is indicated at S634, the method may proceed to S680 where the error may be reported.
  • the method may proceed to S640- 648, where various responses may be received based on the attempt to crawl the designated web site or address. This may include an indication that the requested domain does not have an active web site, as in S640. There may also be an error with the web server responsible for the website that is indicated after the server is found, as in S642. The server or website may also restrict the ability of the web crawler to retrieve content, as in S644, or redirect the web crawler to another site, as in S646. These, and other responses that represent less than full access to the content of the web site, may be reported in S680.
  • the method may proceed to S648, where content from the web site is recognized as found. As described further herein, once content from a web site, or target domain, is found, the method may continue by accessing and analyzing the found content, as in S660. The results of the content retrieval and/or analysis may be reported in S680.
  • Embodiments of the present invention can include systems for implementing the described methods, as well as computer-readable storage medium coded with instructions for causing a computer to execute the described methods.
  • an electronic system 100 including a processor, a memory and an electronic communication device may be configured to identify a status of a first page associated with a domain via DNS server 140.
  • the system 100 may be represent a user computer system, wireless communication devices such as 120, 170, subnetworks such as 130, 190, a server, or any other network-capable device with the requisite functional capabilities.
  • System 100 may operate as part of, or separate from, a DNS server associated with a registry.
  • the first page may be received by the system 100 from server 150 via the electronic communication network 170, such as the internet.
  • the system 100 may identify a plurality of additional pages from the domain based on hyperlinks from the first page.
  • the system 100 may then identify a status of the additional pages via the DNS server 140.
  • the system 100 may also prioritize the hyperlinks based on a comparison with predetermined data, as discussed above.
  • Content from the first page and at least one page of the plurality of additional pages may be extracted via the servers 150, 160, for example from web-hosting servers.
  • the system 100 may then process the content through a signature marker set, that is stored on or otherwise electronically accessed by the system 100, to determine a contextual match.
  • the system 100 can further determine a purpose of the domain according to the status of the first page, the status of the additional pages and results of the processing the content.
  • the system 100 may receive the various user inputs described above, such as selecting identified web pages from which to extract data content, selecting terms, and The results of the described processing performed by system 100 can be displayed, stored and/or sent according to known techniques.
  • the system 100 includes any number of processors (not shown) that are coupled to storage devices including a first storage (not shown, typically a random access memory, or "RAM”), second storage (not shown, typically a read only memory, or "ROM"). Both of these storage devices may include any suitable type of the computer-readable media described and/or mentioned above.
  • a mass storage device (not shown) may also be used to store programs, data and the like and is typically a secondary storage medium, such as a hard disk that is slower than primary storage. It will be appreciated that the information retained within the mass storage device, may, in appropriate cases, be incorporated in standard manner as part of primary storage as virtual memory.
  • a specific mass storage device such as a CD- ROM may also pass data uni-directionally to the processor.
  • the system 100 may also include an interface that includes one or more input/output devices such as such as video monitors, track balls, mice 104, keyboards, microphones, touch-sensitive displays, transducer card readers, magnetic or paper tape readers, tablets, styluses, voice or handwriting recognizers, or other known input devices, including other computers 110.
  • the system 100 may be coupled to a computer or other electronic communication network 170, 180 using a network connection as shown generally at 101.
  • the network can connect various wired, optical, electronic and other known networks to exchange information among computers 110, servers 160, wireless communication devices 120, 170 and sub-networks 190, 130.
  • system 100 and the processor therein may receive information from the network, or may output information to the network in the course of performing the above-described method steps.
  • the above-described devices and materials will be familiar to those of skill in the computer hardware and software arts and need not be individually or exhaustively depicted to be understood by those of skill in the art.
  • the hardware elements described above may be configured (usually temporarily) to act as one or more modules for performing the operations described above.
  • embodiments of the present invention further include computer- readable storage media that include program instructions for performing various computer- implemented operations.
  • the media may also include, alone or in combination with the program instructions, data files, data structures, tables, and the like.
  • the media and program instructions may be those specially designed and constructed for the purposes of the present subject matter, or they may be of the kind available to those having skill in the computer software arts.
  • Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto- optical media such as floptical disks; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM) and random access memory (RAM).
  • Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Information Transfer Between Computers (AREA)
  • Medicines That Contain Protein Lipid Enzymes And Other Medicines (AREA)

Abstract

L'invention porte sur des systèmes et des procédés de création et d'utilisation d'un profil de domaine, qui comprennent l'identification d'un statut d'une première page associée à un domaine. La première page est extraite et des pages supplémentaires du domaine sont identifiées sur la base d'hyperliens provenant de la première page. Le statut des pages supplémentaires est identifié et les hyperliens sont hiérarchisés sur la base du statut et/ou d'une comparaison avec des données prédéterminées. Un contenu est extrait de la première page et de pages sélectionnées parmi les pages supplémentaires. Les pages supplémentaires spécifiques peuvent être sélectionnées sur la base de la hiérarchisation. Le contenu extrait peut être traité par un ensemble de marqueurs de signature pour déterminer une correspondance contextuelle. Un objectif du domaine est déterminé selon le statut de la première page, le statut des pages supplémentaires et des résultats du traitement du contenu. Le profil de domaine peut être affiché, stocké, envoyé et/ou recherché pour identifier des sites Internet ou des attributs d'intérêt.
PCT/US2010/030211 2009-04-07 2010-04-07 Statut objectif et catégories de domaine WO2010118115A1 (fr)

Priority Applications (8)

Application Number Priority Date Filing Date Title
RU2011144859/08A RU2011144859A (ru) 2009-04-07 2010-04-07 Статус, назначение и категории доменов
CA2757833A CA2757833C (fr) 2009-04-07 2010-04-07 Statut objectif et categories de domaine
AU2010234488A AU2010234488B2 (en) 2009-04-07 2010-04-07 Domain status, purpose and categories
CN201080024769.7A CN102460417B (zh) 2009-04-07 2010-04-07 域状态、作用和种类
EP10762358.9A EP2417536A4 (fr) 2009-04-07 2010-04-07 Statut objectif et catégories de domaine
JP2012504817A JP2012523626A (ja) 2009-04-07 2010-04-07 ドメイン状態、目的、およびカテゴリ
KR1020117026116A KR101670700B1 (ko) 2009-04-07 2010-04-07 도메인 상태, 목적 및 범주
BRPI1014177A BRPI1014177A2 (pt) 2009-04-07 2010-04-07 estado de domínio, propósito e categorias

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US16752809P 2009-04-07 2009-04-07
US16752109P 2009-04-07 2009-04-07
US16752309P 2009-04-07 2009-04-07
US61/167,528 2009-04-07
US61/167,521 2009-04-07
US61/167,523 2009-04-07
US12/428,208 2009-04-22
US12/428,208 US9292612B2 (en) 2009-04-22 2009-04-22 Internet profile service

Publications (1)

Publication Number Publication Date
WO2010118115A1 true WO2010118115A1 (fr) 2010-10-14

Family

ID=42936554

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2010/030211 WO2010118115A1 (fr) 2009-04-07 2010-04-07 Statut objectif et catégories de domaine

Country Status (9)

Country Link
EP (1) EP2417536A4 (fr)
JP (1) JP2012523626A (fr)
KR (1) KR101670700B1 (fr)
CN (1) CN102460417B (fr)
AU (1) AU2010234488B2 (fr)
BR (1) BRPI1014177A2 (fr)
CA (1) CA2757833C (fr)
RU (1) RU2011144859A (fr)
WO (1) WO2010118115A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10341290B2 (en) 2013-04-12 2019-07-02 Tencent Technology (Shenzhen) Company Limited Method and system for presenting recommendation information
US10606821B1 (en) 2016-08-23 2020-03-31 Microsoft Technology Licensing, Llc Applicant tracking system integration

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105243073A (zh) * 2014-07-11 2016-01-13 北京金山安全软件有限公司 一种书签访问方法、装置及终端
CN111291284A (zh) * 2018-12-10 2020-06-16 北京京东金融科技控股有限公司 一种多级页面的重定向方法和装置
CN110211581B (zh) * 2019-05-16 2021-04-20 济南市疾病预防控制中心 一种实验室自动语音识别记录标识系统及方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050076222A1 (en) * 2003-09-22 2005-04-07 Secure Data In Motion, Inc. System for detecting spoofed hyperlinks
US20060282883A1 (en) * 2001-03-28 2006-12-14 Geotrust, Inc. Web site identity assurance
US20080028443A1 (en) * 2004-10-29 2008-01-31 The Go Daddy Group, Inc. Domain name related reputation and secure certificates
US20080163369A1 (en) * 2006-12-28 2008-07-03 Ming-Tai Allen Chang Dynamic phishing detection methods and apparatus

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7565402B2 (en) * 2002-01-05 2009-07-21 Eric Schneider Sitemap access method, product, and apparatus
US20030225876A1 (en) * 2002-05-31 2003-12-04 Peter Oliver Method and apparatus for graphically depicting network performance and connectivity
CN100432996C (zh) * 2004-12-07 2008-11-12 国际商业机器公司 基于网页页面布局提取网页核心内容的系统、方法
US7475069B2 (en) * 2006-03-29 2009-01-06 International Business Machines Corporation System and method for prioritizing websites during a webcrawling process
US20080082662A1 (en) * 2006-05-19 2008-04-03 Richard Dandliker Method and apparatus for controlling access to network resources based on reputation

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060282883A1 (en) * 2001-03-28 2006-12-14 Geotrust, Inc. Web site identity assurance
US20050076222A1 (en) * 2003-09-22 2005-04-07 Secure Data In Motion, Inc. System for detecting spoofed hyperlinks
US20080028443A1 (en) * 2004-10-29 2008-01-31 The Go Daddy Group, Inc. Domain name related reputation and secure certificates
US20080163369A1 (en) * 2006-12-28 2008-07-03 Ming-Tai Allen Chang Dynamic phishing detection methods and apparatus

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
PAL ET AL.: "Supporting Safe Content-Inspection of Web Traffic", CROSSTALK, 1 September 2008 (2008-09-01), pages 19 - 23, XP008168521, Retrieved from the Internet <URL:http://www.dist-systems.bbn.com/papers/2008/CrossTalk/0809PalAtighetchi.pdf> [retrieved on 20100523] *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10341290B2 (en) 2013-04-12 2019-07-02 Tencent Technology (Shenzhen) Company Limited Method and system for presenting recommendation information
US10606821B1 (en) 2016-08-23 2020-03-31 Microsoft Technology Licensing, Llc Applicant tracking system integration

Also Published As

Publication number Publication date
CN102460417A (zh) 2012-05-16
CN102460417B (zh) 2015-07-29
JP2012523626A (ja) 2012-10-04
EP2417536A1 (fr) 2012-02-15
AU2010234488A1 (en) 2011-11-17
BRPI1014177A2 (pt) 2016-04-05
CA2757833C (fr) 2018-09-18
AU2010234488B2 (en) 2015-01-22
KR20120005012A (ko) 2012-01-13
KR101670700B1 (ko) 2016-10-31
RU2011144859A (ru) 2013-05-20
EP2417536A4 (fr) 2016-08-31
CA2757833A1 (fr) 2010-10-14

Similar Documents

Publication Publication Date Title
US9742723B2 (en) Internet profile service
US8949251B2 (en) System for and method of identifying closely matching textual identifiers, such as domain names
US8255386B1 (en) Selection of documents to place in search index
US6567800B1 (en) System and method for searching information stored on a network
EP2395443B1 (fr) Réecriture de requête avec détection d&#39;entité
KR101063364B1 (ko) 웹 크롤링 프로세스 동안 웹 사이트에 우선순위를 부여하기위한 시스템 및 방법
US7974970B2 (en) Detection of undesirable web pages
JP5292250B2 (ja) 文書検索装置、文書検索方法、文書検索プログラム
US20050234877A1 (en) System and method for searching using a temporal dimension
US20060167862A1 (en) Method and apparatus for utilizing user feedback to improve signifier mapping
US7962462B1 (en) Deriving and using document and site quality signals from search query streams
KR100892842B1 (ko) 사용자 중심 정보탐색 방법 및 시스템
US9367638B2 (en) Surfacing actions from social data
EP1993045A1 (fr) Système de récupération de document électronique
KR20020025142A (ko) 키워드 광고 서비스를 위한 키워드 추천 시스템 및 방법
CA2757833C (fr) Statut objectif et categories de domaine
JP4761460B2 (ja) 検索装置による情報検索方法、情報検索装置及び情報検索処理プログラム
US20130031075A1 (en) Action-based deeplinks for search results
WO2001055909A1 (fr) Systeme et procede de gestion et d&#39;analyse de signets
WO2009054611A1 (fr) Système et procédé pour gérer une carte d&#39;informations
Barifah et al. Exploring usage patterns of a large-scale digital library
KR20060115488A (ko) 웹 브라우저의 즐겨찾기 리스트를 이용한 개인화 검색 방법및 검색 서버
KR101020895B1 (ko) 지역 정보 검색 결과 제공 방법 및 시스템
CN101923548A (zh) 一种互联网信息搜索方法及一种搜索引擎
KR100909561B1 (ko) 지역 정보 검색 결과 제공 시스템

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 201080024769.7

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10762358

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2012504817

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 2757833

Country of ref document: CA

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 8338/DELNP/2011

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 2010762358

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 20117026116

Country of ref document: KR

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2011144859

Country of ref document: RU

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2010234488

Country of ref document: AU

Date of ref document: 20100407

Kind code of ref document: A

REG Reference to national code

Ref country code: BR

Ref legal event code: B01A

Ref document number: PI1014177

Country of ref document: BR

ENP Entry into the national phase

Ref document number: PI1014177

Country of ref document: BR

Kind code of ref document: A2

Effective date: 20111007