US20160321280A2 - System for automatically generating wrapper for entire websites - Google Patents
System for automatically generating wrapper for entire websites Download PDFInfo
- Publication number
- US20160321280A2 US20160321280A2 US14/871,027 US201514871027A US2016321280A2 US 20160321280 A2 US20160321280 A2 US 20160321280A2 US 201514871027 A US201514871027 A US 201514871027A US 2016321280 A2 US2016321280 A2 US 2016321280A2
- Authority
- US
- United States
- Prior art keywords
- page
- module
- action
- transducers
- website
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000009471 action Effects 0.000 claims abstract description 41
- 230000004044 response Effects 0.000 claims abstract description 3
- 238000004458 analytical method Methods 0.000 claims description 25
- 238000013075 data extraction Methods 0.000 claims description 5
- 230000000007 visual effect Effects 0.000 claims description 4
- 230000001419 dependent effect Effects 0.000 claims description 3
- 238000000605 extraction Methods 0.000 description 4
- 230000003993 interaction Effects 0.000 description 4
- 230000006399 behavior Effects 0.000 description 3
- 238000000034 method Methods 0.000 description 3
- 239000008186 active pharmaceutical agent Substances 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000001427 coherent effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000009193 crawling Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012512 characterization method Methods 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 239000011022 opal Substances 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
-
- G06F17/3089—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G06F17/241—
-
- G06F17/243—
-
- G06F17/30595—
-
- G06F17/30864—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/169—Annotation, e.g. comment data or footnotes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/174—Form filling; Merging
Definitions
- the present application relates to a system for automatically generating a wrapper for an entire website, where the wrapper characterises the structure of the website.
- a system for automatically generating a wrapper for a website the wrapper characterising the structure of the website, the system having a plurality of functional elements, including at least one annotation module to classify components of a page and generate an annotated page, a page classification module to identify functional and informational components of an annotated page, and an action module to identify an action to be taken to further navigate the website, wherein at least one of the annotation module, page classification module and action module is operable in response to a plurality of domain-specific rules, where a domain is understood as a conceptual domain such as real estate, used cars, or electronics.
- the annotation module may be responsive to page structure components and text components to identify a domain-specific datatype and associated data value within the page.
- the page classification module may comprise at least one analysis module and a first control element, the first control element being responsive to the classified page to select an analysis module.
- the at least one analysis module may comprise one or more of a form analysis module, and a results page analysis module.
- the page classification module may include a plurality of analysis modules and the first control module may be able to successively select a plurality of analysis modules.
- the first control element may be operable to pass control to the action module.
- the action module may be operable to select a navigation step and cause a further page of the website to be loaded.
- the action module may be operable to select an action comprising one or more of; selecting a link, invoking a form completion module, performing a crawler action or returning to a previous page.
- the action which may be selected by the action module may be dependent on the result of the operation of the page classification module.
- a plurality of the modules may comprise relational transducers, wherein each relational transducer embodies a set of domain-specific rules defining a relationship between input data and output data.
- the plurality of relational transducers may be disposed in a network, such that the execution order of the network of relational transducers varies depending on the page to be classified.
- the system may further comprise a control transducer, the control transducer being operable to determine control flow within the network.
- the system may further comprise a visual block analysis module to identify graphical elements of the page.
- the system may comprise a data extraction system operable to receive the wrapper and navigate the website in accordance with the wrapper to extract data from the website.
- FIG. 1 is an outline illustration of a system embodying the present invention
- FIG. 2 is a more detailed illustration of the system of FIG. 1 .
- FIG. 3 is a diagram illustrating operation of a part of the system of FIGS. 2 .
- FIG. 4 is a diagram illustrating operation of a further part of the system of FIG. 2 .
- relational transducers refers to any computational element, however implemented, which is defined by particular input and output relations. Relational transducers can be self-contained, in that they interact through a shared memory and have no knowledge of each other. Because transducers are self-contained and are directed to a specific set of relations between inputs and outputs, the individual relational transducers can be relatively simple. Advantageously, in the present example the relational transducers are generally written as Datalog programs. Relational transducers, arranged into a transducer network, form the basic components of the system described herein. The transducer network provides integration and communication between the transducers in a way that represents an ideal trade-off among the system's primary integration goals:
- Transducers communicate through a transactional shared memory and have no other knowledge of each other;
- Transducers can be executed repeatedly, possibly continuing previously suspended computation, e.g., if new input data has become available.
- Transducers are generally Datalog programs with controlled value invention, retaining Datalog's polynomial data complexity but with significant performance benefits.
- the relational transducers are resumable: They can yield processing and may be called again on the same page, returning new facts. Resumption is monotone, i.e., additional calls to a transducer may produce additional output, but never retract previously derived facts.
- Resumable transducers are further distinguished into state- and input-driven transducers. State-driven transducers may produce new facts even if called with the same input, but maintain state between calls. These are typically transducers that iterate over some collection, e.g., all links on a page, and maintain the position in the iteration in their state relations. Input-driven transducers may also be called multiple times, but may only produce new data, if additional input is provided. Typically, these transducers are called only once or twice per page. Resumable transducers are exhausted, if further calls yield no new data.
- Relational transducers described herein are each one of three general types:
- GFSTs Stateful guarded finite state transducers encode finite state transducers where transitions are guarded by first-order formulas. All exploration decisions are delegated to such transducers, which are typically state-driven resumable. The transducers used for form filling are examples of such transducers.
- a system embodying the present invention provides a framework to enable the characterisation of a website, that is generating a wrapper which defines a record and attribute structure for the website.
- the wrapper effectively amounts to a program which specifies actions to navigate through the pages of a target website and to select elements identified as records.
- the framework can be adapted to search websites in different domains.
- the system can be used to search websites relating to real estate, cars, or any other product, or indeed any other structured website with multiple records as may be desired.
- the invention is particularly described with reference to sites encoded with HTML, it will be apparent that the system can be adapted to read structured sites encoded using any appropriate language or structures.
- a system embodying the invention is shown in outline in FIG. 1 at 10 .
- the system 10 comprises an initialisation module 11 , an annotation module 12 , an action module 13 , an implementing module 14 , and a shared memory 15 .
- a control module is generally shown at 16 .
- the control module 16 comprises a relational transducer as discussed above.
- the control transducer 16 is a state-driven resumable transducer for determining control flow and dependencies in the system. If there are multiple transducers which are ready to execute, the control flow is determined by priorities dynamically computed by the control transducer 16 and transducers are executed in order of their priority.
- the control instructions from the control transducer 16 are illustrated by lines 16 , 16 b, 16 c.
- the shared memory 15 is provided such that each of the modules is able to read from and write to the memory 15 as shown by arrow connections 15 a, 15 b, 15 c.
- the shared memory 15 holds a database of the rules used by the transducer network and provides the shared memory through which the transducers interact. When a decision is taken that the site exploration is complete, a wrapper 17 is generated as an output.
- FIG. 2 a more detailed illustration of the system of FIG. 1 shown including a number of individual functional elements which provide each of the respective modules, 11 , 12 , 13 , 14 and which may be appropriately implemented using relational transducers.
- the shared memory 15 and the links thereto are omitted for clarity.
- the control transducer is generally shown at 16 , and individual elements are shown at 16 ′, 16 ′′ as examples of where the control transducer 16 serves to control the processing flow of the system 10 .
- element 110 loads the next page of the website to be examined, whether the initial top-level site URL, or a URL for a subsequent page as described below.
- success of this operation is checked and if not, the page failure element is invoked.
- the loaded HTML document is parsed into a Document Object Model (“DOM”) to facilitate subsequent operations.
- annotation module 12 the received and parsed page is annotated using an annotator element 120 .
- annotation element 120 uses a plurality of labelled and named entity recognisers (“LNER”s) to identify domain-specific datatypes and values within the page. Using rules derived from the relevant domain knowledge, the LNER serve to identify entities of interest, taking into account the HTML structure and CSS formatting, in addition to the text components of the page.
- LNER entity recognisers
- element 200 is invoked in the event of failure.
- a probing element may be invoked to assess the page structure if necessary.
- Visual block identification module 124 identifies graphical and visual elements of the page, such as frames and images.
- the page On completion of the annotation and identification of the various components of the page, the page then is passed to page classification module 13 .
- a plurality of modules 131 , 132 , 133 are shown which may be invoked by first control element 16 ′ of the control transducer 16 , to identify or interact with functional components of the page (such as forms) or information components of the page (such as the results records of interest).
- Each of the analysis modules 131 , 132 , 133 has an associated guard 131 a, 132 a, 133 a which may implement guard rules for the respective analysis module, to indicate whether the respective module is ready or appropriate for execution.
- module 131 is selected when a form identified on the page and, and analyses the form to identify acceptable inputs and resultant behaviour. If result pages identified, one of modules 132 , 133 is run to identify regularities in the structure of a page, typically resulting from data-publishing templates.
- module 132 referred to as AMBER is realised as a phenomenological transducer that encodes domain-independent rules for detecting patterns on websites. These patterns are combined with template discovery, that is the detection of regularities in the structure of a page.
- AMBER's transducer is input-driven resumable: it is called once per page in a sequence of result pages (typically connected by pagination links), each time refining the model of the template underlying the result pages.
- AMBER uses the concept of a pivot attribute.
- a pivot attribute such as PRICE, is a mandatory attributes of an easy to detect type. The system locates these pivot attributes to discard regular structures with irrelevant data or irregular noise in otherwise regular structures, such as advertisements interspersed among records. This increases the accuracy of record and attribute identification compared to existing template discovery approaches. In most product domains, PRICE or product identifier are ideal pivot attributes. In domains with no regular attributes, presentational attributes such as images or details page links may be chosen as pivot attributes. Textual analysis of the results from analysis modules 132 , 133 is then performed by text analysis module 134 .
- actions for further navigation of the website are selected under the control of second control element 16 ′′.
- the action selected may include filling an identified form as shown at 151 , or performing a crawler action as shown in 152 .
- FIG. 3 shows an alternative view of the operation of module 14 , including the options of navigating to a further linked page, returning to a previous page, and identifying an iframe and extracting the contents.
- next link action generator requires the presence of a list of pagination links, as well as set of records (indicating that this is a listings page). Typically only two or three of these have their guards and dependencies satisfied at this stage. Priorities are used to determine which to run: For example, the crawler 152 is always last and iFrames 155 have priority if they are covering a substantial portion of the page. In many cases, if one of the action generators fails (e.g., if it has already attempted to click on all “next links”) some of the other action generators are called next (indicated by the red dashed arrows).
- Back-tracking is essential to the exploration of sites, as their exploration often requires multiple alternative paths. When choosing a path for example by filling a form, the system cannot know yet whether that path will indeed lead to relevant data. Back-tracking is dynamic and based on the transducer dependencies and guards: To back-track, the system goes back over the sequence of executed transducers until it reaches a point where either a transducer is resumable and not yet exhausted, or a prioritised choice of transducers exists with some transducers not yet executed.
- control transducer also allows the specification of explicit back-tracking logic that overwrites the default case and is used, for example., for back actions in the browser.
- form filling is an example of a network of state driven resumable transducers that encode guarded finite state transducers and are called repeatedly to generate queries for a given form. The form is analysed by a phenomenological transducer. The form filling is domain aware, as it uses domain knowledge to generate fillings and to react to feedback from filling (e.g., error messages). Each time the control transducer is executed the network's logical clock is advanced by one step.
- the crawler action element 152 is shown broken down into a number of relational transducers 152 a to 152 f forming a sub-network of relational transducers.). This process is split into six transducers, including the browser interaction transducer 152 e that executes actions and the modification classifier 152 f that analyses how the browser's content has changed due to an action. It returns classifications, such as “new page”, “major page change”, “new form fields”, “new window”, or “alert”.
- the other four transducers are guarded finite state transducer responsible for selecting fields to be filled ( 152 a ); selecting the assumed “behavior” for each field, e.g., that a text field is to be treated as an autocomplete ( 152 b ); selecting the specific value for each field ( 152 c ); and iterating over all fields, feeding the browser interaction transducer with the filling for each field one by one ( 152 d ).
- These four transducers are all resumable and chained: If there are no more fields to be filled in the current iteration, the filling iteration yields control back to the value selection.
- the value selection may return the same sequence of fields with different values or, if there are no additional value combinations to try, may fail and yield control to the behaviour selection.
- execution continues either with the field iteration (typically if there is no change, arrow 1), with the field set selection (if the state of form fields changes, arrow 2), or with the end of the filling phase, for example if a new page is reached (arrow 3).
- generator 160 stores the relevant information to be included in the wrapper. Insert F here If the identified action is successfully performed at step 161 and is checked at step 162 the process repeats.
- the wrapper accumulates information about all identified result pages and the navigation paths leading to them and integrates that information into a coherent wrapper program. It is input driven resumable, called once per page, but accumulating the wrapper information over all the calls for one site. Within a result page sequence, it combines the collected information into a coherent wrapper for the underlying template of these pages, that is likely applicable also to any other page in the sequence.
- the system comprises a plurality of relational transducers which form a synchronised transducer network.
- the network is referred to as being synchronised as its control flow is determined by a central controller, itself a relational transducer.
- a transducer network is a set of transducers with a transactional shared memory which serves as input and output for the transducers. The execution is controlled by the control transducer and a specific area in the memory is reserved for communication between controller and transducers.
- the network is self-adaptive, as the control flow is dynamically determined from transducer dependencies and their guard rules.
- the record identification transducer which has similar dependencies, but is guarded by requiring the presence of pivot attribute annotations. If both guards are satisfied, the transducers may be executed in parallel. Transducers that yield an interaction with the browser cannot be executed in parallel, but must be sequentialised, as parallel access may break server state or Javascript execution. Therefore, for the selection and execution of actions, explicit priorities are used to sequentialise the actions and to prioritise actions with the highest estimated probability to lead to relevant data.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
- This application claims the benefit and priority of U.S. Provisional Application 62/057,395 filed on Sep. 30, 2014, the contents of which are hereby incorporated by reference.
- The present application relates to a system for automatically generating a wrapper for an entire website, where the wrapper characterises the structure of the website.
- The identification and retrieval of data from websites is a difficult and pressing issue. While techniques have been developed for ‘crawling’ the web to classify and index websites, and make the information available for searching and retrieval, extraction of complete data from a site is problematical. It is known for websites to offer APIs to facilitate data extraction, but this is far from universal. Automatic complete extraction of data from a site without a suitable API is problematic as sites are not structured consistently, and page elements such as forms, side bars, and navigation menus can be difficult to correctly identify or interact with. Supervised systems are known, in which a user navigates to a site and identifies relevant data, which the system then uses to direct data extraction, but these are time-consuming and not scalable. Automatic full-site extraction has so far only been successfully used in setting with limited structures, such as title and body extraction from news articles or search engine results. For extracting highly structured data, these approaches are unsuitable.
- According to the present invention there is provided a system for automatically generating a wrapper for a website, the wrapper characterising the structure of the website, the system having a plurality of functional elements, including at least one annotation module to classify components of a page and generate an annotated page, a page classification module to identify functional and informational components of an annotated page, and an action module to identify an action to be taken to further navigate the website, wherein at least one of the annotation module, page classification module and action module is operable in response to a plurality of domain-specific rules, where a domain is understood as a conceptual domain such as real estate, used cars, or electronics.
- The annotation module may be responsive to page structure components and text components to identify a domain-specific datatype and associated data value within the page.
- The page classification module may comprise at least one analysis module and a first control element, the first control element being responsive to the classified page to select an analysis module.
- The at least one analysis module may comprise one or more of a form analysis module, and a results page analysis module.
- The page classification module may include a plurality of analysis modules and the first control module may be able to successively select a plurality of analysis modules.
- The first control element may be operable to pass control to the action module.
- The action module may be operable to select a navigation step and cause a further page of the website to be loaded.
- The action module may be operable to select an action comprising one or more of; selecting a link, invoking a form completion module, performing a crawler action or returning to a previous page.
- The action which may be selected by the action module may be dependent on the result of the operation of the page classification module.
- A plurality of the modules may comprise relational transducers, wherein each relational transducer embodies a set of domain-specific rules defining a relationship between input data and output data.
- The plurality of relational transducers may be disposed in a network, such that the execution order of the network of relational transducers varies depending on the page to be classified.
- The system may further comprise a control transducer, the control transducer being operable to determine control flow within the network.
- The system may further comprise a visual block analysis module to identify graphical elements of the page.
- The system may comprise a data extraction system operable to receive the wrapper and navigate the website in accordance with the wrapper to extract data from the website.
- An embodiment of the invention is described by way of example only with reference to the accompanying drawings, wherein;
-
FIG. 1 is an outline illustration of a system embodying the present invention, -
FIG. 2 is a more detailed illustration of the system ofFIG. 1 , -
FIG. 3 is a diagram illustrating operation of a part of the system ofFIGS. 2 , and -
FIG. 4 is a diagram illustrating operation of a further part of the system ofFIG. 2 . - With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of the preferred embodiments of the present invention only, and are presented in the cause of providing what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the invention. In this regard, no attempt is made to show structural details of the invention in more detail than is necessary for a fundamental understanding of the invention, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice.
- Before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of the components set forth in the following description or illustrated in the drawings. The invention is applicable to other embodiments or of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.
- In the present example, a system embodying the invention will be described with reference to data extraction from sites in a particular knowledge domain, estate agent (real estate) data. However, the invention is applicable to extracting structured information relating to any other domain.
- Throughout this description, we will refer to relational transducers. A relational transducer in this context refers to any computational element, however implemented, which is defined by particular input and output relations. Relational transducers can be self-contained, in that they interact through a shared memory and have no knowledge of each other. Because transducers are self-contained and are directed to a specific set of relations between inputs and outputs, the individual relational transducers can be relatively simple. Advantageously, in the present example the relational transducers are generally written as Datalog programs. Relational transducers, arranged into a transducer network, form the basic components of the system described herein. The transducer network provides integration and communication between the transducers in a way that represents an ideal trade-off among the system's primary integration goals:
- (1) Isolation: Transducers communicate through a transactional shared memory and have no other knowledge of each other;
- (2) Resumable: Transducers can be executed repeatedly, possibly continuing previously suspended computation, e.g., if new input data has become available.
- (3) Complexity: Transducers are generally Datalog programs with controlled value invention, retaining Datalog's polynomial data complexity but with significant performance benefits.
- (4) Data partitioning: The relations (in the shared memory) are strictly partitioned into fine-granular transducer scopes.
- The relational transducers are resumable: They can yield processing and may be called again on the same page, returning new facts. Resumption is monotone, i.e., additional calls to a transducer may produce additional output, but never retract previously derived facts. Resumable transducers are further distinguished into state- and input-driven transducers. State-driven transducers may produce new facts even if called with the same input, but maintain state between calls. These are typically transducers that iterate over some collection, e.g., all links on a page, and maintain the position in the iteration in their state relations. Input-driven transducers may also be called multiple times, but may only produce new data, if additional input is provided. Typically, these transducers are called only once or twice per page. Resumable transducers are exhausted, if further calls yield no new data.
- Relational transducers described herein are each one of three general types:
- (1) Stateless phenomenological transducers encoding phenomenological patterns, such as record and attribute identification. These patterns are domain independent, but query the possibly domain-dependent phenomenological knowledge. These transducers are typically input-driven resumable.
- (2) Stateful guarded finite state transducers (gFSTs) encode finite state transducers where transitions are guarded by first-order formulas. All exploration decisions are delegated to such transducers, which are typically state-driven resumable. The transducers used for form filling are examples of such transducers.
- (3) External programs are required for certain tasks such as interaction with the browser. The corresponding components are designed, such that they can be formalised as relational transducers with an infinite background relation. These transducers are input-driven (e.g., for wrapper induction as discussed in the following example).
- A system embodying the present invention, as illustrated in
FIG. 1 , provides a framework to enable the characterisation of a website, that is generating a wrapper which defines a record and attribute structure for the website. The wrapper effectively amounts to a program which specifies actions to navigate through the pages of a target website and to select elements identified as records. By means of appropriate selection the rules followed by the modules and relational transducers in the framework, the framework can be adapted to search websites in different domains. For example with suitable encoding of domain-specific knowledge, the system can be used to search websites relating to real estate, cars, or any other product, or indeed any other structured website with multiple records as may be desired. Although the invention is particularly described with reference to sites encoded with HTML, it will be apparent that the system can be adapted to read structured sites encoded using any appropriate language or structures. - A system embodying the invention is shown in outline in
FIG. 1 at 10. Thesystem 10 comprises aninitialisation module 11, anannotation module 12, anaction module 13, an implementingmodule 14, and a sharedmemory 15. To provide for control of the network, a control module is generally shown at 16. Thecontrol module 16 comprises a relational transducer as discussed above. Thecontrol transducer 16 is a state-driven resumable transducer for determining control flow and dependencies in the system. If there are multiple transducers which are ready to execute, the control flow is determined by priorities dynamically computed by thecontrol transducer 16 and transducers are executed in order of their priority. The control instructions from thecontrol transducer 16 are illustrated bylines memory 15 is provided such that each of the modules is able to read from and write to thememory 15 as shown byarrow connections memory 15 holds a database of the rules used by the transducer network and provides the shared memory through which the transducers interact. When a decision is taken that the site exploration is complete, awrapper 17 is generated as an output. - With reference to
FIG. 2 , a more detailed illustration of the system ofFIG. 1 shown including a number of individual functional elements which provide each of the respective modules, 11, 12, 13, 14 and which may be appropriately implemented using relational transducers. The sharedmemory 15 and the links thereto are omitted for clarity. The control transducer is generally shown at 16, and individual elements are shown at 16′, 16″ as examples of where thecontrol transducer 16 serves to control the processing flow of thesystem 10. - Within the
initialisation module 11,element 110 loads the next page of the website to be examined, whether the initial top-level site URL, or a URL for a subsequent page as described below. Atelement 111, success of this operation is checked and if not, the page failure element is invoked. The loaded HTML document is parsed into a Document Object Model (“DOM”) to facilitate subsequent operations. Inannotation module 12, the received and parsed page is annotated using anannotator element 120. Advantageously,annotation element 120 uses a plurality of labelled and named entity recognisers (“LNER”s) to identify domain-specific datatypes and values within the page. Using rules derived from the relevant domain knowledge, the LNER serve to identify entities of interest, taking into account the HTML structure and CSS formatting, in addition to the text components of the page. - The success of this step is checked at 121 and
element 200 is invoked in the event of failure. As shown at 122 and 123, a probing element may be invoked to assess the page structure if necessary. Visualblock identification module 124 identifies graphical and visual elements of the page, such as frames and images. - On completion of the annotation and identification of the various components of the page, the page then is passed to
page classification module 13. In the end of this example, a plurality ofmodules first control element 16′ of thecontrol transducer 16, to identify or interact with functional components of the page (such as forms) or information components of the page (such as the results records of interest). Each of theanalysis modules guard 131 a, 132 a, 133 a which may implement guard rules for the respective analysis module, to indicate whether the respective module is ready or appropriate for execution. In this example,module 131, referred to as OPAL, is selected when a form identified on the page and, and analyses the form to identify acceptable inputs and resultant behaviour. If result pages identified, one ofmodules example module 132, referred to as AMBER is realised as a phenomenological transducer that encodes domain-independent rules for detecting patterns on websites. These patterns are combined with template discovery, that is the detection of regularities in the structure of a page. AMBER's transducer is input-driven resumable: it is called once per page in a sequence of result pages (typically connected by pagination links), each time refining the model of the template underlying the result pages. In addition, AMBER uses the concept of a pivot attribute. A pivot attribute, such as PRICE, is a mandatory attributes of an easy to detect type. The system locates these pivot attributes to discard regular structures with irrelevant data or irregular noise in otherwise regular structures, such as advertisements interspersed among records. This increases the accuracy of record and attribute identification compared to existing template discovery approaches. In most product domains, PRICE or product identifier are ideal pivot attributes. In domains with no regular attributes, presentational attributes such as images or details page links may be chosen as pivot attributes. Textual analysis of the results fromanalysis modules text analysis module 134. - In
module 14 as discussed in more detail below, actions for further navigation of the website are selected under the control ofsecond control element 16″. The action selected may include filling an identified form as shown at 151, or performing a crawler action as shown in 152.FIG. 3 shows an alternative view of the operation ofmodule 14, including the options of navigating to a further linked page, returning to a previous page, and identifying an iframe and extracting the contents. In the example ofFIG. 3 there are fivepossible action generators 151. 152. 153, 154, 155. Each of these has a different guard, querying the outcome of the analysis and the previous exploration. For example, the next link action generator requires the presence of a list of pagination links, as well as set of records (indicating that this is a listings page). Typically only two or three of these have their guards and dependencies satisfied at this stage. Priorities are used to determine which to run: For example, thecrawler 152 is always last andiFrames 155 have priority if they are covering a substantial portion of the page. In many cases, if one of the action generators fails (e.g., if it has already attempted to click on all “next links”) some of the other action generators are called next (indicated by the red dashed arrows). For example, if the next link action generator fails, the system either back-tracks to the previous page (arrow 6), if it arrived on the current page through a form; otherwise, it continues crawling relevant links (arrow 7). Back-tracking is essential to the exploration of sites, as their exploration often requires multiple alternative paths. When choosing a path for example by filling a form, the system cannot know yet whether that path will indeed lead to relevant data. Back-tracking is dynamic and based on the transducer dependencies and guards: To back-track, the system goes back over the sequence of executed transducers until it reaches a point where either a transducer is resumable and not yet exhausted, or a prioritised choice of transducers exists with some transducers not yet executed. The control transducer also allows the specification of explicit back-tracking logic that overwrites the default case and is used, for example., for back actions in the browser. In a further example, form filling is an example of a network of state driven resumable transducers that encode guarded finite state transducers and are called repeatedly to generate queries for a given form. The form is analysed by a phenomenological transducer. The form filling is domain aware, as it uses domain knowledge to generate fillings and to react to feedback from filling (e.g., error messages). Each time the control transducer is executed the network's logical clock is advanced by one step. - With reference to
FIG. 4 , thecrawler action element 152 is shown broken down into a number ofrelational transducers 152 a to 152 f forming a sub-network of relational transducers.). This process is split into six transducers, including thebrowser interaction transducer 152 e that executes actions and themodification classifier 152 f that analyses how the browser's content has changed due to an action. It returns classifications, such as “new page”, “major page change”, “new form fields”, “new window”, or “alert”. The other four transducers are guarded finite state transducer responsible for selecting fields to be filled (152 a); selecting the assumed “behavior” for each field, e.g., that a text field is to be treated as an autocomplete (152 b); selecting the specific value for each field (152 c); and iterating over all fields, feeding the browser interaction transducer with the filling for each field one by one (152 d). These four transducers are all resumable and chained: If there are no more fields to be filled in the current iteration, the filling iteration yields control back to the value selection. The value selection may return the same sequence of fields with different values or, if there are no additional value combinations to try, may fail and yield control to the behaviour selection. Depending on the classification of the modification, execution continues either with the field iteration (typically if there is no change, arrow 1), with the field set selection (if the state of form fields changes, arrow 2), or with the end of the filling phase, for example if a new page is reached (arrow 3). - In the event that the operation is successful,
generator 160 stores the relevant information to be included in the wrapper. Insert F here If the identified action is successfully performed atstep 161 and is checked atstep 162 the process repeats. The wrapper accumulates information about all identified result pages and the navigation paths leading to them and integrates that information into a coherent wrapper program. It is input driven resumable, called once per page, but accumulating the wrapper information over all the calls for one site. Within a result page sequence, it combines the collected information into a coherent wrapper for the underlying template of these pages, that is likely applicable also to any other page in the sequence. - Accordingly, the system comprises a plurality of relational transducers which form a synchronised transducer network. The network is referred to as being synchronised as its control flow is determined by a central controller, itself a relational transducer. Intuitively, a transducer network is a set of transducers with a transactional shared memory which serves as input and output for the transducers. The execution is controlled by the control transducer and a specific area in the memory is reserved for communication between controller and transducers. The network is self-adaptive, as the control flow is dynamically determined from transducer dependencies and their guard rules. Rather than relying on one or a few statically defined control flows for exploring a page or site, this allows the network to form different control flows for the exploration of each individual site. Dependency and guard rules, registered by the individual transducers with the control transducer, thus determine for each transducer separately if it can be executed at a given point. This determination is typically based on the already explored portion of the site or page. For example, a form analysis transducer has a dependency on the transducer that produces annotations for Document Object Model (“DOM”) elements and a guard rule that prevents it from running if there are no form elements on the page. It has the same priority as, e.g., the record identification transducer, which has similar dependencies, but is guarded by requiring the presence of pivot attribute annotations. If both guards are satisfied, the transducers may be executed in parallel. Transducers that yield an interaction with the browser cannot be executed in parallel, but must be sequentialised, as parallel access may break server state or Javascript execution. Therefore, for the selection and execution of actions, explicit priorities are used to sequentialise the actions and to prioritise actions with the highest estimated probability to lead to relevant data.
- In the above description, an embodiment is an example or implementation of the invention. The various appearances of “one embodiment”, “an embodiment” or “some embodiments” do not necessarily all refer to the same embodiments.
- Although various features of the invention may be described in the context of a single embodiment, the features may also be provided separately or in any suitable combination. Conversely, although the invention may be described herein in the context of separate embodiments for clarity, the invention may also be implemented in a single embodiment.
- Furthermore, it is to be understood that the invention can be carried out or practiced in various ways and that the invention can be implemented in embodiments other than the ones outlined in the description above.
- Meanings of technical and scientific terms used herein are to be commonly understood as by one of ordinary skill in the art to which the invention belongs, unless otherwise defined.
Claims (14)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/871,027 US10325000B2 (en) | 2014-09-30 | 2015-09-30 | System for automatically generating wrapper for entire websites |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201462057395P | 2014-09-30 | 2014-09-30 | |
US14/871,027 US10325000B2 (en) | 2014-09-30 | 2015-09-30 | System for automatically generating wrapper for entire websites |
Publications (3)
Publication Number | Publication Date |
---|---|
US20160092458A1 US20160092458A1 (en) | 2016-03-31 |
US20160321280A2 true US20160321280A2 (en) | 2016-11-03 |
US10325000B2 US10325000B2 (en) | 2019-06-18 |
Family
ID=55584621
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/871,027 Active 2036-08-12 US10325000B2 (en) | 2014-09-30 | 2015-09-30 | System for automatically generating wrapper for entire websites |
Country Status (1)
Country | Link |
---|---|
US (1) | US10325000B2 (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11288336B2 (en) * | 2018-04-18 | 2022-03-29 | Google Llc | Systems and methods for providing content items in situations involving suboptimal network conditions |
US11321412B1 (en) * | 2020-11-04 | 2022-05-03 | Capital One Services, Llc | Customized navigation flow |
Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020007369A1 (en) * | 1999-12-17 | 2002-01-17 | Thiagarajan Saravanan | Network navigation |
US20020178169A1 (en) * | 2001-05-23 | 2002-11-28 | Nair Sandeep R. | System and method for efficient and adaptive web accesses filtering |
US6549220B1 (en) * | 1999-06-10 | 2003-04-15 | International Business Machines Corporation | Method, system, and program for providing pages of information with navigation and content areas |
US20030167209A1 (en) * | 2000-09-29 | 2003-09-04 | Victor Hsieh | Online intelligent information comparison agent of multilingual electronic data sources over inter-connected computer networks |
US20060117268A1 (en) * | 2004-11-30 | 2006-06-01 | Micheal Talley | System and method for graphical element selection for region of interest compression |
US20070033188A1 (en) * | 2005-08-05 | 2007-02-08 | Ori Levy | Method and system for extracting web data |
US20070073758A1 (en) * | 2005-09-23 | 2007-03-29 | Redcarpet, Inc. | Method and system for identifying targeted data on a web page |
US20070094267A1 (en) * | 2005-10-20 | 2007-04-26 | Glogood Inc. | Method and system for website navigation |
US20090119268A1 (en) * | 2007-11-05 | 2009-05-07 | Nagaraju Bandaru | Method and system for crawling, mapping and extracting information associated with a business using heuristic and semantic analysis |
US20090249251A1 (en) * | 2008-04-01 | 2009-10-01 | International Business Machines Corporation | Generating a user defined page having representations of content in other pages |
US20100198770A1 (en) * | 2009-02-03 | 2010-08-05 | Yahoo!, Inc., a Delaware corporation | Identifying previously annotated web page information |
US20110022552A1 (en) * | 2009-07-22 | 2011-01-27 | Carnegie Mellon University | Systems and Methods for Implementing a Machine-Learning Agent to Retrieve Information in Response to a Message |
US8037048B2 (en) * | 2007-02-13 | 2011-10-11 | Web Lion S.A.S. Di Panarese Marco & Co. | Web site search and selection method |
US8595613B1 (en) * | 2012-07-26 | 2013-11-26 | Viasat Inc. | Page element identifier pre-classification for user interface behavior in a communications system |
US8688545B1 (en) * | 2011-03-24 | 2014-04-01 | Amazon Technologies, Inc. | Fail-safe ordering |
US20140172506A1 (en) * | 2012-12-17 | 2014-06-19 | Microsoft Corporation | Customer segmentation |
US20140208202A1 (en) * | 2013-01-23 | 2014-07-24 | Go Daddy Operating Company, LLC | System for conversion of website content |
US20150100877A1 (en) * | 2012-06-29 | 2015-04-09 | Yahoo! Inc. | Method or system for automated extraction of hyper-local events from one or more web pages |
US9323731B1 (en) * | 2007-11-01 | 2016-04-26 | Google Inc. | Data extraction using templates |
US9423890B2 (en) * | 2013-06-28 | 2016-08-23 | Lenovo (Singapore) Pte. Ltd. | Stylus lexicon sharing |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6851089B1 (en) * | 1999-10-25 | 2005-02-01 | Amazon.Com, Inc. | Software application and associated methods for generating a software layer for structuring semistructured information |
US8793584B2 (en) * | 2006-05-24 | 2014-07-29 | International Business Machines Corporation | Customizable user interface wrappers for web applications |
US8392890B2 (en) * | 2007-10-15 | 2013-03-05 | Software Research, Inc. | Method and system for testing websites |
US8762829B2 (en) * | 2008-12-24 | 2014-06-24 | Yahoo! Inc. | Robust wrappers for web extraction |
US20100199165A1 (en) * | 2009-02-03 | 2010-08-05 | Yahoo!, Inc., a Delaware corporation | Updating wrapper annotations |
US8578010B2 (en) * | 2009-12-17 | 2013-11-05 | Mastercard International Incorporated | Methods and system for tracking web page analytics |
US10115121B2 (en) * | 2013-12-11 | 2018-10-30 | Adobe Systems Incorporated | Visitor session classification based on clickstreams |
-
2015
- 2015-09-30 US US14/871,027 patent/US10325000B2/en active Active
Patent Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6549220B1 (en) * | 1999-06-10 | 2003-04-15 | International Business Machines Corporation | Method, system, and program for providing pages of information with navigation and content areas |
US20020007369A1 (en) * | 1999-12-17 | 2002-01-17 | Thiagarajan Saravanan | Network navigation |
US20030167209A1 (en) * | 2000-09-29 | 2003-09-04 | Victor Hsieh | Online intelligent information comparison agent of multilingual electronic data sources over inter-connected computer networks |
US20020178169A1 (en) * | 2001-05-23 | 2002-11-28 | Nair Sandeep R. | System and method for efficient and adaptive web accesses filtering |
US20060117268A1 (en) * | 2004-11-30 | 2006-06-01 | Micheal Talley | System and method for graphical element selection for region of interest compression |
US20070033188A1 (en) * | 2005-08-05 | 2007-02-08 | Ori Levy | Method and system for extracting web data |
US20070073758A1 (en) * | 2005-09-23 | 2007-03-29 | Redcarpet, Inc. | Method and system for identifying targeted data on a web page |
US20070094267A1 (en) * | 2005-10-20 | 2007-04-26 | Glogood Inc. | Method and system for website navigation |
US8037048B2 (en) * | 2007-02-13 | 2011-10-11 | Web Lion S.A.S. Di Panarese Marco & Co. | Web site search and selection method |
US9323731B1 (en) * | 2007-11-01 | 2016-04-26 | Google Inc. | Data extraction using templates |
US20090119268A1 (en) * | 2007-11-05 | 2009-05-07 | Nagaraju Bandaru | Method and system for crawling, mapping and extracting information associated with a business using heuristic and semantic analysis |
US20090249251A1 (en) * | 2008-04-01 | 2009-10-01 | International Business Machines Corporation | Generating a user defined page having representations of content in other pages |
US20100198770A1 (en) * | 2009-02-03 | 2010-08-05 | Yahoo!, Inc., a Delaware corporation | Identifying previously annotated web page information |
US20110022552A1 (en) * | 2009-07-22 | 2011-01-27 | Carnegie Mellon University | Systems and Methods for Implementing a Machine-Learning Agent to Retrieve Information in Response to a Message |
US8688545B1 (en) * | 2011-03-24 | 2014-04-01 | Amazon Technologies, Inc. | Fail-safe ordering |
US20150100877A1 (en) * | 2012-06-29 | 2015-04-09 | Yahoo! Inc. | Method or system for automated extraction of hyper-local events from one or more web pages |
US8595613B1 (en) * | 2012-07-26 | 2013-11-26 | Viasat Inc. | Page element identifier pre-classification for user interface behavior in a communications system |
US20140172506A1 (en) * | 2012-12-17 | 2014-06-19 | Microsoft Corporation | Customer segmentation |
US20140208202A1 (en) * | 2013-01-23 | 2014-07-24 | Go Daddy Operating Company, LLC | System for conversion of website content |
US9423890B2 (en) * | 2013-06-28 | 2016-08-23 | Lenovo (Singapore) Pte. Ltd. | Stylus lexicon sharing |
Non-Patent Citations (5)
Title |
---|
Embley et al., Conceptual-model-based Data Extraction from Multiple-Record Web Pages, Elsevier 1999, pages 227-251. * |
Laender et al., A Brief Survey of Web Data Extraction Tools, ACM 2002, pages 84-93. * |
Wang et al., Data Extraction and Label Assignment for Web Databases, ACM 2003, pages 187-196. * |
Zhai et al., Web Data Extraction based on Partial Tree Alignment, ACM 2005, pages 76-85. * |
Zheng et al., Joint Optimization of Wrapper Generation and Template Detection, ACM 2007, pages 894-902. * |
Also Published As
Publication number | Publication date |
---|---|
US10325000B2 (en) | 2019-06-18 |
US20160092458A1 (en) | 2016-03-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Chen et al. | A survey on the use of topic models when mining software repositories | |
US10725836B2 (en) | Intent-based organisation of APIs | |
Uzun | A novel web scraping approach using the additional information obtained from web pages | |
Furche et al. | DIADEM: thousands of websites to a single database | |
US10152488B2 (en) | Static-analysis-assisted dynamic application crawling architecture | |
CN101739335B (en) | Recommended application evaluation system | |
JP2022031851A (en) | Automatic mapping system and method of data type used in data flow environment | |
Gunaratna et al. | Gleaning types for literals in rdf triples with application to entity summarization | |
US11263062B2 (en) | API mashup exploration and recommendation | |
CN104331438B (en) | To novel web page contents selectivity abstracting method and device | |
KR102421904B1 (en) | the method to advance the analysis of the causes of disasters | |
US11836331B2 (en) | Mathematical models of graphical user interfaces | |
Omari et al. | Cross-supervised synthesis of web-crawlers | |
US9305261B2 (en) | Knowledge management engine for a knowledge management system | |
US10325000B2 (en) | System for automatically generating wrapper for entire websites | |
Kumar | Apache Solr search patterns | |
Suhas et al. | Firth's penalized logistic regression: A superior approach for analysis of data from India's National Mental Health Survey, 2016 | |
US8843468B2 (en) | Classification of transactional queries based on identification of forms | |
JP4954674B2 (en) | Software development support method, software development support device, software development support program, and computer system | |
Luo et al. | Automated structural semantic annotation for RESTful services | |
US20180239809A1 (en) | Managing content creation of data sources | |
Alam et al. | A data-driven score model to assess online news articles in event-based surveillance system | |
Lüders et al. | On understanding and predicting issue links | |
Gupta et al. | Large-scale information extraction from emails with data constraints | |
US20160117352A1 (en) | Apparatus and method for supporting visualization of connection relationship |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: OXFORD UNIVERSITY INNOVATION LIMITED, GREAT BRITAIN Free format text: CHANGE OF NAME;ASSIGNOR:ISIS INNOVATION LIMITED;REEL/FRAME:039550/0045 Effective date: 20160616 Owner name: OXFORD UNIVERSITY INNOVATION LIMITED, GREAT BRITAI Free format text: CHANGE OF NAME;ASSIGNOR:ISIS INNOVATION LIMITED;REEL/FRAME:039550/0045 Effective date: 20160616 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: MELTWATER NEWS INTERNATIONAL HOLDINGS GMBH, SWITZE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:OXFORD UNIVERSITY INNOVATION LIMITED;REEL/FRAME:049568/0337 Effective date: 20190618 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
AS | Assignment |
Owner name: DNB BANK ASA, AS SECURITY AGENT, NORWAY Free format text: INTELLECTUAL PROPERTY SECURITY AGREEMENT SUPPLEMENT;ASSIGNOR:MELTWATER NEWS US INC.;REEL/FRAME:066159/0082 Effective date: 20240102 |
|
AS | Assignment |
Owner name: MELTWATER NEWS US, INC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MELTWATER NEWS INTERNATIONAL HOLDINGS GMBH;REEL/FRAME:066753/0871 Effective date: 20240220 |