US10769234B2 - Document object model transaction crawler - Google Patents

Document object model transaction crawler Download PDF

Info

Publication number
US10769234B2
US10769234B2 US15/680,418 US201715680418A US10769234B2 US 10769234 B2 US10769234 B2 US 10769234B2 US 201715680418 A US201715680418 A US 201715680418A US 10769234 B2 US10769234 B2 US 10769234B2
Authority
US
United States
Prior art keywords
state
dom
event
transaction log
state change
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US15/680,418
Other languages
English (en)
Other versions
US20180060446A1 (en
Inventor
Thomas Christopher Swedlund
Kevin Allen Williams
Brian Charles King
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Micro Focus LLC
Original Assignee
Micro Focus LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Micro Focus LLC filed Critical Micro Focus LLC
Assigned to HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP reassignment HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: WILLIAMS, Kevin Allen, KING, Brian Charles, SWEDLUND, Thomas Christopher
Priority to US15/680,418 priority Critical patent/US10769234B2/en
Assigned to ENTIT SOFTWARE LLC reassignment ENTIT SOFTWARE LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP
Publication of US20180060446A1 publication Critical patent/US20180060446A1/en
Assigned to MICRO FOCUS LLC reassignment MICRO FOCUS LLC CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: ENTIT SOFTWARE LLC
Assigned to JPMORGAN CHASE BANK, N.A. reassignment JPMORGAN CHASE BANK, N.A. SECURITY AGREEMENT Assignors: BORLAND SOFTWARE CORPORATION, MICRO FOCUS (US), INC., MICRO FOCUS LLC, MICRO FOCUS SOFTWARE INC., NETIQ CORPORATION
Assigned to JPMORGAN CHASE BANK, N.A. reassignment JPMORGAN CHASE BANK, N.A. SECURITY AGREEMENT Assignors: BORLAND SOFTWARE CORPORATION, MICRO FOCUS (US), INC., MICRO FOCUS LLC, MICRO FOCUS SOFTWARE INC., NETIQ CORPORATION
Publication of US10769234B2 publication Critical patent/US10769234B2/en
Application granted granted Critical
Assigned to NETIQ CORPORATION, MICRO FOCUS LLC, MICRO FOCUS SOFTWARE INC. (F/K/A NOVELL, INC.) reassignment NETIQ CORPORATION RELEASE OF SECURITY INTEREST REEL/FRAME 052294/0522 Assignors: JPMORGAN CHASE BANK, N.A.
Assigned to NETIQ CORPORATION, MICRO FOCUS SOFTWARE INC. (F/K/A NOVELL, INC.), MICRO FOCUS LLC reassignment NETIQ CORPORATION RELEASE OF SECURITY INTEREST REEL/FRAME 052295/0041 Assignors: JPMORGAN CHASE BANK, N.A.
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
    • G06F16/986—Document structures and storage, e.g. HTML extensions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2358—Change logging, detection, and notification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/957—Browsing optimisation, e.g. caching or content distillation
    • G06F16/9577—Optimising the visualization of content, e.g. distillation of HTML documents

Definitions

  • Web crawlers identify content of a web to enable entities to automatically determine content of web pages without the need for user interaction with the web page.
  • Web crawlers may identify content of web pages using hypertext markup language (HTML) of the web pages, document object models (DOMs) of the web pages, etc.
  • HTML hypertext markup language
  • DOMs document object models
  • Some web pages may include web applications that enable user interaction with the web page and/or features of the web page via the web application.
  • FIG. 1 illustrates a schematic diagram of an example document object model (DOM) crawler system including a DOM transaction crawler that may be implemented in accordance with an aspect of this disclosure.
  • DOM document object model
  • FIG. 2 a block diagram of an example DOM transaction crawler that may be used to implement the DOM transaction crawler of FIG. 1 .
  • FIG. 3 illustrates an example event tree of a DOM that may be traversed by the DOM transaction crawler of FIG. 2 in accordance with examples herein.
  • FIG. 4 is a flowchart representative of example machine readable instructions that may be executed to implement the DOM transaction crawler of FIG. 2 .
  • FIG. 5 is a flowchart representative of example machine readable instructions that may be executed to implement the DOM transaction crawler of FIG. 2 to enable isolated execution of events by managing transactions and storing state changes between states of a DOM in accordance with an aspect of this disclosure.
  • FIG. 6 is a flowchart representative of example machine readable instructions that may be executed to implement the DOM transaction crawler of FIG. 2 to execute an iteration of traversing an event in accordance with an aspect of this disclosure.
  • FIG. 7 is a block diagram of an example processor platform capable of executing the instructions of FIGS. 4, 5 , and/or 6 to implement the DOM transaction crawler of FIG. 2 .
  • Examples disclosed herein involve a document object model (DOM) transaction crawler to navigate a DOM of a webpage or web application (e.g., a single page application or other dynamic application).
  • DOM document object model
  • Web crawlers in general, are to automatically (without any user interaction, other than to perhaps initiate a crawl) discover content of a web application.
  • Content of large dynamic sites e.g., single page applications (SPAs)
  • SPAs single page applications
  • code e.g., JavaScript
  • the crawler gains knowledge of the DOM to use heuristics to navigate DOM transitions as a user might navigate the web application.
  • a web application's event space can become large enough to make a crawl intractable.
  • a web transaction crawler records atomic state changes when executing an event of a DOM such that the state changes can be reverted to a previous state prior to executing the event. Accordingly, examples herein, provide an effective and efficient means of traversing a DOM of a web application without re-executing the web application to reach each transition or event of the DOM.
  • events are traversed by executing the events, identifying state changes in response to execution of the events, and recording (or storing) the state changes in a transaction log. Accordingly, transactions may be returned to a previous state prior to execution of any event of the application based on the recorded state changes so that any remaining events in the DOM can be identified and executed as if the event was not executed. As such, state changes from execution of events are reverted (e.g., rolled back) and the state changes may be played back (by retrieving and implementing state changes in the DOM) to crawl a DOM without re-execution of the events.
  • Examples herein involve identifying an event in a first state of a document object model.
  • the event is executed to cause a state change from the first state of the document object model to a second state of the document object model.
  • the state change may be stored in a transaction log to store the difference between the first state of the document object model and the second state of the document object model. Examples further involve reverting the document object model from the second state of the document object model to the first state of the document object model for execution of a subsequent event of the first state of the document object model.
  • content of a web page or web application may refer to any text, graphic, picture, user interface, clickable (e.g., a button, drop down, etc.), mouse-over, etc. that is provided and/or available via the web page or web application.
  • a web page or web application may be used interchangeably, however, a web application (or SPA) may be included within a web page.
  • FIG. 1 is a schematic diagram of an example document object model (DOM) crawler system 100 including a DOM transaction crawler 110 implemented in according with examples herein.
  • the example DOM crawler system 100 of FIG. 1 includes the DOM transaction crawler 110 , a DOM creator 120 , and a vulnerability detector 130 .
  • the DOM creator 120 creates a DOM from hypertext markup language (HTML) of a universal resource locator (URL)
  • the DOM transaction crawler 110 determines content of a web page or a web application of the URL using the generated DOM
  • the vulnerability detector 130 detects vulnerabilities in the content of the web page or web application of the URL.
  • the example DOM creator 120 generates a DOM of a web page using HTML of a received or retrieved URL.
  • the DOM creator 120 may be implemented by a headless browser to build a DOM from the HTML for the web page or web application.
  • the DOM creator 120 may parse the HTML of a URL and run any code (e.g., JavaScript) necessary for the SPA to load.
  • the example DOM creator 120 may modify any data objects involved with the DOM and the code runtime by tagging properties of the object to represent the object. Such properties may be adjusted each time the properties change as a result of triggering an event in the DOM. Accordingly, the generated DOM may be used to independently detect, execute, and analyze content of the web application without affecting the actual web application.
  • the DOM transaction crawler 110 of FIG. 1 crawls the DOM generated by the DOM creator 120 accordingly to examples herein.
  • the DOM transaction crawler 110 discovers events within the generated DOM.
  • the DOM transaction crawler 110 may identify events of interest, such as user interactive events.
  • the events of a generated DOM may form a tree architecture, such that certain events are triggered in response to triggering other events.
  • the example DOM transaction crawler 110 may then execute the events of the DOM (independently from executing any events of the actual web page or web application) to determine the content of the web page or web application.
  • the DOM transaction crawler 110 utilizes a transaction mechanism to record changes (e.g., in a transaction log) in the state of the DOM in response to triggering an event, and reverting the DOM to a previous state after triggering each event.
  • changes e.g., in a transaction log
  • the example recorded changes may be used to efficiently and effectively navigate the transitions of the DOM.
  • the vulnerability detector 130 may then detect vulnerabilities in the content of the web application by attacking or accessing portions of the web content with a securities scanner. Any suitable technique may be used to perform a security scan and detect vulnerabilities in the web content.
  • additional components or alternative components to the vulnerability detector 130 may be included in the example system 100 of FIG. 1 to perform additional/alternative analysis on the web content identified by the DOM transaction crawler 110 . Accordingly, examples herein may allow for detection of vulnerabilities of a web page or web application or any other type of analysis by efficiently crawling a DOM generated from HTML of the web page or web application.
  • FIG. 2 is a block diagram of an example DOM transaction crawler 110 that may be used to implement the DOM transaction crawler 110 of FIG. 1 .
  • the example DOM transaction crawler 110 of FIG. 2 includes an event identifier 210 , an event executor 220 , a transaction manager 230 , and a state change recorder 240 .
  • the event identifier 210 identifies events in a DOM of a web application
  • the event executor 220 executes the identified events
  • the transaction manager 230 maintains a state of the DOM after executing the events
  • the state change recorder 240 logs changes in the state of the DOM based on the execution of the events.
  • the example event identifier 210 may scan or search a DOM (e.g., a DOM generated by the DOM creator 120 of FIG. 1 ) for events of interest.
  • Example events of interest may include user interactive events, such as button clicks, mouse-overs, or form submissions.
  • the event identifier 110 may identify static events by traversing nodes of the DOM and scanning for event attributes (e.g., from tagged properties of the event).
  • a ⁇ button> element of the DOM may include code (e.g., JavaScript) within an attribute (e.g., an ‘onclick’ attribute).
  • events may be applied to elements dynamically using a call (e.g., to ‘addEventListener’), which may be tracked so that the events may be stored and executed.
  • anchor tags may include code in a definition (e.g., a ‘href’ definition) by prepending ‘javascript:’ to the code.
  • the event identifier 210 may identify events in a DOM, such as a DOM generated by the DOM creator 120 of FIG. 1 .
  • the example event executor 220 executes the events to elicit DOM state transitions, which may make additional events available and/or identifiable.
  • a new state of the DOM is generated.
  • the example new state of the DOM may reflect the state of the DOM as a result of executing the event.
  • the event identifier 210 may then scan the new state of the DOM to identify any new or other events in the new state of the DOM. These new events may be subsequent events to be executed in a subsequent iteration. Accordingly, this iterative process may build a tree of DOM events that may be traversed to discover content of a web application.
  • the event executor 220 and event identifier 210 may use heuristics such as a breadth-first crawl or a depth-first crawl of an event tree of the DOM.
  • a breadth-first crawl involves executing each sibling event of a tier of an event tree of a DOM before advancing to a next tier of the event tree of the DOM
  • a depth-first crawl involves executing the sibling events of a branch of an event tree of the DOM before advancing to a next branch of the event tree of the DOM (see FIG. 3 and accompanying description below).
  • event executor 220 may effectively pair an event-based crawl with a traffic-based crawl to traverse the events of a DOM and enable access to the content of the web application.
  • the example transaction manager 230 of FIG. 2 identifies state changes between states (e.g., tiers or branches of an event tree) of a DOM in response to the execution of each of the events by the event executor 220 and reverts/returns the generated DOM to a previous state prior to the event executor 220 executing each event without re-executing the entire web application. Accordingly, the transaction manager 230 enables the DOM transaction crawler 110 to crawl events of the DOM without keeping the effects of previous event traversal by playing back state changes (retrieving stored state changes). For example, when traversing the event in a depth-first manner, sibling events (events from the same state of the DOM) may be run without affecting the triggering of another event.
  • states e.g., tiers or branches of an event tree
  • the example transaction manager 230 may achieve this by, after the event executor 220 executes the event, the transaction manager 230 determines the state changes to the DOM (which are recorded by the state change recorder 240 ) and returns the DOM back to the state of the DOM prior to executing that event, then running the sibling event. The transaction manager 230 may return the DOM to the previous state by simply rolling back the state changes.
  • the transaction manager 230 may refer to a log of the state changes maintained by the state change recorder 240 .
  • the state change recorder 240 records atomic state changes to the DOM after the event executor 220 executes each event of the DOM in a transaction log. Accordingly, the state change recorder 240 may include or have access to a database for maintaining the transaction log.
  • the transaction manager 230 may then refer to the state changes in the transaction log to revert or roll back the state of the DOM after execution of each event so that sibling events may be executed without affecting one another. In other words, the state changes from executing one sibling event will not carry over to the execution of another sibling event.
  • the event executor 220 (and the DOM transaction crawler 110 , generally) are able to traverse and execute sibling events as if none of the sibling events had yet been executed, though some of them may actually have been executed.
  • the transaction manager 230 may use the transaction log to play back state changes to a subsequent state of the DOM after execution of all sibling events of a state of the DOM. For example, the transaction manager may retrieve state changes corresponding to events in the transaction log of the state change recorder and implement those changes in the DOM to reach the desired event of the DOM.
  • the transaction manager 230 may traverse a DOM using state changes in the transaction log to reach a particular state of the DOM. Accordingly, subsequent events in subsequent states of the DOM may be reached without re-executing the entire web application or events of the web application (e.g., from the beginning or an initial state of the DOM).
  • the transaction manager 230 By recording only the state changes to the DOM in the transaction log, the transaction manager 230 enables the DOM transaction crawler 110 to effectively and efficiently traverse the events of a DOM. Recording the state changes to the DOM after each execution of an event and rolling back the state changes allows for the isolation of event executions, which facilitates traversing the events of the DOM without accumulating state transitions. Furthermore, the transaction manager 230 and the state change recorder 240 assist in crawl tractability by allowing DOM transitions to occur without having to re-traverse the web application. Furthermore, the transaction manager 230 and state change recorder 240 may advance tractability by persisting transactions to persistent storage and relieving memory pressure on the DOM transaction crawler 110 .
  • FIG. 2 While an example manner of implementing the DOM transaction crawler 110 of FIG. 1 is illustrated in FIG. 2 , at least one of the elements, processes and/or devices illustrated in FIG. 2 may be combined, divided, re-arranged, omitted, eliminated and/or implemented in any other way. Further, the event identifier 210 , the event executor 220 , the transaction manager 230 , the state change recorder 240 , and/or, more generally, the DOM transaction crawler 110 of FIG. 2 may be implemented by hardware and/or any combination of hardware and executable instructions (e.g., software and/or firmware).
  • hardware and executable instructions e.g., software and/or firmware
  • any of the event identifier 210 , the event executor 220 , the transaction manager 230 , the state change recorder 240 , and/or, more generally, the DOM transaction crawler 110 of FIG. 2 could be implemented by at least one of an analog or digital circuit, a logic circuit, a programmable processor, an application specific integrated circuit (ASIC), a programmable logic device (PLD) and/or a field programmable logic device (FPLD).
  • ASIC application specific integrated circuit
  • PLD programmable logic device
  • FPLD field programmable logic device
  • At least one of the event identifier 210 , the event executor 220 , the transaction manager 230 , and/or the state change recorder 240 is/are hereby expressly defined to include a tangible machine readable storage device or storage disk such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc. storing the executable instructions.
  • the example DOM transaction manager 110 of FIG. 2 may include at least one element, process, and/or device in addition to, or instead of, those illustrated in FIG. 2 , and/or may include more than one of any or all of the illustrated elements, processes and devices.
  • FIG. 3 illustrates an example event tree 310 of a DOM 320 that may be traversed by a DOM transaction crawler 110 , which may be implemented by the DOM transaction manager 110 of FIG. 2 .
  • the example event tree 310 includes ten (numbered 1-10) events that may be executed by the event executor 220 to access content of a web application of the DOM 320 .
  • the DOM transaction crawler 110 may traverse the events of the DOM 320 via a breadth-first crawl or a depth-first crawl.
  • an example breadth-first crawl of the event tree 310 involves the event executor 220 executing the events across tiers of the event tree.
  • the event executor 220 executes event 1
  • the state change recorder 240 records any change in the state of the DOM after the execution of event 1 to a transaction log
  • the transaction manager 230 reverts the state of the DOM 320 to the state of the DOM prior to execution of event 1 (as if event 1 had not been executed).
  • the event identifier 210 and/or event executor 220 may determine whether there are any remaining sibling events of events 1 (breadth-first sibling events) to be executed. Identifying event 2 as breadth-first crawl sibling event, the event executor 220 may then execute event 2, record any state changes, and revert back to the original state of the DOM 320 prior to executing event 2. With all sibling events of that tier of the event tree having been executed, the event executor 220 may move on to execute the next event.
  • the event executor 220 may refer to the transaction log to determine state changes of executing event 1, which may indicate that events 3 and 4 are created in response to executing event 1.
  • the event executor 220 may then execute event 3, the state change recorder 240 may store any changes in the state of the DOM 320 in the transaction log, and the transaction manager 230 revert back the DOM 320 back the state of the DOM prior to executing event 3. From there, the event executor 220 , state change recorder 240 , and transaction manager 230 may perform the same iteration for event 4 (identifying event 4 as a sibling of event 3).
  • the event executor 220 may play back the state changes from events 1 and 3 and perform the same iterations for event 6, before playing back states changes of 1 and 4 to perform iterations on event 7, then event 8, then event 10, before returning to the branch of event 2 to play back the state changes of event 2 and perform iterations on events 5 and 9.
  • the DOM transaction crawler 110 e.g., via the transaction manager
  • the DOM transaction crawler 110 may traverse the event tree 310 of the DOM 320 via a depth-first crawl in accordance with examples herein.
  • the example event executor 220 may perform iterations on event 1, then event 3, then event 6, before reverting back to the state of the DOM 320 and playing back the state changes from executing event 1, to perform iterations on event 4, event 7, event 8, and event 10.
  • the DOM transaction crawler in a depth-first crawl may then play back the DOM back to the original state of the DOM 320 (by referring to the transaction log of the state change recorder 240 ) to perform the iterations on the branch of the event tree extending from event 2 (i.e., event 2, event 5, and event 9).
  • the order of iterations for a breadth-first crawl of the example event tree 310 may be 1, 2, 3, 4, 6, 7, 8, 10, 5, 9, while the order of iterations for a depth-first crawl of the example event tree 310 may be 1, 3, 6, 4, 7, 8, 10, 2, 5, 9.
  • User settings or preferences may be used to select a traversal method. Accordingly, regardless of the selected method, the DOM transaction crawler 110 may effectively and efficiently traverse the event tree by storing state changes, retrieving state changes, and loading the state changes after all sibling events for a state of a DOM have been executed.
  • FIGS. 4, 5, and 6 Flowcharts representative of example machine readable instructions for implementing the DOM transaction crawler 110 of FIG. 2 are shown in FIGS. 4, 5, and 6 .
  • the machine readable instructions comprise a program(s)/process(es) for execution by a processor such as the processor 712 shown in the example processor platform 700 discussed below in connection with FIG. 7 .
  • the program(s)/process(es) may be embodied in executable instructions (e.g., software) stored on a tangible machine readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD), a Blu-ray disk, or a memory associated with the processor 712 , but the entire program/process and/or parts thereof could alternatively be executed by a device other than the processor 712 and/or embodied in firmware or dedicated hardware.
  • a tangible machine readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD), a Blu-ray disk, or a memory associated with the processor 712 , but the entire program/process and/or parts thereof could alternatively be executed by a device other than the processor 712 and/or embodied in firmware or dedicated hardware.
  • the example program(s)/process(es) is/are described with reference to the flowchart illustrated in FIGS. 4, 5 ,
  • the example process 400 of FIG. 4 begins with an initiation of the DOM transaction crawler 110 (e.g., upon startup, upon instructions from a user, upon startup of a device implementing the DOM transaction crawler 110 (e.g., the DOM crawler system 100 ), etc.).
  • the example process 400 of FIG. 4 may be executed to traverse an event of a DOM in accordance with examples herein.
  • the event identifier 210 identifies an event in a first state of a DOM.
  • the event identifier 210 may identify user interactive events (e.g., using attributes for the events stored in the DOM).
  • the event executor 220 executes the event to determine state change from the first state of the DOM to a second state of the DOM.
  • the state change recorder 240 stores the state change comprising a difference between the first state of the DOM and the second state of the DOM in a transaction log.
  • the transaction manager 230 reverts the DOM from the second state of the DOM to the first state of the DOM for execution of a subsequent event of the first state of the DOM.
  • the example process 500 of FIG. 5 begins with an initiation of the DOM transaction crawler 110 .
  • the example process 500 of FIG. 5 may be executed to enable isolated execution of events by managing transactions and storing state changes between states of a DOM.
  • the transaction manager 230 identifies a first state change in a DOM of an application in response to executing an event of a first state of the DOM.
  • the state change of the DOM in block 510 is a difference between a first state of the DOM and a second state of the DOM.
  • the state change recorder 240 stores the state changes in a transaction log.
  • the event identifier 210 and/or event executor 220 determines whether there are any remaining sibling events in the first state of the DOM. If there are no remaining sibling events in the first state of the DOM, the example process 500 ends (e.g., the event executor 220 traverses the DOM to a new branch of an event tree of the DOM). If there are remaining sibling events in the first state of the DOM (block 530 ), then the transaction manager 230 reverts the DOM from the second state of the DOM to the first state of the DOM at block 540 .
  • the event executor 220 executes the sibling event.
  • the state change recorder 240 stores a second state change in the DOM in the log in response to executing the sibling event. After block 560 , the example process 500 ends.
  • the example process 600 of FIG. 6 begins with an initiation of the DOM transaction crawler 110 .
  • the example process 600 may be iteratively executed to perform iterations of traversing an event in accordance with examples herein.
  • the event executor 220 executes a next event of a state of a DOM.
  • the state change recorder 240 records a state change (e.g., based on a state change identified by the transaction manager 230 after the event executor 220 executed the event).
  • the event identifier 210 and/or the event executor 220 determines whether there are any remaining sibling events in the state of the DOM.
  • control returns to block 610 for a subsequent iteration. If there are remaining sibling events in the DOM, at block 640 , the event identifier 210 and/or event executor 220 determines whether there are any remaining events in the DOM. If there are no remaining events in the DOM at block 640 , the example process 600 ends. If, at block 640 , there are remaining events in the DOM, then at block 650 , the transaction manager 230 retrieves state change data from a transaction log for a next state of the DOM and reverts the DOM to a state corresponding to the next event of the DOM. After block 650 , control returns to block 610 for a subsequent iteration.
  • FIGS. 4, 5 , and/or 6 may be implemented using coded instructions (e.g., computer and/or machine readable instructions) stored on a tangible machine readable storage medium such as a hard disk drive, a flash memory, a read-only memory (ROM), a compact disk (CD), a digital versatile disk (DVD), a cache, a random-access memory (RAM) and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information).
  • coded instructions e.g., computer and/or machine readable instructions
  • a tangible machine readable storage medium such as a hard disk drive, a flash memory, a read-only memory (ROM), a compact disk (CD), a digital versatile disk (DVD), a cache, a random-access memory (RAM) and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently
  • tangible machine readable storage medium is expressly defined to include any type of machine readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media.
  • computer readable storage medium and “machine readable storage medium” are used interchangeably. Additionally or alternatively, the example processes of FIGS.
  • Non-transitory computer and/or machine readable medium such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a cache, a random-access memory and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information).
  • a non-transitory computer and/or machine readable medium such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a cache, a random-access memory and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information).
  • a non-transitory machine readable medium is expressly defined to include any type of machine readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission
  • FIG. 7 is a block diagram of an example processor platform 700 capable of executing the instructions of FIGS. 4, 5 , and/or 6 to implement the DOM transaction manager 110 of FIG. 2 .
  • the example processor platform 700 may be or may be included in any type of apparatus, such as a server, a personal computer, a mobile device or any other type of computing device.
  • the processor platform 700 of the illustrated example of FIG. 7 includes a processor 712 .
  • the processor 712 of the illustrated example is hardware.
  • the processor 712 can be implemented by at least one integrated circuit, logic circuit, microprocessor or controller from any desired family or manufacturer.
  • the processor 712 of the illustrated example includes a local memory 713 (e.g., a cache).
  • the processor 712 of the illustrated example is in communication with a main memory including a volatile memory 714 and a non-volatile memory 716 via a bus 718 .
  • the volatile memory 714 may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM) and/or any other type of random access memory device.
  • the non-volatile memory 716 may be implemented by flash memory, a persistent, byte-addressable memory accessible via a memory fabric and/or any other desired type of non-volatile memory device. Access to the main memory 714 , 716 is controlled by a memory controller.
  • the processor platform 700 of the illustrated example also includes an interface circuit 720 .
  • the interface circuit 720 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), and/or a peripheral component interconnect (PCI) express interface.
  • At least one input device 722 is connected to the interface circuit 720 .
  • the input device(s) 722 permit(s) a user to enter data and commands into the processor 712 .
  • the input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, and/or a voice recognition system.
  • At least one output device 724 is also connected to the interface circuit 720 of the illustrated example.
  • the output device(s) 724 can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display, a cathode ray tube display (CRT), a touchscreen, a tactile output device, a light emitting diode (LED), a printer and/or speakers).
  • the interface circuit 720 of the illustrated example thus, may include a graphics driver card, a graphics driver chip or a graphics driver processor.
  • the interface circuit 720 of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem and/or network interface card to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network 726 (e.g., an Ethernet connection, a digital subscriber line (DSL), a telephone line, coaxial cable, a cellular telephone system, etc.).
  • a communication device such as a transmitter, a receiver, a transceiver, a modem and/or network interface card to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network 726 (e.g., an Ethernet connection, a digital subscriber line (DSL), a telephone line, coaxial cable, a cellular telephone system, etc.).
  • DSL digital subscriber line
  • the processor platform 700 of the illustrated example also includes at least one mass storage device 728 for storing executable instructions (e.g., software) and/or data.
  • executable instructions e.g., software
  • Examples of such mass storage device(s) 728 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, RAID systems, and digital versatile disk (DVD) drives.
  • the coded instructions 732 implementing the processes of FIGS. 4, 5 , and/or 6 may be stored in the mass storage device 728 , in the local memory 713 in the volatile memory 714 , in the non-volatile memory 716 , and/or on a removable tangible machine readable storage medium such as a CD or DVD.
  • a web crawler may navigate between events of a DOM without loading and executing an entire web application for each transition between states of the DOM.
  • a transaction mechanism is disclosed herein that records atomic state changes that are replayed or rolled back to facilitate DOM transition navigation without application re-execution.
  • examples herein allow for isolation of event executions that facilitates traversing an applications event space without accumulating state transitions. Tractability may be furthered using examples herein by persisting transactions to disk to relieve memory pressure on a web crawler.
US15/680,418 2016-08-31 2017-08-18 Document object model transaction crawler Active 2038-11-27 US10769234B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/680,418 US10769234B2 (en) 2016-08-31 2017-08-18 Document object model transaction crawler

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201662382149P 2016-08-31 2016-08-31
US15/680,418 US10769234B2 (en) 2016-08-31 2017-08-18 Document object model transaction crawler

Publications (2)

Publication Number Publication Date
US20180060446A1 US20180060446A1 (en) 2018-03-01
US10769234B2 true US10769234B2 (en) 2020-09-08

Family

ID=59738256

Family Applications (1)

Application Number Title Priority Date Filing Date
US15/680,418 Active 2038-11-27 US10769234B2 (en) 2016-08-31 2017-08-18 Document object model transaction crawler

Country Status (3)

Country Link
US (1) US10769234B2 (zh)
EP (1) EP3291109A1 (zh)
CN (1) CN107798051A (zh)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107146082B (zh) * 2017-05-27 2021-01-29 北京小米移动软件有限公司 交易记录信息获取方法、装置及计算机可读存储介质
CN110290114B (zh) * 2019-06-04 2020-09-08 武汉大学 一种基于预警信息的漏洞自动化防护方法及系统
US11562037B2 (en) 2019-09-18 2023-01-24 International Business Machines Corporation Crawlability of single page applications
CN111078519A (zh) * 2019-12-13 2020-04-28 杭州安恒信息技术股份有限公司 异常监控行为回溯的方法、装置和电子设备

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070006078A1 (en) * 2005-07-01 2007-01-04 Microsoft Corporation Declaratively responding to state changes in an interactive multimedia environment
US20130290786A1 (en) * 2012-04-26 2013-10-31 International Business Machines Corporation Automated testing of applications with scripting code
US20140075563A1 (en) 2011-05-31 2014-03-13 Shawn Morgan Simpson Automated security testing
WO2014190427A1 (en) 2013-05-28 2014-12-04 International Business Machines Corporation Identifying client states
US9208235B1 (en) 2013-03-11 2015-12-08 Symantec Corporation Systems and methods for profiling web applications
US10372785B2 (en) * 2016-07-29 2019-08-06 Microsoft Technology Licensing, Llc Client-side modularization of a requested webpage
US10430212B1 (en) * 2016-07-13 2019-10-01 Screenshare Technology Ltd. Method for recording, editing and reproduction of computer session

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8630997B1 (en) * 2009-03-05 2014-01-14 Cisco Technology, Inc. Streaming event procesing
CN105824965A (zh) * 2016-04-01 2016-08-03 无锡中科富农物联科技有限公司 基于动态爬虫技术的数据源发现方法

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070006078A1 (en) * 2005-07-01 2007-01-04 Microsoft Corporation Declaratively responding to state changes in an interactive multimedia environment
US20140075563A1 (en) 2011-05-31 2014-03-13 Shawn Morgan Simpson Automated security testing
US20130290786A1 (en) * 2012-04-26 2013-10-31 International Business Machines Corporation Automated testing of applications with scripting code
US9208235B1 (en) 2013-03-11 2015-12-08 Symantec Corporation Systems and methods for profiling web applications
WO2014190427A1 (en) 2013-05-28 2014-12-04 International Business Machines Corporation Identifying client states
US20160110455A1 (en) * 2013-05-28 2016-04-21 International Business Machines Corporation Identifying client states
US10430212B1 (en) * 2016-07-13 2019-10-01 Screenshare Technology Ltd. Method for recording, editing and reproduction of computer session
US10372785B2 (en) * 2016-07-29 2019-08-06 Microsoft Technology Licensing, Llc Client-side modularization of a requested webpage

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
Acunetix Inc., Deep Crawling of Html5 & Javascript Websites with Acunetix Deepscan Technology, [downloaded from the Internet May 17, 2015], 5 pages.
Bezemer, C-P., et al., Automated Security Testing of Web Widget Interactions, Delft University of Technology, Report TUD-SERG-2009-011, Aug. 23-28, 2009, 14 pages.
Deursen, A.V., et al., Crawl-based Analysis of Web Applications: Prospects and Challenges, Delft University of Technology, 2014, 14 pages.
European Patent Office, EP Search Report for Appl. No. 17188285.5-1222 dated Jan. 24, 2018 (10 pages).
Mesbah et al; Crawling Ajax-Based Web Application through Dynamic Analysis of User Interface State Changes; ACM Transactions on the Web, vol. 6, No. 1, Article 3, Publication date: Mar. 2012 (30 pages).
Pellegrino, G., et al., JÂk: Using Dynamic Analysis to Crawl and Test Modern Web Applications, Saarland University, Sep. 8, 2015, 22 pages.
Zhang et al; AJAX Crawling Scheme Based on Document Object Model; 2012 Fourth International Conference on Computational and Information Sciences; Dept. of Information Engineering; Xuzhou College of Industrial Technology; 2012 IEEE (4 pages).

Also Published As

Publication number Publication date
US20180060446A1 (en) 2018-03-01
EP3291109A1 (en) 2018-03-07
CN107798051A (zh) 2018-03-13

Similar Documents

Publication Publication Date Title
US10769234B2 (en) Document object model transaction crawler
US20180052940A1 (en) Serializing plug-in data in a web page
US9418243B2 (en) Invoking a private browsing mode by selection of a visual control element within a browser tab
US9600400B1 (en) Performance testing of web application components using image differentiation
US10749927B2 (en) Webpage loading method, apparatus and system
US8326922B2 (en) Method for server-side logging of client browser state through markup language
US9195572B2 (en) Systems and methods for identifying user interface (UI) elements
US10353721B2 (en) Systems and methods for guided live help
CN110069683B (zh) 一种基于浏览器爬取数据的方法及装置
US20130212465A1 (en) Postponed rendering of select web page elements
CN109376291B (zh) 一种基于网络爬虫的网站指纹信息扫描的方法及装置
US9910992B2 (en) Presentation of user interface elements based on rules
JP2015135680A (ja) Ａｊａｘウェブページコンテンツを取得する方法およびシステム
CN104956362A (zh) 分析web应用程序的结构
US20140236919A1 (en) Executing a fast crawl over a computer-executable application
US20130018912A1 (en) Method and system for searching for a web document
WO2018010573A1 (zh) 一种脚本生成方法与装置
EP3745292A1 (en) Hidden link detection method and apparatus for website
EP3851981A1 (en) Page processing method and apparatus, electronic device and computer readable medium
US20210334113A1 (en) Method and device for lazy loading of js scripts
CN113868502A (zh) 一种页面爬虫方法、装置、电子设备及可读存储介质
CN113076501A (zh) 一种页面处理方法、存储介质及设备
CN111949903A (zh) 一种网页数据采集方法、装置、设备及可读存储介质
CN104268246A (zh) 生成访问互联网站点指令脚本的方法及访问方法和装置
US10289613B2 (en) Element identifier generation

Legal Events

Date Code Title Description
AS Assignment

Owner name: HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP, TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SWEDLUND, THOMAS CHRISTOPHER;WILLIAMS, KEVIN ALLEN;KING, BRIAN CHARLES;SIGNING DATES FROM 20160831 TO 20161117;REEL/FRAME:043333/0609

AS Assignment

Owner name: ENTIT SOFTWARE LLC, CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP;REEL/FRAME:043675/0001

Effective date: 20170405

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

AS Assignment

Owner name: MICRO FOCUS LLC, CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:ENTIT SOFTWARE LLC;REEL/FRAME:050004/0001

Effective date: 20190523

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

AS Assignment

Owner name: JPMORGAN CHASE BANK, N.A., NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNORS:MICRO FOCUS LLC;BORLAND SOFTWARE CORPORATION;MICRO FOCUS SOFTWARE INC.;AND OTHERS;REEL/FRAME:052294/0522

Effective date: 20200401

Owner name: JPMORGAN CHASE BANK, N.A., NEW YORK

Free format text: SECURITY AGREEMENT;ASSIGNORS:MICRO FOCUS LLC;BORLAND SOFTWARE CORPORATION;MICRO FOCUS SOFTWARE INC.;AND OTHERS;REEL/FRAME:052295/0041

Effective date: 20200401

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: NETIQ CORPORATION, WASHINGTON

Free format text: RELEASE OF SECURITY INTEREST REEL/FRAME 052295/0041;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:062625/0754

Effective date: 20230131

Owner name: MICRO FOCUS SOFTWARE INC. (F/K/A NOVELL, INC.), MARYLAND

Free format text: RELEASE OF SECURITY INTEREST REEL/FRAME 052295/0041;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:062625/0754

Effective date: 20230131

Owner name: MICRO FOCUS LLC, CALIFORNIA

Free format text: RELEASE OF SECURITY INTEREST REEL/FRAME 052295/0041;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:062625/0754

Effective date: 20230131

Owner name: NETIQ CORPORATION, WASHINGTON

Free format text: RELEASE OF SECURITY INTEREST REEL/FRAME 052294/0522;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:062624/0449

Effective date: 20230131

Owner name: MICRO FOCUS SOFTWARE INC. (F/K/A NOVELL, INC.), WASHINGTON

Free format text: RELEASE OF SECURITY INTEREST REEL/FRAME 052294/0522;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:062624/0449

Effective date: 20230131

Owner name: MICRO FOCUS LLC, CALIFORNIA

Free format text: RELEASE OF SECURITY INTEREST REEL/FRAME 052294/0522;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:062624/0449

Effective date: 20230131

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4