US9405692B2 - Data processing performance enhancement in a distributed file system - Google Patents

Data processing performance enhancement in a distributed file system Download PDF

Info

Publication number
US9405692B2
US9405692B2 US13/426,466 US201213426466A US9405692B2 US 9405692 B2 US9405692 B2 US 9405692B2 US 201213426466 A US201213426466 A US 201213426466A US 9405692 B2 US9405692 B2 US 9405692B2
Authority
US
United States
Prior art keywords
data
machine
system
access request
method
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US13/426,466
Other versions
US20130254246A1 (en
Inventor
Todd Lipcon
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cloudera Inc
Original Assignee
Cloudera Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Cloudera Inc filed Critical Cloudera Inc
Priority to US13/426,466 priority Critical patent/US9405692B2/en
Assigned to Cloudera, Inc. reassignment Cloudera, Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LIPCON, TODD
Publication of US20130254246A1 publication Critical patent/US20130254246A1/en
Application granted granted Critical
Publication of US9405692B2 publication Critical patent/US9405692B2/en
Application status is Active legal-status Critical
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/08Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0804Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches with main memory updating
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/08Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0866Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/08Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0866Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
    • G06F12/0871Allocation or management of cache space
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14Handling requests for interconnection or transfer
    • G06F13/20Handling requests for interconnection or transfer for access to input/output bus
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems
    • G06F16/1824Distributed file systems implemented using Network-attached Storage [NAS] architecture
    • G06F16/183Provision of network file services by network file servers, e.g. by using NFS, CIFS
    • G06F17/30203
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/21Employing a record carrier using a specific recording technology
    • G06F2212/214Solid state disk
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/60Details of cache memory
    • G06F2212/603Details of cache memory of operating mode, e.g. cache mode or local memory mode

Abstract

Systems and methods of data processing performance enhancement are disclosed. One embodiment includes, invoking operating system calls to optimize cache management by an I/O component; wherein, the operating system calls are invoked to perform one or more of; proactive triggering of readaheads for sequential read requests of a disk; purging data out of buffer cache after writing to the disk or performing sequential reads from the desk; and/or eliminating a delay between when a write is performed and when written data from the write is flushed to the disk from the buffer cache.

Description

BACKGROUND OF THE DISCLOSURE

1. Technical Field

The present disclosure relates generally to distributed computing, and more specifically, to techniques for enhancing the performance of a distributed file system in a computer cluster.

2. Description of the Related Art

Increasingly large amounts of data are generated every day online by users interacting with one another, with applications, data, websites, etc. Although distributed computing has been adopted for management analysis of large amounts of data, continuous optimizations to improve system performance remains critical to keep up with the rapidity with which data is being generated.

SUMMARY OF THE DISCLOSURE

Embodiments of the present disclosure include systems and methods for enhancing data processing performance in a distributed file system. Some embodiments of the present disclosure include a method that comprises invoking operating system calls to optimize cache management by an I/O component. The operating system calls can be invoked to perform, for example, one or more of: (1) proactive triggering of readaheads for sequential read requests of a disk: (2) purging data out of buffer cache after writing to the disk or performing sequential reads from the desk; and/or (3) eliminating a delay between when a write is performed and when written data from the write is flushed to the disk from the buffer cache.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of client devices that generate datasets (log data) to be collected for storage and processing via interacting nodes in various tiers in the computing cluster, in some instances, through a network.

FIG. 2A depicts sections of a disk from which readahead can be performed for sequential read requests by the I/O scheduler.

FIG. 2B depicts a portion of the cache with buffered data written to or read from the disk that can be dropped by the I/O scheduler.

FIG. 2C depicts buffer cached written data being immediately flushed to disk.

FIG. 3A depicts a diagram of a single request being processed in a single connection between a client and a data node.

FIG. 3B depicts a diagram of multiple requests being processed in a single connection.

FIG. 4 depicts an example block diagram of the components of a machine in machine cluster able to enhance the performance of data processing by the distributed file system.

FIG. 5A depicts a flowchart of an example process for deterministically instructing readahead in a sequential read request.

FIG. 5B depicts a flowchart of an example process for dropping the buffer cache behind writes and sequential reads.

FIG. 5C depicts a flowchart of an example process for immediate flush of written data to disk.

FIG. 6 depicts a flowchart of an example process for reusing a connection for multiple client requests at a data node.

FIG. 7 shows a diagrammatic representation of a machine in the example form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed.

DETAILED DESCRIPTION

The following description and drawings are illustrative and are not to be construed as limiting. Numerous specific details are described to provide a thorough understanding of the disclosure. However, in certain instances, well-known or conventional details are not described in order to avoid obscuring the description. References to one or an embodiment in the present disclosure can be, but not necessarily are, references to the same embodiment; and, such references mean at least one of the embodiments.

Reference in this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Moreover, various features are described which may be exhibited by some embodiments and not by others. Similarly, various requirements are described which may be requirements for some embodiments but not other embodiments.

The terms used in this specification generally have their ordinary meanings in the art, within the context of the disclosure, and in the specific context where each term is used. Certain terms that are used to describe the disclosure are discussed below, or elsewhere in the specification, to provide additional guidance to the practitioner regarding the description of the disclosure. For convenience, certain terms may be highlighted, for example using italics and/or quotation marks. The use of highlighting has no influence on the scope and meaning of a term; the scope and meaning of a term is the same, in the same context, whether or not it is highlighted. It will be appreciated that same thing can be said in more than one way.

Consequently, alternative language and synonyms may be used for any one or more of the terms discussed herein, nor is any special significance to be placed upon whether or not a term is elaborated or discussed herein. Synonyms for certain terms are provided. A recital of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification including examples of any terms discussed herein is illustrative only, and is not intended to further limit the scope and meaning of the disclosure or of any exemplified term. Likewise, the disclosure is not limited to various embodiments given in this specification.

Without intent to further limit the scope of the disclosure, examples of instruments, apparatus, methods and their related results according to the embodiments of the present disclosure are given below. Note that titles or subtitles may be used in the examples for convenience of a reader, which in no way should limit the scope of the disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In the case of conflict, the present document, including definitions will control.

Embodiments of the present disclosure include systems and methods for enhancing data processing performance in a distributed file system.

FIG. 1 illustrates a block diagram of client devices 102A-N that generate datasets (log data) to be collected for storage and processing via interacting nodes in various tiers in the computing cluster 100, in some instances, through a network 106.

The client devices 102A-N can be any system and/or device, and/or any combination of devices/systems that is able to establish a connection with another device, a server and/or other systems. The client devices 102A-N typically include display or other output functionalities to present data exchanged between the devices to a user. For example, the client devices and content providers can be, but are not limited to, a server desktop, a desktop computer, a thin-client device, an internet kiosk, a computer cluster, a mobile computing device such as a notebook, a laptop computer, a handheld computer, a mobile phone, a smart phone, a PDA, a Blackberry device, a Treo, a tablet, an iPad, a thin client, and/or an iPhone, etc. In one embodiment, the client devices 102A-N are coupled to a network 106. In some embodiments, the client devices may be directly connected to one another.

In one embodiment, users interact with user devices 102A-N (e.g., machines or devices). As a results of the user interaction, the devices 102A-N can generate datasets such as log files to be collected and aggregated. The file can include logs, information, and other metadata about clicks, feeds, status updates, data from applications, and associated properties and attributes.

User devices 102A-N can have nodes executing or running thereon that collect the datasets that are user-generated or machine-generated, for example, based on user-interaction with applications or websites running on the devices. Such nodes can interact and/or communicate with one or more other nodes (e.g., either running on the same device/machine or another device/machine (e.g., machine/device 104) to facilitate collection and aggregation of datasets thus generated. In one embodiment, the datasets are eventually written to a file and stored, for example, in storage (e.g., repository 130) on a physical disk.

Additionally, functionalities and properties can be assigned to the nodes such that various analytics can be performed on the collected dataset and additional information can be extracted or embedded. The dataflow among nodes can be configured at a master node. In one embodiment, the nodes executed on the machines 102 or 104 can contact the master(s) to obtain configuration information, which have been set by default or configured by a user The master can be executed on the same devices 102A-N, 104, or at the computing cluster 100. One or multiple masters can be involved in the mapping of data flow among the nodes and various machines.

The repository 130 may be managed by a file system. The file system can be distributed (e.g., the Hadoop Distributed File System (HDFS)). Results of any analytics performed by the machines 104 and/or the computing cluster 100 can also be written to storage. Data/metadata extracted from the collected dataset may be written to storage 130 as well.

The network 106, over which the client devices 102A-N, 104, host, and the nodes and masters therein communicate may be a telephonic network, an open network, such as the Internet, or a private network, such as an intranet and/or the extranet. For example, the Internet can provide file transfer, remote log in, email, news, RSS, and other services through any known or convenient protocol, such as, but is not limited to the TCP/IP protocol, Open System Interconnections (OSI), FTP, UPnP, iSCSI, NSF, ISDN, PDH, RS-232, SDH, SONET, etc.

The network 106 can be any collection of distinct networks operating wholly or partially in conjunction to provide connectivity to the client devices, host server, and may appear as one or more networks to the serviced systems and devices. In one embodiment, communications to and from the client devices 102A-N can be achieved by, an open network, such as the Internet, or a private network, such as an intranet and/or the extranet. In one embodiment, communications can be achieved by a secure communications protocol, such as secure sockets layer (SSL), or transport layer security (TLS).

The term “Internet” as used herein refers to a network of networks that uses certain protocols, such as the TCP/IP protocol, and possibly other protocols such as the hypertext transfer protocol (HTTP) for hypertext markup language (HTML) documents that make up the World Wide Web (the web). Content is often provided by content servers, which are referred to as being “on” the Internet. A web server, which is one type of content server, is typically at least one computer system which operates as a server computer system and is configured to operate with the protocols of the World Wide Web and is coupled to the Internet. The physical connections of the Internet and the protocols and communication procedures of the Internet and the web are well known to those of skill in the relevant art. For illustrative purposes, it is assumed the network 106 broadly includes anything from a minimalist coupling of the components illustrated in the example of FIG. 1, to every component of the Internet and networks coupled to the Internet.

In addition, communications can be achieved via one or more wireless networks, such as, but is not limited to, one or more of a Local Area Network (LAN), Wireless Local Area Network (WLAN), a Personal area network (PAN), a Campus area network (CAN), a Metropolitan area network (MAN), a Wide area network (WAN), a Wireless wide area network (WWAN), Global System for Mobile Communications (GSM), Personal Communications Service (PCS), Digital Advanced Mobile Phone Service (D-Amps), Bluetooth, Wi-Fi, Fixed Wireless Data, 2G, 2.5G, 3G networks, enhanced data rates for GSM evolution (EDGE), General packet radio service (GPRS), enhanced GPRS, messaging protocols such as, TCP/IP, SMS, MMS, extensible messaging and presence protocol (XMPP), real time messaging protocol (RTMP), instant messaging and presence protocol (IMPP), instant messaging, USSD, IRC, or any other wireless data networks or messaging protocols.

The client devices 102A-N can be coupled to the network (e.g., Internet) via a dial up connection, a digital subscriber loop (DSL, ADSL), cable modem, and/or other types of connection. Thus, the client devices 102A-N can communicate with remote servers (e.g., web server, host server, mail server, and instant messaging server) that provide access to user interfaces of the World Wide Web via a web browser, for example.

The repository 130 can store software, descriptive data, images, system information, drivers, collected datasets, aggregated datasets, log files, analytics of collected datasets, enriched datasets, user data, metadata, etc. The repository may be managed by a database management system (DBMS), for example but not limited to, Oracle, DB2, Microsoft Access, Microsoft SQL Server, MySQL, FileMaker, etc.

The repositories can be implemented via object-oriented technology and/or via text files, and can be managed by a distributed database management system, an object-oriented database management system (OODBMS) (e.g., ConceptBase, FastDB Main Memory Database Management System, JDOInstruments, ObjectDB, etc.), an object-relational database management system (ORDBMS) (e.g., Informix, OpenLink Virtuoso, VMDS, etc.), a file system, and/or any other convenient or known database management package.

In one embodiment, the repository 130 is managed by a distributed file system or network file system that allows access to files from multiple hosts/machines over a network. The distributed file system can include by way of example, the Hadoop Distributed File system (HDFS). Other file systems can be used as well, for example, through integration of Hadoop's interface which provides an abstraction layer for the file system. For example, a local file system where a node resides can be used. The HDFS native distributed file system can also be used. In addition, S3 (a remote file system hosted by Amazon web services), FTP, and KFS (Kosmos file system—another distributed file system) can also be used. Clients can also write to network file systems (NFS), or other distributed file systems.

In general, the user devices 102 and 104 are able to write files (e.g., files including by way of example, collected and aggregated datasets/logs/log files) to the repository 130, either through the network 106 or without utilizing the network 106. Any device 102 or machine 104, or a machine in the machine cluster 100 can be implemented on a known or convenient computer system, such as is illustrated in FIG. 7.

FIG. 2A depicts sections of a disk 210 from which readahead can be performed for sequential read requests by the I/O scheduler.

Sequential read requests can be optimized for reads and writes in large scale data processing systems (e.g., systems like Hadoop MapReduce, Apache Hive, Apache Pig, etc). For example, for a sequential read of disk 210 to read datablock 202, readahead of data blocks 204 and/or 206, and or later data blocks can be proactively triggered. The proactive triggering causes the readaheads to be deterministically performed in lieu being of heuristically performed thus allowed the readahead to be performed quicker than it otherwise would, thus enhancing system performance. In distributed file systems where a large portion of the read commands are part of sequential reads of large files (e.g., data chunks between 1-5 MB or 5-10 MB, >10 MB, >50 MB, >100 MB, or more generally on the order of 100's MB), automatic readahead can significantly reduce read time and system through put by allowing the disk to seek less (e.g., the underlying disk hardware sees larger units of reads/writes). This allows the system to handle more concurrent streams of data given the same hardware resources.

FIG. 2B depicts a portion of the cache 250 with buffered data 252 written to or read from the disk 210 that can be dropped by the I/O scheduler.

The OS can be instructed to purged data out of buffer cache after writing to the disk or performing sequential reads from the desk. Some operating systems by default will automatically buffer recently read/written data to optimize a re-read or re-write that occurs after the read/write. However, in systems such as a distributed file system or the Hadoop distributed file system or other applications where the files being read/written are large, this caching becomes an overhead. As such, in this application, the I/O scheduler is explicitly instructed to purge the buffered data 252 which was cached in 250 as a result of the data 212 in disk 210 being read/written. Another advantage of purging the buffer cache frees up the buffer cache for other more useful uses—in the MapReduce example, the “intermediate output files” can now use the buffer cache more effectively. Therefore, purging the buffer cache speeds up the system by enabling other processes to use the buffer.

FIG. 2C depicts buffer cached 250 written data 252 being immediately flushed to disk 210.

The immediate flushing eliminates the delay that is implemented by default in I/O schedulers of some operating systems to support re-writes of the written data (e.g., the written data 252 stored in the buffer cache 250). However, in a distributed file system (e.g., Hadoop distributed file system) where re-writes are not supported, the written data 252 is immediately flushed from the buffer cache 250 to the disk 210 to enhance performance by decreasing write time. The written data then occupies storage location 212 in the disk 210. In one embodiment, this is performed by instructing the operating system to drop all data from offset 0 through the beginning of the current window (a data chunk of a predetermined size (e.g., 8 MB) preceding a current offset that is being written to in the file). In addition, the system also instructs the OS to drop the data in the current window (or the preceding 8 MB of written data) from the cache. The window size to drop cached data can be 8 MB (e.g., the size of the ‘current window’) can be larger or smaller (e.g., 1 MB, 2 MB, 16 MB, 32 MB, 64 MB, or larger).

FIG. 3A depicts a diagram 300 of a single request being processed in a single connection between a client 302 and a data node 322. For example, each read request is sent over different connections between the client 302 and the data node 322. In the case that an operation leaves the stream in a well-defined state (e.g., a client reads to the end of the requested region of data) the same connection could be reused for a second operation. For example, in the random read case, each request is usually only ˜64-100 KB, and generally not entire blocks. This optimization improves random read performance significantly. The optimization for random read performance enhancement is illustrated in FIG. 3B where an established connection between a client 352 and the data node 372 is held open for one read operation and for a subsequent read operation. FIG. 3B depicts a diagram of multiple requests being processed in a single connection.

FIG. 4 depicts an example block diagram of the components of a machine 400 in a machine cluster having components able to enhance data processing performance of the distributed file system.

The machine 400 can include, a network interface 402, an I/O manager 404 having an I/O scheduler 405, a readahead engine 406, a buffer purge engine 408, a write synchronization engine 410, and/or a connection manager 412. Additional or less modules/components may be included.

As used in this paper, a “module,” a “manager”, a “handler”, or an “engine” includes a dedicated or shared processor and, typically, firmware or software modules that are executed by the processor. Depending upon implementation-specific or other considerations, the module, manager, hander, or engine can be centralized or its functionality distributed. The module, manager, hander, or engine can include special purpose hardware, firmware, or software embodied in a computer-readable medium for execution by the processor. As used in this paper, a computer-readable medium or computer-readable storage medium is intended to include all mediums that are statutory (e.g., in the United States, under 35 U.S.C. 101), and to specifically exclude all mediums that are non-statutory in nature to the extent that the exclusion is necessary for a claim that includes the computer-readable (storage) medium to be valid. Known statutory computer-readable mediums include hardware (e.g., registers, random access memory (RAM), non-volatile (NV) storage, to name a few), but may or may not be limited to hardware.

The machine 400 receives and processes client requests through the network interface 402. The machine 400 can also interact with other machines in a computing cluster and/or with disks (e.g., disk 434) over the network interface 402. In general, computing cluster which the machine 400 is a part of, is a distributed file system cluster (e.g., Hadoop distributed file system). The machine 400 can include a processor (not illustrated), a disk 434, and memory having stored there on instructions which when executed by the processor, is able to perform various optimization techniques in enhancing data processing performance in the distributed file system, for example, by way of the I/O manager 404 and/or the connection manager 412.

In one embodiment, the readahead engine 406 can be managed by the I/O manager 404 or scheduler 405 to proactively trigger readaheads for sequential read requests or any other read requests, whether or not it can be determined that the read is part of a sequential read (as shown in the diagram of FIG. 2A). The write synchronization engine 410 can, for example, commit the written data to disk 434 immediately by eliminating any delay between when a write is performed and when written data from the write is flushed from the buffer 432 to the disk (as shown in the diagram of FIG. 2B). In one embodiment, the buffer purge engine 408 can actively purge data out of buffer cache 432 after writing to the disk 434 or performing sequential reads from the disk 434, for example, at the instruction of the I/O manager 404 (as shown in the diagram of FIG. 2C).

In one embodiment, the connection manager 412 is able to manage connections of the machine 400 with clients and optimize the performance of random reads by a client. For example, for a given client, the connection manager 412 can optimize random read performance of data on the disk 434 by holding an established connection between a client and the machine 400 used for one operation for one or more subsequent operations (as example of this process is illustrated in the example of FIG. 3B).

In one embodiment, the distributed file system performance is further enhanced by decreasing checksum overhead in speeding up the distributed file system read path. This can be implemented by performing the checksum in hardware (e.g., by a processor supporting CRC 32). In some instances, the checksum implementation can be modified to use zlib polynomial from iSCSI polynomial. For example, the specific optimization can be switched from using CRC32 to CRC32C such that hardware support in SSE4.2-enabled processors could be taken advantage of.

This and other modules or engines described in this specification are intended to include any machine, manufacture, or composition of matter capable of carrying out at least some of the functionality described implicitly, explicitly, or inherently in this specification, and/or carrying out equivalent functionality.

FIG. 5A depicts a flowchart of an example process for deterministically instructing readahead in a sequential read request.

In process 502, a sequential read request is detected in a distributed file system. When a read request is detected in DFS or the Hadoop distributed file system, operating system calls (e.g., Linux calls) can be invoked to proactively trigger readaheads in response to the read request. The proactive triggering causes the readaheads to be deterministically performed in lieu of being heuristically performed, for example, by the operating system. In some cases, operating systems may have heuristics for detecting sequential reads from random reads and apply readahead in the case that certain criteria is met based on observations of multiple read requests. By performing deterministic trigger, this ensures that readahead is initiated and that it is initiated with minimal time delay resulting from the need apply the heuristics before deciding to readahead.

To proactively trigger readaheads, in Linux or Unix-based operating systems or other systems conforming to (or partially conforming to) the POSIX.1-2001 standard, the posix_fadvise( ) call can be used with the POSIX_FADV_WILLNEED flag to indicate data also expected in the future based on the current read request. In other operating systems, any command which allows an application to tell the kernel how it expects to use a file handle, so that the kernel can choose appropriate read-ahead and caching techniques for access to the corresponding can be used to trigger proactive readahead.

In process 504, a byte range of the file to read ahead of the current request is specified. In process 506, the OS is instructed to read the file at the specified byte range in advance of the current request. An example of readahead data sets to be read in advance of the data requested in a given read command is illustrated in the example of FIG. 2A.

The specified byte range can be predetermined, set by default, dynamically adjusted/determined, and/or (re) configurable by a client or system administrator, or a master node in the cluster. In one embodiment, given that any read request in the Hadoop distributed file system is likely a sequential read request, the proactive readahead is typically performed/requested to proactively retrieve data from the disk in advance of subsequent requests to speed up reads from HDFS.

FIG. 5B depicts a flowchart of an example process for dropping the buffer cache behind writes and sequential reads.

In process 512, data read or write event is detected in a distributed file system. Upon a read or write event, operating system calls (e.g., operating system calls which are native to the operating system) can be invoked to optimize cache management by an I/O component. In one embodiment, the cache management is optimized for reads and writes in Hbase or for large sequential reads and writes. In general, sequential read requests can be optimized for reads and writes in large scale data processing systems (including by way of example systems like Hadoop MapReduce, Apache Hive, Apache Pig, etc) “Large” sequential reads and writes, in general, can include reads and writes of data chunks between 1-5 MB or 5-10 MB, >10 MB, >50 MB, or in the 100's MB range, or data chunks greater than 500 kB, or data chunks greater than ˜250 kB, depending on the application or system resources/availability. In one embodiment, if a read is larger than a threshold size (e.g., 256 KB), readahead is performed. However, the readahead will generally not pass the client's requested boundary. For example: if a client requests bytes 0 through 256 MB the system can readahead 4 MB chunks ahead of the reader's current position, as the reader streams through the file, if a client requests bytes 0 through 1 MB, the system will readahead only 1 MB (even if the file itself is larger than 1 MB).

In process 514, read or written data is cached in the OS buffer cache. In general, the operating system will automatically cache read or written data into the buffer assuming that there will be subsequent commands to re-read the previously read or written data. A diagram showing read or written data in the disk and the corresponding cached data in the buffer is illustrated in the example of FIG. 2B.

In process 516, when it is detected that a specified file size has been read or written (e.g., on the order of 1 MB, 2 MB, 4 MB, or >10 MB, for example), in process 518, the OS is instructed to purge the buffer cache of the read or written data. In one embodiment, the data size of the data that is purged from the buffer cache is configurable or dynamically adjustable. For example, data size of the data that is purged from the buffer cache can be between 4-8 MB, 8-16 MB, 16-32 MB, or other sizes.

FIG. 5C depicts a flowchart of an example process for immediate flush of written data to disk.

In process 522, a write event is detected in a distributed file system. In general, in some operating systems, the I/O scheduler will automatically hold the written data in the buffer cache for some amount of time (e.g., 10-40 seconds) before writing (e.g., committing or flushing) the written data to the disk. This can be an optimization technique in the event that frequent rewrites occur such that one write can be performed instead of writing and rewriting the data to the disk multiple times in the event of re-writes.

In a distributed file system such as the Hadoop distributed file system or other file systems where re-writes are not allowed or not supported, this delay in committing data to the disk becomes an overhead since it unnecessarily occupies the cache and slows the write event. Therefore, in one embodiment, to enhance data processing performance, in process 524, the OS is instructed to flush the written data to disk without delay. Therefore, in operation, a write event can be expedited since the delay between when a write is performed and when written data from the write is flushed to the disk from the buffer cache is now eliminated. Another advantage of purging the buffer cache frees up the buffer cache for other more useful uses—in the MapReduce example, the “intermediate output files” can now use the buffer cache more effectively. Therefore, purging the buffer cache speeds up the system by enabling other processes to use the buffer.

In Linux, the SYNC FILE RANGE command can be used with a specified file byte size range to instruct the I/O scheduler to immediately start flushing the data to the disk. The “msync” API can be used along with the MS_ASYNC flag for a similar implementation on other systems conforming to POSIX.1-2001. This will enable the I/O scheduler to also expedite the scheduling of subsequent writes without delay and overhead cache use. A diagram showing flushing of written data to disk is depicted in the example of FIG. 2C.

FIG. 6 depicts a flowchart of an example process for reusing a connection for multiple client requests at a datanode to optimize a distributed file system for random read performance.

The random read performance can be optimized by holding an established connection with the given machine used for one read operation for a subsequent read operation. In general, random read performance can be optimized for reading data that is less than 50 kB, less than 100 kB, or less than 1 MB. In process 602, a client sends a random read request to a node in a cluster. In process 604, a connection with the node is established. In process 606, the operation which was requested by the client is completed.

In process 608, the connection is held open, for example, in the case that an operation leaves the stream in a well-defined state (e.g., if a client successfully reads to the end of a block), for use by additional operations. A well-defined state is any state in which the server is able to successfully respond to the entirety of the request, and the client fully reads the response from the server. An example of an undefined state is if the client initially requests to read 3 MB, but then only reads 1 MB of the response. At that point, it can't issue another request because there is still data coming across the pipe. Another example of an unclean state is if the client receives a timeout or another error. In the case of an error, the connection is closed and a new one is established.

In process 610, subsequent operations are sent from the client to the node using the same connection. In process 612, the connection is closed after timeout. In one embodiment, the client and the given machine have same or similarly configured timeouts. For example, the established connection is held for 0.5-1 seconds, 1-2, seconds, or 2-5 seconds for optimization of the random read performance, such that all requests from the client to the same machine within the timeout period can use the same connection. In one embodiment, the timeout is measured from the end of the last successful operation. In process 614, a connection is re-established when the client next sends a request to the node, and the process can continue at step 606. A diagrammatic example of using a single connection for multiple operations between a client and a data node is illustrated in the example of FIG. 3B.

FIG. 7 shows a diagrammatic representation of a machine in the example form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed.

In the example of FIG. 7, the computer system or machine 700 includes a processor, memory, disk, non-volatile memory, and an interface device. Various common components (e.g., cache memory) are omitted for illustrative simplicity. The computer system 700 is intended to illustrate a hardware device on which any of the components depicted in the example of FIG. 1 (and any other components described in this specification) can be implemented. The computer system or machine 700 can be of any applicable known or convenient type. The components of the computer system 700 can be coupled together via a bus or through some other known or convenient device.

The processor may be, for example, a conventional microprocessor such as an Intel Pentium microprocessor or Motorola power PC microprocessor. One of skill in the relevant art will recognize that the terms “machine-readable (storage) medium” or “computer-readable (storage) medium” include any type of device that is accessible by the processor.

The memory is coupled to the processor by, for example, a bus. The memory can include, by way of example but not limitation, random access memory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM). The memory can be local, remote, or distributed.

The bus also couples the processor to the non-volatile memory and drive unit. The non-volatile memory is often a magnetic floppy or hard disk, a magnetic-optical disk, an optical disk, a read-only memory (ROM), such as a CD-ROM, EPROM, or EEPROM, a magnetic or optical card, or another form of storage for large amounts of data. Some of this data is often written, by a direct memory access process, into memory during execution of software in the computer 900. The non-volatile storage can be local, remote, or distributed. The non-volatile memory is optional because systems can be created with all applicable data available in memory. A typical computer system will usually include at least a processor, memory, and a device (e.g., a bus) coupling the memory to the processor.

Software is typically stored in the non-volatile memory and/or the drive unit. Indeed, for large programs, it may not even be possible to store the entire program in the memory. Nevertheless, it should be understood that for software to run, if necessary, it is moved to a computer readable location appropriate for processing, and for illustrative purposes, that location is referred to as the memory in this paper. Even when software is moved to the memory for execution, the processor will typically make use of hardware registers to store values associated with the software, and local cache that, ideally, serves to speed up execution. As used herein, a software program is assumed to be stored at any known or convenient location (from non-volatile storage to hardware registers) when the software program is referred to as “implemented in a computer-readable medium.” A processor is considered to be “configured to execute a program” when at least one value associated with the program is stored in a register readable by the processor.

The bus also couples the processor to the network interface device. The interface can include one or more of a modem or network interface. It will be appreciated that a modem or network interface can be considered to be part of the computer system 1900. The interface can include an analog modem, isdn modem, cable modem, token ring interface, satellite transmission interface (e.g., “direct PC”), or other interfaces for coupling a computer system to other computer systems. The interface can include one or more input and/or output devices. The I/O devices can include, by way of example but not limitation, a keyboard, a mouse or other pointing device, disk drives, printers, a scanner, and other input and/or output devices, including a display device. The display device can include, by way of example but not limitation, a cathode ray tube (CRT), liquid crystal display (LCD), or some other applicable known or convenient display device. For simplicity, it is assumed that controllers of any devices not depicted in the example of FIG. 7 reside in the interface.

In operation, the machine 700 can be controlled by operating system software that includes a file management system, such as a disk operating system. One example of operating system software with associated file management system software is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Wash., and their associated file management systems. Another example of operating system software with its associated file management system software is the Linux operating system and its associated file management system. The file management system is typically stored in the non-volatile memory and/or drive unit and causes the processor to execute the various acts required by the operating system to input and output data and to store data in the memory, including storing files on the non-volatile memory and/or drive unit.

Some portions of the detailed description may be presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the methods of some embodiments. The required structure for a variety of these systems will appear from the description below. In addition, the techniques are not described with reference to any particular programming language, and various embodiments may thus be implemented using a variety of programming languages.

In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a laptop computer, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, an iPhone, a Blackberry, a processor, a telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.

While the machine-readable medium or machine-readable storage medium is shown in an exemplary embodiment to be a single medium, the term “machine-readable medium” and “machine-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “machine-readable medium” and “machine-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the presently disclosed technique and innovation.

In general, the routines executed to implement the embodiments of the disclosure, may be implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions referred to as “computer programs.” The computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and that, when read and executed by one or more processing units or processors in a computer, cause the computer to perform operations to execute elements involving the various aspects of the disclosure.

Moreover, while embodiments have been described in the context of fully functioning computers and computer systems, those skilled in the art will appreciate that the various embodiments are capable of being distributed as a program product in a variety of forms, and that the disclosure applies equally regardless of the particular type of machine or computer-readable media used to actually effect the distribution.

Further examples of machine-readable storage media, machine-readable media, or computer-readable (storage) media include but are not limited to recordable type media such as volatile and non-volatile memory devices, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD ROMS), Digital Versatile Disks, (DVDs), etc.), among others, and transmission type media such as digital and analog communication links.

Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” As used herein, the terms “connected,” “coupled,” or any variant thereof, means any connection or coupling, either direct or indirect, between two or more elements; the coupling of connection between the elements can be physical, logical, or a combination thereof. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word “or,” in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

The above detailed description of embodiments of the disclosure is not intended to be exhaustive or to limit the teachings to the precise form disclosed above. While specific embodiments of, and examples for, the disclosure are described above for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. For example, while processes or blocks are presented in a given order, alternative embodiments may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, or may be performed at different times. Further any specific numbers noted herein are only examples: alternative implementations may employ differing values or ranges.

The teachings of the disclosure provided herein can be applied to other systems, not necessarily the system described above. The elements and acts of the various embodiments described above can be combined to provide further embodiments.

Any patents and applications and other references noted above, including any that may be listed in accompanying filing papers, are incorporated herein by reference. Aspects of the disclosure can be modified, if necessary, to employ the systems, functions, and concepts of the various references described above to provide yet further embodiments of the disclosure.

These and other changes can be made to the disclosure in light of the above Detailed Description. While the above description describes certain embodiments of the disclosure, and describes the best mode contemplated, no matter how detailed the above appears in text, the teachings can be practiced in many ways. Details of the system may vary considerably in its implementation details, while still being encompassed by the subject matter disclosed herein. As noted above, particular terminology used when describing certain features or aspects of the disclosure should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the disclosure with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the disclosure to the specific embodiments disclosed in the specification, unless the above Detailed Description section explicitly defines such terms. Accordingly, the actual scope of the disclosure encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the disclosure under the claims.

While certain aspects of the disclosure are presented below in certain claim forms, the inventors contemplate the various aspects of the disclosure in any number of claim forms. For example, while only one aspect of the disclosure is recited as a means-plus-function claim under 35 U.S.C. §112, ¶6, other aspects may likewise be embodied as a means-plus-function claim, or in other forms, such as being embodied in a computer-readable medium. (Any claims intended to be treated under 35 U.S.C. §112, ¶6 will begin with the words “means for”.) Accordingly, the applicant reserves the right to add additional claims after filing the application to pursue such additional claim forms for other aspects of the disclosure.

Claims (21)

What is claimed is:
1. A method for enhancing performance for data processing in a distributed file system, the method comprising:
instantiating an input/output (I/O) manager on a machine among a plurality of machines that implement the distributed file system; and
utilizing the I/O manager to perform cache management optimization including:
(a) determining that the machine employs a heuristic for triggering readaheads for sequential read requests;
overriding the heuristic so as to deterministically trigger the readaheads for all sequential read requests;
(b) determining that the machine is configured to automatically cache data into a buffer on the machine after the data is accessed;
detecting that a specific size of data has been accessed;
instructing the machine to invalidate the cached data in the buffer; and
(c) determining that the machine is configured to include a time delay before committing data from the buffer to a disk on the machine;
overriding the time delay so that the machine commits the data from the buffer to the disk without the time delay.
2. The method of claim 1, wherein, the cache management is optimized for reads and writes in Hbase or MapReduce.
3. The method of claim 1, wherein, the cache management is optimized for large sequential reads and writes.
4. The method of claim 1, wherein an amount of the readaheads for sequential read requests is dynamically adjusted based on the sequential read requests.
5. The method of claim 1, wherein the method further comprising:
instantiating a connection manager on the machine, wherein the connection manager is configured to instruct the machine perform steps including:
upon receiving an access request from a client, establishing a connection with a node associated with the access request in response to the access request;
determining that the access request is associated with data that is smaller than a select size;
detecting whether operations requested by the access request have been completed;
after the operations requested by the access request have been completed, holding the connection with the node open for a select time period,
receiving, by the connection manager from the client, a subsequent access request that is associated with the node associated with the access request before the select time period expires; and
performing operations requested by the subsequent access request through the held open connection.
6. The method of claim 1, wherein, data size of the data that is invalidated from the buffer is configurable or dynamically adjustable.
7. The method of claim 1, wherein, data size of the data that is invalidated from the buffer is between 4-8 MB or 8-16 MB.
8. The method of claim 1, wherein the cache management optimization are performed through operating system calls are native to an operating system running on the machine.
9. The method of claim 1, further comprising decreasing checksum overhead in a read path of the distributed file system by modifying a polynomial that is used in generating the checksum.
10. The method of claim 9, wherein, the checksum is performed in hardware by a processor supporting CRC32.
11. The method of claim 1, wherein, the operating system is Linux or Unix-based operating systems.
12. The method of claim 1, wherein, the operating system conforms to POSIX.1-2001.
13. The method of claim 1, wherein, the distributed file system is the Hadoop distributed file system.
14. A system for distributed computing, the system comprising:
a set of machines forming a distributed file system cluster, a given machine in the set of machines having:
a processor;
a disk;
memory having stored there on instructions which when executed by the processor, causes the given machine to perform:
(a) determining that the machine employs a heuristic for triggering readaheads for sequential read requests;
overriding the heuristic so as to deterministically trigger the readaheads for all sequential read requests;
(b) determining that the machine is configured to automatically cache data into a buffer on the machine after the data is accessed;
detecting that a specific size of data has been accessed;
instructing the machine to invalidate the cached data in the buffer; and
(c) determining that the machine is configured to include a time delay before committing data from the buffer to a disk on the machine;
overriding the time delay so that the machine commits the data from the buffer to the disk without the time delay.
15. The system of claim 14, wherein the given machine is further caused to perform:
instantiating a connection manager on the machine, wherein the connection manager is configured to instruct the machine perform steps including:
upon receiving an access request from a client, establishing a connection with node associated with the access request in response to the access request;
determining that the access request is associated with data that is smaller than a select size;
detecting whether operations requested by the access request have been completed;
after the operations requested by the access request have been completed, holding the connection with the node open for a select time period,
receiving, by the connection manager from the client, a subsequent access request that is associated with the node associated with the access request before the select time period expires; and
performing operations requested by the subsequent access request through the held open connection.
16. The system of claim 14, wherein the connection is held open for 0.5-1 seconds, 1-2, seconds, or 2-5 seconds.
17. The system of claim 14, wherein an amount of the readaheads for sequential read requests is dynamically adjusted based on the sequential read requests.
18. The system of claim 14, wherein, the distributed file system is the Hadoop distributed file system (HDFS).
19. The system of claim 14, wherein, the cache management is optimized for reads and writes in Hbase or MapReduce.
20. The system of claim 14, wherein the given system is further cased to perform:
decreasing checksum overhead in a read path of the distributed file system by modifying a polynomial that is used in generating the checksum.
21. The system of claim 20, wherein, the checksum uses CRC32C algorithm.
US13/426,466 2012-03-21 2012-03-21 Data processing performance enhancement in a distributed file system Active 2034-03-29 US9405692B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US13/426,466 US9405692B2 (en) 2012-03-21 2012-03-21 Data processing performance enhancement in a distributed file system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US13/426,466 US9405692B2 (en) 2012-03-21 2012-03-21 Data processing performance enhancement in a distributed file system
US15/225,533 US9600492B2 (en) 2012-03-21 2016-08-01 Data processing performance enhancement in a distributed file system

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/225,533 Continuation US9600492B2 (en) 2012-03-21 2016-08-01 Data processing performance enhancement in a distributed file system

Publications (2)

Publication Number Publication Date
US20130254246A1 US20130254246A1 (en) 2013-09-26
US9405692B2 true US9405692B2 (en) 2016-08-02

Family

ID=49213346

Family Applications (2)

Application Number Title Priority Date Filing Date
US13/426,466 Active 2034-03-29 US9405692B2 (en) 2012-03-21 2012-03-21 Data processing performance enhancement in a distributed file system
US15/225,533 Active US9600492B2 (en) 2012-03-21 2016-08-01 Data processing performance enhancement in a distributed file system

Family Applications After (1)

Application Number Title Priority Date Filing Date
US15/225,533 Active US9600492B2 (en) 2012-03-21 2016-08-01 Data processing performance enhancement in a distributed file system

Country Status (1)

Country Link
US (2) US9405692B2 (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9405692B2 (en) 2012-03-21 2016-08-02 Cloudera, Inc. Data processing performance enhancement in a distributed file system
CN104978336A (en) * 2014-04-08 2015-10-14 云南电力试验研究院(集团)有限公司电力研究院 Unstructured data storage system based on Hadoop distributed computing platform
US20150347450A1 (en) * 2014-05-28 2015-12-03 Bluedata Software, Inc. Inter-version mapping of distributed file systems
CN105205082A (en) 2014-06-27 2015-12-30 国际商业机器公司 Method and system for processing file storage in HDFS
CN105446896B (en) 2014-08-29 2018-05-04 国际商业机器公司 Cache management method and apparatus for mapping of simple applications
US9628486B2 (en) * 2014-10-23 2017-04-18 Vormetric, Inc. Access control for data blocks in a distributed filesystem
US20170060754A1 (en) * 2015-08-25 2017-03-02 Tarun Nakra Capturing temporal store streams into cpu caches by dynamically varying store streaming thresholds
US9778879B2 (en) 2015-10-23 2017-10-03 Microsoft Technology Licensing, Llc Flushless transactional layer

Citations (122)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US1017916A (en) 1910-10-31 1912-02-20 Electromagnetic Tool Company Power-hammer.
US5325522A (en) 1986-10-15 1994-06-28 United States Data Corporation Apparatus and method for communicating between devices trough use of a real time data base
US5671385A (en) * 1993-04-29 1997-09-23 Southwestern Bell Technology Resources, Inc. Memory subsystem with disk meshing, controller meshing, and efficient cache buffer lookup
US5737536A (en) * 1993-02-19 1998-04-07 Borland International, Inc. System and methods for optimized access in a multi-user environment
US5825877A (en) 1996-06-11 1998-10-20 International Business Machines Corporation Support for portable trusted software
US20020055989A1 (en) 2000-11-08 2002-05-09 Stringer-Calvert David W.J. Methods and apparatus for scalable, distributed management of virtual private networks
US20020073322A1 (en) 2000-12-07 2002-06-13 Dong-Gook Park Countermeasure against denial-of-service attack on authentication protocols using public key encryption
US20020138762A1 (en) 2000-12-01 2002-09-26 Horne Donald R. Management of log archival and reporting for data network security systems
US20020174194A1 (en) 2001-05-18 2002-11-21 Eoin Mooney Providing access to a plurality of message accounts from a single web-based interface
US20030051036A1 (en) 2000-09-04 2003-03-13 International Business Machines Corporation Computer network system, computer system, method for communication between computer systems, method for measuring computer system performance, and storage medium
US20030055868A1 (en) 2001-09-19 2003-03-20 International Business Machines Corporation Building distributed software services as aggregations of other services
US6542930B1 (en) 2000-03-08 2003-04-01 International Business Machines Corporation Distributed file system with automated file management achieved by decoupling data analysis and movement operations
US6553476B1 (en) * 1997-02-10 2003-04-22 Matsushita Electric Industrial Co., Ltd. Storage management based on predicted I/O execution times
US20030093633A1 (en) 2001-09-27 2003-05-15 Seagate Technology Llc Method and system for data path verification
US6651242B1 (en) 1999-12-14 2003-11-18 Novell, Inc. High performance computing system for distributed applications over a computer
US20040003322A1 (en) 2002-06-28 2004-01-01 Collins David L. Method and apparatus for maintaining data integrity using a system management processor
US6678828B1 (en) 2002-07-22 2004-01-13 Vormetric, Inc. Secure network file access control system
US6687847B1 (en) 1999-04-21 2004-02-03 Cornell Research Foundation, Inc. Failure detector with consensus protocol
US20040034746A1 (en) * 2002-08-19 2004-02-19 Horn Robert L. Method of increasing performance and manageablity of network storage systems using optimized cache setting and handling policies
US20040059728A1 (en) 2000-04-27 2004-03-25 Todd Miller Method and system for retrieving search results from multiple disparate databases
US20040103166A1 (en) 2002-11-27 2004-05-27 International Business Machines Corporation Semi-hierarchical system and method for administration of clusters of computer resources
US20040172421A1 (en) 2002-12-09 2004-09-02 Yasushi Saito Namespace consistency for a wide-area file system
US20040186832A1 (en) 2003-01-16 2004-09-23 Jardin Cary A. System and method for controlling processing in a distributed system
US20050044311A1 (en) * 2003-08-22 2005-02-24 Oracle International Corporation Reducing disk IO by full-cache write-merging
US20050071708A1 (en) 2003-09-29 2005-03-31 International Business Machines (Ibm) Corporation Method, system, and program for recovery from a failure in an asynchronous data copying system
US20050091244A1 (en) * 1999-11-18 2005-04-28 International Business Machines Corporation Method, system and program products for reducing data movement within a computing environment
US6910099B1 (en) * 2001-10-31 2005-06-21 Western Digital Technologies, Inc. Disk drive adjusting read-ahead to optimize cache memory allocation
US20050138111A1 (en) 2003-10-15 2005-06-23 Microsoft Corporation On-line service/application monitoring and reporting system
US20050171983A1 (en) 2000-11-27 2005-08-04 Microsoft Corporation Smart card with volatile memory file subsystem
US6931530B2 (en) 2002-07-22 2005-08-16 Vormetric, Inc. Secure network file access controller implementing access control and auditing
US20050182749A1 (en) 2004-02-12 2005-08-18 Yoshiro Matsui Process editing apparatus and method and process management apparatus and method
US20060020854A1 (en) 2004-07-22 2006-01-26 International Business Machines Corporation Method and apparatus for high-speed network adapter failover
US20060050877A1 (en) 2004-09-07 2006-03-09 Mitsuhiro Nakamura Information processing apparatus and method, program, and recording medium
US7031981B1 (en) 2001-12-21 2006-04-18 Unisys Corporation Tool supporting system log file reporting
US7069497B1 (en) 2002-09-10 2006-06-27 Oracle International Corp. System and method for applying a partial page change
US20060143453A1 (en) 2002-06-19 2006-06-29 Secured Communications, Inc Inter-authentication method and device
US20060156018A1 (en) 2004-12-30 2006-07-13 Craig Lauer Template access control lists
US7107323B2 (en) 2003-03-17 2006-09-12 Hitachi, Ltd. System and method of file distribution for a computer system in which partial files are arranged according to various allocation rules
US20060224784A1 (en) * 2005-04-04 2006-10-05 Akira Nishimoto Storage system providing stream-oriented performance assurance
US20060248278A1 (en) * 2005-05-02 2006-11-02 International Business Machines Corporation Adaptive read ahead method of data recorded on a sequential media readable via a variable data block size storage device
US20060247897A1 (en) 2005-03-16 2006-11-02 Lin Jia Y Method and system for recording a monitoring log
US7143288B2 (en) 2002-10-16 2006-11-28 Vormetric, Inc. Secure file system server architecture and methods
US20070100913A1 (en) 2005-10-12 2007-05-03 Sumner Gary S Method and system for data backup
US20070113188A1 (en) 2005-11-17 2007-05-17 Bales Christopher E System and method for providing dynamic content in a communities framework
US20070136442A1 (en) 2005-12-08 2007-06-14 International Business Machines Corporation Seamless reflection of model updates in a visual page for a visual channel in a composite services delivery system
US20070180255A1 (en) 2006-01-30 2007-08-02 Toru Hanada Information processing apparatus and authentication control method
US20070177737A1 (en) 2005-02-18 2007-08-02 Samsung Electronics Co., Ltd. Network and domain-creating method thereof
US20070186112A1 (en) 2005-01-28 2007-08-09 Microsoft Corporation Controlling execution of computer applications
US20070226488A1 (en) 2006-03-22 2007-09-27 Hon Hai Precision Industry Co., Ltd. System and method for protecting digital files
US20070234115A1 (en) 2006-04-04 2007-10-04 Nobuyuki Saika Backup system and backup method
US20070255943A1 (en) 2006-04-18 2007-11-01 Kern David S Method and system for automating the recovery of a credential store
US20070282988A1 (en) 2006-05-31 2007-12-06 Christof Bornhoevd Device registration in a hierarchical monitor service
US20080104579A1 (en) 2006-10-31 2008-05-01 Sap Ag Systems and methods of transforming XML schemas
US20080140630A1 (en) 2006-12-11 2008-06-12 Fujitsu Limited Network control system, network control method, and computer product
US7392421B1 (en) 2002-03-18 2008-06-24 Symantec Operating Corporation Framework for managing clustering and replication
US20080168135A1 (en) 2007-01-05 2008-07-10 Redlich Ron M Information Infrastructure Management Tools with Extractor, Secure Storage, Content Analysis and Classification and Method Therefor
US20080244307A1 (en) 2007-03-26 2008-10-02 Oracle International Corporation Method to avoid continuous application failovers in a cluster
US20080256486A1 (en) 2006-09-14 2008-10-16 Kenji Hagiwara Providing device usage information through layout diagram
US20080263006A1 (en) 2007-04-20 2008-10-23 Sap Ag Concurrent searching of structured and unstructured data
US20080270706A1 (en) * 2007-04-27 2008-10-30 Fair Robert L Block Reallocation Planning During Read-Ahead Processing
US20080276130A1 (en) 2007-05-01 2008-11-06 Almoustafa Ahmed M Distributed logging apparatus system and method
US20080307181A1 (en) 2007-06-08 2008-12-11 Kuszmaul Bradley C Disk-Resident Streaming Dictionary
US20090013029A1 (en) 2007-07-03 2009-01-08 Childress Rhonda L Device, system and method of operating a plurality of virtual logical sites
US7487228B1 (en) 2003-01-30 2009-02-03 Red Hat, Inc. Metadata structures and related locking techniques to improve performance and scalability in a cluster file system
US7496829B2 (en) 2000-04-21 2009-02-24 Microsoft Corporation Method and apparatus for displaying multiple contexts in electronic documents
US20090177697A1 (en) 2008-01-08 2009-07-09 International Business Machines Corporation Correlation and parallelism aware materialized view recommendation for heterogeneous, distributed database systems
US20090259838A1 (en) 2008-04-15 2009-10-15 Authenex, Inc. Hardware-Bonded Credential Manager Method and System
US7631034B1 (en) 2008-09-18 2009-12-08 International Business Machines Corporation Optimizing node selection when handling client requests for a distributed file system (DFS) based on a dynamically determined performance index
US20090307783A1 (en) 2006-04-24 2009-12-10 Manabu Maeda Data processing device, method, program, integrated circuit, and program generating device
US7640512B1 (en) 2000-12-22 2009-12-29 Automated Logic Corporation Updating objects contained within a webpage
US20100010968A1 (en) 2008-07-10 2010-01-14 Redlich Ron M System and method to identify, classify and monetize information as an intangible asset and a production model based thereon
US20100008509A1 (en) 2008-07-11 2010-01-14 Kabushiki Kaisha Toshiba Communication apparatus, key server, and management server
US7653668B1 (en) 2005-11-23 2010-01-26 Symantec Operating Corporation Fault tolerant multi-stage data replication with relaxed coherency guarantees
US20100070769A1 (en) 2007-03-28 2010-03-18 Shigeyoshi Shima Log acquisition system, log collection terminal, log acquisition terminal, and log acquisition method and program using the same system and terminals
US7685109B1 (en) 2005-12-29 2010-03-23 Amazon Technologies, Inc. Method and apparatus for data partitioning and replication in a searchable data service
US7698321B2 (en) 2005-09-30 2010-04-13 At&T Intellectual Property I, L.P. Methods, systems, and products for updating web content
US20100131817A1 (en) 2007-04-19 2010-05-27 Lg Eelctronics Inc. Method of automatic repeat request (arq) in communication system
US20100179855A1 (en) * 2009-01-09 2010-07-15 Ye Chen Large-Scale Behavioral Targeting for Advertising over a Network
US20100198972A1 (en) 2009-02-04 2010-08-05 Steven Michael Umbehocker Methods and Systems for Automated Management of Virtual Resources In A Cloud Computing Environment
US7818313B1 (en) 2005-07-18 2010-10-19 Sybase, Inc. Method for distributing processing of queries over a cluster of servers in a continuous processing system
US7831991B1 (en) 1999-02-19 2010-11-09 Chaincast, Inc. Method and system for ensuring continuous data flow between re-transmitters within a chaincast communication system
US20100296652A1 (en) 2003-03-24 2010-11-25 Sony Corporation Information recording medium, information processing apparatus, information processing method, and computer program
US20100306286A1 (en) 2009-03-05 2010-12-02 Chi-Hsien Chiu Distributed steam processing
US20100325713A1 (en) 2005-11-18 2010-12-23 Taro Kurita Mobile terminal, data communication method, and computer program
US20100332373A1 (en) 2009-02-26 2010-12-30 Jason Crabtree System and method for participation in energy-related markets
US20110055578A1 (en) 2009-08-27 2011-03-03 Cleversafe, Inc. Verification of dispersed storage network access control information
US20110078549A1 (en) 2008-05-26 2011-03-31 Nxp B.V. Decoupling of measuring the response time of a transponder and its authentication
US7937482B1 (en) 2008-03-27 2011-05-03 Amazon Technologies, Inc. Scalable consensus protocol
US20110119328A1 (en) 2009-11-18 2011-05-19 Microsoft Corporation Load Balancing In A Distributed Computing Environment
US7984043B1 (en) 2007-07-24 2011-07-19 Amazon Technologies, Inc. System and method for distributed query processing using configuration-independent query plans
US8024560B1 (en) 2004-10-12 2011-09-20 Alten Alex I Systems and methods for securing multimedia transmissions over the internet
US20110228668A1 (en) 2010-03-22 2011-09-22 Infosys Technologies Limited Method and system for automatic failover of distributed query processing using distributed shared memory
US20110236873A1 (en) 2001-11-13 2011-09-29 Clarke Daniel Bowers Method and system for computer based testing using customizable templates
US20110246816A1 (en) 2010-03-31 2011-10-06 Cloudera, Inc. Configuring a system to collect and aggregate datasets
US20110246826A1 (en) 2010-03-31 2011-10-06 Cloudera, Inc. Collecting and aggregating log data with fault tolerance
US20110258378A1 (en) * 2010-04-14 2011-10-20 International Business Machines Corporation Optimizing a File System for Different Types of Applications in a Compute Cluster Using Dynamic Block Size Granularity
US20110276495A1 (en) 2010-05-10 2011-11-10 Computer Associates Think, Inc. One-time use password systems and methods
US20110276396A1 (en) 2005-07-22 2011-11-10 Yogesh Chunilal Rathod System and method for dynamically monitoring, recording, processing, attaching dynamic, contextual and accessible active links and presenting of physical or digital activities, actions, locations, logs, life stream, behavior and status
US8069267B2 (en) 2005-06-28 2011-11-29 Microsoft Corporation Advertisement refresh rules for network applications
US20110302417A1 (en) 2007-10-15 2011-12-08 Adobe Systems Incorporated Imparting cryptographic information in network communications
US20110307534A1 (en) 2009-03-25 2011-12-15 Zte Corporation Distributed file system supporting data block dispatching and file processing method thereof
US8108338B2 (en) 2005-06-10 2012-01-31 International Business Machines Corporation Method and system for model-based replication of data
US8108771B2 (en) 2007-08-27 2012-01-31 Yahoo! Inc. Secure inter-module communication mechanism
US20120036357A1 (en) 2003-11-14 2012-02-09 Marinus Struik Cryptographic method and apparatus
US20120036146A1 (en) 2010-10-26 2012-02-09 ParElastic Corporation Apparatus for elastic database processing with heterogeneous data
US20120102072A1 (en) 2009-07-07 2012-04-26 Zte Corporation Distributed management monitoring system, monitoring method and creating method thereof
US20120130874A1 (en) 2010-11-22 2012-05-24 Network Appliance, Inc. Providing security in a cloud storage environment
US20120131341A1 (en) 2010-11-22 2012-05-24 Network Appliance, Inc. Method and system for improving storage security in a cloud computing environment
US8306919B2 (en) 2002-06-10 2012-11-06 Ken Sakamura IC card and authentication method in electronic ticket distribution system
US20130031240A1 (en) 2011-07-29 2013-01-31 Cisco Technology, Inc. Capacity Evaluation of Computer Network Capabilities
US20130041872A1 (en) * 2011-08-12 2013-02-14 Alexander AIZMAN Cloud storage system with distributed metadata
US20130054976A1 (en) 2011-08-23 2013-02-28 International Business Machines Corporation Lightweight document access control using access control lists in the cloud storage or on the local file system
US8468244B2 (en) 2007-01-05 2013-06-18 Digital Doors, Inc. Digital information infrastructure and method for security designated data and with granular data stores
US8484716B1 (en) 2009-08-07 2013-07-09 Adobe Systems Incorporated Hosting a server application on multiple network tiers
US20130218840A1 (en) 2012-02-17 2013-08-22 Charles Smith System and method for building a point-in-time snapshot of an eventually-consistent data store
US8631403B2 (en) 2010-01-04 2014-01-14 Vmware, Inc. Method and system for managing tasks by dynamically scaling centralized virtual center in virtual infrastructure
US8655939B2 (en) 2007-01-05 2014-02-18 Digital Doors, Inc. Electromagnetic pulse (EMP) hardened information infrastructure with extractor, cloud dispersal, secure storage, content analysis and classification and method therefor
US8667267B1 (en) 2011-01-31 2014-03-04 Gazzang, Inc. System and method for communicating with a key management system
US8732406B1 (en) * 2011-03-15 2014-05-20 Netapp, Inc. Mechanism for determining read-ahead length in a storage system
US8732674B1 (en) 2012-11-14 2014-05-20 Microsoft Corporation Revertable managed execution image instrumentation
US20140188841A1 (en) 2012-12-29 2014-07-03 Futurewei Technologies, Inc. Method for Two-Stage Query Optimization in Massively Parallel Processing Database Clusters
US20140195558A1 (en) 2013-01-07 2014-07-10 Raghotham Murthy System and method for distributed database query engines

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6209058B1 (en) * 1999-01-27 2001-03-27 Quantum Corp. Cache management for data transfer control from target disk areas
US8489820B1 (en) * 2008-03-18 2013-07-16 Netapp, Inc Speculative copying of data from main buffer cache to solid-state secondary cache of a storage server
WO2013025655A2 (en) 2011-08-12 2013-02-21 Mcalister Technologies, Llc Systems and methods for providing supplemental aqueous thermal energy
US9405692B2 (en) 2012-03-21 2016-08-02 Cloudera, Inc. Data processing performance enhancement in a distributed file system

Patent Citations (129)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US1017916A (en) 1910-10-31 1912-02-20 Electromagnetic Tool Company Power-hammer.
US5325522A (en) 1986-10-15 1994-06-28 United States Data Corporation Apparatus and method for communicating between devices trough use of a real time data base
US5737536A (en) * 1993-02-19 1998-04-07 Borland International, Inc. System and methods for optimized access in a multi-user environment
US5671385A (en) * 1993-04-29 1997-09-23 Southwestern Bell Technology Resources, Inc. Memory subsystem with disk meshing, controller meshing, and efficient cache buffer lookup
US5825877A (en) 1996-06-11 1998-10-20 International Business Machines Corporation Support for portable trusted software
US6553476B1 (en) * 1997-02-10 2003-04-22 Matsushita Electric Industrial Co., Ltd. Storage management based on predicted I/O execution times
US7831991B1 (en) 1999-02-19 2010-11-09 Chaincast, Inc. Method and system for ensuring continuous data flow between re-transmitters within a chaincast communication system
US6687847B1 (en) 1999-04-21 2004-02-03 Cornell Research Foundation, Inc. Failure detector with consensus protocol
US20050091244A1 (en) * 1999-11-18 2005-04-28 International Business Machines Corporation Method, system and program products for reducing data movement within a computing environment
US6651242B1 (en) 1999-12-14 2003-11-18 Novell, Inc. High performance computing system for distributed applications over a computer
US6542930B1 (en) 2000-03-08 2003-04-01 International Business Machines Corporation Distributed file system with automated file management achieved by decoupling data analysis and movement operations
US7496829B2 (en) 2000-04-21 2009-02-24 Microsoft Corporation Method and apparatus for displaying multiple contexts in electronic documents
US20040059728A1 (en) 2000-04-27 2004-03-25 Todd Miller Method and system for retrieving search results from multiple disparate databases
US20030051036A1 (en) 2000-09-04 2003-03-13 International Business Machines Corporation Computer network system, computer system, method for communication between computer systems, method for measuring computer system performance, and storage medium
US20020055989A1 (en) 2000-11-08 2002-05-09 Stringer-Calvert David W.J. Methods and apparatus for scalable, distributed management of virtual private networks
US20050171983A1 (en) 2000-11-27 2005-08-04 Microsoft Corporation Smart card with volatile memory file subsystem
US20020138762A1 (en) 2000-12-01 2002-09-26 Horne Donald R. Management of log archival and reporting for data network security systems
US20020073322A1 (en) 2000-12-07 2002-06-13 Dong-Gook Park Countermeasure against denial-of-service attack on authentication protocols using public key encryption
US7640512B1 (en) 2000-12-22 2009-12-29 Automated Logic Corporation Updating objects contained within a webpage
US20020174194A1 (en) 2001-05-18 2002-11-21 Eoin Mooney Providing access to a plurality of message accounts from a single web-based interface
US20030055868A1 (en) 2001-09-19 2003-03-20 International Business Machines Corporation Building distributed software services as aggregations of other services
US20030093633A1 (en) 2001-09-27 2003-05-15 Seagate Technology Llc Method and system for data path verification
US6910099B1 (en) * 2001-10-31 2005-06-21 Western Digital Technologies, Inc. Disk drive adjusting read-ahead to optimize cache memory allocation
US20110236873A1 (en) 2001-11-13 2011-09-29 Clarke Daniel Bowers Method and system for computer based testing using customizable templates
US7031981B1 (en) 2001-12-21 2006-04-18 Unisys Corporation Tool supporting system log file reporting
US7392421B1 (en) 2002-03-18 2008-06-24 Symantec Operating Corporation Framework for managing clustering and replication
US8306919B2 (en) 2002-06-10 2012-11-06 Ken Sakamura IC card and authentication method in electronic ticket distribution system
US20060143453A1 (en) 2002-06-19 2006-06-29 Secured Communications, Inc Inter-authentication method and device
US20040003322A1 (en) 2002-06-28 2004-01-01 Collins David L. Method and apparatus for maintaining data integrity using a system management processor
US6931530B2 (en) 2002-07-22 2005-08-16 Vormetric, Inc. Secure network file access controller implementing access control and auditing
US6678828B1 (en) 2002-07-22 2004-01-13 Vormetric, Inc. Secure network file access control system
US20040034746A1 (en) * 2002-08-19 2004-02-19 Horn Robert L. Method of increasing performance and manageablity of network storage systems using optimized cache setting and handling policies
US7069497B1 (en) 2002-09-10 2006-06-27 Oracle International Corp. System and method for applying a partial page change
US7143288B2 (en) 2002-10-16 2006-11-28 Vormetric, Inc. Secure file system server architecture and methods
US20040103166A1 (en) 2002-11-27 2004-05-27 International Business Machines Corporation Semi-hierarchical system and method for administration of clusters of computer resources
US8311980B2 (en) 2002-12-09 2012-11-13 Hewlett-Packard Development Company, L.P. Namespace consistency for a wide-area file system
US20040172421A1 (en) 2002-12-09 2004-09-02 Yasushi Saito Namespace consistency for a wide-area file system
US20040186832A1 (en) 2003-01-16 2004-09-23 Jardin Cary A. System and method for controlling processing in a distributed system
US7487228B1 (en) 2003-01-30 2009-02-03 Red Hat, Inc. Metadata structures and related locking techniques to improve performance and scalability in a cluster file system
US7107323B2 (en) 2003-03-17 2006-09-12 Hitachi, Ltd. System and method of file distribution for a computer system in which partial files are arranged according to various allocation rules
US7620698B2 (en) 2003-03-17 2009-11-17 Hitachi, Ltd. File distribution system in which partial files are arranged according to various allocation rules associated with a plurality of file types
US7325041B2 (en) 2003-03-17 2008-01-29 Hitachi, Ltd. File distribution system in which partial files are arranged according to various allocation rules associated with a plurality of file types
US20100296652A1 (en) 2003-03-24 2010-11-25 Sony Corporation Information recording medium, information processing apparatus, information processing method, and computer program
US20050044311A1 (en) * 2003-08-22 2005-02-24 Oracle International Corporation Reducing disk IO by full-cache write-merging
US20050071708A1 (en) 2003-09-29 2005-03-31 International Business Machines (Ibm) Corporation Method, system, and program for recovery from a failure in an asynchronous data copying system
US20050138111A1 (en) 2003-10-15 2005-06-23 Microsoft Corporation On-line service/application monitoring and reporting system
US20120036357A1 (en) 2003-11-14 2012-02-09 Marinus Struik Cryptographic method and apparatus
US20050182749A1 (en) 2004-02-12 2005-08-18 Yoshiro Matsui Process editing apparatus and method and process management apparatus and method
US20060020854A1 (en) 2004-07-22 2006-01-26 International Business Machines Corporation Method and apparatus for high-speed network adapter failover
US20060050877A1 (en) 2004-09-07 2006-03-09 Mitsuhiro Nakamura Information processing apparatus and method, program, and recording medium
US8024560B1 (en) 2004-10-12 2011-09-20 Alten Alex I Systems and methods for securing multimedia transmissions over the internet
US20060156018A1 (en) 2004-12-30 2006-07-13 Craig Lauer Template access control lists
US20070186112A1 (en) 2005-01-28 2007-08-09 Microsoft Corporation Controlling execution of computer applications
US20070177737A1 (en) 2005-02-18 2007-08-02 Samsung Electronics Co., Ltd. Network and domain-creating method thereof
US20060247897A1 (en) 2005-03-16 2006-11-02 Lin Jia Y Method and system for recording a monitoring log
US20060224784A1 (en) * 2005-04-04 2006-10-05 Akira Nishimoto Storage system providing stream-oriented performance assurance
US20060248278A1 (en) * 2005-05-02 2006-11-02 International Business Machines Corporation Adaptive read ahead method of data recorded on a sequential media readable via a variable data block size storage device
US8108338B2 (en) 2005-06-10 2012-01-31 International Business Machines Corporation Method and system for model-based replication of data
US8069267B2 (en) 2005-06-28 2011-11-29 Microsoft Corporation Advertisement refresh rules for network applications
US7818313B1 (en) 2005-07-18 2010-10-19 Sybase, Inc. Method for distributing processing of queries over a cluster of servers in a continuous processing system
US20110276396A1 (en) 2005-07-22 2011-11-10 Yogesh Chunilal Rathod System and method for dynamically monitoring, recording, processing, attaching dynamic, contextual and accessible active links and presenting of physical or digital activities, actions, locations, logs, life stream, behavior and status
US7698321B2 (en) 2005-09-30 2010-04-13 At&T Intellectual Property I, L.P. Methods, systems, and products for updating web content
US20070100913A1 (en) 2005-10-12 2007-05-03 Sumner Gary S Method and system for data backup
US20070113188A1 (en) 2005-11-17 2007-05-17 Bales Christopher E System and method for providing dynamic content in a communities framework
US20100325713A1 (en) 2005-11-18 2010-12-23 Taro Kurita Mobile terminal, data communication method, and computer program
US7653668B1 (en) 2005-11-23 2010-01-26 Symantec Operating Corporation Fault tolerant multi-stage data replication with relaxed coherency guarantees
US20070136442A1 (en) 2005-12-08 2007-06-14 International Business Machines Corporation Seamless reflection of model updates in a visual page for a visual channel in a composite services delivery system
US7685109B1 (en) 2005-12-29 2010-03-23 Amazon Technologies, Inc. Method and apparatus for data partitioning and replication in a searchable data service
US20070180255A1 (en) 2006-01-30 2007-08-02 Toru Hanada Information processing apparatus and authentication control method
US20070226488A1 (en) 2006-03-22 2007-09-27 Hon Hai Precision Industry Co., Ltd. System and method for protecting digital files
US20070234115A1 (en) 2006-04-04 2007-10-04 Nobuyuki Saika Backup system and backup method
US20070255943A1 (en) 2006-04-18 2007-11-01 Kern David S Method and system for automating the recovery of a credential store
US20090307783A1 (en) 2006-04-24 2009-12-10 Manabu Maeda Data processing device, method, program, integrated circuit, and program generating device
US20070282988A1 (en) 2006-05-31 2007-12-06 Christof Bornhoevd Device registration in a hierarchical monitor service
US20080256486A1 (en) 2006-09-14 2008-10-16 Kenji Hagiwara Providing device usage information through layout diagram
US20080104579A1 (en) 2006-10-31 2008-05-01 Sap Ag Systems and methods of transforming XML schemas
US20080140630A1 (en) 2006-12-11 2008-06-12 Fujitsu Limited Network control system, network control method, and computer product
US8468244B2 (en) 2007-01-05 2013-06-18 Digital Doors, Inc. Digital information infrastructure and method for security designated data and with granular data stores
US8655939B2 (en) 2007-01-05 2014-02-18 Digital Doors, Inc. Electromagnetic pulse (EMP) hardened information infrastructure with extractor, cloud dispersal, secure storage, content analysis and classification and method therefor
US20130304761A1 (en) 2007-01-05 2013-11-14 Digital Doors, Inc. Digital Information Infrastruture and Method for Security Designated Data and with Granular Data Stores
US20080168135A1 (en) 2007-01-05 2008-07-10 Redlich Ron M Information Infrastructure Management Tools with Extractor, Secure Storage, Content Analysis and Classification and Method Therefor
US20080244307A1 (en) 2007-03-26 2008-10-02 Oracle International Corporation Method to avoid continuous application failovers in a cluster
US20100070769A1 (en) 2007-03-28 2010-03-18 Shigeyoshi Shima Log acquisition system, log collection terminal, log acquisition terminal, and log acquisition method and program using the same system and terminals
US20100131817A1 (en) 2007-04-19 2010-05-27 Lg Eelctronics Inc. Method of automatic repeat request (arq) in communication system
US20080263006A1 (en) 2007-04-20 2008-10-23 Sap Ag Concurrent searching of structured and unstructured data
US20080270706A1 (en) * 2007-04-27 2008-10-30 Fair Robert L Block Reallocation Planning During Read-Ahead Processing
US7734961B2 (en) 2007-05-01 2010-06-08 International Business Machines Corporation Distributed logging apparatus system and method
US20080276130A1 (en) 2007-05-01 2008-11-06 Almoustafa Ahmed M Distributed logging apparatus system and method
US20080307181A1 (en) 2007-06-08 2008-12-11 Kuszmaul Bradley C Disk-Resident Streaming Dictionary
US20090013029A1 (en) 2007-07-03 2009-01-08 Childress Rhonda L Device, system and method of operating a plurality of virtual logical sites
US7984043B1 (en) 2007-07-24 2011-07-19 Amazon Technologies, Inc. System and method for distributed query processing using configuration-independent query plans
US8108771B2 (en) 2007-08-27 2012-01-31 Yahoo! Inc. Secure inter-module communication mechanism
US20110302417A1 (en) 2007-10-15 2011-12-08 Adobe Systems Incorporated Imparting cryptographic information in network communications
US20090177697A1 (en) 2008-01-08 2009-07-09 International Business Machines Corporation Correlation and parallelism aware materialized view recommendation for heterogeneous, distributed database systems
US7937482B1 (en) 2008-03-27 2011-05-03 Amazon Technologies, Inc. Scalable consensus protocol
US20090259838A1 (en) 2008-04-15 2009-10-15 Authenex, Inc. Hardware-Bonded Credential Manager Method and System
US20110078549A1 (en) 2008-05-26 2011-03-31 Nxp B.V. Decoupling of measuring the response time of a transponder and its authentication
US20100010968A1 (en) 2008-07-10 2010-01-14 Redlich Ron M System and method to identify, classify and monetize information as an intangible asset and a production model based thereon
US20100008509A1 (en) 2008-07-11 2010-01-14 Kabushiki Kaisha Toshiba Communication apparatus, key server, and management server
US7631034B1 (en) 2008-09-18 2009-12-08 International Business Machines Corporation Optimizing node selection when handling client requests for a distributed file system (DFS) based on a dynamically determined performance index
US20100179855A1 (en) * 2009-01-09 2010-07-15 Ye Chen Large-Scale Behavioral Targeting for Advertising over a Network
US20100198972A1 (en) 2009-02-04 2010-08-05 Steven Michael Umbehocker Methods and Systems for Automated Management of Virtual Resources In A Cloud Computing Environment
US20100332373A1 (en) 2009-02-26 2010-12-30 Jason Crabtree System and method for participation in energy-related markets
US20100306286A1 (en) 2009-03-05 2010-12-02 Chi-Hsien Chiu Distributed steam processing
US20110307534A1 (en) 2009-03-25 2011-12-15 Zte Corporation Distributed file system supporting data block dispatching and file processing method thereof
US20120102072A1 (en) 2009-07-07 2012-04-26 Zte Corporation Distributed management monitoring system, monitoring method and creating method thereof
US8484716B1 (en) 2009-08-07 2013-07-09 Adobe Systems Incorporated Hosting a server application on multiple network tiers
US20110055578A1 (en) 2009-08-27 2011-03-03 Cleversafe, Inc. Verification of dispersed storage network access control information
US7970861B2 (en) 2009-11-18 2011-06-28 Microsoft Corporation Load balancing in a distributed computing environment
US20110119328A1 (en) 2009-11-18 2011-05-19 Microsoft Corporation Load Balancing In A Distributed Computing Environment
US8631403B2 (en) 2010-01-04 2014-01-14 Vmware, Inc. Method and system for managing tasks by dynamically scaling centralized virtual center in virtual infrastructure
US20110228668A1 (en) 2010-03-22 2011-09-22 Infosys Technologies Limited Method and system for automatic failover of distributed query processing using distributed shared memory
US20110246816A1 (en) 2010-03-31 2011-10-06 Cloudera, Inc. Configuring a system to collect and aggregate datasets
US20110246826A1 (en) 2010-03-31 2011-10-06 Cloudera, Inc. Collecting and aggregating log data with fault tolerance
US20110258378A1 (en) * 2010-04-14 2011-10-20 International Business Machines Corporation Optimizing a File System for Different Types of Applications in a Compute Cluster Using Dynamic Block Size Granularity
US20110276495A1 (en) 2010-05-10 2011-11-10 Computer Associates Think, Inc. One-time use password systems and methods
US20120036146A1 (en) 2010-10-26 2012-02-09 ParElastic Corporation Apparatus for elastic database processing with heterogeneous data
US20120131341A1 (en) 2010-11-22 2012-05-24 Network Appliance, Inc. Method and system for improving storage security in a cloud computing environment
US20120130874A1 (en) 2010-11-22 2012-05-24 Network Appliance, Inc. Providing security in a cloud storage environment
US8667267B1 (en) 2011-01-31 2014-03-04 Gazzang, Inc. System and method for communicating with a key management system
US8788815B1 (en) 2011-01-31 2014-07-22 Gazzang, Inc. System and method for controlling access to decrypted data
US8732406B1 (en) * 2011-03-15 2014-05-20 Netapp, Inc. Mechanism for determining read-ahead length in a storage system
US20130031240A1 (en) 2011-07-29 2013-01-31 Cisco Technology, Inc. Capacity Evaluation of Computer Network Capabilities
US20130041872A1 (en) * 2011-08-12 2013-02-14 Alexander AIZMAN Cloud storage system with distributed metadata
US20130054976A1 (en) 2011-08-23 2013-02-28 International Business Machines Corporation Lightweight document access control using access control lists in the cloud storage or on the local file system
US20130218840A1 (en) 2012-02-17 2013-08-22 Charles Smith System and method for building a point-in-time snapshot of an eventually-consistent data store
US8732674B1 (en) 2012-11-14 2014-05-20 Microsoft Corporation Revertable managed execution image instrumentation
US20140188841A1 (en) 2012-12-29 2014-07-03 Futurewei Technologies, Inc. Method for Two-Stage Query Optimization in Massively Parallel Processing Database Clusters
US20140195558A1 (en) 2013-01-07 2014-07-10 Raghotham Murthy System and method for distributed database query engines

Non-Patent Citations (48)

* Cited by examiner, † Cited by third party
Title
Babaoglu, O., and K. Marzullo, "Consistent global states of distributed systems: Fundamental concepts and mechanisms," Distributed Systems, 53 pages, Jan. 1993.
Beomseok Nam et al: "Spatial indexing of distributed multidimensional datasets", Cluster Computing and the Grid, 2005. CCGRID 2005. IEEE International Symposium on Cardiff~Wales, UK May 9-12, 2005, Piscataway, NJ, USA,IEEE, Piscataway, NJ, USA, vol. 2, May 9, 2005, pp. 743-750.
Beomseok Nam et al: "Spatial indexing of distributed multidimensional datasets", Cluster Computing and the Grid, 2005. CCGRID 2005. IEEE International Symposium on Cardiff˜Wales, UK May 9-12, 2005, Piscataway, NJ, USA,IEEE, Piscataway, NJ, USA, vol. 2, May 9, 2005, pp. 743-750.
Canada Exam Report for application No. 2843459, mailing date Aug. 5, 2014, 3 pages.
Chapter 25: Distributed Databases Ed -; ; Ramez Elmasri; Shamkant B Navathe (eds), Jan. 1, 2011, Fundamentals of Database Systems (Sixth Edition), Addison-Wesley, p. 877-927.
Cheng, Security Attack Safe Mobil and Cloud-Based One-Time Password Tokens Using Rubbing Encryption Algorithm, ACM, Jun. 2011, pp. 304-336.
Corbett et al., "Spanner: Google's Globally Distributed Database," Transactions on Computer Systems (TOGS), vol. 31, No. 3, 14 pages, Aug. 2013.
Du et al., "A Pairwise Key Predistribution Scheme for Wireless Sensor Networks," ACM, May 2005, vol. 8 Issue 2, pp. 228-258.
European Search Report for European Application No. 14157984.7, mailing date Jun. 6, 2014, 10 pages.
Exam Report for GB1403929.1, Applicant: Cloudera, Inc, Mailed May. 2, 2014, 6 pages.
Kim et al., Design and Implementation of a Private and Public Key Crypto Processor and Its Applicaiton to a Security System, IEEE, Feb. 2004, vol. %0, Issue 1, pp. 214-224.
Kossmann D: "The State of the Art in Distributed Query Processing", ACM Computing Surveys, ACM, New York, NY, us, vol. 32, No. 4, Dec. 1, 2000, pp. 422-469.
Lamport, L., "Time, clocks, and the ordering of events in a distributed system," Communications of the ACM, vol. 21, No. 7, pp. 558-565, Jul. 1978.
Non-Final Office Action for U.S. Appl. No. 13/362,695, mailed Apr. 29, 2013, 23 pgs.
Provisional U.S. Appl. No. 61/319,797, filed on Mar. 31, 2010, titled Collecting and Aggregating Datasets for Analysis.
Provisional U.S. Appl. No. 61/319,816, filed on Mar. 31, 2010, titled Configuring a System to Collect and Aggregate Datasets.
Provisional U.S. Appl. No. 61/319,831, filed on Mar. 31, 2010, titled Collecting and Aggregating Log Data with Fault Tolerance.
Provisional U.S. Appl. No. 61/319,834, filed on Mar. 31, 2010, titled Dynamically Processing an Event Using and Extensible Data Model.
Provisional U.S. Appl. No. 61/588,036, filed on Jan. 18, 2012, titled Memory Allocation Buffer for Reduction of Heap Fragmentation.
Provisional U.S. Appl. No. 61/636,452, filed on Apr. 20, 2012, titled Automatic Repair of Corrupt HBases.
Provisional U.S. Appl. No. 61/642,937, filed on May 4, 2012, titled Configuring Hadoop Security with Cloudera Manager.
Stoller, S.D., "Detecting global predicates in distributed systems with clocks," Distributed Computing, vol. 13, No. 2, pp. 85-98, Feb. 2000.
Tamer Dzsu et al: "Principles of Distributed Database Systems", Principles of Distributed Database Systems, XX, XX, Jan. 1, 1991, pp. 74-93.
U.S. Appl. No. 12/877,878, filed on Sep. 8, 2010, titled Collecting and Aggregating Datasets for Analysis.
U.S. Appl. No. 12/877,902, filed on Sep. 8, 2010, titled Configuring a System to Collect and Aggregate Datasets.
U.S. Appl. No. 12/877,913, filed on Sep. 8, 2010, titled Collecting and Aggregating Log Data with Fault Tolerance.
U.S. Appl. No. 12/877,930, filed on Sep. 8, 2010, now U.S. Pat. No. 8,812,457, titled Dynamically Processing an Event Using an Extensible Data Model.
U.S. Appl. No. 12/877,930, filed on Sep. 8, 2010, titled Dynamically Processing an Event Using and Extensible Data Model.
U.S. Appl. No. 13/077,878, filed Mar. 31, 2011, titled Interactive User Interface Implementation and Development Environment Therefor.
U.S. Appl. No. 13/077,878, filed on Mar. 31, 2011, titled Interactive User Interface Implementation and Development Environment Therefor.
U.S. Appl. No. 13/077,909, filed on Mar. 31, 2011, titled. User Interface Implementation for Partial Display Update.
U.S. Appl. No. 13/077,990, filed on Mar. 31, 2011, now U.S. Pat. No. 8,880,592, titled User Interface Implementation for Partial Display Update.
U.S. Appl. No. 13/566,943, filed on Aug. 3, 2012, titled Centralized Configuration and Monitoring of a Distributed Computing Cluster.
U.S. Appl. No. 13/745,461, filed on Jan. 18, 2013, titled Memory Allocation Buffer for Reduction of Heap Fragmentation.
U.S. Appl. No. 13/800,280, filed on Mar. 13, 2013, titled Low Latency Query Engine for Apache Hadoop.
U.S. Appl. No. 13/842,967, filed on Mar. 15, 2013, titled Automatic Repair of Corrup HBases.
U.S. Appl. No. 13/842,967, filed on Mar. 15, 2013, titled Automatic Repair of Corrupt HBases.
U.S. Appl. No. 13/854,773, filed Apr. 1, 2013, Kirkland et al.
U.S. Appl. No. 13/854,773, filed on Apr. 1, 2013, titled System and Method for Secure Release of Secret Information Over a Network.
U.S. Appl. No. 14/024,585, filed on Sep. 11, 2013, titled Data Node Fencing in a Distributed File System.
U.S. Appl. No. 14/043,753, filed on Oct. 1, 2013, titled Background Format Optimization for Enhanced SQL-Like Queries in Hadoop.
U.S. Appl. No. 14/462,395, filed on Aug. 18, 2014, titled Dynamically Processing an Event Using an Extensible Data Model.
U.S. Appl. No. 14/462,445, filed on Aug. 18, 2014, titled Hybridtime and Hybridlocks for Clock Uncertainty Reduction in a Distributed Computing Environment.
U.S. Appl. No. 14/509,300, filed on Oct. 8, 2014, titled Centralized Configuration of a Distributed Computing Cluster.
U.S. Appl. No. 14/510,006, filed on Oct. 8, 2014, titled Querying Operating System State on Multiple Machines Declaratively.
U.S. Appl. No. 14/526,372, filed on Oct. 28, 2014, titled Virtual Machine Image Encryption.
U.S. Appl. No. 14/527,563, filed on Oct. 18, 2014, titled Manifest-Based Snapshots in Distributed Computing Environments.
U.S. Appl. No. 14/588,250, filed on Dec. 31, 2014 titled Resource Management in a Distributed Computing Environment.

Also Published As

Publication number Publication date
US20160342619A1 (en) 2016-11-24
US9600492B2 (en) 2017-03-21
US20130254246A1 (en) 2013-09-26

Similar Documents

Publication Publication Date Title
US8484260B2 (en) Method for a cloud-based meta-file system to virtually unify remote and local files across a range of devices' local file systems
US8966066B2 (en) Application and network-based long poll request detection and cacheability assessment therefor
CA2737571C (en) Pre-caching web content for a mobile device
US7941609B2 (en) HTTP acceleration by prediction and pre-fetching
KR100908746B1 (en) System, and computer readable media for managing objects stored in the cache
US9332063B2 (en) Versatile application configuration for deployable computing environments
US10264039B2 (en) Streaming content and placeholders
US9811329B2 (en) Cloud based file system surpassing device storage limits
US9798730B2 (en) Systems and methods for folder redirection
US8812451B2 (en) Programming model for synchronizing browser caches across devices and web services
US8769206B2 (en) Methods and systems for implementing transcendent page caching
US9298747B2 (en) Deployable, consistent, and extensible computing environment platform
WO2012145544A2 (en) Device resource sharing for network resource conservation
WO2013116852A1 (en) User as an end point for profiling and optimizing the delivery of content and data in a wireless network
WO2014011216A1 (en) Dynamic bandwidth adjustment for browsing or streaming activity in a wireless network based on prediction of user behavior when interacting with mobile applications
JP2016506580A (en) System and method for automatic synchronization recently modified data
CA2855209A1 (en) Session transfer and suspension in a remote access application framework
US9672267B2 (en) Hybrid data management system and method for managing large, varying datasets
US8949719B2 (en) Methods and systems for user interface event snooping and prefetching
US20090216789A1 (en) Management of time-variant data schemas in data warehouses
US8799409B2 (en) Server side data cache system
US9032000B2 (en) System and method for geolocation of social media posts
CN102375882A (en) Method, device and browser for rapidly accessing webpage
US8972690B2 (en) Methods and apparatuses for usage based allocation block size tuning
US9384297B2 (en) Systems and methods of accelerating delivery of remote content

Legal Events

Date Code Title Description
AS Assignment

Owner name: CLOUDERA, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LIPCON, TODD;REEL/FRAME:029255/0846

Effective date: 20120511

CC Certificate of correction