US20130124483A1 - System and method for operating a big-data platform - Google Patents
System and method for operating a big-data platform Download PDFInfo
- Publication number
- US20130124483A1 US20130124483A1 US13/671,896 US201213671896A US2013124483A1 US 20130124483 A1 US20130124483 A1 US 20130124483A1 US 201213671896 A US201213671896 A US 201213671896A US 2013124483 A1 US2013124483 A1 US 2013124483A1
- Authority
- US
- United States
- Prior art keywords
- data
- storage system
- client
- query
- client data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/221—Column-oriented storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Definitions
- This invention relates generally to the data-intensive distributed systems field, and more specifically to a new and useful system and method for operating a big-data platform in the distributed systems field.
- FIG. 1 is a schematic representation of a system of a preferred embodiment of the invention
- FIG. 2 is an exemplary schematic representation of an implementation of a system of a preferred embodiment of the invention.
- FIG. 3 is a detailed schematic of an example of a data agent integrating with a client application
- FIG. 4 is a representation of the conversion of user data to row and columnar formats for real-time and archived storage
- FIG. 5 is an exemplary sample of a command line interface
- FIG. 6 is a schematic representation of a method of a preferred embodiment of the invention.
- FIG. 7 is a schematic representation of a variation cooperatively querying a real-time storage system and an archive storage system of a preferred embodiment of the invention.
- a system 100 for operating a big-data platform of a preferred embodiment of the invention includes a client data agent 110 integrated with a discrete event channel, a remote distributed storage system 120 , a query processing cluster 130 , and a query interface 140 .
- the remote distributed storage system may additionally include a real-time storage system 122 and an archive storage system 124 .
- the system functions to enable applications built on any infrastructure to benefit from the data processing capabilities of a specialized and dedicated infrastructure. As the system functions independently from the application of a client, the big data processing requirements are abstracted to a simple interface for the client.
- the system preferably uses a columnar-based archive storage system 124 and a data-intensive query processing cluster 130 deployed on a distributed computing environment to enable fast data interactions on scalable system.
- the system may additionally be adapted to function as a multitenant service so that numerous clients can utilize the system simultaneously.
- Data of various users is preferably stored securely and in a format isolated from other user data.
- the query processing cluster is preferably shared by users of the multitenant infrastructure.
- the system 100 is suitable for use with big datasets such as datasets measured in the terabytes, petabytes, or even greater data size.
- the system has the benefits of using generally discrete and unorganized data encountered periodically during the operation of an application and converting such unstructured data into a structured data from large dataset.
- the system enables existing applications and data-sources to easily and organically benefit from the big-data platform. This may be true even for existing application and data-sources that were designed without any considerations for big-data processing. Put another way, the system enables traditional applications to benefit from analysis of data that requires non-traditional data management tools.
- the system can preferably be built utilizing Hadoop and cloud hosting services for some base components.
- the system may alternatively include any suitable components or be built on any suitable infrastructure.
- the client data agent no of a preferred embodiment functions to integrate with client systems to communicate small portions of data to the remote distributed storage system.
- An event channel or interface preferably enables a website, application, service, or any suitable networked program to supply event data.
- the client data (e.g., event data) is preferably discrete event data.
- Discrete event data preferably has an association to the time of an event, the kind of event, and/or any suitable additional event attributes.
- An event channel may be a database log, app log, syslog, sensor data stream, web content, or any suitable source of data.
- an application will generate client data in response to actions of a user of the application, and the client data will include information of the action.
- the client data is formatted as a plurality of key-value pairs.
- a sequence of client data is preferably organized in row format.
- the client data may have any suitable format.
- the client data agent 110 is preferably a daemon providing a gateway to the data-intensive infrastructure.
- the client data agent is preferably a plugin.
- the client data agent may be a ruby gem, but may alternatively be a web service, a library, a script, an application programming interface (API), or any suitable agent with an interface to client systems and a communication channel to the storage system 120 .
- the data agent may be integrated into multiple portions of an application as shown in FIG. 3 .
- the client data agent 110 preferably includes networking interface to periodically communicate with the storage system 120 .
- the client data agent may additionally include a data compressor 112 that functions to compress event data before communicating to the storage system 120 .
- the data compressor 112 preferably is a data binary serializer, and preferably converts data to MessagePack or any suitable data interchange format.
- MessagePack is a small and compact binary serialization format with a JSON-like syntax.
- the client data agent no may include configuration to identify the client, user account, or application when communicating with the remote storage system. A private key may be used to uniquely identify event data and distinguish the event data from other clients using the multitenant system.
- the client data agent no may additionally include a pluggable architecture to enable arbitrary data sources to be used in the client data agent.
- the pluggable architecture preferably includes an input plugin interface and an output plugin interface.
- the input plugin interface is preferably configured to allow plugins from third party developers or any suitable entity to be developed to pull data from any suitable source. For example, an input plugin may be written to pull data in from a particular site.
- the output plugin interface is preferably configured to allow plugins from third party developers or any suitable entity to be developed to customize the output of data from the client data agent.
- the output plugin may translate data, customize the transmission of data, or provide any customized functionality for the output of the client data agent 110 .
- the output plugin may convert the data units or language.
- the output plugin may be used to transmit the data to multiple destinations.
- the remote distributed storage system 120 of a preferred embodiment functions to store the event data.
- the event data coming into the distributed storage system is preferably discrete (e.g., organized by time based events) and substantially unorganized.
- the storage system 120 in combination with the query processing cluster enables organized big data analysis to be output from the system. Big data may be characterized as greater than a terabyte of data, a petabyte of data, or any suitable size. In one exemplary application, big data is characterized by greater than 1 terabyte of data storing a billion web clicks, but may alternatively be characterized by any suitable sized data.
- the storage system is preferably hosted on a distributed cloud storage system such as Amazon's S3/EC2, but may alternatively be any suitable infrastructure.
- the data is preferably kept in a compressed format (e.g., MessagePack format).
- the remote distributed storage system preferably functions to store data for both real-time querying and for large batch data querying, but may alternatively partition and group data in any suitable way.
- the remote distributed storage system preferably includes a real-time storage system 122 and an archive storage system 124 .
- the remote distributed storage system 120 is preferably structured to initially store discrete data sent from the client data agent 110 in the real-time storage system 122 , and after a temporal delay, a data converter merges or otherwise prepares the data for storage in the archive storage system 124 .
- the real-time storage system 122 preferably stores the client data as row data
- the archive storage system 124 preferably stores the data in a columnar data format.
- the data is preferable merged and converted to columnar-style using a MapReduce job.
- Row file format has the benefit of being easy to append new rows, add new keys, and is suitable for writing data.
- Columnar file format has the benefit of enabling unnecessary keys and values to be skipped, can be highly compressed due to coherency, and is suitable for archiving and reading.
- the columnar data format additionally can provide optimizations and efficiencies when querying the large data sets of the archive storage system. If the system is implemented for multitenant application, the data may additionally be stored according to an account identifier.
- the data may additionally be encrypted or suitable secured. Data for different users is preferably kept isolated from access from other unauthorized user accounts.
- the query processing cluster 130 of a preferred embodiment functions to convert a query instruction communicated through a query interface to a data processing method. Depending on the query instruction, real-time data and/or archived data may be processed.
- the query processing cluster 130 is preferably configured on a remote distributed computing infrastructure such as Amazon's Elastic Compute Cloud.
- the query processing cluster 130 may be any suitable collection of a plurality of processing units, virtual or physical, that are preferably distributed but may be co-located or positioned in any suitable configuration. In a multitenant implementation of the system, queries from a plurality of different user accounts are preferably distributed amongst the shared resources of the query processing cluster 130 .
- the query processing cluster 130 is preferably a data-intensive processing cluster such as Hadoop.
- the data-intensive processing cluster is preferably configured to execute data mapping and reduction processes in querying data. More preferably the query processing cluster 130 is a Hadoop cluster implementing MapReduce processes.
- a single query processing module preferably queries the real-time storage system 122 and the archive storage system 124 . More preferably, a query processing module of the query processing cluster 130 preferably queries the real-time storage system 122 and the archive storage system 124 in a cooperative and coordinated manner to return a unified, cohesive query result that incorporates real-time and archived data.
- the query processing cluster 130 may include a real-time query module and a batch query module.
- a real-time query module preferably performs queries on the real-time storage system to provide time relevant time analysis.
- the real-time query module may use a real-time optimized Hadoop base for performing queries.
- a batch query module functions to perform big data queries on the archived storage system 124 .
- the batch query module preferably utilizes a Hadoop infrastructure to facilitate working with the big data.
- the archived storage system is optimized to provide large volumes of data in a format efficient for query processing.
- the query interface 140 of a preferred embodiment functions to provide a user access to the stored data.
- a query interface 140 is preferably used by a user to obtain information about data captured by the client data agent 110 .
- the query interface 140 may be used for requesting information but may also be used for interacting with the data such as creating a database or table, importing data, scheduling a query, setting a schema for the data, or any suitable action.
- the query interface takes user actions and preferably communicates them to the query processing cluster 130 .
- the query interface is a command line interface.
- a command line interface preferably enables users to issue relational database-like query commands to interact with data.
- the command line interface is preferably modeled to be similar to that of working with a relational database such as MySQL even though the backend infrastructure is drastically different.
- a method for integrating with a data-intensive infrastructure of a preferred embodiment includes retrieving discrete, client data S 110 ; storing discrete, client data in a remote storage system S 120 ; wherein storing the data includes storing client data in a real-time storage system S 122 ; and storing client data in a columnar format in an archive storage system S 124 ; receiving a data request through a user interface S 130 ; and selectively interacting with the data from the real-time storage system and the archive storage system S 140 .
- the method functions to enable applications built on any infrastructure to benefit from the data processing capabilities of a big data specialized infrastructure.
- the method preferably leverages the columnar-based distributed archive storage system and a data intensive processing cluster (e.g., Hadoop) to provide fast and scalable data analytics and services.
- the method takes unorganized discrete event recordings of a wide range of client types and transforms the data into queryable organized data.
- the method is preferably used in big data applications (e.g., datasets larger than terabyte or in some cases a petabyte).
- the method is preferably utilized by a data processing service provider.
- the service provider preferably enables a plurality of clients to utilize the infrastructure in a multi-tenant environment.
- the method is preferably implemented on a system as described above, but any suitable system may alternatively implement the method.
- Step S 110 which includes retrieving discrete, client data, functions to receive small pieces of client data from a client application.
- the discrete, client data is preferably collected by a plug-in, library, or application programming interface integrated with a client application.
- the client data is preferably event data including a timestamp associated with the parameters of the client data.
- Client data is preferably received at a distributed storage system. More preferably a data import module receives the data from a client data agent through an application programming interface (API) and appropriately adds the client data to the storage system.
- API application programming interface
- a client agent integrated with the client application preferably retrieves events from a data source, and more preferably the events are retrieved as the events occur.
- the client data may alternatively be any suitable form of data.
- An application of a client can preferably easily integrate with such a client agent, without making system level changes to the application.
- this may include adding a data agent gem, and performing some initial setup such as specifying a schema.
- the data agent can then relay appropriate information.
- Retrieving discrete, client data may include at a client data agent retrieving logs, converting relational databases of an application, receiving event notifications, retrieving online data, retrieving sensor data, and/or performing any suitable action to obtain data.
- the client agent may additionally interface with an input plugin.
- the input plugin may be configured to retrieve any suitable form of data. In some situations a plurality of sources are used to retrieve discrete, client event data.
- Client event data is preferable key-value pair data characterizing an event or an object to be stored.
- retrieving client event data may include compressing client event data.
- Compressing preferably includes serializing the event data, which may comprise of transforming the data from a human-readable format to a retrievable binary serialization in a JSON-like format.
- MessagePack library is one suitable data interchange format.
- the data agent may additionally buffer communication to avoid overloading a data importing system of the storage system.
- the method may additionally enable dynamic schema. Dynamic schema functions to enable flexibly setting data schema. One possible benefit is that schema information for client data is not required prior to adding data.
- a generic schema may be applied to data by default. The schema of data may later be updated through a data query request and interacting with the data, which can result in enhanced performances for subsequent queries.
- Step S 120 which includes storing discrete, client data in a remote storage system, functions to transfer data from an application to storage infrastructure of a big data specialized system.
- a client agent preferably periodically communicates client event data to the storage system. Alternatively, the client agent can continuously communicate client event data to the storage system.
- the event data is preferably in a compressed format for communication. Storing the event data in the storage system preferably enables the event data to be organized for optimized analytics.
- the client data is preferably stored as time series data. The data may include a timestamp for the collection of the data, for the storing in real-time storage system, for the storing in archive storage, and/or any suitable timestamp.
- storing client event data additionally includes storing client event data in a real-time storage system S 122 and storing client data in a columnar format in an archive storage system S 124 , which function to store data in formats suitable for real-time applications and big data applications respectively.
- the storage system preferably initially stores event data in a real-time storage system.
- data is moved from the real-time data storage system to the archived storage system, though event data may alternatively be replicated within sub-storage systems. Storing in these two sub-storage systems enables data to be formatted for different applications.
- Real-time data may have more queries to look at events as a whole.
- the event data is preferably stored in row format in the real-time storage system.
- Row format preferably simplifies appending a new row to the data, adding new keys, and is preferably suitable for writing.
- Initially storing in the real-time data storage additionally acts as a buffer as data is merged or otherwise converted into a structure suitable for archived storage.
- the event data is preferably converted to a MapReduce format for storing in the archive storage system, which functions to make the archived event data to be more accessible for batch queries.
- the event data is preferably stored in a columnar format.
- Columnar format preferably enables unnecessary keys and values to be skipped when reading.
- data requests are received through a command line interface and the requests may additionally be converted from a relational database-like query command format to a format compatible for the data storage and query processing cluster.
- SQL like database commands are preferably converted to a MapReduce format suitable for use with Hadoop.
- data requests are received through a business intelligence tool interface.
- the business intelligence tool interface is preferably a driver for a Java Database Connectivity (JDBC) or Open Database Connectivity (ODBC) interface.
- JDBC Java Database Connectivity
- ODBC Open Database Connectivity
- Such business intelligence tool interfaces preferably enables data access and interactions from various business intelligence tools such as Jasper Soft, Tableau, Excel, Pentaho, or any suitable tool.
- Step S 140 which includes selectively interacting with the data from the real-time storage system and the archive storage system, functions to perform the data request.
- a query-processing cluster preferably selectively queries the real-time storage system and/or the archived storage system.
- the real-time storage system is preferably queried.
- the archived storage system is queried.
- Data in the archived storage system is preferably processed quicker and more efficiently because the data is preferably stored in a columnar format.
- a query mechanism preferably allows part of the data to be selectively queried as opposed to querying the whole columnar dataset.
- a range-get operation for a query on a single parameter of client event data point can be retrieved substantially faster and efficiently than retrieving such data arranged in a row format.
- the system may alternatively select all of a columnar dataset.
- the query processing cluster may additionally analyze the query request and limit the queried data. Such query oversight can preferably provide performance improvements.
- the query processing cluster preferably canonizes the data using the specified schema.
- the query processing cluster may decompress the data for delivery to the user. For example, if data was compressed using MessagePack, the data is preferably converted to the original format.
- results or sub-calculations of a data request may be cached for future data requests.
- the method may include the step of returning structured data, which functions to return results of a data query.
- the data results are preferably communicated through the query interface used to make the data request.
- the results may be represented in an infographic, text based result summary, as table styled data, a spreadsheet, a csv-data file, or in any suitable data format.
- the selective interaction preferably includes cooperatively querying the real-time storage system and the archive storage system, which functions to distribute the query over multiple storage systems to retrieve a single cohesive query result.
- Query processing cluster preferably leverages the timestamps of the various client data in the real-time storage system and the archive storage system to query non-redundant client data.
- merge status information preferably indicates what client data has been added to the archive storage system.
- the merge status information may be a merge log, an indicator of latest merge block, or any suitable parameter(s) that can indicate the redundancy of data between the real-time storage and the archive storage.
- this is a log of time blocks that have completed merging, in the progress of merging, and/or awaiting merging.
- the query cluster preferably uses the merge status information and client data timestamps to appropriately skip data from either the real-time storage system or the archive storage system if the skipped data will be accounted for in the other storage system as shown in FIG. 7 .
- An alternative embodiment preferably implements the above methods in a computer-readable medium storing computer-readable instructions.
- the instructions are preferably executed by computer-executable components preferably integrated with a data agent, a storage system, a query processing cluster, and an interface.
- the computer-readable medium may be stored on any suitable computer readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device.
- the computer-executable component is preferably a processor but the instructions may alternatively or additionally be executed by any suitable dedicated hardware device.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A system and method for operating a big-data platform that includes at a data analysis platform, receiving discrete client data; storing the client data in a network accessible distributed storage system that includes: storing the client data in a real-time storage system; and merging the client data into a columnar-based distributed archive storage system; receiving a data query request through a query interface; and selectively interfacing with the client data from the real-time storage system and archive storage system according to the query.
Description
- This application claims the benefit of U.S. Provisional Application No. 61/558,395, filed 10 Nov. 2011, which is incorporated in its entirety by this reference.
- This invention relates generally to the data-intensive distributed systems field, and more specifically to a new and useful system and method for operating a big-data platform in the distributed systems field.
- Developments in technology have led to an increasing amount of available data. Such data has numerous useful applications and many companies could benefit from being able to analyze that data. Specialized systems have been developed to work with large datasets. Hadoop is one exemplary platform designed for large datasets that leverages MapReduce. However, many websites and applications are built on system infrastructures that differ greatly from those that can take advantage of large quantities of data. To take advantage of platforms such as Hadoop, a system would have to be re-engineered for the new platform. Such infrastructure changes are time and cost prohibitive. Additionally, a platform designed for large datasets requires highly specialized and knowledgeable personnel. Many of the people working on existing websites and applications do not have such skills. Thus, there is a need in the data-intensive distributed systems field to create a new and useful system and method for integrating with a data-intensive infrastructure. This invention provides such a new and useful system and method.
-
FIG. 1 is a schematic representation of a system of a preferred embodiment of the invention; -
FIG. 2 is an exemplary schematic representation of an implementation of a system of a preferred embodiment of the invention; -
FIG. 3 is a detailed schematic of an example of a data agent integrating with a client application; -
FIG. 4 is a representation of the conversion of user data to row and columnar formats for real-time and archived storage; -
FIG. 5 is an exemplary sample of a command line interface; -
FIG. 6 is a schematic representation of a method of a preferred embodiment of the invention; and -
FIG. 7 is a schematic representation of a variation cooperatively querying a real-time storage system and an archive storage system of a preferred embodiment of the invention. - The following description of the preferred embodiments of the invention is not intended to limit the invention to these preferred embodiments, but rather to enable any person skilled in the art to make and use this invention.
- As shown in
FIG. 1 , asystem 100 for operating a big-data platform of a preferred embodiment of the invention includes aclient data agent 110 integrated with a discrete event channel, a remotedistributed storage system 120, aquery processing cluster 130, and aquery interface 140. The remote distributed storage system may additionally include a real-time storage system 122 and anarchive storage system 124. The system functions to enable applications built on any infrastructure to benefit from the data processing capabilities of a specialized and dedicated infrastructure. As the system functions independently from the application of a client, the big data processing requirements are abstracted to a simple interface for the client. The system preferably uses a columnar-basedarchive storage system 124 and a data-intensivequery processing cluster 130 deployed on a distributed computing environment to enable fast data interactions on scalable system. The system may additionally be adapted to function as a multitenant service so that numerous clients can utilize the system simultaneously. Data of various users is preferably stored securely and in a format isolated from other user data. The query processing cluster is preferably shared by users of the multitenant infrastructure. Thesystem 100 is suitable for use with big datasets such as datasets measured in the terabytes, petabytes, or even greater data size. The system has the benefits of using generally discrete and unorganized data encountered periodically during the operation of an application and converting such unstructured data into a structured data from large dataset. As a related benefit the system enables existing applications and data-sources to easily and organically benefit from the big-data platform. This may be true even for existing application and data-sources that were designed without any considerations for big-data processing. Put another way, the system enables traditional applications to benefit from analysis of data that requires non-traditional data management tools. As shown inFIG. 2 , the system can preferably be built utilizing Hadoop and cloud hosting services for some base components. The system may alternatively include any suitable components or be built on any suitable infrastructure. - The client data agent no of a preferred embodiment functions to integrate with client systems to communicate small portions of data to the remote distributed storage system. An event channel or interface preferably enables a website, application, service, or any suitable networked program to supply event data. The client data (e.g., event data) is preferably discrete event data. Discrete event data preferably has an association to the time of an event, the kind of event, and/or any suitable additional event attributes. An event channel may be a database log, app log, syslog, sensor data stream, web content, or any suitable source of data. Typically, an application will generate client data in response to actions of a user of the application, and the client data will include information of the action. Preferably, the client data is formatted as a plurality of key-value pairs. A sequence of client data is preferably organized in row format. Alternatively, the client data may have any suitable format. The
client data agent 110 is preferably a daemon providing a gateway to the data-intensive infrastructure. The client data agent is preferably a plugin. For example the client data agent may be a ruby gem, but may alternatively be a web service, a library, a script, an application programming interface (API), or any suitable agent with an interface to client systems and a communication channel to thestorage system 120. Additionally, the data agent may be integrated into multiple portions of an application as shown inFIG. 3 . Theclient data agent 110 preferably includes networking interface to periodically communicate with thestorage system 120. The client data agent may additionally include a data compressor 112 that functions to compress event data before communicating to thestorage system 120. The data compressor 112 preferably is a data binary serializer, and preferably converts data to MessagePack or any suitable data interchange format. MessagePack is a small and compact binary serialization format with a JSON-like syntax. Additionally the client data agent no may include configuration to identify the client, user account, or application when communicating with the remote storage system. A private key may be used to uniquely identify event data and distinguish the event data from other clients using the multitenant system. - The client data agent no may additionally include a pluggable architecture to enable arbitrary data sources to be used in the client data agent. The pluggable architecture preferably includes an input plugin interface and an output plugin interface. The input plugin interface is preferably configured to allow plugins from third party developers or any suitable entity to be developed to pull data from any suitable source. For example, an input plugin may be written to pull data in from a particular site. The output plugin interface is preferably configured to allow plugins from third party developers or any suitable entity to be developed to customize the output of data from the client data agent. The output plugin may translate data, customize the transmission of data, or provide any customized functionality for the output of the
client data agent 110. In one example, the output plugin may convert the data units or language. In another example, the output plugin may be used to transmit the data to multiple destinations. - The remote distributed
storage system 120 of a preferred embodiment functions to store the event data. The event data coming into the distributed storage system is preferably discrete (e.g., organized by time based events) and substantially unorganized. Thestorage system 120 in combination with the query processing cluster enables organized big data analysis to be output from the system. Big data may be characterized as greater than a terabyte of data, a petabyte of data, or any suitable size. In one exemplary application, big data is characterized by greater than 1 terabyte of data storing a billion web clicks, but may alternatively be characterized by any suitable sized data. The storage system is preferably hosted on a distributed cloud storage system such as Amazon's S3/EC2, but may alternatively be any suitable infrastructure. The data is preferably kept in a compressed format (e.g., MessagePack format). The remote distributed storage system preferably functions to store data for both real-time querying and for large batch data querying, but may alternatively partition and group data in any suitable way. - The remote distributed storage system preferably includes a real-
time storage system 122 and anarchive storage system 124. The remote distributedstorage system 120 is preferably structured to initially store discrete data sent from theclient data agent 110 in the real-time storage system 122, and after a temporal delay, a data converter merges or otherwise prepares the data for storage in thearchive storage system 124. As shown inFIG. 4 , the real-time storage system 122 preferably stores the client data as row data, and thearchive storage system 124 preferably stores the data in a columnar data format. The data is preferable merged and converted to columnar-style using a MapReduce job. Row file format has the benefit of being easy to append new rows, add new keys, and is suitable for writing data. Columnar file format has the benefit of enabling unnecessary keys and values to be skipped, can be highly compressed due to coherency, and is suitable for archiving and reading. The columnar data format additionally can provide optimizations and efficiencies when querying the large data sets of the archive storage system. If the system is implemented for multitenant application, the data may additionally be stored according to an account identifier. The data may additionally be encrypted or suitable secured. Data for different users is preferably kept isolated from access from other unauthorized user accounts. - The
query processing cluster 130 of a preferred embodiment functions to convert a query instruction communicated through a query interface to a data processing method. Depending on the query instruction, real-time data and/or archived data may be processed. Thequery processing cluster 130 is preferably configured on a remote distributed computing infrastructure such as Amazon's Elastic Compute Cloud. Thequery processing cluster 130 may be any suitable collection of a plurality of processing units, virtual or physical, that are preferably distributed but may be co-located or positioned in any suitable configuration. In a multitenant implementation of the system, queries from a plurality of different user accounts are preferably distributed amongst the shared resources of thequery processing cluster 130. Thequery processing cluster 130 is preferably a data-intensive processing cluster such as Hadoop. The data-intensive processing cluster is preferably configured to execute data mapping and reduction processes in querying data. More preferably thequery processing cluster 130 is a Hadoop cluster implementing MapReduce processes. A single query processing module preferably queries the real-time storage system 122 and thearchive storage system 124. More preferably, a query processing module of thequery processing cluster 130 preferably queries the real-time storage system 122 and thearchive storage system 124 in a cooperative and coordinated manner to return a unified, cohesive query result that incorporates real-time and archived data. - In an alternative embodiment, the
query processing cluster 130 may include a real-time query module and a batch query module. A real-time query module preferably performs queries on the real-time storage system to provide time relevant time analysis. The real-time query module may use a real-time optimized Hadoop base for performing queries. A batch query module functions to perform big data queries on thearchived storage system 124. The batch query module preferably utilizes a Hadoop infrastructure to facilitate working with the big data. The archived storage system is optimized to provide large volumes of data in a format efficient for query processing. - The
query interface 140 of a preferred embodiment functions to provide a user access to the stored data. Aquery interface 140 is preferably used by a user to obtain information about data captured by theclient data agent 110. Thequery interface 140 may be used for requesting information but may also be used for interacting with the data such as creating a database or table, importing data, scheduling a query, setting a schema for the data, or any suitable action. The query interface takes user actions and preferably communicates them to thequery processing cluster 130. In one variation, the query interface is a command line interface. A command line interface preferably enables users to issue relational database-like query commands to interact with data. The command line interface is preferably modeled to be similar to that of working with a relational database such as MySQL even though the backend infrastructure is drastically different. This functions to allow users without in-depth knowledge of the data-intensive infrastructure (e.g., Hadoop) to issue to control and interact with the data using familiar commands as shown inFIG. 5 . In a second variation, thequery interface 140 is a web console interface. The web console is preferably website or application providing a graphical user interface. Data visualizations, analytics reports, and any suitable form of data representation may be part of such an interface. In a third variation, thequery interface 140 may be an API used by third party applications. In a fourth variation, thequery interface 140 may be business intelligence tool driver. For example, a Java Database Connectivity (JDBC) or Open Database Connectivity (ODBC) interface can enable access from various business intelligence tools such as Jasper Soft, Tableau, Excel, Pentaho, or any suitable tool. Any of the above variations or other suitable query interfaces may be used in any suitable combination. - 2. Method for Integrating with a Data-Intensive Infrastructure
- As shown in
FIG. 6 , a method for integrating with a data-intensive infrastructure of a preferred embodiment includes retrieving discrete, client data S110; storing discrete, client data in a remote storage system S120; wherein storing the data includes storing client data in a real-time storage system S122; and storing client data in a columnar format in an archive storage system S124; receiving a data request through a user interface S130; and selectively interacting with the data from the real-time storage system and the archive storage system S140. The method functions to enable applications built on any infrastructure to benefit from the data processing capabilities of a big data specialized infrastructure. The method preferably leverages the columnar-based distributed archive storage system and a data intensive processing cluster (e.g., Hadoop) to provide fast and scalable data analytics and services. The method takes unorganized discrete event recordings of a wide range of client types and transforms the data into queryable organized data. The method is preferably used in big data applications (e.g., datasets larger than terabyte or in some cases a petabyte). The method is preferably utilized by a data processing service provider. The service provider preferably enables a plurality of clients to utilize the infrastructure in a multi-tenant environment. The method is preferably implemented on a system as described above, but any suitable system may alternatively implement the method. - Step S110, which includes retrieving discrete, client data, functions to receive small pieces of client data from a client application. The discrete, client data is preferably collected by a plug-in, library, or application programming interface integrated with a client application. The client data is preferably event data including a timestamp associated with the parameters of the client data. Client data is preferably received at a distributed storage system. More preferably a data import module receives the data from a client data agent through an application programming interface (API) and appropriately adds the client data to the storage system. A client agent integrated with the client application preferably retrieves events from a data source, and more preferably the events are retrieved as the events occur. The client data may alternatively be any suitable form of data. An application of a client can preferably easily integrate with such a client agent, without making system level changes to the application. In the case of a Ruby on Rails application, this may include adding a data agent gem, and performing some initial setup such as specifying a schema. The data agent can then relay appropriate information. Retrieving discrete, client data may include at a client data agent retrieving logs, converting relational databases of an application, receiving event notifications, retrieving online data, retrieving sensor data, and/or performing any suitable action to obtain data. The client agent may additionally interface with an input plugin. The input plugin may be configured to retrieve any suitable form of data. In some situations a plurality of sources are used to retrieve discrete, client event data. Client event data is preferable key-value pair data characterizing an event or an object to be stored.
- Additionally, retrieving client event data may include compressing client event data. Compressing preferably includes serializing the event data, which may comprise of transforming the data from a human-readable format to a retrievable binary serialization in a JSON-like format. MessagePack library is one suitable data interchange format. As the data agent communicates event data to a storage system, the data agent may additionally buffer communication to avoid overloading a data importing system of the storage system. The method may additionally enable dynamic schema. Dynamic schema functions to enable flexibly setting data schema. One possible benefit is that schema information for client data is not required prior to adding data. A generic schema may be applied to data by default. The schema of data may later be updated through a data query request and interacting with the data, which can result in enhanced performances for subsequent queries.
- Step S120, which includes storing discrete, client data in a remote storage system, functions to transfer data from an application to storage infrastructure of a big data specialized system. A client agent preferably periodically communicates client event data to the storage system. Alternatively, the client agent can continuously communicate client event data to the storage system. As discussed above, the event data is preferably in a compressed format for communication. Storing the event data in the storage system preferably enables the event data to be organized for optimized analytics. The client data is preferably stored as time series data. The data may include a timestamp for the collection of the data, for the storing in real-time storage system, for the storing in archive storage, and/or any suitable timestamp.
- As two preferred sub-steps, storing client event data additionally includes storing client event data in a real-time storage system S122 and storing client data in a columnar format in an archive storage system S124, which function to store data in formats suitable for real-time applications and big data applications respectively. The storage system preferably initially stores event data in a real-time storage system. Preferably, after some temporal delay, data is moved from the real-time data storage system to the archived storage system, though event data may alternatively be replicated within sub-storage systems. Storing in these two sub-storage systems enables data to be formatted for different applications. Real-time data may have more queries to look at events as a whole. The event data is preferably stored in row format in the real-time storage system. Row format preferably simplifies appending a new row to the data, adding new keys, and is preferably suitable for writing. Initially storing in the real-time data storage additionally acts as a buffer as data is merged or otherwise converted into a structure suitable for archived storage. The event data is preferably converted to a MapReduce format for storing in the archive storage system, which functions to make the archived event data to be more accessible for batch queries. In converting to a MapReduce format, the event data is preferably stored in a columnar format. Columnar format preferably enables unnecessary keys and values to be skipped when reading. Columnar format additionally has improved compression rate and is suitable for archiving and reading. Conversion to the MapReduce format preferably occurs within the storage system to alleviate clients from performing the difficult conversion process.
- Step S130, which includes receiving a data request through a user interface, functions to obtain instructions indicating the intensions of a user or system. The data request is preferably obtained through an interface, and then subsequently communicated to a query processing cluster such as a Hadoop enabled cluster. The data request is preferably communicated through a user interfaces such as a command line interface, a web console, an API, a business intelligence tool interface, or through any suitable mechanism. Preferably, a data request is a query request that requires a response of corresponding structured data from the storage system. Alternatively, a data request may be an instruction requesting the modification to the stored data such as by creating a database or table, setting a schema, importing data interactively, or performing any suitable action. In one variation, data requests are received through a command line interface and the requests may additionally be converted from a relational database-like query command format to a format compatible for the data storage and query processing cluster. For example, SQL like database commands are preferably converted to a MapReduce format suitable for use with Hadoop. In another variation, data requests are received through a business intelligence tool interface. The business intelligence tool interface is preferably a driver for a Java Database Connectivity (JDBC) or Open Database Connectivity (ODBC) interface. Such business intelligence tool interfaces preferably enables data access and interactions from various business intelligence tools such as Jasper Soft, Tableau, Excel, Pentaho, or any suitable tool.
- Step S140, which includes selectively interacting with the data from the real-time storage system and the archive storage system, functions to perform the data request. For data queries, a query-processing cluster preferably selectively queries the real-time storage system and/or the archived storage system. When the query is for real-time information, the real-time storage system is preferably queried. When the query is a batch query, the archived storage system is queried. Data in the archived storage system is preferably processed quicker and more efficiently because the data is preferably stored in a columnar format. A query mechanism preferably allows part of the data to be selectively queried as opposed to querying the whole columnar dataset. For example, leveraging the columnar format and the time-series configuration of the archive data, a range-get operation for a query on a single parameter of client event data point can be retrieved substantially faster and efficiently than retrieving such data arranged in a row format. The system may alternatively select all of a columnar dataset. The query processing cluster may additionally analyze the query request and limit the queried data. Such query oversight can preferably provide performance improvements. In the variation where the data request is setting a schema, the query processing cluster preferably canonizes the data using the specified schema. Additionally, the query processing cluster may decompress the data for delivery to the user. For example, if data was compressed using MessagePack, the data is preferably converted to the original format. Additionally, the results or sub-calculations of a data request may be cached for future data requests. The method may include the step of returning structured data, which functions to return results of a data query. The data results are preferably communicated through the query interface used to make the data request. The results may be represented in an infographic, text based result summary, as table styled data, a spreadsheet, a csv-data file, or in any suitable data format.
- In querying data, the selective interaction preferably includes cooperatively querying the real-time storage system and the archive storage system, which functions to distribute the query over multiple storage systems to retrieve a single cohesive query result. Query processing cluster preferably leverages the timestamps of the various client data in the real-time storage system and the archive storage system to query non-redundant client data. When client data is being merged or otherwise structured for storage in the archive storage system, merge status information preferably indicates what client data has been added to the archive storage system. The merge status information may be a merge log, an indicator of latest merge block, or any suitable parameter(s) that can indicate the redundancy of data between the real-time storage and the archive storage. Preferably this is a log of time blocks that have completed merging, in the progress of merging, and/or awaiting merging. The query cluster preferably uses the merge status information and client data timestamps to appropriately skip data from either the real-time storage system or the archive storage system if the skipped data will be accounted for in the other storage system as shown in
FIG. 7 . - An alternative embodiment preferably implements the above methods in a computer-readable medium storing computer-readable instructions. The instructions are preferably executed by computer-executable components preferably integrated with a data agent, a storage system, a query processing cluster, and an interface. The computer-readable medium may be stored on any suitable computer readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device. The computer-executable component is preferably a processor but the instructions may alternatively or additionally be executed by any suitable dedicated hardware device.
- As a person skilled in the art will recognize from the previous detailed description and from the figures and claims, modifications and changes can be made to the preferred embodiments of the invention without departing from the scope of this invention defined in the following claims.
Claims (19)
1. A method for operating a big-data platform comprising:
at a data analysis platform, receiving discrete client data;
storing the client data in a network accessible distributed storage system that includes:
storing the client data in a real-time storage system;
merging the client data into a columnar-based distributed archive storage system;
receiving a data query request through a query interface; and
selectively interfacing with the client data from the real-time storage system and archive storage system according to the query.
2. The method of claim 1 , wherein selectively interfacing with the client data includes at a data-intensive processing cluster, processing the data query according to a data mapping and reduction processes in querying data from the real-time storage system and archive storage system.
3. The method of claim 2 , wherein the data-intensive processing cluster is a Hadoop cluster and the data mapping and reduction process is a MapReduce implementation.
4. The method of claim 1 , wherein discrete client data is received and stored with dynamic schema.
5. The method of claim 4 , wherein the data query request includes a schema definition and wherein selectively interfacing with the client data includes applying the schema definition to the dynamic schema.
6. The method of claim 1 , further comprising at a client data agent collecting client data and transmitting the client data to the data analysis platform.
7. The method of claim 6 , wherein the client data agent is integrated into an event channel from which client data is collected.
8. The method of claim 7 , wherein the event channel is selected from the list consisting of syslog, a relational database, cloud data, and sensor data.
9. The method of claim 6 , further comprising at the client data agent serializing data into a binary serialization data-interchange that is transmitted to the data analysis platform.
10. The method of claim 6 , wherein collecting client data is collected through a client agent data-input plugin.
11. The method of claim 1 , wherein the columnar-based distributed archive storage system stores client data in time series order, and wherein selectively interfacing with client data includes querying data from distributed storage system.
12. The method of claim 11 , wherein querying client data from distributed storage system includes cooperatively querying the real-time storage system and the archive storage system for a cohesive query result.
13. The method of claim 1 , wherein receiving a data query includes converting relational database styled query to data-intensive cluster query process.
14. The method of claim 1 , wherein the data query request is received through a infographics interface and further comprising returning an infographic from the selectively interfaced client data.
15. The method of claim 1 , wherein receiving a data query includes receiving the data query through a business intelligence tool driver and further comprising returning data analytics results to the business intelligence tool driver.
16. The method of claim 1 , wherein client data is associated with a user account through unique identifier.
17. The method of claim 16 , [scalable querying and isolated data storage], wherein client data merged into the archive data storage is isolated according to the user account associated with the client data and a query processing cluster interfaces with the distributed storage system, and the query processing cluster is shared between by a plurality of user accounts.
18. The method of claim 1 , further comprising at a client data agent collecting client data and transmitting the client data to the data analysis platform; wherein the columnar-based distributed archive storage system stores client data in time series order with a dynamic schema, and wherein selectively interfacing with client data includes cooperatively querying data from the real-time storage system and the archive storage system for a cohesive query result.
19. The method of claim 18 , wherein distributed storage system includes over one petabyte of data.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/671,896 US20130124483A1 (en) | 2011-11-10 | 2012-11-08 | System and method for operating a big-data platform |
US15/147,790 US9582528B2 (en) | 2011-11-10 | 2016-05-05 | System and method for operating a big-data platform |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201161558395P | 2011-11-10 | 2011-11-10 | |
US13/671,896 US20130124483A1 (en) | 2011-11-10 | 2012-11-08 | System and method for operating a big-data platform |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/147,790 Continuation US9582528B2 (en) | 2011-11-10 | 2016-05-05 | System and method for operating a big-data platform |
Publications (1)
Publication Number | Publication Date |
---|---|
US20130124483A1 true US20130124483A1 (en) | 2013-05-16 |
Family
ID=48281607
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/671,896 Abandoned US20130124483A1 (en) | 2011-11-10 | 2012-11-08 | System and method for operating a big-data platform |
US15/147,790 Active US9582528B2 (en) | 2011-11-10 | 2016-05-05 | System and method for operating a big-data platform |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/147,790 Active US9582528B2 (en) | 2011-11-10 | 2016-05-05 | System and method for operating a big-data platform |
Country Status (3)
Country | Link |
---|---|
US (2) | US20130124483A1 (en) |
JP (1) | JP5818394B2 (en) |
WO (1) | WO2013070873A1 (en) |
Cited By (77)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130139166A1 (en) * | 2011-11-24 | 2013-05-30 | Alibaba Group Holding Limited | Distributed data stream processing method and system |
US20130198186A1 (en) * | 2012-01-28 | 2013-08-01 | Microsoft Corporation | Determination of relationships between collections of disparate media types |
US20140237554A1 (en) * | 2013-02-15 | 2014-08-21 | Infosys Limited | Unified platform for big data processing |
US20140244628A1 (en) * | 2011-12-22 | 2014-08-28 | Sap Ag | Hybrid Database Table Stored as Both Row and Column Store |
CN104077398A (en) * | 2014-06-30 | 2014-10-01 | 中华电信股份有限公司 | System and method for job assignment based on Hadoop multi-cluster environment |
US20140298041A1 (en) * | 2013-03-29 | 2014-10-02 | Dropbox, Inc. | Portable computing device with methodologies for client-side analytic data collection |
CN104112010A (en) * | 2014-07-16 | 2014-10-22 | 深圳市国泰安信息技术有限公司 | Data storage method and device |
CN104156465A (en) * | 2014-08-22 | 2014-11-19 | 金石易诚(北京)科技有限公司 | Real-time webpage synchronization and background distributed data storage system |
US20150032743A1 (en) * | 2013-07-18 | 2015-01-29 | CIMCON Software, Inc. | Analyzing files using big data tools |
US20150073958A1 (en) * | 2013-09-12 | 2015-03-12 | Bank Of America Corporation | RESEARCH REPORT RECOMMENDATION ENGINE ("R+hu 3 +lE") |
WO2015038224A1 (en) * | 2013-09-13 | 2015-03-19 | Nec Laboratories America, Inc. | Systems and methods for tuning multi-store systems to speed up big data query workload |
US20150095308A1 (en) * | 2013-10-01 | 2015-04-02 | Cloudera, Inc. | Background format optimization for enhanced sql-like queries in hadoop |
WO2015070236A1 (en) * | 2013-11-11 | 2015-05-14 | Amazon Technologies, Inc. | Partition-based data stream processing framework |
US20150134796A1 (en) * | 2013-11-11 | 2015-05-14 | Amazon Technologies, Inc. | Dynamic partitioning techniques for data streams |
US9082127B2 (en) | 2010-03-31 | 2015-07-14 | Cloudera, Inc. | Collecting and aggregating datasets for analysis |
US9081888B2 (en) | 2010-03-31 | 2015-07-14 | Cloudera, Inc. | Collecting and aggregating log data with fault tolerance |
WO2015125225A1 (en) * | 2014-02-19 | 2015-08-27 | 株式会社日立製作所 | Data processing system and data processing method |
TWI506456B (en) * | 2013-05-23 | 2015-11-01 | Chunghwa Telecom Co Ltd | System and method for dispatching hadoop jobs in multi-cluster environment |
US9201910B2 (en) | 2010-03-31 | 2015-12-01 | Cloudera, Inc. | Dynamically processing an event using an extensible data model |
WO2015183344A1 (en) * | 2014-05-28 | 2015-12-03 | Bluedata Software, Inc. | Inter-version mapping of distributed file systems |
US20150382209A1 (en) * | 2013-02-14 | 2015-12-31 | Nokia Solutions And Networks Oy | Method of adapting operation of self-organizing network functions |
WO2016065198A1 (en) * | 2014-10-22 | 2016-04-28 | Qubole, Inc. | High performance hadoop with new generation instances |
US9338008B1 (en) | 2012-04-02 | 2016-05-10 | Cloudera, Inc. | System and method for secure release of secret information over a network |
US9342557B2 (en) | 2013-03-13 | 2016-05-17 | Cloudera, Inc. | Low latency query engine for Apache Hadoop |
AU2015201361A1 (en) * | 2014-11-14 | 2016-06-02 | Tata Consultancy Services Limited | A method and system for efficient performance prediction of structured query for big data |
US9361263B1 (en) * | 2011-12-21 | 2016-06-07 | Emc Corporation | Co-located clouds, vertically integrated clouds, and federated clouds |
US20160314147A1 (en) * | 2015-04-24 | 2016-10-27 | Dell Software, Inc. | Partitioning target data to improve data replication performance |
US9537941B2 (en) | 2013-12-17 | 2017-01-03 | Wistron Corporation | Method and system for verifying quality of server |
CN107016128A (en) * | 2017-05-16 | 2017-08-04 | 郑州云海信息技术有限公司 | A kind of data processing method and device |
US9794135B2 (en) | 2013-11-11 | 2017-10-17 | Amazon Technologies, Inc. | Managed service for acquisition, storage and consumption of large-scale data streams |
US9811938B2 (en) | 2013-03-14 | 2017-11-07 | Business Objects Software Ltd. | Methods, apparatus and system for analytics replay utilizing random sampling |
US9898217B2 (en) | 2015-05-08 | 2018-02-20 | Sonicwall Inc. | Two stage memory allocation using a cache |
WO2018043767A1 (en) * | 2016-08-30 | 2018-03-08 | 디포커스(주) | Data collection method using data agent container and system therefor |
CN107895046A (en) * | 2017-11-30 | 2018-04-10 | 广东奥飞数据科技股份有限公司 | A kind of Heterogeneous Database Integration Platform |
US10046457B2 (en) | 2014-10-31 | 2018-08-14 | General Electric Company | System and method for the creation and utilization of multi-agent dynamic situational awareness models |
US10078683B2 (en) | 2013-07-02 | 2018-09-18 | Jpmorgan Chase Bank, N.A. | Big data centralized intelligence system |
US10097659B1 (en) * | 2014-12-16 | 2018-10-09 | Amazon Technologies, Inc. | High performance geographically distributed data storage, retrieval and update |
US10133800B2 (en) | 2013-09-11 | 2018-11-20 | Microsoft Technology Licensing, Llc | Processing datasets with a DBMS engine |
US10210173B2 (en) | 2014-06-27 | 2019-02-19 | International Business Machines Corporation | File storage processing in HDFS |
US10235780B2 (en) | 2015-12-18 | 2019-03-19 | Dropbox, Inc. | Integrated data exploration, modeling and visualization |
US10303786B2 (en) * | 2015-08-05 | 2019-05-28 | International Business Machines Corporation | Retrieving and displaying data from a columnar database |
CN109981698A (en) * | 2017-12-27 | 2019-07-05 | 博元森禾信息科技(北京)有限公司 | Number networking cross-domain data access standardized system and method based on metadata |
US10423493B1 (en) | 2015-12-21 | 2019-09-24 | Amazon Technologies, Inc. | Scalable log-based continuous data protection for distributed databases |
US10459881B2 (en) | 2015-02-27 | 2019-10-29 | Podium Data, Inc. | Data management platform using metadata repository |
US10496545B2 (en) * | 2015-11-24 | 2019-12-03 | Hewlett Packard Enterprise Development Lp | Data caching in a large-scale processing environment |
US10567500B1 (en) | 2015-12-21 | 2020-02-18 | Amazon Technologies, Inc. | Continuous backup of data in a distributed data store |
US10621049B1 (en) | 2018-03-12 | 2020-04-14 | Amazon Technologies, Inc. | Consistent backups based on local node clock |
CN111490999A (en) * | 2019-01-28 | 2020-08-04 | 上海菲碧文化传媒有限公司 | Big data processing analysis platform based on Hadoop |
US10733024B2 (en) | 2017-05-24 | 2020-08-04 | Qubole Inc. | Task packing scheduling process for long running applications |
US10754844B1 (en) | 2017-09-27 | 2020-08-25 | Amazon Technologies, Inc. | Efficient database snapshot generation |
US10768830B1 (en) | 2018-07-16 | 2020-09-08 | Amazon Technologies, Inc. | Streaming data service with isolated read channels |
US10798140B1 (en) | 2018-07-16 | 2020-10-06 | Amazon Technologies, Inc. | Stream data record reads using push-mode persistent connections |
US10831614B2 (en) | 2014-08-18 | 2020-11-10 | Amazon Technologies, Inc. | Visualizing restoration operation granularity for a database |
US10853536B1 (en) * | 2014-12-11 | 2020-12-01 | Imagars Llc | Automatic requirement verification engine and analytics |
US10853182B1 (en) | 2015-12-21 | 2020-12-01 | Amazon Technologies, Inc. | Scalable log-based secondary indexes for non-relational databases |
US10855754B1 (en) | 2018-07-16 | 2020-12-01 | Amazon Technologies, Inc. | Isolated read channel categories at streaming data service |
US10956246B1 (en) | 2018-07-16 | 2021-03-23 | Amazon Technologies, Inc. | Isolated read channel management interfaces at streaming data service |
US10990581B1 (en) | 2017-09-27 | 2021-04-27 | Amazon Technologies, Inc. | Tracking a size of a database change log |
US11042454B1 (en) | 2018-11-20 | 2021-06-22 | Amazon Technologies, Inc. | Restoration of a data source |
US11042503B1 (en) | 2017-11-22 | 2021-06-22 | Amazon Technologies, Inc. | Continuous data protection and restoration |
US11070600B1 (en) | 2018-07-16 | 2021-07-20 | Amazon Technologies, Inc. | Optimization techniques to support lagging readers at streaming data service |
US11075984B1 (en) | 2018-07-16 | 2021-07-27 | Amazon Technologies, Inc. | Workload management at streaming data service supporting persistent connections for reads |
US11080207B2 (en) | 2016-06-07 | 2021-08-03 | Qubole, Inc. | Caching framework for big-data engines in the cloud |
US11113121B2 (en) | 2016-09-07 | 2021-09-07 | Qubole Inc. | Heterogeneous auto-scaling big-data clusters in the cloud |
US11120034B2 (en) * | 2018-11-13 | 2021-09-14 | Sap Se | Automatic routing of connection requests in business intelligence (BI) tools |
US11126505B1 (en) | 2018-08-10 | 2021-09-21 | Amazon Technologies, Inc. | Past-state backup generator and interface for database systems |
US11144360B2 (en) | 2019-05-31 | 2021-10-12 | Qubole, Inc. | System and method for scheduling and running interactive database queries with service level agreements in a multi-tenant processing system |
CN113515610A (en) * | 2021-06-21 | 2021-10-19 | 中盾创新档案管理(北京)有限公司 | File management method based on object-oriented language processing |
US11182372B1 (en) | 2017-11-08 | 2021-11-23 | Amazon Technologies, Inc. | Tracking database partition change log dependencies |
US11228489B2 (en) | 2018-01-23 | 2022-01-18 | Qubole, Inc. | System and methods for auto-tuning big data workloads on cloud platforms |
US11269731B1 (en) | 2017-11-22 | 2022-03-08 | Amazon Technologies, Inc. | Continuous data protection |
US11385969B2 (en) | 2009-03-31 | 2022-07-12 | Amazon Technologies, Inc. | Cloning and recovery of data volumes |
US11436667B2 (en) | 2015-06-08 | 2022-09-06 | Qubole, Inc. | Pure-spot and dynamically rebalanced auto-scaling clusters |
US11474874B2 (en) | 2014-08-14 | 2022-10-18 | Qubole, Inc. | Systems and methods for auto-scaling a big data system |
US11704316B2 (en) | 2019-05-31 | 2023-07-18 | Qubole, Inc. | Systems and methods for determining peak memory requirements in SQL processing engines with concurrent subtasks |
US11747978B2 (en) | 2019-07-23 | 2023-09-05 | International Business Machines Corporation | Data compaction in distributed storage system |
US11755415B2 (en) | 2014-05-09 | 2023-09-12 | Amazon Technologies, Inc. | Variable data replication for storage implementing data backup |
Families Citing this family (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6357243B2 (en) * | 2013-11-11 | 2018-07-11 | アマゾン・テクノロジーズ・インコーポレーテッド | Data stream ingestion and persistence policy |
US9892179B2 (en) | 2014-05-21 | 2018-02-13 | International Business Machines Corporation | Managing queries in business intelligence platforms |
US9798762B2 (en) | 2015-11-30 | 2017-10-24 | International Business Machines Corporation | Real time big data master data management |
US10067817B2 (en) | 2016-05-25 | 2018-09-04 | International Business Machines Corporation | Equipment failure risk detection and prediction in industrial process |
US10614092B2 (en) | 2017-01-24 | 2020-04-07 | International Business Machines Corporation | Optimizing data retrieval operation in big-data processing systems |
CN107665233B (en) * | 2017-07-24 | 2020-07-31 | 深圳壹账通智能科技有限公司 | Database data processing method and device, computer equipment and storage medium |
US10719508B2 (en) * | 2018-04-19 | 2020-07-21 | Risk Management Solutions, Inc. | Data storage system for providing low latency search query responses |
CN109542871A (en) * | 2018-10-23 | 2019-03-29 | 高新兴科技集团股份有限公司 | Multi-source data management method, system and data management apparatus |
US11061856B2 (en) | 2019-07-03 | 2021-07-13 | Bank Of America Corporation | Data ingestion system |
US11507622B2 (en) | 2020-03-25 | 2022-11-22 | The Toronto-Dominion Bank | System and method for automatically managing storage resources of a big data platform |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060112123A1 (en) * | 2004-11-24 | 2006-05-25 | Macnica, Inc. | Spreadsheet user-interfaced business data visualization and publishing system |
US20080208820A1 (en) * | 2007-02-28 | 2008-08-28 | Psydex Corporation | Systems and methods for performing semantic analysis of information over time and space |
US20110153603A1 (en) * | 2009-12-17 | 2011-06-23 | Yahoo! Inc. | Time series storage for large-scale monitoring system |
US20110191361A1 (en) * | 2010-01-30 | 2011-08-04 | International Business Machines Corporation | System and method for building a cloud aware massive data analytics solution background |
US20120110004A1 (en) * | 2010-11-03 | 2012-05-03 | Microsoft Corporation | Homomorphism lemma for efficiently querying databases |
US20130151563A1 (en) * | 2011-12-07 | 2013-06-13 | Cisco Technology, Inc. | Network-based dynamic data management |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5966126A (en) * | 1996-12-23 | 1999-10-12 | Szabo; Andrew J. | Graphic user interface for database system |
US6081810A (en) * | 1998-02-03 | 2000-06-27 | Electronic Data Systems Corporation | Report database system |
US7783679B2 (en) * | 2005-01-12 | 2010-08-24 | Computer Associates Think, Inc. | Efficient processing of time series data |
US7877370B2 (en) * | 2006-05-15 | 2011-01-25 | Algebraix Data Corporation | Systems and methods for data storage and retrieval using algebraic relations composed from query language statements |
US7827125B1 (en) * | 2006-06-01 | 2010-11-02 | Trovix, Inc. | Learning based on feedback for contextual personalized information retrieval |
US8271429B2 (en) * | 2006-09-11 | 2012-09-18 | Wiredset Llc | System and method for collecting and processing data |
US9626421B2 (en) * | 2007-09-21 | 2017-04-18 | Hasso-Plattner-Institut Fur Softwaresystemtechnik Gmbh | ETL-less zero-redundancy system and method for reporting OLTP data |
JP5198929B2 (en) * | 2008-04-25 | 2013-05-15 | 株式会社日立製作所 | Stream data processing method and computer system |
US8239389B2 (en) * | 2008-09-29 | 2012-08-07 | International Business Machines Corporation | Persisting external index data in a database |
US7917463B2 (en) * | 2008-10-10 | 2011-03-29 | Business.Com, Inc. | System and method for data warehousing and analytics on a distributed file system |
JP4939568B2 (en) * | 2009-04-28 | 2012-05-30 | インターナショナル・ビジネス・マシーンズ・コーポレーション | Method for synchronizing data between databases, and computer system and computer program thereof |
US20110004589A1 (en) * | 2009-07-06 | 2011-01-06 | Rockwell Automation Technologies, Inc. | Diagnostics in a distributed directory system |
WO2011111532A1 (en) * | 2010-03-10 | 2011-09-15 | 日本電気株式会社 | Database system |
JP5464017B2 (en) * | 2010-04-01 | 2014-04-09 | 日本電気株式会社 | Distributed memory database system, database server, data processing method and program thereof |
US8954967B2 (en) * | 2011-05-31 | 2015-02-10 | International Business Machines Corporation | Adaptive parallel data processing |
-
2012
- 2012-11-08 US US13/671,896 patent/US20130124483A1/en not_active Abandoned
- 2012-11-08 WO PCT/US2012/064089 patent/WO2013070873A1/en active Application Filing
- 2012-11-08 JP JP2014541254A patent/JP5818394B2/en active Active
-
2016
- 2016-05-05 US US15/147,790 patent/US9582528B2/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060112123A1 (en) * | 2004-11-24 | 2006-05-25 | Macnica, Inc. | Spreadsheet user-interfaced business data visualization and publishing system |
US20080208820A1 (en) * | 2007-02-28 | 2008-08-28 | Psydex Corporation | Systems and methods for performing semantic analysis of information over time and space |
US20110153603A1 (en) * | 2009-12-17 | 2011-06-23 | Yahoo! Inc. | Time series storage for large-scale monitoring system |
US20110191361A1 (en) * | 2010-01-30 | 2011-08-04 | International Business Machines Corporation | System and method for building a cloud aware massive data analytics solution background |
US20120110004A1 (en) * | 2010-11-03 | 2012-05-03 | Microsoft Corporation | Homomorphism lemma for efficiently querying databases |
US20130151563A1 (en) * | 2011-12-07 | 2013-06-13 | Cisco Technology, Inc. | Network-based dynamic data management |
Non-Patent Citations (1)
Title |
---|
"IBM InfoSphere Release Notes" [online]. May 27, 2011. URL: <http://pic.dhe.ibm.com/infocenter/bigins/v1r1/index.jsp?topic=%2Fcom.ibm.swg.im.infosphere.biginsights.doc%2Fdoc%2Fr0057893.html>. * |
Cited By (111)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11914486B2 (en) | 2009-03-31 | 2024-02-27 | Amazon Technologies, Inc. | Cloning and recovery of data volumes |
US11385969B2 (en) | 2009-03-31 | 2022-07-12 | Amazon Technologies, Inc. | Cloning and recovery of data volumes |
US9082127B2 (en) | 2010-03-31 | 2015-07-14 | Cloudera, Inc. | Collecting and aggregating datasets for analysis |
US9201910B2 (en) | 2010-03-31 | 2015-12-01 | Cloudera, Inc. | Dynamically processing an event using an extensible data model |
US9081888B2 (en) | 2010-03-31 | 2015-07-14 | Cloudera, Inc. | Collecting and aggregating log data with fault tolerance |
US9727613B2 (en) * | 2011-11-24 | 2017-08-08 | Alibaba Group Holding Limited | Distributed data stream processing method and system |
US9250963B2 (en) * | 2011-11-24 | 2016-02-02 | Alibaba Group Holding Limited | Distributed data stream processing method and system |
US20130139166A1 (en) * | 2011-11-24 | 2013-05-30 | Alibaba Group Holding Limited | Distributed data stream processing method and system |
US20160179898A1 (en) * | 2011-11-24 | 2016-06-23 | Alibaba Group Holding Limited | Distributed data stream processing method and system |
US9361263B1 (en) * | 2011-12-21 | 2016-06-07 | Emc Corporation | Co-located clouds, vertically integrated clouds, and federated clouds |
US20140244628A1 (en) * | 2011-12-22 | 2014-08-28 | Sap Ag | Hybrid Database Table Stored as Both Row and Column Store |
US9864817B2 (en) * | 2012-01-28 | 2018-01-09 | Microsoft Technology Licensing, Llc | Determination of relationships between collections of disparate media types |
US20130198186A1 (en) * | 2012-01-28 | 2013-08-01 | Microsoft Corporation | Determination of relationships between collections of disparate media types |
US9338008B1 (en) | 2012-04-02 | 2016-05-10 | Cloudera, Inc. | System and method for secure release of secret information over a network |
US11304070B2 (en) * | 2013-02-14 | 2022-04-12 | Nokia Solutions And Networks Oy | Method of adapting operation of self-organizing network functions |
US20150382209A1 (en) * | 2013-02-14 | 2015-12-31 | Nokia Solutions And Networks Oy | Method of adapting operation of self-organizing network functions |
US9418241B2 (en) * | 2013-02-15 | 2016-08-16 | Infosys Limited | Unified platform for big data processing |
US20140237554A1 (en) * | 2013-02-15 | 2014-08-21 | Infosys Limited | Unified platform for big data processing |
US9990399B2 (en) | 2013-03-13 | 2018-06-05 | Cloudera, Inc. | Low latency query engine for apache hadoop |
US9342557B2 (en) | 2013-03-13 | 2016-05-17 | Cloudera, Inc. | Low latency query engine for Apache Hadoop |
US9811938B2 (en) | 2013-03-14 | 2017-11-07 | Business Objects Software Ltd. | Methods, apparatus and system for analytics replay utilizing random sampling |
US10445920B2 (en) | 2013-03-14 | 2019-10-15 | Business Objects Software Ltd. | Methods, apparatus and system for analytics replay utilizing random sampling |
US9813393B2 (en) | 2013-03-29 | 2017-11-07 | Dropbox, Inc. | System and method for client-side analytic data collection |
US8996889B2 (en) * | 2013-03-29 | 2015-03-31 | Dropbox, Inc. | Portable computing device with methodologies for client-side analytic data collection |
US20140298041A1 (en) * | 2013-03-29 | 2014-10-02 | Dropbox, Inc. | Portable computing device with methodologies for client-side analytic data collection |
TWI506456B (en) * | 2013-05-23 | 2015-11-01 | Chunghwa Telecom Co Ltd | System and method for dispatching hadoop jobs in multi-cluster environment |
US10078683B2 (en) | 2013-07-02 | 2018-09-18 | Jpmorgan Chase Bank, N.A. | Big data centralized intelligence system |
US20150032743A1 (en) * | 2013-07-18 | 2015-01-29 | CIMCON Software, Inc. | Analyzing files using big data tools |
US10133800B2 (en) | 2013-09-11 | 2018-11-20 | Microsoft Technology Licensing, Llc | Processing datasets with a DBMS engine |
US20150073958A1 (en) * | 2013-09-12 | 2015-03-12 | Bank Of America Corporation | RESEARCH REPORT RECOMMENDATION ENGINE ("R+hu 3 +lE") |
WO2015038224A1 (en) * | 2013-09-13 | 2015-03-19 | Nec Laboratories America, Inc. | Systems and methods for tuning multi-store systems to speed up big data query workload |
US20170032003A1 (en) * | 2013-10-01 | 2017-02-02 | Cloudera, Inc. | Background format optimization for enhanced sql-like queries in hadoop |
US20150095308A1 (en) * | 2013-10-01 | 2015-04-02 | Cloudera, Inc. | Background format optimization for enhanced sql-like queries in hadoop |
US10706059B2 (en) * | 2013-10-01 | 2020-07-07 | Cloudera, Inc. | Background format optimization for enhanced SQL-like queries in Hadoop |
AU2014240211B2 (en) * | 2013-10-01 | 2017-02-02 | Cloudera, Inc. | Background format optimization for enhanced sql-like queries in hadoop |
US11567956B2 (en) * | 2013-10-01 | 2023-01-31 | Cloudera, Inc. | Background format optimization for enhanced queries in a distributed computing cluster |
EP3901781A1 (en) * | 2013-10-01 | 2021-10-27 | Cloudera, Inc. | Background format optimization for enhanced sql-like queries in hadoop |
EP2857995A3 (en) * | 2013-10-01 | 2015-08-12 | Cloudera, Inc. | Background format optimization for enhanced SQL-like queries in hadoop |
US9477731B2 (en) * | 2013-10-01 | 2016-10-25 | Cloudera, Inc. | Background format optimization for enhanced SQL-like queries in Hadoop |
US11630830B2 (en) * | 2013-10-01 | 2023-04-18 | Cloudera Inc. | Background format optimization for enhanced queries in a distributed computing cluster |
US20150134796A1 (en) * | 2013-11-11 | 2015-05-14 | Amazon Technologies, Inc. | Dynamic partitioning techniques for data streams |
US10691716B2 (en) | 2013-11-11 | 2020-06-23 | Amazon Technologies, Inc. | Dynamic partitioning techniques for data streams |
US10635644B2 (en) | 2013-11-11 | 2020-04-28 | Amazon Technologies, Inc. | Partition-based data stream processing framework |
WO2015070236A1 (en) * | 2013-11-11 | 2015-05-14 | Amazon Technologies, Inc. | Partition-based data stream processing framework |
US9720989B2 (en) * | 2013-11-11 | 2017-08-01 | Amazon Technologies, Inc. | Dynamic partitioning techniques for data streams |
US9794135B2 (en) | 2013-11-11 | 2017-10-17 | Amazon Technologies, Inc. | Managed service for acquisition, storage and consumption of large-scale data streams |
US9537941B2 (en) | 2013-12-17 | 2017-01-03 | Wistron Corporation | Method and system for verifying quality of server |
WO2015125225A1 (en) * | 2014-02-19 | 2015-08-27 | 株式会社日立製作所 | Data processing system and data processing method |
US11755415B2 (en) | 2014-05-09 | 2023-09-12 | Amazon Technologies, Inc. | Variable data replication for storage implementing data backup |
US11080244B2 (en) | 2014-05-28 | 2021-08-03 | Hewlett Packard Enterprise Development Lp | Inter-version mapping of distributed file systems |
WO2015183344A1 (en) * | 2014-05-28 | 2015-12-03 | Bluedata Software, Inc. | Inter-version mapping of distributed file systems |
US10210173B2 (en) | 2014-06-27 | 2019-02-19 | International Business Machines Corporation | File storage processing in HDFS |
CN104077398A (en) * | 2014-06-30 | 2014-10-01 | 中华电信股份有限公司 | System and method for job assignment based on Hadoop multi-cluster environment |
CN104112010A (en) * | 2014-07-16 | 2014-10-22 | 深圳市国泰安信息技术有限公司 | Data storage method and device |
US11474874B2 (en) | 2014-08-14 | 2022-10-18 | Qubole, Inc. | Systems and methods for auto-scaling a big data system |
US10831614B2 (en) | 2014-08-18 | 2020-11-10 | Amazon Technologies, Inc. | Visualizing restoration operation granularity for a database |
CN104156465A (en) * | 2014-08-22 | 2014-11-19 | 金石易诚(北京)科技有限公司 | Real-time webpage synchronization and background distributed data storage system |
WO2016065198A1 (en) * | 2014-10-22 | 2016-04-28 | Qubole, Inc. | High performance hadoop with new generation instances |
US10606478B2 (en) | 2014-10-22 | 2020-03-31 | Qubole, Inc. | High performance hadoop with new generation instances |
US10046457B2 (en) | 2014-10-31 | 2018-08-14 | General Electric Company | System and method for the creation and utilization of multi-agent dynamic situational awareness models |
AU2015201361B2 (en) * | 2014-11-14 | 2017-03-02 | Tata Consultancy Services Limited | A method and system for efficient performance prediction of structured query for big data |
AU2015201361A1 (en) * | 2014-11-14 | 2016-06-02 | Tata Consultancy Services Limited | A method and system for efficient performance prediction of structured query for big data |
US10853536B1 (en) * | 2014-12-11 | 2020-12-01 | Imagars Llc | Automatic requirement verification engine and analytics |
US10097659B1 (en) * | 2014-12-16 | 2018-10-09 | Amazon Technologies, Inc. | High performance geographically distributed data storage, retrieval and update |
US10459881B2 (en) | 2015-02-27 | 2019-10-29 | Podium Data, Inc. | Data management platform using metadata repository |
US10671565B2 (en) * | 2015-04-24 | 2020-06-02 | Quest Software Inc. | Partitioning target data to improve data replication performance |
US20160314147A1 (en) * | 2015-04-24 | 2016-10-27 | Dell Software, Inc. | Partitioning target data to improve data replication performance |
US9898217B2 (en) | 2015-05-08 | 2018-02-20 | Sonicwall Inc. | Two stage memory allocation using a cache |
US11436667B2 (en) | 2015-06-08 | 2022-09-06 | Qubole, Inc. | Pure-spot and dynamically rebalanced auto-scaling clusters |
US10303786B2 (en) * | 2015-08-05 | 2019-05-28 | International Business Machines Corporation | Retrieving and displaying data from a columnar database |
US10496545B2 (en) * | 2015-11-24 | 2019-12-03 | Hewlett Packard Enterprise Development Lp | Data caching in a large-scale processing environment |
US10235780B2 (en) | 2015-12-18 | 2019-03-19 | Dropbox, Inc. | Integrated data exploration, modeling and visualization |
US10423493B1 (en) | 2015-12-21 | 2019-09-24 | Amazon Technologies, Inc. | Scalable log-based continuous data protection for distributed databases |
US11153380B2 (en) | 2015-12-21 | 2021-10-19 | Amazon Technologies, Inc. | Continuous backup of data in a distributed data store |
US10853182B1 (en) | 2015-12-21 | 2020-12-01 | Amazon Technologies, Inc. | Scalable log-based secondary indexes for non-relational databases |
US10567500B1 (en) | 2015-12-21 | 2020-02-18 | Amazon Technologies, Inc. | Continuous backup of data in a distributed data store |
US11080207B2 (en) | 2016-06-07 | 2021-08-03 | Qubole, Inc. | Caching framework for big-data engines in the cloud |
WO2018043767A1 (en) * | 2016-08-30 | 2018-03-08 | 디포커스(주) | Data collection method using data agent container and system therefor |
US11113121B2 (en) | 2016-09-07 | 2021-09-07 | Qubole Inc. | Heterogeneous auto-scaling big-data clusters in the cloud |
CN107016128A (en) * | 2017-05-16 | 2017-08-04 | 郑州云海信息技术有限公司 | A kind of data processing method and device |
US10733024B2 (en) | 2017-05-24 | 2020-08-04 | Qubole Inc. | Task packing scheduling process for long running applications |
US10990581B1 (en) | 2017-09-27 | 2021-04-27 | Amazon Technologies, Inc. | Tracking a size of a database change log |
US10754844B1 (en) | 2017-09-27 | 2020-08-25 | Amazon Technologies, Inc. | Efficient database snapshot generation |
US11182372B1 (en) | 2017-11-08 | 2021-11-23 | Amazon Technologies, Inc. | Tracking database partition change log dependencies |
US11042503B1 (en) | 2017-11-22 | 2021-06-22 | Amazon Technologies, Inc. | Continuous data protection and restoration |
US11860741B2 (en) | 2017-11-22 | 2024-01-02 | Amazon Technologies, Inc. | Continuous data protection |
US11269731B1 (en) | 2017-11-22 | 2022-03-08 | Amazon Technologies, Inc. | Continuous data protection |
CN107895046A (en) * | 2017-11-30 | 2018-04-10 | 广东奥飞数据科技股份有限公司 | A kind of Heterogeneous Database Integration Platform |
CN107895046B (en) * | 2017-11-30 | 2021-02-23 | 广东奥飞数据科技股份有限公司 | Heterogeneous data integration platform |
CN109981698A (en) * | 2017-12-27 | 2019-07-05 | 博元森禾信息科技(北京)有限公司 | Number networking cross-domain data access standardized system and method based on metadata |
US11228489B2 (en) | 2018-01-23 | 2022-01-18 | Qubole, Inc. | System and methods for auto-tuning big data workloads on cloud platforms |
US10621049B1 (en) | 2018-03-12 | 2020-04-14 | Amazon Technologies, Inc. | Consistent backups based on local node clock |
US11070600B1 (en) | 2018-07-16 | 2021-07-20 | Amazon Technologies, Inc. | Optimization techniques to support lagging readers at streaming data service |
US11675501B2 (en) | 2018-07-16 | 2023-06-13 | Amazon Technologies, Inc. | Streaming data service with isolated read channels |
US10768830B1 (en) | 2018-07-16 | 2020-09-08 | Amazon Technologies, Inc. | Streaming data service with isolated read channels |
US10855754B1 (en) | 2018-07-16 | 2020-12-01 | Amazon Technologies, Inc. | Isolated read channel categories at streaming data service |
US11075984B1 (en) | 2018-07-16 | 2021-07-27 | Amazon Technologies, Inc. | Workload management at streaming data service supporting persistent connections for reads |
US11509700B2 (en) | 2018-07-16 | 2022-11-22 | Amazon Technologies, Inc. | Stream data record reads using push-mode persistent connections |
US10798140B1 (en) | 2018-07-16 | 2020-10-06 | Amazon Technologies, Inc. | Stream data record reads using push-mode persistent connections |
US11621999B2 (en) | 2018-07-16 | 2023-04-04 | Amazon Technologies, Inc. | Isolated read channel categories at streaming data service |
US10956246B1 (en) | 2018-07-16 | 2021-03-23 | Amazon Technologies, Inc. | Isolated read channel management interfaces at streaming data service |
US12013764B2 (en) | 2018-08-10 | 2024-06-18 | Amazon Technologies, Inc. | Past-state backup generator and interface for database systems |
US11126505B1 (en) | 2018-08-10 | 2021-09-21 | Amazon Technologies, Inc. | Past-state backup generator and interface for database systems |
US11579981B2 (en) | 2018-08-10 | 2023-02-14 | Amazon Technologies, Inc. | Past-state backup generator and interface for database systems |
US11120034B2 (en) * | 2018-11-13 | 2021-09-14 | Sap Se | Automatic routing of connection requests in business intelligence (BI) tools |
US11042454B1 (en) | 2018-11-20 | 2021-06-22 | Amazon Technologies, Inc. | Restoration of a data source |
CN111490999A (en) * | 2019-01-28 | 2020-08-04 | 上海菲碧文化传媒有限公司 | Big data processing analysis platform based on Hadoop |
US11704316B2 (en) | 2019-05-31 | 2023-07-18 | Qubole, Inc. | Systems and methods for determining peak memory requirements in SQL processing engines with concurrent subtasks |
US11144360B2 (en) | 2019-05-31 | 2021-10-12 | Qubole, Inc. | System and method for scheduling and running interactive database queries with service level agreements in a multi-tenant processing system |
US11747978B2 (en) | 2019-07-23 | 2023-09-05 | International Business Machines Corporation | Data compaction in distributed storage system |
CN113515610A (en) * | 2021-06-21 | 2021-10-19 | 中盾创新档案管理(北京)有限公司 | File management method based on object-oriented language processing |
Also Published As
Publication number | Publication date |
---|---|
WO2013070873A1 (en) | 2013-05-16 |
US20160246824A1 (en) | 2016-08-25 |
JP2015501976A (en) | 2015-01-19 |
JP5818394B2 (en) | 2015-11-18 |
US9582528B2 (en) | 2017-02-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9582528B2 (en) | System and method for operating a big-data platform | |
CN107038162B (en) | Real-time data query method and system based on database log | |
CN109471863B (en) | Information query method and device based on distributed database and electronic equipment | |
CN109074377B (en) | Managed function execution for real-time processing of data streams | |
CN108536778B (en) | Data application sharing platform and method | |
US20210248143A1 (en) | Automatically executing graphql queries on databases | |
US10122783B2 (en) | Dynamic data-ingestion pipeline | |
EP3654204B1 (en) | Low-latency database analysis using external data sources | |
US20140222843A1 (en) | Systems, Methods, and computer Program Products to Ingest, Process, and Output Large Data | |
US20150088807A1 (en) | System and method for granular scalability in analytical data processing | |
US10877810B2 (en) | Object storage system with metadata operation priority processing | |
WO2012125166A1 (en) | Data source analytics | |
CN107622055B (en) | Method for rapidly realizing data service release | |
US20190361999A1 (en) | Data analysis over the combination of relational and big data | |
CN112988741B (en) | Real-time service data merging method and device and electronic equipment | |
CN113282611B (en) | Method, device, computer equipment and storage medium for synchronizing stream data | |
CN113312428A (en) | Multi-source heterogeneous training data fusion method, device and equipment | |
CN110457307B (en) | Metadata management system, user cluster creation method, device, equipment and medium | |
US11567957B2 (en) | Incremental addition of data to partitions in database tables | |
Kuderu et al. | Relational database to NoSQL conversion by schema migration and mapping | |
CN113190517B (en) | Data integration method and device, electronic equipment and computer readable medium | |
US9256641B1 (en) | Dynamic optimization of data aggregation | |
WO2024001029A1 (en) | Method and apparatus for maintaining blockchain data, electronic device, and storage medium | |
KR20100132752A (en) | Distributed data processing system | |
CN117390106B (en) | Data processing method, device, storage medium and computing equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: TREASURE DATA, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:FURUHASHI, SADAYUKI;YOSHIKAWA, HIRONOBU;OTA, KAZUKI;SIGNING DATES FROM 20121115 TO 20121116;REEL/FRAME:029319/0645 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |