DE112014005183T5 - Store service network - Google Patents

Abstract

A storage service network, comprising: an application programming interface (API) mechanism that receives requests from data sources to store data, the requests each including an indication of the type of data to be stored; at least one data processing engine configured to process the data type, the processing by the at least one data processing engine converting the data into processed data having a format suitable for consumer use; a plurality of databases storing the processed data and providing the processed data to consumers; and a Pull API mechanism called by the consumers to retrieve the processed data.

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of US Provisional Application No. 61 / 903,650, filed on Nov. 13, 2013, entitled "STORAGE UTILITY NETWORK", which is hereby incorporated by reference in its entirety.

GENERAL PRIOR ART

Recording and storing large amounts of data is extremely inefficient. For example, many data centers are often used to provide access to large amounts of data. However, this results in high operating costs and lacks a centralized, scalable architecture. In addition, there are often duplications and inconsistencies between the data in the multiple data centers. These data centers often lack visibility in terms of data access, making it difficult for clients to retrieve the data, which makes each data center work as an island without being fully aware of the other data centers. When common data centers handle large amounts of data, there are latencies that can affect the availability of the data, so they may no longer be relevant.

SUMMARY

Disclosed are systems and methods for providing a scalable storage network. In some aspects, there is provided a storage service network including: an application programming interface (API) mechanism that receives requests from data sources to store data, the requests each including an indication of the type of data to be stored; at least one data processing engine configured to process the data type, the processing by the at least one data processing engine converting the data into processed data having a format suitable for consumer use; a plurality of databases storing the processed data and providing the processed data to consumers; and a Pull API mechanism called by the consumers to retrieve the processed data.

In other aspects, a method of storing and providing data is provided. The method includes: receiving a request to an application programming interface (API) mechanism from data sources for storing data, the requests each including an indication of a data type to be stored; Processing the data on a data processing engine configured to process the data type to convert the data into processed data having a format suitable for consumer use; Storing the processed data at one of a plurality of databases that further provide the processed data to consumers; and receiving a call from a consumer at a pull API mechanism to retrieve the processed data.

Other systems, methods, features, and / or advantages will become apparent to those skilled in the art upon consideration of the following drawings and detailed description. It is intended that all such additional systems, methods, features and advantages be included within this description, be protected by the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The components in the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding parts throughout the several views.

1 FIG. 10 illustrates an exemplary memory service network (SDN) architecture according to the present disclosure; FIG.

2 illustrates an exemplary data acquisition architecture;

 Fig. 10 illustrates an exemplary data processing engine (DVE);

4 FIG. 10 illustrates an example operation of the processes performed to capture input data from the SDN 1 to be received;

5 illustrates exemplary client access to the storage service network using a geographic location based API; and

6 FIG. 3 illustrates an example computing device. FIG.

DETAILED DESCRIPTION

Unless defined otherwise, all technical and scientific terms have the same meaning as commonly accepted by one of ordinary skill in the art. In practice or for testing the present Disclosure may be made of methods and materials similar or equivalent to those described herein.

The present disclosure relates to a storage service network (SDN) that provides a centralized source of data entry, storage, and distribution.

The SDN provides non-blocking data collection, pull and push data service, datacenter data center data center data center replication, use of real-time data storage-based caching (cache), low latency, easy scalability, high availability and easy maintenance records. The SDN may be geographically distributed such that each site geographically stores relevant data for faster processing. The SDN is scalable to billions of data requests per day, delivering the data at a low latency of, for example, 10 ms-100 ms. As will be described, the SDN 100 Measure and authenticate low latency API calls by processing multiple TB of data daily, storing petabytes of data, and having a flexible data capture platform to manage hundreds of data feeds from external parties.

With the above summary as an introduction reference is now made to 1 illustrating an exemplary implementation of the storage service network (SDN) 100 of the present disclosure. The SDN 100 includes a capture API mechanism 102 that input data from different sources 101 receives an API management component 104 ; a caching layer 106 ; Data storage elements 108a - 108d ; virtual machines 110 ; a process, framework and organizational layer 112 ; and a pull API mechanism 114 , the output data to different data consumers 116 provides. The data consumers 116 may be broadcasters, cable systems, web-based information providers (e.g., news and weather websites) and other information or data disseminators.

The recording API 102 is from the SDN 100 disclosed to meet requirements z. For example, to receive a published Uniform Resource Identifier (URI) to receive data of a particular type in the SDN 100 save. Further details of the recording API 102 be referring to 2 described. The API Management Component 104 is used to authenticate, measure and throttle application programming interface (API) requests for data stored in the SDN 100 are saved or retrieved from it. Non-limiting examples of the API Management Component 104 are Mashery and Layer 7. The API Management Component 104 also provides customer engagement, enforcement of access policies, and provision of services. The API Management Component 104 can make the APIs accessible to different classes of end users by applying security and usage policies to data and services. The API Management Component 104 can also provide analytics to determine the use of services to support business or technical goals. Details of the API Management Component 104 are disclosed in US Patent Application No. 61 / 954,688, filed Mar. 18, 2014, entitled "LOW LATENCY, HIGH PAYLOAD, HIGH VOLUME API GATEWAY," which is incorporated by reference herein in its entirety

The caching layer 106 is a location in the memory of the SDN 100 Received data and server data that are sent to the data consumer 116 (ie clients) of the SDN 100 to be sent. To the data storage elements 108 may include, but are not limited to, a relational database management system (RDBVS). 108a , a big data file system 108b (eg Hadoop Distributed File System (HDFS) or similar) and a NoSQL database (eg a NoSQL Document Store database 108c or a NoSQL Key Value database 108d ) belong. As will be described below, data is provided by the recording API 102 are received, depending on which data type in the request to the recording API 102 in a non-blocking manner in one of the data storage elements 108 saved.

According to the present disclosure, the elements in the SDN 100 on the virtual machines 110 hosted. For example, the data processing engines 210 ( 2 ) are created and destroyed by starting and stopping the virtual machines to receive incoming data from the caching layer 106 retrieve the data and process the data for storage. As one of ordinary skill in the art will understand, the virtual machines are 110 Software computers running an operating system and applications such as a physical computing device. Each virtual machine is supported by the physical resources of a host computing device and has the same functions as physical hardware, but with the advantages of portability, manageability, and security. For example, virtual machines can be created and destroyed to meet the resource requirements of the SDN 100 without having to add physical hardware to meet this need. An example of the host computing device will be described with reference to FIG 6 described.

The process, framework and organizational layer 112 provides data quality, data governance, customer engagement and an interface to other systems. Data Service Governance includes business decisions for recommendations about which data products and services are on the SDN 100 should build up when and in what order data products and services should be built, and the distribution channels for those products and services. Data quality ensures that the SDN 100 processed data is always valid and consistent.

The pull API mechanism 114 is used by the consumer to obtain data from the SDN 100 used. Similar to the recording API 102 becomes the pull API mechanism 114 from the SDN 100 disclosed to meet requirements z. For example, to receive a published Uniform Resource Identifier (URI) in the SDN 100 retrieve stored data associated with a particular product or type.

The SDN 100 can be implemented in a public cloud infrastructure such as Amazon Web Services, Microsoft Azure, the Google Cloud Platform or others to provide high-availability services to the users of the SDN 100 provide.

With reference to 2 - 4 Now the operation of the SDN 100 described in more detail. In particular, presents 2 an exemplary data acquisition architecture 200 within the SDN 100 represents. 3 represents an exemplary data processing engine (DVE) 210a - 210n represents. 4 FIG. 10 illustrates an example operation of the processes performed to capture input data provided by the SDN 100 be received.

As noted above, the data acquisition architecture 200 a non-blocking architecture for processing the data received from the SDN 100 be received. The data acquisition architecture 200 has load balancing devices 202a - 202n on, the workload on the computational resources in the architecture 200 to distribute. For example, if an input data source is the recording API 102 calls, from the SDN 100 is received (at 402 ), determine the load balancing devices 202a - 202n which resources that are associated with the called API should be used to estimate the response time associated with the components in the data acquisition architecture 200 to minimize. Calling the recording API 102 Contains information about the data type, that is from the input data source to the data acquisition architecture 200 should be transmitted. This information may come from the load balancing devices 202a - 202n can be used to determine which of the representational state transfer (REST) APIs 204a - 204n programmatic access to write the input data into the data acquisition architecture 200 will provide (at 404 ).

The REST APIs 204a - 204n provide an interface to an associated direct broker 206a - 206n ready to put data in an appropriate message queue 208a - 208c to submit (at 406 ) to enable it from a data processing engine (DVE) farm 210 be processed (at 408 ). In accordance with aspects of the present disclosure, each DVE 210a - 201n configured to process a particular type of input data. For example, the input data may be observation data provided by the REST API 204a or 204b be received. With this information, the observation data can be queued 208a the DVE 210a which is responsible for the processing of the observation data. That's how the SDN tries 100 To forward data in a way that each DVE always processes data of the same type. However, if a DVE 210a - 201n Receives data of an unknown type, directs the DVE 210a - 201n According to some aspects of the present disclosure, the data is queued to another DVE 210a - 201n who can process the data.

3 represents an exemplary data processing engine (DVE) 210a - 201n The DVE is a universal computational resource containing the input data 101 receives and to a suitable data storage element 108 writes. The DVE can z. B. JAVA and on one of the virtual machines 110 be executed. During instantiation, the DVE notifies its associated message queue (for example, message queue 208a for DVE 210a ), which is active.

A data pump 302 in the DVE reads a message from a queue and passes the message to a handler 304 , As shown, the handler 304 be multi-stranded and multiple handlers 304a - 304n include. The dealer 304 sends the data to a data cartridge for processing 306 , The data cartridge 306 "Programs" the functions of the DVE according to a configuration file 308 , For example, a separate data cartridge 306 for each data type that the SDN 100 receives. The data cartridge 306 formatted the message z. A JavaScript Object Notation (JSON) document, determines keys and values for each message, performs data pre-processing, transforms data based on operational logic, and provides data quality. The conversion of the data puts them in a state in which they are intended for consumption by one or more of the data consumers 116 to be ready.

With reference to 2 and 3 gives the data cartridge 306 after processing the message, the processed message back to the handler 304 which then sends the processed message to a DB interface 310 and / or a message queue switch (e.g. 212b ) (at 410 ). For example, the DB interface 310 the message from the handler 304a received and, according to key-values (or other information) defined in the message, into a database (ie one of the data storage elements 108 ) write. Additionally or alternatively, a selection of the database type may be made based on the data type stored therein. Although not in 3 shown is the DB interface 310 for a specific database type (eg Redis) specific, which is why several DB interfaces 310 may be present. Thus, the DB interface provides 310 ensure that the data is written to a database (such as Redis) for optimal storage and retrieval.

In another example, the handler 304a the data to the message queue switch 212a / 212b transmit the data to a suitable output queue 2141- 214n / 216a - 216n for consumption by data consumers 116 stores. Thus, the data acquisition architecture 200 input data 101 for data consumers 116 with extremely low latency, as the data from the DVE farm 210 recorded and processed and can be output in substantially real time.

As an example of data processing by the SDN 100 can the input data 101 rasterized data such as observation data. This data is usually used in weather forecasting to create geographically specific weather forecasts that are sent to the data consumer 116 to be provided. These data are extensive and time-dependent, especially in changeable weather conditions. The SDN 100 provides a platform through which to retrieve this data from the data acquisition architecture 200 can be processed quickly, such that the data output to the data consumer 116 is done promptly.

5 illustrates an exemplary client access to the storage service network using a geographic location based API. In accordance with the present disclosure, a client application may 500 through a published Uniform Resource Identifier (URI), the inclusion API 102 by meeting pre-agreed location parameters 502 was assigned to the SDN 100 access. A geopositioning service 504 can provide a geohashing algorithm. Geohashing algorithms use short URLs to uniquely identify sites on Earth to make referencing these sites more convenient. To obtain a geohash, a user provides an address or latitude and longitude coordinates for geocoding in a single input field (with the most common latitude and longitude pairs accepted) and makes the request.

6 FIG. 10 illustrates an example computing environment in which example embodiments and aspects may be implemented. FIG. The computing system environment is just one example of a suitable computing environment and is not intended to imply any limitations on the scope of its use or operation.

Many other general or specialized computing system environments or configurations may be used. Examples of known computing systems, environments, and / or configurations that may be suitable for use include, but are not limited to, personal computers, servers, handheld or laptop devices, multiprocessor systems, microprocessor based systems, network PCs, minicomputers , Mainframe computers, embedded systems, distributed computing environments including any of the foregoing systems or devices, and the like.

Computer-executable instructions, such as program modules, that are executed by a computer may be used. Generally, program modules include routines, programs, objects, components, and data structures, etc. that perform certain tasks or implement particular abstract data types. Distributed computing environments may be used in which tasks are performed by remote processing devices that are connected via a communication network or other data transmission medium. In a distributed computing environment, program modules and other data may be located on both local and remote computer storage media, including memory storage devices.

With reference to 6 For example, one example system for implementing aspects described herein includes a computing device, such as a computing device 600 , in its simplest configuration includes the computing device 600 usually at least one processing unit 602 and a memory 604 , Depending on the exact configuration and type of computing device, the memory may 604 be volatile (such as random access memory (RAM)), non-volatile (such as read-only memory (ROM), flash memory, etc.) or a combination of the two. This basic configuration is in 6 through the dashed line 606 shown.

The computing device 600 may have further features / functions. For example, the computing device 600 additional storage (removable storage / non-removable storage), including, but not limited to, magnetic or optical disks or tapes. This extra memory is in 6 through the removable storage 608 and the non-removable memory 610 shown.

The computing device 600 typically includes several tangible computer-readable media. Computer-readable media can be any tangible media upon which the device 600 and include both volatile and non-volatile, removable and non-removable media.

Tangible computer storage media include both volatile and nonvolatile, removable and non-removable media implemented with any method or technique for storing information such as computer readable instructions, data structures, program modules, or other data. The memory 604 , Removable storage 608 and non-removable memory 610 are all examples of computer storage media. Access to tangible computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory and other memory technology, CD-ROMs, Digital Versatile Disks (DVD) or other optical storage media, magnetic cassettes, magnetic tapes, magnetic disks Storage media or other magnetic storage devices or any other media that may be used to store the desired information and to which the computing device 600 can access. Any of these computer storage media may be part of the computing device 600 be.

The computing device 600 can communication connection (s) 612 so that the device can communicate with other devices. The computing device 600 can also input device (s) 614 such as keyboard, mouse, pen, voice input device, touch input device, etc. It could also output device (s) 616 such as display, speakers, printers, etc. may be present. All of these devices are known in the art and need not be discussed in detail here.

It should be understood that the various techniques described herein may be implemented in conjunction with hardware or software or, where appropriate, a combination of both. The methods and apparatus of the subject matter disclosed herein, or certain aspects or portions thereof, may thus take the form of program code (ie instructions) embodied on tangible media such as floppy disks, CD-ROMs, hard disks, or any other machine-readable storage media Loading the program code into a machine such as a computer and, when executed, the machine becomes an apparatus for practicing the subject matter disclosed herein. In the case of program code execution on programmable computers, the computing device generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory elements), at least one input device, and at least one output device. One or more programs may implement or utilize the processes described in connection with the subject matter disclosed herein, e.g. Through the use of an application programming interface (API), reusable controls, or the like. These programs may be implemented in a higher level procedural or object oriented programming language to communicate with a computer system. However, the program (s) may be implemented in assembly or machine language as needed. In any case, the language may be a compiled or interpreted language and may be combined with hardware implementations.

Although the subject matter has been described in terms specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features or acts described above are disclosed as example forms for implementing the claims.

Claims (20)

A memory device, comprising: an application programming interface (API) mechanism that receives requests from data sources for storing data, the requests each including an indication of a data type to be stored; at least one data processing engine configured to process the data type, the processing by the at least one data processing engine converting the data into processed data having a format suitable for consumer use; a plurality of databases storing the processed data and providing the processed data to consumers; and a pull API mechanism called by the consumers to retrieve the processed data. The apparatus of claim 1, further comprising an API management component that authenticates, measures and throttles the requests and the calls to the host API mechanism and the pull API mechanism. The apparatus of claim 2, wherein the capture API mechanism and the pull API mechanism are exposed by the storage device to receive requests for respective Uniform Resource Identifiers (URI). The apparatus of claim 1, wherein data received by the host API is processed in a non-blocking manner and stored in one of the databases, depending on which type of data is specified in the request to the host API mechanism. The apparatus of claim 1, wherein the capture API mechanism further comprises load balancers that determine resources to use in the storage device to minimize the response time for storing the processed data in the databases. The apparatus of claim 1, wherein the capture API mechanism deposits the data in a predetermined message queue for processing by a respective data processing engine associated with the data type, depending on which data type is specified in the request. The apparatus of claim 1, wherein the at least one data processing engine further comprises: a data pump reading a message from a queue; a handler receiving messages from the queue; a data cartridge that configures the data processing engine to process the data from the handler to convert the data into the processed data; a database interface that writes the processed data to a given database of the plurality of databases; and an exchange mechanism that provides processed data directly to consumers, wherein the predetermined database is selected based on the data type to be stored therein. The apparatus of claim 7, wherein when the respective data processing engine receives data of an unknown type, the respective data processing engine places the data in a queue from another of the at least one data processing engine capable of processing the data. The apparatus of claim 1, wherein the data is rasterized data provided by the data sources. The apparatus of claim 9, wherein the data type is one of pollen data, satellite data, predictive models, wind data, flash data, air quality data, user data, temperature data, or weather station data. A method for storing and providing data, comprising: Receiving a request at a recording programming interface (API) mechanism from data sources for storing data, the requests each including an indication of a data type to be stored; Processing the data on a data processing engine configured to process the data type to convert the data into processed data having a format suitable for consumer use; Storing the processed data at one of a plurality of databases that further provide the processed data to consumers; and Receive a call from a consumer to a Pull API mechanism to retrieve the processed data. The method of claim 11, further comprising authenticating, measuring and throttling the requests and calls to the ingest API mechanism and the pull api mechanism using an API management component. The method of claim 12, further comprising exposing the ingest API mechanism and the pull API mechanism to respective Uniform Resource Identifiers (URI). The method of claim 11, further comprising storing data in a non-blocking manner in one of the plurality of databases, depending on which data type is specified in the request to the capture API mechanism. The method of claim 11, further comprising providing load balancing devices that determine resources to use in the storage device to minimize the response time for storing the processed data in the one of the plurality of databases. The method of claim 11, further comprising storing the data dependent on which data type is specified in the request for processing by a respective data processing engine associated with the data type in a given message queue. The method of claim 11, further comprising providing the data processing engine with a data pump reading messages from a queue, a handler receiving messages from the queue, a data cartridge configuring the data processing engine to retrieve the data from the handler to convert the data into the processed data, a database interface that writes the processed data to a given database of the plurality of databases, and an exchange mechanism that provides processed data directly to the consumers. The method of claim 17, further comprising: Determining whether the respective data processing engine is receiving data of an unknown type; and Storing the data in a queue of another of the at least one data processing engine that can process the data. The method of claim 11, wherein the data is rasterized data provided by the data sources. The method of claim 19, wherein the data type is one of pollen data, satellite data, predictive models, wind data, flash data, air quality data, user data, temperature data, or weather station data.

Images