KR102508177B1 - Credentialless external stage for database integration - Google Patents

Abstract

A storage integration object is created within the database of the data warehouse system. The storage integration object identifies the storage location within the storage platform of the cloud storage provider system and the cloud identity object maintained by the network-based data warehouse system. A cloud identity object is associated with a proxy identity object that has been granted permission to access a storage location in the cloud storage provider's storage platform. An external stage object is created based on the storage integration object. The external stage object identifies the storage location and includes an association with the storage integration object. A command is received to load or unload data from a storage location. In response to the command, data is loaded or unloaded from the storage location, via the proxy identity object, using the external stage object.

Description

Credentialless external stage for database integration

Cross reference to priority application

This application claims priority to the application filed as U.S. Patent Application Serial No. 16/683,641 on November 14, 2019 and issued as U.S. Patent No. 10,715,524 on June 14, 2020, the contents of which are hereby incorporated in their entirety. included by reference.

technology field

Embodiments of the disclosure relate generally to network-based data warehouses, and more specifically to credential-less external stages for data warehouse storage integration.

A cloud data warehouse (also referred to as a "network-based data warehouse" or simply "data warehouse") is a network-based system used for data analysis and reporting that contains a central repository of consolidated data from one or more disparate sources. am. Cloud data warehouses can store current and historical data that can be used to generate analytics reports for businesses. To this end, data warehouses typically provide business intelligence tools, tools to extract, transform, and load data into storage, and tools to manage and retrieve metadata.

An external stage is a component within a cloud data warehouse that facilitates integration between a cloud data warehouse system and a customer-managed storage location (referred to herein as “storage integration”). Typically, external stages are used to load data into and unload data from customer-managed storage locations. In existing implementations, external stages must be provided with secret security credentials to read and write data to these storage locations. However, exchanging secret security credentials creates vulnerabilities that can lead to unauthorized access to data by exposing secret security credentials. Additionally, in existing implementations, cloud data warehouse account managers have limited ability to prevent organizational members from creating external stages, which could potentially be used to exfiltrate confidential data to private locations. Also, the storage owner cannot fine-grain control over permission to access the storage location. Existing external stages are also restricted to use on a single file path and cannot be used on different file paths, even if the credentials used to create the external stage can be applied to other file paths.

The present disclosure will be more fully understood from the detailed description given below and accompanying drawings of various embodiments of the disclosure.
1 illustrates an example computing environment that includes a network-based data warehouse system in communication with a cloud storage provider system, in accordance with some embodiments of the present disclosure.
2 is a data flow diagram illustrating the use of a credential-free external stage object within a computing environment to load or unload data from a storage location within a cloud storage provider system to a network-based data warehouse system, in accordance with some embodiments of the present disclosure; am.
3 illustrates a diagram between components of a computing environment loading or unloading data from a storage location within a cloud storage provider system to a network-based data warehouse system without exchanging security credentials associated with the storage location, according to some embodiments of the present disclosure. It is an interaction diagram showing the interaction of
4 and 5 are flow diagrams illustrating operation of a network-based data warehouse performing a method of loading or unloading data from an external storage platform using an external stage object without credentials, according to some embodiments of the present disclosure. .
6 is a block diagram illustrating components of a computing service manager, in accordance with some embodiments of the present disclosure.
7 is a block diagram illustrating components of an execution platform, in accordance with some embodiments of the present disclosure.
8 illustrates a diagrammatic representation of a machine in the form of a computer system from which sets of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein, in accordance with some embodiments of the present disclosure.

Reference will now be made in detail to specific exemplary embodiments for carrying out the subject matter of the invention. Examples of these specific embodiments are illustrated in the accompanying drawings, and specific details are set forth in the following description in order to provide a thorough understanding of the claimed subject matter. It will be understood that these examples are not intended to limit the scope of the claims to the illustrated embodiments. On the contrary, they are intended to cover such alternatives, modifications and equivalents as may be included within the scope of the disclosure.

As pointed out above, external stages are used in network-based data warehouses to load and unload data from customer-managed storage locations, and traditional external stages must be provided with secret security credentials to access these storage locations. , which can create security vulnerabilities for data. Aspects of the present disclosure are credential-less systems that do not require users to share secret, secure credentials with a network-based data warehouse to facilitate loading and unloading of data from a storage location in an external cloud storage provider system. ) solves the above and other deficiencies of previous data warehouse functions by creating an external stage object. The credentialless external stage object described herein also allows client account administrators to fine-tune access permissions to prevent data exfiltration.

According to some embodiments, the network-based data warehouse is an external cloud storage provider system (e.g., Amazon Web Services® (AWS)) that provides access to load and unload data to the network-based data warehouse. ), the storage platform of Microsoft Azure Blob Storage® or Google Cloud Storage), the integration object containing the identifier of the storage location (e.g., the Uniform Resource Identifier (URL)). generate The integration object further includes an identifier of a proxy identity object maintained by the external cloud storage provider system. Once created, the network-based data warehouse associates the integration object with a cloud identity object and the cloud storage provider system associates it with a proxy identity object. A proxy identity object defines a proxy identity that can be assumed to be granted access to the storage location and to allow the cloud identity object to load and unload data from the storage location.

The data warehouse creates integration objects based on commands that create storage integrations. The instructions may be provided by, for example, an administrative user of the client account data warehouse. The cloud identity object associated with the integration object corresponds to the client account to which the user belongs. The storage integration definition includes the identifier of the storage location, the identifier of the proxy identity object and the identifier of the cloud storage provider system. The storage integration definition may further specify one or more storage locations to which access is allowed or denied, as the case may be. A storage definition object can specify specific segments within a storage location to be denied access. For example, a storage location may be identified as a file path corresponding to a storage resource within a storage platform, such as a bucket or folder, and the command may specify subfolders within the file path to which access is denied. In another example, a command can specify one or more file paths to which access is allowed, in which case access to all other file paths is denied by default.

Data warehouses create external stage objects based on storage integration objects to load or unload data from storage locations. The external stage object includes the identifier of the storage location and the identifier of the storage integration object. The data warehouse creates an external stage object based on, for example, a command to create an external stage object provided by a user who provided a storage integration definition.

A network-based data warehouse may receive commands to load or unload data from a storage location. The command contains the identifier of the external stage object. In response to commands, the data warehouse utilizes external stage objects to load or unload data from storage locations on the external cloud storage provider's storage platform. During this process, the network-based data warehouse accesses the credentials using the security credentials associated with the cloud identity object, allowing the cloud identity object to load or unload data assuming a proxy identity. In this way, an external stage object may exchange security credentials associated with a storage location with a network-based data warehouse system or load or unload data from a storage location without storing security credentials associated with the storage location.

As described herein, an external stage object without credentials separates the process of granting permissions to a storage location from the use of the storage location to load and unload data. Credentialless external stage objects also allow organizations to grant network-based data warehouses permission to use data locations instead of providing security credentials to the data warehouse. Organizations can specify roles that can create and use storage locations for access, separate from who can create and use predefined stages. For example, an organization can allow account administrators to create connections to storage locations, and since only account administrators can create storage integrations, they cannot create additional storage integrations to export data, so confidential data is unknown. It can prevent leakage to the missing location. Once created, non-administrator users can be granted read and write permissions from a fixed storage location to external stage objects they create. Users with low privileges can only use existing stages.

Users with permission to create storage aggregations can control the paths under the default location that can be accessed using that aggregation. Giving account administrators the ability to create storage integrations and designate who can use them allows organizations to control where internal data flows or completely contain data exfiltration.

Credentialless external stage objects also allow access permissions to storage to be managed by cloud storage providers, allowing organizations using data warehouses to leverage storage providers to manage data access by network-based data warehouses. provides the advantage of If the account administrator decides to revoke the data warehouse's access to the storage location, it can do so immediately using the access controls provided by the storage provider.

1 illustrates an example computing environment 100 that includes a network-based data warehouse system 102 in communication with a cloud storage provider system 104, according to some embodiments of the present disclosure. In order to avoid obscuring the inventive subject matter with unnecessary detail, various functional components not relevant to conveying an understanding of the inventive subject matter have been omitted from FIG. 1 . However, those skilled in the art will readily appreciate that various additional functional components may be included as part of the computing environment 100 to facilitate additional functionality not specifically described herein.

As shown, computing environment 100 includes a network-based data warehouse system 102 and a cloud storage provider system 104 (eg, AWS®, Microsoft Azure Blob Storage®, or Google Cloud Storage). . The network-based data warehouse system 102 is a network-based system used for reporting and analysis of consolidated data from one or more disparate sources including one or more storage locations within the cloud storage provider system 104 . The cloud storage provider system 104 includes a plurality of computing machines and provides on-demand computer system resources such as data storage and computing power to the network-based data warehouse system 102 .

The network-based data warehouse system 102 includes an access management system 110 , a computing service manager 112 , an execution platform 114 , and a database 116 . The network-based data warehouse system 102 hosts and provides data reporting and analysis services to multiple client accounts. The access management system 110 allows administrative users of client accounts to manage access to resources and services provided by the network-based data warehouse system 102 . Administrative users create and manage identities (eg users, roles and groups) and grant or deny identities access to resources and services.

Computing service manager 112 coordinates and manages the operation of network-based data warehouse system 102 . Computing services manager 112 manages clusters of computing services (also referred to as “virtual warehouses”) that perform query optimization and compilation as well as provide computing resources. Computing service manager 112 may be used by end users to provide data storage and retrieval requests, system managers that manage the systems and methods described herein, and any other components/devices that interact with computing service manager 112. A number of client accounts can be supported.

Computing service manager 112 is also coupled to database 116 , which is associated with data stored in computing environment 100 . Database 116 stores data relating to various functions and aspects associated with network-based data warehouse system 102 and its users. For example, database 116 stores external stage objects 108 without one or more credentials. Typically, the external stage object 108 specifies a storage location (e.g., a URL) where the data files are stored so that the data in the file is loaded into a table stored internally by the data warehouse 102 or the data in the table is The data warehouse 102 allows it to be unloaded as an internally stored data file. An external stage object 108 without one or more credentials allows the network-based data warehouse system 102 to access a storage location within the cloud storage provider system 104 without storing, using, or otherwise accessing the security credentials associated with the storage location. give access to

In some embodiments, database 116 includes summaries of data stored in remote data storage systems as well as data available from local caches. Additionally, database 116 may include information about how data is organized in remote data storage systems (eg, cloud storage provider system 104 ) and local caches. Database 116 enables systems and services to determine whether a piece of data should be accessed without loading or accessing the actual data from the storage device.

Computing service manager 112 is further coupled to execution platform 114, which provides multiple computing resources to execute various data storage and data retrieval tasks. The execution platform 114 is coupled to the storage platform 122 of the cloud storage provider system 104 . Storage platform 122 includes a number of data storage devices 124-1 through 124-N. In some embodiments, data storage devices 124-1 through 124-N are cloud-based storage devices located in one or more geographic locations. For example, data storage devices 124-1 through 124-N may be part of a public cloud infrastructure or a private cloud infrastructure. Data storage devices 124-1 through 124-N may be hard disk drives (HDDs), solid state drives (SSDs), storage clusters, Amazon S3(TM) storage systems, or any other data storage technology. Additionally, the cloud storage provider system 104 may include a distributed file system (eg, Hadoop Distributed File Systems (HDFS)), an object storage system, and the like.

Execution platform 114 includes a plurality of computing nodes. A set of processes on the computing node executes the query plan compiled by the computing service manager 112 . The set of processes include: a first process that executes the query plan; a second process for monitoring and deleting micro-partition files using a least recently used (LRU) policy and implementing an out of memory (OOM) error mitigation process; a third process for extracting health information from the process log and status and sending it back to the computing service manager 112; a fourth process for establishing communication with the computing service manager 112 after system boot; and a fifth process that handles all communication with the computing cluster for a given task provided by the computing services manager 112 and passes information back to the computing services manager 112 and other computing nodes in the execution platform 114. include

In addition to the storage platform 122 , the cloud storage provider system 104 also includes an authentication and identity management system 118 . The authentication and identity management system 118 allows users to create and manage identities (eg, users, roles, and groups) and use permissions to allow or deny identities access to cloud services and resources. The access management system 110 of the network-based data warehouse system 102 and the authentication and identity management system 118 of the cloud storage provider system 104 communicate and share information so that the network-based data warehouse system 102 and It may enable users on both sides of the cloud storage provider system 104 to access and manage shared resources and services.

In some embodiments, communication links between elements of computing environment 100 are implemented through one or more data communication networks. Such data communication networks may use any communication protocol and any type of communication medium. In some embodiments, a data communication network is a combination of two or more data communication networks (or subnetworks) coupled together. In alternative embodiments, these communication links are implemented using any type of communication medium and any communication protocol.

As shown in FIG. 1 , data storage devices 124 - 1 through 124 -N are separate from the computing resources associated with execution platform 114 . This architecture supports dynamic changes to the network-based data warehouse system 102 based on changing data storage/retrieval needs as well as changing needs of users and systems. Support for dynamic change allows the network-based data warehouse system 102 to scale rapidly in response to changing demands on the systems and components within the network-based data warehouse system 102 . Separating the computing resources from the data storage device supports storage of large amounts of data without requiring a correspondingly large amount of computing resources. Similarly, this separation of computing resources supports a significant increase in the computing resources utilized at any given time without requiring a corresponding increase in available data storage resources.

Computing services manager 112, database 116, execution platform 114, storage platform 122, and authentication and identity management system 118 are shown as separate components in FIG. However, computing service manager 112, database 116, execution platform 114, storage platform 122, and authentication and identity management system 118 each are distributed systems (e.g., multiple distributed across systems/platforms). In addition, the computing service manager 112, database 116, execution platform 114, storage platform 122, authentication and identity management system 118, respectively, change and network-based data warehouse system 102 for received requests. ) can be expanded or contracted (independently of each other) according to the changing needs of Thus, in the described embodiment, the network-based data warehouse system 102 is dynamic and supports regular changes to meet current data processing needs.

During normal operation, network-based data warehouse system 102 processes a number of tasks determined by computing services manager 112 . These jobs are scheduled and managed by computing services manager 112 to determine when and how to execute the jobs. For example, computing services manager 112 may divide the work into a number of discrete tasks, and may determine the data required to execute each of the number of discrete tasks. Computing services manager 112 may assign each of a plurality of individual tasks to one or more nodes of execution platform 114 to process the task. Computing service manager 112 may determine the data needed to process the job and may further determine which node within execution platform 114 is best suited to process the job. Some nodes may already have cached the data needed to process the task, so they are good candidates for processing the task. The metadata stored in database 116 assists computing service manager 112 in determining which nodes within execution platform 114 have already cached at least some of the data needed to process a task. One or more nodes of the execution platform 114 process tasks using data cached by the nodes and, if necessary, data retrieved from the cloud storage provider system 104 . It is desirable to retrieve as much data as possible from the cache within the execution platform 114 because the speed of retrieval is generally much faster than retrieving data from the cloud storage provider system 104 .

As shown in FIG. 1 , computing environment 100 separates execution platform 114 from storage platform 122 . In this arrangement, the processing and cache resources of the execution platform 114 operate independently of the data storage devices 124-1 through 124-N of the cloud storage. Thus, computing resources and cache resources are not limited to specific data storage devices 124-1 through 124-N. Instead, all computing resources and all cache resources can retrieve data from and store data to any data storage resource in the cloud storage provider system 104 .

2 is a data flow diagram illustrating the use of an external stage object 200 within a computing environment 100, in accordance with some embodiments of the present disclosure. External stage object 200 is an example of the credential-free external stage object(s) 108 illustrated in FIG. 1 . External stage object 200 is created by computing service manager 112 and stored in database 116 . The external stage object 200 is created by the computing service manager 112 within the client account 204 . Computing service manager 112 communicates with network-based data warehouse system 102 to create external stage objects 200 based on input received from computing devices. For example, user 205 of client account 204 may utilize a command line or other user interface provided by network-based data warehouse system 102 to computing device 206 to create external stage object 200. You can provide commands to create them.

External stage object 200 is a component used to load or unload data from storage locations within storage platform 122 to network-based data warehouse system 102 . In this particular example, external stage object 200 designates a storage location corresponding to storage resource 208 within storage platform 122 as a location from which data may be loaded or unloaded. Storage resource 208 resides on one or more of storage devices 124 - 1 through 124 -N of storage platform 122 . The external stage object 200 further includes a reference (eg, pointer) to the storage integration object 202 .

The storage integration object 202 is created in the client account 204 by the computing service manager 112 and stored in the database 116 . Computing service manager 112 communicates with network-based data warehouse system 102 to create external stage object 200 based on input received from computing device 206 of user 205 of client account 204. do. For example, user 205 may utilize a command line or other user interface provided to computing device 206 by network-based data warehouse system 102 to provide commands to create storage integration object 202. can do.

It should be understood that the user providing the command for creating the external stage object 200 may be a different user than the user providing the command for creating the storage integration object 202 . For example, a first user—an administrative user—having administrator privileges can provide a command to create a storage integration object 202, and as part of the command, a storage integration object 202 to create an external stage object. ) may be granted to the second user. In this example, the second user may provide a command to create an external stage object 200 .

The storage integration object 202 defines storage integration between the network-based data warehouse system 102 and an externally managed storage location on the storage platform 122 . More specifically, the storage integration object 202 provides storage integration attributes between the network-based data warehouse system 102 and customer-managed storage resources 208 (e.g., folders, data buckets, or other storage resources). describe The storage integration object 202 includes an identifier (eg, URL) of a storage location corresponding to the storage resource 208 and an identifier of the cloud storage provider system 104 . In some embodiments, storage integration object 202 may further specify one or more storage locations to which access to data is denied. For example, the external stage object 200 may use the file path to identify the default storage location to which access is to be granted, and the storage integration object 202 may use a sub-path of the file path to identify the default storage location to which access is to be granted or denied. Some of the storage locations may be further identified.

Once created, the computing service manager 112 associates the storage integration object 202 with a service account maintained by the network-based data warehouse system 102 and the authentication and identity management system 118 associated with the client account 204 ( 212) with the cloud identity object 210. The cloud identity object 210 is an identity within the cloud storage provider system 104 associated with the client account 204 . The cloud identity object 210 may be created when the client account 204 is created. A unique identifier (eg, Amazon® Resource Name (ARN)) is associated with the cloud identity object 210 upon creation. A storage provider administrator may utilize the authentication and identity management system 118 to grant the cloud identity object 210 permission to access storage using the identifier of the cloud identity object 210 .

Computing service manager 112 may store the cloud storage provider identity identifier in database 116 in encrypted form. Computing service manager 112 may further store in encrypted form in database 116 the security credentials associated with each cloud storage provider identity.

The cloud storage provider system 104 creates a proxy identity object 214 within the client account 216 of the cloud storage provider system 104 . Client account 216 is an account of a client that corresponds to client account 204 in cloud storage provider system 104 . The cloud storage provider system 104 creates the proxy identity object 214 based on input specified by the administrative user of the client account 216 . In some examples, the administrative user of client account 216 is user 205 .

A proxy identity object 214 defines a proxy identity with an associated trust policy for making service requests within the cloud storage provider system 104 . More specifically, proxy identity object 214 includes a permission set that allows cloud identity object 210 to read data from and write data to storage resource 208 assuming a proxy identity. Proxy identity object 214 defines a proxy identity that can be assumed by multiple users rather than uniquely associated with one person, such as a user.

In some cases, proxy identities defined by proxy identity object 214 do not have long-term security credentials, in which case other identities that assume proxy identities are temporary security credentials provided by authentication and identity management system 118. Use the name to access the proxy identity. According to this embodiment, temporary security credentials may expire after an expiration time.

The cloud storage provider system 104 assigns a unique identifier (eg, Amazon Resource Name (ARN)) to the proxy identity object 214 . The unique identifier of the proxy identity object 214 is used by the storage manager to authorize access to storage.

Loading data from a storage location corresponding to storage resource 208 into an internally managed storage resource (eg, table) or unloading data from an internally managed storage resource to a storage location corresponding to storage resource 208 In response to receiving the command to do so, the network-based data warehouse system 102 uses the external stage object 200 to load or unload data. In particular, the compute service manager 112 uses the external stage object 200 to identify and access the storage integration object 202 and uses the storage integration object 202 to access security credentials associated with the cloud identity object 210. approach Computing service manager 112 uses the security credentials associated with cloud identity object 210 to access security credentials from authentication and identity management system 118 so that cloud identity object 210 is associated with proxy identity object 214. allow loading or unloading data between the internal storage resource and the storage resource 208 assuming a proxy identity defined by

3 illustrates an external stage object 200 for loading or unloading data from a storage resource 208 in a cloud storage provider system 104 to a network-based data warehouse system 102, in accordance with some embodiments of the present disclosure. An interaction diagram illustrating the interaction between a network-based data warehouse system 102 and a cloud storage provider system 104 in a method 300 of use. For ease of explanation, method 300 is described below with reference to components shown in FIGS. 1 and 2 and described above.

At operation 302 , cloud storage provider system 104 creates proxy identity object 214 in client account 216 . The cloud storage provider system 104 creates the proxy identity object 214 based on input specified by the administrative user of the client account 216 . As noted above, the proxy identity object 214 includes a set of permissions that allow the cloud identity object 210 to read data from and write data to the storage resource 208 assuming a proxy identity.

At operation 304 , the computing service manager 112 of the network-based data warehouse system 102 creates the storage integration object 202 in the database 116 . Computing service manager 112 creates storage integration object 202 based on a command provided by a first user (eg, user 205 ) of client account 204 . The first user may be an administrative user of the client account 204 . As mentioned above, the storage integration object 202 includes: an integration name; an identifier of an externally managed storage location, such as a URL corresponding to the storage resource 208; identifier of the cloud storage provider system 104; and an identifier of the proxy identity object 214. In some embodiments, the storage integration object 200 may further designate one or more storage locations to which access to data is denied. One or more storage locations to which access to data is denied may correspond to portions of storage locations to which access to the network-based data warehouse system 102 is provided. For example, the external stage object 200 may identify a basic storage location to which access is allowed using a file path, and the storage integration object 200 may use a sub-path of the file path to identify a basic storage location. You can identify the parts to be rejected.

At operation 306 , computing service manager 112 associates storage integration object 202 with cloud identity object 210 . The computing service manager 112 associates the storage integration object 202 with the cloud identity object 210 based on the association between the client account 204 and the cloud identity object 210 .

At operation 308 , the access management system 110 sets permissions for the storage integration object 202 . Setting permission to use the integration object may include granting permission to use the storage integration object to a second user. In some embodiments, access management system 110 may provide permissions for each user associated with an identity by granting permissions to identities corresponding to multiple users.

At operation 310 , computing service manager 112 creates external stage object 200 . The computing service manager 112 creates an external stage object 200 based on an external stage creation command for loading or unloading data from a storage location. The storage location may be the same as the storage location specified in the storage integration object 202 or may include a portion of the storage location specified in the storage integration object 202 . The external stage object 200 includes an identifier corresponding to the storage location (eg, a URL corresponding to the storage location) and a reference to the storage integration object 202 (eg, a pointer).

At operation 312 , the access management system 110 sets permissions for the external stage object 200 . The access management system 110 may set usage permission based on input received from the second user. Setting use permission for the integrated object may include granting use permission for a third identity. The third identity may correspond to a single user or may be associated with multiple users. When permission is granted to the third identity, one or more users associated with the third identity are permitted to load or unload data using the external stage object.

At operation 314, the cloud storage provider administrator sets permissions on the cloud storage provider system 104 for the proxy identity object. When setting permissions for a proxy identity object, the cloud storage provider system 104 grants the cloud identity object permission to load and unload data from a storage location using the proxy identity object.

At operation 316, computing services manager 112 receives an instruction to load data from a storage location into an internally managed storage resource (eg, a table) or unload data from an internally managed storage resource to a storage location. . The command includes the identifier of the external stage object 200 (eg, the integration name). The command may be received from a third user's computing device associated with the third identity. In response to the command, the computing services manager 112, at operation 318, uses the external stage object 200 to execute the command. In executing the instruction, the network-based data warehouse system 102 authenticates the proxy identity object 214 using the security credentials associated with the proxy identity object 214 to obtain a proxy defined by the proxy identity object 214. Assume identity. The network-based data warehouse system 102, at operation 322, assumes the proxy identity object 214 to load data from a storage location into an internally managed storage resource (e.g., a table) or load data from an internally managed storage resource (e.g., table). Unload data from a resource to a storage location.

4 and 5 illustrate a network-based data warehouse when performing a method 400 for loading or unloading data from a storage platform 122 using an external stage object 200, according to some embodiments of the present disclosure. A flow diagram illustrating the operation of system 102 is provided. Method 400 is a computer for execution by one or more hardware components (eg, one or more processors) such that the operations of method 400 can be performed by components of network-based data warehouse system 102. It can be implemented as readable instructions. Accordingly, the method 400 is described below by way of example with reference thereto. However, it should be understood that the method 400 may be deployed in a variety of other hardware configurations and is not intended to be limited to deployment within the network-based data warehouse system 102 .

At operation 405 , computing service manager 112 receives a command to create a storage integration object (also referred to as a “storage integration creation command”). The command to create the storage integration is received from a computing device communicating with the data warehouse 102 and specified by a first user via a command line or UI provided to the computing device by the network-based data warehouse system 102 . The first user is an administrative user belonging to a client account of the network-based data warehouse system 102. Typically, commands specify which storage locations are allowed or blocked as part of storage consolidation. For example, the instructions may include an identifier corresponding to a first storage location within the storage platform 122 of the cloud storage provider system 104 to which the network-based data warehouse is permitted access to load and unload data (eg, URL) may be included. The first storage location corresponds to a data storage resource such as a data folder or data bucket. The create storage integration command further specifies the identifier of the cloud storage provider system 104 , the integration name and proxy identity object maintained by the cloud storage provider system 104 . A proxy identity object defines a proxy identity associated with permission to access the first storage location. The proxy identity object further includes permissions allowing the cloud identity to load and unload data from the first storage location assuming the proxy identity.

In some cases, the first storage location may be a default storage location and the create storage integration command may further specify one or more blocked storage locations that are denied access by the network-based data warehouse system 102 . The blocked location may correspond to one or more portions of the first storage location. For example, the first storage location may correspond to a storage bucket (eg, an S3 storage bucket) and the create storage integration command may specify one or more folders within the storage bucket that are denied access.

At operation 410 , computing services manager 112 creates a storage integration object in database 116 based on the storage integration creation command. The storage integration object specifies the primary storage location (eg URL), the cloud storage provider system 104 and the proxy identity object. More specifically, the storage integration object includes a first identifier (eg, URL) corresponding to the first storage location, an identifier of the cloud storage provider system 104 and an identifier of the proxy identity object (eg, ARN). do. Once created, the network-based data warehouse system 102, in operation 415, associates the integration object with the cloud identity object associated with the proxy identity object. A cloud identity object defines a cloud identity used to access cloud services provided by the cloud storage provider system 104 . The cloud identity object is associated with the storage integration object based on the association of the cloud identity object with a client account to which the first user belongs.

In operation 420, the access management system 110 sets permissions for the storage integration object. The access management system 110 may set usage permission based on input received from the first user. Setting usage permission for the integration object may include granting the second identity permission to use the storage integration object to create one or more external stage objects. In some embodiments, the second identity corresponds to at least a second user. In some embodiments, access management system 110 may provide permissions for each user associated with an identity by granting permissions to identities corresponding to multiple users.

At operation 425, computing service manager 112 receives a command to create an external stage object (also referred to as a “create external stage command”). The external stage creation command is received from a computing device communicating with the data warehouse 102 and specified by a second user via a command line or UI provided to the computing device by the network-based data warehouse system 102 . The external stage creation command includes an identifier of the second storage location (eg URL) and an identifier of the storage integration object (eg name of the storage integration object). In some cases, the second storage location may be the same as the first storage location, while in other cases, the second storage location may correspond to a location within the first storage location. That is, the second storage location may correspond to a portion of the first storage location.

At operation 430, the computing service manager 112 creates an external stage object based on the external stage creation command to load or unload data from the second storage location. The external stage object identifies the secondary storage location and includes an association with the storage integration object. More specifically, the external stage object includes an identifier of the second storage location (eg, a URL corresponding to the second storage location) and a reference to the storage integration object (eg, a pointer).

At operation 435, the access management system 110 sets permissions for the external stage object. The access management system 110 may set permissions based on input provided by the second user. Setting permission for use of the integration object may include granting permission for use to a third identity associated with at least a third user. If permission is granted to the third identity, at least the third user may use the external stage object.

At operation 440, computing services manager 112 receives a command to unload from an internal data resource (eg, table) to a third storage location or to load data from a third storage location to an internal data resource. The command contains the identifier of the outer stage object (eg, the name assigned to the outer stage object). The command may be received from the third user's computing device. In response to the command, the computing service manager 112, in operation 445, uses the external stage object 200 to load or unload data from the third storage location of the external cloud storage provider's storage platform 122. In the first example, computing service manager 112 copies data from a storage location to a table maintained at a storage location within network-based data warehouse system 102 . In a second example, computing services manager 112 copies data from a table maintained in an internal storage location to a storage location specified in an instruction. The third storage location corresponds to the second storage location. For example, the third storage location may be the same as the second storage location or may correspond to a location within the second storage location.

As shown in FIG. 5 , method 400 may further include operations 505 , 510 , 515 , 520 and 525 in some embodiments. According to this embodiment, operations 505 and 510 are performed after operation 440 in which the computing service manager 112 receives a command to load or unload data from a storage location.

At operation 505, the computing services manager 112 identifies the storage integration object based on its association with the external stage object. For example, as noted above, external stage objects referenced in instructions that load or unload data contain references (eg, pointers) to integration objects.

At operation 510, computing services manager 112 works with access management system 110 to verify user permissions associated with the third user. In verifying the user rights associated with the third user, computing service manager 112 verifies that the third user has permission to use the external stage object and the storage integration object.

According to such an embodiment, operations 515, 520, and 525 may be performed as part of operation 445 (e.g., as a subroutine or sub-action) in which computing service manager 112 loads or unloads data from a storage location. there is.

At operation 515, the computing service manager 112 verifies that the third storage location is allowed by the storage integration object. That is, the computing service manager 112 checks the third storage location against the first storage location identified in the storage integration object to verify whether the third storage location is within the first storage location. Service manager 112 may further check the third storage location against any blocked storage locations specified by the storage integration object to determine whether the third storage location is allowed by the storage integration object.

At operation 520, the computing service manager 112 accesses the security credentials that will be used to authenticate with the cloud storage provider system 104 to assume the proxy identity defined by the proxy identity object. In some embodiments, security credentials are temporary and may expire after reaching a time limit (eg, 1 hour) and may be limited in scope, particularly for use in loading or unloading data from the first storage location. .

According to some embodiments, computing service manager 112 may obtain security credentials by sending a request for security credentials to authentication and identity management system 118 of cloud storage provider system 104 . The request may include or indicate a first identifier corresponding to the cloud identity object, a second identifier corresponding to the proxy identity object, and security credentials associated with the cloud identity object. The security credentials associated with the cloud identity object may be stored in encrypted form in database 116 . The authentication and identity management system 118 of the cloud storage provider system 104 provides security credentials in response to the request.

At operation 525 , computing services manager 112 uses the credentials obtained from authentication and identity management system 118 to cause the cloud identity to assume the proxy identity defined by the proxy identity object. That is, the cloud identity may interact with the storage location via the proxy identity (eg, by loading or unloading data) and exchange data with the computing service manager 112 using the proxy identity.

6 is a block diagram illustrating components of computing services manager 112, in accordance with some embodiments of the present disclosure. As shown in FIG. 6 , request processing service 602 manages received data storage requests and data retrieval requests (eg, operations to be performed on database data). For example, request processing service 602 may determine data necessary to process a received query (eg, a data storage request or data retrieval request). Data may be stored in a cache within the execution platform 114 or a data storage device in the cloud storage provider system 104 .

Management console service 604 supports access to various systems and processes by administrators and other system administrators. Additionally, management console service 604 can monitor workloads on the system and receive requests to run jobs.

Computing services manager 112 also includes a task compiler 606 , task optimizer 608 and task executor 610 . Job compiler 606 parses the job into a number of discrete tasks and generates executable code for each of the number of discrete tasks. Job optimizer 608 determines the best way to execute a number of individual tasks based on the data to be processed. Job optimizer 608 also handles various data pruning operations and other data optimization techniques to improve the speed and efficiency of job execution. Task executor 610 executes executable code for tasks received from a queue or determined by computing services manager 112 .

Job scheduler and coordinator 612 directs received jobs to appropriate services or systems for compilation, optimization, and dispatch to execution platform 114 . For example, tasks may be prioritized and processed in that prioritized order. In one embodiment, task scheduler and coordinator 612 is a computing service manager along with other “external” tasks such as user queries that may be scheduled by other systems in the database but may utilize the same processing resources on execution platform 114. (112) determines the priority for internal tasks scheduled. In some embodiments, task scheduler and coordinator 612 identifies or assigns specific nodes on execution platform 114 to handle specific tasks. The virtual warehouse manager 614 manages the operation of multiple virtual warehouses implemented in the execution platform 114 . As discussed below, each virtual warehouse includes a number of execution nodes each including a cache and a processor.

Computing services manager 112 also includes a configuration and metadata manager 616 that manages information related to data stored on remote data storage devices and local caches (eg, caches of execution platform 114 ). The configuration and metadata manager 616 uses metadata to determine which data micro-partitions must be accessed to retrieve data to process a particular task or job. The monitor and workload analyzer 618 oversees the processes performed by the computing services manager 112 and manages the distribution of tasks (e.g., workloads) across the virtual warehouse and execution nodes of the execution platform 114. do. The monitor and workload analyzer 618 also redistributes tasks, as needed, based on changes in workload across the data warehouse 102, and users (which may also be processed by the execution platform 114). For example, tasks may be further redistributed based on the “external”) query workload. Configuration and metadata manager 616 and monitor and workload analyzer 618 are coupled to data storage device 620 . Data storage device 620 of FIG. 6 represents any data storage device within data warehouse 102 . For example, data storage device 620 may represent a cache of execution platform 114 , a storage device of cloud storage provider system 104 , or any other storage device.

7 is a block diagram illustrating components of an execution platform 114, in accordance with some embodiments of the present disclosure. As shown in FIG. 7 , execution platform 114 includes several virtual warehouses, including Virtual Warehouse 1, Virtual Warehouse 2, and Virtual Warehouse N. Each virtual warehouse contains several execution nodes, each containing a data cache and a processor. A virtual warehouse can run multiple jobs in parallel using multiple execution nodes. As discussed herein, execution platform 114 may add new virtual warehouses and delete existing virtual warehouses in real time based on the current processing needs of the system and users. This flexibility allows execution platform 114 to quickly deploy large amounts of computing resources when needed without continuing to pay for computing resources when they are no longer needed. All virtual warehouses can access data on all data storage devices (eg, all storage devices in the cloud storage provider system 104 ).

Each virtual warehouse shown in Figure 7 includes three execution nodes, but a particular virtual warehouse may include any number of execution nodes. Also, the number of running nodes in a virtual warehouse is dynamic, so new running nodes are created when there is additional demand, and old running nodes are deleted when they are no longer needed.

Each virtual warehouse can access any of the data storage devices 124-1 through 124-N shown in FIG. Thus, the virtual warehouse is not necessarily assigned to a specific data storage device 124-1 through 124-N, but instead any of the data storage devices 124-1 through 124-N within the cloud storage provider system 104. of data can be accessed. Similarly, each execution node shown in FIG. 7 can access data from any of the data storage devices 124-1 through 124-N. In some embodiments, a particular virtual warehouse or particular execution node may be temporarily assigned to a particular data storage device, but the virtual warehouse or execution node may later access data on any other data storage device.

In the example of Figure 7, virtual warehouse 1 includes three execution nodes 702-1, 702-2 and 702-N. Execution node 702-1 includes a cache 704-1 and a processor 706-1. Execution node 702-2 includes a cache 704-2 and a processor 706-2. Execution node 702-N includes a cache 704-N and a processor 706-N. Each execution node 702-1, 702-2 and 702-N is associated with handling one or more data storage and/or data retrieval tasks. For example, a virtual warehouse may handle data storage and data retrieval tasks associated with internal services such as clustering services, materialized view refresh services, file compression services, stored procedure services, or file upgrade services. In another implementation, a particular virtual warehouse may handle data storage and data retrieval jobs associated with a particular data storage system or particular category of data.

Similar to virtual warehouse 1 discussed above, virtual warehouse 2 includes three execution nodes 712-1, 712-2 and 712-N. Execution node 712-1 includes a cache 714-1 and a processor 716-1. Execution node 712-2 includes a cache 714-2 and a processor 716-2. Execution node 712-N includes a cache 714-N and a processor 716-N. In addition, virtual warehouse 3 includes three execution nodes 722-1, 722-2 and 722-N. Execution node 722-1 includes a cache 724-1 and a processor 726-1. Execution node 722-2 includes a cache 724-2 and a processor 726-2. Execution node 722-N includes a cache 724-N and a processor 726-N.

In some embodiments, the execution node shown in FIG. 7 is stateless with respect to the data that the execution node caches. For example, these execution nodes do not store or otherwise maintain state information about data cached by the execution node or by a particular execution node. Therefore, when a failure occurs in a running node, the failed node can be transparently replaced with another node. Since there is no state information associated with the failed execution node, a new (replacement) execution node can easily replace the failed node without concern for rewriting any particular state.

Although each execution node shown in FIG. 7 includes one data cache and one processor, alternative embodiments may include any number of processors and any number of execution nodes that include any number of caches. Also, the size of the cache may be different for each execution node. The cache shown in FIG. 7 stores, on the local execution node, data retrieved from one or more data storage devices of the cloud storage provider system 104 . Thus, caches reduce or eliminate bottlenecks in platforms that continuously retrieve data from remote storage systems. Instead of repeatedly accessing data from a remote storage device, the systems and methods described herein access data from the cache of the executing node, which is much faster and avoids the bottleneck problem discussed above. In some embodiments, the cache is implemented using high-speed memory devices that provide fast access to cached data. Each cache may store data from any storage device in the cloud storage provider system 104 .

Also, cache resources and computing resources may differ between different execution nodes. For example, one execution node may include significant computing resources and minimal cache resources, so that the execution node is useful for tasks requiring considerable computing resources. Other execution nodes can include significant cache resources and minimal computing resources, and these execution nodes are useful for tasks that require large amounts of data caching. Another execution node may contain cache resources that provide faster I/O operations useful for tasks that require fast scanning of large amounts of data. In some embodiments, cache resources and computing resources associated with a particular execution node are determined when the execution node is created based on expected work to be performed by the execution node.

In addition, cache resources and computing resources associated with a particular execution node may change over time based on changing tasks performed by the execution node. For example, as the work performed by an execution node becomes more processor intensive, more processing resources may be allocated to the execution node. Similarly, more cache resources may be allocated to an execution node if the work being performed by the execution node requires greater cache capacity.

Although virtual warehouses 1, 2 and N are associated with the same execution platform 114, virtual warehouses can be implemented using multiple computing systems in multiple geographic locations. For example, virtual warehouse 1 may be implemented by a computing system in a first geographic location, while virtual warehouses 2 and N are implemented by other computing systems in a second geographic location. In some embodiments, these different computing systems are cloud-based computing systems maintained by one or more different entities.

Additionally, each virtual warehouse is shown in FIG. 7 as having multiple execution nodes. The multiple execution nodes associated with each virtual warehouse may be implemented using multiple computing systems in multiple geographic locations. For example, an instance of Virtual Warehouse 1 implements execution nodes 702-1 and 702-2 on one computing platform in a geographic location, and implements execution nodes 702-N on a different computing platform in another geographic location. do. The selection of a particular computing system to implement an Execution Node depends on the level of resources required for that particular Execution Node (e.g., processing resource requirements and cache requirements), the resources available on the particular computing system, and within or between geographic locations. may depend on a variety of factors, such as the communication capabilities of the network and the computing systems already implementing other execution nodes within the virtual warehouse.

The execution platform 114 is also fault tolerant. For example, if one virtual warehouse fails, that virtual warehouse is quickly replaced by a different virtual warehouse in a different geographic location.

A particular execution platform 114 may include any number of virtual warehouses. Additionally, the number of virtual warehouses within a particular execution platform is dynamic, so new virtual warehouses are created when additional processing and/or cache resources are needed. Similarly, an existing virtual warehouse may be deleted when the resources associated with the virtual warehouse are no longer needed.

In some embodiments, the virtual warehouses may operate on the same data in the cloud storage provider system 104, but each virtual warehouse has its own execution node with independent processing and cache resources. This configuration allows requests for different virtual warehouses to be processed independently without interference between requests. Combined with the ability to dynamically add and remove virtual warehouses, this independent processing supports adding new processing capacity for new users without impacting the performance observed by existing users.

8 illustrates a schematic representation of a machine 800 in the form of a computer system on which a set of instructions may be executed to cause the machine 800 to perform any one or more of the methodologies discussed herein, according to an example embodiment. do. Specifically, FIG. 8 is a diagrammatic representation of a machine 800, which is an exemplary form of a computer system, within which instructions 816 ( eg, software, program, application, applet, app, or other executable code) may be executed. For example, instructions 816 may cause machine 800 to execute any one or more operations of any one or more methods 300 or 400 . As another example, instruction 816 can cause machine 800 to implement a portion of the data flow illustrated in any one or more of FIGS. 3-5 . In this manner, instructions 816 direct a general unprogrammed machine to a specific machine 800 (e.g., access management system 110) that is specifically configured to perform any of the functions described and illustrated in the manner described herein. , computing services manager 112, execution platform 114, authentication and identity management system 118, and computing device 206).

In alternative embodiments, machine 800 may operate as a standalone device or may be connected (eg, networked) to other machines. In a network deployment, machine 800 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 800 may be a server computer, client computer, personal computer (PC), tablet computer, laptop computer, netbook, smart phone, mobile device, network router, network switch, network bridge, or an action to be taken by the machine 800. any machine capable of sequentially or otherwise executing the instructions 816 you specify, but is not limited thereto. Further, while only a single machine 800 is illustrated, the term "machine" also refers to a machine 800 that individually or jointly executes instructions 816 to perform any one or more of the methodologies discussed herein. It is considered to contain a set.

Machine 800 includes a processor 810, memory 830, and input/output (I/O) components 850 configured to communicate with each other over a bus 802. In an exemplary embodiment, a processor 810 (e.g., a central processing unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Radio-frequency Integrated Circuit (RFIC), Other processors or any suitable combination thereof may include, for example, processor 812 and processor 814 capable of executing instructions 816 . The term "processor" is intended to include multi-core processor 810, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions 816 concurrently. 8 depicts multiple processors 810, machine 800 may include a single processor with a single core, a single processor with multiple cores (eg, a multi-core processor), multiple processors with a single core, multiple cores. It may include multiple processors having or a combination thereof.

Memory 830 may include main memory 832 , static memory 834 and storage unit 836 , all accessible to processor 810 via bus 802 . Main memory 832 , static memory 834 , and storage unit 836 store instructions 816 implementing any one or more of the methodologies or functions described herein. Instructions 816 may also, while being executed by machine 800, wholly or partially reside within main memory 832, within static memory 834, within storage unit 836, within at least one of processor 810. (eg, within the cache memory of a processor) or any suitable combination thereof.

I/O components 850 include components for receiving input, providing output, generating output, transferring information, exchanging information, capturing measurements, and the like. The particular I/O components 850 included in a particular machine 800 will depend on the type of machine. For example, portable machines such as mobile phones are likely to include touch input devices or other such input mechanisms, whereas headless server machines are likely not to include such touch input devices. It will be appreciated that I/O component 850 may include many other components not shown in FIG. 8 . I/O components 850 are grouped according to function only to simplify the following discussion and the grouping is by no means limiting. In various exemplary embodiments, I/O component 850 may include output component 852 and input component 854 . The output component 852 may include a visual component (eg, a display such as a plasma display panel (PDF), light emitting diode (LED) display, liquid crystal display (LCD), projector, or cathode ray tube (CRT)), an acoustic component (eg, a speaker), other signal generators, and the like. Input element 854 may include an alphanumeric input element (eg, a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input element), a point-based input element (e.g., mouse, touchpad, trackball, joystick, motion sensor, or other pointing tool), tactile input components (e.g., physical buttons, touch screens that provide touch or position and/or force of touch gestures; or other tactile input components), audio input components (eg, a microphone), and the like.

Communication may be implemented using a variety of technologies. I/O component 850 may include a communication component 864 operable to connect machine 800 to network 880 or device 870 via connection 882 and connection 872, respectively. there is. For example, communication component 864 may include a network interface component or other suitable device for interfacing with network 880 . In further examples, communications component 864 may include wired communications components, wireless communications components, cellular communications components, and other communications components to provide communications over other modalities. Device 870 may be another machine or any of a variety of peripherals (eg, peripherals connected via Universal Serial Bus (USB)). For example, as noted above, machine 800 may correspond to any one of access management system 110, computing services manager 112, execution platform 114, authentication and identity management system 118, and Device 870 may include computing device 206 or any other computing device described herein as communicating with network-based data warehouse system 102 or cloud storage provider system 104 .

executable instructions and machine storage media

The various memories (e.g., 830, 832, 834 and/or memory and/or storage unit 836 of processor(s) 810) are implemented by any one or more of the methodologies or functions described herein, or It may store one or more sets of instructions 816 and data structures (eg, software) that are utilized. These instructions 816, when executed by processor(s) 810, result in various actions to implement the disclosed embodiments.

As used herein, the terms “machine-storage medium,” “device-storage medium,” and “computer-storage medium” mean the same thing and may be used interchangeably in this disclosure. The term refers to single or multiple storage devices and/or media (eg, centralized or distributed databases and/or associated caches and servers) that store executable instructions and/or data. The term is thus considered to include, but not be limited to, solid state memory, optical and magnetic media, including memory internal to or external to the processor. Specific examples of machine-storage media, computer-storage media, and/or device-storage media include semiconductor memory devices such as erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), non-volatile memory including field programmable gate arrays (FPGAs) and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disk; and CD-ROM and DVD-ROM disks. The terms “machine-storage medium,” “computer-storage medium,” and “device-storage medium” specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are discussed below as “signal media.” " is included in the term

transmission medium

In various exemplary embodiments, one or more portions of network 880 may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN) ) Wireless WAN (WWAN), Metropolitan Area Network (MAN), Internet, part of the Internet, part of the Public Switched Telephone Network (PSTN), Common Old Telephone Service (POTS) network, cellular telephone network, wireless network, Wi-Fi® network, It may be a different type of communication network or a combination of two or more such networks. For example, network 880 or a portion of network 880 may include a wireless or cellular network, and the connection 882 utilized may be a Code Division Multiple Access (CDMA) connection, a telecommunications global It may be a Global System for Mobile Communications (GSM) connection or another type of cellular or wireless connection. In this example, the connections are Single Carrier Radio Transmission Technology (lxRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, GSM evolution Enhanced Data rates for GSM Evolution (EDGE) technology, third Generation Partnership Project (3GPP) including 3G, fourth generation wireless (4G) networks, Universal Mobile Telecommunications System UMTS), High-Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standards, and various standard settings It may implement any of various types of data transmission technologies, such as other agency-defined, other long-range protocols, or other data transmission technologies.

Instructions 816 use a transmission medium over a network interface device (e.g., a network interface component included within communication component 864) and use any one of a number of well-known transport protocols (e.g., hyperlinks). may be transmitted or received over the network 880 utilizing the Text Transfer Protocol (HTTP). Similarly, commands 816 can be transmitted or received using a transmission medium over connection 872 to device 870 (eg, a peer-to-peer connection). The terms “transmission medium” and “signal medium” mean the same thing and may be used interchangeably in this disclosure. The terms “transmission medium” and “signal medium” should be considered to include any intangible medium capable of storing, encoding or conveying instructions 816 for execution by machine 800, and include digital or analog communication signals or and other intangible media that facilitate communication of such software. Accordingly, the terms "transmission medium" and "signal medium" should be taken to include all forms of modulated data signals, carrier waves, and the like. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a way as to encode information in the signal.

computer readable media

The terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” mean the same thing and may be used interchangeably in this disclosure. The term is defined to include both machine storage media and transmission media. The term thus includes both storage devices/media and carrier waves/modulated data signals.

Various operations of example methods described herein may be performed at least in part by one or more processors that are either temporarily configured (eg, by software) or permanently configured to perform the associated operations. Similarly, the methods described herein may be at least partially processor implemented. For example, at least some of the operations of methods 300 and 400 may be performed by one or more processors. Execution of certain operations may be distributed across one or more processors, not only residing within a single system, but also distributed across multiple systems. In some demonstrative embodiments, a processor or processors may be located in a single location (eg, within a home environment, office environment, or server farm), while in other embodiments a processor may be distributed across multiple locations. there is.

Although embodiments of the present disclosure have been described with reference to specific exemplary embodiments, it will be apparent that various modifications and changes may be made to these embodiments without departing from the broader scope of the claimed subject matter. Accordingly, the specification and drawings are to be regarded in an illustrative rather than restrictive sense. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which form a part of this application, show by way of example and not limitation, specific embodiments in which the claimed subject matter may be practiced. The illustrated embodiments are described in sufficient detail to enable any person skilled in the art to practice the teachings disclosed herein. Other embodiments may be used and derived therefrom so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Accordingly, this detailed description is not to be taken in a limiting sense, and the scope of the various embodiments is defined only by the appended claims, along with the full range of equivalents given thereto.

Although more than one of these embodiments of claimed subject matter are in fact disclosed, they are individually and/or collectively referred to as "the invention" for convenience only and without the intention of voluntarily limiting the scope of this application to any single invention or inventive concept. may be referred to herein as generically. Thus, while specific embodiments have been illustrated and described herein, it should be understood that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiment shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments and other embodiments not specifically described herein will become apparent to those skilled in the art upon reviewing the above description.

In this document, the terms "a" or "an" are used generically in patent documents, independently of any other occurrence or use of "at least one" or "one or more". , used to include one or more than one. In this document, the term "or" is used to refer to a non-exclusive, so that "A or B" is "A but not B", "B but not A", and "A and B", unless otherwise indicated. includes In the appended claims, the terms "including" and "in which" are used as the plain English equivalents of the terms "comprising" and "wherein" respectively. . Also, in the following claims the terms "including" and "comprising" are open-ended. That is, systems, devices, articles, or processes that include elements other than those listed after such terms in a claim are still deemed to fall within the scope of that claim.

example

Example 1 includes: at least one hardware processor; and at least one hardware processor: creating a storage integration object in the database that identifies a storage location in the storage platform of the cloud storage provider system and a cloud identity object, the cloud identity object being stored in the storage platform of the cloud storage provider system. associated with a proxy identity object granted permission to access the location; In the database, creating an external stage object based on the storage integration object, the external stage object identifying the storage location and including an association with the storage integration object; receiving a command from the computing device to load or unload data from a storage location; and a memory storing instructions that cause, in response to instructions, to perform operations including, via proxy identity objects, loading or unloading data from storage locations using external stage objects. .

In Example 2, the subject matter of Example 1 is: setting permissions associated with an integration object; and optionally further comprising setting permission associated with the external stage object.

In Example 3, the subject matter of any of Examples 1 and 2 is: granting a first user permission to use a storage integration object; and optionally further comprising granting permission to use the external stage object to the second user.

In Example 4, the subject matter of any of Examples 1-3 optionally further includes: receiving a command to create a storage integration object, wherein the command identifies a storage location and a cloud storage provider system; The creation of is based on the command to create a storage integration object.

In Example 5, the subject matter of any of Examples 1-4 optionally further comprises receiving a command to create an external stage object, the command including an identifier corresponding to the storage location and an identifier corresponding to the integration object. include

Example 6 includes the subject matter of any of Examples 1-5, wherein the loading or unloading of data from the storage location comprises: a first security to access the proxy identity object using a second security credential associated with the cloud identity object; access to credentials; and optionally accessing the proxy identity object using security credentials.

In Example 7, the subject matter of any of Examples 1-6 sends, to an access management system of a cloud storage provider system, a first security credential request—the request is to send a second security credential associated with a cloud identity object. Including - Optionally further includes.

In Example 8, the subject matter of any of Examples 1-7 optionally further includes verifying that the storage location is allowed to the storage integration object based on information included in the storage integration object.

In Example 9, the subject matter of any of Examples 1-8 optionally further includes verifying user authorization of a user associated with the command.

In Example 10, the subject matter of any of Examples 1-9 is to: verify that a user has permission to use a storage integration object; and optionally verifying that the user has permission to use the external stage object.

Example 11 includes the subject matter of any of Examples 1-10, wherein the storage integration object optionally includes a first identifier corresponding to a storage location, and the external stage object includes a second identifier corresponding to a portion of the storage location. optionally include

Example 12: Creating, by at least one hardware processor of the machine, a storage integration object in a database of a network-based data warehouse system - the storage integration object is a storage location in a storage platform of a cloud storage provider system and a cloud identity object. identify, and the cloud identity object is associated with a proxy identity object that has been granted permission to access the storage location, in the storage platform of the cloud storage provider; In the database, creating an external stage object based on the storage integration object, the external stage object identifying the storage location and including an association with the storage integration object; receiving a command from the computing device to load or unload data from a storage location; and, in response to the command, loading or unloading data from the storage location, via the proxy identity object, using the external Stage object.

In Example 13, the subject matter of Example 12 is to: set permissions associated with an Integration Object; and optionally further comprising setting permission associated with the external stage object.

In Example 14, the subject matter of any of Examples 12 and 13 is to: grant a first user permission to use a storage integration object; and optionally further comprising granting permission to use the external stage object to the second user.

In Example 15, the subject matter of any of Examples 12-14 optionally further includes receiving an instruction to create a storage integration object, wherein the instruction identifies a storage location and a cloud storage provider system, and optionally further comprising: a storage integration object The creation of is based on the command to create a storage integration object.

In Example 16, the subject matter of any of Examples 12-15 optionally further receives a command to create an external stage object, the command including an identifier corresponding to the storage location and an identifier corresponding to the integration object. include

Example 17 includes the subject matter of any of Examples 12-16, wherein the loading or unloading of data from the storage location optionally includes accessing the proxy identity object using security credentials associated with the proxy identity object.

In Example 18, the subject matter of any of Examples 12-17 optionally further includes verifying user permission of a user associated with the command.

In Example 19, the subject matter of any of Examples 12-18 is to: verify that a user has permission to use a storage integration object; or optionally further comprising verifying that the user has permission to use the external stage object.

Example 20, when executed by a processing device, causes the processing device to: create a storage integration object in a database of a network-based data warehouse system - the storage integration object identifies a cloud identity object and a storage location within a storage platform of a cloud storage provider system. identify, and the cloud identity object is associated with a proxy identity object that has been granted permission to access the storage location, in the storage platform of the cloud storage provider; In the database, creating an external stage object based on the storage integration object, the external stage object identifying the storage location and including an association with the storage integration object; receiving a command from the computing device to load or unload data from a storage location; and, in response to the command, instructions configured to perform an operation including loading or unloading data from a storage location using an external stage object, via the proxy identity object.

In Example 21, the subject matter of Example 20 is to: set permissions associated with an Integration Object; and optionally further comprising setting permission associated with the external stage object.

In Example 22, the subject matter of any one or more of Examples 20 and 21 optionally further comprises receiving a command to create a storage integration object, wherein the command identifies a storage location and a cloud storage provider system, and optionally further comprising: a storage integration object The creation of is based on the command to create a storage integration object.

In Example 23, the subject matter of any one or more of Examples 20-22 optionally receives a command to create an external stage object, wherein the command includes an identifier corresponding to a storage location and an identifier corresponding to a storage integration object. contains more

Example 24 includes the subject matter of any of Examples 20-23, wherein the loading or unloading of data from the storage location uses a second security credential associated with the cloud identity object to access the proxy identity object with a first security credential. access to people; and optionally accessing the proxy identity object using security credentials.

In Example 25, the subject matter of any of Examples 20-24 optionally further includes verifying that the storage location is allowed to the storage integration object based on information included within the storage integration object.

Claims (25)

at least one hardware processor; and
The at least one hardware processor is:
In the database, creating a cloud identity object, a proxy identity object, and a storage integration object that identifies a storage location within a storage platform of a cloud storage provider system - the cloud identity An object, in the storage platform of the cloud storage provider, is associated with the proxy identity object, the proxy identity object corresponding to a proxy identity granted permission to access the storage location, the proxy identity object being the cloud identity object. contains a set of permissions allowing loading or unloading of data from a storage location assuming this proxy identity;
In the database, creating an external stage object based on the storage integration object, the external stage object identifying the storage location and including an association with the storage integration object, the external stage object is operable to load or unload data from the storage location based on the cloud identity assuming the proxy identity;
receiving a command from a computing device to load or unload data from the storage location; and
In response to the command, loading or unloading the data from the storage location, via the proxy identity object, using the external stage object.
A network-based data warehouse system comprising a memory for storing instructions for performing operations including. The method of claim 1 , wherein the operation:
setting permissions associated with the storage integration object; and
The network-based data warehouse system further comprising setting permissions associated with the external stage object. According to claim 2:
the setting of the usage permission associated with the storage integration object includes granting permission to use the storage integration object to a first user;
and the setting of the usage permission associated with the external stage object includes granting permission to use the external stage object to a second user. The method of claim 1 , wherein the operation:
further comprising receiving a command to create the storage integration object, wherein the command identifies the storage location and the cloud storage provider system, wherein the creation of the storage integration object comprises the command to create the storage integration object. A command-based, network-based data warehouse system. The method of claim 1 , wherein the operation:
receiving a command to create the external stage object, the command including an identifier corresponding to the storage location and an identifier corresponding to the storage integration object. 2. The method of claim 1, wherein the loading or unloading of the data from the storage location:
accessing a first security credential for accessing the proxy identity object using a second security credential associated with the cloud identity object; and
and accessing the proxy identity object using the security credentials. 7. The method of claim 6, wherein the access of the security credentials:
network-based data comprising sending a request for the first security credential to an access management system of the cloud storage provider system, the request including the second security credential associated with the cloud identity object. warehouse system. The network-based data warehouse system of claim 1 , wherein the operation further comprises verifying that the storage location is permitted to the storage integration object based on information included in the storage integration object. 2. The network-based data warehouse system of claim 1, wherein the operations further include verifying user permissions of a user associated with the command. 10. The method of claim 9, wherein the verification of the user permission:
verifying that the user has permission to use the storage integration object; or
and verifying that the user has permission to use the external stage object. According to claim 1:
the storage integration object includes a first identifier corresponding to the storage location;
and the external stage object includes a second identifier corresponding to a portion of the storage location. creating, by at least one hardware processor of the machine, a storage integration object in a database of a network-based data warehouse system, the storage integration object comprising: a cloud identity object, a proxy identity object, and a storage location within a storage platform of the cloud storage provider system; identify a cloud identity object, in the storage platform of the cloud storage provider, is associated with the proxy identity object, the proxy identity object corresponding to a proxy identity granted permission to access the storage location; a proxy identity object contains a set of permissions allowing the cloud identity to load or unload data from a storage location assuming the proxy identity;
In the database, creating an external stage object based on the storage integration object, the external stage object identifying the storage location and including an association with the storage integration object, the external stage object identifying the proxy identity. operable to load or unload data from the storage location based on the assumed cloud identity;
receiving a command from the computing device to load or unload data from the storage location; and
in response to the command, via the proxy identity object, loading or unloading the data from the storage location using the external Stage object. According to claim 12:
setting permissions associated with the storage integration object; and
The method further comprising setting a permission associated with the external stage object. According to claim 13:
the setting of the usage permission associated with the storage integration object includes granting permission to use the storage integration object to a first user;
wherein the setting of the usage permission associated with the external Stage object comprises granting permission to use the external Stage object to a second user. According to claim 12:
further comprising receiving a command to create the storage integration object, wherein the command identifies the storage location and the cloud storage provider system, wherein the creation of the storage integration object comprises the command to create the storage integration object. Command-based method. According to claim 12:
receiving a command to create the external stage object, the command including an identifier corresponding to the storage location and an identifier corresponding to the storage integration object. 13. The method of claim 12, wherein the loading or unloading of the data from the storage location:
and accessing the proxy identity object using security credentials associated with the proxy identity object. 13. The method of claim 12, further comprising verifying user permissions of a user associated with the command. 19. The method of claim 18, wherein the verification of the user permission:
verifying that the user has permission to use the storage integration object; or
and verifying that the user has permission to use the external stage object. When executed by at least one hardware processor of a machine, the machine:
creating, by at least one hardware processor of the machine, a storage integration object in a database of a network-based data warehouse system, the storage integration object comprising: a cloud identity object, a proxy identity object, and a storage location within a storage platform of the cloud storage provider system; identify a cloud identity object, in the storage platform of the cloud storage provider, is associated with the proxy identity object, the proxy identity object corresponding to a proxy identity granted permission to access the storage location; a proxy identity object contains a permission set allowing the cloud identity to load or unload data from the storage location assuming the proxy identity;
In the database, creating an external stage object based on the storage integration object, the external stage object identifying the storage location and including an association with the storage integration object, the external stage object identifying the proxy identity. operable to load or unload data from the storage location based on the assumed cloud identity;
receiving a command from the computing device to load or unload data from the storage location; and
In response to the command, loading or unloading the data from the storage location, via the proxy identity object, using the external stage object.
A computer storage medium comprising instructions configured to perform an operation comprising: 21. The method of claim 20, wherein the instructions cause the machine to:
setting permissions associated with the storage integration object; and
A computer storage medium configured to perform additional operations including setting permissions associated with the external stage object. 21. The method of claim 20, wherein the instructions cause the machine to:
receive a command to create the storage integration object, wherein the command identifies the storage location and the cloud storage provider system; A computer storage medium based on the instructions to create an integration object. 21. The method of claim 20, wherein the instructions cause the machine to:
receiving a command to create the external stage object, the command including an identifier corresponding to the storage location and an identifier corresponding to the storage integration object. 21. The method of claim 20, wherein the instructions cause the machine to:
accessing a first security credential for accessing the proxy identity object using a second security credential associated with the cloud identity object; and
A computer storage medium configured to perform an additional operation comprising accessing the proxy identity object using the security credentials. 21. The method of claim 20, wherein the instructions cause the machine to:
and performing an additional operation comprising verifying that the storage location is allowed to the storage integration object based on information included in the storage integration object.

Images