WO2023114164A1 - Processus de certification de données pour plateforme de base de données en nuage - Google Patents

Processus de certification de données pour plateforme de base de données en nuage Download PDF

Info

Publication number
WO2023114164A1
WO2023114164A1 PCT/US2022/052622 US2022052622W WO2023114164A1 WO 2023114164 A1 WO2023114164 A1 WO 2023114164A1 US 2022052622 W US2022052622 W US 2022052622W WO 2023114164 A1 WO2023114164 A1 WO 2023114164A1
Authority
WO
WIPO (PCT)
Prior art keywords
rules
database
data
access
records
Prior art date
Application number
PCT/US2022/052622
Other languages
English (en)
Inventor
Nagender Gurram
Yudhish Batra
Syed Shamaz Salim
Praveen Kandukuri
Original Assignee
Capital One Services, Llc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/550,032 external-priority patent/US11392715B1/en
Priority claimed from US17/550,036 external-priority patent/US11392587B1/en
Priority claimed from US17/550,040 external-priority patent/US11829367B2/en
Application filed by Capital One Services, Llc filed Critical Capital One Services, Llc
Publication of WO2023114164A1 publication Critical patent/WO2023114164A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6227—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database where protection concerns the structure of data, e.g. records, types, queries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G06F21/6254—Protecting personal data, e.g. for financial or medical purposes by anonymising data, e.g. decorrelating personal data from the owner's identification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/2141—Access rights, e.g. capability lists, access control lists, access tables, access matrices

Definitions

  • aspects of the disclosure relate generally to data storage and retrieval. More specifically, aspects of the disclosure relate to a data certification process for implementing privacy and data restrictions on a cloud database platform that provides access to shared databases.
  • Cloud database platforms such as the Snowflake architecture, produced by Snowflake Inc. of San Mateo, CA, permit organizations to logically separate but natively integrate storage, computing, and services.
  • Snowflake and similar “data warehouse as a service” platforms may provide users access to cloud database storage, whereby storage of data is maintained in separate servers. This process allows data creators to share their data with a wide variety of consumers. Given the complexity and size of many data warehouses, the task of executing queries and collecting the results of those queries is often tasked to computing devices specially configured for that purpose.
  • Such computing devices may be, as is the case with Snowflake, one or more servers which may instantiate virtual warehouses for a user to conduct searches within.
  • This process also allows users and companies to offload complex and expensive data warehousing and query operations to a cloud provider. For example, a user seeking to query a multi-terabyte data warehouse may, rather than trying to execute the query and collect results on their laptop, send instructions to a virtual warehouse in the cloud that causes one or more servers to, via a virtual warehouse, perform the query on their behalf. This allows the user to access the results of the data (e.g., in a user interface) from a relatively underpowered computing device.
  • Snowflake architecture In addition to avoiding resource limitations associated with queries, another advantage of the Snowflake architecture is that it allows users to collect data in a way that is resilient. Because a user’s laptop may be relatively underpowered, queries that request significant amounts of data may crash the laptop. Moreover, because a single device collects the results of a query, unexpected technical issues (e.g., power loss, Internet disconnects) may cause the entire query to fail.
  • the Snowflake architecture is equipped with built-in replication and failover/failback procedures which avoid such crashes, thereby ensuring that data continuity may be preserved.
  • Snowflake architecture improves conventional query execution is that Snowflake allows virtual warehouses to be created, modified, and destroyed as desired. This allows multiple queries to be executed simultaneously but separately. For example, the Snowflake architecture allows a first user from an organization to execute a first query in a first virtual warehouse at the same time that a second user from the same organization executes a second query in a second virtual warehouse. To preserve computing resources, the different virtual warehouses may be configured with different computing resources.
  • One useful feature in Snowflake is the ability to share data without needing to copy that data over from one storage device for another.
  • This process might be referred to as a “zero copy” process, referring to the fact that the underlying data need not be copied for it to be shared.
  • an owner of data (which might also be referred to as a data producer and/or data creator) might sell access to all or portions of their data to one or more consumers, such that the one or more consumers might use virtual warehouses to access and execute queries against that data. In this manner, the consumers gain quick and easy access to the data, while the owner maintains control of the data.
  • this means that needless copies of the data are not created, which means that updates to the data are available to all users.
  • Snowflake data sharing functionality
  • different consumers of data within the environment might need the data to be pre-processed and/or otherwise certified for different scenarios.
  • one consumer of financial data might want only portions of data that are particularly accurate and/or reliable, whereas another consumer of the data might be legally restricted to accessing only portions of the financial data.
  • the existing manner in which virtual warehouses access data in the Snowflake environment does not account for these various needs, which can introduce problems into the data sharing process. For instance, if a consumer is legally permitted to only access a certain type of data stored in a database, then the provider of that data might be forced to generate an entirely new database comprising that data, effectively nullifying the various benefits of the Snowflake data sharing platform.
  • the process of sharing data with third party consumers can become a cumbersome and time-consuming process, requiring a significant amount of time and computing resources be devoted to data extraction, processing, and loading.
  • aspects described herein relate to providing access to records of a database stored on a database server by generating a limited consumer view definition via which a consumer of data might access data.
  • Data producers may create data and store it in a database on a database server in a cloud database platform.
  • a company might generate financial records data through its operations, then store that data in the Snowflake platform.
  • that company may not only store their own data in the cloud (which has its own benefits, particularly with respect to the use of virtual warehouses), but may also readily share the data with others (e.g., consumers of that data, such as other organizations).
  • the data might be provided through a data marketplace, whereby users might exchange (e.g., sell) access to their data as stored in the cloud database platform.
  • This process may advantageously allow the data producer to share its data with other organizations (e.g., for a fee) in a manner which provides those consumers ready and convenient access to that data. That said, in many circumstances, the data producer might not want to provide the entirety of the data to consumers. As one example, the consumer might request only particularly reliable portions of the data producer’s data. As another example, the consumer might be legally permitted to access only certain portions of data producer’s data. In such circumstances, rules might be determined that limit consumer access to records of the data producer’s data.
  • Those rules might be set by the producer of the data (e.g., preventing the consumer from accessing confidential information) and/or by the consumer (e.g., a rule requesting only valid data). Then, based on those rules, a table of certification results might be generated, and a limited consumer view definition might be generated. That limited consumer view definition might be usable by the consumer to access a particular portion of the data stored by the database (and, in turn, might exclude a different portion of the data stored by the database). In this manner, the data producer can provide its data without having to modify and/or copy its data, the consumer has access to the latest form of the data, and the limited consumer view definition may be leveraged to ensure that the consumer receives appropriate data.
  • a computing device may determine a shared view definition for access to the database stored on the database server, wherein the shared view definition is configured to provide access to all records stored by the database and to enable execution of queries against the database using processing resources of one or more virtual warehouses provided by the cloud database platform.
  • the computing device may determine one or more first rules that specify criteria, associated with consumer permissions to access the database via the cloud database platform, that limit consumer access to the records stored by the database.
  • the computing device may perform, based on the one or more first rules, a data access certification process on the records stored by the database to generate a table of certification results by accessing all records stored by the database using the shared view definition, generating a data certification result for each record based on determining, for each record, whether a given record satisfies the criteria of the one or more first rules based on one or more fields of the given record, and generating, based on the data certification result for each record, the table of certification results that indicates, for each record, whether the record satisfies the criteria of the one or more first rules.
  • the computing device may generate, based on an intersection of the table of certification results and the shared view definition, and without modifying the records stored by the database, a limited consumer view definition configured to provide access to a first portion of the records in compliance with the criteria of the one or more first rules and exclude a second portion of the records not in compliance with the criteria of the one or more first rules without modifying the records stored by the database.
  • the computing device may then cause a first virtual warehouse, of the one or more virtual warehouses, to execute a query on the first portion of the records in compliance with the criteria of the one or more first rules via the limited consumer view definition.
  • the computing device may then cause output of a result of the query to a consumer authorized to access the database through the limited customer view definition.
  • the computing device may determine a shared view definition for access to the database stored on the database server, wherein the shared view definition is configured to provide access to all records stored by the database and to enable execution of queries against the database using processing resources of one or more virtual warehouses provided by the cloud database platform.
  • the computing device may determine one or more attributes of the database and provide, to a user device and based on the one or more attributes of the database, a user interface enabling creation of rules that specify criteria, associated with consumer permissions to access the database via the cloud database platform, that limit consumer access to the records stored by the database.
  • the computing device may then generate, based on criteria received via the user interface, one or more first rules that limit the output of the data.
  • the computing device may then perform, based on the one or more first rules, a data access certification process on the records stored by the database to generate a table of certification results by accessing all records stored by the database using the shared view definition, generating a data certification result for each record based on determining, for each record, whether a given record satisfies the criteria of the one or more first rules based on one or more fields of the given record, and generating, based on the data certification result for each record, the table of certification results that indicates, for each record, whether the record satisfies the criteria of the one or more first rules.
  • the computing device may generate, based on an intersection of the table of certification results and the shared view definition, and without modifying the records stored by the database, a limited consumer view definition configured to provide access to a first portion of the records in compliance with the criteria of the one or more first rules and exclude a second portion of the records not in compliance with the criteria of the one or more first rules without modifying the records stored by the database. Then, the computing device may cause a first virtual warehouse, of the one or more virtual warehouses, to execute a query on the records in compliance with the criteria of the one or more first rules via the limited consumer view definition.
  • aspects described herein may also relate to a process that addresses updates to the data.
  • the computing device may detect, via the shared view definition, an update to at least one record of the records stored by the database.
  • the computing device may then perform the data access certification process on the updated records to generate a second table of certification results.
  • the computing device may generate, based on the first table of certification results, the second table of certification results, and the shared view definition, a updated limited consumer view definition different from the limited consumer view definition.
  • the computing device may cause a first virtual warehouse, of the one or more virtual warehouses, to execute a query on the first portion of the records in compliance with the criteria of the one or more first rules via the updated limited consumer view definition.
  • FIG. 1 shows an example of a system in which one or more aspects described herein may be implemented.
  • FIG. 2 shows an example computing device in accordance with one or more aspects described herein.
  • FIG. 3 depicts computing devices, virtual warehouse servers, and data warehouses working in conjunction to enable execution of queries.
  • FIG. 4 shows a flow chart which may be performed to generate a limited consumer view definition based on a data access certification process.
  • FIG. 5 shows a flow chart which may be performed to prompt users for rules based on attributes of a database.
  • FIG. 6 shows a flow chart which may be performed to generate a updated limited consumer view definition based on an update to data.
  • FIG. 7 depicts an example deep neural network architecture for a model according to one or more aspects of the disclosure.
  • FIG. 8 depicts an illustrative application of a shared view definition, an illustrative first limited consumer view definition, and an illustrative second limited consumer view definition with respect to illustrative data.
  • FIG. 9 depicts an illustrative user interface enabling creation of rules that specify criteria that limit consumer access to records stored by a database.
  • FIG. 10 depicts illustrative rules.
  • FIG. 11 depicts an illustrative architecture for a data certification process.
  • aspects discussed herein may relate to methods and techniques for allowing data producers to share data with consumers in a data sharing marketplace, and in particular a manner in which limits can be placed on consumer access to shared data.
  • This functionality is effectuated via limited consumer view definitions which limits consumers to portions of data in compliance with one or more rules. Those rules may be established by the data producer, the consumer, and/or other parties.
  • the data stored by the cloud database platform might be freely shared by the data producer without requiring that the data itself be duplicated, modified, and/or otherwise processed to be shared.
  • This process might be referred to as a “zero copy” process, whereby data might be shared near-instantaneously and without requiring that the data be copied or otherwise modified for the consumer’s use.
  • the aforementioned limited consumer view definitions provide limits on data provided to consumers via the cloud database platform when such consumers perform queries via virtual warehouses.
  • these limited consumer view definitions can allow consumers to access portions of data to which they are legally permitted to access while preventing those same consumers from inadvertently gaining access to portions of the data to which they are not legally permitted to access. This avoids the need to maintain additional cloud storage (and/or file transfer protocol setups), reduces the staff expense to prepare and send data, removes the need to pay for storage or a database to house duplicative data, and generally just results in an easier-to-maintain marketplace for data sharing.
  • One advantage of the present disclosure is that the limited consumer view definitions generated herein need not modify and/or copy any data stored in the Snowflake environment.
  • This approach has numerous benefits.
  • the data need not be copied over to a separate database, one single copy of the data may be stored, and thus updates to various records of the data need only be performed once (and, e.g., all consumers of the data have access to the latest copy of the data at any given time).
  • the limited consumer view definitions reflect rules (and, e.g., not the data itself), changes to the rules might be made over time. For example, a first rule might provide that a consumer is permitted to access only the last four digits of a credit card number.
  • That rule might be later changed to provide that a consumer is permitted to access the last eight digits of the credit card number.
  • an entirely new copy of the database may need to be generated: after all, a first copy of the database comporting with the first rule might have had data deleted from it, such that the first copy of the database no longer contains the last eight digits of the credit card number.
  • the limited consumer view definition might be modified, and the underlying data need not be changed.
  • the present disclosure is significantly different than conventional data filtering and organization processes at least because it is fundamentally rooted in a cloud database platform that features zero-copy data sharing and view definitions leveraged by virtual warehouses.
  • a cloud database platform such as Snowflake, enables the use of limited consumer view definitions and virtual warehouses in a manner which permits the application of rules in a manner separate from storage of the underlying data itself.
  • limited consumer view definitions can serve to limit the consumer’s access to that data regardless of the nature of the query, the nature of the processing resources used, or the like.
  • the present disclosure also improves the functioning of computers by improving the manner in which queries are executed with respect to one or more data warehouses.
  • Conventional (e.g., non-cloud) data storage approaches can be wasteful, particularly when data is shared between different consumers. For example, for a data creator to share data with a consumer, that creator might send the entirety of the data over to the consumer. This can waste unnecessary storage space and computing resources, and introduces a large number of other concerns (e.g., versioning, privacy control, etc.).
  • the present disclosure avoids these issues by maintaining a single version of the data, while providing limited consumer view definitions that nonetheless allow consumers limited access to that data. This avoids the unnecessary (e.g., duplicative) storage of additional copies of the data, ensures that all consumers have access to the latest form of the data, and allows for the rules underpinning limited consumer view definitions to be changed as desired.
  • the present disclosure is also fundamentally rooted in computing devices and, in particular, an environment with virtual warehouses.
  • Snowflake’s architecture is unique in that it allows for the cloud storage of data, with consumers of that data able to access the data through virtual warehouses.
  • other database systems rely on monolithic systems to handle all enterprise needs. It is precisely this architecture of Snowflake (and similar virtual warehouse systems) that is leveraged by the improvements discussed herein.
  • FIG. 1 shows a system 100.
  • the system 100 may include one or more computing devices 110, one or more data warehouses 120, and/or one or more virtual warehouse servers 130 in communication via a network 140.
  • network connections shown are illustrative and any means of establishing a communications link between the computers may be used.
  • the existence of any of various network protocols such as TCP/IP, Ethernet, FTP, HTTP and the like, and of various wireless communication technologies such as GSM, CDMA, WiFi, and LTE, is presumed, and the various computing devices described herein may be configured to communicate using any of these network protocols or technologies. Any of the devices and systems described herein may be implemented, in whole or in part, using one or more computing systems described with respect to FIG. 2.
  • the computing devices 110 may, for example, provide queries to the virtual warehouse servers 130 and/or receive query results from the virtual warehouse servers 130, as described herein.
  • the data warehouses 120 may store data and provide, in response to queries, all or portions of the stored data, as described herein.
  • the data warehouses 120 may include, but are not limited to relational databases, hierarchical databases, distributed databases, in-memory databases, flat file databases, XML databases, NoSQL databases, graph databases, and/or a combination thereof.
  • the virtual warehouse servers 130 may execute, manage, resize, and otherwise control one or more virtual warehouses, as described herein.
  • one or more of the computing devices 110 may send a request to execute a query to one or more of the virtual warehouse servers 130, and one or more virtual warehouses of the virtual warehouse servers 130 may perform steps which effectuate that query with respect to one or more of the data warehouses 120.
  • the network 140 may include a local area network (LAN), a wide area network (WAN), a wireless telecommunications network, and/or any other communication network or combination thereof.
  • the virtual warehouse servers 130 and/or the data warehouses 120 may be all or portions of a cloud system.
  • the computing devices 110 may be located in a first location (e.g., the offices of a corporation), and the virtual warehouse servers 130 and/or the data warehouses 120 may be located in a variety of locations (e.g., distributed in a redundant manner across the globe). This may protect business resources: for example, if the Internet goes down in a first location, the distribution and redundancy of various devices may allow a business to continue operating despite the outage.
  • the virtual warehouse servers 130 may be all or portions of a virtual warehouse as a service system, such as is provided via the Snowflake architecture.
  • the computing devices 110 and/or the data warehouses 120 may be managed by an organization.
  • the virtual warehouse servers 130 may be managed by a different entity, such as Snowflake Inc.
  • a third party e.g., Snowflake
  • a data warehouse such as any one of the data warehouses 120, may be one or more databases or other devices which store data.
  • a data warehouse may be a single database, a collection of databases, or the like.
  • a data warehouse may be structured and/or unstructured, such that, for example, a data warehouse may comprise a data lake.
  • a data warehouse may store data in a variety of formats and in a variety of manners.
  • a data warehouse may comprise textual data in a table, image data as stored in various file system folders, and the like.
  • the data transferred to and from various computing devices in a system 100 may include secure and sensitive data, such as confidential documents, consumer personally identifiable information, and account data. Therefore, it may be desirable to protect transmissions of such data using secure network protocols and encryption, and/or to protect the integrity of the data when stored on the various computing devices.
  • a file-based integration scheme or a service-based integration scheme may be utilized for transmitting data between the various computing devices.
  • Data may be transmitted using various network communication protocols.
  • Secure data transmission protocols and/or encryption may be used in file transfers to protect the integrity of the data, for example, File Transfer Protocol (FTP), Secure File Transfer Protocol (SFTP), and/or Pretty Good Privacy (PGP) encryption.
  • FTP File Transfer Protocol
  • SFTP Secure File Transfer Protocol
  • PGP Pretty Good Privacy
  • one or more web services may be implemented within the various computing devices.
  • Web services may be accessed by authorized external devices and users to support input, extraction, and manipulation of data between the various computing devices in the system 100.
  • Web services built to support a personalized display system may be cross-domain and/or cross-platform, and may be built for enterprise use. Data may be transmitted using the Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocol to provide secure connections between the computing devices.
  • Web services may be implemented using the WS-Security standard, providing for secure SOAP messages using XML encryption.
  • Specialized hardware may be used to provide secure web services.
  • secure network appliances may include built-in features such as hardware- accelerated SSL and HTTPS, WS-Security, and/or firewalls. Such specialized hardware may be installed and configured in the system 100 in front of one or more computing devices such that any external devices may communicate directly with the specialized hardware.
  • the computing device 200 may be the same or similar as any one of the computing devices 110, the virtual warehouse servers 130, and/or the data warehouses 120 of FIG. 1.
  • the computing device 200 may include a processor 203 for controlling overall operation of the computing device 200 and its associated components, including RAM 205, ROM 207, input/output device 209, communication interface 211, and/or memory 215.
  • a data bus may interconnect processor(s) 203, RAM 205, ROM 207, memory 215, VO device 209, and/or communication interface 211.
  • computing device 200 may represent, be incorporated in, and/or include various devices such as a desktop computer, a computer server, a mobile device, such as a laptop computer, a tablet computer, a smart phone, any other types of mobile computing devices, and the like, and/or any other type of data processing device.
  • I/O device 209 may include a microphone, keypad, touch screen, and/or stylus through which a user of the computing device 200 may provide input, and may also include one or more of a speaker for providing audio output and a video display device for providing textual, audiovisual, and/or graphical output.
  • Software may be stored within memory 215 to provide instructions to processor 203 allowing computing device 200 to perform various actions.
  • memory 215 may store software used by the computing device 200, such as an operating system 217, application programs 219, and/or an associated internal database 221.
  • the various hardware memory units in memory 215 may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data.
  • Memory 215 may include one or more physical persistent memory devices and/or one or more non- persistent memory devices.
  • Memory 215 may include, but is not limited to, random access memory (RAM) 205, read only memory (ROM) 207, electronically erasable programmable read only memory (EEPROM), flash memory or other memory technology, optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store the desired information and that may be accessed by processor 203.
  • Communication interface 211 may include one or more transceivers, digital signal processors, and/or additional circuitry and software for communicating via any network, wired or wireless, using any protocol as described herein.
  • Processor 203 may include a single central processing unit (CPU), which may be a singlecore or multi-core processor, or may include multiple CPUs. Processor(s) 203 and associated components may allow the computing device 200 to execute a series of computer-readable instructions to perform some or all of the processes described herein. Although not shown in FIG.
  • various elements within memory 215 or other components in computing device 200 may include one or more caches, for example, CPU caches used by the processor 203, page caches used by the operating system 217, disk caches of a hard drive, and/or database caches used to cache content from database 221.
  • the CPU cache may be used by one or more processors 203 to reduce memory latency and access time.
  • a processor 203 may retrieve data from or write data to the CPU cache rather than reading/writing to memory 215, which may improve the speed of these operations.
  • a database cache may be created in which certain data from a database 221 is cached in a separate smaller database in a memory separate from the database, such as in RAM 205 or on a separate computing device.
  • a database cache on an application server may reduce data retrieval and data manipulation time by not needing to communicate over a network with a back- end database server.
  • caches and others may be included in various embodiments, and may provide potential advantages in certain implementations of devices, systems, and methods described herein, such as faster response times and less dependence on network conditions when transmitting and receiving data.
  • FIG. 3 shows a system comprising a data sharing platform 307 (which comprises, e.g., the computing devices 110 of FIG. 1) and a cloud database platform 306 (which comprises, e.g., the virtual warehouse servers 130 and the data warehouses 120 of FIG. 1).
  • FIG. 3 may depict all or portions of a system configured according to the Snowflake architecture or a similar architecture, which provides access to cloud databases (in a database-as-a-service format) via which users may share via a data marketplace and/or may submit queries using one or more virtual warehouses.
  • FIG. 3 also depicts various elements which may be portions of those computing devices, as well as transmissions between those devices.
  • the computing devices 110 are shown having a request application 301
  • the virtual warehouse servers 130 are shown having a virtual warehouse manager application 302 and three virtual warehouses (a virtual warehouse A 303a, a virtual warehouse B 303b, and a virtual warehouse C 303c)
  • the data warehouses 120 are shown comprising a data warehouse A 304a and a data warehouse B 304b. All or portions of these devices may be part of the Snowflake architecture or another architecture.
  • the computing devices 110 may be users’ personal computing devices
  • the virtual warehouse servers 130 may be cloud servers managed by Snowflake Inc., of San Mateo, CA.
  • the data sharing platform 307 and cloud database platform 306 are shown as separate in FIG. 3.
  • the data sharing platform 307 and the cloud database platform 306 may be managed by the same or different entities.
  • the cloud database platform 306 may correspond to preexisting Snowflake architecture managed by Snowflake Inc. of San Mateo, CA, whereas the data sharing platform 307 may be managed by another organization.
  • some of the computing devices, networks, and other aspects of the data sharing platform 307 and/or the cloud database platform 306 may overlap.
  • some of the devices managed by one entity might be located in offices managed by Snowflake, and/or the devices in the data sharing platform 307 may be communicatively coupled to devices in the cloud database platform 306 via a private network.
  • the request application 301 may transmit, to the virtual warehouse manager application 302, a request for a query.
  • the transmitted request may be in a variety of formats which indicate a request for a query to be executed.
  • the request may comprise a structured query which may be directly executed on one or more of the data warehouses 120 (such as an SQL query), and/or may comprise a vaguer request for data (e.g., a natural language query, such as a request for “all data in the last month”).
  • the request application 301 may be any type of application which may transmit a request to the virtual warehouse manager application 302, such as a web browser (c.g, showing a web page associated with the virtual warehouse manager application 302), a special-purpose query application (c.g, as part of a secure banking application, such as may execute on a tablet or smartphone), an e-mail application (e.g., such that the request to the virtual warehouse manager application 302 may be transmitted via e-mail), or the like.
  • the request may be input by a user in a user interface of the request application 301 and using, for example, a keyboard, a mouse, voice commands, a touchscreen, or the like.
  • the virtual warehouse manager application 302 may select one of a plurality of available virtual warehouses (in this case, the virtual warehouse C 303C) to execute the query. As part of this process, the virtual warehouse manager application may determine which of a plurality of virtual warehouses should address the request received in step 305. The virtual warehouse manager application 302 may identify an execution plan for the query by determining one or more sub-queries to be executed with respect to one or more of the data warehouses 120. For example, the request may comprise querying both the data warehouse A 304a and the data warehouse B 304b for different portions of data. The virtual warehouse manager application 302 may, based on the query and the execution plan, predict a processing complexity of the query.
  • the processing complexity of the query may correspond to a time to complete the query (e.g., the time required to perform all steps of the execution plan), a quantity of computing resources (e.g., processor time, memory) required to execute the query, or the like.
  • the virtual warehouse manager application 302 may additionally and/or alternatively determine an operating status of the plurality of virtual warehouses and/or processing capabilities of the plurality of virtual warehouses.
  • the virtual warehouse A 303a is shown as being large (e.g., having relatively significant processing capabilities) but having a utilization of 99% (that is, being quite busy), the virtual warehouse B 303b is shown as being large and having a utilization of 5% (that is, being quite free), and the virtual warehouse C 303c is shown as being small and having a utilization of 5%.
  • a subset of the plurality of virtual warehouses may be selected.
  • that subset may comprise both the virtual warehouse B 303b and the virtual warehouse C 303c, at least because both have a low utilization rate and thus may be capable of handling the request received from the request application 301.
  • one or more virtual warehouses may be selected to execute the query.
  • the virtual warehouse C 303c has been selected to address the query. This may be because, for example, the query may be small (that is, the execution plan may be simple or otherwise quick to handle), such that executing the query on the virtual warehouse C 303c may be cheaper and may free up the virtual warehouse B 303b for handling larger, more complex queries.
  • Virtual warehouses such as the virtual warehouse A 303a, the virtual warehouse B 303b, and/or the virtual warehouse C 303c, may comprise a respective set of computing resources.
  • each virtual warehouse may execute on one or a plurality of servers (e.g, the virtual warehouse servers 130), and each virtual warehouse may be apportioned a particular quantity of computing resources (e.g, computing processor speed, memory, storage space, bandwidth, or the like).
  • such quantities of computing resources may be referred to via “t-shirt sizes,” such that one virtual warehouses may be referred to as “large,” whereas another may be referred to as “small.”
  • Virtual warehouses may be resized such that, for example, the virtual warehouse A 303a (which is large) may be shrunk down to a smaller size to save money and/or to allocate resources to another virtual warehouse.
  • Virtual warehouses may also have different utilization rates. For example, a virtual warehouse using substantially all of its resources to execute a query may be said to be fully occupied (that is, to have a utilization rate of approximately 100%), whereas a virtual warehouse not performing any tasks may be said to be free (that is, to have a utilization rate of approximately 0%).
  • the size of the virtual warehouses may affect the utilization rate: for example, a larger virtual warehouse may be capable of handling more queries at the same time as compared to a relatively smaller virtual warehouse.
  • virtual warehouses may be configured to execute one or more queries with respect to at least a portion of the data warehouses 120, collect results from the one or more queries, and provide, to one or more computing devices, access to the collected results.
  • the size and/or utilization of a particular virtual warehouse may impact its ability to enable execution of queries, collect results, and provide those results.
  • Virtual warehouses may use one or more view definitions to retrieve content from the databases 120.
  • a virtual warehouse might use a view definition to specify which portion(s) of data stored in the databases 120 should be displayed to a user.
  • view definitions might be established such that, for example, a consumer of data might not have access to all data stored by a database, but rather might be limited to a portion of that data.
  • the virtual warehouse manager application 302 is shown as part of the virtual warehouse servers 130, the virtual warehouse manager application 302 may execute on a wide variety of computing devices.
  • the virtual warehouse manager application may execute on one or more of the computing devices 110, such as the same computing device 110 hosting the request application 301.
  • the virtual warehouse manager application may execute on an entirely separate computing device. Because the virtual warehouse manager application 302 may perform steps above and beyond conventional virtual warehouse functionality, the application may execute on an entirely separate computing device and may interface with preexisting virtual warehouse systems, e.g., Snowflake.
  • the selected virtual warehouse may execute the query requested by the request application 301. As shown in FIG. 3, this entails querying both the data warehouse A 304a and the data warehouse B 304b.
  • the data warehouses 120 such as the data warehouse A 304a and the data warehouse B 304b, need not be the same: for example, the data warehouse A 304a may have an entirely different format, may have entirely different schedules which affect their size at any given time, and may have an entirely different structure as compared to the data warehouse B 304b.
  • the data warehouse A 304a may comprise a SQL database
  • the data warehouse B 304b may comprise a file server which stores files according to the File Allocation Table (FAT) file system.
  • the virtual warehouse C 303c may receive, store, and/or organize results from the data warehouses 120.
  • the virtual warehouse C 303c may receive query results from the data warehouse A 304a and the data warehouse B 304b, may store those results in memory, and then may encrypt those results for security purposes.
  • step 305e the virtual warehouse C 303c provides the collected results to the virtual warehouse manager application 302. Then, as part of step 305f, the virtual warehouse manager application 302 provides the results to one or more of the computing devices 110.
  • step 305f the virtual warehouse manager application 302 provides the results to one or more of the computing devices 110.
  • This process is optional, as the virtual warehouse C 303c may, in some instances, provide the results directly to one or more of the computing devices 110.
  • the results need not be provided back to the request application 301 : for example, the results may be provided to an entirely different computing device (e.g., such that the request may have been received from a smartphone but the results may be delivered to an associated laptop) and/or may be provided to an entirely different application (e.g., such that the request may have been received via the request application 301, but the results may be received by a separate application, such as a spreadsheet application, executing on one or more of the computing devices 110).
  • a separate application such as a spreadsheet application, executing on one or more of the computing devices 110.
  • step 305a is reflected as an arrow directly leading from the request application 301 to one or more of the virtual warehouse servers 130, the request may in fact be routed through various other computing devices as part of the network 140.
  • the query process reflected in step 305c and step 305d may involve a plurality of different transmissions between the virtual warehouse C 303 c and the data warehouses 120.
  • one or more rules might be used to generate a limited consumer view definition which limits the ability of that consumer to access the data.
  • a computing device might perform a data access certification process, by which a shared view definition of the data (which may display all data) is used to generate a table of certification results by processing the data and determine which record(s) of the data comply with the one or more rules.
  • FIG. 4 depicts a flowchart with steps which may be performed by a computing device, such as one or more of the computing devices 110, the virtual warehouse servers 130, and/or the data warehouses 120.
  • a computing device such as one or more of the computing devices 110, the virtual warehouse servers 130, and/or the data warehouses 120.
  • One or more non-transitory computer-readable media may store instructions that, when executed by one or more processors of a computing device, cause performance of one or more of the steps of FIG. 4.
  • the steps depicted in FIG. 4 may operate on a Snowflake environment or other virtual warehouse environment, such that they may be performed by a computing device within or external to such an environment.
  • the steps depicted in FIG. 4 may be performed on a user device external to the cloud database platform.
  • a computing device may determine a shared view definition.
  • the computing device may determine a shared view definition for access to the database stored on the database server.
  • a shared view definition may be configured to provide access to all records stored by the database.
  • the shared view definition might be a default view with which an owner of data can access their own data. In this manner, records of the data might not be excluded when viewed via the shared view definition.
  • the shared view definition may be additionally and/or alternatively configured to enable execution of queries against the database using processing resources of one or more virtual warehouses provided by the cloud database platform. For example, a user might, via the shared view definition, use the virtual warehouse 303a to query one or more of the databases 120.
  • the computing device may determine one or more rules.
  • a rule may specify criteria which relates to limits to a consumer’s access to a database.
  • a computing device may determine these rules to determine limits on consumer access to records stored by a database.
  • the computing device may determine one or more first rules that specify criteria, associated with consumer permissions to access the database via the cloud database platform, that limit consumer access to the records stored by the database.
  • the one or more first rules might prevent output of invalid values.
  • Some consumers of data might want to receive (e.g., view) only data which is valid and/or reliable.
  • “NaN” (not a number) values might be excluded if such values are included in fields expected to have numbers.
  • the one or more first rules might prevent output of values outside of a predefined range.
  • some consumers of data might want to receive (e.g., view) only data which is valid and/or reliable. For example, data indicating a birthdate after the current day might be excluded because such data is almost certainly inaccurate (or, at least, speculative).
  • data indicating a birthdate after the current day might be excluded because such data is almost certainly inaccurate (or, at least, speculative).
  • values under zero or over one hundred and fifty might be excluded.
  • the one or more first rules might prevent output of values that do not match a regular expression pattern. Certain data might be in a predefined format such that values not comporting with that format might be considered invalid. For example, for a column corresponding to a date and time, values that do not match conventional date/time formats might be excluded. As another example, because credit card numbers are conventionally sixteen digits, at least one of the one or more first rules might specify that values in a column corresponding to credit card numbers that are not sixteen digits should be excluded.
  • the one or more first rules may prevent output of data that was not created, modified, or otherwise processed within a time period. Certain consumers might want to gain access to data within a certain time period. As such, for example, the one or more first rules may prevent output of data outside a time period specified by the at least one rule (e.g., older than three years).
  • At least one of the one or more first rules might cause generation of an alert.
  • Rules may limit consumers’ access to data to an undesirably extreme degree, to the point where the data received by users might be unusable and/or undesirably fragmented.
  • alerts might be generated in circumstances where insufficient quantities of data are output to a consumer.
  • at least one of the one or more first rules may be configured to cause output of an alert based on a determination that more than a predetermined percentage of the records is not output based on the one or more first rules.
  • the creator of the one or more rules might be alerted that the rules might be over-restricting consumer access to data.
  • Such an alert might, for example, prompt the creator of the one or more rules to modify the rules.
  • the computing device may perform a data access certification process based on the one or more rules to generate a table of certification results.
  • the data access certification process may be configured to generate the table of certification results so as to indicate whether data complies with the one or more first rules.
  • the computing device may perform, based on the one or more first rules, a data access certification process on the records stored by the database to generate a table of certification results.
  • the computing device may access all records stored by the database using the shared view definition.
  • the computing device might retrieve, from a database, all records of data stored by the database.
  • the computing device may use the shared view definition, which may provide access to all records of data stored by the database.
  • the computing device may generate a data certification result for each record based on determining, for each record, whether a given record satisfies the criteria of the one or more first rules based on one or more fields of the given record.
  • the computing device might iterate through all records stored in the data stored by the database to determine, for each record, compliance with the criteria of the one or more first rules.
  • This process might correspond to each record (e.g., each row of the database) as well as each field (e.g., each column of the database). For example, a record might be considered to not satisfy the criteria of the one or more first rules because a field in the record does not satisfy the one or more first rules.
  • the computing device may generate, based on the data certification result for each record, the table of certification results that indicates, for each record, whether the record satisfies the criteria of the one or more first rules.
  • the table of certification results may be in some ways similar to the data stored by the database itself, such that the table of certification results indicates, for each field of each record of the data, whether the field satisfied the one or more first rules.
  • the table of certification results might comprise a plurality of Boolean values for each field in the data, with each indicating whether or not a particular field satisfied the one or more rules.
  • Generating the table of certification results may comprise adding, to the table of certification results and based on at least one of the one or more first rules, an indication that a first value in a first field should be replaced with a second value.
  • a value in a field of a record of the data might satisfy a rule if it is replaced with a second value.
  • Such replacement might comprise truncating and/or otherwise modifying the original data. For example, a sixteen-digit credit card number might be replaced with only the last four digits of the sixteen-digit credit card.
  • Such replacement might comprise entirely replacing the original data with a summary of the data. For example, a specific state (e.g., “Maine”) might be replaced with a generic identifier of a region (e.g., “Northeast”).
  • the computing device may generate a limited consumer view definition based on the table of certification results.
  • the limited consumer view definition may be a view definition configured to limit access to the data based on the table of certification results. That limited consumer view definition might be configured to prevent access to portions of the data that did not satisfy the one or more rules.
  • the computing device may generate, based on an intersection of the table of certification results and the shared view definition, and without modifying the records stored by the database, a limited consumer view definition configured to provide access to a first portion of the records in compliance with the criteria of the one or more first rules and exclude a second portion of the records not in compliance with the criteria of the one or more first rules without modifying the records stored by the database.
  • An intersection of the table of certification results and the shared view definition may be generated using an algorithm that returns common data sets.
  • the shared view definition and the table of certification results may each comprise a plurality of rows, and the intersection of those two objects may comprise common rows shared between the two objects. For example, assume the shared view definition comprises rows A, B, and C, whereas the table of certification results comprises rows B and C. In such a circumstance, the intersection of the table of certification results and the shared view definition may comprise rows B and C, but not row A.
  • the computing device may implement a limited consumer view definition.
  • Implementing the limited consumer view definition may comprise providing the limited consumer view definition to one or more consumers.
  • the computing device may cause a first virtual warehouse, of the one or more virtual warehouses, to execute a query on the first portion of the records in compliance with the criteria of the one or more first rules via the limited consumer view definition.
  • the computing device may cause output of a result of the query to a consumer authorized to access the database through the limited customer view definition.
  • Step 406 through step 409 describe a process whereby the computing device might respond to rules being updated.
  • one advantage of the present disclosure is that it allows the rapid implementation of rule modifications without requiring that the underlying data be modified.
  • the computing device may detect whether a rule has been updated. For example, the computing device may receive an update to the one or more first rules.
  • a rule update might be any change to the one or more rules, such as the addition of a new rule, the removal of a rule, the modification of an existing rule, or the like. For example, a rule requiring that data be younger than one year old might be replaced with a rule requiring that data be younger than six months old. If an update to a rule is detected, the flow chart proceeds to step 407. Otherwise, the computing device may continue to wait, and/or the flow chart may end.
  • the computing device may perform the data access certification process based on the updated rules to generate a second and/or updated table of certification results.
  • the data access certification process might be the same or similar as that discussed with respect to step 403 of FIG. 4.
  • the computing device may perform, based on the update to the one or more first rules, the data access certification process to update the table of certification results generated as part of step 403, and/or may generate a second table of certification results.
  • the second table of certification results might be similar to the table of certification results and based on at least one of the one or more first rules generated in step 403, albeit with differences based on the update to the rule detected in step 406.
  • the decisions as to whether to generate a new table of certification results versus update an existing table of certification results may be based on a variety of different factors. For example, a new (second) table of certification results might be generated if, for example, an existing table of certification results is preserved for logging, security, and/or if the existing table of certification results is used for a variety of different customers. As another example, an existing table of certification results may be generated in circumstances where, for example, each customer has their own table of certification results.
  • step 408 the computing device may generate a updated limited consumer view definition based on the second table of certification results. For example, the computing device may generate, based on the second table of certification results, a updated limited consumer view definition.
  • This step may be the same or similar as step 404, albeit in view of the second table of certification results generated in step 407.
  • step 409 the computing device may implement the second limited consumer view definition. This step may be the same or similar as step 405, albeit using the updated limited consumer view definition generated in step 408.
  • the computing device may be configured to detect updates to data and react by generating a updated limited consumer view definition.
  • the computing device may detect an update to at least one record.
  • the computing device may then perform the data access certification process on the update generate a second table of certification results.
  • the computing device may then generate, based on the table of certification results and the second table of certification results, a updated limited consumer view definition. In this manner, updates to the data might be reflected in the updated limited consumer view definition. This advantageously may ensure that consumers of data have the latest version of the data.
  • the consumer when a data producer shares data with a consumer, the consumer may be onboarded in a manner which generates a limited consumer view definition governing their access to that data. For example, as part of sharing data with a new consumer, a creator of that data might be prompted to generate rules for that consumer. In this manner, the consumer might be provided the data which they asked/paid for (e.g., and no more than the data which they are entitled to).
  • FIG. 5 depicts a flowchart with steps which may be performed by a computing device, such as one or more of the computing devices 110, the virtual warehouse servers 130, and/or the data warehouses 120.
  • a computing device such as one or more of the computing devices 110, the virtual warehouse servers 130, and/or the data warehouses 120.
  • One or more non-transitory computer-readable media may store instructions that, when executed by one or more processors of a computing device, cause performance of one or more of the steps of FIG. 5.
  • the steps depicted in FIG. 5 may operate on a Snowflake environment or other virtual warehouse environment, such that they may be performed by a computing device within or external to such an environment.
  • the steps depicted in FIG. 5 may be performed on a user device external to the cloud database platform.
  • the computing device may determine a shared view definition. This step may be the same or similar as step 401 of FIG. 4. For example, the computing device may determine a shared view definition for access to the database stored on the database server. As was the case with step 401, the shared view definition may be configured to provide access to all records stored by the database and to enable execution of queries against the database using processing resources of one or more virtual warehouses provided by the cloud database platform.
  • the computing device may determine one or more attributes of the database.
  • An attribute of the database may be any aspect describing the database, including a size of the database, names of columns of the database, information about data stored by the database, or the like.
  • the attributes might indicate what kind of data is stored by the database. For example, the attributes might indicate that the database stores demographic information for voters in the United States, organized by city and state. As another example, the attributes might comprise a list of columns of a table as well as an age of each record (e.g., each row) of the table.
  • the computing device may, based on the attributes, prompt the user for rules.
  • the computing device might provide a user the opportunity to create rules which limit access to data stored by the database.
  • the computing device may provide, to a user device and based on the one or more attributes of the database, a user interface enabling creation of rules that specify criteria, associated with consumer permissions to access the database via the cloud database platform, that limit consumer access to the records stored by the database.
  • the computing device may determine that the database comprises a table with a column comprising values that indicate the last name of a user.
  • the computing device may permit a user to specify one or more rule(s) which might limit access to that last name.
  • the user might be allowed to make a rule obfuscating the last name (e.g., replacing the last name with a blank value, “LASTNAME,” or the like), removing the last name entirely, permitting the last name to be included only if it satisfies a regular expression, or the like.
  • the prompt to the user for rules might entail use of a trained machine learning model, such as may be implemented via the deep neural network 700 of FIG. 7 (discussed below). Because of the complexity and volume of data that might be stored by the database, it may be valuable to help users think of different rules that might be implemented with respect to a database. To perform this task, a machine learning model might be trained to recommend rules. For example, the computing device may train a machine learning model to output recommended rules using training data that comprises a history of rules, applied to different databases, created by one or more users. In this manner, the machine learning model might be trained using information about what other kind of rules users have applied.
  • the computing device may then provide, as input to the trained machine learning model, account metadata associated with the user and retrieve, as output from the trained machine learning model, one or more recommended rules.
  • the account metadata might indicate, for example, an identity of the user (e.g., whether the user is a data owner, data consumer, what sort of limitations the user might have in terms of data they can access), and the like.
  • the computing device may then cause display, in the user interface, of an indication of the one or more recommended rules. In this manner, the user might then select recommended rules with ease.
  • the computing device may generate the rules. For example, the computing device may generate, based on criteria received via the user interface, one or more first rules that limit the output of the data. This process might be the same or similar as step 402 of FIG. 4, except that at least some of the rules might be generated via the prompt in step 503.
  • the user interface might enable a user to select criteria, and a rule might be generated based on that criteria. In this manner, while some of the rules might be received via the prompt in step 503, others might be received from external sources.
  • the rules might be generated based on input (e.g., criteria specified via) the user interface. For example, at least one rule might be generated based on user input, via the user interface, specifying a range of values. As another example, at least one rule might be generated based on user input, via the user interface, specifying a category of values.
  • All or portions of the rules might originate from contract data. Consumers of data may be contractually and/or legally limited with respect to the data they may access. For example, for privacy reasons, a contract might provide that a consumer cannot access personally identifying information associated with users, but might instead provide that the consumer could access obfuscated versions of that data. In such a circumstance, one or more rules might exist that obfuscate personally identifying data (e.g., social security numbers, last names, addresses) by removing and/or replacing that data (e.g., removing the social security numbers, replacing all last names with “LASTNAME,” and replacing the addresses with generic indication of geographic location). For example, the computing device may identify contract data associated with a contract for access to the database.
  • personally identifying data e.g., social security numbers, last names, addresses
  • the computing device may identify contract data associated with a contract for access to the database.
  • the computing device may identify that contract data by receiving, via a user interface, the contract data. For example, a user might upload text data corresponding to the contract via the user interface.
  • the contract data may be stored in a database associated with a user.
  • the contract data might comprise, for example, text from a contract, a listing of rules from a contract, or the like.
  • the computing device may then generate, based on processing the contract data, at least one of the one or more first rules. For example, based on using Natural Language Processing (NLP) algorithms on text corresponding to a contract, the computing device might determine the one or more first rules.
  • NLP Natural Language Processing
  • the rules might be added to a catalog.
  • a catalog of user access to data stored by the cloud database platform may be maintained. That catalog might be used to record information about access to data, including one or more rules governing access to that data.
  • the computing device may update a catalog that stores information about access, by one or more users, to databases by adding, to the catalog, an indication of the one or more first rules.
  • the computing device may perform a data access certification process based on the rules to generate a table of certification results. This process may be the same or similar as step 403 of FIG. 4.
  • step 506 the computing device may generate a limited consumer view definition based on the table of certification results. This process may be the same or similar as step 404 of FIG. 4.
  • step 507 the computing device may implement the limited consumer view definition. This process may be the same or similar as step 405 of FIG. 4.
  • updates to the data stored by a database may occur over time. For example, records might be added, modified, or removed. In such circumstances, updates to the data might cause generation of updated limited consumer view definitions which reflect the changes to the data. In this way, consumers of that data receive the latest form of that data, albeit limited by the rules governing their access to that data. For example, based on a new record being added to data, a updated limited consumer view definition might be generated such that the updated limited consumer view definition indicates whether the new record satisfies one or more rules.
  • FIG. 6 depicts a flowchart with steps which may be performed by a computing device, such as one or more of the computing devices 110, the virtual warehouse servers 130, and/or the data warehouses 120.
  • a computing device such as one or more of the computing devices 110, the virtual warehouse servers 130, and/or the data warehouses 120.
  • One or more non-transitory computer-readable media may store instructions that, when executed by one or more processors of a computing device, cause performance of one or more of the steps of FIG. 6.
  • the steps depicted in FIG. 6 may operate on a Snowflake environment or other virtual warehouse environment, such that they may be performed by a computing device within or external to such an environment.
  • the steps depicted in FIG. 6 may be performed on a user device external to the cloud database platform.
  • a computing device may determine a shared view definition. For example, the computing device may determine a shared view definition for access to the database stored on the database server. As was discussed in FIG. 4, the shared view definition may be configured to provide access to all records stored by the database and to enable execution of queries against the database using processing resources of one or more virtual warehouses provided by the cloud database platform. This step may be the same or similar as step 401 of FIG. 4.
  • the computing device may determine one or more rules. For example, the computing device may determine one or more first rules that specify criteria, associated with consumer permissions to access the database via the cloud database platform, that limit consumer access to the records stored by the database. This step may be the same or similar as step 402 of FIG. 4.
  • the computing device may perform a data certification process based on the one or more rules to generate a table of certification results. For example, the computing device may perform, based on the one or more first rules, a data access certification process on the records stored by the database to generate a first table of certification results.
  • This step may be the same or similar as step 403 of FIG. 4.
  • the computing device may generate a limited consumer view definition based on the table of certification results. For example, the computing device may generate, based on an intersection of the first table of certification results and the shared view definition, a limited consumer view definition configured to provide access to a first portion of the records in compliance with the criteria of the one or more first rules and exclude a second portion of the records not in compliance with the criteria of the one or more first rules without modifying the records stored by the database.
  • This step may be the same or similar as step 404 of FIG. 4.
  • the computing device may implement the limited consumer view definition.
  • the computing device may cause a first virtual warehouse, of the one or more virtual warehouses, to execute a query on the records in compliance with the criteria of the one or more first rules via the limited consumer view definition.
  • This step may be the same or similar as step 405 of FIG. 4.
  • the computing device may determine whether it has detected an update to the data. For example, the computing device may detect, via the shared view definition, an update to at least one record of the records stored by the database.
  • An update to the data may be any addition to, modification of, and/or removal of any portion of the data. For example, a record e.g., a row of a table) may be added, removed, and/or one or more fields of the record may be modified.
  • Detecting the update to the data may comprise receiving, via a cloud database platform, an indication of an update to data.
  • the update may be detected using existing Snowflake functionality, such as the Streams functionality of Snowflake.
  • the computing device may perform the data certification process on the update to generate a second table of certification results.
  • the computing device may perform the data access certification process on the updated records to generate a second table of certification results.
  • the second table of certification results may indicate whether the update satisfies the one or more rules.
  • both the original table of certification results and the second table of certification results may be used to generate a second limited consumer view definition (as discussed below).
  • the new limited customer view definition (discussed below) might be generated based on a comparison of a first table of certification results and a second table of certification results.
  • the second table of certification results may indicate whether all data, including the update detected in step 606, satisfies the one or more rules.
  • the second limited consumer view definition might be generated (as discussed below) based on the second table of certification results, and the previous table of certification results might be discarded if desired.
  • step 608 the computing device may generate a second limited consumer view definition based on the second table of certification results.
  • a updated limited consumer view definition might be generated in view of the update detected in step 606.
  • the computing device may generate, based on the first table of certification results, the second table of certification results, and the shared view definition, a updated limited consumer view definition different from the limited consumer view definition.
  • This step may be the same or similar as step 404 of FIG. 4, albeit in view of the second table of certification results.
  • the computing device may implement the second limited consumer view definition.
  • the computing device may cause a first virtual warehouse, of the one or more virtual warehouses, to execute a query on the first portion of the records in compliance with the criteria of the one or more first rules via the updated limited consumer view definition.
  • This step may be the same or similar as step 405 of FIG. 4.
  • FIG. 7 depicts an example deep neural network architecture 700.
  • This architecture is one example of how a machine learning model might be implemented in order to, for example, prompt to the user for rules in accordance with step 503 of FIG. 5.
  • the architecture depicted in FIG. 7 need not be performed on a single computing device, and may be performed by, e.g., a plurality of computers (e.g., any one of the devices depicted in FIG. 1).
  • An artificial neural network may be a collection of connected nodes, with the nodes and connections each having assigned weights used to generate predictions. Each node in the artificial neural network may receive input and generate an output signal. The output of a node in the artificial neural network may be a function of its inputs and the weights associated with the edges.
  • the trained model may be provided with input beyond the training set and used to generate predictions regarding the likely results.
  • Artificial neural networks may have many applications, including object classification, image recognition, speech recognition, natural language processing, text recognition, regression analysis, behavior modeling, and others.
  • An artificial neural network may have an input layer 710, one or more hidden layers 720, and an output layer 730.
  • a deep neural network as used herein, may be an artificial network that has more than one hidden layer. Illustrated network architecture 700 is depicted with three hidden layers, and thus may be considered a deep neural network. The number of hidden layers employed in deep neural network 700 may vary based on the particular application and/or problem domain. For example, a network model used for image recognition may have a different number of hidden layers than a network used for speech recognition. Similarly, the number of input and/or output nodes may vary based on the application. Many types of deep neural networks are used in practice, such as convolutional neural networks, recurrent neural networks, feed forward neural networks, combinations thereof, and others.
  • the weights of each connection and/or node may be adjusted in a learning process as the model adapts to generate more accurate predictions on a training set.
  • the weights assigned to each connection and/or node may be referred to as the model parameters.
  • the model may be initialized with a random or white noise set of initial model parameters.
  • the model parameters may then be iteratively adjusted using, for example, stochastic gradient descent algorithms that seek to minimize errors in the model.
  • FIG. 8 depicts examples of applications of a shared view definition 800, a first limited consumer view definition 803a, and a second limited consumer view definition 803b with respect to each illustrative data. Each of these views are depicted as executed on an illustrative set of data. Particularly, as shown in FIG. 8, the shared view definition 800 shows all data from a database, whereas the first limited consumer view definition 803a and the second limited consumer view definition 803b show different subsets of the data based on the application of one or more rules.
  • the shared view definition 800, first limited consumer view definition 803a, and/or the second limited consumer view definition 803b may be represented in data in entirely different formats.
  • the shared view definition 800 might be represented as a query that causes a database to return all records of data, whereas the first limited consumer view definition might be represented in data as a listing of fields of records to be replaced and/or otherwise obfuscated.
  • the shared view definition 800 is shown outputting three columns: a name column 801a, a credit card number column 801b, and a location column 801c.
  • the shared view definition 800 is shown outputting three rows: a first record 802a, a second record 802b, and a third record 802c.
  • the shared view definition 800 in this example is showing all fields of all records of data, such that no data is shown as obfuscated and/or otherwise modified.
  • the first limited consumer view definition 803a is shown outputting the same columns (the name column 801a, the credit card number column 801b, and the location column 801c), except that fields in all three records (e.g., the first record 802a, the second record 802b, and the third record 802c) have been modified according to one or more rules.
  • the last names in the name column 801a have been replaced with “LASTNAME,” the credit card numbers in the credit card number column 801b, have been truncated to their last four digits, and the locations in the location column 801c have been modified to reflect a general region of the United States.
  • the second limited consumer view definition 803b is shown outputting the same columns (the name column 801a, the credit card number column 801b, and the location column 801c), except that an entire column (the credit card number column 801b) has been removed. This may reflect a rule that prevents access, by a consumer, to credit card numbers. As was the case with the first limited consumer view definition 803a, the underlying data stored by the database need not change: rather, all that has changed is that a consumer is prevented from accessing the column.
  • FIG. 9 depicts an illustrative user interface 900, such as might be generated as part of step 503 of FIG. 5.
  • the user interface 900 depicts various options which might be presented to a user as part of making rules for that specify criteria, associated with consumer permissions to access the database via the cloud database platform, that limit consumer access to the records stored by the database.
  • the user interface 900 shows that a user might be able to select options relating to what sort of geographic regions data will be shown from (e.g., New York City, Boston, Virginia), whether credit cards will be shown in whole or in part, and whether last names should be obfuscated. Based on user selections in this user interface 900, one or more rules may be generated.
  • FIG. 10 depicts an illustrative listing of rules 1000.
  • the rules 1000 may include a first rule 1001a indicating that no invalid values should be output. For example, this rule might prevent output of “NaN” (not a number) values in fields expected to contain numbers.
  • the rules 1000 may include a second rule 1001b indicating that no birthdates after a current date should be displayed. This rule may, in practice, prevent output of potentially invalid (or, at least, speculative) data.
  • the rules 1000 may include a third rule 1001c indicating that no non- 16-digit credit card numbers should be displayed. Such a rule may, as with the previous rule, prevent output of potentially invalid data.
  • the rules 1000 may include a fourth rule 100 Id indicating that no data older than 3 years old should be displayed. This may, in practice, prevent output of particularly old data.
  • FIG. 11 depicts an illustrative architecture for a data certification process discussed in FIG 4, FIG. 5, and FIG. 6.
  • the architecture depicted in FIG. 11 might be implemented through one or more computing devices, such as those depicted in any one or more of FIG. 1, FIG. 2, and/or FIG. 3.
  • One or more aspects of the architecture may be implemented through one or more non-transitory computer-readable memory storing instructions that, when executed by one or more processors, cause a computing device to perform one or more steps, such as the steps depicted in any one of FIG. 4, FIG. 5, and/or FIG. 6.
  • a cloud database platform 306, such as the Snowflake platform, may comprise one or more data producer accounts, such as the producer account 1104.
  • the producer account 1104 may manage (e.g., create, edit) one or more databases, such as databases 1105.
  • the databases 1105 may be the same or similar as the databases 120 of FIG. 3.
  • the databases 1105 may be shared, as shared databases 1106, via a data sharing platform 307.
  • the data sharing platform 307 may be, for example, a multi-tenant virtual private server.
  • the data sharing platform 307 need not be managed by the same entity as the cloud database platform 306.
  • the cloud database platform 306 might be managed by a first entity, and the data sharing platform 307 may be managed by a different entity.
  • a table of certification results 1107 may be generated via the data sharing platform 307 in accordance with, e.g., step 403 of FIG. 4, step 505 of FIG. 5, and/or step 603 of FIG. 6. Then, a limited customer view definition 1108 may be generated via the data sharing platform 307.
  • the data sharing platform 307 may be used to implement the sharing of data.
  • a data producer may be willing to provide (e.g., for a fee) access to its data on the cloud database platform 306.
  • the cloud database platform 306 and/or the data sharing platform 307 may thereby implement a data sharing marketplace, whereby producers and consumers of data may exchange data as desired.
  • each producer and/or consumer of data may maintain their own producer account(s) (e.g., the producer account 1104) and their own database(s) (e.g., the databases 1105), and may provide access to those databases as desired via the marketplace.
  • a data producer associated with the producer account 1104 agrees to share their database(s) with a consumer.
  • the data producer may, via the cloud database platform 306 and/or the data sharing platform 307, cause the database(s) to be shared as the shared database(s) 1106.
  • this process need not entail changes to the data: as indicated above, the zero copy functionality of Snowflake would permit such sharing without the need for data to be copied.
  • the data producer, the consumer, and/or other entities may specify one or more rules, which may be used (as described above in, e.g., FIG. 4, FIG. 5, and/or FIG. 6) to generate the table of certification results 1107.
  • the limited customer view definition 1108 may then be used to limit access, by the consumer, to the databases 1105 shared as the shared databases 1106.
  • One or more aspects discussed herein may be embodied in computer-usable or readable data and/or computer-executable instructions, such as in one or more program modules, executed by one or more computers or other devices as described herein.
  • program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types when executed by a processor in a computer or other device.
  • the modules may be written in a source code programming language that is subsequently compiled for execution, or may be written in a scripting language such as (but not limited to) HTML or XML.
  • the computer executable instructions may be stored on a computer readable medium such as a hard disk, optical disk, removable storage media, solid- state memory, RAM, and the like.
  • the functionality of the program modules may be combined or distributed as desired in various embodiments.
  • the functionality may be embodied in whole or in part in firmware or hardware equivalents such as integrated circuits, field programmable gate arrays (FPGA), and the like.
  • Particular data structures may be used to more effectively implement one or more aspects discussed herein, and such data structures are contemplated within the scope of computer executable instructions and computer-usable data described herein.
  • Various aspects discussed herein may be embodied as a method, a computing device, a system, and/or a computer program product.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Databases & Information Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

Sont décrits dans la description, des procédés, des systèmes et des appareils pour fournir un accès à des dossiers d'une base de données stockée sur un serveur de base de données dans une plateforme de base de données en nuage. Une plateforme de partage de données peut déterminer une définition de visualisation partagée pour un accès à la base de données. La plateforme de partage de données peut déterminer des règles qui spécifient des critères qui limitent l'accès aux dossiers stockés par la base de données. La ou les premières règles peuvent être reçues par l'intermédiaire d'une interface utilisateur. La plateforme de partage de données peut effectuer, sur la base des règles, un processus de certification d'accès aux données sur les dossiers stockés par la base de données pour générer une table de résultats de certification. La plateforme de partage de données peut générer, sur la base de la table de résultats de certification, et sans modifier les dossiers stockés par la base de données, une définition de visualisation de consommateur limitée. Sur la base de mises à jour des dossiers, une nouvelle définition de visualisation de consommateur limitée peut être générée.
PCT/US2022/052622 2021-12-14 2022-12-13 Processus de certification de données pour plateforme de base de données en nuage WO2023114164A1 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US17/550,040 2021-12-14
US17/550,032 US11392715B1 (en) 2021-12-14 2021-12-14 Data certification process for cloud database platform
US17/550,032 2021-12-14
US17/550,036 2021-12-14
US17/550,036 US11392587B1 (en) 2021-12-14 2021-12-14 Rule generation and data certification onboarding process for cloud database platform
US17/550,040 US11829367B2 (en) 2021-12-14 2021-12-14 Data certification process for updates to data in cloud database platform

Publications (1)

Publication Number Publication Date
WO2023114164A1 true WO2023114164A1 (fr) 2023-06-22

Family

ID=85222444

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2022/052622 WO2023114164A1 (fr) 2021-12-14 2022-12-13 Processus de certification de données pour plateforme de base de données en nuage

Country Status (1)

Country Link
WO (1) WO2023114164A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0991005A2 (fr) * 1998-10-02 2000-04-05 Ncr International Inc. Base de données avec une confidentialité élevée
US9141659B1 (en) * 2014-09-25 2015-09-22 State Farm Mutual Automobile Insurance Company Systems and methods for scrubbing confidential insurance account data
US20210117567A1 (en) * 2019-10-21 2021-04-22 International Business Machines Corporation Preventing leakage of selected information in public channels
US20210141920A1 (en) * 2019-11-08 2021-05-13 Okera, Inc. Dynamic view for implementing data access control policies

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0991005A2 (fr) * 1998-10-02 2000-04-05 Ncr International Inc. Base de données avec une confidentialité élevée
US9141659B1 (en) * 2014-09-25 2015-09-22 State Farm Mutual Automobile Insurance Company Systems and methods for scrubbing confidential insurance account data
US20210117567A1 (en) * 2019-10-21 2021-04-22 International Business Machines Corporation Preventing leakage of selected information in public channels
US20210141920A1 (en) * 2019-11-08 2021-05-13 Okera, Inc. Dynamic view for implementing data access control policies

Similar Documents

Publication Publication Date Title
US11036771B2 (en) Data processing systems for generating and populating a data inventory
US11240273B2 (en) Data processing and scanning systems for generating and populating a data inventory
US11138336B2 (en) Data processing systems for generating and populating a data inventory
US11921894B2 (en) Data processing systems for generating and populating a data inventory for processing data access requests
US10438016B2 (en) Data processing systems for generating and populating a data inventory
US10282370B1 (en) Data processing systems for generating and populating a data inventory
US11966489B2 (en) Data certification process for cloud database platform
US10970675B2 (en) Data processing systems for generating and populating a data inventory
US11392587B1 (en) Rule generation and data certification onboarding process for cloud database platform
US11829367B2 (en) Data certification process for updates to data in cloud database platform
US20240232421A1 (en) Data Certification Process for Cloud Database Platform
WO2023114164A1 (fr) Processus de certification de données pour plateforme de base de données en nuage
US20230401181A1 (en) Data Management Ecosystem for Databases
WO2019023511A1 (fr) Systèmes de traitement de données permettant de générer et de remplir un stock de données

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22856881

Country of ref document: EP

Kind code of ref document: A1