WO2023173908A1

WO2023173908A1 - Method, apparatus and system for accessing file, and storage medium

Info

Publication number: WO2023173908A1
Application number: PCT/CN2023/070167
Authority: WO
Inventors: 黄爽; 许田立
Original assignee: 华为云计算技术有限公司
Priority date: 2022-03-17
Filing date: 2023-01-03
Publication date: 2023-09-21

Abstract

A method, apparatus and system for accessing a file, and a storage medium. The method comprises: receiving a data access request, wherein the data access request comprises access demand information, the access demand information is used for indicating content in a first file that a first user needs to access, and the first file is stored in an object file storage system; when it is determined, on the basis of the access demand information, that the granularity of the content in the first file to be accessed is a first granularity, accessing the first file on the basis of account information of the first user and the access demand information; and when it is determined, on the basis of the access demand information, that the granularity of the content in the first file to be accessed is a second granularity, accessing the first file on the basis of specified administrator account information and the access demand information, wherein the second granularity is less than the first granularity.

Description

Methods, devices, systems and storage media for accessing files

This application claims priority to the Chinese patent application with application number 202210264898.8 and the invention title "An efficient fine-grained data lake authority management solution" submitted on March 17, 2022, the entire content of which is incorporated herein by reference. Applying. And, this application also claims priority to the Chinese patent application with application number 202210511098.1 and the invention title "Method, device, system and storage medium for accessing files" submitted on May 11, 2022, the entire content of which is incorporated by reference. in this application.

Technical field

The present application relates to the field of computers, and in particular to a method, device, system and storage medium for accessing files.

Background technique

For a data lake that separates storage and computing, the data lake includes a structured query language (SQL) engine and an object file storage system. The object file storage system includes at least one file, each file is used to store data. The SQL engine receives a SQL statement from the user, and the SQL statement is used to indicate to the SQL engine the file that the user needs to access. The SQL engine accesses the file in the object file storage system based on the SQL statement.

Usually the files in the object file storage system are structured data files, which store data in list form. For example, the object file storage system has a file used to save an employee data table. The file includes four columns. The first column is used to store the employee's name, the second column is used to store the employee's address, and the third column is used to store the employee's department. The fourth column is used to store the employee's position. Each row of the file stores the employee's name, address, department, and position.

Currently, users can use the SQL engine to access the entire file located in the object file storage system. That is to say, the access granularity provided by the SQL engine to the user is the entire file. The SQL engine provides file-granular access services to users, but the access services provided to users are too single.

Contents of the invention

This application provides a method, device, system and storage medium for accessing files to enrich the access services provided to users. The technical solutions are as follows:

In the first aspect, embodiments of the present application provide a method for accessing files. In the method, a data access request is received. The data access request includes access requirement information. The access requirement information is used to indicate that the first user needs to access. The content of the first file is stored in the object file storage system. When it is determined based on the access requirement information that the granularity of accessing the content in the first file is the first granularity, the first file is accessed based on the account information of the first user and the access requirement information. When the granularity of accessing the content in the first file is determined to be the second granularity based on the access requirement information, the first file is accessed based on the specified administrator account information and the access requirement information, and the second granularity is smaller than the first granularity.

Wherein, the granularity of accessing the content in the first file is determined based on the access requirement information. When the determined granularity is the first granularity, the first file is accessed based on the account information of the first user and the access requirement information. When the determined granularity is the second granularity, the first file is accessed based on the specified administrator account information and the access requirement information. In this way, the first-granularity access service and the second-granularity access service can be provided to the user, which enriches the access services provided to the user.

Since the first granularity is greater than the second granularity, when the determined granularity is the first granularity, the first file is accessed based on the first user's account information and the access requirement information. In this way, there is no need to borrow the administrator account information to access the first file, which improves the efficiency. The efficiency of accessing the first file and the performance of reading and writing the first file.

If the first user is configured with permission to access the content at the second granularity, the first user can not only access the content in the first file, but also access other content in the first file except the content. This gives the first user The permissions configured by the user are automatically expanded to access any content in the first file, which results in too much permission expansion and is not conducive to permission management. However, in this application, when the determined granularity is the second granularity, the first file is accessed based on the specified administrator account information and the access requirement information, so the administrator account information is used to replace the first user's account information, and Using the administrator account information to access the first file eliminates the need to configure permissions for the first user to access the second granularity, thereby avoiding extending the first user's access permissions and facilitating permission management.

In a possible implementation, the access requirement information includes the identification information of the first file, and the first granularity is the file granularity; or, the access requirement information includes the identification information of the first file and the identification of the partition in the first file. Information, the first granularity is the partition granularity. The first granularity may be file granularity or partition granularity, thereby providing users with file-granular access services, or providing partition-granular access services, thereby enriching the provided access services. In addition, the partition granularity is a newly defined granularity, which means that this application can also provide partition access services.

In one possible implementation, file granularity refers to the need to access all contents of the first file. Partition granularity refers to the need to access the entire contents of a partition in the first file.

In another possible implementation, an authentication request is sent to the file path authentication module. The authentication request includes authentication information, and the authentication information is used to indicate the first user, the file path of the first file, and the first file path. The user accesses the file path. The authentication information is obtained based on the access requirement information and the first user's account information. The authentication request is used to trigger the file path authentication module based on the first permission information and the authentication The information authenticates the first user's permission to access the file path using the access operation. The file path is used to indicate the storage location of the first file. The first permission information is used to indicate the identity and access operation of the user who can access the file path. . Receive an authentication response sent by the file path authentication module after passing the permission authentication. The authentication response includes a temporary credential. The temporary credential, the file path, and the operation type of the access operation are correspondingly stored in the object file storage system. Based on the temporary credentials, the access requirement information and the file path, access the first file.

Since the temporary credential sent by the file path authentication module after passing the permission authentication is received, the temporary credential is used to access the first file in the object file storage system, thereby improving the security of accessing the first file.

In another possible implementation, the first file is a structured data file, the first file uses a list form to store data, the access requirement information includes identification information of the first file and first information, and the first information is used to Indicates at least one column of the first file and/or at least one row of the first file, and the second granularity is row and column granularity. Alternatively, the first file is a semi-structured data file, the first file includes at least one data fragment, the data fragment is used to save data with the same business attributes, and the access requirement information includes the identification information of the first file and the Identification information of one or more data fragments, the second granularity is the data fragment granularity. The second granularity may be row-column granularity or data fragment granularity, thereby providing users with row-column granular access services, or providing data fragment granular access services, thereby enriching the provided access services.

In another possible implementation, an access instruction is sent to the data filtering engine. The access instruction includes the access requirement information. The data filtering engine includes administrator account information. The access instruction is used to trigger the data filtering engine based on the administrator account. Information and the access requirement information, access the first file.

Since the data filtering engine includes administrator account information, the access instruction is sent to the data filtering engine, so that the data filtering engine accesses the first file based on the specified administrator account information and the access requirement information. In this way, the administrator account information is used to replace the account information of the first user and the administrator account information is used to access the first file. There is no need to configure the first user with permissions to access the second granularity, thus avoiding the expansion of the first file. User's access rights to facilitate rights management.

In another possible implementation, the first user's permission to access the content is authenticated based on the second permission information, the first user's account information and the access requirement information, and the second permission information is used to indicate that the first user can The identity and access actions of the user who accessed the content. After the first user's permission to access the content is authenticated, the granularity of accessing the content in the first file is determined based on the access requirement information.

After the first user's permission to access the content is authenticated, the granularity of accessing the content in the first file is determined, and then different methods are used to access the first file based on different granularities, thereby improving the security of accessing the first file.

In another possible implementation, the first permission information is generated based on the second permission information. The first permission information is used to indicate the user identity and access operation of the file path that can access the first file. The file path is used to indicate the third file path. The storage location of a file. Therefore, the first authority information can be automatically generated, which improves the efficiency of obtaining the first authority information and reduces the cost of obtaining the first authority information.

In the second aspect, this application provides an access system, which includes: a computing engine and an object file storage system.

The computing engine is configured to receive a data access request, where the data access request includes access requirement information. The access requirement information is used to indicate the content of the first file that the first user needs to access. The first file is stored in the object file storage system.

The computing engine is also configured to access the first file based on the account information of the first user and the access requirement information when it is determined that the granularity of accessing the content in the first file is the first granularity based on the access requirement information.

The computing engine is also configured to access the first file based on the specified administrator account information and the access requirement information, and the second granularity is smaller than First granularity.

The computing engine determines the granularity of accessing the content in the first file based on the access requirement information. When the determined granularity is the first granularity, the first file is accessed based on the account information of the first user and the access requirement information. When the determined granularity is the second granularity, the first file is accessed based on the specified administrator account information and the access requirement information. In this way, the first-granularity access service and the second-granularity access service can be provided to the user, which enriches the access services provided to the user.

Since the first granularity is greater than the second granularity, when the determined granularity is the first granularity, the computing engine accesses the first file based on the first user's account information and the access requirement information, so that there is no need to borrow the administrator account information to access the first file. , improve the efficiency of accessing the first file and the performance of reading and writing the first file.

If the first user is configured with permission to access the content at the second granularity, the first user can not only access the content in the first file, but also access other content in the first file except the content. This gives the first user The permissions configured by the user are automatically expanded to access any content in the first file, which results in too much permission expansion and is not conducive to permission management. However, in this application, when the determined granularity is the second granularity, the computing engine accesses the first file based on the specified administrator account information and the access requirement information, so that the administrator account information is used to replace the first user's account information. , and use the administrator account information to access the first file, so that there is no need to configure permissions for the first user to access the second granularity, thereby avoiding extending the first user's access permissions and facilitating permission management.

In another possible implementation, the system also includes a file path authentication module,

A computing engine configured to send an authentication request to the file path authentication module, where the authentication request includes authentication information, and the authentication information is used to indicate the first user, the file path of the first file, and the first user's access to the file path. For an access operation, the authentication information is obtained based on the access requirement information and the first user's account information, and the file path is used to indicate the storage location of the first file.

A file path authentication module, configured to authenticate the first user's permission to access the file path using the access operation based on the first permission information and the authentication information. The first permission information is used to indicate users who can access the file path. Identity and access operations, after passing the permission authentication, send an authentication response to the computing engine. The authentication response includes temporary credentials.

The object file storage system is used to store the temporary credentials, the file path, and the operation type of the access operation.

The calculation engine is also used to access the first file based on the temporary credential, the access requirement information and the file path.

Since the computing engine receives the temporary credential sent by the file path authentication module after passing the permission authentication, it uses the temporary credential to access the first file in the object file storage system, thereby improving the security of accessing the first file.

In another possible implementation, the system further includes a data filtering engine, where the data filtering engine includes administrator account information.

The computing engine is configured to send an access instruction to the data filtering engine. The access instruction includes the file path of the first file and the access requirement information. The file path is used to indicate the storage location of the first file.

A data filtering engine is used to access the first file based on the administrator account information, the file path and the access requirement information.

Since the data filtering engine includes administrator account information, the computing engine sends an access instruction to the data filtering engine, and the data filtering engine accesses the first file based on the specified administrator account information and the access requirement information. In this way, the administrator account information is used to replace the account information of the first user and the administrator account information is used to access the first file. There is no need to configure the first user with permissions to access the second granularity, thus avoiding the expansion of the first file. User's access rights to facilitate rights management.

In another possible implementation, the computing engine is also used to authenticate the first user's permission to access the content based on the second permission information, the first user's account information, and the access requirement information. The second permission Information used to indicate the identity and access actions of users who have access to the content. After the first user's permission to access the content is authenticated, the granularity of accessing the content in the first file is determined based on the access requirement information. After the first user's permission to access the content is authenticated, the computing engine determines the granularity of accessing the content in the first file, and then accesses the first file in different ways based on different granularities, thereby improving the security of accessing the first file. sex.

In another possible implementation, the system also includes a linkage authority module,

A linkage permission module, configured to generate first permission information based on the second permission information. The first permission information is used to indicate the user identity and access operation of the file path that can access the first file. The file path is used to indicate the storage of the first file. Location. Therefore, the first authority information can be automatically generated, which improves the efficiency of obtaining the first authority information and reduces the cost of obtaining the first authority information.

In a third aspect, this application provides a device for accessing a file, for performing the method in the first aspect or any possible implementation of the first aspect. Specifically, the apparatus includes a unit for performing the method in the first aspect or any possible implementation of the first aspect.

In a fourth aspect, the present application provides a device for accessing files, where the device includes a processor and a memory. The processor and the memory may be connected through internal connections. The memory is used to store programs, and the processor is used to execute the programs in the memory, so that the device completes the method in the first aspect or any possible implementation of the first aspect.

In a fifth aspect, the present application provides a computer program product. The computer program product includes a computer program stored in a computer-readable storage medium, and the computing program is loaded by a processor to implement the first aspect or the third aspect. On the one hand any possible implementation method.

In a sixth aspect, the present application provides a computer-readable storage medium for storing a computer program, which is loaded by a processor to execute the method of the above-mentioned first aspect or any possible implementation of the first aspect.

In a seventh aspect, this application provides a chip, including a memory and a processor. The memory is used to store computer instructions. The processor is used to call and run the computer instructions from the memory to execute the first aspect or any possible method of the first aspect. Ways to implement it.

Description of the drawings

Figure 1 is a schematic structural diagram of an access system provided by an embodiment of the present application;

Figure 2 is a schematic diagram of a file provided by an embodiment of the present application;

Figure 3 is a schematic structural diagram of another access system provided by an embodiment of the present application;

Figure 4 is a flow chart of a method for accessing files provided by an embodiment of the present application;

Figure 5 is a schematic structural diagram of another access system according to the embodiment of the present application;

Figure 6 is a schematic structural diagram of another access system according to an embodiment of the present application;

Figure 7 is a flow chart of a method for obtaining first permission information provided by an embodiment of the present application;

Figure 8 is a schematic structural diagram of a device for accessing files provided by an embodiment of the present application;

Figure 9 is a schematic structural diagram of another device for accessing files provided by an embodiment of the present application.

Detailed ways

The embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

Referring to Figure 1, an embodiment of the present application provides an access system 100. The access system 100 includes a computing engine 101 and an object file storage system 102. The computing engine 101 communicates with the object file storage system 102.

In some embodiments, the access system 100 is a database system with separate storage and calculation, in which the object file storage system 102 is responsible for data storage, and the computing engine 101 is responsible for data calculation.

In some embodiments, the access system 100 is applied in scenarios such as data lakes where storage and computing are separated, and in the field of big data processing.

The object file storage system 102 is used to store at least one file, and for any file saved in the object file storage system 102, the file is used to store data.

In some embodiments, the file may be a structured data file. The structured data file uses a list form to store data, so the structured data file is a data table. For any column in the file, the column holds data with the same business attributes.

For a structured data file, the file is essentially a data table, and the identification information of the file is the identification information of the data table. For example, the identification information of the file is the file name of the file, that is, the identification information of the file is the table name of the data table.

For example, the file shown in Table 1 below is a structured data file. The file is a data table. The data table includes five columns of data. The data table is used to store company information. The table name of the data table is "Company information" , the file name of this file is also "Company information". This file is a company information table, and the table name and file name are the same.

See Table 1 below, each column of this file is used to save data with the same business attributes. As shown in Table 1 below, the data stored in the first column are all row numbers, that is, the business attributes of each data stored in the first column are row numbers. The data stored in the second column are all company names, that is, the business attributes of each data stored in the second column are company names. The data stored in the third column are all industry names, that is, the business attributes of each data stored in the third column are industry names. The data stored in the fourth column are all cities, that is, the business attributes of each data stored in the fourth column are city names. The data stored in the fifth column are all countries, that is, the business attributes of each data stored in the fifth column are country names.

Table 1: Company information (company information)

In some embodiments, the file is a semi-structured data file, and the file includes at least one data fragment. For any data fragment in the file, the data fragment is used to save data with the same business attributes.

For example, referring to the semi-structured data file shown in Figure 2, the semi-structured data file includes four data fragments, namely a first data fragment, a second data fragment, a third data fragment and a fourth data fragment. The data saved in the first data fragment are all company names. For example, the data saved in the first data fragment include "Company 1", "Company 2", "Company 3", "Company 4", "Company 5" and "Company 6" , that is, the business attribute of each data stored in the first data fragment is the company name. The data saved in the second data fragment are all industry names. For example, the data saved in the second data fragment include "Internet", "Internet", "Communication", "Logistics", "Communication" and "Logistics", that is, the second data fragment The business attribute of each data stored is the industry name. The data saved in the third data fragment are all cities. For example, the data saved in the third data fragment include "City 1", "City 1", "City 2", "City 2", "City 1" and "City 3". That is, the business attributes of each data stored in the third data fragment are city names. The data stored in the fourth data fragment are all countries. For example, the data stored in the fourth data fragment include "Country 1", "Country 2", "Country 1", "Country 1", "Country 1" and "Country 3". That is, the business attributes of each data stored in the fourth data fragment are country names.

In some embodiments, the semi-structured data file is an extensible markup language (XML) file, etc., and the tag blocks in the XML file are data fragments.

In some embodiments, the file may also include at least one partition.

In some embodiments, the file is stored in the object file storage system 102, and the file path of the file is used to indicate the storage location of the file in the object file storage system 102. For example, assume that the file path of the file shown in Table 1 is "C:\windows\system32\Company information". This file path is used to indicate the storage location of the file shown in Table 1 in the object file storage system 102.

Referring to FIG. 1 , a first user has a need to access content in a file, which is a file stored in the object file storage system 102 . The granularity with which the first user accesses the content in the file may be the first granularity or the second granularity, and the second granularity is smaller than the first granularity.

In some embodiments, the first granularity may be a file granularity, that is, the first user needs to access the entire content of the file; or the first granularity may be a partition granularity, that is, the first user needs to access a partition of the file. File granularity refers to the need to access the entire contents of the first file. Partition granularity refers to the need to access the entire contents of a partition in the first file.

In some embodiments, the file is a structured data file, and the second granularity is row-column granularity, that is, the first user needs to access at least one row and/or at least one column of the file. Alternatively, the file is a semi-structured data file, and the second granularity is data fragment granularity, that is, the first user needs to access the data fragments in the file.

Therefore, it is possible to provide users with access services at file granularity, or provide users with access services at partition granularity, or provide users with access services at row and column granularity, or provide users with access services at data fragment granularity, which enriches the provision of services to users. access services.

The first user is a user with data access requirements, and can also be called a business user. Alternatively, the first user may be an application or the like.

When the first user needs to access data, he sends a data access request to the computing engine 101. The data access request includes access requirement information. The access requirement information is used to indicate the content of the first file that the first user needs to access. The first file is a file stored in the object file storage system 102.

The computing engine 101 is configured to receive the data access request, and determine the granularity of accessing the content in the first file based on the access requirement information. When the determined granularity is the first granularity, based on the account information of the first user and the access requirement Information, access the first file. When the determined granularity is the second granularity, the first file is accessed based on the specified administrator account information and the access requirement information, and the second granularity is smaller than the first granularity.

In some embodiments, the computing engine 101 includes an interface that the first user can call to send the data access request to the computing engine 101 through the interface. Optionally, the interface includes a Java database connectivity (JDBC) interface or an open database connectivity (ODBC) interface.

In some embodiments, the first file is a structured data file, the access requirement information includes identification information of the first file, and the access requirement information does not include identification information and first information of the partition in the first file, and the first The information is used to indicate at least one column and/or at least one row in the first file. The granularity determined by the computing engine 101 based on the access requirement information is file granularity.

In some embodiments, the first file is a semi-structured data file, the access requirement information includes the identification information of the first file, and the access requirement information does not include the identification information of the partition in the first file and the identification information of the partition in the first file. Identification information of the data fragment. The granularity determined by the computing engine 101 based on the access requirement information is file granularity.

In some embodiments, the first file is a structured data file, the access requirement information includes identification information of the first file and identification information of partitions in the first file, and the access requirement information does not include the first information. The granularity determined by the computing engine 101 based on the access requirement information is partition granularity.

In some embodiments, the first file is a semi-structured data file, the access requirement information includes identification information of the first file and identification information of partitions in the first file, and the access requirement information does not include identification information of the first file. Identification information of the data fragment. The granularity determined by the computing engine 101 based on the access requirement information is partition granularity.

In some embodiments, the first file is a structured data file, and the access requirement information includes identification information of the first file and first information, and the first information is used to indicate at least one column and/or at least one row in the first file. The granularity determined by the computing engine 101 based on the access requirement information is row-column granularity.

In some embodiments, the first file is a semi-structured data file, and the access requirement information includes identification information of the first file and identification information of the data fragments in the first file. The granularity determined by the computing engine 101 based on the access requirement information is the data fragment granularity.

In some embodiments, referring to Figures 1 and 3, the computing engine 101 includes a computing module 1011 and a routing module 1012. The computing module 1011 in the computing engine 101 receives the data access request, and the routing module 1012 in the computing engine 101 is based on the access request. Requirement information determines the granularity of accessing the content in the first file.

In some embodiments, the access requirement information further includes a first operation type, and the first operation type is used to indicate a first access operation for the first user to access the first file. Optionally, the first access operation includes query, update, insert or delete, etc.

In some embodiments, the operation of the computing engine 101 to access the first file may be: query the content in the first file, and return the queried content to the first user. Alternatively, the access requirement information includes content to be updated, and the operation of the computing engine 101 to access the first file may be: updating all or part of the content in the first file to the content to be updated. Alternatively, the access requirement information includes content to be inserted, and the operation of the computing engine 101 to access the first file may be: inserting the content to be inserted into the first file. Alternatively, the operation of the computing engine 101 to access the first file may be: deleting all or part of the content in the first file, etc.

In some embodiments, the data access request also includes the first user's account information. Alternatively, the data access request may not include the first user's account information. The communication connection between the computing engine 101 and the first user is bound to the first user's account information, and the computing engine 101 obtains the first user's account information bound to the communication connection. Optionally, the communication connection is a session between the first user and the computing engine 101.

In some embodiments, the access system 100 includes one or more computing engines 101 . Optionally, the computing engine 101 is a Hive engine or a Spark engine, that is, the access system 100 includes one or more Hive engines, and/or, one or more Spark engines, etc.

In some embodiments, the Hive engine is a data warehouse tool based on Hadoop (a distributed system infrastructure), which can map structured data files into a table and provide query functions.

In some embodiments, the Spark engine is a fast and general computing engine designed for large-scale data processing.

In some embodiments, referring to Figure 3, the access system 100 also includes a file path authentication module 103. The file path authentication module 103 communicates with the computing engine 101 and the object file storage system 102 respectively.

The computing engine 101 is configured to send an authentication request to the file path authentication module 103 when the determined granularity is the first granularity. The authentication request includes authentication information, and the authentication information is used to indicate the first user, the first The file path of the file and the second access operation of the first user to access the file path, the authentication information is obtained based on the access requirement information and the first user's account information;

The file path authentication module 103 is configured to receive the authentication request, and authenticate the first user's permission to access the file path using the second access operation based on the first permission information and the authentication information. The first permission information is used to Indicates the identity of the user who can access the file path and the third access operation that can access the file path. After passing the authorization authentication, an authentication response is sent to the computing engine 101. The authentication response includes a temporary credential and a request to the object file. The storage system 102 sends storage information, which includes the temporary credential, the file path, and a second operation type. The second operation type is the operation type of the second access operation;

The object file storage system 102 is used to receive the storage information and correspondingly save the temporary voucher, the file path and the second operation type;

The computing engine 101 is also used to access the first file based on the temporary credential, the access requirement information and the file path.

The second access operation is an operation obtained by mapping the first access operation, and the second access operation is an operation capable of accessing the object file storage system 102 . Usually the second access operation includes a read operation and/or a write operation.

For example, the first access operation is a query, and the second access operation mapped to the query operation is a read operation. Assuming that the content in the first file needs to be queried, the first file is read from the object file storage system 102, and the content that needs to be queried is obtained from the read first file.

For another example, the first access operation is an update, and the second access operation mapped to the update operation includes a read operation and a write operation. Assume that part of the content in the first file needs to be updated to the content to be updated, then the first file is read from the object file storage system 102, the part of the content in the first file is updated to the content to be updated, and the updated third file is updated to the content to be updated. A file is written to the object file storage system 102 to overwrite the first file saved in the object file storage system 102 .

In some embodiments, the authentication information includes the user identity of the first user, the file path of the first file, and the second operation type. Among them, the user identity of the first user is obtained by the computing engine 101 based on the first user's account information, the second operation type is obtained by mapping the first operation type, and the file path of the first file is obtained by the computing engine 101 based on the first operation type. The identification information of the file is obtained.

In some embodiments, the authentication information includes account information of the first user, identification information of the first file, and the first operation type.

In some embodiments, the user identity of the first user includes a user group to which the first user belongs and/or a role of the first user, etc.

In some embodiments, the first permission information includes the file path, a user identity that can access the file path, and a third operation type, and the third operation type is a type of a third access operation that can access the file path.

In some embodiments, referring to Figure 3, the access system 100 also includes a linkage authority module 104, which communicates with the computing engine 101 and the file path authentication module 103 respectively. The linkage authority module 104 stores the above-mentioned first authority information.

After receiving the authentication request, the file path authentication module 103 obtains the file path of the first file, the user identity of the first user and the second operation type of the second access operation based on the authentication information included in the authentication request. The permission module 104 obtains the first permission information including the file path. If the user identity of the first user is the same as the user identity included in the first permission information and the second operation type of the second access operation is the same as the third operation type of the third access operation included in the first permission information, then the permission is authenticated. Passed permission means that the first user has permission to use the second access operation to access the file path.

In some embodiments, the linkage permission module 104 includes a first read-write interface. The file path authentication module 103 calls the first read-write interface of the linkage permission module 104, and obtains the information including the linkage permission module 104 from the linkage permission module 104 through the first read-write interface. The first permission information of the file path.

In some embodiments, the authentication information includes the user identity of the first user, the file path of the first file, and the second operation type. The file path authentication module 103 directly obtains the file path of the first file from the authentication information. , the user identity of the first user and the second operation type of the second access operation.

In some embodiments, the authentication information includes the first user's account information, the first file's identification information, and the first operation type. The file path authentication module 103 obtains the first user's user identity based on the first user's account information. , map the first operation type to obtain the second operation type, and obtain the file path of the first file based on the identification information of the first file.

In some embodiments, referring to Figure 3, the access system 100 also includes a data filtering engine 105, which includes specified administrator account information; the data filtering engine 105 communicates with the computing engine 101 and the object file storage system 102 respectively.

The computing engine 101 is configured to send an access instruction to the data filtering engine 105 when the determined granularity is the second granularity, where the access instruction includes the file path of the first file and the access requirement information;

The data filtering engine 105 is used to access the first file based on the administrator account information, the file path and the access requirement information.

Optionally, the data filtering engine 105 also communicates with the linkage authority module 104.

Referring to Figure 3, in some embodiments, after receiving the data access request, the computing module 1011 of the computing engine 101 allows the first user to access the content based on the second permission information, the first user's account information and the access requirement information. Authentication of permissions. The second permission information is used to indicate the user identity and the fourth access operation that can access the content.

After the authentication is passed, if the granularity determined by the routing module 1012 of the computing engine 101 is the first granularity, the routing module 1012 of the computing engine 101 sends an authentication request to the file path authentication module 103 . If the granularity determined by the routing module 1012 of the computing engine 101 is the second granularity, the routing module 1012 of the computing engine 101 sends an access instruction to the data filtering engine 105 .

In some embodiments, the second permission information includes content identification information of the content, a user identity that can access the content, and a fourth operation type of a fourth access operation that can access the content.

In some embodiments, the linkage permission module 104 stores the second permission information, and the linkage permission module 104 includes a second read-write interface. After receiving the data access request, the computing module 1011 of the computing engine 101 obtains the content identification information of the content based on the access requirement information, and obtains the user identity of the first user based on the first user's account information. The second read-write interface in the linkage permission module 104 is called, and the second permission information including the content identification information is obtained from the linkage permission module 104 through the second read-write interface. If the user identity of the first user is the same as the user identity included in the second permission information and the first operation type of the first access operation is the same as the fourth operation type of the fourth access operation included in the second permission information, then the first user If the authorization to access the content is passed, it means that the first user has the authorization to access the content.

The content identification information of the content is part of the access requirement information.

In some embodiments, when the content is the entire content of the first file, the content identification information of the content includes identification information of the first file. Alternatively, when the content is a partition of the first file, the content identification information of the content includes identification information of the first file and identification information of the partition. Alternatively, when the content is at least one column or at least one row in the first file, the content identification information of the content includes the identification information of the first file and the column identification of the at least one column, or the content identification information of the content includes the first file The identification information and the line number of at least one line. Alternatively, the content is at least one data fragment in the first file, and the content identification information of the content includes identification information of the first file and identification information of each data fragment in the at least one data fragment.

In some embodiments, for the authentication operation of authenticating the first user's permission to access the content, and for the determination operation of determining the granularity of accessing the content in the first file, the computing module 1011 of the computing engine 101 receives After the data access request, the authentication operation may be performed first, and then the routing module 102 of the computing engine 101 may perform the determination operation. That is, the computing module 1011 of the computing engine 101 may first authenticate the first user's permission to access the content. . After the authentication is passed, the routing module 1012 of the computing engine 101 determines the granularity of accessing the content in the first file based on the access requirement information. or,

After the computing module 1011 of the computing engine 101 receives the data access request, the routing module 1012 of the computing engine 101 may first perform the determination operation, and then the computing module 1011 of the computing engine 101 may perform the authentication operation. That is, the routing module 1012 of the computing engine 101 may first determine the granularity of accessing the content in the first file based on the access requirement information, and then the computing module 1011 of the computing engine 101 authenticates the first user's permission to access the content. or,

After the computing module 1011 of the computing engine 101 receives the data access request, the computing module 1011 of the computing engine 101 performs the authentication operation, and at the same time, the routing module 1012 of the computing engine 101 performs the determination operation, that is, the authentication operation and the determination operation. executed simultaneously.

In some embodiments, referring to Figure 3, the access system 100 also includes an identity authentication center 106, which is used to save the corresponding relationship between the user's account information and the user's identity.

In some embodiments, the operation of the computing module 101 of the computing engine 101 to obtain the user identity of the first user is: the computing module 101 of the computing engine 101 queries the first user's identity from the identity authentication center 106 based on the first user's account information. User ID.

In some embodiments, the file path authentication module 103 obtains the user identity of the first user as follows: the file path authentication module 103 queries the first user's user identity from the identity authentication center 106 based on the first user's account information. .

In some embodiments, referring to FIG. 3 , the access system 100 further includes a metadata center 107 , which is configured to receive and save the metadata of the first file input by the second user, where the metadata includes the metadata of the first file. Identification information, the type of operation that needs to be performed on the first file and the file path of the first file. Optionally, the operation type may be creating the first file, deleting the first file, querying the first file, or modifying the first file, etc.

In some embodiments, the first file is a structured data file, and the metadata of the first file also includes one or more of the following: a column identifier of each column in the first file, a column type of each column in the first file , the row separator of the first file, or the column separator of the first file, etc.

In some embodiments, the first file is a semi-structured data file, and the metadata of the first file also includes one or more of the following: identification information of each data fragment in the first file, each The type of the data fragment, or the line delimiter of the first file, etc. The line delimiter is used to distinguish each line of data in any data fragment in the first file.

In some embodiments, the operation of the computing engine 101 to obtain the file path of the first file is: the computing engine 101 obtains metadata including the identification information of the first file from the metadata center 107, and obtains the first file from the metadata. file path.

In some embodiments, the operation of the file path authentication module 103 to obtain the file path of the first file is: the file path authentication module 103 obtains metadata including the identification information of the first file from the metadata center 107, and obtains the metadata from the metadata center 107. Get the file path of the first file in the data.

In some embodiments, the metadata center 107 displays a first interface to the second user, in which the second user can input the metadata of the first file, and receives the metadata of the first file input by the second user through the first interface. metadata. Optionally, the first interface includes network product interface design (website user interface, Web UI), etc.

In some embodiments, when receiving the metadata of the first file, the metadata center 107 also obtains the account information of the second user, and verifies the metadata of the first file based on the account information of the second user. When implemented,

The metadata center 107 verifies the legitimacy of the second user based on the second user's account information. When verifying that the second user is legitimate, the user identity of the second user is obtained, and the operation type that the second user can operate is obtained based on the user identity of the second user. If the metadata includes an operation type that needs to operate on the first file, The operation type that the second user can operate is to pass the verification of the metadata of the first file, and then save the metadata of the first file.

In some embodiments, the operation of the metadata center 107 to verify the legitimacy of the second user and obtain the user identity of the second user is:

The identity authentication center 106 stores the corresponding relationship between the account information and the user's identity. The metadata center 107 queries the identity authentication center 106 to see whether the second user's account information is stored in the identity authentication center 106. If the identity authentication center 106 stores the second user's account information, Verify that the second user is a legitimate user. Query the user identity of the second user from the identity authentication center 106 based on the second user's account information.

In some embodiments, the metadata center 10 obtains the operation type that the second user can operate as:

The metadata center 107 stores the corresponding relationship between the user identity and the operation type. Based on the user identity of the second user, the metadata center 107 obtains the corresponding operation type from the corresponding relationship between the user identity and the operation type as the operation type that the second user can operate. Operation type.

In some embodiments, the operation of the metadata center 107 to save the metadata of the first file is: the metadata center 107 queries whether the metadata including the identification information of the first file has been saved, and if the metadata has been saved, it will be saved. The metadata is updated to the metadata of the first file. If the metadata is not saved, the metadata of the first file is saved directly.

Metadata center 107 includes designated administrator account information. After the metadata of the first file is verified, if the operation type included in the metadata is to create the first file, the metadata center 107 creates the first file in the object file storage system 102 based on the specified administrator account information. The file path, the storage location corresponding to the file path is used to save the first file. If the operation type included in the metadata is to delete the first file, the metadata center 107 determines the first file in the object file storage system 102 based on the specified administrator account information and the file path of the first file, and deletes the determined file. First document. If the operation type included in the metadata is to query the first file, the metadata center 107 determines the first file in the object file storage system 102 based on the specified administrator account information and the file path of the first file, and obtains the first file. description information and/or attribute information, etc., and return the obtained content to the second user. If the operation type included in the metadata is to modify the first file, the metadata center 107 determines the first file in the object file storage system 102 based on the specified administrator account information and the file path of the first file, and modifies the first file. description information and/or attribute information.

Referring to FIG. 3 , the linkage permission module 104 is also configured to receive second permission information configured by the permission administrator. The second permission information is used to indicate the identity of the user who can access the content in the first file and the fourth access operation. The first permission information is generated based on the second permission information, and the first permission information is used to indicate a user identity that can access the file path of the first file and a third access operation that can access the file path of the first file. Save the second authority information and the first authority information.

In some embodiments, the linkage permission module 104 is also used to obtain the metadata report of the first file from the metadata center 107, obtain at least one user identity from the identity authentication center 106, and display the second interface to the permission administrator. The second interface includes metadata of the first file and the at least one user identity.

In this way, the rights administrator selects the content identification information of the content in the first file from the metadata of the first file, selects a user identity that can access the content from the at least one user identity, and inputs the user identity that can access the content into the second interface. The fourth operation type of the fourth access operation of the content is used to obtain the second permission information. The second permission information includes content identification information of the content, the selected user identity and the input fourth operation type. Optionally, the second interface includes Web UI, etc.

In some embodiments, the first file is a structured data file, the metadata of the first file includes a file identifier of the first file and a column identifier of each column of the first file, and the content identification information of the content includes a file identifier of the first file. File identification, or the content identification information of the content includes the file identification of the first file and the column identification of at least one column in the first file, or the content identification information of the content includes the file identification of the first file and the column identification of at least one column in the first file. The line number of at least one line.

In some embodiments, the first file is a semi-structured data file, the metadata of the first file includes a file identification of the first file and identification information of each data fragment of the first file, and the content identification information of the content includes the third file. The file identification of a file, or the content identification information includes the file identification of the first file and the identification information of at least one data fragment in the first file.

In some embodiments, the operation of the linkage permission module 104 to generate the first permission information is:

(1): The linkage permission module 104 obtains the file path of the first file based on the content identification information of the content in the second permission information.

In some embodiments, the content identification information of the content includes identification information of the first file, and metadata including the identification information of the first file is obtained from the metadata center 107. The metadata is metadata of the first file, from Obtain the file path of the first file from the metadata of the first file.

(2): The linkage authority module 104 maps the fourth operation type included in the second authority information to obtain the third operation type.

(3): The linkage permission module 104 reads the user identity from the second permission information, and combines the file path of the first file, the user identity and the third operation type into the first permission information.

In this embodiment of the present application, the computing engine receives the data access request, and determines the granularity of accessing the content in the first file based on the access requirement information in the data access request. When the determined granularity is the first granularity, based on the first user's Account information and the access requirement information, access the first file. When the determined granularity is the second granularity, the first file is accessed based on the specified administrator account information and the access requirement information. In this way, the first-granularity access service and the second-granularity access service can be provided to the user, which enriches the access services provided to the user. Since the first granularity is greater than the second granularity, when the determined granularity is the first granularity, the computing engine accesses the first file based on the first user's account information and the access requirement information, so that there is no need to borrow the administrator account information to access the first file. , improve the efficiency of accessing the first file and the performance of reading and writing the first file.

In addition, if the permission to access the second granular content is configured for the first user, in addition to the content in the first file, the first user can also access other content in the first file except the content. The user's permissions are automatically expanded to access any content in the first file, which results in too much permission expansion and is not conducive to permission management. However, in this application, when the determined granularity is the second granularity, the computing engine accesses the first file based on the specified administrator account information and the access requirement information, so that the administrator account information is used to replace the first user's account information. , and use the administrator account information to access the first file, so that there is no need to configure the first user with permissions to access the second granularity, thereby avoiding expanding the first user's access permissions and facilitating permission management.

Referring to Figure 4, this embodiment of the present application provides a method 400 for accessing files. The method 400 is applied to the access system 100 shown in Figure 1 or Figure 3. The method 400 includes the following steps 401 to 410.

Step 401: The computing engine receives a data access request. The data access request includes access requirement information. The access requirement information is used to indicate the content of the first file that the first user needs to access. The first file is stored in the object file storage system.

In some embodiments, the first user is a business user who performs data access services, and the first user sends a data access request to the computing engine.

In some embodiments, the access requirement information is an access statement for accessing the database, for example, the access requirement information is a SQL statement, etc.

In some embodiments, the access requirement information includes content identification information of the content and a first operation type, and the first operation type is used to indicate a first access operation for accessing the content in the first file. Optionally, the first access operation is querying the first file, updating the first file, or deleting the first file, etc.

In some embodiments, when the first access operation indicated by the first operation type is to update the first file, the access requirement information also includes content to be updated.

In some embodiments, the data access request may also include the first user's account information.

In some embodiments, the access requirement information may include the following types of information. The following types of access requirement information will be described respectively.

Type 1, the access requirement information includes identification information of the first file and the first operation type.

The access requirement information does not include identification information of the partition of the first file. When the first file is a structured data file, the access requirement information does not include the first information, and the first information is used to indicate at least one column and/or at least one row in the first file. When the first file is a semi-structured data file, the access requirement information does not include identification information of partitions in the first file.

In this case, the content identification information of the content is the identification information of the first file.

For type 1 access requirement information, the content is the entire content of the first file, indicating that the first user needs to access all the content of the first file, and the granularity of the first user's access to the content in the first file is file granularity.

For example, the access requirement information is: Select*From Company information. The access requirement information includes the identification information "Company information" of the first file as shown in Table 1 and the first operation type "Select". The first operation type "Select" is to query the first file.

Type 2: the access requirement information includes identification information of the first file, identification information of the partition in the first file, and the first operation type.

When the first file is a structured data file, the access requirement information does not include the first information, and the first information is used to indicate at least one column and/or at least one row in the first file. When the first file is a semi-structured data file, the access requirement information does not include identification information of partitions in the first file.

In this case, the content identification information of the content includes identification information of the first file and identification information of the partition in the first file.

For type 2 access requirement information, the content is the partition of the first file, indicating that the first user needs to access the partition of the first file, and the granularity of the first user accessing the content in the first file is the partition granularity.

Type 3, the first file is a structured data file, the access requirement information includes the identification information of the first file, the first information and the first operation type, the first information is used to indicate at least one column in the first file and/or at least One line.

For type 3 access requirement information, the content is the at least one column or the at least one row of the first file, indicating that the first user needs to access the at least one column or the at least one row of the first file, and the first user accesses the at least one column or the at least one row of the first file. The granularity of this content is row-column granularity.

In some embodiments, the first information includes a column identifier of the at least one column in the first file, the content is the at least one column in the first file, indicating that the first user needs to access the at least one column of the first file. In this case, the content identification information of the content includes identification information of the first file and a column identification of the at least one column in the first file.

For example, the access requirement information is: Select Name, City From Company information. The access requirement information includes the identification information "Company information" of the first file as shown in Table 1, the column identification "Name" of the second column of the first file, the column identification "City" of the fourth column of the first file, and The first operation type "Select" is to query the first file.

In some embodiments, the first information includes a column identifier of at least one column in the first file and row filtering information corresponding to each column in the at least one column, and the content is at least one row in the first file, indicating that the first user needs Access at least one line of the first file. In this case, the content identification information of the content includes identification information of the first file and a column identification of the at least one column in the first file.

For any column in the at least one column, one or more rows whose content of the column is the row of filter information can be located from the first file. The content is the content of the located row or rows.

For example, referring to the above-mentioned Table 1, assuming that the first information includes the column identifier "City" of the fourth column in Table 1 and the row filtering information "City 1" corresponding to the fourth column, the first information can be obtained from the information shown in Table 1 based on the first information. In the first file, the City in the fourth column is located as the first, second and fifth rows of "City 1". The access requirement information is: Select*From Company information Where City=City 1, and the first operation type "Select" is to query the first file.

In some embodiments, the first information includes a line number of at least one line in the first file, the content is the at least one line in the first file, indicating that the first user needs to access the at least one line in the first file. In this case, the content identification information of the content includes identification information of the first file and a line number of the at least one line in the first file.

Type 4, the first file is a semi-structured data file, and the access requirement information includes identification information of the first file, identification information of at least one data fragment in the first file, and the first operation type.

In this case, the content identification information of the content includes identification information of the first file and identification information of the at least one data fragment in the first file.

For type 4 access requirement information, the content is the at least one data fragment of the first file, indicating that the first user needs to access the at least one data fragment of the first file, and the granularity of the first user's access to the content in the first file. is the data fragment granularity.

For type 4 access requirement information, the data access request is a remote procedure call (RPC) request, and the RPC request includes type 4 access requirement information.

For example, the data access information in the RPC request includes the identification information "Company information" of the first file, the identification information "Name" of the first data fragment of the first file, and the third data of the first file as shown in Figure 2 The identification information of the fragment "Country" and the first operation type "query the first file".

To sum up, the access requirement information at least includes the identification information of the first file and the first operation type, and may also include the first information, the identification information of the partition in the first file, or the identification information of the data fragment in the first file. wait.

Step 402: The computing engine authenticates the first user's permission to access the content based on the second permission information, the first user's account information and the access requirement information, and then authenticates the first user's permission to access the content. Afterwards, step 403 is executed.

The second permission information is used to indicate the user identity that can access the content and the fourth access operation that can access the content.

Referring to Figure 5, in some embodiments, the computing engine is a Hive engine, and the Hive engine authenticates the first user's permission to access the content through the following operations 4021 to 4023.

4021: Regarding the content identification information of the content in the access requirement information, the Hive engine determines whether the content exists in the object file storage system based on the content identification information. If the content exists in the object file storage system, perform the following operation 4022.

In 4021, the content identification information includes the identification information of the first file. The Hive engine obtains the metadata of the first file from the metadata center based on the identification information of the first file. The metadata of the first file includes the metadata of the first file. Identification information. The operation type included in the metadata of the first file is to delete the first file, which means that the metadata center has deleted the first file in the object file storage system. The Hive engine determines that the object file storage system does not exist based on the metadata of the first file. the content. The operation type included in the metadata of the first file is creating the first file, modifying the first file, or querying the first file, which means that the object file storage system stores the first file. The Hive engine determines whether the content exists in the object file storage system based on the metadata of the first file and the content identification information.

In some embodiments, the operation of the Hive engine to obtain the metadata of the first file is:

The Hive engine sends a first acquisition command to the metadata center, where the first acquisition command includes identification information of the first file. The metadata center receives the first acquisition command, acquires metadata including the identification information of the first file from the saved metadata, and the acquired metadata is the metadata of the first file, and sends a first acquisition response to the Hive engine. The get response includes metadata for the first file. or,

The Hive engine sends the first fetch command to the metadata center. The metadata center receives the first acquisition command, acquires each saved metadata, and sends a first acquisition response to the Hive engine, where the first acquisition response includes each metadata. The Hive engine receives the first acquisition response, and acquires metadata including the identification information of the first file from each metadata, and the acquired metadata is the metadata of the first file.

In some embodiments, when the metadata of the first file includes an operation type of creating the first file, modifying the first file, or querying the first file, the Hive engine based on the metadata of the first file and the content identification information, The operation to determine whether the content exists in the object file storage system is:

If the content identification information includes the identification information of the first file and the identification information of the partition in the first file, and the metadata of the first file also includes the identification information of the partition in the first file, it is determined that the object file storage system exists the content. The metadata in the first file does not include the identification information of the partition in the first file, and it is determined that the content does not exist in the object file storage system.

If the content identification information includes the identification information of the first file and the column identification of at least one column in the first file, and the metadata of the first file also includes the column identification of the at least one column, it is determined that the content exists in the object file storage system. If the metadata of the first file does not include the column identifier of the at least one column, it is determined that the content does not exist in the object file storage system.

If the content identification information includes the identification information of the first file and the identification information of the data fragment in the first file, and the metadata of the first file also includes the identification information of the data fragment, it is determined that the content exists in the object file storage system. The metadata of the first file does not include the identification information of the data fragment, and it is determined that the content does not exist in the object file storage system.

If the content identification information includes the identification information of the first file and the line number of at least one line in the first file, when it is determined that the first file is stored in the object file storage system, the content can be considered to exist in the object file storage system.

The metadata of the first file includes the file path of the first file, so the calculation engine reads the file path of the first file from the metadata of the first file.

The operation of 4021 is an optional operation, that is to say, you can directly perform the following operation of 4022 without performing the operation of 4021. Alternatively, you can also perform the operation 4021, and then perform the following operation 4022.

4022: The Hive engine obtains the second permission information including the content identification information of the content from the linkage permission module. The second permission information includes the content identification information of the content, the identity of the user who can access the content, and the fourth user who can access the content. The fourth operation type of access operation.

In some embodiments, the linkage permission module includes a second read-write interface. The Hive engine reads each second permission information saved in the linkage permission module through the second read-write interface. Based on the content identification information of the content, from each Obtain the second permission information corresponding to the content from the second permission information.

In some embodiments, the Hive engine sends a second acquisition command to the linkage permission module, and the second acquisition command includes the content identification information of the content. The linkage permission module receives the second acquisition command, obtains the second permission information corresponding to the content from each saved second permission information based on the content identification information, and sends a second acquisition response to the Hive engine. The second acquisition response includes acquisition second permission information.

Wherein, if the content identification information includes the identification information of the first file, and the content identification information does not include other information, the obtained second permission information is the second permission information including the identification information of the first file.

If the content identification information includes the identification information of the first file and the line number of at least one line in the first file, and the content identification information does not include other information, the obtained second permission information includes the identification information of the first file. Second authority information.

If the content identification information includes the identification information of the first file and the column identification of at least one column in the first file, and the content identification information does not include other information, the obtained second permission information includes the identification information of the first file and the column identification of the first file. At least one column identifies the secondary permission information.

If the content identification information includes the identification information of the first file and the identification information of the partition in the first file, and the content identification information does not include other information, the obtained second permission information includes the identification information of the first file and the identification information of the first file. The second permission information of the partition identification information.

If the content identification information includes the identification information of the first file and the identification information of the data fragment in the first file, and the content identification information does not include other information, the obtained second permission information includes the identification information of the first file and The second permission information of this data fragment.

4023: The Hive engine authenticates the first user's permission to access the content based on the second permission information, the first user's account information and the first operation type.

In 4023, the Hive engine determines the user identity of the first user based on the account information of the first user, compares the user identity of the first user with the user identity included in the second permission information, and compares the first operation type with the user identity included in the second permission information. The fourth operation type. If the user identity of the first user is compared with the user identity included in the second permission information, and the first operation type is compared with the fourth operation type included in the second permission information, then the permission for the first user to access the content is Authentication passed. If it is compared that the user identity of the first user is different from the user identity included in the second permission information, and/or it is compared that the first operation type is different from the fourth operation type included in the second permission information, then the first user accesses the Content permission authentication failed.

In some embodiments, the identity authentication center stores the corresponding relationship between the user's account information and the user's identity. The Hive engine queries the first user's user identity from the identity authentication center based on the first user's account information.

Referring to Figure 6, in some embodiments, the computing engine is a Spark engine, and the Spark engine authenticates the first user's permission to access the content through the following operations 4121 to 4125.

4121: The Spark engine sends the access requirement information to the metadata center.

In some embodiments, the Spark engine also sends the first user's account information to the metadata center.

4122: Regarding the content identification information of the content in the access requirement information, the metadata center determines whether the content exists in the object file storage system based on the content identification information. If the content exists in the object file storage system, perform the following operations 4123. .

For the detailed process of the metadata center determining whether the content exists in the object file storage system, please refer to the detailed process of the Hive engine determining whether the content exists in the object file storage system in the above operation 4021, which will not be described in detail here.

If the Spark engine also sends the first user's account information to the metadata center, when the metadata center determines that the content exists in the object file storage system, it obtains the second permission information including the content identification information of the content from the linkage permission module. Based on the second permission information, the first user's account information and the first operation type, the first user's permission to access the content is authenticated. After the first user's permission to access the content is authenticated, the following 4123 is performed. operation.

For the process of the metadata center obtaining the second permission information and authenticating the first user's permission to access the content, refer to the above 4022 and 4023 for the Hive engine to obtain the second permission information and authenticate the first user's permission to access the content. The process will not be described in detail here.

4123: The metadata center sends confirmation information to the Spark engine.

Among them, if the content does not exist in the object file storage system, a denial message is sent to the Spark engine. Or, if the first user's permission to access the content fails to be authenticated, a denial message is sent to the Spark engine.

4124: The Spark engine receives the confirmation information and obtains the second permission information including the content identification information of the content from the linkage permission module. The second permission information includes the content identification information of the content, the identity of the user who can access the content, and the identity of the user who can access the content. A fourth operation type of a fourth access operation for the content.

For the process of the Spark engine obtaining the second permission information, please refer to the process of the Hive engine obtaining the second permission information in 4022 above, which will not be explained in detail here.

If the Spark engine receives the denial information, the operation ends.

4125: The Spark engine authenticates the first user's permission to access the content based on the second permission information, the first user's account information and the first operation type.

For the process of the Spark engine authenticating the first user's permission to access the content, please refer to the process of the Hive engine authenticating the first user's permission to access the content in 4023 above, which will not be described in detail here.

Among them, the above-mentioned operations 4121-4123 are optional operations, that is, you do not need to perform the above-mentioned operations 4121-4123. Spark directly performs the operations 4124-4125, that is, the Spark engine obtains the content identification information including the content from the linkage permission module. The second permission information is used to authenticate the first user's permission to access the content based on the second permission information, the first user's account information and the first operation type.

Step 403: The computing engine determines the granularity of accessing the content of the first file based on the access requirement information. If the determined granularity is the first granularity, step 404 is executed. If the determined granularity is the second granularity, step 409 is executed.

In step 403, the computing engine determines the granularity of accessing the content of the first file based on the content identification information of the content included in the access requirement information.

The above describes the four types of access requirement information. Next, the process of determining the granularity is explained for each type of access requirement information.

For the above type 1 access requirement information, the access requirement information includes identification information of the first file. However, the access requirement information does not include identification information of the partition of the first file. Moreover, when the first file is a structured data file, the access requirement information does not include the first information. When the first file is a semi-structured data file, the access requirement information does not include identification information of the data fragments in the first file. At this time, the content identification information of the content is the identification information of the first file, and the computing engine determines that the granularity of accessing the content of the first file is file granularity.

For the above type 2 access requirement information, the access requirement information includes identification information of the first file and identification information of the partition in the first file. Moreover, when the first file is a structured data file, the access requirement information does not include the first information. When the first file is a semi-structured data file, the access requirement information does not include identification information of partitions in the first file. At this time, the content identification information of the content is the identification information of the first file and the identification information of the partition in the first file, and the computing engine determines that the granularity of accessing the content of the first file is the partition granularity.

The first granularity is file granularity or partition granularity, so if the determined granularity is the first granularity, the content identification information of the content includes the identification information of the first file, or the content identification information of the content includes the identification of the first file. information and the identification information of the partition in the first file.

For the above type 3 access requirement information, the first file is a structured data file. The access requirement information includes identification information of the first file and first information. The first information is used to indicate at least one line in the first file and/or At least one column. At this time, the content identification information of the content is the identification information of the first file and the column identification of the at least one column in the first file, or the content identification information of the content is the identification information of the first file and the column identification of the first file. The line number of at least one line, the calculation engine determines that the granularity of accessing the content of the first file is the row and column granularity.

For the above type 4 access requirement information, the first file is a semi-structured data file, and the access requirement information includes identification information of the first file and identification information of the data fragments in the first file. At this time, the content identification information of the content is the identification information of the first file and the identification information of the data fragment in the first file, and the computing engine determines that the granularity of accessing the content of the first file is the data fragment granularity.

The second granularity is row-column granularity or data fragment granularity. Therefore, if the determined granularity is the second granularity, the content identification information of the content includes the identification information of the first file and the column identification of the at least one column in the first file. Optionally, the access requirement information may also include row filtering information corresponding to the at least one column. Alternatively, the content identification information of the content includes identification information of the first file and the line number of the at least one line in the first file. Alternatively, the content identification information of the content includes identification information of the first file and identification information of the data fragment in the first file.

The computing engine includes computing module and routing module. The above step 402 is executed by the computing module, and the above step 403 is executed by the routing module. The computing module may first perform the above step 402, and then the routing module may perform the above step 403. Alternatively, the routing module first performs the above step 403, and then the computing module can perform the above step 402. Alternatively, while the routing module performs the above step 403, the computing module also performs the above step 402. After the computing engine passes the authentication of the first user's permission to access the content, and if the granularity determined by the routing module is the first granularity, the following step 404 is executed. If the granularity determined by the routing module is the second granularity, the following step 409 is executed.

Step 404: The computing engine sends an authentication request to the file path authentication module. The authentication request includes authentication information. The authentication information is used to indicate the first user, the file path of the first file, and the first user's access to the file path. second access operation.

In some embodiments, the authentication information includes the file path of the first file, the user identity of the first user, and the second operation type. The second operation type is an operation type corresponding to the first operation type that can access the object file storage system. The computing engine maps the first operation type to obtain the second operation type. The second operation type includes read operations and/or write operations.

In some embodiments, the authentication information includes identification information of the first file, account information of the first user, and the first operation type. The authentication information may also include other information, which will not be listed here.

Referring to Figure 5 or Figure 6, in step 404, the routing module of the computing engine (Hive engine or Spark engine) sends an authentication request to the file path authentication module.

Step 405: The file path authentication module receives the authentication request, and authenticates the first user's permission to access the file path using the second access operation based on the first permission information and the authentication information. The second access operation is the second The access operation corresponding to the operation type.

The linkage permission module stores the first permission information corresponding to at least one file. For any file, the first permission information corresponding to the file includes the file path of the file, the identity of the user who can access the file path of the file and the user who can access the file path. The third operation type for the third access operation of the file path of the file.

In some embodiments, the authentication information includes the file path of the first file, the user identity of the first user, and the second operation type. In step 405, the file path authentication module reads the first permission information including the file path from the linkage permission module based on the file path, and the read first permission information corresponds to the first file. Comparing the user identity of the first user with the user identity included in the read first permission information, and comparing the second operation type with the third operation type included in the read first permission information. If the user identity of the first user is compared with the user identity included in the read first permission information, and the second operation type is compared with the third operation type included in the read first permission information, then the first The user's permission to access the file path using the second access operation is authenticated and passed. If the comparison shows that the user identity of the first user is different from the user identity included in the read first permission information, and/or the comparison shows that the second operation type is different from the third operation type included in the read first permission information, then The permission authentication for the first user to access the file path using the second access operation failed.

In some embodiments, the authentication information includes identification information of the first file, account information of the first user, and the first operation type. The file path authentication module first obtains the file path of the first file based on the identification information of the first file, obtains the user identity of the first user based on the account information of the first user, and maps the first operation type to obtain the second operation type. Then, the first user's permission to access the file path using the second access operation is authenticated.

In some embodiments, the file path authentication module obtains metadata including identification information of the first file from the metadata center, the metadata is metadata of the first file, and reads the metadata of the first file from the metadata of the first file. The file path of a file. And, the file path authentication module obtains the user identity of the first user from the identity authentication center based on the account information of the first user.

Step 406: When the file path authentication module passes the permission authentication for the first user to access the file path using the second access operation, it sends storage information to the object file storage system and an authentication response to the computing engine. The storage information includes The temporary credential, the file path and the second operation type, the authentication response includes the temporary credential.

In step 406, the file path authentication module allocates a temporary credential when the first user's permission to access the file path through the second access operation is authenticated.

Step 407: The object file storage system receives the storage information and saves the corresponding relationship between the temporary certificate, the file path and the second operation type.

The object file storage system stores the correspondence between temporary credentials, file paths, and operation types. In step 407, the object file storage system receives the storage information and stores the temporary certificate, the file path and the second operation type in the corresponding relationship between the temporary certificate, the file path and the operation type.

In the correspondence between temporary credentials, file paths and operation types, if the storage time of the temporary credentials reaches the specified length, the object file storage system deletes the temporary credentials from the correspondence between temporary credentials, file paths and operation types. Record.

Step 408: The computing engine receives the authentication response, and based on the temporary credential, the access requirement information and the file path, accesses the first file in the object file storage system.

The access requirement information includes content identification information of the content and a first operation type of the first access operation. The content identification information of the content includes identification information of the first file, or the content identification information of the content includes the identification of the first file. information and identification information of the partition in the first file.

Assume that the first access operation is to query the first file, and the second access operation mapped by the first access operation includes a read operation. In step 408, the first file is accessed according to the following process.

1-1: The computing engine sends a read request to the object file storage system. The read request includes the temporary credential and the file path of the first file.

For example, the access requirement information listed above is: Select*From Company information. In this example, the first operation type "Select" is to query the first file, the content identification information of the content is "Company information", and the second access operation mapped by the computing engine to "Select" includes a read operation.

Assume that the temporary credential assigned by the file path authentication module is "P1", and the corresponding relationship between the temporary credential, file path and operation type as shown in Table 2 below is stored in the object file storage system. The first record in this correspondence includes temporary credentials 1, the file path of the first file "C:\windows\system32\Company information" as shown in Table 1 and the second access operation of the second access operation to the file path. Operation type "read operation".

Table 2

临时凭证Temporary credentials	文件路径file path	操作类型Operation type
P1P1	C:\windows\system32\Company informationC:\windows\system32\Company information	读操作Read operation
………	………	………

In 1-1, the computing engine sends a read request to the object file storage system. The read request includes the temporary credential "P1" and the file path of the first file "C:\windows\system32\Company information".

1-2: The object file storage system receives the read request, and based on the temporary credentials included in the read request, obtains the corresponding file path and second operation type from the correspondence between the temporary credentials, file path, and operation type.

For example, the object file storage system receives the read request, which includes the temporary credential "P1" and the file path of the first file "C:\windows\system32\Company information". Based on the temporary credential "P1", obtain the corresponding file path "C:\windows\system32\Company information" and the second operation type "read operation" from the corresponding relationship between the temporary credential, file path and operation type shown in Table 2. ".

1-3: If the file path included in the read request is the same as the obtained file path and the second access operation corresponding to the second operation type includes a read operation, the object file storage system reads the first file based on the file path of the first file, Return the first file to the calculation engine.

The file path "C:\windows\system32\Company information" included in the read request is the same as the obtained corresponding file path "C:\windows\system32\Company information", and the obtained second operation type (read operation) corresponds to The second access operation includes a read operation, so the object file storage system reads the first file as shown in Table 1 based on the file path "C:\windows\system32\Company information" of the first file, and returns to the computing engine as shown in Table 1 The first file shown.

1-4: The computing engine receives the first file, the content identification information includes the identification information of the first file, and returns the first file to the first user; the content identification information includes the identification information of the first file and the first file identification information of the partition, obtain the contents of the partition from the first file, and return the contents of the partition to the first user.

The content identification information includes identification information of the first file, indicating that the first user needs to query the entire content of the first file. The content identification information includes identification information of the first file and identification information of the partition in the first file, indicating that the first user needs to query the content of the partition in the first file.

For example, the computing engine receives the first file shown in Table 1, where the content identification information includes the identification information "Company information" of the file shown in Table 1, and returns the first file shown in Table 1 to the first user. .

Assume that the first access operation is to update the first file, the second access operation mapped by the first access operation includes a read operation and a write operation, and the access requirement information includes content to be updated. In step 408, the first file is accessed according to the following process.

2-1: The computing engine sends a read request to the object file storage system, and the read request includes the temporary credential and the file path.

2-2: The object file storage system receives the read request, and based on the temporary credentials included in the read request, obtains the corresponding file path and second operation type from the correspondence between the temporary credentials, file path, and operation type.

2-3: If the file path included in the read request is the same as the obtained file path and the second access operation corresponding to the second operation type includes a read operation, the object file storage system reads the first file based on the file path and reports it to the computing engine. Return to the first file.

2-4: The computing engine receives the first file, the content identification information includes the identification information of the first file, and updates the content in the first file to the content to be updated; the content identification information includes the identification information of the first file and The identification information of the partition in the first file is used to update the content of the partition in the first file to the content to be updated.

2-5: The computing engine sends a write request to the object file storage system. The write request includes the temporary certificate, the first file, and the file path.

2-6: The object file storage system receives the write request, and based on the temporary credentials included in the write request, obtains the corresponding file path and second operation type from the correspondence between the temporary credentials, file path, and operation type.

2-7: If the file path included in the write request is the same as the obtained file path and the second access operation corresponding to the second operation type includes a write operation, the object file storage system replaces the first file saved at the file path with this Write the first file included in the request.

The first access operation may also be other operations, for example, the first access operation may be deleting the first file, etc., which will not be listed one by one here.

Step 409: The computing engine sends an access instruction to the data filtering engine, where the access instruction includes the file path of the first file and the access requirement information.

Referring to Figure 5 or Figure 6, in step 409, the routing module of the computing engine (Hive engine or Spark engine) sends an access instruction to the data filtering engine.

Step 410: The data filtering engine receives the access instruction and accesses the first file based on the administrator account information, the file path and the access requirement information.

In some embodiments, the data filtering engine authenticates the first user's permission to access the content based on the second permission information, the first user's account information, and the access requirement information. After the first user's permission to access the content is authenticated, the first file is accessed based on the administrator account information, the file path and the access requirement information. The second permission information is used to indicate the user identity that can access the content and the fourth access operation that can access the content.

For the detailed process of the data filtering engine authenticating the first user's permission to access the content, please refer to the detailed process of the computing engine authenticating the first user's permission to access the content in the above step 402, which will not be described in detail here. .

The first file is a structured data file, and the access requirement information includes identification information of the first file, first information and a first operation type of the first access operation, and the first information includes a column identifier of at least one column of the first file, or , the first information includes a column identifier of at least one column of the first file and row filtering information corresponding to the at least one column, or the first information includes a line number of the at least one line in the first file. Alternatively, the first file is a semi-structured data file, and the content identification information of the content includes identification information of the first file and identification information of the data fragments in the first file.

Assume that the first access operation is to query the first file, and the second access operation mapped by the first access operation includes a read operation. In step 410, the first file is accessed according to the following process.

3-1: The data filtering engine sends a read request to the object file storage system. The read request includes the administrator account information and the file path of the first file.

For example, the access requirement information listed above is: Select*From Company information Where City=City 1. In this example, the first operation type "Select" is to query the first file. The content identification information of the content includes "Company information" and the column identification "City" in the fourth column. The row filter information corresponding to the fourth column is "City 1". ”, the second access operation mapped by the calculation engine to “Select” includes a read operation.

The data filtering engine sends a read request to the object file storage system. The read request includes the administrator account information "administrators" and the file path of the first file "C:\windows\system32\Company information".

3-2: When the object file storage system receives the read request and determines that the account information included in the read request is the administrator account information, it reads the first file based on the file path of the first file and returns the first file to the data filtering engine.

The administrator has greater authority, so when the object file storage system determines that the account information included in the read request is the administrator's account information, it can directly read the first file based on the file path of the first file.

For example, the object file storage system receives the read request, reads the first file shown in Table 1 based on the file path "C:\windows\system32\Company information" of the first file, and returns the information shown in Table 1 to the data filtering engine. The first file shown.

3-3: The data filtering engine receives the first file, the access requirement information includes the first information, and obtains the content in the first file based on the first information; the content identification information includes identification information of the data fragments in the first file. , obtain the content in the first file from the first file, and the content is the content in the data fragment.

In some embodiments, the first information includes a column identifier of at least one column in the first file, the data filtering engine obtains the content of the at least one column from the first file based on the first information, and returns the content of the at least one column to the computing engine.

In some embodiments, the first information includes a column identifier and row filtering information of at least one column in the first file, and the data filtering engine obtains the content of the at least one column from the first file based on the first information as the row filtering information. The content of one or more lines is returned to the calculation engine.

For example, the data filtering engine receives the first file shown in Table 1 and filters the information "City 1" based on the column identifier "City" in the fourth column and the row corresponding to the fourth column "City 1". Obtain the three rows whose content in the fourth column is "City 1". These three rows are the first row, the second row and the fifth row in Table 1. Return the contents of the three rows to the calculation engine. The calculation engine returns the content to the first user. Return the three lines of content.

In some embodiments, the first information includes a line number of at least one line in the first file, the data filtering engine obtains the content of the at least one line from the first file based on the first information, and returns the content of the at least one line to the computing engine.

In some embodiments, the first information includes identification information of the data fragment in the first file, the data filtering engine obtains the content of the data fragment from the first file based on the first information, and returns the content of the data fragment to the computing engine.

3-4: The data filtering engine returns the content in the first file to the calculation engine.

3-5: The calculation engine receives the content in the first file and returns the content in the first file to the first user.

Assume that the first access operation is to update the first file, the second access operation mapped by the first access operation includes a read operation and a write operation, and the access requirement information includes content to be updated. In step 410, the first file is accessed according to the following process.

4-1: The data filtering engine sends a read request to the object file storage system. The read request includes the administrator account information and the file path of the first file.

4-2: When the object file storage system receives the read request and determines that the account information included in the read request is the administrator account information, it reads the first file based on the file path of the first file and returns the first file to the data filtering engine.

4-3: The data filtering engine receives the first file, where the access requirement information includes the first information, and updates at least one column or at least one row indicated by the first information in the first file to the content to be updated; where the access requirement information includes The identification information of the data fragment in the first file updates the content in the data fragment in the first file to the content to be updated.

In some embodiments, the first information includes a column identifier of at least one column in the first file, and the data filtering engine updates the at least one column in the first file to the content to be updated.

In some embodiments, the first information includes a column identifier and row filtering information of at least one column in the first file, and the data filtering engine determines from the first file based on the first information that the content in the at least one column is the row filtering information. The content of one or more rows is updated to the content to be updated.

In some embodiments, the first information includes a line number of at least one line in the first file, and the data filtering engine updates the content of the at least one line in the first file to the content to be updated based on the first information.

In some embodiments, the first information includes identification information of the data fragment in the first file, and the data filtering engine updates the content of the data fragment in the first file to the content to be updated based on the identification information of the data fragment.

4-4: The data filtering engine sends a write request to the object file storage system. The write request includes administrator account information, the file path of the first file, and the updated first file.

4-5: When the object file storage system receives the write request and determines that the account information included in the write request is the administrator account information, it replaces the first file saved in the file path with the first file included in the write request.

In some embodiments, for the first permission information and the second permission information in the linkage permission module, the linkage permission module receives the second permission information configured by the permission administrator, and generates the first permission information based on the second permission information.

In this way, the permission administrator authorizes access to the content of the file by configuring the second permission information, and the linkage permission module generates the first permission information based on the second permission information, so that the computing engine in the access system uses the second permission information for authentication. The file path authentication module uses the first permission information for authentication. This only requires one authorization from the authority administrator to enable two-dimensional authentication.

In some embodiments, the first user accesses the first file in an open source manner, that is, the first user sends authentication information to the file path authentication module through the client, and the authentication information includes the file path of the first file. The user identity and the second operation type of the second access operation. The file path authentication module authenticates the first user's permission to access the file path using the second access operation based on the authentication information. After passing the authentication, it sends an authentication response to the client. The authentication response includes the temporary credentials. , and sending storage information to the object file storage system, where the storage information includes the temporary certificate, the file path and the second operation type. The client receives the authentication response, and based on the temporary credential, the file path and the access requirement information, accesses the first file in the object file storage system. This achieves transparent file access.

In the embodiment of the present application, since the computing engine determines the granularity of accessing the content in the first file based on the access requirement information, when the determined granularity is the first granularity, the computing engine requests the file path authentication module to access the first file for the first user. The permissions of the file path of the file are authenticated. After the authentication is passed, the temporary credentials assigned by the file path authentication module are obtained. Based on the temporary credentials, the access requirement information and the file path access the object file storage system. Since the computing engine directly accesses the object file storage system, the file reading and writing performance is improved. When the determined granularity is the second granularity, the computing engine requests the data filtering engine to access the object file storage system. The data filtering engine includes the specified administrator account information, so that the first file can be read from the object file storage system. Based on the second Granularity: segment the data that the first user needs to access from the first file, thereby providing the user with access services smaller than the file granularity, and enriching the access services provided by the user.

Referring to Figure 7, an embodiment of the present application provides a method 700 for obtaining first permission information. The first permission information in the above-mentioned embodiment shown in FIG. 1 or FIG. 3 , or the first permission information in the above-mentioned embodiment shown in FIG. 4 is obtained through the method 700 . The method 700 includes the following steps:

Step 701: The linkage permission module receives second permission information. The second permission information is used to indicate the identity of the user who can access the content in the first file and the fourth access operation.

The linkage authority module obtains the metadata report of the first file from the metadata center, and the metadata of the first file is any metadata stored in the metadata center. Obtain at least one user identity from the identity authentication center, and display a second interface to the authority administrator, where the second interface includes metadata of the first file and the at least one user identity.

In this way, the rights administrator selects the content identification information of the content in the first file from the metadata of the first file, selects a user identity that can access the content from the at least one user identity, and inputs the user identity that can access the content into the second interface. The fourth operation type of the fourth access operation of the content is used to obtain the second permission information. The second permission information includes content identification information of the content, the selected user identity and the input fourth operation type. The linkage permission module reads the second permission information from the second interface.

In some embodiments, the first file is a structured data file, and the metadata of the first file includes a file identifier of the first file and a column identifier of each column of the first file. Optionally, the content identification information of the content selected by the rights administrator includes the file identification of the first file, or the content identification information of the content selected by the rights administrator includes the file identification of the first file and at least one column in the first file. The column identification, or content identification information of the content selected by the rights administrator includes a file identification of the first file and a line number of at least one line in the first file.

In some embodiments, the first file is a semi-structured data file, and the metadata of the first file includes a file identification of the first file and identification information of each data fragment of the first file. Optionally, the content identification information of the content selected by the rights administrator includes the file identification of the first file, or the content identification information of the content selected by the rights administrator includes the file identification of the first file and at least one data in the first file. Identification information for the fragment.

Step 702: The linkage authority module generates first authority information based on the second authority information.

In step 702, the linkage permission module generates the first permission information through the following operations 7021-7023. The operations 7021-7023 are:

7021: The linkage permission module obtains the file path of the first file based on the content identification information of the content in the second permission information.

In some embodiments, the content identification information of the content includes the identification information of the first file, and the linkage permission module obtains metadata including the identification information of the first file from the metadata center, and the metadata is the metadata of the first file. , obtain the file path of the first file from the metadata of the first file.

7022: The linkage permission module maps the fourth operation type included in the second permission information to obtain the third operation type.

The access operation corresponding to the fourth operation type is the fourth access operation configured by the administrator to be able to access the content in the first file. The fourth access operation may be querying the first file, updating the first file, or deleting the first file, etc. The third operation type is an access operation corresponding to the fourth operation type that can access the object file storage system. The third operation type includes read operations and/or write operations, etc.

7023: The linkage permission module reads the user identity from the second permission information, and combines the file path of the first file, the user identity and the third operation type into the second permission information.

Step 703: The linkage authority module saves the first authority information and the second authority information.

The above-mentioned steps 701-703 can be repeatedly executed, so that the linkage authority module generates a large amount of first authority information and second authority information.

In the embodiment of this application, the linkage permission module receives the second permission information configured by the permission administrator, and generates the first permission information based on the second permission information. The first permission information is used to indicate the identity of the user who can access the file path of the first file. and access operations. Therefore, the first authority information can be automatically generated, which improves the efficiency of obtaining the first authority information and reduces the cost of obtaining the first authority information. Since the permission administrator only needs to configure the second permission information, the linkage permission module automatically generates the first permission information based on the second permission information. The second permission information is used to authenticate the user's permission to access the content in the first file. The first The permission information is used to authenticate the user's permission to access the file path of the first file. In this way, the authority administrator only needs to authorize once (configure the second authority information), and the access system uses the second authority information and the first authority information for two-dimensional authentication.

Referring to Figure 8, an embodiment of the present application provides a device 800 for accessing files. The device 800 can be deployed on the computing engine in the system shown in Figure 1 or Figure 3, or deployed on the system shown in Figure 4, Figure 5 or Figure 6 on the compute engine in the embodiment shown. The device 800 includes:

Communication unit 801, configured to receive a data access request. The data access request includes access requirement information. The access requirement information is used to indicate the content of the first file that the first user needs to access. The first file is stored in the object file storage system. ;

The processing unit 802 is configured to access the first file based on the account information of the first user and the access requirement information when it is determined that the granularity of accessing the content in the first file is the first granularity based on the access requirement information;

The processing unit 802 is also configured to access the first file at the second granularity based on the specified administrator account information and the access requirement information when it is determined that the granularity of accessing the content in the first file is the second granularity based on the access requirement information. smaller than the first particle size.

Optionally, for the detailed implementation process of the communication unit 801 receiving the data access request, please refer to the relevant content of step 401 of the embodiment shown in Figure 4, which will not be described in detail here.

Optionally, the processing unit 802 accesses the first file based on the account information of the first user and the access requirement information. For a detailed implementation process, please refer to the relevant content of steps 405-408 of the embodiment shown in Figure 4, which will not be detailed here. illustrate.

Optionally, the processing unit 802 accesses the first file based on the specified administrator account information and the access requirement information. For a detailed implementation process, please refer to the relevant content of steps 409-410 of the embodiment shown in Figure 4, which will not be detailed here. illustrate.

Optionally, the access requirement information includes identification information of the first file, and the first granularity is file granularity; or,

Optionally, the access requirement information includes identification information of the first file and identification information of a partition in the first file, and the first granularity is a partition granularity.

Optionally, the communication unit 801 is also configured to send an authentication request to the file path authentication module. The authentication request includes authentication information, and the authentication information is used to indicate the first user, the file path of the first file, and the file path of the first file. A user accesses the file path. The authentication information is obtained based on the access requirement information and the first user's account information. The authentication request is used to trigger the file path authentication module based on the first permission information and the authentication The permission information authenticates the first user's permission to access the file path using the access operation. The file path is used to indicate the storage location of the first file. The first permission information is used to indicate the identity and access of the user who can access the file path. operate;

The communication unit 801 is also used to receive an authentication response sent by the file path authentication module after the permission has been authenticated. The authentication response includes a temporary credential. The temporary credential, the file path and the operation type of the access operation are in the object. Corresponding storage in the file storage system;

The processing unit 802 is configured to access the first file based on the temporary credential, the access requirement information and the file path.

Optionally, for the detailed implementation process of the communication unit 801 sending the authentication request to the file path authentication module, please refer to the relevant content of step 404 of the embodiment shown in Figure 4, which will not be described in detail here.

Optionally, for the detailed implementation process of the communication unit 801 receiving the authentication response, please refer to the relevant content of step 408 of the embodiment shown in Figure 4, which will not be described in detail here.

Optionally, the processing unit 802 accesses the first file based on the temporary credential, the access requirement information and the file path. For a detailed implementation process, refer to the relevant content of step 408 of the embodiment shown in Figure 4, which will not be described in detail here. .

Optionally, the first file is a structured data file, and the first file uses a list form to store data. The access requirement information includes identification information of the first file and first information, and the first information is used to indicate at least one of the first files. A column and/or at least one row of the first file, and the second granularity is the column granularity; or,

The first file is a semi-structured data file. The first file includes at least one data fragment. The data fragment is used to save data with the same business attributes. The access requirement information includes identification information of the first file and one of the first files. or identification information of multiple data fragments, and the second granularity is the data fragment granularity.

Optionally, the communication unit 801 is also used to send an access instruction to the data filtering engine. The access instruction includes the access requirement information. The data filtering engine includes the administrator account information. The access instruction is used to trigger the data filtering engine based on the administrator account. Information and the access requirement information, access the first file.

Optionally, for the detailed implementation process of the communication unit 801 sending the access instruction to the data filtering engine, please refer to the relevant content of step 409 of the embodiment shown in Figure 4, which will not be described in detail here.

Optionally, the processing module 802 is also used to:

Based on the second permission information, the first user's account information and the access requirement information, authenticate the first user's permission to access the content, and the second permission information is used to indicate the identity and access operation of the user who can access the content;

After the first user's permission to access the content is authenticated, the granularity of accessing the content in the first file is determined based on the access requirement information.

Optionally, the processing unit 802 is also used to:

The first permission information is generated based on the second permission information. The first permission information is used to indicate the user identity and access operation that can access the file path of the first file. The file path is used to indicate the storage location of the first file.

Optionally, for a detailed implementation process of the processing unit 802 generating the first permission information, please refer to the relevant content of step 702 of the embodiment shown in FIG. 7, which will not be described in detail here.

In the embodiment of this application, since the first granularity is larger than the second granularity, when the determined granularity is the first granularity, the processing unit accesses the first file based on the first user's account information and the access requirement information, so that there is no need to borrow the administrator's permission. Account information is used to access the first file, thereby improving the efficiency of accessing the first file and the performance of reading and writing the first file. Because when the determined granularity of the processing unit is the second granularity, the processing unit accesses the first file based on the specified administrator account information and the access requirement information, so that the administrator account information is used to replace the first user's account information, and uses the management User account information is used to access the first file, so that there is no need to configure permissions for the first user to access the second granularity, thereby avoiding expansion of the first user's access permissions and facilitating permission management.

Referring to Figure 9, an embodiment of the present application provides a schematic diagram of a device 900 for accessing files. The device 900 may be the computing engine in any of the above embodiments, for example, it may be the computing engine provided by the embodiment shown in FIG. 1, FIG. 3, FIG. 4, FIG. 5 or FIG. 6. The device 900 includes at least one processor 901, internal connections 902, memory 903 and at least one transceiver 904.

The device 900 is a device with a hardware structure and can be used to implement the functional modules in the device 800 described in Figure 8 . For example, those skilled in the art can imagine that the processing unit 802 in the device 800 shown in Figure 8 can be implemented by calling the code in the memory 903 through the at least one processor 901, and the communication unit 801 in the device 800 shown in Figure 8 can be implemented. This is achieved through the transceiver 904.

Optionally, the device 900 can also be used to implement the functions of the computing engine in any of the above embodiments.

Optionally, the above-mentioned processor 901 can be a general central processing unit (CPU), a network processor (network processor, NP), a microprocessor, an application-specific integrated circuit (ASIC) , or one or more integrated circuits used to control the execution of the program of this application.

The internal connection 902 may include a path for transmitting information between the components. Optionally, the internal connection 902 is a single board or a bus, etc.

The above-mentioned transceiver 904 is used to communicate with other devices or communication networks.

The above-mentioned memory 903 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (random access memory, RAM) or other types that can store information and instructions. Type of dynamic storage device, it can also be electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage, optical disc Storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store the desired program code in the form of instructions or data structures and can be used by Any other media accessible by a computer, but not limited to this. The memory can exist independently and be connected to the processor through a bus. Memory can also be integrated with the processor.

Among them, the memory 903 is used to store the application program code for executing the solution of the present application, and the processor 901 controls the execution. The processor 901 is used to execute the application program code stored in the memory 903, and cooperate with at least one transceiver 904, so that the device 900 implements the functions in the patent method.

In specific implementation, as an embodiment, the processor 901 may include one or more CPUs, such as CPU0 and CPU1 in FIG. 9 .

In specific implementation, as an embodiment, the device 900 may include multiple processors, such as the processor 901 and the processor 907 in Figure 9 . Each of these processors may be a single-CPU processor or a multi-CPU processor. A processor here may refer to one or more devices, circuits, and/or processing cores for processing data (eg, computer program instructions).

Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned The storage media mentioned can be read-only memory, magnetic disks or optical disks, etc.

The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application. Inside.

Claims

A method for accessing files, characterized in that the method includes:

Receive a data access request, the data access request includes access requirement information, the access requirement information is used to indicate the content in the first file that the first user needs to access, and the first file is stored in the object file storage system;

When it is determined that the granularity of accessing the content in the first file is the first granularity based on the access requirement information, access the first file based on the account information of the first user and the access requirement information;

When it is determined based on the access requirement information that the granularity of accessing the content in the first file is the second granularity, the first file is accessed based on the specified administrator account information and the access requirement information, and the second granularity is determined based on the access requirement information. The particle size is smaller than said first particle size.
The method of claim 1, wherein the access requirement information includes identification information of the first file, and the first granularity is file granularity; or,

The access requirement information includes identification information of the first file and identification information of partitions in the first file, and the first granularity is partition granularity.
The method of claim 2, wherein accessing the first file based on the first user's account information and the access requirement information includes:

Send an authentication request to the file path authentication module, where the authentication request includes authentication information, and the authentication information is used to indicate the first user, the file path of the first file, and the first user's access For the access operation of the file path, the authentication information is obtained based on the access requirement information and the account information of the first user, and the authentication request is used to trigger the file path authentication module based on the first user's account information. The permission information and the authentication information authenticate the first user's permission to access the file path using the access operation. The file path is used to indicate the storage location of the first file. The first Permission information is used to indicate the user identity and access operations that can access the file path;

Receive an authentication response sent by the file path authentication module after passing the permission authentication. The authentication response includes a temporary credential, and the temporary credential, the file path and the operation type of the access operation are in the Corresponding storage in the above object file storage system;

Based on the temporary credentials, the access requirement information and the file path, the first file is accessed.
The method of claim 1, wherein the first file is a structured data file, the first file uses a list form to store data, and the access requirement information includes identification information of the first file. and first information, the first information is used to indicate at least one column of the first file and/or at least one row of the first file, and the second granularity is row and column granularity; or,

The first file is a semi-structured data file, the first file includes at least one data fragment, the data fragment is used to save data with the same business attributes, and the access requirement information includes the identification of the first file. information and identification information of one or more data fragments in the first file, and the second granularity is data fragment granularity.
The method of claim 4, wherein accessing the first file based on the specified administrator account information and the access requirement information includes:

Send an access instruction to the data filtering engine. The access instruction includes the access requirement information. The data filtering engine includes the administrator account information. The access instruction is used to trigger the data filtering engine based on the administrator account. information and the access requirement information to access the first file.
The method according to any one of claims 1 to 5, characterized in that the method further includes:

Based on the second permission information, the first user's account information and the access requirement information, the first user's permission to access the content is authenticated, and the second permission information is used to indicate that the first user can access the content. User identity and access operations for content;

After the first user's permission to access the content is authenticated, the granularity of accessing the content in the first file is determined based on the access requirement information.
The method of claim 6, further comprising:

First permission information is generated based on the second permission information. The first permission information is used to indicate the user identity and access operation that can access the file path of the first file. The file path is used to indicate the first The location where the file is stored.
An access system, characterized in that the system includes: a computing engine and an object file storage system;

The computing engine is configured to receive a data access request. The data access request includes access requirement information. The access requirement information is used to indicate the content of the first file that the first user needs to access. The first file is stored in In the object file storage system;

The computing engine is further configured to, when it is determined based on the access requirement information that the granularity of accessing the content in the first file is the first granularity, based on the account information of the first user and the access requirement information, access the first document;

The computing engine is further configured to, when it is determined based on the access requirement information that the granularity of accessing the content in the first file is the second granularity, access the content based on the specified administrator account information and the access requirement information. For a first file, the second granularity is smaller than the first granularity.
The system of claim 8, wherein the access requirement information includes identification information of the first file, and the first granularity is file granularity; or,

The access requirement information includes identification information of the first file and identification information of partitions in the first file, and the first granularity is partition granularity.
The system of claim 9, wherein the system further includes a file path authentication module,

The computing engine is configured to send an authentication request to the file path authentication module. The authentication request includes authentication information, and the authentication information is used to indicate the first user and the first file. The file path and the access operation of the first user to access the file path. The authentication information is obtained based on the access requirement information and the account information of the first user. The file path is used to indicate the The storage location of the first file;

The file path authentication module is used to authenticate the first user's permission to access the file path using the access operation based on the first permission information and the authentication information. The first permission information is In order to indicate the user identity and access operation that can access the file path, send an authentication response to the computing engine after passing the authentication of the permission, and the authentication response includes a temporary credential;

The object file storage system is used to correspondingly store the temporary credentials, the file path and the operation type of the access operation;

The computing engine is also configured to access the first file based on the temporary credentials, the access requirement information and the file path.
The system of claim 8, wherein the first file is a structured data file, the first file uses a list form to store data, and the access requirement information includes identification information of the first file. and first information, the first information is used to indicate at least one column of the first file and/or at least one row of the first file, and the second granularity is row and column granularity; or,

The first file is a semi-structured data file, the first file includes at least one data fragment, the data fragment is used to save data with the same business attributes, and the access requirement information includes the identification of the first file. information and identification information of one or more data fragments in the first file, and the second granularity is data fragment granularity.
The system of claim 11, wherein the system further includes a data filtering engine, and the data filtering engine includes the administrator account information;

The computing engine is configured to send an access instruction to the data filtering engine. The access instruction includes the file path of the first file and the access requirement information. The file path is used to indicate the access requirement of the first file. storage location;

The data filtering engine is used to access the first file based on the administrator account information, the file path and the access requirement information.
The system according to any one of claims 8-12, characterized in that the calculation engine is also used to:

Based on the second permission information, the first user's account information and the access requirement information, the first user's permission to access the content is authenticated, and the second permission information is used to indicate that the first user can access the content. User identity and access operations for content;

After the first user's permission to access the content is authenticated, the granularity of accessing the content in the first file is determined based on the access requirement information.
The system according to claim 13, characterized in that the system further includes a linkage authority module,

The linkage permission module is used to generate first permission information based on the second permission information. The first permission information is used to indicate the user identity and access operation that can access the file path of the first file. The path is used to indicate the storage location of the first file.
A device for accessing files, characterized in that the device includes:

A communication unit configured to receive a data access request, the data access request including access requirement information, the access requirement information being used to indicate the content of the first file that the first user needs to access, the first file being stored in the object file in the storage system;

A processing unit configured to, when it is determined based on the access requirement information that the granularity of accessing the content in the first file is a first granularity, based on the account information of the first user and the access requirement information, access the third a document;

The processing unit is further configured to, when it is determined based on the access requirement information that the granularity of accessing the content in the first file is the second granularity, access the content based on the specified administrator account information and the access requirement information. For a first file, the second granularity is smaller than the first granularity.
The device of claim 15, wherein the access requirement information includes identification information of the first file, and the first granularity is file granularity; or,

The access requirement information includes identification information of the first file and identification information of partitions in the first file, and the first granularity is partition granularity.
The device according to claim 16, characterized in that:

The communication unit is also configured to send an authentication request to the file path authentication module, where the authentication request includes authentication information, and the authentication information is used to indicate the first user, the file of the first file path and the access operation of the first user to access the file path. The authentication information is obtained based on the access requirement information and the account information of the first user. The authentication request is used to trigger the The file path authentication module authenticates the first user's permission to access the file path using the access operation based on the first permission information and the authentication information. The file path is used to indicate the first file The storage location, the first permission information is used to indicate the user identity and access operation that can access the file path;

The communication unit is also configured to receive an authentication response sent by the file path authentication module after passing the permission authentication. The authentication response includes a temporary credential, the temporary credential, the file path and the The operation type of the access operation is correspondingly stored in the object file storage system;

The processing unit is configured to access the first file based on the temporary credentials, the access requirement information and the file path.
The device of claim 15, wherein the first file is a structured data file, the first file uses a list form to store data, and the access requirement information includes identification information of the first file. and first information, the first information is used to indicate at least one column of the first file and/or at least one row of the first file, and the second granularity is row and column granularity; or,

The first file is a semi-structured data file, the first file includes at least one data fragment, the data fragment is used to save data with the same business attributes, and the access requirement information includes the identification of the first file. information and identification information of one or more data fragments in the first file, and the second granularity is data fragment granularity.
The device according to claim 18, characterized in that:

The communication unit is also used to send an access instruction to the data filtering engine. The access instruction includes the access requirement information. The data filtering engine includes the administrator account information. The access instruction is used to trigger the data. The filtering engine accesses the first file based on the administrator account information and the access requirement information.
The device according to any one of claims 15 to 19, characterized in that the processing module is also used to:

Based on the second permission information, the first user's account information and the access requirement information, the first user's permission to access the content is authenticated, and the second permission information is used to indicate that the first user can access the content. User identity and access operations for content;

After the first user's permission to access the content is authenticated, the granularity of accessing the content in the first file is determined based on the access requirement information.
The device according to claim 20, characterized in that the processing module is also used to:

First permission information is generated based on the second permission information. The first permission information is used to indicate the user identity and access operation that can access the file path of the first file. The file path is used to indicate the first The location where the file is stored.
A computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a computer, the method according to any one of claims 1-7 is implemented.
A computer program product, characterized in that the computer program product includes a computer program stored in a computer-readable storage medium, and the computing program is loaded by a processor to implement any one of claims 1-7 method described.