CN113392068A - Data processing method, device and system - Google Patents
Data processing method, device and system Download PDFInfo
- Publication number
- CN113392068A CN113392068A CN202110721882.0A CN202110721882A CN113392068A CN 113392068 A CN113392068 A CN 113392068A CN 202110721882 A CN202110721882 A CN 202110721882A CN 113392068 A CN113392068 A CN 113392068A
- Authority
- CN
- China
- Prior art keywords
- node
- file
- model file
- target
- attribute
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosed embodiment provides a data processing method, a device and a system, which respectively store logical relations between objects of a neural network in a data management module and a file storage module, wherein on one hand, the logical relations between the objects are stored through a tree structure, so that the hierarchical relations between the objects are reserved, different authorities can be conveniently set for different objects based on the hierarchical relations between the objects, and the flexibility of authority setting is improved; on the other hand, the model file is stored in the file storage module in a flattened mode in an object storage mode, so that the model file can be efficiently accessed. The scheme of the embodiment of the disclosure gives consideration to the flexibility of authority management and the efficiency of file access.
Description
Technical Field
The present disclosure relates to the field of data storage technologies, and in particular, to a data processing method, apparatus, and system.
Background
In the traditional full-chain business process of deep learning training, management of different types of model files and related metadata at different stages mostly depends on a lower-level file system. However, the conventional file system has difficulty in taking into account the flexibility of the rights management mechanism and the efficiency of file access.
Disclosure of Invention
The disclosure provides a data processing method, device and system.
In a first aspect, an embodiment of the present disclosure provides a data processing system, where the system includes: the data management module is used for storing a logic tree, the logic tree is established based on the hierarchical relation among objects of a neural network, one node of the logic tree corresponds to one object of the neural network, and the object comprises a model file and a model folder; and the file storage module is used for providing an object storage service so as to store the model file of the neural network, and the model file pair is associated with the node of the model file corresponding to the logic tree.
In some embodiments, the data management module is further to: storing attribute tables corresponding to the objects, wherein the attribute table of one object is associated with the node corresponding to the object.
In some embodiments, the data management module is further to: reading first attribute information of the object, and writing the first attribute information into the attribute table; and/or editing the attributes in the attribute table based on the attribute editing instruction in response to receiving the attribute editing instruction.
In some embodiments, the data management module is further to: acquiring a target data access authority of an object corresponding to a target node in the logic tree; and setting the data access authority of each node on a subtree taking the target node as a root node in the logic tree as the target data access authority.
In some embodiments, each node corresponds to a node attribute, and the node attributes include file attributes and folder attributes; the attribute of a node not associated with a model file is a folder attribute, and the attribute of a node associated with at least one model file is a file attribute.
In some embodiments, each node includes a node identifier for uniquely identifying the node, each file includes a file identifier for uniquely identifying the file; the data management module is further configured to: creating a new node in the logical tree in response to a file store instruction for a target model file and generating a node identifier for the new node; requesting a file identifier of the object model file from the file storage module based on the hash value of the object model file and the size of the object model file; and associating the node identifier of the new node with the file identifier of the target model file.
In some embodiments, the file storage module is further configured to: searching the target model file in the stored model file based on the hash value of the target model file and the size of the target model file; and if the target model file is not found, acquiring the target model file uploaded by the user, and storing the target model file.
In some embodiments, the data management module is further to: acquiring a downloading request of a target model file, wherein the downloading request carries node identification information of a node corresponding to the target model file, and the node identification information of the node is used for uniquely identifying the node; responding to the downloading request, and requesting a downloading address of the target model file from the file storage module; and pushing the download address to a user so that the user downloads the target model file.
In some embodiments, the logical tree includes a plurality of subtrees, wherein: the neural networks to which the objects corresponding to different subtrees belong are used for providing different services; or the neural networks to which the objects corresponding to different subtrees belong are used for providing services for different users.
In a second aspect, an embodiment of the present disclosure provides a data processing method, where the method includes: receiving a creation request for a target object; creating a new node in the logical tree stored in the data management module in response to the creation request; wherein the logic tree is established based on a hierarchical relationship between objects of a neural network, one node of the logic tree corresponds to one object of the neural network, the object comprises a model file and a model folder, and the model file is stored in a file storage module providing an object storage service; and associating the new node with the target object.
In some embodiments, the method further comprises: under the condition that the target object is a target model file, searching the target model file in the file storage module based on the hash value of the target model file and the size of the target model file; and if the target model file is not found, acquiring the target model file uploaded by the user, and storing the target model file to the file storage module.
In some embodiments, each object includes an attribute table, the attribute table being stored in the data management module, the attribute table of an object being associated with the node to which the object corresponds; the method further comprises the following steps: reading first attribute information of the target object, and writing the first attribute information into an attribute table of the target object; and/or in response to receiving a property editing instruction for the target object, editing the property in the property table of the target object based on the property editing instruction.
In some embodiments, the method further comprises: acquiring a target data access authority of an object corresponding to a target node in the logic tree; and setting the data access authority of each node on a subtree taking the target node as a root node in the logic tree as the target data access authority.
In some embodiments, each node corresponds to a node attribute, and the node attributes include file attributes and folder attributes; the attribute of a node not associated with a model file is a folder attribute, and the attribute of a node associated with at least one model file is a file attribute.
In some embodiments, the logical tree includes a plurality of subtrees, wherein: the neural networks to which the objects corresponding to different subtrees belong are used for providing different services; or the neural networks to which the objects corresponding to different subtrees belong are used for providing services for different users.
In a third aspect, an embodiment of the present disclosure provides a data processing method, where the method includes: acquiring a downloading request of a target model file, wherein the downloading request carries node identification information of a node corresponding to the target model file, and the node identification information of the node is used for uniquely identifying the node; responding to the downloading request, and acquiring a downloading address of the target model file from a file storage module for providing object storage service; pushing the downloading address to a user so that the user downloads the target model file; wherein the files in the file storage module are associated with nodes of a logical tree stored in a data management module, the logical tree is established based on a hierarchical relationship between objects of a neural network, one node of the logical tree corresponds to one object of the neural network, and the objects comprise model files and model folders.
In a fourth aspect, an embodiment of the present disclosure provides a data interaction apparatus, where the apparatus includes: the receiving module is used for receiving a creation request of a target object; a creation module for creating a new node in the logical tree stored in the data management module in response to the creation request; wherein the logic tree is established based on a hierarchical relationship between objects of a neural network, one node of the logic tree corresponds to one object of the neural network, the object comprises a model file and a model folder, and the model file is stored in a file storage module providing an object storage service; and the association module is used for associating the new node with the target object.
In some embodiments, the apparatus further comprises: the searching module is used for searching the target model file in the file storage module based on the hash value of the target model file and the size of the target model file under the condition that the target object is the target model file; and the storage module is used for acquiring the target model file uploaded by the user and storing the target model file to the file storage module if the target model file is not found.
In some embodiments, each object includes an attribute table, the attribute table being stored in the data management module, the attribute table of an object being associated with the node to which the object corresponds; the device further comprises: the reading module is used for reading first attribute information of the target object and writing the first attribute information into an attribute table of the target object; and/or the editing module is used for responding to the received attribute editing instruction of the target object and editing the attributes in the attribute table of the target object based on the attribute editing instruction.
In some embodiments, the apparatus further comprises: the authority acquisition module is used for acquiring the target data access authority of the object corresponding to the target node in the logic tree; and the authority setting module is used for setting the data access authority of each node on the subtree taking the target node as the root node in the logic tree as the target data access authority.
In some embodiments, each node corresponds to a node attribute, and the node attributes include file attributes and folder attributes; the attribute of a node not associated with a model file is a folder attribute, and the attribute of a node associated with at least one model file is a file attribute.
In some embodiments, the logical tree includes a plurality of subtrees, wherein: the neural networks to which the objects corresponding to different subtrees belong are used for providing different services; or the neural networks to which the objects corresponding to different subtrees belong are used for providing services for different users.
In a fifth aspect, an embodiment of the present disclosure provides a data interaction apparatus, where the apparatus includes: the system comprises a first acquisition module, a first storage module and a second acquisition module, wherein the first acquisition module is used for acquiring a downloading request of a target model file, the downloading request carries node identification information of a node corresponding to the target model file, and the node identification information of the node is used for uniquely identifying the node; a second obtaining module, configured to, in response to the download request, obtain a download address of the target model file from a file storage module configured to provide an object storage service; the pushing module is used for pushing the downloading address to a user so that the user can download the target model file; wherein the files in the file storage module are associated with nodes of a logical tree stored in a data management module, the logical tree is established based on a hierarchical relationship between objects of a neural network, one node of the logical tree corresponds to one object of the neural network, and the objects comprise model files and model folders.
In a sixth aspect, the embodiments of the present disclosure provide a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the method of any one of the embodiments.
In a seventh aspect, the embodiments of the present disclosure provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the program to implement the method according to any one of the embodiments.
According to the embodiment of the disclosure, the logical relations between the objects of the neural network are respectively stored in the data management module and the file storage module, on one hand, the logical relations between the objects are stored through the tree structure, so that the hierarchical relations between the objects are reserved, different authorities can be conveniently set for different objects based on the hierarchical relations between the objects, and the flexibility of authority setting is improved; on the other hand, the model file is stored in the file storage module in a flattened mode in an object storage mode, so that the model file can be efficiently accessed. The scheme of the embodiment of the disclosure gives consideration to the flexibility of authority management and the efficiency of file access.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.
FIG. 1A is a schematic illustration of a file-based model management approach of some embodiments.
FIG. 1B is a schematic illustration of an object-based model management approach of some embodiments.
Fig. 2A and 2B are schematic diagrams of a data processing system according to an embodiment of the present disclosure, respectively.
Fig. 2C is a schematic diagram of a storage manner of an embodiment of the present disclosure.
Fig. 3 is a schematic diagram of a setting process of data access rights according to an embodiment of the present disclosure.
Fig. 4 is a flow chart of a data processing method of an embodiment of the present disclosure.
Fig. 5 is a flowchart of a data processing method according to another embodiment of the present disclosure.
Fig. 6 is a block diagram of a data processing apparatus of an embodiment of the present disclosure.
Fig. 7 is a block diagram of a data processing apparatus of another embodiment of the present disclosure.
Fig. 8 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
In order to make the technical solutions in the embodiments of the present disclosure better understood and make the above objects, features and advantages of the embodiments of the present disclosure more comprehensible, the technical solutions in the embodiments of the present disclosure are described in further detail below with reference to the accompanying drawings.
It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.
There are two general file management methods for file systems in the related art, one is a file-based model management method, and the other is an object-based model management method. As shown in fig. 1A, in the file-based model management manner, a file or another folder may be stored in one folder, for example, folder B, folder C, and file a may be stored in folder a of the diskettes, and another file or folder may be stored in folder B, such as folder D, file B, and file C in the figure. The files themselves and the hierarchical relationship among the files are distinguished by the folder in which the files are located, so that different files have different hierarchies. However, in this way, the access efficiency of the file is low, for example, when accessing the file c, the folder a needs to be accessed first, then the folder B needs to be accessed from the folder a, and then the file c needs to be accessed from the folder B. In the object-based model management method shown in fig. 1B, files are stored in buckets (buckets) in a flattened storage manner, and the files in a bucket have no hierarchical relationship, which can improve the access efficiency of the files, but is inconvenient to set different data access rights for the files. In summary, it is difficult for the file system in the related art to consider both the flexibility of the rights management mechanism and the efficiency of file access.
Based on this, the disclosed embodiment provides a data processing system, referring to fig. 2A and 2B, the system includes:
a data management module 201, configured to store a logic tree, where the logic tree is established based on a hierarchical relationship between objects of a neural network, and one node of the logic tree corresponds to one object of the neural network, and the object includes a model file and a model folder;
a file storage module 202, configured to provide an object storage service to store a model file of the neural network, where the model file pair is associated with a node of the model file corresponding to the logical tree.
The logical tree in this embodiment is a tree structure, and includes at least one root node, and the root node may further include at least one child node. Each node corresponds to an object of the neural network. Taking the folders and files shown in fig. 1A as an example, a logical tree shown in fig. 2C may be established, where the logical tree includes a root node a, a child node B, a child node C, and a child node a corresponding to the folder a and the root node a, respectively, and the child node B further includes three child nodes, i.e., a node D, a node B, and a node C, respectively corresponding to the folder D, the file B, and the file C. By building a logical tree, the hierarchy and dependencies between individual files and folders can be represented.
The folders may be model folders in a neural network, and the files may be model files in the neural network. A root folder may generally be a large project, and folders corresponding to the project may include folders of different stages, such as a pre-training stage, a testing stage, an inference stage, a delivery deployment stage, and the like of a neural network, and each model folder may include different versions of specific model files.
The data management module only stores the logical relationship between the files and folders, not the actual files themselves. The file itself may be stored in a file storage module. The file storage module may provide an object storage service, that is, store each file in an object-based model management manner. Multiple buckets (e.g., bucket K and bucket J shown in FIG. 2B) may be included in the file storage module, and different files may be stored in different buckets. As shown in fig. 2C, the file storage module stores a file a, a file b, and a file C, and these three files are stored in the file storage module in a flattened manner, so that the access efficiency of the files is high.
The model file pairs in the file storage module are associated with the nodes of the model files corresponding to the logical trees in the data management module, so that the model files corresponding to the nodes can be found in the file storage module based on the logical trees in the data management module.
Further, the data management module is further configured to store an attribute table corresponding to each object, where the attribute table of an object is associated with a node corresponding to the object. The attribute information may include, but is not limited to, creation time, size, type (including file type and folder type) of the object corresponding to the node, and other customized attribute information. Both the folders and files may include at least one piece of attribute information, which may be stored in an attribute table. Since only files and no folders are included in the file storage module, the attribute table can be stored in the data management module. Each node may be associated with an attribute table to facilitate recording attributes of the object to which the node corresponds. The attribute information may be stored in the form of key-value pairs, where a key is an attribute name and a value is a value of an attribute.
In some embodiments, the data management module may read first attribute information of an object and write the first attribute information to an attribute table of the object. The first attribute information may be inherent attributes of the object, such as creation time, size, type, etc., which may be attribute information included in each object. The data management module may read these inherent attributes directly and write them to the attribute table.
In other embodiments, the data management module may further edit the attributes in the attribute table based on the attribute editing instructions in response to receiving the attribute editing instructions. The editing includes adding, deleting, and modifying properties. Different nodes may include different attributes, e.g., version number of the delivery model, chip type. By the method, various attribute information required by the user can be recorded in the attribute table according to actual needs, and the attribute table of each object can be conveniently expanded and edited.
Each node may include a globally unique node Identifier (ID) with which the corresponding attribute table may be conveniently stored and looked up. For example, when the attribute table of the file a needs to be searched, the ID of the node a is obtained, and the data management module performs searching based on the ID of the node a (assumed to be ID2 shown in fig. 2B), so as to obtain attribute key-value pairs such as { key21, value21}, { key22, value22 }. Furthermore, when new attribute information needs to be added to the file a, the ID of the node a is also acquired, the corresponding attribute table is found in the data management module based on the ID of the node a (assumed to be ID2 shown in fig. 2B), and a newly added attribute key value pair, for example, { key23, value23} is inserted into the found attribute table.
In some embodiments, the data management module is further configured to obtain a target data access permission of an object corresponding to a target node in the logical tree; and setting the data access authority of each node on a subtree taking the target node as a root node in the logic tree as the target data access authority. The permissions may include, but are not limited to, viewing an object or attribute table corresponding to the node (assumed to be permission 1), downloading a file corresponding to the node (assumed to be permission 2), modifying or deleting the node and an object corresponding to the node (assumed to be permission 3), and the like. Assuming that a node P is a root node of a subtree T in the logical tree, the permissions of all nodes on the subtree T can be set by setting the permissions of the node P, that is, the permissions of all nodes on the subtree can be automatically covered by the permissions of the node P, so that the permissions of other nodes on the subtree T do not need to be set one by one. In this way, the efficiency of authority setting is improved.
As shown in fig. 3, the authority of the node P0 on the edit tree may be set to AC0, so that the authority of each node on the subtree T0 with the node P0 as the root node is set to AC 0. Further, the authority of the node P1 on the logical tree may be individually set to AC1, so that the authority of each node on the subtree T1 with the node P1 as the root node is set to AC1, while the authority of each node not on the subtree T1 remains as AC 0. Further, the authority of the node P2 on the logical tree may be individually set to AC2, so that the authority of each node on the subtree T1 with the node P2 as the root node is set to AC2, while the authority of each node not on the subtree T2 remains as AC0 or AC 1. Different authorities are marked by different colors in the graph, and it can be seen that different authorities can be set for different nodes in the logic tree respectively through the setting mode, batch setting is supported by taking subtrees as units, and the authority setting mode is flexible and efficient.
In some embodiments, different permissions correspond to different permission levels. For example, the authority level of authority 3 is higher than that of authority 2, and the authority level of authority 2 is higher than that of authority 1. Assuming that the original privilege level of a node is privilege level ACi, the privilege level of the node can only be changed to a privilege level higher than privilege level ACi. For example, in the embodiment shown in FIG. 3, the original privilege level of node P1 is AC0, then the modified privilege level of node P1, AC1, should be higher than AC 0. That is, the privilege level of a node is not higher than the privilege levels of its children nodes.
In this case, the authority of each node in the logical tree may be checked upward. And if the permission level set for the child node of the node in the received permission setting instruction is lower than the permission level of the node, returning error information, otherwise, setting the corresponding permission for the child node of the node based on the permission setting instruction.
In some embodiments, each node corresponds to a node attribute, which includes a file attribute and a folder attribute. The attribute of the node which is not associated with the model file is a folder attribute, and the attribute of the node which is associated with at least one model file is a file attribute. Because only files are stored in the file storage module and folders are not stored in the file storage module, objects corresponding to nodes can be distinguished as files by setting different attributes for the nodes, and therefore the objects corresponding to the corresponding nodes can be stored in the file storage module.
In some embodiments, each node includes a node identifier, such as id1, id2 shown in FIG. 2B, for uniquely identifying the node; each file includes a file identifier, such as fid1, fid2, and fid3 shown in FIG. 2B, for uniquely identifying the file. The data management module may upload and download files based on the node identifiers and the file identifiers.
In the process of uploading the file, the data management module firstly responds to a file storage instruction of the target model file, creates a new node of the file type in the logic tree and generates a node identifier of the new node. The file identifier of the object model file is then requested from the file storage module based on the hash value of the object model file (e.g., the MD5 value, which may be generated by a file digest hashing algorithm) and the size of the object model file. The node identifier of the new node is then associated with the file identifier of the target model file. By the method, the decoupling uploading of the logical relationship of the file and the file is realized, so that the file can be stored in a flat object storage mode, and the logical relationship of the file can be stored in a tree structure with distinct hierarchies.
It should be noted that the same model file can be referenced by multiple nodes in the logical tree, and when referenced by different nodes, different file identifiers can be generated for the file. For example, if a node with node identifier id2 and a node with node identifier id3 each request file a from the file storage module, the file storage module may generate two different file identifiers for file a, as shown in fid1 and fid2 in FIG. 2B, and return fid1 and fid2 to the node with node identifier id2 and the node with node identifier id3 in the data management module, respectively.
Further, the file storage module may search the object model file in the stored model files based on the hash value of the object model file and the size of the object model file; and if the target model file is not found, acquiring the target model file uploaded by the user, and storing the target model file. In this embodiment, the target model file only needs to be uploaded if the target model file is not found in the file storage module based on the hash value of the file and the size of the target model file, otherwise, the target model file does not need to be uploaded repeatedly, and the deduplication processing of the target model file is realized. A link can be returned to a User Interface (UI) of a User through an Application Programming Interface (API) of the data management module, and the link is used for uploading the target model file.
In the process of downloading the file, the data management module may obtain a downloading request of a target model file, where the downloading request carries node identification information of a node corresponding to the target model file, and the node identification information of the node is used to uniquely identify the node; responding to the downloading request, and requesting a downloading address of the target model file from the file storage module; and pushing the download address to a user so that the user downloads the target model file. By the method, the file can be downloaded based on the logical relation of the file under the condition that the logical relation of the file is decoupled from the file.
In some embodiments, the logical tree includes a plurality of subtrees, wherein neural networks to which objects corresponding to different subtrees belong are used to provide different services; or the neural networks to which the objects corresponding to different subtrees belong are used for providing services for different users. Different subtrees may belong to the same root node. Different users may have different rights to different subtrees. In this way, different permissions can be set for different users, respectively.
The file, the folder and the like are abstracted into nodes in a file system, are defined into different types, are endowed with globally unique node identifiers, and are managed and maintained by a relational database in a data management module. Meanwhile, in the data management module, the attribute table of the metadata of the node is managed and maintained by combining the node identifier.
For the nodes of the file type, the real file data is stored in a file storage module in an object storage mode, object storage management service is provided through a data management module and a tree structure at the upper layer of object storage, the maintenance and the configuration management of the meta information of the file object are realized, the globally unique file identifier is associated with the node identifier of the node of the file type, and the file data is uploaded and downloaded by an API of the object storage management service provided by the data management module.
The method separates node service of a tree structure managed by a nested model from file service stored based on an object, defines types and attributes for the node service of the tree structure to flexibly manage metadata related to a neural network, can conveniently manage files and folders in the neural network in a pre-training stage, a training stage and a delivery stage of different versions under different business scenes in a multi-level classification mode by a user based on the scheme of the embodiment, simultaneously sets different attributes for the model folders and the model files by using an extended metadata management method, sets different file views through the attributes to view and mark the neural network, and automatically produces, manages and uploads and downloads the model files of the neural network by using an API managed by the neural network. The embodiment of the disclosure improves the management automation capability of the neural network, reduces the communication cost in the neural network delivery process, shortens the source tracing path of the neural network, and improves the production and delivery efficiency of the neural network.
As shown in fig. 4, an embodiment of the present disclosure further provides a data processing method, where the method includes:
step 401: receiving a creation request for a target object;
step 402: creating a new node in the logical tree stored in the data management module in response to the creation request; wherein the logic tree is established based on a hierarchical relationship between objects of a neural network, one node of the logic tree corresponds to one object of the neural network, the object comprises a model file and a model folder, and the model file is stored in a file storage module providing an object storage service;
step 403: and associating the new node with the target object.
In some embodiments, the method further comprises: under the condition that the target object is a target model file, searching the target model file in the file storage module based on the hash value of the target model file and the size of the target model file; and if the target model file is not found, acquiring the target model file uploaded by the user, and storing the target model file to the file storage module.
In some embodiments, each object includes an attribute table, the attribute table being stored in the data management module, the attribute table of an object being associated with the node to which the object corresponds; the method further comprises the following steps: reading first attribute information of the target object, and writing the first attribute information into an attribute table of the target object; and/or in response to receiving a property editing instruction for the target object, editing the property in the property table of the target object based on the property editing instruction.
In some embodiments, the method further comprises: acquiring a target data access authority of an object corresponding to a target node in the logic tree; and setting the data access authority of each node on a subtree taking the target node as a root node in the logic tree as the target data access authority.
In some embodiments, each node corresponds to a node attribute, and the node attributes include file attributes and folder attributes; the attribute of a node not associated with a model file is a folder attribute, and the attribute of a node associated with at least one model file is a file attribute.
In some embodiments, the logical tree includes a plurality of subtrees, wherein: the neural networks to which the objects corresponding to different subtrees belong are used for providing different services; or the neural networks to which the objects corresponding to different subtrees belong are used for providing services for different users.
The method of the embodiment of the present disclosure may be used to store the model file in a file storage module of the aforementioned data processing system, and store the hierarchical relationship between the stored model file and other model files and model folders of the neural network in a data management module of the aforementioned data processing system. Details of the embodiments of the present disclosure are given in the foregoing embodiments of the data processing system, and are not described herein again.
As shown in fig. 5, an embodiment of the present disclosure further provides a data processing method, where the method includes:
step 501: acquiring a downloading request of a target model file, wherein the downloading request carries node identification information of a node corresponding to the target model file, and the node identification information of the node is used for uniquely identifying the node;
step 502: responding to the downloading request, and acquiring a downloading address of the target model file from a file storage module for providing object storage service;
step 503: pushing the downloading address to a user so that the user downloads the target model file;
wherein the files in the file storage module are associated with nodes of a logical tree stored in a data management module, the logical tree is established based on a hierarchical relationship between objects of a neural network, one node of the logical tree corresponds to one object of the neural network, and the objects comprise model files and model folders.
The method of the embodiments of the present disclosure may be used to download the model file from the file storage module of the data processing system, the model file may be stored into the file storage module of the data processing system based on the data processing method in any of the embodiments, and the hierarchical relationship between the model file and other model files and model folders of the neural network may be stored into the data management module of the data processing system based on the data processing method in any of the embodiments. Details of the embodiments of the present disclosure are given in the foregoing embodiments of the data processing system and the data processing method, and are not described herein again.
It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.
As shown in fig. 6, an embodiment of the present disclosure further provides a data interaction apparatus, where the apparatus includes:
a receiving module 601, configured to receive a creation request for a target object;
a creating module 602, configured to create a new node in the logical tree stored in the data management module in response to the creation request; wherein the logic tree is established based on a hierarchical relationship between objects of a neural network, one node of the logic tree corresponds to one object of the neural network, the object comprises a model file and a model folder, and the model file is stored in a file storage module providing an object storage service;
an associating module 603, configured to associate the new node with the target object.
In some embodiments, the apparatus further comprises: the searching module is used for searching the target model file in the file storage module based on the hash value of the target model file and the size of the target model file under the condition that the target object is the target model file; and the storage module is used for acquiring the target model file uploaded by the user and storing the target model file to the file storage module if the target model file is not found.
In some embodiments, each object includes an attribute table, the attribute table being stored in the data management module, the attribute table of an object being associated with the node to which the object corresponds; the device further comprises: the reading module is used for reading first attribute information of the target object and writing the first attribute information into an attribute table of the target object; and/or the editing module is used for responding to the received attribute editing instruction of the target object and editing the attributes in the attribute table of the target object based on the attribute editing instruction.
In some embodiments, the apparatus further comprises: the authority acquisition module is used for acquiring the target data access authority of the object corresponding to the target node in the logic tree; and the authority setting module is used for setting the data access authority of each node on the subtree taking the target node as the root node in the logic tree as the target data access authority.
In some embodiments, each node corresponds to a node attribute, and the node attributes include file attributes and folder attributes; the attribute of a node not associated with a model file is a folder attribute, and the attribute of a node associated with at least one model file is a file attribute.
In some embodiments, the logical tree includes a plurality of subtrees, wherein: the neural networks to which the objects corresponding to different subtrees belong are used for providing different services; or the neural networks to which the objects corresponding to different subtrees belong are used for providing services for different users.
As shown in fig. 7, an embodiment of the present disclosure further provides a data interaction apparatus, where the apparatus includes:
a first obtaining module 701, configured to obtain a download request of a target model file, where the download request carries node identification information of a node corresponding to the target model file, and the node identification information of the node is used to uniquely identify the node;
a second obtaining module 702, configured to, in response to the download request, obtain a download address of the target model file from a file storage module for providing an object storage service;
a pushing module 703, configured to push the download address to a user, so that the user downloads the target model file;
wherein the files in the file storage module are associated with nodes of a logical tree stored in a data management module, the logical tree is established based on a hierarchical relationship between objects of a neural network, one node of the logical tree corresponds to one object of the neural network, and the objects comprise model files and model folders.
In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.
Embodiments of the present specification also provide a computer device, which at least includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method according to any of the foregoing embodiments when executing the program.
Fig. 8 is a schematic diagram illustrating a more specific hardware structure of a computing device according to an embodiment of the present disclosure, where the computing device may include: a processor 801, a memory 802, an input/output interface 803, a communication interface 804, and a bus 805. Wherein the processor 801, the memory 802, the input/output interface 803 and the communication interface 804 are communicatively connected to each other within the device via a bus 805.
The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present specification. The processor 801 may further include a graphics card, which may be an Nvidia titan X graphics card or a 1080Ti graphics card, etc.
The Memory 802 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 802 may store an operating system and other application programs, and when the technical solution provided by the embodiments of the present specification is implemented by software or firmware, the relevant program codes are stored in the memory 802 and called to be executed by the processor 801.
The input/output interface 803 is used for connecting an input/output module to realize information input and output. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The communication interface 804 is used for connecting a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
It should be noted that although the above-mentioned device only shows the processor 801, the memory 802, the input/output interface 803, the communication interface 804 and the bus 805, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present description, and not necessarily all of the components shown in the figures.
The embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the method of any of the foregoing embodiments.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
From the above description of the embodiments, it is clear to those skilled in the art that the embodiments of the present disclosure can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the embodiments of the present specification may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments of the present specification.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus embodiment, since it is substantially similar to the method embodiment, it is relatively simple to describe, and reference may be made to some descriptions of the method embodiment for relevant points. The above-described apparatus embodiments are merely illustrative, and the modules described as separate components may or may not be physically separate, and the functions of the modules may be implemented in one or more software and/or hardware when implementing the embodiments of the present disclosure. And part or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The foregoing is only a specific embodiment of the embodiments of the present disclosure, and it should be noted that, for those skilled in the art, a plurality of modifications and decorations can be made without departing from the principle of the embodiments of the present disclosure, and these modifications and decorations should also be regarded as the protection scope of the embodiments of the present disclosure.
Claims (20)
1. A data processing system, characterized in that the system comprises:
the data management module is used for storing a logic tree, the logic tree is established based on the hierarchical relation among objects of a neural network, one node of the logic tree corresponds to one object of the neural network, and the object comprises a model file and a model folder;
and the file storage module is used for providing an object storage service so as to store the model file of the neural network, and the model file pair is associated with the node of the model file corresponding to the logic tree.
2. The system of claim 1, wherein the data management module is further configured to:
storing attribute tables corresponding to the objects, wherein the attribute table of one object is associated with the node corresponding to the object.
3. The system of claim 2, wherein the data management module is further configured to: reading first attribute information of the object, and writing the first attribute information into the attribute table; and/or
In response to receiving a property editing instruction, editing the property in the property table based on the property editing instruction.
4. The system of any of claims 1-3, wherein the data management module is further configured to:
acquiring a target data access authority of an object corresponding to a target node in the logic tree;
and setting the data access authority of each node on a subtree taking the target node as a root node in the logic tree as the target data access authority.
5. The system according to any one of claims 1-4, wherein each node corresponds to a node attribute, and the node attributes comprise file attributes and folder attributes;
the attribute of a node not associated with a model file is a folder attribute, and the attribute of a node associated with at least one model file is a file attribute.
6. The system of any of claims 1-5, wherein each node includes a node identifier for uniquely identifying the node, and each file includes a file identifier for uniquely identifying the file; the data management module is further configured to:
creating a new node in the logical tree in response to a file store instruction for a target model file and generating a node identifier for the new node;
requesting a file identifier of the object model file from the file storage module based on the hash value of the object model file and the size of the object model file;
and associating the node identifier of the new node with the file identifier of the target model file.
7. The system of claim 6, wherein the file storage module is further configured to:
searching the target model file in the stored model file based on the hash value of the target model file and the size of the target model file;
and if the target model file is not found, acquiring the target model file uploaded by the user, and storing the target model file.
8. The system of any of claims 1-7, wherein the data management module is further configured to:
acquiring a downloading request of a target model file, wherein the downloading request carries node identification information of a node corresponding to the target model file, and the node identification information of the node is used for uniquely identifying the node;
responding to the downloading request, and requesting a downloading address of the target model file from the file storage module;
and pushing the download address to a user so that the user downloads the target model file.
9. The system of any of claims 1-8, wherein the logical tree comprises a plurality of subtrees, wherein:
the neural networks to which the objects corresponding to different subtrees belong are used for providing different services; or
And the neural networks to which the objects corresponding to the different subtrees belong are used for providing services for different users.
10. A method of data processing, the method comprising:
receiving a creation request for a target object;
creating a new node in the logical tree stored in the data management module in response to the creation request; wherein the logic tree is established based on a hierarchical relationship between objects of a neural network, one node of the logic tree corresponds to one object of the neural network, the object comprises a model file and a model folder, and the model file is stored in a file storage module providing an object storage service;
and associating the new node with the target object.
11. The method of claim 10, further comprising:
under the condition that the target object is a target model file, searching the target model file in the file storage module based on the hash value of the target model file and the size of the target model file;
and if the target model file is not found, acquiring the target model file uploaded by the user, and storing the target model file to the file storage module.
12. The method according to claim 10 or 11, wherein each object comprises an attribute table, the attribute table being stored in the data management module, the attribute table of an object being associated with the node to which the object corresponds; the method further comprises the following steps:
reading first attribute information of the target object, and writing the first attribute information into an attribute table of the target object; and/or
In response to receiving a property editing instruction for the target object, editing a property in a property table of the target object based on the property editing instruction.
13. The method according to any one of claims 10-12, further comprising:
acquiring a target data access authority of an object corresponding to a target node in the logic tree;
and setting the data access authority of each node on a subtree taking the target node as a root node in the logic tree as the target data access authority.
14. The method according to any one of claims 10-13, wherein each node corresponds to a node attribute, and the node attributes include file attributes and folder attributes;
the attribute of a node not associated with a model file is a folder attribute, and the attribute of a node associated with at least one model file is a file attribute.
15. The method of any of claims 10-14, wherein the logical tree comprises a plurality of subtrees, wherein:
the neural networks to which the objects corresponding to different subtrees belong are used for providing different services; or
And the neural networks to which the objects corresponding to the different subtrees belong are used for providing services for different users.
16. A method of data processing, the method comprising:
acquiring a downloading request of a target model file, wherein the downloading request carries node identification information of a node corresponding to the target model file, and the node identification information of the node is used for uniquely identifying the node;
responding to the downloading request, and acquiring a downloading address of the target model file from a file storage module for providing object storage service;
pushing the downloading address to a user so that the user downloads the target model file;
wherein the files in the file storage module are associated with nodes of a logical tree stored in a data management module, the logical tree is established based on a hierarchical relationship between objects of a neural network, one node of the logical tree corresponds to one object of the neural network, and the objects comprise model files and model folders.
17. A data interaction apparatus, the apparatus comprising:
the receiving module is used for receiving a creation request of a target object;
a creation module for creating a new node in the logical tree stored in the data management module in response to the creation request; wherein the logic tree is established based on a hierarchical relationship between objects of a neural network, one node of the logic tree corresponds to one object of the neural network, the object comprises a model file and a model folder, and the model file is stored in a file storage module providing an object storage service;
and the association module is used for associating the new node with the target object.
18. A data interaction apparatus, the apparatus comprising:
the system comprises a first acquisition module, a first storage module and a second acquisition module, wherein the first acquisition module is used for acquiring a downloading request of a target model file, the downloading request carries node identification information of a node corresponding to the target model file, and the node identification information of the node is used for uniquely identifying the node;
a second obtaining module, configured to, in response to the download request, obtain a download address of the target model file from a file storage module configured to provide an object storage service;
the pushing module is used for pushing the downloading address to a user so that the user can download the target model file;
the files in the file storage module are associated with nodes of a logical tree stored in a data management module, the logical tree is established based on a hierarchical relationship between objects of a neural network, one node of the logical tree corresponds to one object of the neural network, and the objects include model files and model folders.
19. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the method of any one of claims 10 to 16.
20. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any one of claims 10 to 16 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110721882.0A CN113392068A (en) | 2021-06-28 | 2021-06-28 | Data processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110721882.0A CN113392068A (en) | 2021-06-28 | 2021-06-28 | Data processing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113392068A true CN113392068A (en) | 2021-09-14 |
Family
ID=77624371
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110721882.0A Pending CN113392068A (en) | 2021-06-28 | 2021-06-28 | Data processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113392068A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114168536A (en) * | 2021-11-27 | 2022-03-11 | 深圳市连用科技有限公司 | Method for uploading file and terminal equipment |
| CN114254068A (en) * | 2022-02-28 | 2022-03-29 | 杭州未名信科科技有限公司 | Data transfer method and system |
| US20220198446A1 (en) * | 2019-04-11 | 2022-06-23 | Aleksey Vladislavovich POTANIN | Method for Recording Data Related to Product Manufacture and Sales, and Related System |
Citations (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6098072A (en) * | 1997-03-27 | 2000-08-01 | International Business Machines Corporation | Source code files in a file directory system having multiple hierarchies representing contextual views |
| CN101127067A (en) * | 2007-09-13 | 2008-02-20 | 深圳市融合视讯科技有限公司 | Electronic document attribute dynamic setting method |
| CN101141476A (en) * | 2007-10-09 | 2008-03-12 | 创新科存储技术（深圳）有限公司 | File storing, downloading method and device |
| CN101699822A (en) * | 2009-08-06 | 2010-04-28 | 腾讯科技（深圳）有限公司 | File uploading method and device, and mass storage system |
| CN102243660A (en) * | 2011-07-18 | 2011-11-16 | 中兴通讯股份有限公司 | Data access method and device |
| CN103731483A (en) * | 2013-12-25 | 2014-04-16 | 侯金涛 | Virtual file system based on cloud computing |
| CN105468689A (en) * | 2015-11-17 | 2016-04-06 | 广东电网有限责任公司电力科学研究院 | Power grid object level authority configuration and inheritance method |
| US20160323359A1 (en) * | 2015-04-30 | 2016-11-03 | Brandon Camping | Electronic file transfer and modification system and method |
| US20170091296A1 (en) * | 2015-09-25 | 2017-03-30 | Netapp, Inc. | Object storage backed file system |
| US20170270138A1 (en) * | 2015-06-30 | 2017-09-21 | Yandex Europe Ag | Method and system for managing data associated with a hierarchical structure |
| US20170286707A1 (en) * | 2016-03-30 | 2017-10-05 | International Business Machines Corporation | Unified file and object storage architecture for clustered file systems |
| US20180113862A1 (en) * | 2014-12-29 | 2018-04-26 | Workshare, Ltd. | Method and System for Electronic Document Version Tracking and Comparison |
| CN109471894A (en) * | 2018-10-29 | 2019-03-15 | 深圳市瑞驰信息技术有限公司 | A kind of system and method for novel decentralized file and the unified storage of object |
| CN110402441A (en) * | 2017-04-21 | 2019-11-01 | 谷歌有限责任公司 | Quote accesses control list |
| CN110569657A (en) * | 2019-09-10 | 2019-12-13 | 北京字节跳动网络技术有限公司 | Data access method, device, equipment and storage medium |
| CN111427841A (en) * | 2020-02-26 | 2020-07-17 | 平安科技（深圳）有限公司 | Data management method and device, computer equipment and storage medium |
| US20200265023A1 (en) * | 2019-02-19 | 2020-08-20 | Oracle International Corporation | System for transition from a hierarchical file system to an object store |
| CN111611220A (en) * | 2019-02-26 | 2020-09-01 | 宁波创元信息科技有限公司 | File sharing method and system based on hierarchical nodes |
| CN112231291A (en) * | 2019-07-15 | 2021-01-15 | 广联达科技股份有限公司 | Method and device for multi-branch version management of cloud files |
| CN112236758A (en) * | 2018-05-31 | 2021-01-15 | 微软技术许可有限责任公司 | Cloud storage distributed file system |
| US20210019285A1 (en) * | 2019-07-16 | 2021-01-21 | Citrix Systems, Inc. | File download using deduplication techniques |
-
2021
- 2021-06-28 CN CN202110721882.0A patent/CN113392068A/en active Pending
Patent Citations (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6098072A (en) * | 1997-03-27 | 2000-08-01 | International Business Machines Corporation | Source code files in a file directory system having multiple hierarchies representing contextual views |
| CN101127067A (en) * | 2007-09-13 | 2008-02-20 | 深圳市融合视讯科技有限公司 | Electronic document attribute dynamic setting method |
| CN101141476A (en) * | 2007-10-09 | 2008-03-12 | 创新科存储技术（深圳）有限公司 | File storing, downloading method and device |
| CN101699822A (en) * | 2009-08-06 | 2010-04-28 | 腾讯科技（深圳）有限公司 | File uploading method and device, and mass storage system |
| CN102243660A (en) * | 2011-07-18 | 2011-11-16 | 中兴通讯股份有限公司 | Data access method and device |
| CN103731483A (en) * | 2013-12-25 | 2014-04-16 | 侯金涛 | Virtual file system based on cloud computing |
| US20180113862A1 (en) * | 2014-12-29 | 2018-04-26 | Workshare, Ltd. | Method and System for Electronic Document Version Tracking and Comparison |
| US20160323359A1 (en) * | 2015-04-30 | 2016-11-03 | Brandon Camping | Electronic file transfer and modification system and method |
| US20170270138A1 (en) * | 2015-06-30 | 2017-09-21 | Yandex Europe Ag | Method and system for managing data associated with a hierarchical structure |
| US20170091296A1 (en) * | 2015-09-25 | 2017-03-30 | Netapp, Inc. | Object storage backed file system |
| CN105468689A (en) * | 2015-11-17 | 2016-04-06 | 广东电网有限责任公司电力科学研究院 | Power grid object level authority configuration and inheritance method |
| US20170286707A1 (en) * | 2016-03-30 | 2017-10-05 | International Business Machines Corporation | Unified file and object storage architecture for clustered file systems |
| CN110402441A (en) * | 2017-04-21 | 2019-11-01 | 谷歌有限责任公司 | Quote accesses control list |
| CN112236758A (en) * | 2018-05-31 | 2021-01-15 | 微软技术许可有限责任公司 | Cloud storage distributed file system |
| CN109471894A (en) * | 2018-10-29 | 2019-03-15 | 深圳市瑞驰信息技术有限公司 | A kind of system and method for novel decentralized file and the unified storage of object |
| US20200265023A1 (en) * | 2019-02-19 | 2020-08-20 | Oracle International Corporation | System for transition from a hierarchical file system to an object store |
| CN111611220A (en) * | 2019-02-26 | 2020-09-01 | 宁波创元信息科技有限公司 | File sharing method and system based on hierarchical nodes |
| CN112231291A (en) * | 2019-07-15 | 2021-01-15 | 广联达科技股份有限公司 | Method and device for multi-branch version management of cloud files |
| US20210019285A1 (en) * | 2019-07-16 | 2021-01-21 | Citrix Systems, Inc. | File download using deduplication techniques |
| CN110569657A (en) * | 2019-09-10 | 2019-12-13 | 北京字节跳动网络技术有限公司 | Data access method, device, equipment and storage medium |
| CN111427841A (en) * | 2020-02-26 | 2020-07-17 | 平安科技（深圳）有限公司 | Data management method and device, computer equipment and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220198446A1 (en) * | 2019-04-11 | 2022-06-23 | Aleksey Vladislavovich POTANIN | Method for Recording Data Related to Product Manufacture and Sales, and Related System |
| CN114168536A (en) * | 2021-11-27 | 2022-03-11 | 深圳市连用科技有限公司 | Method for uploading file and terminal equipment |
| CN114254068A (en) * | 2022-02-28 | 2022-03-29 | 杭州未名信科科技有限公司 | Data transfer method and system |
| CN114254068B (en) * | 2022-02-28 | 2022-08-09 | 杭州未名信科科技有限公司 | Data transfer method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113392068A (en) | Data processing method, device and system | |
| US9009108B2 (en) | Minimal extensions required for multi-master offline and collaboration for devices and web services | |
| CN100468402C (en) | Sort data storage and split catalog inquiry method based on catalog tree | |
| TWI420328B (en) | Device specific content indexing for optimized device operation | |
| US8108360B2 (en) | Database object update order determination | |
| CN108470040B (en) | Method and device for warehousing unstructured data | |
| CN108255915B (en) | File management method and device and machine-readable storage medium | |
| TWI550513B (en) | Brokered item access for isolated applications | |
| TW202027455A (en) | Method and apparatus for performing evidence storage on structured work based on blockchain | |
| CN105808428A (en) | Method for performing unified performance test on distributed file system | |
| CN107016047A (en) | Document query, document storing method and device | |
| CN103914290A (en) | Operating command processing method and device | |
| US10691639B1 (en) | Hybrid metadata and folder based file access | |
| CN113704248B (en) | Block chain query optimization method based on external index | |
| CN111158650A (en) | Report template, report template and report generation method and device | |
| CN105843809B (en) | Data processing method and device | |
| CN112835638A (en) | Configuration information management method and device based on embedded application program | |
| CN111427863A (en) | Data migration method, device and equipment based on domain model | |
| CN115543428A (en) | Simulated data generation method and device based on strategy template | |
| CN109683887A (en) | A kind of construction method and device for supporting the customized web project of multi-scheme | |
| CN1828596A (en) | File system represented inside a database | |
| CN112861185A (en) | Data automatic deformation transmission method based on Hive data warehouse | |
| CN112182115A (en) | Relationship display method and device, storage medium and electronic device | |
| CN112948593A (en) | Knowledge graph generation method, device, equipment and medium | |
| Brocco | Delta-State JSON CRDT: Putting Collaboration on Solid Ground |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |