CN114356873A - Data sharing system and method - Google Patents

Abstract

The invention discloses a data sharing system and method, and relates to the technical field of big data. One embodiment of the system comprises: the user interaction layer is used for receiving a data processing request, wherein the data processing request comprises a data writing request, and the data writing request comprises data to be written; the data storage layer is used for persistently storing the data to be written according to the data writing request; a multi-level distributed cache layer having a plurality of cache nodes, each cache node having at least one tag; the multi-level distributed cache layer is used for performing label adaptation on data to be written, so as to determine a first target cache node from a plurality of cache nodes, and cache the data to be written to the first target cache node. The implementation method can carry out efficient data sharing by utilizing the customized and personalized tags without changing the existing storage pattern, and the management is carried out through a uniform interface, so that the details are hidden for users, and the development and the maintenance are convenient.

Description

Data sharing system and method

technical field

The invention relates to the technical field of big data, and in particular, to a data sharing system and method.

Background technique

With the advent of the era of big data, the speed of data generation is getting faster and faster, and there are more and more types. It is widely distributed inside and outside the organization, and it is constantly flowing at any time. The location of these data storage may also be located in different geographical locations. This situation brings a lot of difficulties to the data users. In addition, the data users They are usually located in different geographical locations, and the demand for data exchange, sharing and use is becoming more and more frequent, which has caused great difficulties in the efficient distribution and use of data.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present invention provide a data sharing system and method, which can use customized and personalized tags for efficient data sharing without changing the existing storage format, and realize flexible caching by dynamically combining tags Strategies to cache data in more suitable cache nodes for fast data reading and analysis, improve data sharing efficiency and data use value; and manage through a unified interface. In addition, users do not need to write code for data sharing, but can be achieved through the most common read and write commands, and details are hidden from users, which is convenient for development and maintenance.

To achieve the above purpose, according to an aspect of the embodiments of the present invention, a data sharing system is provided, including:

a user interaction layer, configured to receive a data processing request, where the data processing request includes a data writing request, and the data writing request includes data to be written;

a data storage layer, configured to persistently store the data to be written according to the data write request;

A multi-level distributed cache layer, the multi-level distributed cache layer has a plurality of cache nodes, and each of the cache nodes has at least one tag; the multi-level distributed cache layer is used to perform a Label adaptation is used to determine a first target cache node from the plurality of cache nodes, and cache the to-be-written data to the first target cache node.

Optionally, the multi-level distributed cache layer further includes a label configuration module, configured to receive label configuration information, and determine at least one label of the cache node according to the label configuration information.

Optionally, the tag configuration information includes one or more of the following: physical location of the cache node, computer room location, read and write speed, hardware configuration information, data source type, data usage, and remaining cache space.

Optionally, the multi-level distributed cache layer further includes a hierarchical management unit configured to perform hierarchical division on the cache resources of the cache node.

Optionally, the number of levels of each of the cache nodes is the same.

Optionally, the data storage layer further includes a first file system and at least one second file system, and the at least one second file system is mounted under a global directory of the first file system.

Optionally, the data processing request further includes a data reading request;

The multi-level distributed cache layer is further configured to determine a second target cache node according to the data read request, and read target data from the second target cache node.

Optionally, the system further includes a metadata service layer, configured to record a storage path and a cache path of the data to be written.

To achieve the above object, according to another aspect of the embodiments of the present invention, a data sharing method is provided. The data sharing method is applied to the data sharing system according to the embodiments of the present invention, and the data sharing method includes:

receiving a data processing request, where the data processing request includes a data write request, and the data write request includes data to be written;

According to the data writing request, persistently store the data to be written;

Perform tag adaptation on the data to be written to determine a first target cache node from multiple cache nodes, and cache the data to be written to the first target cache node; wherein the multiple cache nodes Each of the cache nodes has at least one label.

Optionally, before receiving the data processing request, the method further includes:

receiving label configuration information, and determining at least one label of the cache node according to the label configuration information.

Optionally, the tag configuration information includes one or more of the following: physical location of the cache node, computer room location, read and write speed, hardware configuration information, data source type, data usage, and remaining cache space.

Optionally, the data processing request further includes a data reading request;

The method further includes: determining a second target cache node according to the data read request, and reading target data from the second target cache node.

To achieve the above object, according to yet another aspect of the embodiments of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, when the one or more A plurality of programs are executed by the one or more processors, so that the one or more processors implement the data sharing method of the embodiment of the present invention.

In order to achieve the above object, according to another aspect of the embodiments of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the data sharing method of the embodiment of the present invention is implemented .

An embodiment of the above invention has the following advantages or beneficial effects: because the user data processing request is received through a unified user interaction layer, the user does not directly interact with the underlying data storage, does not need to understand the underlying data storage logic, and only needs to write Data writing or reading command is enough; multi-dimensional marking is performed on the cache nodes in the multi-level distributed cache layer, at least one tag of each cache node is determined, and flexible caching strategies are implemented by dynamically combining tags, so that the data Cache to a more suitable cache node to facilitate fast data reading and analysis, improve data sharing efficiency and data use value. Using the existing file system as the persistent storage layer, and mounting other file systems under the file system of the persistent storage layer, it can use customized and personalized labels to perform storage without changing the existing storage structure. Efficient data sharing, flexible caching strategy is realized through dynamic combination of tags, so that data is cached in more suitable cache nodes, so that data can be read and analyzed quickly, and data sharing efficiency and data use value are improved; and through a unified interface to manage. In addition, users do not need to write code for data sharing, but can be achieved through the most common read and write commands, and details are hidden from users, which is convenient for development and maintenance.

Further effects of the above non-conventional alternatives will be described below in conjunction with specific embodiments.

Description of drawings

The accompanying drawings are used for better understanding of the present invention and do not constitute an improper limitation of the present invention. in:

1 is a schematic structural diagram of a centralized data storage method in the prior art;

2 is a schematic structural diagram of a distributed data storage method in the prior art;

3 is a schematic structural diagram of a data sharing system according to an embodiment of the present invention;

4 is a schematic structural diagram of a data storage layer of a data sharing system according to an embodiment of the present invention;

5 is a schematic structural diagram of a data sharing system according to another embodiment of the present invention;

6 is a schematic diagram of a main flow of a data sharing method according to an embodiment of the present invention;

FIG. 7 is an exemplary system architecture diagram to which an embodiment of the present invention may be applied;

FIG. 8 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

In the big data environment, the scale and complexity of data often increase very rapidly, and the requirements for data storage systems are also getting higher and higher. At present, if data needs to be shared on a large scale and across regions, there are usually two ways: centralized storage and distributed storage. Among them, as shown in Figure 1, centralized storage needs to aggregate the shared data level by level. The data storage of each layer may be distributed in different geographical locations, and finally the data sharing area provides services. However, the data link of this method is too long, the data timeliness is poor, and the data occupies a lot of storage and computation. However, distributed storage is shown in Figure 2. Each data user will initiate a request for the data it needs according to its own needs, and then the other party will transmit the data to the local for use. Compared with centralized storage, distributed storage is more flexible and can quickly respond to needs, but data is more dependent on each other. With the growth of time, the connection between data becomes more and more closely, the dependence between data sources will become more and more difficult to manage, and the management cost will increase sharply.

In order to solve at least one of the above technical problems, the embodiment of the present invention constructs a data sharing system. There is a unified access method in the data sharing system, that is, the data sharing system provides a unified service interface for users to read and write data. In this data sharing system, there are also a variety of data storage media, and most of them are distributed file systems. These distributed file systems may be physically located in different geographical locations, and the data to be shared is also scattered in different data in the storage medium. Moreover, these different data storage media all have at least one label, and the label can be set according to different scene requirements, for example, it can be marked according to physical location, read and write speed, and the like. During the data writing operation, the tag of the storage medium is adapted according to the source, function, read and write requirements, etc. of the data to be written, so as to determine the target storage medium from multiple storage mediums. By flexibly combining tags, the data sharing system of the embodiment of the present invention can obtain an accurate caching strategy that meets the requirements of the scenario, and can meet complex data sharing requirements.

FIG. 3 is a schematic structural diagram of a data sharing system according to an embodiment of the present invention. As shown in FIG. 3 , the data sharing system 300 includes a user interaction layer 301 , a data storage layer 302 and a multi-level distributed cache layer 303 .

The user interaction layer 301 is configured to receive a data processing request, where the data processing request includes a data writing request, and the data writing request includes data to be written. The data processing request also includes a data read request. In an optional embodiment, the user interaction layer may include a user interaction interface, through which a user's request for writing or reading data is received. Users do not need to adapt or even understand the specific data access protocol and path, but only need to know the data storage path to perform data read and write operations, which greatly facilitates data maintenance.

The data storage layer 302 is configured to persistently store the data to be written according to the data writing request. In this embodiment, the data storage layer 302 is used to persistently store the data to be written, and the multi-level distributed cache layer 303 is used to cache the data in the data storage layer 302, so as to improve the speed of reading and writing data and improve the data analysis. efficiency and increase the value of data.

In an optional embodiment, the data storage layer may use any standard file system as the data storage layer, and the file system may be distributed. For example, HDFS (Hadoop Distributed File S8stem, distributed file system), NFS (Network File S8stem, network file system), Ceph, AWS S3 (Amazon Web Services Simple Storage Service, simple storage service), OSS (Object Storage Service, object storage) )Wait.

In an optional embodiment, the data storage layer 302 further includes a first file system and at least one second file system, and the at least one second file system is mounted under a global directory of the first file system. In this embodiment, as shown in FIG. 4 , after any standard file system (ie, the first file system) is used as the data storage layer, other file systems (ie, the second file system) may also be mounted to the data storage layer. Under the standard file system (ie, the first file system), the data storage layer can be expanded horizontally, and it is not necessary to make major changes to the existing architecture, which is convenient for maintenance and management. The user can access the first file system or the second file system through the user interaction layer 301. The user does not need to adapt or even understand the specific data access protocol and path, but only needs to know the data storage path to perform data read and write operations. It greatly facilitates data maintenance.

The multi-level distributed cache layer 303 is used to cache data. The multi-level distributed cache layer 303 has multiple cache nodes. The multi-level distributed cache layer 303 is configured to perform tag adaptation on the data to be written, so as to determine a first target cache node from the plurality of cache nodes, and cache the data to be written to the first target cache node. A target cache node. Among the storage resources under the data storage layer 302, there are generally additional storage resources, such as the memory of these data storage nodes (that is, on the machine where the file system is located or on the storage medium where the file system is located), high-speed solid-state hard disk area, These storage resources are independent of the file system and are managed by the host operating system where the file system is located. In this embodiment, these high-speed storage resources independent of the file system can be used as a multi-level distributed cache layer of the data sharing system. In an optional embodiment, each cache node of the multi-level distributed cache layer 303 may be further divided into levels, that is, different storage resources in the cache nodes may be regarded as different levels, for example, may be divided according to read and write speeds. As an example, the first level can read and write the fastest storage medium, such as memory, the second level is the storage medium with slower read and write speed, such as the directory corresponding to the SSD, and so on. It is worth noting that the number of layers of each cache node is consistent, but the directory corresponding to each layer may or may not be consistent. For example, the number of layers set by all cache nodes is the same, and there will not be a situation where one cache node has 3 layers and the other cache node has 4 layers; for different nodes, the set directories may be inconsistent, for example, a cache node of node A has different layers. The first-level directory is memory, the second-level directory is SSD, the first-level directory of Node B is SSD, and the second-level directory is memory.

Further, each cache node in this embodiment has at least one label. In this embodiment, it is necessary to perform multi-dimensional marking on each cache node to cache different data in different cache nodes, so as to share data in an easy-to-manage and simple manner and improve data reading. speed and use value.

In an optional embodiment, the multi-level distributed cache layer 303 further includes a tag configuration module, configured to receive tag configuration information, and determine at least one tag of the cache node according to the tag configuration information. The tag configuration information includes, but is not limited to, one or more of the following: the physical location of the cache node, the location of the computer room, the read/write speed, hardware configuration information, data source type, data usage, and remaining cache space.

As a specific example, if the label of the cache node is determined according to the physical location, the geographic location where the cache node is located may be used as the label, such as Beijing and Shanghai. If the label of the cache node is determined according to the location of the computer room, the location of the computer room where the cache node is located or the identifier of the computer room can be used as the label, such as Beijing No. 1 computer room or Beijing 0001 computer room. If the label of the cache node is determined according to the read and write speed, different read and write speeds can be set to different levels, such as high speed, medium speed, low speed, etc. The label is high speed read and write, medium speed read and write or low speed read and write. If the label of the cache node is determined according to the hardware configuration information, the system type and total storage space of the cache node may be used as the label. If the label of the cache node is determined according to the data source type, the data source or the name of the business or project related to the data is used as the label. If the label of the cache node is determined according to the use of the data, the use of the data can be used as the label, such as real-time analysis, offline calculation, item recommendation or user portrait. If the label of the cache node is determined according to the remaining cache space, the current remaining cache space size of the cache node can be used as the label, or the remaining cache space of different sizes can be divided in advance, for example, the remaining cache space larger than 5T is used as the remaining cache space. The cache space is sufficient, the remaining cache space is between 2T-5T as the remaining cache space is moderate, and the remaining cache space is less than 2T as the remaining cache space is tight. It is worth noting that the remaining cache space is automatically updated. Whenever data is written to the cache node, the remaining cache space of the cache node will decrease, and the corresponding label will also be updated.

In this embodiment, tags can be flexibly and dynamically combined, and data specified by tags is cached in a more suitable cache node to obtain better performance. For example, the data in city A is stored in HDFS, and a data analyst in city B needs to use this data for data mining, then the data sharing system in this embodiment of the present invention can cache this data on a cache node in city B or at a distance B The nearest cache node in the city to improve the data reading speed.

In an optional embodiment, after a data writing request is received, the data to be written is stored in the data storage layer 302 for persistent storage. After the data to be written is stored in the data storage layer 302 , the data to be written can be cached in an adapted cache node synchronously or asynchronously. After receiving the data read request, the multi-level distributed cache layer 303 determines a second target cache node according to the data read request, and reads target data from the second target cache node. If the target data is not queried in the second target cache node, it is read from the data storage layer 302 . That is, after receiving the data read request, data is preferentially read from the multi-level distributed cache layer 303 , and if the multi-level distributed cache layer 303 does not query corresponding data, it is read from the data storage layer 302 .

In the data sharing system of the embodiment of the present invention, the user's data processing request is received uniformly through the user interaction layer. The user does not directly interact with the underlying data storage, and does not need to understand the underlying data storage logic, and only needs to write data to write or read. The command is enough; perform multi-dimensional marking on the cache nodes in the multi-level distributed cache layer, determine at least one label of each cache node, and implement a flexible cache strategy by dynamically combining the labels, so as to cache the data to a more suitable cache In the node, it is convenient to read and analyze data quickly, and improve the efficiency of data sharing and the value of data use. The data sharing system according to the embodiment of the present invention can also mount other file systems on the data storage layer, so that customized labels can be used to perform efficient data exchange without changing the existing storage format, and a unified interface can be used to exchange data efficiently. to manage. In addition, users do not need to write code for data sharing, but can be achieved through the most common read and write commands, and the details are hidden from users, and the experience will be better.

In an optional embodiment, the data sharing system of the embodiment of the present invention may be implemented based on the open source software Alluxio. Among them, Alluxio is an open source data orchestration and storage system for hybrid cloud environments. Mounting at least one second file system in the global directory of the first file system can be implemented through the Alluxio mount command, such as alluxio fsmount/test s3a://aaa/aa. Among them, alluxio is the command name (if the data sharing system is implemented through Alluxio, it is alluxio. If it is other open source software or custom development, the command name can be set randomly and declared in the program), fs, mount All are command parameters, /test is the mount point, that is, the directory, and s3a://aaa/aa is the URL (Uniform Resource Locator) of the file system that needs to be mounted.

FIG. 5 is a schematic structural diagram of a data sharing system 500 according to another embodiment of the present invention. As shown in FIG. 5 , the data sharing system 500 includes:

The user interaction layer 501 is configured to receive a data processing request, where the data processing request includes a data writing request, and the data writing request includes data to be written;

A data storage layer 502, configured to persistently store the data to be written according to the data writing request;

A multi-level distributed cache layer, the multi-level distributed cache layer has a plurality of cache nodes, and each of the cache nodes has at least one tag; the multi-level distributed cache layer is used to perform a Label adaptation is used to determine a first target cache node from the plurality of cache nodes, and cache the to-be-written data to the first target cache node.

The metadata service layer 503 is used to record the storage path and the cache path of the data to be written.

In this embodiment, the storage locations (storage paths and cache paths) of data are uniformly managed by the metadata service layer, which is convenient for subsequent data maintenance.

FIG. 6 is a schematic diagram of the main process of a data sharing method according to an embodiment of the present invention. As shown in FIG. 6 , the method includes:

Step S601: Receive a data processing request, where the data processing request includes a data write request, and the data write request includes data to be written and a data read request;

Step S602: Persistently store the data to be written according to the data writing request;

Step S603: Perform label adaptation on the data to be written, so as to determine a first target cache node from a plurality of cache nodes, and cache the data to be written to the first target cache node; Each cache node in the plurality of cache nodes has at least one label;

Step S604: Determine a second target cache node according to the data read request, and read target data from the second target cache node.

The data sharing method according to the embodiment of the present invention is applied to the above data sharing system. When performing data writing operations, the cache address can be individually specified through different tag combinations, such as geographic location = city A, cache speed = high speed, etc., and almost arbitrary cache can be achieved by custom tags Strategy. When a read operation is performed, it is preferentially obtained from the cache.

In an optional embodiment, before receiving the data processing request, the method further includes: receiving label configuration information, and determining at least one label of the cache node according to the label configuration information. The tag configuration information includes one or more of the following: the physical location of the cache node, the location of the computer room, read and write speed, hardware configuration information, data source type, data usage and remaining cache space.

As a specific example, if the label of the cache node is determined according to the physical location, the geographic location where the cache node is located may be used as the label, such as Beijing and Shanghai. If the label of the cache node is determined according to the location of the computer room, the location of the computer room where the cache node is located or the identifier of the computer room can be used as the label, such as Beijing No. 1 computer room or Beijing 0001 computer room. If the label of the cache node is determined according to the read and write speed, different read and write speeds can be set to different levels, such as high speed, medium speed, low speed, etc. The label is high speed read and write, medium speed read and write or low speed read and write. If the label of the cache node is determined according to the hardware configuration information, the system type and total storage space of the cache node may be used as the label. If the label of the cache node is determined according to the data source type, the data source or the name of the business or project related to the data is used as the label. If the label of the cache node is determined according to the use of the data, the use of the data can be used as the label, such as real-time analysis, offline calculation, item recommendation or user portrait. If the label of the cache node is determined according to the remaining cache space, the current remaining cache space size of the cache node can be used as the label, or the remaining cache space of different sizes can be divided in advance, for example, the remaining cache space larger than 5T is used as the remaining cache space. The cache space is sufficient, the remaining cache space is between 2T-5T as the remaining cache space is moderate, and the remaining cache space is less than 2T as the remaining cache space is tight. It is worth noting that the remaining cache space is automatically updated. Whenever data is written to the cache node, the remaining cache space of the cache node will decrease, and the corresponding label will also be updated.

In this embodiment, tags can be flexibly and dynamically combined, and data specified by tags is cached in a more suitable cache node to obtain better performance.

In the data sharing method of the embodiment of the present invention, efficient data sharing is performed through customized and personalized tags, and flexible caching strategies are realized through dynamic combination of tags, so that data is cached in a more suitable cache node, so that data can be read quickly. , analysis, improve data sharing efficiency and data use value.

FIG. 7 shows an exemplary system architecture 700 of a data sharing method or data sharing system to which embodiments of the present invention may be applied.

As shown in FIG. 7 , the system architecture 700 may include terminal devices 701 , 702 , and 703 , a network 704 and a server 705 . The network 704 is the medium used to provide the communication link between the terminal devices 701 , 702 , 703 and the server 705 . Network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can use the terminal devices 701, 702, 703 to interact with the server 705 through the network 704 to receive or send messages and the like. Various communication client applications may be installed on the terminal devices 701 , 702 and 703 , such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, and the like.

The terminal devices 701, 702, 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

The server 705 may be a server that provides various services, such as a background management server that provides support for shopping websites browsed by the terminal devices 701 , 702 , and 703 . The background management server can analyze and process the received product information query request and other data, and feed back the processing results (eg, target push information, product information) to the terminal device.

It should be noted that the data sharing method provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the data sharing system is generally set in the server 705 .

It should be understood that the numbers of terminal devices, networks and servers in FIG. 7 are only illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

Referring next to FIG. 8 , it shows a schematic structural diagram of a computer system 800 suitable for implementing a terminal device according to an embodiment of the present invention. The terminal device shown in FIG. 8 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

As shown in FIG. 8, a computer system 800 includes a central processing unit (CPU) 801, which can be loaded into a random access memory (RAM) 803 according to a program stored in a read only memory (ROM) 802 or a program from a storage section 808 Instead, various appropriate actions and processes are performed. In the RAM 803, various programs and data necessary for the operation of the system 800 are also stored. The CPU 801 , the ROM 802 , and the RAM 803 are connected to each other through a bus 804 . An input/output (I/O) interface 805 is also connected to bus 804 .

The following components are connected to the I/O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc. ; and a communication section 809 including a network interface card such as a LAN card, a modem, and the like. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I/O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

In particular, the processes described above with reference to the flowcharts may be implemented as computer software programs in accordance with the disclosed embodiments of the present invention. For example, embodiments disclosed herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 809, and/or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-described functions defined in the system of the present invention are performed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The modules involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described modules can also be provided in the processor, for example, it can be described as: a processor includes a sending module, an obtaining module, a determining module and a first processing module. Among them, the names of these modules do not constitute a limitation on the unit itself under certain circumstances. For example, the sending module can also be described as "a module that sends a request for image acquisition to the connected server".

As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist alone without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by a device, the device includes:

receiving a data processing request, where the data processing request includes a data write request, and the data write request includes data to be written and a data read request;

According to the data writing request, persistently store the data to be written;

Perform tag adaptation on the data to be written to determine a first target cache node from multiple cache nodes, and cache the data to be written to the first target cache node; wherein the multiple cache nodes Each of the cache nodes has at least one label.

According to the technical solution of the embodiment of the present invention, when performing data writing operations, the cache address can be individually specified through the combination of different tags, such as geographic location=Beijing, cache speed=high speed, etc., and can be customized tags to achieve nearly arbitrary caching strategies. When a read operation is performed, it is preferentially obtained from the cache.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (14)

1. a data sharing system, is characterized in that, comprises: a user interaction layer, configured to receive a data processing request, where the data processing request includes a data writing request, and the data writing request includes data to be written; a data storage layer, configured to persistently store the data to be written according to the data write request; A multi-level distributed cache layer, the multi-level distributed cache layer has a plurality of cache nodes, and each of the cache nodes has at least one tag; the multi-level distributed cache layer is used to perform a Label adaptation is used to determine a first target cache node from the plurality of cache nodes, and cache the to-be-written data to the first target cache node. 2. The system according to claim 1, wherein the multi-level distributed cache layer further comprises a tag configuration module for: receive tag configuration information; At least one label of the cache node is determined according to the label configuration information. 3. The system according to claim 2, wherein the tag configuration information includes one or more of the following: physical location of the cache node, location of the computer room, read/write speed, hardware configuration information, data source type, data usage and remaining cache space. 4 . The system according to claim 1 , wherein the multi-level distributed cache layer further comprises a hierarchical management unit, which is used for hierarchically dividing the cache resources of the cache nodes. 5 . 5. The system according to claim 4, wherein the number of levels of each of the cache nodes is the same. 6. The system according to claim 1, wherein the data storage layer further comprises a first file system and at least one second file system, and the at least one second file system is mounted on the first file in the global directory of the system. 7. The system according to any one of claims 1-6, wherein the data processing request further comprises a data reading request; The multi-level distributed cache layer is further configured to determine a second target cache node according to the data read request, and read target data from the second target cache node. 8 . The system according to claim 7 , wherein the system further comprises a metadata service layer, configured to record the storage path and the cache path of the data to be written. 9 . 9. A data sharing method, wherein the data sharing method is applied to the data sharing system according to any one of claims 1-8, and the data sharing method comprises: receiving a data processing request, where the data processing request includes a data write request, and the data write request includes data to be written; According to the data writing request, persistently store the data to be written; Perform tag adaptation on the data to be written to determine a first target cache node from multiple cache nodes, and cache the data to be written to the first target cache node; wherein the multiple cache nodes Each of the cache nodes has at least one label. 10. The method according to claim 9, wherein before receiving the data processing request, the method further comprises: receiving label configuration information, and determining at least one label of the cache node according to the label configuration information. 11. The method according to claim 10, wherein the label configuration information comprises one or more of the following: the physical location of the cache node, the location of the computer room, read and write speed, hardware configuration information, file system type, data Source type, data usage, and remaining cache space. 12. The method according to claim 9, wherein the data processing request further comprises a data reading request; The method further includes: determining a second target cache node according to the data read request, and reading target data from the second target cache node. 13. An electronic device, characterized in that, comprising: one or more processors; storage means for storing one or more programs, The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 9-12. 14. A computer-readable medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the method according to any one of claims 9-12 is implemented.

Images