CN113311999A - Object storage device and method based on intelligent campus middleboxes - Google Patents
Object storage device and method based on intelligent campus middleboxes Download PDFInfo
- Publication number
- CN113311999A CN113311999A CN202110609147.0A CN202110609147A CN113311999A CN 113311999 A CN113311999 A CN 113311999A CN 202110609147 A CN202110609147 A CN 202110609147A CN 113311999 A CN113311999 A CN 113311999A
- Authority
- CN
- China
- Prior art keywords
- storage
- file
- data
- node information
- proxy server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 192
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000005520 cutting process Methods 0.000 claims abstract description 21
- 230000011218 segmentation Effects 0.000 claims abstract description 16
- 238000004891 communication Methods 0.000 claims abstract description 9
- 238000004458 analytical method Methods 0.000 claims description 14
- 238000004422 calculation algorithm Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 7
- 238000005192 partition Methods 0.000 claims description 2
- 230000007246 mechanism Effects 0.000 abstract description 6
- 230000008569 process Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 9
- 238000007726 management method Methods 0.000 description 7
- 238000010276 construction Methods 0.000 description 6
- 238000007405 data analysis Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000007792 addition Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an object storage device and method based on a smart campus middlebox, wherein the device is arranged on a proxy server of an object storage system, the proxy server is in communication connection with a user terminal, and the method comprises the following steps: the control module is arranged at a client of the object storage device and used for appointing a storage server for storing the object file; the block cutting module is used for cutting the storage object file into a plurality of data subfiles; and the storage module is used for creating an ordered list file and reading/writing the data subfile segmentation node information and the storage node information. The invention provides placement control for the object storage system by constructing a location-aware writing mechanism, realizes efficient object renaming in the object storage system, and solves the problem of data writing performance.
Description
[ technical field ] A method for producing a semiconductor device
The invention relates to the technical field of big data object storage system design, in particular to an object storage device and method based on a smart campus middesk.
[ background of the invention ]
The intelligent campus middle platform is used for constructing an education basic data warehouse, realizing the collection and aggregation of user data, integrating a telecommunication charging channel, realizing the unified charging management of users, constructing an interconnection and intercommunication system of education data, constructing an information basic support platform, taking information application as guidance, providing support and service for the construction of the whole education information construction framework, and realizing 'unified user authentication, unified data center and third-party application access management'. The front-end campus portal and the back-end smart campus application integration are realized, and the application implementation and charging management and control are realized. Information isolated islands and application blanks are eliminated, the working efficiency is improved, the management efficiency is improved, and the decision-making efficiency is improved. The construction target is as follows: 1) the access of background application and the unified access and management of application suppliers and third-party applications are realized; 2) the data of the campus user are unified, the authentication and the application authorization are unified, and meanwhile, the data synchronization of a service order and a telecom business navigation platform is realized; 3) the campus foreground systems such as wing school and the like are enabled to provide application, and a campus user can enter an application function interface accessed by a middle station through one-time authentication of an original campus application portal; 4) the method has the advantages that intensive management of the development and operation of the services of the telecom campus is accelerated, the data center is built to precipitate large data of the services, the open platform is built to introduce external cooperation, intelligent education is developed, deep fusion of information technology and education is promoted, deep change of education management, teaching and learning under the background of the mobile internet is promoted, construction and public demands of smart cities and learning cities are served, high-quality education resources are widely shared, the fair level of education is improved, and the development of education modernization is promoted.
A data center system for construction of a platform project in a smart campus realizes convergence of education and teaching process data and unstructured data such as teaching resource text type, video type and file type from an education application ecological manufacturer through a data exchange subsystem, and realizes rapid warehousing of data and access of large and small files at any time through distributed storage service for storing massive files provided by a data storage subsystem. Object-based storage systems (e.g., AliOSS, Baidu BOS, Amazon S3, OpenStack Swift, etc.) have become an important solution for large-scale storage due to their virtually unlimited scalability and high economic efficiency. As businesses deposit more data in object storage systems, it is becoming increasingly important to perform big data analysis on such data.
The distributed object storage system provides quick reading and reduces access delay so as to meet the actual requirements of projects, is an attractive storage solution due to the advantages of scalability and low cost, and is widely deployed in project construction. In order to analyze data stored in an object storage system while avoiding the expensive process of copying data to a distributed file system (e.g., HDFS), it is natural to use them directly as the storage back-end of a data parallel analysis framework (e.g., Spark or MapReduce). However, executing a data-parallel framework on an object store presents serious performance problems: 1. the Hash mapping process from the object name to the storage node prevents the file writing in when the object is renamed, and increases the time delay; 2. the granularity of the objects is coarser compared to conventional file blocks, reducing parallelism during reading. Existing solutions attempt to address write performance issues in data parallel analysis work by avoiding expensive data operations, particularly avoiding renaming operations, resulting in limited operations.
[ summary of the invention ]
The invention aims to provide an object storage device and method based on a smart campus middle desk, which aims to overcome the defects in the prior art, provides placement control for an object storage system by constructing a location-aware writing mechanism, realizes efficient object renaming in the object storage system, and solves the problem of data writing performance; meanwhile, the data of the object is cut into blocks when the object is stored, and then a plurality of copies of the cut blocks are stored on different servers in the cluster, so that the problem of data reading performance is solved. Finally, the storage nodes and the computing nodes are arranged on the same cluster, an analysis framework and an object storage system are seamlessly integrated without causing obvious performance loss, and extra time delay of remote data reading and writing is avoided, so that the maximum data analysis performance is achieved.
Additional features and advantages of the invention will be set forth in the detailed description which follows, or may be learned by practice of the invention.
In a first aspect, the present invention provides an object storage device based on a smart campus middlebox, the object storage device being disposed in a proxy server of an object storage system, the proxy server being communicatively connected to a user terminal, the object storage device comprising:
the control module is arranged at a client of the object storage device and used for appointing a storage server for storing the object file;
the block cutting module is used for cutting the storage object file into a plurality of data subfiles;
and the storage module is used for creating an ordered list file and reading/writing the data subfile segmentation node information and the storage node information.
Preferably, the control module includes:
the analysis unit is used for analyzing the ordered list file of the storage object and extracting the number of blocks of the object file;
and the updating unit updates the storage node information of the storage object through a consistent hash algorithm.
Preferably, the block-cutting module transparently cuts the original object file into a plurality of data subfiles with arbitrary fixed sizes according to the client instruction.
Preferably, the storage module includes:
a list file storage unit for storing the ordered list file with the original object as a name,
and the node storage unit is used for storing the segmentation node information and the storage node information of the data subfile.
In a second aspect, the present invention provides an object storage method based on a smart campus middlebox, the method is performed by a client-specified proxy server in an object storage system, the proxy server is in communication connection with a user terminal, and the method includes:
dividing the received storage object file into a plurality of data subfiles;
creating an ordered list file with an original object file as a name;
and acquiring the segmentation node information and the storage node information of the data subfile, and writing the segmentation node information and the storage node information into the sequence list file.
Preferably, the step of dividing the received original object file into a plurality of data subfiles includes:
the method comprises the steps of transparently cutting specified object data into a plurality of data subfiles with any fixed size;
and naming the data subfiles independently according to the positions of the data subfiles in the cluster.
Preferably, the step of obtaining the storage node information of the data subfile includes:
respectively hashing the original object file and the data subfile to obtain respective hash values;
and calculating storage nodes of the original object file and the data subfiles by adopting a consistent hash algorithm according to the hash value.
Preferably, before the step of dividing the received storage object file into a plurality of data subfiles, the method further includes:
judging whether the received storage object file has a storage record, if so, the storage object file is an original object file, directly analyzing the original object file, and then acquiring the segmentation node information and the storage node information of the data subfile again;
if not, the storage object file name is hashed, and an ordered list file with the storage object file as the name is created according to the hash value.
In a third aspect, the present invention provides a server comprising a processor and a memory, the memory storing computer-executable instructions capable of being executed by the processor, the processor executing the computer-executable instructions to implement the steps of the object storage method of any one of the preceding first aspects.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, performs the steps of the object storage method provided in any one of the foregoing first aspects.
The invention has the advantages that:
1. the object storage device is arranged on the proxy server of the object storage system, and is in communication connection with the user terminal through the proxy server, so that seamless integration between an analysis frame and the object storage system is realized, and obvious performance loss can be avoided
2. The object storage device constructs a position sensing write-in mechanism between the client and the proxy server through the control module and the block module, provides placement control for the object storage system, controls the object storage position, thereby eliminating unnecessary I/O operation caused by renaming during the completion of operation, and breaks the dependency relationship between the object name and the hash calculation position thereof under the condition of not introducing a centralized component, thereby not influencing scalability and improving the operation analysis speed
3. The method of the invention constructs a cutting mechanism between the client and the object storage system through the cutting module, transparently cuts the storage object file into small blocks to improve the data locality, then stores a plurality of copies of the cut blocks on different servers in the cluster, and improves the reading speed.
4. The object storage device of the system is arranged on the proxy server of the object storage system, the proxy server is used as an entry point of the cluster, a user interacts with the cluster through a proxy, in the analysis work of parallel reading and writing of a plurality of tasks, a plurality of proxy server copies are established, different proxy server copies are operated on each node in the cluster, and the reading/writing tasks are completed through the local examples, so that the performance bottleneck of the proxy server is eliminated.
[ description of the drawings ]
In order to more clearly illustrate the embodiments or the prior art solutions of the present invention, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without inventive labor.
FIG. 1 is a schematic diagram of an object storage system architecture composed of a proxy server, a client and a storage system.
FIG. 2 is a schematic block diagram of an object storage device based on a smart campus staging platform according to the present invention;
fig. 3 is a schematic flow chart of an object storage method based on a smart campus staging according to an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating an embodiment of a block cutting process for an object file;
FIG. 5 is a diagram illustrating a process of the proxy server in the object storage method according to an embodiment of the present invention;
FIG. 6 is a schematic diagram illustrating a renaming procedure of an object file according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a server according to an embodiment of the present invention. .
[ detailed description ] embodiments
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail and completely with reference to the following embodiments and accompanying drawings. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention discloses an object storage device based on a smart campus middle desk, which is an object storage device oriented to a big data analysis framework, is an effective storage layer of a data parallel analysis framework (such as Spark and MapReduce), and is a technical scheme for solving the reading performance problem and the writing performance problem aiming at the performance problem existing when the analysis framework is operated on an object storage system. The method comprises the following steps that a position sensing writing mechanism is built in an object-oriented storage system through a control module, and an object storage position is controlled, so that unnecessary I/O (input/output) operation caused by renaming during operation completion is eliminated, and the operation analysis speed is increased; the method comprises the steps of providing a transparent object file block cutting mechanism through a block cutting module, transparently cutting object data into data with fixed arbitrary size, storing a plurality of copies of blocks on different servers in a cluster, and improving the reading and writing performance of the object data by matching with a control module.
In the embodiment shown in fig. 1-2, the present invention provides an object storage device based on a smart campus middleware, the object storage device is disposed in a proxy server of an object storage system, the proxy server is communicatively connected to a user terminal, as shown in fig. 1, in the object storage system, a file system uses file blocks as storage units, and data is stored in the object storage system with BLOB granularity. A BLOB may be any type of data including an image or document. Objects are stored (PUT), retrieved (GET) and Deleted (DELETE) through interaction of the RESTful API with the storage system and through their key values. The object storage system uses a flat name space in which a key is an arbitrary identifier of an object, and the key may be the object name or its ID. The storage system also stores multiple copies of one data to ensure reliability. The object storage has scalability, and the object storage system is enabled to store metadata in a distributed mode through the following two characteristics, so that single-point failure and performance bottleneck are avoided:
1. because the consistent hash algorithm is used for positioning the object, a centralized metadata server is not required to be arranged. In the consistent hashing algorithm, storage nodes are randomly assigned locations in a logical ring structure. The ring is closed at the minimum and maximum hash values. The system obtains the Hash ID by carrying out consistent Hash calculation on the object, and distributes the Hash ID to the node with the position closest to the Hash value of the object on the ring to determine the position of the object;
2. all object metadata (e.g., its creation time or checksum) is stored with the object, rather than on a separate metadata server.
The object storage system architecture established based on the above two characteristics, referring to fig. 1, the process of performing the search operation specifically includes: when the client needs to retrieve object f, it contacts the proxy server. A proxy server is a single entry point to a storage cluster that is stateless and therefore arbitrarily extensible. After receiving the client request, the proxy server determines the storage server of the object through a Hash algorithm, forwards the retrieval request to the object storage server, and returns the retrieval result to the client.
When the analysis framework runs on the object storage system, the object storage system uses the proxy server as an entry point of the cluster, a user interacts with the cluster through a proxy, and the analysis framework interacts with the distributed storage in two stages: (1) when reading input data; (2) in writing an output result, in order to eliminate the two performance problems, the proxy server may have a read performance problem and a write performance problem, and referring to fig. 2, the present invention provides an object storage device based on a central station of a smart campus, including: the control module is arranged at a client of the object storage device and used for appointing a storage server for storing the object file; the block cutting module is used for cutting the storage object file into a plurality of data subfiles, and for the original object file, the block cutting module is used for transparently cutting the original object file into a plurality of data subfiles with any fixed size according to a client instruction; and the storage module is used for creating an ordered list file and reading/writing the data subfile segmentation node information and the storage node information.
In particular, the control module, for the problem of renaming in the object storage system, comprises: the analysis unit is used for analyzing the ordered list file of the storage object and extracting the number of blocks of the object file; and the updating unit updates the storage node information of the storage object through a consistent hash algorithm. According to the scheme, the storage nodes of the renamed object file are updated by analyzing the manifest file of the storage object, the object server for storing the new object can be specified definitely, and the object replication process caused by object renaming is avoided.
Specifically, the storage module includes: the node storage unit is used for storing the segmentation node information and the storage node information of the data subfile. According to the technical scheme, the segmented data subfiles are simultaneously linked to a manifest file stored by an original object name, the manifest file comprises an ordered list of all object blocks, different parts of the segmented data subfiles have different internal names according to the positions of the data subfiles in the whole cluster, and storage nodes of the files can be found through a consistent hash algorithm and are filled into the manifest file.
The process schematic diagram of the object storage method based on the intelligent campus middlebox realized by the object storage device based on the intelligent campus middlebox is shown in the attached figures 3-6, the method is executed by a proxy server designated by a client in an object storage system, the proxy server is in communication connection with a user terminal, and the method comprises the following steps:
dividing the received storage object file into a plurality of data subfiles;
creating an ordered list file with an original object file as a name;
and acquiring the segmentation node information and the storage node information of the data subfile, and writing the segmentation node information and the storage node information into the sequence list file.
When uploading an object file, the client may specify that the object be chunked through the above-described object storage method, which will result in the proxy server cutting the incoming data into portions, preferably into fixed arbitrary size data subfiles, such as 128MB chunks in HDFS. Different parts have different internal names depending on their location in the whole cluster. In the process, the proxy server simultaneously establishes a manifest file with the original object as a name, wherein the manifest file comprises an ordered list of all object blocks. By the storage method, after a GET request of a client is received, the proxy server retrieves and analyzes the manifest file, and then the object blocks are returned to the client.
Referring to fig. 4, in a preferred embodiment, the step of dividing the received original object file into a plurality of data subfiles specifically includes:
the method comprises the steps of transparently cutting specified object data into a plurality of data subfiles with any fixed size;
and naming the data subfiles independently according to the positions of the data subfiles in the cluster.
By the method, each object block is linked to the manifest file stored by the original object name, the data reading parallelism is improved, a plurality of proxy server copies can be established, different proxy server copies are operated on each node in the cluster, and the reading/writing task is completed through the local instance of the proxy server, so that the performance bottleneck of the proxy server is eliminated.
In the above embodiment, the step of obtaining the storage node information of the data subfile includes:
respectively hashing the original object file and the data subfile to obtain respective hash values;
and calculating storage nodes of the original object file and the data subfiles by adopting a consistent hash algorithm according to the hash value.
In the above-mentioned process of object file blocking illustrated in fig. 5, the processing procedure of the proxy server in the object storage method is as follows:
step one, after receiving a file f from a client, a proxy server divides the file into N data subfiles;
step two, if N is 3, the several data subfiles are respectively f1/f2/f 3;
step three, the proxy server hashes the original file to obtain fm, and hashes the three data files to obtain h1/h2/h3 respectively;
step four, according to the hash value, the proxy server searches by using fm/h1/h2/h3 as a keyword through a consistent hash algorithm, and can find out the storage nodes of each file to be nm/n1/n2/n3 respectively;
step five, the proxy server creates a manifest file f.manifest, and the manifest file and each data sub-division and storage node information are filled in the manifest file;
step six, the proxy server sends the manfiest file to the client.
It should be noted that, after the original object files are stored in sequence at night, the client saves the manifest file as a stub, and uses the manifest file as an access index of the data file. Therefore, before the step of dividing the received storage object file into a plurality of data subfiles, the method further comprises:
judging whether the received storage object file has a storage record, if so, the storage object file is an original object file, directly analyzing the original object file, and then acquiring the segmentation node information and the storage node information of the data subfile again;
if not, the storage object file name is hashed, and an ordered list file with the storage object file as the name is created according to the hash value.
When a client requests to access an object, whether a received storage object file has a storage record or not is judged, namely whether the client stores a manifest file of the object or not is judged, if the client stores the manifest file, the client can directly deliver the manifest file to a proxy server, the proxy server analyzes the manifest file, the file block number and each block storage node are extracted, each storage block is obtained, and then the storage blocks are returned to the client; if the client does not store the manifest file, the client can submit the file name to be requested to the proxy server, the proxy server hashes the file name, obtains the manifest file according to the hash value, then enters the first step to obtain the file and returns the file to the client.
When an object needs to be renamed, a step of judging whether the storage object file has a storage record is needed, for renaming the object, a manifest file of the object stored by a client can be obtained, the storage object file is an original object file, the partition node information and the storage node information of the data subfile are obtained again after the storage object file is directly analyzed, and the updated manifest file is sent to the client by a proxy server.
For example, the object file with the file name Data _ object is in manifest file Data _ object. If the Data _ object is renamed to Data _ object _ new _ name. The client sends a Put (f _ new _ name, Data _ object _ new _ name. manifest) request to the proxy server. Referring to fig. 6 specifically, which is a schematic diagram illustrating a renaming process of an object file in this embodiment, after receiving the request and parsing a manifest file, the proxy server executes the following steps:
deleting Data _ object.
Step two, updating the manifest file received from the client, wherein the specific updating process comprises the following steps:
"manifest _ file.name" is updated to "Data _ object _ new _ name.manifest";
update of manifest _ file. hash _ ID to new hash value: hash (Data _ object _ new _ name);
c. and searching the storage node through a consistent hash algorithm of the object storage system, and updating the manifest _ file.
Uploading a new manifest file to the updated manifest _ file.storage _ node;
and step four, the proxy server sends the updated manifest file to the client.
Further, an embodiment of the present invention provides a server, which includes a processor and a memory, the memory storing computer executable instructions capable of being executed by the processor, the processor executing the computer executable instructions to implement the steps of the object storage method according to any one of the above embodiments.
Fig. 7 is a schematic structural diagram of a server according to an embodiment of the present invention, where the server includes: the system comprises a processor, a memory, a bus and a communication interface, wherein the processor, the communication interface and the memory are connected through the bus; the processor is used to execute executable modules, such as computer programs, stored in the memory. The Memory may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network and the like can be used. The bus may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 7, but this does not indicate only one bus or one type of bus. The memory is used for storing a program, and the processor executes the program after receiving an execution instruction, and the method executed by the apparatus defined by the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to or implemented by the processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.
The file storage method, the file storage device, and the computer program product of the server provided in the embodiments of the present invention include a computer-readable storage medium storing a nonvolatile program code executable by a processor, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the method in the foregoing method embodiments is executed.
The embodiments of the present invention are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and additions can be made without departing from the method of the present invention, and these modifications and additions should also be regarded as the protection scope of the present invention.
Claims (10)
1. An object storage device based on a smart campus staging platform, the object storage device being provided in a proxy server of an object storage system, the proxy server being communicatively connected to a user terminal, the object storage device comprising:
the control module is arranged at a client of the object storage device and used for appointing a storage server for storing the object file;
the block cutting module is used for cutting the storage object file into a plurality of data subfiles;
and the storage module is used for creating an ordered list file and reading/writing the data subfile segmentation node information and the storage node information.
2. The smart campus based object storage device of claim 1, wherein the control module comprises:
the analysis unit is used for analyzing the ordered list file of the storage object and extracting the block number of the object file;
and the updating unit is used for updating the storage node information of the storage object through a consistent hash algorithm.
3. The smart campus middesk-based object storage device of claim 1 or 2 wherein the chunking module transparently partitions the original object file into a number of data subfiles of arbitrary fixed size according to client instructions.
4. The smart campus based object storage device of claim 3, wherein the storage module comprises:
a list file storage unit for storing the ordered list file with the original object as a name,
and the node storage unit is used for storing the segmentation node information and the storage node information of the data subfile.
5. An object storage method based on a smart campus middesk, characterized in that the method is executed by a client-specified proxy server in an object storage system, the proxy server is in communication connection with a user terminal, and the method comprises:
dividing the received storage object file into a plurality of data subfiles;
creating an ordered list file with an original object file as a name;
and acquiring the segmentation node information and the storage node information of the data subfile, and writing the segmentation node information and the storage node information into the sequence list file.
6. The method of claim 5, wherein the step of segmenting the received original object file into a plurality of data subfiles comprises:
the method comprises the steps of transparently cutting specified object data into a plurality of data subfiles with any fixed size;
and naming the data subfiles independently according to the positions of the data subfiles in the cluster.
7. The method of claim 6, wherein the step of obtaining the storage node information of the data subfile comprises:
respectively hashing the original object file and the data subfile to obtain respective hash values;
and calculating storage nodes of the original object file and the data subfiles by adopting a consistent hash algorithm according to the hash value.
8. The method of claim 7, wherein the step of segmenting the received storage object file into a plurality of data subfiles further comprises:
judging whether the received storage object file has a storage record, if so, the storage object file is an original object file, directly analyzing the original object file, and then acquiring the segmentation node information and the storage node information of the data subfile again;
if not, the storage object file name is hashed, and an ordered list file with the storage object file as the name is created according to the hash value.
9. A server comprising a processor and a memory, the memory storing computer-executable instructions executable by the processor to perform the steps of the object storage method of any one of claims 5 to 8.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the object storage method according to any one of claims 5 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110609147.0A CN113311999A (en) | 2021-06-01 | 2021-06-01 | Object storage device and method based on intelligent campus middleboxes |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110609147.0A CN113311999A (en) | 2021-06-01 | 2021-06-01 | Object storage device and method based on intelligent campus middleboxes |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113311999A true CN113311999A (en) | 2021-08-27 |
Family
ID=77376937
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110609147.0A Pending CN113311999A (en) | 2021-06-01 | 2021-06-01 | Object storage device and method based on intelligent campus middleboxes |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113311999A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104050102A (en) * | 2014-06-26 | 2014-09-17 | 北京思特奇信息技术股份有限公司 | Object storing method and device in telecommunication system |
| CN106156289A (en) * | 2016-06-28 | 2016-11-23 | 北京百迈客云科技有限公司 | The method of the data in a kind of read-write object storage system and device |
| CN110032543A (en) * | 2019-04-15 | 2019-07-19 | 苏州浪潮智能科技有限公司 | A kind of management method of storage file system |
| CN112698783A (en) * | 2019-10-22 | 2021-04-23 | 北京金山云网络技术有限公司 | Object storage method, device and system |
-
2021
- 2021-06-01 CN CN202110609147.0A patent/CN113311999A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104050102A (en) * | 2014-06-26 | 2014-09-17 | 北京思特奇信息技术股份有限公司 | Object storing method and device in telecommunication system |
| CN106156289A (en) * | 2016-06-28 | 2016-11-23 | 北京百迈客云科技有限公司 | The method of the data in a kind of read-write object storage system and device |
| CN110032543A (en) * | 2019-04-15 | 2019-07-19 | 苏州浪潮智能科技有限公司 | A kind of management method of storage file system |
| CN112698783A (en) * | 2019-10-22 | 2021-04-23 | 北京金山云网络技术有限公司 | Object storage method, device and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7113040B2 (en) | Versioned hierarchical data structure for distributed data stores | |
| US11086531B2 (en) | Scaling events for hosting hierarchical data structures | |
| US10353742B2 (en) | Tracking large numbers of moving objects in an event processing system | |
| US11201936B2 (en) | Input and output schema mappings | |
| CN110275983B (en) | Retrieval method and device of traffic monitoring data | |
| CN104778270A (en) | Storage method for multiple files | |
| CN104679898A (en) | Big data access method | |
| CN110390739B (en) | Vehicle data processing method and vehicle data processing system | |
| CN102169507A (en) | Distributed real-time search engine | |
| CN113485962B (en) | Log file storage method, device, equipment and storage medium | |
| US20180203874A1 (en) | Space-efficient secondary indexing on distributed data stores | |
| CN115114374B (en) | Transaction execution method and device, computing equipment and storage medium | |
| JP2022002092A (en) | Geographic information system engine system, implementing method thereof, device, and storage medium | |
| CN111404932A (en) | Method for accessing medical institution system to smart medical cloud service platform | |
| CN113760847A (en) | Log data processing method, device, equipment and storage medium | |
| CN111723161A (en) | Data processing method, device and equipment | |
| CN111651424B (en) | Data processing method, device, data node and storage medium | |
| CN114610680A (en) | Method, device and equipment for managing metadata of distributed file system and storage medium | |
| CN111125248A (en) | Big data storage analysis query system | |
| CN111881086B (en) | Big data storage method, query method, electronic device and storage medium | |
| CN111930684A (en) | Small file processing method, device and equipment based on HDFS (Hadoop distributed File System) and storage medium | |
| CN116775712A (en) | Method, device, electronic equipment, distributed system and storage medium for inquiring linked list | |
| CN116048728A (en) | Container mirror acceleration method based on-demand delay loading and application | |
| CN107291875B (en) | Metadata organization management method and system based on metadata graph | |
| CN113311999A (en) | Object storage device and method based on intelligent campus middleboxes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210827 |