CN116737367A - Minio-based data preservation method, device and medium - Google Patents

Minio-based data preservation method, device and medium Download PDF

Info

Publication number
CN116737367A
CN116737367A CN202310567257.4A CN202310567257A CN116737367A CN 116737367 A CN116737367 A CN 116737367A CN 202310567257 A CN202310567257 A CN 202310567257A CN 116737367 A CN116737367 A CN 116737367A
Authority
CN
China
Prior art keywords
data
minio
information set
uploading
storage
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310567257.4A
Other languages
Chinese (zh)
Other versions
CN116737367B (en
Inventor
付兴东
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inspur Intelligent Technology Co Ltd
Original Assignee
Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inspur Intelligent Technology Co Ltd filed Critical Inspur Intelligent Technology Co Ltd
Priority to CN202310567257.4A priority Critical patent/CN116737367B/en
Publication of CN116737367A publication Critical patent/CN116737367A/en
Application granted granted Critical
Publication of CN116737367B publication Critical patent/CN116737367B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2228—Indexing structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G06F9/5022—Mechanisms to release resources
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The application provides a data preservation method, equipment and medium based on Minio, wherein the method acquires an uploading information set of service data corresponding to a current time node; the uploading information set at least comprises an uploading address of the business data after the last time node finishes data preservation and an uploading address of the business data from the last time node to the current time node. And sending a business data freshness keeping instruction to a pre-connected Minio file server to obtain a storage bucket information set corresponding to the current time node in the Minio file server. The storage bucket information set at least comprises storage relative paths of all uploaded service data in all storage buckets. Based on the comparison result of the uploading information set and the storage bucket information set, invalid service data in the storage bucket information set under the current time node is determined to be unreachable data, so that a data preservation instruction is generated, and the unreachable data is removed from the Minio file server.

Description

Minio-based data preservation method, device and medium
Technical Field
The application relates to the technical field of computers, in particular to a data preservation method, device and medium based on Minio.
Background
Minio is a high-performance, distributed object storage system. It is a software product that can run 100% on standard hardware, i.e. X86, etc. low cost machines can also run Minio well.
At present, when using a Minio file server to perform services such as website registration, business service and the like, the services often have some file collection requirements, and an uploading interface for file collection can be exposed to a user, so that the user can upload corresponding files conveniently.
In the process of uploading files by a user, repeated operations or wrong operations may occur in the uploading operation of the user, and invalid files under the operations are recorded by the Minio file server, but the invalid files are not used by the user because of no effective index, so that resources of the Minio file server are occupied by the invalid files, the resource utilization rate of the file server is low, and the response speed of the Minio file server is influenced, and the use efficiency and the use experience of the user are influenced.
Disclosure of Invention
The embodiment of the application provides a data preservation method, device and medium based on Minio, which are used for solving the problems that the current file server is easy to occupy resources by invalid files, the resource utilization rate is low, and the use efficiency and the use experience of the file server of a user are affected.
In one aspect, an embodiment of the present application provides a Minio-based data preservation method, where the method includes:
acquiring an uploading information set of service data corresponding to a current time node; the uploading information set at least comprises an uploading address of the business data after the last time node finishes data preservation and an uploading address of the business data from the last time node to the current time node;
a business data fresh-keeping instruction is sent to a pre-connected Minio file server so as to obtain a storage bucket information set corresponding to the current time node in the Minio file server; the storage bucket information set at least comprises storage relative paths of all uploaded service data in all storage buckets;
based on the comparison result of the uploading information set and the storage bucket information set, determining invalid service data in the storage bucket information set under the current time node as unreachable data, so as to generate a data preservation instruction, and rejecting the unreachable data from the Minio file server; wherein the invalid service data includes uploaded service data whose stored relative path does not match the upload address.
In one implementation manner of the present application, obtaining an upload information set of service data corresponding to a current time node specifically includes:
under the condition that the current time node meets a preset timing fresh-keeping task, determining a stored data record of the service data corresponding to the current time node from a preset service data uploading database; the stored data record stores the uploading address of the service data in a data line; the uploading address at least comprises a storage relative path of the service data;
and generating the uploading information set according to the stored data record.
In one implementation manner of the present application, a service data freshness keeping instruction is sent to a pre-connected Minio file server to obtain a bucket information set corresponding to the current time node in the Minio file server, which specifically includes:
and sending a business data freshness preservation instruction to a pre-connected Minio file server, calling a storage bucket traversing interface of the Minio file server through the business data freshness preservation instruction so as to traverse a storage bucket in the Minio file server, and adding the storage relative path of the uploaded business data in the traversed storage bucket to the storage bucket information set.
In one implementation manner of the present application, based on a comparison result of the uploading information set and the bucket information set, invalid service data in the bucket information set at the current time node is determined to be unreachable data, which specifically includes:
comparing the uploading information set with the storage bucket information set, and calculating a difference set of the storage bucket information set and the uploading information set;
and in the case that the difference set is not an empty set, taking the storage relative path in the difference set as the storage relative path of the unreachable data, and determining the unreachable data.
In one implementation of the present application, comparing the upload information set with the bucket information set specifically includes:
and comparing each uploading address in the uploading information set with each storage relative path in the storage bucket information set, and determining whether the uploading address is consistent with the storage relative path or not.
In one implementation of the present application, the set of upload information corresponds to at least one user of the Minio file server; the Minio file server corresponds to a plurality of users.
In one implementation of the present application, the Minio file server stores at least one bucket, and each bucket has at least one uploaded service data and a corresponding storage relative path.
In one implementation of the present application, the unreachable data includes at least: repeating the uploaded service data and the service data of the uploading address which is not stored by the data line of the service data uploading database.
On the other hand, the embodiment of the application also provides a Minio-based data preservation device, which comprises:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
acquiring an uploading information set of service data corresponding to a current time node; the uploading information set at least comprises an uploading address of the business data after the last time node finishes data preservation and an uploading address of the business data from the last time node to the current time node;
a business data fresh-keeping instruction is sent to a pre-connected Minio file server so as to obtain a storage bucket information set corresponding to the current time node in the Minio file server; the storage bucket information set at least comprises storage relative paths of all uploaded service data in all storage buckets;
based on the comparison result of the uploading information set and the storage bucket information set, determining invalid service data in the storage bucket information set under the current time node as unreachable data, so as to generate a data preservation instruction, and rejecting the unreachable data from the Minio file server; wherein the invalid service data includes uploaded service data whose stored relative path does not match the upload address.
In yet another aspect, an embodiment of the present application further provides a Minio-based data preservation non-volatile computer storage medium, storing computer-executable instructions configured to:
acquiring an uploading information set of service data corresponding to a current time node; the uploading information set at least comprises an uploading address of the business data after the last time node finishes data preservation and an uploading address of the business data from the last time node to the current time node;
a business data fresh-keeping instruction is sent to a pre-connected Minio file server so as to obtain a storage bucket information set corresponding to the current time node in the Minio file server; the storage bucket information set at least comprises storage relative paths of all uploaded service data in all storage buckets;
based on the comparison result of the uploading information set and the storage bucket information set, determining invalid service data in the storage bucket information set under the current time node as unreachable data, so as to generate a data preservation instruction, and rejecting the unreachable data from the Minio file server; wherein the invalid service data includes uploaded service data whose stored relative path does not match the upload address.
Through the scheme, the data preservation processing is carried out on the Minio file server by utilizing the uploading address in the client system and the storage relative path of the service data in the Minio file server. On one hand, occupied resources of invalid files in the Minio file server are released, and the resource utilization rate of the file server is improved; on the other hand, the Minio file server can be optimized regularly, high-efficiency response of the Minio file server is guaranteed, and the use experience of a user is improved. Therefore, the problems that the current file server is easy to occupy resources by invalid files, the resource utilization rate is low, and the use efficiency and the use experience of the file server of a user are affected are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
fig. 1 is a schematic flow chart of a Minio-based data preservation method in an embodiment of the application;
fig. 2 is a schematic structural diagram of a Minio-based data preservation device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Minio is a high-performance, distributed object storage system that is a software product. When the website provides registration, a service of a sponsor and the like, there is a need for some file collection, and at this time, some interfaces for uploading files (for file collection) are exposed. In general, when a file is uploaded to the Minio file server, the data line of the database may go to the relative path of the stored file, but in this case, there may be a case that the file is repeatedly uploaded to the Minio file server, or a case that the data line directly uploads the file to the Minio file server without going to the relative path of the stored file, where there may occur some invalid files (i.e. no valid reference or index can be used to use the current file) for the user itself, where the invalid files may not be the files used by any user of the Minio file server.
The repeated uploading refers to that, if the path of the file uploading to the file server is assumed to be per usr/local/1.Jpg, the path of the user accessing the picture is www.XXXX.com/url/local/1.Jpg, the url/local/1.Jpg is a relative path, the repeated uploading is compared with 1.Jpg and 2.Jpg, and at this time, the data line of the user stores the relative path of 2.Jpg, and for the user, 1.Jpg is the picture without business meaning.
Based on the above, the embodiment of the application provides a data preservation method, device and medium based on Minio, which are used for solving the problems that the current file server is easy to occupy resources by invalid files, the resource utilization rate is low, and the use efficiency and the use experience of the file server of a user are affected.
Various embodiments of the present application are described in detail below with reference to the attached drawing figures.
The embodiment of the application provides a data preservation method based on Minio, as shown in fig. 1, the method can comprise the steps of S101-S103:
s101, a server acquires an uploading information set of service data corresponding to a current time node.
The uploading information set at least comprises an uploading address of the business data after the last time node finishes data preservation and an uploading address of the business data from the last time node to the current time node.
The server is a client for a user to transact and upload service data, and serves as an execution subject of the Minio-based data preservation method, and the execution subject is not limited to the server, but is not particularly limited thereto.
The current time node can be understood as the current time triggering the data preservation of the application, for example, a certain time preservation task is set to be one month, the last time node is 12 points of 1 month and 1 day, and the current time node is 12 points of 2 months and 1 day. The business data refers to data uploaded by a user to a Minio file server at a server, such as uploading registered business licenses, and is business data; and uploading advertisement pictures of the vendor information as service data. The service data uploading address in the uploading information set can be a storage relative path of the service data in the database. If the previous time node finishes the data preservation of the Minio file server, the uploading information set comprises the uploading address of the business data after the previous time node finishes the data preservation and the uploading address of the business data from the previous time node to the current time node; if the current time node is to keep the data fresh for the first time for the Minio file server, the uploading information set contains uploading addresses of all business data from the initial data storage to the current time node in the database.
In the embodiment of the application, the method for acquiring the uploading information set of the service data corresponding to the current time node specifically comprises the following steps:
and under the condition that the current time node meets the preset timing fresh-keeping task, the server uploads the service data corresponding to the current time node to a database from preset service data, and determines a stored data record of the service data. The stored data record stores the upload address of the service data in a data row. The upload address comprises at least a stored relative path of the traffic data. The server generates an upload information set from the stored data record.
In other words, the server may traverse the stored data records of the business data in the business data upload database while performing the timed freshness task. The store data record is an upload address (a relative path stored to the Minio file server) that stores the service data in the form of a data line. After traversing all the uploading addresses of the business data, adding the uploading addresses into the uploading information set.
The uploading information set at least corresponds to at least one user of the Minio file server, and in the actual use process, a plurality of users using the Minio file server can exist, specifically, different users can respectively have own file storage accounts in the Minio file server, and the Minio file server can store files (service data) uploaded by the plurality of users. In the actual use process, the uploading information set may be uploading addresses of service data uploaded to the Minio file server by all users, that is, users, and the uploading addresses of the service data uploaded by the users may be all stored in a preset service data uploading database.
S102, the server sends a business data fresh-keeping instruction to a pre-connected Minio file server to obtain a storage bucket information set corresponding to a current time node in the Minio file server.
The storage bucket information set at least comprises storage relative paths of all uploaded service data in all storage buckets.
In the embodiment of the application, a business data fresh-keeping instruction is sent to a pre-connected Minio file server to obtain a storage bucket information set corresponding to a current time node in the Minio file server, which specifically comprises the following steps:
the server sends a business data refreshing instruction to a pre-connected Minio file server, and calls a storage barrel traversing interface of the Minio file server through the business data refreshing instruction so as to traverse a storage barrel in the Minio file server, and adds a storage relative path of uploaded business data in the traversed storage barrel to the storage barrel information set.
That is, when the data preservation of the Minio file server is performed, the server may first send a service data preservation instruction to the Minio file server, where the service data preservation instruction can call a bucket traversal interface in the Minio file server, where the bucket traversal interface is used to traverse all buckets in the Minio file server, and traverse the uploaded service data and the storage relative path in each bucket. Where a Bucket (Bucket) is a carrier of objects, it is understood as a "container" that holds objects, and there is no upper limit to the capacity of the "container". Objects are stored in buckets in a flattened structure, without the concept of folders and directories, and a user may choose to store objects in a single or multiple buckets.
The bucket traversal interface may be an application programming interface (Application Programming Interface, API) of a template minitemplate in mini.
In the embodiment of the application, the Minio file server stores at least one storage bucket, and each storage bucket is internally provided with at least one uploaded service data and a corresponding storage relative path.
S103, the server determines invalid service data in the storage barrel information set under the current time node as unreachable data based on the comparison result of the uploading information set and the storage barrel information set, so as to generate a data preservation instruction, and eliminates the unreachable data from the Minio file server. Thereby completing the data preservation in the Minio file server.
Wherein the invalid traffic data includes uploaded traffic data for which the stored relative path does not match the upload address.
In the embodiment of the application, based on the comparison result of the uploading information set and the storage bucket information set, invalid service data in the storage bucket information set under the current time node is determined to be unreachable data, and the method specifically comprises the following steps:
the server compares the uploaded information set with the bucket information set and calculates a difference set between the bucket information set and the uploaded information set. In the case where the difference set is not an empty set, the stored relative path within the difference set is taken as the stored relative path of the unreachable data, and the unreachable data is determined.
The unreachable data includes at least: repeating the uploaded service data and the service data of the uploading address which is not stored by the data line of the service data uploading database.
Comparing the uploaded information set with the storage bucket information set, wherein the method specifically comprises the following steps:
and comparing each uploading address in the uploading information set with each storage relative path in the storage bucket information set, and determining whether the uploading address is consistent with the storage relative path. If the addresses are consistent, the uploading addresses are matched with the storage relative addresses, otherwise, the addresses are not matched.
In other words, after obtaining the bucket information set, the server may compare each storage relative path with the uploading address in the uploading information set to determine a difference set between the bucket information set and the uploading information set. The difference set is a storage relative path without uploading address recorded by uploading information set, and the storage relative path of the difference set part is generated when the user is operated to repeatedly upload or directly upload business data. Based on the above, after the server obtains the difference set, the server can perform processing of deleting unreachable data on the Minio file server so as to realize the data preservation function of the Minio file server.
Through the scheme, the data preservation processing is carried out on the Minio file server by utilizing the uploading address in the client system and the storage relative path of the service data in the Minio file server. On one hand, occupied resources of invalid files in the Minio file server are released, and the resource utilization rate of the file server is improved; on the other hand, the Minio file server can be optimized regularly, high-efficiency response of the Minio file server is guaranteed, and the use experience of a user is improved.
Fig. 2 is a schematic structural diagram of a Minio-based data preservation device according to an embodiment of the present application, where the device includes:
at least one processor; and a memory communicatively coupled to the at least one processor. Wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to:
and acquiring an uploading information set of service data corresponding to the current time node. The uploading information set at least comprises an uploading address of the business data after the last time node finishes data preservation and an uploading address of the business data from the last time node to the current time node. And sending a business data freshness keeping instruction to a pre-connected Minio file server to obtain a storage bucket information set corresponding to the current time node in the Minio file server. The storage bucket information set at least comprises storage relative paths of all uploaded service data in all storage buckets. Based on the comparison result of the uploading information set and the storage bucket information set, invalid service data in the storage bucket information set under the current time node is determined to be unreachable data, so that a data preservation instruction is generated, and the unreachable data is removed from the Minio file server. Wherein the invalid traffic data includes uploaded traffic data for which the stored relative path does not match the upload address.
The embodiment of the application also provides a data preservation nonvolatile computer storage medium based on Minio, which stores computer executable instructions, wherein the computer executable instructions are set as follows:
and acquiring an uploading information set of service data corresponding to the current time node. The uploading information set at least comprises an uploading address of the business data after the last time node finishes data preservation and an uploading address of the business data from the last time node to the current time node. And sending a business data freshness keeping instruction to a pre-connected Minio file server to obtain a storage bucket information set corresponding to the current time node in the Minio file server. The storage bucket information set at least comprises storage relative paths of all uploaded service data in all storage buckets. Based on the comparison result of the uploading information set and the storage bucket information set, invalid service data in the storage bucket information set under the current time node is determined to be unreachable data, so that a data preservation instruction is generated, and the unreachable data is removed from the Minio file server. Wherein the invalid traffic data includes uploaded traffic data for which the stored relative path does not match the upload address.
In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable Gate Array, FPGA)) is an integrated circuit whose logic function is determined by the programming of the device by a user. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented by using "logic compiler" software, which is similar to the software compiler used in program development and writing, and the original code before the compiling is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but not just one of the hdds, but a plurality of kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), lava, lola, myHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.
The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.
The apparatus set forth in the above embodiments may be embodied by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
It will be apparent to those skilled in the art that the present embodiments may be provided as a method, apparatus, or computer program product. Accordingly, the present specification embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description embodiments may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present description is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
The description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for apparatus, non-volatile computer storage medium embodiments, the description is relatively simple, as it is substantially similar to method embodiments, with reference to the section of the method embodiments being relevant.
The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
The foregoing is merely one or more embodiments of the present description and is not intended to limit the present description. Various modifications and alterations to one or more embodiments of this description will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, or the like, which is within the spirit and principles of one or more embodiments of the present description, is intended to be included within the scope of the claims of the present description.

Claims (10)

1. A Minio-based data preservation method, the method comprising:
acquiring an uploading information set of service data corresponding to a current time node; the uploading information set at least comprises an uploading address of the business data after the last time node finishes data preservation and an uploading address of the business data from the last time node to the current time node;
a business data fresh-keeping instruction is sent to a pre-connected Minio file server so as to obtain a storage bucket information set corresponding to the current time node in the Minio file server; the storage bucket information set at least comprises storage relative paths of all uploaded service data in all storage buckets;
based on the comparison result of the uploading information set and the storage bucket information set, determining invalid service data in the storage bucket information set under the current time node as unreachable data, so as to generate a data preservation instruction, and rejecting the unreachable data from the Minio file server; wherein the invalid service data includes uploaded service data whose stored relative path does not match the upload address.
2. The Minio-based data preservation method according to claim 1, wherein the step of obtaining the uploading information set of the service data corresponding to the current time node comprises the following steps:
under the condition that the current time node meets a preset timing fresh-keeping task, determining a stored data record of the service data corresponding to the current time node from a preset service data uploading database; the stored data record stores the uploading address of the service data in a data line; the uploading address at least comprises a storage relative path of the service data;
and generating the uploading information set according to the stored data record.
3. The Minio-based data preservation method according to claim 1, wherein a service data preservation instruction is sent to a Minio file server connected in advance to obtain a bucket information set corresponding to the current time node in the Minio file server, and specifically comprises:
and sending a business data freshness preservation instruction to a pre-connected Minio file server, calling a storage bucket traversing interface of the Minio file server through the business data freshness preservation instruction so as to traverse a storage bucket in the Minio file server, and adding the storage relative path of the uploaded business data in the traversed storage bucket to the storage bucket information set.
4. The Minio-based data preservation method according to claim 1, wherein invalid service data in the bucket information set at the current time node is determined to be unreachable data based on a comparison result of the uploaded information set and the bucket information set, and specifically comprises:
comparing the uploading information set with the storage bucket information set, and calculating a difference set of the storage bucket information set and the uploading information set;
and in the case that the difference set is not an empty set, taking the storage relative path in the difference set as the storage relative path of the unreachable data, and determining the unreachable data.
5. The Minio-based data preservation method according to claim 4, wherein comparing the uploaded information set with the bucket information set comprises:
and comparing each uploading address in the uploading information set with each storage relative path in the storage bucket information set, and determining whether the uploading address is consistent with the storage relative path or not.
6. The Minio-based data preservation method according to claim 1, wherein the uploaded information set corresponds to at least one user of the Minio file server; the Minio file server corresponds to a plurality of users.
7. The method of claim 6, wherein the Minio file server stores at least one bucket, each bucket having at least one uploaded service data and a corresponding stored relative path.
8. The Minio-based data preservation method according to claim 1, wherein the unreachable data comprises at least: repeating the uploaded service data and the service data of the uploading address which is not stored by the data line of the service data uploading database.
9. A Minio-based data preservation apparatus, the apparatus comprising:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a Minio-based data preservation method according to any one of claims 1-8.
10. A Minio-based data preservation non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are capable of performing a Minio-based data preservation method as claimed in any one of claims 1-8.
CN202310567257.4A 2023-05-17 2023-05-17 Minio-based data preservation method, device and medium Active CN116737367B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310567257.4A CN116737367B (en) 2023-05-17 2023-05-17 Minio-based data preservation method, device and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310567257.4A CN116737367B (en) 2023-05-17 2023-05-17 Minio-based data preservation method, device and medium

Publications (2)

Publication Number Publication Date
CN116737367A true CN116737367A (en) 2023-09-12
CN116737367B CN116737367B (en) 2024-01-16

Family

ID=87905359

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310567257.4A Active CN116737367B (en) 2023-05-17 2023-05-17 Minio-based data preservation method, device and medium

Country Status (1)

Country Link
CN (1) CN116737367B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110703643A (en) * 2019-09-25 2020-01-17 西安天和防务技术股份有限公司 Military and civil fusion type national defense mobilization comprehensive information system and method
CN111639068A (en) * 2020-05-24 2020-09-08 中信银行股份有限公司 Multi-system-based public data pool generation method, device, equipment and readable storage medium
US11070620B1 (en) * 2020-03-26 2021-07-20 EMC IP Holding Company LLC Efficient transfer to and from a deduplicated cloud storage system
CN114049385A (en) * 2021-10-15 2022-02-15 哈尔滨工业大学（威海） Self-adaptive registration-based multi-laser radar autonomous network building system, method and terminal
US20220214810A1 (en) * 2021-03-26 2022-07-07 Ian F. Adams Near-data processing in sharded storage environments
CN115150419A (en) * 2022-09-05 2022-10-04 杭州华卓信息科技有限公司 Configuration and access method and system for hybrid cloud object storage

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110703643A (en) * 2019-09-25 2020-01-17 西安天和防务技术股份有限公司 Military and civil fusion type national defense mobilization comprehensive information system and method
US11070620B1 (en) * 2020-03-26 2021-07-20 EMC IP Holding Company LLC Efficient transfer to and from a deduplicated cloud storage system
CN111639068A (en) * 2020-05-24 2020-09-08 中信银行股份有限公司 Multi-system-based public data pool generation method, device, equipment and readable storage medium
US20220214810A1 (en) * 2021-03-26 2022-07-07 Ian F. Adams Near-data processing in sharded storage environments
CN114049385A (en) * 2021-10-15 2022-02-15 哈尔滨工业大学（威海） Self-adaptive registration-based multi-laser radar autonomous network building system, method and terminal
CN115150419A (en) * 2022-09-05 2022-10-04 杭州华卓信息科技有限公司 Configuration and access method and system for hybrid cloud object storage

Also Published As

Publication number Publication date
CN116737367B (en) 2024-01-16

Similar Documents

Publication Publication Date Title
EP3547129A1 (en) Block chain consensus method and device
CN108628688B (en) Message processing method, device and equipment
CN107786630B (en) Web application package processing method, device and equipment
CN110807013B (en) Data migration method and device for distributed data storage cluster
CN110263050B (en) Data processing method, device, equipment and storage medium
CN111767499A (en) Page configuration method and device
CN115617799A (en) Data storage method, device, equipment and storage medium
CN110046052B (en) Service calling method and device
CN116737367B (en) Minio-based data preservation method, device and medium
CN110083602B (en) Method and device for data storage and data processing based on hive table
CN108769152B (en) Service refresh policy registration method, service refresh request method, device and equipment
CN108536759B (en) Sample playback data access method and device
CN112464049B (en) Method, device and equipment for downloading number detail list
CN108428189B (en) Social resource processing method and device and readable medium
CN108628615B (en) Method, device and equipment for detecting abandoned codes
CN109582938B (en) Report generation method and device
CN110502551A (en) Data read-write method, system and infrastructure component
CN116644090B (en) Data query method, device, equipment and medium
CN117081931B (en) Online capacity expansion method and device for heterogeneous distributed storage system
CN117555697B (en) Distributed training-oriented cache loading system, method, device and equipment
CN117348999B (en) Service execution system and service execution method
CN117041980B (en) Network element management method and device, storage medium and electronic equipment
CN116226225A (en) Single number generation method, device and equipment
CN113011165B (en) Method, device, equipment and medium for identifying blocked keywords
CN117519912B (en) Mirror image warehouse deployment method, device, storage medium and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant