CN114691727A - Data processing method, device and system, electronic equipment and storage medium - Google Patents

Data processing method, device and system, electronic equipment and storage medium Download PDF

Info

Publication number
CN114691727A
CN114691727A CN202011627337.7A CN202011627337A CN114691727A CN 114691727 A CN114691727 A CN 114691727A CN 202011627337 A CN202011627337 A CN 202011627337A CN 114691727 A CN114691727 A CN 114691727A
Authority
CN
China
Prior art keywords
data
information
storage
instruction
stored
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011627337.7A
Other languages
Chinese (zh)
Inventor
钟亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Intellifusion Technologies Co Ltd
Original Assignee
Shenzhen Intellifusion Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Intellifusion Technologies Co Ltd filed Critical Shenzhen Intellifusion Technologies Co Ltd
Priority to CN202011627337.7A priority Critical patent/CN114691727A/en
Publication of CN114691727A publication Critical patent/CN114691727A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

The embodiment of the invention provides a data processing method, a device, a system, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a data processing instruction from the front end of a first system, wherein the data processing instruction comprises a data query instruction or a data storage instruction; analyzing the data processing instruction to obtain an instruction analysis result, wherein the instruction analysis result comprises data query information or data storage information, and the data storage information comprises data to be stored; if the instruction analysis result comprises the data storage information, storing the data to be stored to a second system according to the data storage information, wherein the first system and the second system operate independently; and if the instruction analysis result comprises the data query information, acquiring data from the first system or the second system according to the data query information and returning. By integrating the resources of the first system and the second system, the resource utilization rate of the system can be improved.

Description

Data processing method, device and system, electronic equipment and storage medium
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method, apparatus, system, electronic device, and storage medium.
Background
Fastdfs is an open-source lightweight distributed file system, aims to realize high availability and high concurrency of file operation, is particularly suitable for storage operation of small and medium files, is particularly suitable for online network services taking files as carriers, such as photo album websites, video websites and the like, and is widely applied to various domestic Internet enterprises due to the three characteristics of unique lightweight, group storage and peer-to-peer structure. Usually, storage nodes of a single Fastdfs cluster are deployed on a storage server (StorageServer) of an enterprise intranet, when the enterprise has a large scale and has more data, a plurality of data storage systems are often deployed, and the data storage systems may be a plurality of Fastdfs clusters between the intranet or other Fastdfs clusters of an extranet. Because the Fastdfs is only responsible for uploading and downloading files and does not support data sharing and storage space sharing among Fastdfs clusters of a plurality of internal and external networks, the disaster tolerance capability of the system is weak, and the resources of the Fastdfs clusters cannot be integrated, so that the resource utilization rate of the storage system is low.
Disclosure of Invention
The embodiment of the invention provides a data processing method which can improve the resource utilization rate of a storage system.
In a first aspect, an embodiment of the present invention provides a data processing method, including the following steps:
acquiring a data processing instruction from the front end of a first system, wherein the data processing instruction comprises a data query instruction or a data storage instruction;
analyzing the data processing instruction to obtain an instruction analysis result, wherein the instruction analysis result comprises data query information or data storage information, and the data storage information comprises data to be stored;
if the instruction analysis result comprises the data storage information, storing the data to be stored into a second system according to the data storage information, wherein the first system and the second system operate independently and share the data;
and if the instruction analysis result comprises the data query information, acquiring data from the first system or the second system according to the data query information and returning.
Optionally, the data storage information further includes second path information, and the storing the data to be stored to the second system according to the data storage information includes:
acquiring at least one second scheduling information from the second system according to the second path information;
and storing the data to be stored to at least one storage module of the second system according to the second scheduling information, and acquiring a storage result.
Optionally, the data storage information further includes first path information, and if the storage result includes failure information, the method further includes:
acquiring at least one piece of first scheduling information from the first system according to the first path information;
and storing the data to be stored to at least one storage module of the first system according to the first scheduling information.
Optionally, the data query information includes path information and a data source identifier to be queried, where the path information includes the first system identifier or the second system identifier.
Optionally, if the storage result includes success information, the method further includes:
and carrying out data synchronization on the data to be stored among the storage modules of the second system according to a preset synchronization rule.
Optionally, the step of querying information according to the data to the first system or the second system to obtain data includes:
acquiring the path information and the identifier of the data source to be queried in the data query information;
if the path information comprises the second system identification, determining a storage address of the data to be queried in the second system according to the identification of the data to be queried, and acquiring data in the second system according to the storage address;
and if the path information comprises the identifier of the first system, directly acquiring data in the first system.
In a second aspect, an embodiment of the present invention provides a data processing apparatus, including:
the system comprises a first acquisition module, a second acquisition module and a data processing module, wherein the first acquisition module is used for acquiring a data processing instruction from the front end of a first system, and the data processing instruction comprises a data query instruction or a data storage instruction;
the analysis module is used for analyzing the data processing instruction to obtain an instruction analysis result, wherein the instruction analysis result comprises data query information or data storage information, and the data storage information comprises data to be stored;
the storage module is used for storing the data to be stored to a second system according to the data storage information if the instruction analysis result comprises the data storage information, and the first system and the second system operate independently and share the data;
and the second acquisition module is used for acquiring data from the first system or the second system according to the data query information and returning the data if the instruction analysis result comprises the data query information.
In a third aspect, an embodiment of the present invention provides a data processing system, including: the system comprises a first system, a second system and the data processing device, wherein the first system comprises a front-end module and a rear-end module, and the front-end module is used for providing a human-computer interaction interface and acquiring an input instruction of a user; the data processing device executes the data processing method, acquires and analyzes the input instruction of the front-end module of the first system, and then judges whether to upload data to the back-end module or the second system of the first system or acquire data from the first system or the second system and return the data
In a fourth aspect, an embodiment of the present invention provides an electronic device, including: the data processing method comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor executes the computer program to realize the steps in the data processing method provided by the embodiment of the invention.
In a fifth aspect, the embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements the steps in the data processing method provided by the embodiment of the present invention.
In the embodiment of the invention, a data processing instruction is obtained from the front end of a first system, wherein the data processing instruction comprises a data query instruction or a data storage instruction; analyzing the data processing instruction to obtain an instruction analysis result, wherein the instruction analysis result comprises data query information or data storage information, and the data storage information comprises data to be stored; if the instruction analysis result comprises the data storage information, storing the data to be stored to a second system according to the data storage information, wherein the first system and the second system operate independently; and if the instruction analysis result comprises the data query information, acquiring data from the first system or the second system according to the data query information and returning. The method comprises the steps of obtaining a data processing instruction from the front end of a first system, analyzing data storage information from the data processing instruction, storing data to be stored to a second system which operates independently according to the data storage information, enabling the second system to share the data to be stored to the first system, or analyzing data query information, obtaining data from the first system or the second system according to the data query information, integrating resources of the first system and the second system and sharing the resources, and therefore the disaster tolerance capability of the system is improved, and meanwhile the resource utilization rate of the system is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without any creative effort.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a data storage method according to an embodiment of the present invention;
FIG. 3 is a flow chart of another data storage method provided by an embodiment of the invention;
FIG. 4 is a flow chart of a data acquisition method according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a data storage device according to an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of another data storage device according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a data acquisition apparatus according to an embodiment of the present invention;
FIG. 9 is a block diagram of a data processing system according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures, are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the above-described drawings are used for distinguishing between different objects and not for describing a particular order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The following description is provided to illustrate the relevant background information in the embodiments of the present invention. The first system and the second system in the embodiment of the invention are realized based on the existing single Fastdfs distributed file system, the Fastdfs file system is mainly used for file storage, file synchronization, file access (file uploading and file downloading) and the like, and the system has three roles: a tracking server (tracker server), a storage server (storageserver), and a client (client). The client is used for providing a system access entrance, such as uploading and downloading files; the tracking server mainly performs scheduling work, plays a role in load balancing and is responsible for managing all storage servers and all storage nodes in the storage servers; each storage node is divided into a group, and after being started, each storage node is connected with a tracking server to inform the tracking server of the information of the group to which the storage node belongs, and periodic heartbeat information is kept to be sent to the tracking server, and the tracking server establishes a mapping table from the storage node group to the storage server according to the heartbeat information of the storage node; the plurality of tracking servers are in a peer-to-peer relationship, so that the tracking server service is very easy to expand, all the tracking servers receive heartbeat information of the storage nodes and generate metadata information in the tracking servers to provide read-write service; the storage servers mainly provide storage capacity and data backup service, and also can take groups as units, each group can contain a plurality of storage servers, and data in the groups are mutually backed up; the storage space is organized by taking the group as a unit, application isolation, load balancing and backup number customization can be conveniently carried out, if the machine in the group is broken down, data recovery can be synchronized again through other machines in the group, and the exception handling capacity of the system can be improved.
The Fastdfs system of the first system in the embodiment of the invention is mainly used for temporary caching of data and storage of partial data during large data processing, the Fastdfs system of the second system is used for preferentially storing the data and serving as a backup of the first system, the first system and the second system are only convenient to distinguish, and actually the positions and functions of the two systems can be interchanged.
Referring to fig. 1, fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, as shown in fig. 1, including the following steps:
101. data processing instructions are obtained from a front end of a first system.
The data processing instruction comprises a data query instruction or a data storage instruction.
In an embodiment of the present invention, the front end of the first system may be a visual interface of the first system based on the Fastdfs file storage system, such as a GUI or a WEB page, or may be a command line provided by the first system; the data processing instruction may be obtained through a command line entry at the front end of the first system, may also be obtained through a visual interface of the first system, such as a GUI or a WEB page, and may also be obtained and converted through a front end voice recognition of the first system. The data processing instruction may include at least one data query instruction or at least one data storage instruction, and may also include other data processing instructions, such as data deletion, addition, and the like; the data query instruction may be a data acquisition instruction, such as a file download instruction; the data storage instruction may be a data uploading instruction, such as a picture data uploading instruction; the data processing instruction may be in the form of a character string, or in the form of other digital formats set according to a preset rule, such as an instruction in a binary format.
102. And analyzing the data processing instruction to obtain an instruction analysis result.
The instruction analysis result comprises data query information or data storage information, and the data storage information comprises data to be stored.
In the embodiment of the present invention, the instruction analysis may be performed after the data processing instruction is obtained in step 101, so as to obtain a corresponding analysis result, that is, data query information or storage information including data to be stored may be obtained. For example, firstly, format analysis is performed on the data processing instruction, if the format of the data processing instruction is a character string, such as a hypertext transfer protocol HTTP type data query instruction, a corresponding file downloading instruction may be a host/port/downLoad/system1/Fastdfs/tracker. For another example, the data storage instruction in the string format includes upload/system2/Fastdfs/tracker, storage, group, aadoc, which may be performed through a preset parsing rule, and if the field between "/" is extracted through a regular expression, the data storage information including the data to be stored is obtained and then temporarily stored in a memory in a certain format, for example, a key-value format is used for caching, for example, an upload identifier of data stored in an upload key field, path information of a path field, data field cache of data to be stored, and the like.
103. And if the instruction analysis result comprises data storage information, storing the data to be stored to the second system according to the data storage information.
The first system and the second system operate independently and share data.
In the embodiment of the present invention, the instruction analysis result obtained in step 102 may be judged to determine whether the next data processing flow is data uploading or downloading; specifically, if the obtained instruction analysis result includes the data storage information, for example, the analyzed information includes a character string upload field or other preset instruction identifiers (e.g., 0001, # # # and the like), a data uploading process is performed, data to be stored is obtained from a data field in the data storage information, a storage path is obtained according to a path field in the data storage information, and the data to be stored is preferentially stored in a corresponding position in the second system according to the storage path.
Further, in the embodiment of the present invention, the first system is mainly configured to provide a front-end entry for data processing and a back-end for large data processing and data caching, and is independently deployed in an intranet or an extranet, and can be accessed through the front end of the visual WEB page or the command line; the second system is independently deployed in the intranet and used for preferentially storing data and serving as a data backup of the first system. The first system and the second system can be respectively deployed on different computer server hardware and different operating systems in a docker mirror image mode and independently run, such as a windows system and a linux system, the two systems can communicate through addresses (such as IP addresses) in a network to realize synchronous data sharing and storage space sharing, and resources of the first system and the second system are integrated, so that the disaster tolerance capability of the systems for data processing and the utilization rate of the resources can be improved.
Optionally, referring to fig. 2, fig. 2 is a flowchart of a data storage method according to an embodiment of the present invention, as shown in fig. 2, where the data storage information further includes second path information, and the step of storing the data to be stored in the second system according to the data storage information specifically includes:
201. and acquiring at least one piece of second scheduling information from the second system according to the second path information.
In the embodiment of the present invention, the second path information may be used to identify storage path information of data to be stored, such as a stored system, a specific storage location, and the like; the second scheduling information includes second system address information of the second system in the network and access path information of the Fastdfs storage system in the second system, and the access path information includes address information of a tracking server and address information of a storage server of the Fastdfs storage system and group information. Specifically, first, the second path information, such as system2/Fastdfs/tracker storage group, is obtained from a path field in the data storage information, and the second path information of the character string is divided to obtain system2 which can be used as an identifier of the second system, Fastdfs which can be used as an identifier of the Fastdfs storage system, and tracker storage group which can be used as optional target storage path information (which can be empty); and finally, obtaining a running example of the corresponding Fastdfs system from the second system according to the Fastdfs storage system identification, and cutting a packet field to obtain the address information of the tracking server, the address information of the storage server and the group information.
202. And storing the data to be stored to at least one storage module of the second system according to the second scheduling information, and acquiring a storage result.
In the embodiment of the present invention, a target storage location of the data to be stored in the second system may be obtained based on the second scheduling information, then the size of the data to be stored is calculated, and a storage module with the same size is obtained from a storage space of the second system and the data to be stored is written into the storage module. Specifically, the data to be stored may be obtained from a data field in the data storage information, and may be a picture, a segment of video, or a word document, and may be binary data or encoded character string data. Then, positioning the position of the second system in the internal network according to second system address information in the second scheduling information, and acquiring an operation example of the Fastdfs system from the second system, such as a process identifier of the Fastdfs system, wherein the second system address information can be an intranet IP address; further, the corresponding tracking server, storage server and group can be obtained from the running instance of the Fastdfs system according to the address information of the tracking server, the address information of the storage server and the group information in the second scheduling information, and then the data to be stored is transmitted to the storage server through the tracking server and written into the storage space in the corresponding group, so that the data to be stored is written into the storage module of the second system, and successful information (such as 1, success and the like) is returned as the storage result; if the data storage fails, for example, the acquisition of the running instance of the Fastdfs system fails, or the data to be stored exceeds the storage space of the storage module, the failed information (such as 0, fai, etc.) is returned as the storage result. If the address information of the tracking server, the address information of the storage server and the group information comprise null information, the data to be stored can be stored in a group with larger storage space on the storage server of the Fastdfs system operation example, so that the storage space is fully utilized, and the resource utilization rate is improved.
Further, the second system may synchronize the data to be stored to the first system for data sharing, specifically, address information (e.g., an IP address) of the first system in the network may be stored in the second system, and when data synchronization is required, an interface of the Fastdfs storage system of the first system is obtained according to the address information of the first system, and then the data is transmitted to the Fastdfs storage system of the first system in the network through the interface, so that data sharing between the two systems is achieved.
Optionally, if the storage result includes success information, the data processing method further includes:
and carrying out data synchronization on the data to be stored among the storage modules of the second system according to a preset synchronization rule.
In the embodiment of the present invention, the storage module of the second system includes storage nodes in the storage server of the Fastdfs system, through the data storage step, the data to be stored is finally written into the storage nodes, a plurality of storage nodes form a storage node group (i.e. the storage module), different storage node groups can store data of different data sources and can use names of the data sources as identifiers of corresponding storage node groups, and when storing data, the identifiers of the storage node groups can be used as prefixes of storage names of the data to be stored to distinguish different data sources; in the storage server, each storage node group has at least one backup group with equal status, and when data is stored in one storage node group, the data can be correspondingly stored in another backup group according to a preset synchronization rule, so that the data is synchronized between the storage node groups (storage modules). The preset synchronization rule may be an incremental synchronization rule, that is, when the storage server of the Fastdfs system is in an appropriate time (for example, when there is no data read/write), the newly added data is asynchronously copied from one storage node group to a corresponding backup group, so as to improve the stability of the system.
Optionally, referring to fig. 3, fig. 3 is a flowchart of another data storage method provided in an embodiment of the present invention, and as shown in fig. 3, the data storage information further includes first path information, and if the storage result includes failure information, the data storage method further includes:
301. and acquiring at least one piece of first scheduling information from the first system according to the first path information.
In the embodiment of the present invention, the first path information may be used to identify storage path information of data to be stored, such as a stored system, a specific storage location, and the like; the first scheduling information comprises first system address information of a first system in a network and access path information of a Fastdfs storage system in the first system, and the access path information comprises address information of a tracking server and address information of a storage server of the Fastdfs storage system and group information. Specifically, first, the first path information, such as system1/Fastdfs/tracker storage group, is obtained from a path field in the data storage information, and the first path information of the character string is divided to obtain system1 which can be used as an identifier of the first system, Fastdfs which can be used as an identifier of the Fastdfs storage system, and tracker storage group which can be used as optional target storage path information (which can be empty); the address information of the first system is obtained from the first system according to the first system identifier, the address (such as an IP address, an MAC address, and the like) of the first system in the network can be obtained, finally, the running instance corresponding to the Fastdfs system can be obtained from the first system according to the Fastdfs storage system identifier, and the packet field is cut to obtain the address information of the tracking server, the address information of the storage server, and the group information.
302. And storing the data to be stored to at least one storage module of the first system according to the first scheduling information.
In the embodiment of the present invention, a target storage location of the data to be stored in the first system may be obtained based on the first scheduling information, then the size of the data to be stored is calculated, and a storage module with the same size is obtained from a storage space of the first system and the data to be stored is written into the storage module. Specifically, the data to be stored may be obtained from a data field in the data storage information, and may be a picture, a segment of video, a word document, binary data, or encoded character string data. Then, the position of the first system in the internal network is located according to the first system address information in the first scheduling information, and an operation example of the Fastdfs system, such as a process identifier of the Fastdfs system, is obtained from the first system, wherein the first system address information can be an intranet IP address; further, the corresponding tracking server, storage server and group can be obtained from the operation instance of the Fastdfs system according to the address information of the tracking server, the address information of the storage server and the group information in the first scheduling information, and then the data to be stored is transmitted to the storage server through the tracking server and written into the storage space in the corresponding group, so that the data to be stored is written into the storage module of the first system, and successful information (such as 1, success and the like) is returned as the storage result; if the data storage fails, for example, the acquisition of the running instance of the Fastdfs system fails, or the data to be stored exceeds the storage space of the storage module, the failed information (such as 0, fai, etc.) is returned as the storage result. If the address information of the tracking server, the address information of the storage server and the group information comprise null information, the data to be stored can be stored in a group with larger storage space on the storage server of the Fastdfs system operation example, so that the storage space is fully utilized, and the resource utilization rate is improved.
Further, the first system may synchronize the data to be stored to the second system for data sharing, specifically, address information (e.g., an IP address) of the second system in a network may be stored in the first system, and when data synchronization is required, the first system may communicate with the second system according to the address information of the second system and obtain an interface of the Fastdfs storage system of the second system, and then transmit the data to the Fastdfs storage system of the second system in the network through the interface, thereby implementing data sharing between the two systems.
Optionally, the data query information includes path information and an identifier of a data source to be queried, where the path information includes an identifier of the first system or an identifier of the second system.
The data query information is obtained by parsing a data query instruction obtained from the front end of the first system, for example, in step 102, the data query instruction host/port/downLoad/system1/Fastdfs/tracker storage.group.2122.jpg in a string format may be parsed according to a path parsing method of a hypertext transfer protocol to obtain the data query information, and the data query information is further obtained by performing a segmentation process on the data query information, where the data query information includes path information and a to-be-queried data source identifier, the path information includes an identifier of the first system (e.g., system1 field, or 0001 field in the data query instruction) or an identifier of the second system (e.g., system2 field, or 0010 field in the data query instruction), and the to-be-queried data source identifier may be an identifier of the set of storage nodes of the fastfs storage system in which the to-be-queried data is located (e.g., the to-be-queried data source identifier may be an identifier of the set of the fastfs storage nodes in which the to-be-queried data is located The group.2122 field in the data query instruction).
104. And if the instruction analysis result comprises data query information, acquiring data from the first system or the second system according to the data query information and returning.
In this embodiment of the present invention, the instruction analysis result obtained in step 102 is determined, and if the instruction analysis result includes data query information, it indicates that the original data processing instruction includes a data acquisition instruction, such as a string-format hypertext transfer protocol HTTP-type data acquisition instruction host/port/downLoad/system1/Fastdfs/tracker.storage.group.2122.jpg, and the data query information may be obtained by splitting the data acquisition instruction according to a "/", where a host/port field indicates a local host address and port information, and a downLoad field may be used to indicate a data acquisition identifier; the data query information includes data to be queried and a specific storage path of the data, for example, an identifier of a storage system may be obtained according to a system1 field and Fastdfs, and a specific storage path and data to be queried may be obtained according to a tracker.
Optionally, referring to fig. 4, fig. 4 is a flowchart of a data obtaining method according to an embodiment of the present invention, and as shown in fig. 4, the step of obtaining data from the first system or the second system according to the data query information includes:
401. and acquiring path information and a to-be-queried data source identifier in the data query information.
402. If the path information comprises the identifier of the second system, determining the storage address of the data to be inquired in the second system according to the identifier of the data to be inquired, and acquiring the data in the second system according to the storage address.
403. And if the path information comprises the identification of the first system, directly acquiring the data in the first system.
In the embodiment of the invention, the path information and the identifier of the data source to be queried are obtained from the data query information, then, judging whether the path information contains the identifier of the second system or the identifier of the first system, if the path information includes the identifier of the second system (i.e. system2 field in the original data query instruction, or 0010, etc.), first obtaining the address information of the second system in the network, further determining the storage address of the data to be queried in the second system according to the identifier of the data to be queried (e.g. group.2122 field in the data query instruction), namely the storage node group identification of the Fastdfs running instance in the second system, a complete data acquisition path can be obtained through the address information of the second system in the network and the storage address, and then reading the data to be acquired from the corresponding storage node in the second system based on the data acquisition path.
If the path information includes the identifier of the first system (i.e. system1 field in the original data query instruction, or 0001, etc.), then, similarly, a complete data acquisition path is obtained through the address information of the first system in the network and the storage address, and then the data to be acquired is read from the corresponding storage node in the first system based on the data acquisition path.
To sum up, the data processing instruction is obtained from the front end of the first system, the data storage information is analyzed from the data processing instruction, the data to be stored is stored in the second system which operates independently according to the data storage information, or the data is obtained from the first system or the second system according to the data query information after the data query information is analyzed, and the resources of the first system and the second system are integrated, so that the disaster tolerance capability of the system is improved, and the resource utilization rate of the system is improved.
It should be noted that the data processing method provided in the embodiment of the present invention may be applied to a device that can perform data processing, such as a mobile phone, a monitor, a computer, and a server.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention, and as shown in fig. 5, the apparatus 500 includes:
a first obtaining module 501, configured to obtain a data processing instruction from a front end of a first system, where the data processing instruction includes a data query instruction or a data storage instruction;
an analysis module 502, configured to analyze the data processing instruction to obtain an instruction analysis result, where the instruction analysis result includes data query information or data storage information, and the data storage information includes data to be stored;
a storage module 503, configured to store the data to be stored in a second system according to the data storage information if the instruction parsing result includes the data storage information, where the first system and the second system operate independently and perform data sharing;
a second obtaining module 504, configured to, if the instruction parsing result includes the data query information, obtain data from the first system or the second system according to the data query information, and return the data.
Optionally, as shown in fig. 6, fig. 6 is a schematic structural diagram of a data storage device according to an embodiment of the present invention, where the data storage information further includes second path information, and the data storage device 600 includes:
an obtaining module 601, configured to obtain at least one piece of second scheduling information from the second system according to the second path information;
the storage module 602 is configured to store the data to be stored in the at least one storage module of the second system according to the second scheduling information, and obtain a storage result.
Optionally, as shown in fig. 7, fig. 7 is a schematic structural diagram of another data storage device according to an embodiment of the present invention, where the data storage information further includes first path information, and if the storage result includes failure information, the data storage device 700 further includes:
an obtaining module 701, configured to obtain at least one first scheduling information from a first system according to the first path information;
a storage module 702, configured to store data to be stored to at least one storage module of the first system according to the first scheduling information.
Optionally, as shown in fig. 8, fig. 8 is a schematic structural diagram of a data acquisition apparatus according to an embodiment of the present invention, where the data acquisition apparatus 800 includes:
a first obtaining module 801, configured to obtain path information and a to-be-queried data source identifier in data query information;
a second obtaining module 802, configured to determine, according to the identifier of the data to be queried, a storage address of the data to be queried in the second system if the path information includes the identifier of the second system, and obtain the data in the second system according to the storage address;
a third obtaining module 803, configured to directly obtain data in the first system if the path information includes an identifier of the first system.
It should be noted that the data processing apparatus provided in the embodiment of the present invention may be applied to a device that can perform data processing, such as a mobile phone, a monitor, a computer, and a server.
The data processing device provided by the embodiment of the invention can realize each process realized by the data processing method in the method embodiment, and can achieve the same beneficial effect. To avoid repetition, it is not described here in detail.
Referring to fig. 9, fig. 9 is a schematic structural diagram of a data processing system according to an embodiment of the present invention, and as shown in fig. 9, the data processing system 900 includes: the system comprises a first system 901, a second system 902 and the data processing device 500, wherein the first system 901 comprises a front-end module 9011 and a rear-end module 9012, and the front-end module 9011 is used for providing a human-computer interaction interface and acquiring an input instruction of a user; the data processing apparatus 500 executes the data processing method, acquires and analyzes the input command of the first system front-end module 9011, and determines whether to upload data to the back-end module 9012 or the second system 902 of the first system 901 or acquire data from the first system 901 or the second system 902 and return the data.
The first system 901 and the second system 902 are both constructed based on a Fastdfs storage system, and can be deployed independently in the same intranet or in different networks, and perform data sharing through the networks; the first system 901 may be a first system in the data processing method, and is mainly used for performing large data processing and performing temporary caching of data and storing part of data; the second system 902 may be a second system in the data processing method, and is configured to preferentially store data and serve as a backup of the first system 901. By integrating the resources of the first system 901 and the second system 902, the disaster tolerance capability of the system 900 is improved, and the resource utilization rate of the system 900 is also improved.
Referring to fig. 10, fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, as shown in fig. 10, including: a memory 1002, a processor 1001 and a computer program stored on said memory 1002 and executable on said processor 1001, wherein:
the processor 1001 is used for calling the computer program stored in the memory 1002, and executes the following steps:
acquiring a data processing instruction from the front end of a first system, wherein the data processing instruction comprises a data query instruction or a data storage instruction;
analyzing the data processing instruction to obtain an instruction analysis result, wherein the instruction analysis result comprises data query information or data storage information, and the data storage information comprises data to be stored;
if the instruction analysis result comprises the data storage information, storing the data to be stored into a second system according to the data storage information, wherein the first system and the second system operate independently and perform data sharing;
and if the instruction analysis result comprises the data query information, acquiring data from the first system or the second system according to the data query information and returning.
Optionally, the data storage information further includes second path information, and the storing, performed by the processor 1001, the data to be stored in the second system according to the data storage information includes:
acquiring at least one second scheduling information from the second system according to the second path information;
and storing the data to be stored to at least one storage module of the second system according to the second scheduling information, and acquiring a storage result.
Optionally, the data storage information further includes first path information, and if the storage result includes failure information, the method executed by the processor 1001 further includes:
acquiring at least one piece of first scheduling information from the first system according to the first path information;
and storing the data to be stored to at least one storage module of the first system according to the first scheduling information.
Optionally, if the storage result includes success information, the method performed by the processor 1001 further includes:
and carrying out data synchronization on the data to be stored among the storage modules of the second system according to a preset synchronization rule.
Optionally, the step of acquiring data from the first system or the second system according to the data query information, which is executed by the processor 1001, includes:
acquiring the path information and the to-be-queried data source identifier in the data query information;
if the path information comprises the second system identification, determining a storage address of the data to be queried in the second system according to the data identification to be queried, and acquiring the data in the second system according to the storage address;
and if the path information comprises the identifier of the first system, directly acquiring data in the first system.
The electronic device may be a device that can be applied to a mobile phone, a monitor, a computer, a server, or the like that can perform data processing.
The electronic device provided by the embodiment of the invention can realize each process realized by the data processing method in the method embodiment, and can achieve the same beneficial effects, and in order to avoid repetition, the details are not repeated.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements each process of the data processing method provided in the embodiment of the present invention, and can achieve the same technical effect, and is not described herein again to avoid repetition.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above may be implemented by a computer program, which may be stored in a computer readable storage medium and executed by a computer to implement the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-only memory (ROM), a Random Access Memory (RAM), or the like.
The above disclosure is only for the purpose of illustrating the preferred embodiments of the present invention, and it is therefore to be understood that the invention is not limited by the scope of the appended claims.

Claims (10)

1. A data processing method, characterized by comprising the steps of:
acquiring a data processing instruction from the front end of a first system, wherein the data processing instruction comprises a data query instruction or a data storage instruction;
analyzing the data processing instruction to obtain an instruction analysis result, wherein the instruction analysis result comprises data query information or data storage information, and the data storage information comprises data to be stored;
if the instruction analysis result comprises the data storage information, storing the data to be stored into a second system according to the data storage information, wherein the first system and the second system operate independently and share the data;
and if the instruction analysis result comprises the data query information, acquiring data from the first system or the second system according to the data query information and returning.
2. The method of claim 1, wherein the data storage information further comprises second path information, and wherein storing the data to be stored to a second system according to the data storage information comprises:
acquiring at least one second scheduling information from the second system according to the second path information;
and storing the data to be stored to at least one storage module of the second system according to the second scheduling information, and acquiring a storage result.
3. The method of claim 2, wherein the data storage information further includes first path information, and if the storage result includes failure information, the method further comprises:
acquiring at least one piece of first scheduling information from the first system according to the first path information;
and storing the data to be stored to at least one storage module of the first system according to the first scheduling information.
4. The method of claim 3, wherein the data query information comprises path information and a data source identification to be queried, the path information comprising an identification of the first system or an identification of the second system.
5. The method of claim 2, wherein if success information is included in the stored result, the method further comprises:
and carrying out data synchronization on the data to be stored among the storage modules of the second system according to a preset synchronization rule.
6. The method of claim 4, wherein the step of querying information from the data for data at the first system or the second system comprises:
acquiring the path information and the to-be-queried data source identifier in the data query information;
if the path information comprises the identifier of the second system, determining the storage address of the data to be queried in the second system according to the identifier of the data to be queried, and acquiring the data in the second system according to the storage address;
and if the path information comprises the identifier of the first system, directly acquiring data in the first system.
7. A data processing apparatus, characterized by comprising:
the system comprises a first acquisition module, a second acquisition module and a data processing module, wherein the first acquisition module is used for acquiring a data processing instruction from the front end of a first system, and the data processing instruction comprises a data query instruction or a data storage instruction;
the analysis module is used for analyzing the data processing instruction to obtain an instruction analysis result, wherein the instruction analysis result comprises data query information or data storage information, and the data storage information comprises data to be stored;
the storage module is used for storing the data to be stored to a second system according to the data storage information if the instruction analysis result comprises the data storage information, and the first system and the second system operate independently and share the data;
and the second acquisition module is used for acquiring data from the first system or the second system according to the data query information and returning the data if the instruction analysis result comprises the data query information.
8. A data processing system, comprising: the data processing device comprises a first system, a second system and a data processing device according to claim 7, wherein the first system comprises a front-end module and a back-end module, and the front-end module is used for providing a human-computer interaction interface and acquiring input instructions of a user; the data processing device executes the data processing method according to any one of claims 1 to 6, acquires and analyzes an input instruction of the front-end module of the first system, and then determines whether to upload data to the back-end module of the first system or the second system, or acquire data from the first system or the second system and return the data.
9. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, the processor implementing the steps in the data processing method according to any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, carries out the steps in the data processing method according to any one of claims 1 to 6.
CN202011627337.7A 2020-12-30 2020-12-30 Data processing method, device and system, electronic equipment and storage medium Pending CN114691727A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011627337.7A CN114691727A (en) 2020-12-30 2020-12-30 Data processing method, device and system, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011627337.7A CN114691727A (en) 2020-12-30 2020-12-30 Data processing method, device and system, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN114691727A true CN114691727A (en) 2022-07-01

Family

ID=82133598

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011627337.7A Pending CN114691727A (en) 2020-12-30 2020-12-30 Data processing method, device and system, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114691727A (en)

Similar Documents

Publication Publication Date Title
US9311326B2 (en) Virtual file system for automated data replication and review
CN110971655B (en) Offline client playback and synchronization
CN1988535A (en) Synchronous method, system for file storage and customer terminal
CN105138571B (en) Distributed file system and method for storing massive small files
US10831612B2 (en) Primary node-standby node data transmission method, control node, and database system
US10069941B2 (en) Scalable event-based notifications
KR20210005043A (en) Input and output schema mapping
US20120278429A1 (en) Cluster system, synchronization controlling method, server, and synchronization controlling program
EP3786802A1 (en) Method and device for failover in hbase system
CN107347062A (en) A kind of method, electronic equipment and the readable storage medium storing program for executing of daily record data processing
CN104980307A (en) Processing method of data access requests, processing device of data access requests and database server
CN112100152A (en) Service data processing method, system, server and readable storage medium
CN105610917B (en) Method and system for realizing synchronous data repair in system
CN112347192A (en) Data synchronization method, device, platform and readable medium
CN112491609A (en) System application architecture discovery method and system based on business service
US8090695B2 (en) Dynamic restoration of message object search indexes
US8412676B2 (en) Forgetting items with knowledge based synchronization
CN112148745B (en) Multi-HBase cluster access method, device and storage medium
CN116501700B (en) APP formatted file offline storage method, device, equipment and storage medium
US9852031B2 (en) Computer system and method of identifying a failure
CN114691727A (en) Data processing method, device and system, electronic equipment and storage medium
CN115374078A (en) Data storage and reading system and working method thereof
CN116821232A (en) Data synchronization method and related device
CN111209938A (en) Automatic progress monitoring method, electronic equipment and storage medium
US11379147B2 (en) Method, device, and computer program product for managing storage system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination