CN115374078A

CN115374078A - Data storage and reading system and working method thereof

Info

Publication number: CN115374078A
Application number: CN202211058175.9A
Authority: CN
Inventors: 马涛; 郝伟; 刘加瑞; 吴璇
Original assignee: Beijing Huayun'an Software Co ltd
Current assignee: Beijing Huayun'an Software Co ltd
Priority date: 2022-08-30
Filing date: 2022-08-30
Publication date: 2022-11-22

Abstract

The embodiment of the disclosure provides a data storage and reading system and a working method thereof, belonging to the technical field of big data transmission. The data storage method comprises the steps that a data conversion module receives and converts data uploaded by data production equipment; the data forwarding and recording module uploads the converted data to the data storage system, receives file retrieval information fed back by the data storage system and writes the file retrieval information into the file retrieval information database; and the file retrieval information database receives and stores the file retrieval information written by the data forwarding and recording module. In this way, the high availability and high performance of data acquisition and transmission can be improved, and the method is suitable for data collection and summarization in various distributed data production environments.

Description

Data storage and reading system and working method thereof

Technical Field

The present disclosure relates to the field of big data transmission technologies, and in particular, to the field of data storage and data access technologies.

Background

Large-scale data acquisition using network for data transmission is an important requirement in various fields. A common method is to collect data using a centralized or semi-centralized server. In the prior art, data acquisition is mostly carried out by one data receiving server, the usability is poor, and acquisition errors can be caused once a problem occurs in a summary server. In order to expand data downloading nodes, a user needs to deploy the same program and environment to a new machine, the deployment is troublesome, various different complex scenes need to be considered, the expansibility of the program is poor, the dynamic extension is not suitable, and the process of updating the version is complicated.

Disclosure of Invention

The present disclosure provides a data storage and reading system and a working method thereof.

According to a first aspect of the present disclosure, there is provided a data storage and reading system comprising:

a data forwarding module, a recording module, a data storage system, a file retrieval information database, and a front-end display module,

the data forwarding modules are one or more and are used for receiving data uploaded by data production equipment and uploading the data to the data storage system;

the recording module is used for writing the file retrieval information fed back by the data storage system into the file retrieval information database;

the data storage system is used for storing the data uploaded by the data forwarding module and feeding back file retrieval information to the recording module; the system is also used for returning the appointed file according to the front-end requirement;

the file retrieval information database is used for receiving and storing the file retrieval information written by the recording module; the front-end display module is also used for receiving an access request submitted by a user through the front-end display module;

the front-end display module is used for submitting an access request of a user to the file retrieval information database and receiving a corresponding retrieval result; and downloading the corresponding file from the data storage system according to the retrieval result.

In some implementations of the first aspect, comprising:

the data forwarding module and the recording module are deployed in a POD (platform digital POD) of a K8S-based cloud native platform;

the data storage system is a cluster-based HDFS storage system.

According to a second aspect of the present disclosure, there is provided a method of operating a data storage and reading system, the method comprising:

the data forwarding module receives data uploaded by the data production equipment and uploads the data to the data storage system;

the recording module receives the file retrieval information fed back by the data storage system and writes the file retrieval information into the file retrieval information database;

the file retrieval information database receives and stores the file retrieval information written by the recording module;

in some implementations of the second aspect, the receiving, by the data forwarding module, the data uploaded by the data production device includes:

one or more data forwarding modules respectively receive data uploaded by one or more data production devices through SSH service and store the data in a temporary file buffer area;

in some implementations of the second aspect, the data forwarding module uploading the received data to the data storage system includes:

and the data forwarding uploads the received data to the data storage system according to the timing task.

According to a third aspect of the present disclosure, there is provided a data reading method, the method including:

the file retrieval information database receives an access request submitted by a user through the front-end display module and retrieves according to a keyword in the access request;

and the front-end display module downloads a corresponding file from the data storage system according to the result retrieved by the file retrieval information database.

As for the above-mentioned aspect and any possible implementation manner, there is further provided an implementation manner, where the file retrieval information includes: file name, size, location, upload time, file MD5.

According to the method, data storage and reading are combined, so that data generated by multiple devices can be uploaded to a distributed storage space based on an HDFS (Hadoop distributed file system) through a data forwarding environment of a K8S-based cloud native platform, and retrieval information of the uploaded data is written into a file retrieval information database. When accessing or reading data, a user can acquire the required data through a front-end display page or a query page, and can also acquire the required data by calling an HDFS access data interface. High availability of data acquisition, summarization and storage and access is satisfied.

It should be understood that the statements herein reciting aspects are not intended to limit the critical or essential features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The above and other features, advantages and aspects of embodiments of the present disclosure will become more apparent upon consideration of the following detailed description, taken in conjunction with the accompanying drawings. The accompanying drawings are included to provide a further understanding of the present disclosure, and are not intended to limit the disclosure thereto, and the same or similar reference numerals will be used to indicate the same or similar elements, where:

FIG. 1 illustrates a block diagram of a data storage and reading system according to an embodiment of the present disclosure;

FIG. 2 illustrates a schematic structural diagram of a data storage and reading system according to an embodiment of the present disclosure;

FIG. 3 illustrates a flow chart of a data storage method of a data storage and reading system according to an embodiment of the present disclosure;

FIG. 4 illustrates a flow chart of a data reading method of a data storage and reading system according to an embodiment of the present disclosure;

FIG. 5 illustrates a block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure.

Detailed Description

To make the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions in the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without inventive step, are intended to be within the scope of the present disclosure.

In addition, the term "and/or" herein is only one kind of association relationship describing an associated object, and means that there may be three kinds of relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

According to the method and the device, data storage and a reading system are combined, so that data generated by a plurality of devices can be uploaded to a distributed storage space based on an HDFS (Hadoop distributed file system) through a data forwarding environment of a K8S-based cloud native platform, and retrieval information of the uploaded data is written into a file retrieval information database. When a user accesses or reads data, the user can acquire the required data through a front-end display page or a query page, and download the required file by calling an HDFS access data interface. High availability of data acquisition, summarization and storage and data access convenience are realized.

FIG. 1 shows a block diagram of a data storage and reading system according to an embodiment of the present disclosure.

As shown in fig. 1, the data storage and reading system 100 includes:

the data forwarding module 101 is configured to receive data uploaded by data production equipment and upload the received data to the data storage system;

the recording module 102 is configured to write the file retrieval information fed back by the data storage system into the file retrieval information database;

the data storage system 103 is used for storing the data uploaded by the data forwarding module and feeding back file retrieval information to the recording module; the system is also used for returning the appointed file according to the front-end requirement;

a file retrieval information database 104 for receiving and storing the file retrieval information written by the recording module; the front-end display module is also used for receiving an access request submitted by a user through the front-end display module;

a front-end display module 105, configured to submit an access request of a user to the file retrieval information database, and receive a corresponding retrieval result; and downloading the corresponding file from the data storage system according to the retrieval result.

The above is a description of an embodiment of the apparatus, and the following is a further description of the embodiments of the present disclosure by way of example of the method.

FIG. 2 illustrates a schematic structural diagram of a data storage and reading system according to an embodiment of the present disclosure.

As shown in fig. 2, the present disclosure provides a data storage and reading system.

Wherein, data storage and reading system includes: the system comprises a data production environment, a data forwarding server, a data forwarding module and a recording module which are deployed on a K8S POD, a database for recording characteristic information such as file paths, a distributed storage space based on an HDFS, and a front-end display module.

The data production environment comprises three data production devices, namely a Windows server, a Linux server and an Arm device; the three data production devices comprise different operating systems, a transmission protocol uses a safe SCP protocol, the Linux operating system running on the Linux server and the Arm device is provided with an SCP function, and the Windows server uses Putty to transmit and call the SCP.

The data forwarding server is a K8S-based cloud native platform, operates on a multi-host cluster, and transmits and summarizes data through a plurality of hosts;

the data forwarding module is deployed on a plurality of PODs (digital POD storage) on the K8S, and meanwhile, a receiving service and a data buffer area based on SSHD (secure Shell storage) are contained in a container of each POD and are used for receiving data and forwarding the data to the data storage system; the data storage system is a distributed storage space based on an HDFS (Hadoop distributed file system);

the recording module is deployed in a plurality of PODs on the K8S and used for writing the file retrieval information into the file retrieval information database; wherein, the file retrieval information database is a MySQL database;

the front-end display module comprises a display page and a query page, and a user accesses the file retrieval information database through the front-end query page; the display page is used for counting data conditions and checking detailed information;

the front-end display module can also download corresponding files from a data storage system by calling an HDFS data access interface;

in the present disclosure, the working method of the data storage and reading system includes: the data conversion module receives and converts data uploaded by the data production equipment; the data forwarding and recording module uploads the converted data to the data storage system, receives file retrieval information fed back by the data storage system and writes the file retrieval information into the file retrieval information database; the file retrieval information database receives and stores the file retrieval information written by the data forwarding and recording module;

the file retrieval information database receives an access request submitted by a user through the front-end display module and retrieves according to a keyword in the access request; and the front-end display module downloads a corresponding file from the data storage system according to the result retrieved by the file retrieval information database.

method 300 may be performed by data storage and reading system 100 in FIG. 1.

As shown in fig. 3, the working method 300 of the data storage and reading system includes:

s301, a data forwarding module receives data uploaded by data production equipment and uploads the received data to a data storage system;

s302, a recording module receives file retrieval information fed back by a data storage system and writes the file retrieval information into a file retrieval information database;

and S303, receiving and storing the file retrieval information written by the recording module by a file retrieval information database.

In step S301, the receiving, by the data forwarding module, data uploaded by the data production device and uploading the received data to the data storage system includes:

the one or more data forwarding modules respectively receive the data uploaded by the one or more data production devices through the SSH service and store the data in the temporary file buffer area.

The data forwarding module uploads the received data to the data storage system and comprises:

and the data forwarding module uploads the received data to the data storage system according to the timing task.

According to the embodiment of the disclosure, data transmission is realized by using a file transmission method-SCP command based on SSH. One or more data production devices push data to the K8S management platform through SCP remote file transmission. And the data forwarding module deployed on the POD of the K8S receives the data uploaded by one or more data production devices through the SSH service and stores the received data in the temporary file buffer.

The SCP commands use SSH to transmit data, and use the same authentication mode as SSH to provide the same security guarantee, the SSH is a protocol which is reliable at present and provides security for remote login session and other network services, and the information leakage problem in the remote management process can be effectively prevented by using the SSH protocol.

In some embodiments, other SSH file transfer commands are also used for file transfer, such as: rsync: popular commands for file synchronization; sshfs: mounting the remote directory through SSH;

in the data forwarding server based on the K8S cloud native platform, the data forwarding server runs on a cluster with multiple hosts, data are transmitted and summarized through the multiple hosts, and high availability of the data is achieved.

According to the embodiment of the disclosure, the characteristic that the data storage system adopts the HDFS architecture is that the HDFS architecture adopts a low-cost hardware design mode, has high fault tolerance, can be deployed on a common server with low price, and forms an HDFS cluster through a plurality of servers so as to provide the capability of distributed storage. HDFS provides high throughput data access and is well suited for large-scale dataset-based applications.

In some embodiments, with the HDFS high availability architecture, two namenodes (to manage the namespace of the file system) will be running simultaneously in the HDFS high availability architecture cluster, one as an Active Namenode and one as a Standby Namenode. The namespace of the Standby Namenode and the Active Namenode are synchronized in real time, so when the Active Namenode fails and stops serving, the Standby Namenode can be immediately switched to an Active state, and HDFS cluster service is not influenced.

Specifically, the problem of expandability of traditional centralized storage is effectively solved by distributed storage, the scale can be expanded to thousands of nodes, the capacity is expanded to hundreds of PB (lead-back) or even EB (Electron Beam) levels, and the performance is linearly improved along with the capacity. After on-line capacity expansion according to needs, data rebalancing is automatically realized. A plurality of storage nodes of distributed storage can provide read-write service at the same time, so that the throughput rate is high and can reach dozens of GB/s.

It can be understood that, in the design of the distributed architecture, there are multiple nodes in nature, and it is easy to implement calculation and storage redundancy backup by means of primary and standby (HA), redundancy, hash, and the like, thereby implementing high availability.

In step S302, the recording module receives the file retrieval information fed back by the data storage system and writes the file retrieval information into the file retrieval information database.

In some embodiments, the method further includes a specific data processing process, the data processing process is a core link for storing the operation of the cloud system, and the specific way to implement the process is as follows: all data information files in the storage cloud contain a prime index file, and the two files are connected closely and located in the same node. When the data is stored on the storage cloud system, the storage system feeds the index information of the file back to the corresponding recording module.

According to the embodiment of the disclosure, the data forwarding and recording module is deployed on the K8S, the data forwarding module is used for uploading data stored in the temporary file buffer area to the HDFS-based distributed storage system through a timing task, when the data is uploaded to the data storage system, the data storage system feeds file retrieval information stored in the data back to the recording module, and the recording module receives the file retrieval information fed back by the data storage system and writes the file retrieval information into the file retrieval information database. Therefore, the data storage is carried out by utilizing the distributed storage space, the high availability of the data storage is realized, and meanwhile, the data retrieval and access by users are facilitated.

In step S303, the file search information database receives and stores the file search information written by the recording module.

In the present disclosure, the file retrieval information includes: file name, size, location, upload time, file MD5, etc.

According to the embodiment of the disclosure, the MySQL database receives the file retrieval information written by the recording module and stores the file retrieval information into the corresponding database table, so that when a subsequent user accesses data, the data can be directly obtained by accessing the file recording table of the database.

method 400 may be performed by data storage and reading system 100 in FIG. 1.

As shown in fig. 4, the data reading method 400 includes:

s401: the file retrieval information database receives an access request submitted by a user through a front-end display module, and retrieves according to a keyword in the access request;

s402: and the front-end display module downloads a corresponding file from the data storage system according to the result retrieved by the file retrieval information database.

In step S401, the file search information database receives an access request submitted by a user through the front-end display module, and searches according to a keyword in the access request.

MySQL is a relational database management system in this disclosure that maintains data in different tables instead of putting all data in one large repository, which increases speed and flexibility.

According to the embodiment of the disclosure, the file retrieval information is written into the MySQL database, when a user accesses data, the file record table in the MySQL database can be accessed through inputting keywords, statistics of data conditions and detailed data information can be checked on a front-end query page, and data display is performed through a front-end display page. Therefore, the functions of data analysis, statistics and the like are better realized, and data display is carried out.

In step S402, the downloading, by the front-end display module, the corresponding file from the data storage system according to the result retrieved by the file retrieval information database includes:

and the front-end display module downloads a corresponding file from the data storage system by calling an HDFS data access interface.

It is known that HDFS clusters have two types of nodes and operate in supervisor and worker mode, namely one NameNode and multiple datanodes (a software node running on a separate machine in the HDFS instance). In the HDFS file downloading process, a client requests to a NameNode to download a file, the NameNode finds a DataNode address where a file block is located by inquiring metadata, and selects a DataNode (a near principle and then a random) server to request to read data. The DataNode starts to transmit data to the client (reading data from the disk and putting stream, and checking with packet as unit), the client receives the data with packet as unit, and then caches the data locally, and writes the data into the target file.

Specifically, the DataNode is slave, the NameNode issues a command, and the DataNode executes actual operation. The DataNode in the cluster is responsible for managing storage on the node where the DataNode is located, and is mainly responsible for storing actual data blocks and performing read/write operations of the data blocks.

According to the embodiment of the disclosure, a user submits an access request on a front-end page, the file retrieval information database receives a corresponding retrieval result, and the data file is downloaded by calling an HDFS data access interface.

It is noted that while for simplicity of explanation, the foregoing method embodiments have been described as a series of acts or combination of acts, it will be appreciated by those skilled in the art that the present disclosure is not limited by the order of acts, as some steps may, in accordance with the present disclosure, occur in other orders and concurrently. Further, those skilled in the art will appreciate that the embodiments described in the specification are exemplary embodiments and that acts and modules are not necessarily required for the disclosure.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the described module may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 5 illustrates a block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

The device 500 comprises a computing unit 501 which may perform various suitable actions and processes in accordance with a computer program stored in a Read Only Memory (ROM) 502 or a computer program loaded from a storage unit 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The calculation unit 501, the ROM 502, and the RAM 503 are connected to each other by a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.

A number of components in the device 500 are connected to the I/O interface 505, including: an input unit 506 such as a keyboard, a mouse, or the like; an output unit 507 such as various types of displays, speakers, and the like; a storage unit 508, such as a magnetic disk, optical disk, or the like; and a communication unit 509 such as a network card, modem, wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

The computing unit 501 may be a variety of general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The computing unit 501 performs the various methods and processes described above, such as the method 300 or the method 400. For example, in some embodiments, method 300 or method 400 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 500 via ROM 502 and/or communications unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the method 300 or the method 400 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the method 300 or the method 400 by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel or sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions of the present disclosure can be achieved.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A data storage and reading system, the system comprising:

the system comprises a data forwarding module, a recording module, a data storage system, a file retrieval information database and a front-end display module; wherein the content of the first and second substances,

2. The data storage and reading system of claim 1,

the data forwarding module and the recording module are deployed in a POD (POD) of a K8S-based cloud native platform;

the data storage system is a cluster-based HDFS storage system.

3. A method of operating a data storage and read system according to claim 1 or 2, comprising:

and the file retrieval information database receives and stores the file retrieval information written by the recording module.

4. The method of claim 3, wherein the receiving, by the data forwarding module, the data uploaded by the data production device comprises:

5. The method of claim 3, wherein uploading data to the data storage system by a data forwarding module comprises:

and the data forwarding module uploads the data to the data storage system according to the timing task.

6. The method of claim 3,

the file retrieval information includes: file name, size, location, upload time, file MD5.

7. The method of claim 3, further comprising:

8. An electronic device, comprising:

at least one processor;

and a memory communicatively coupled to the at least one processor;

wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 3-7.

9. A non-transitory computer readable storage medium storing computer instructions, wherein,

the computer instructions are for causing the computer to perform the method of any one of claims 3-7.