CN115729445A

CN115729445A - Data storage method, related device and equipment

Info

Publication number: CN115729445A
Application number: CN202111028492.1A
Authority: CN
Inventors: 曾加
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-09-02
Filing date: 2021-09-02
Publication date: 2023-03-03

Abstract

The embodiment of the application discloses a data storage method, a related device and equipment, which are used for avoiding resource waste of data storage. The method in the embodiment of the application comprises the following steps: the method comprises the steps of obtaining data to be transmitted, carrying out fragmentation processing on the data to be transmitted according to the type of target data to obtain K fragmentation data and an actual storage space corresponding to each fragmentation data, distributing a preset storage space for each fragmentation data according to the actual storage space corresponding to each fragmentation data, determining a storage address of each fragmentation data if a data label of each fragmentation data exists in a historical storage record, wherein the historical storage record comprises a mapping relation between the data label and the storage address in historical time, sending a resource creation request to a server according to the storage address of each fragmentation data to enable the server to create a storage resource matched with the preset storage space for each fragmentation data, and sending K fragmentation data to the server to enable the server to store the K fragmentation data.

Description

Data storage method, related device and equipment

Technical Field

The embodiment of the application relates to the technical field of data processing, in particular to a data storage method, a related device and equipment.

Background

With the rapid development of computers and internet technologies, the internet develops more and more rapidly, the amount of data generated every day is larger and larger, and people can process the data through a distributed storage cluster technology in order to avoid network breakdown. The distributed storage cluster is formed by constructing a network topology by a plurality of storage servers to form a storage server cluster, and provides data reading, writing and disk-dropping functions in the storage server cluster through related algorithms, wherein the distributed storage cluster is formed by sharing storage load by utilizing a plurality of storage servers.

The existing distributed storage cluster technology based data processing method is to fixedly allocate a storable disk space in advance, then use the allocated disk space on each machine in a storage server cluster through a network, and store the received data according to the fixedly allocated storable disk space, but because the size of the received data is uncontrollable, the situation that the disk space is unreasonably allocated is likely to occur, so that the resource waste of the disk space or the data cannot be smoothly stored.

Disclosure of Invention

The embodiment of the application provides a data storage method, a related device and equipment, which are used for dividing data to be transmitted with large data volume into K pieces of fragmented data with small data volume for decentralized transmission by performing fragmentation processing on the data to be transmitted so as to reduce data storage load, and enabling a server to create storage resources matched with the preset storage volume for each piece of fragmented data by distributing the preset storage volume for each piece of fragmented data so as to store the K pieces of fragmented data, thereby avoiding resource waste.

An embodiment of the present application provides a data storage method, including:

acquiring data to be transmitted, wherein the data to be transmitted carries a target data type, and the target data type is a time type or a file type;

according to the type of the target data, carrying out fragmentation processing on the data to be transmitted to obtain K fragmentation data and an actual storage amount corresponding to each fragmentation data, wherein each fragmentation data corresponds to one data label, and K is an integer greater than 1;

distributing a preset storage amount for each piece of fragment data according to the actual storage amount corresponding to each piece of fragment data, wherein the preset storage amount is greater than or equal to the corresponding actual storage amount;

if the data label of each piece of fragmented data exists in the history storage record, determining the storage address of each piece of fragmented data, wherein the history storage record comprises the mapping relation between the data label and the storage address in the history time;

according to the storage address of each piece of fragmented data, a resource creation request is sent to a server, so that the server creates a storage resource matched with a preset storage amount for each piece of fragmented data, wherein the resource creation request carries the preset storage amount of the piece of fragmented data;

and sending the K pieces of fragment data to the server so that the server stores the K pieces of fragment data.

Another aspect of the present application provides an apparatus for storing data, including:

the device comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring data to be transmitted, the data to be transmitted carries a target data type, and the target data type is a time type or a file type;

the processing unit is used for carrying out fragmentation processing on data to be transmitted according to the type of the target data to obtain K fragmentation data and an actual storage capacity corresponding to each fragmentation data, wherein each fragmentation data corresponds to one data tag, and K is an integer greater than 1;

the processing unit is further used for distributing a preset storage amount to each piece of fragment data according to the actual storage amount corresponding to each piece of fragment data, wherein the preset storage amount is greater than or equal to the corresponding actual storage amount;

the determining unit is used for determining the storage address of each piece of sliced data if the data label of each piece of sliced data exists in the historical storage record, wherein the historical storage record comprises the mapping relation between the data label and the storage address in the historical time;

the processing unit is further used for sending a resource creation request to the server according to the storage address of each piece of fragmented data, so that the server creates a storage resource matched with the preset storage amount for each piece of fragmented data, wherein the resource creation request carries the preset storage amount of the piece of fragmented data;

and the processing unit is further used for sending the K pieces of fragment data to the server so as to enable the server to store the K pieces of fragment data.

In one possible design, in one implementation of another aspect of an embodiment of the present application,

the processing unit is further used for sending an address allocation request to the server if the data label of the fragmented data does not exist in the history storage record, so that the server allocates a corresponding storage address for the fragmented data according to the address allocation request;

and the receiving unit is used for receiving an address allocation response sent by the server, wherein the address allocation response carries a storage address allocated for the fragment data.

In a possible design, in an implementation manner of another aspect of the embodiment of the present application, the processing unit may be specifically configured to:

determining a transmission identifier of each piece of sliced data according to the data label of each piece of sliced data, wherein the transmission identifier is used for indicating the transmission sequence of the piece of sliced data;

arranging the transmission identifiers of each piece of sliced data to obtain a data transmission sequence, wherein the data transmission sequence comprises K transmission identifiers which are arranged in a descending order or an ascending order;

and sequentially sending K pieces of fragment data to the server according to the data transmission sequence.

the receiving unit is further used for receiving K sections of storage progress sent by the server in sequence, wherein one section of fragmented data corresponds to one section of storage progress;

and the processing unit is also used for sequentially sending the K sections of storage progress to the client according to the data transmission sequence so as to enable the client to display the K sections of storage progress.

In a possible design, in an implementation manner of another aspect of the embodiment of the present application, the obtaining unit may be specifically configured to:

receiving a data uploading request sent by a client, wherein the data uploading request carries a target data type;

responding to the data uploading request, and acquiring data to be processed;

and performing data conversion on the data to be processed according to the type of the target data to obtain the data to be transmitted.

if the target data type is a time type, converting the data to be processed into the data to be transmitted in a time-data value format;

and if the target data type is the file type, converting the data to be processed into the data to be transmitted in the file name-file object format.

the receiving unit is further configured to receive a first reading request sent by the client, where the first reading request carries a data reading range;

the determining unit is further used for responding to the first reading request and determining T data tags according to the data reading range, wherein T is an integer which is greater than or equal to 1 and less than or equal to K;

the determining unit is further used for determining a storage address corresponding to each data tag in the T data tags according to the historical storage record;

the processing unit is further configured to send a second reading request to the server according to the storage address corresponding to each data tag, so that the server responds to the second reading request to obtain T fragmented data;

and the receiving unit is further used for receiving the T pieces of sliced data sent by the server.

Another aspect of the present application provides a computer device, including: a memory, a transceiver, a processor, and a bus system;

wherein, the memorizer is used for storing the procedure;

the processor, when executing the program in the memory, implements the methods of the aspects described above;

the bus system is used for connecting the memory and the processor so as to enable the memory and the processor to communicate.

Another aspect of the present application provides a computer-readable storage medium having stored therein instructions, which when executed on a computer, cause the computer to perform the method of the above-described aspects.

In another aspect of the application, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the network device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, so that the network device executes the method provided by the above aspects.

According to the technical scheme, the embodiment of the application has the following advantages:

the method comprises the steps of obtaining data to be transmitted, carrying out fragmentation processing on the data to be transmitted according to the type of target data carried by the data to be transmitted, obtaining K fragmentation data and actual storage space corresponding to each fragmentation data, distributing preset storage space for each fragmentation data according to the actual storage space corresponding to each fragmentation data, determining the storage address of each fragmentation data when the data label of each fragmentation data exists in a historical storage record, sending a resource creation request to a server according to the storage address of each fragmentation data, enabling the server to create storage resources matched with the preset storage space for each fragmentation data, and then sending K fragmentation data to the server, so that the server stores the K fragmentation data. Through the mode, the data to be transmitted with large data volume is divided into K pieces of fragment data with small data volume for scattered transmission by carrying out fragment processing on the data to be transmitted, so that the data storage load is reduced, and further, the preset storage volume can be distributed to each piece of fragment data, so that the server creates storage resources matched with the preset storage volume for each piece of fragment data, the K pieces of fragment data are stored, and the resource waste is avoided.

Drawings

FIG. 1 is a schematic diagram of an architecture of a data object control system in an embodiment of the present application;

FIG. 2 is a flow chart of one embodiment of a method for data storage in an embodiment of the present application;

FIG. 3 is a flow chart of another embodiment of a method for data storage in an embodiment of the present application;

FIG. 4 is a flow chart of another embodiment of a method for data storage in an embodiment of the present application;

FIG. 5 is a flow chart of another embodiment of a method for data storage in an embodiment of the present application;

FIG. 6 is a flow chart of another embodiment of a method for data storage in an embodiment of the present application;

FIG. 7 is a flow chart of another embodiment of a method for data storage in an embodiment of the present application;

FIG. 8 is a flow chart of another embodiment of a method of data storage in an embodiment of the present application;

FIG. 9 is a schematic flow chart diagram of a method of data storage in an embodiment of the present application;

fig. 10 (a) is a schematic diagram of a data uploading structure of the data storage method in the embodiment of the present application;

FIG. 10 (b) is a schematic diagram of a data acquisition structure of the data storage method in the embodiment of the present application;

FIG. 10 (c) is a schematic diagram of a message structure of the data storage method in the embodiment of the present application;

FIG. 11 is a message interaction diagram of a method of data storage in an embodiment of the present application;

FIG. 12 is a schematic diagram of an embodiment of a data storage device in the embodiment of the present application;

FIG. 13 is a schematic diagram of an embodiment of a computer device in the embodiment of the present application.

Detailed Description

The embodiment of the application provides a data storage method, a related device and equipment, which are used for dividing data to be transmitted with large data volume into K pieces of fragmented data with small data volume for decentralized transmission by fragmenting the data to be transmitted, so as to reduce data storage load, and enabling a server to establish storage resources matched with the preset storage volume for each piece of fragmented data by distributing the preset storage volume for each piece of fragmented data, so as to store the K pieces of fragmented data, and avoid resource waste.

The terms "first," "second," "third," "fourth," and the like in the description and in the claims and drawings of the present application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "corresponding" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It should be understood that the method for entity identification provided by the present application may be applied to a scenario in which data storage is completed through data fragmentation, for example, the storage of weather change data is completed through fragmentation of weather change data of a day. As another example, storage of stock float data is accomplished, for example, by slicing a month of stock float data. As another example, for example, the quarterly product sales volume report is fragmented to complete storage of quarterly product sales volume report data, in various scenarios described above, in order to complete data storage, a solution provided in the prior art is to fixedly allocate a storable disk space in advance, then use the allocated disk space on each machine in the storage server cluster through a network, and store the received data according to the fixedly allocated disk space, but because the size of the received data is uncontrollable, an unreasonable allocation of disk space is likely to occur, which results in resource waste of disk space or data storage failure.

In order to solve the above problem, the present application provides a data storage method, which is applied to a data object system shown in fig. 1, please refer to fig. 1, where fig. 1 is an architecture schematic diagram of a data object control system in an embodiment of the present application, and as shown in fig. 1, by acquiring data to be transmitted at a client, performing fragmentation processing on the data to be transmitted according to a target data type carried by the data to be transmitted, to obtain K pieces of fragmented data and an actual storage amount corresponding to each piece of fragmented data, allocating a preset storage amount to each piece of fragmented data according to the actual storage amount corresponding to each piece of fragmented data, determining a storage address of each piece of fragmented data when a data tag of each piece of fragmented data exists in a history storage record, and sending a resource creation request to a server according to the storage address of each piece of fragmented data, so that the server creates a storage resource matching the preset storage amount for each piece of fragmented data, and then sending K pieces of fragmented data to the server, so that the server stores K pieces of fragmented data. Through the mode, the data to be transmitted with large data volume is divided into K pieces of fragment data with small data volume for scattered transmission by carrying out fragment processing on the data to be transmitted, so that the data storage load is reduced, and further, the preset storage volume can be distributed to each piece of fragment data, so that the server creates storage resources matched with the preset storage volume for each piece of fragment data, the K pieces of fragment data are stored, and the resource waste is avoided.

It should be understood that fig. 1 only shows one kind of terminal device of the client, and in an actual scenario, a greater variety of terminal devices may participate in the data processing process, such as a Personal Computer (PC), and the specific number and variety depend on the actual scenario, and are not limited herein. In addition, although a server is shown in fig. 1, in an actual scenario, a plurality of servers may participate, and particularly in a scenario of multi-model training interaction, the number of servers depends on the actual scenario, which is not limited herein.

It should be noted that in this embodiment, the server may be an independent physical server, may also be a server cluster or a distributed system formed by a plurality of physical servers, and may also be a cloud server that provides basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and an artificial intelligence platform, and the like. The terminal device may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal device and the server may be directly or indirectly connected through a wired or wireless communication manner, and the terminal device and the server may be connected to form a block chain network, which is not limited herein.

In order to solve the above problems, the present application proposes a method of data storage, which is generally performed by a server or a terminal device, and accordingly, an apparatus applied to data storage is generally provided in the server or the terminal device.

It is understood that the method, related device and apparatus for data storage as disclosed herein, wherein a plurality of server/terminal devices can be combined into a blockchain, and the server/terminal devices are nodes on the blockchain. In practical application, data sharing between nodes may be required in a block chain, and each node may store time data, file data, and the like.

Referring to fig. 2, a method for storing data in the present application will be described, where an embodiment of the method for storing data in the present application includes:

in step S101, data to be transmitted is obtained, where the data to be transmitted carries a target data type, and the target data type is a time type or a file type;

in this embodiment, when a user wants to upload data to a server for storage, the user may perform data upload operation through an application interface of a client to generate a data upload request, and may upload data to be transmitted to an Actor model of the server for processing, so that the data to be transmitted with a large data volume may be subsequently fragmented and fragmented, and the data to be transmitted with a large data volume may be divided into K fragmented data with a small data volume for decentralized transmission, thereby reducing a data storage load to a certain extent.

Wherein the Actor model is a most basic computational element, which can be used to receive a message and perform computations based thereon, and is stateful, can control itself or can be controlled by the message sender.

Specifically, as shown in fig. 9, the user may perform a data uploading operation through the application interface of the client, and generate a data uploading request according to the message structure shown in fig. 10 (c), for example, if the user wants to upload weather change data of one week, the data uploading request including weather change data with an event name, a current schedule of the data uploading request being null, and one week of data transmission may be generated.

The message structure may be transmitted between the client and the server through a Socket (Socket), as shown in fig. 11, a user may perform a data upload operation through an application interface of the client to generate a data upload request, and send the data upload request to the server according to the StartSocket shown in fig. 11.

Socket is a Socket which uses TCP to provide a communication mechanism between two computers, that is, a client program creates a Socket and tries to connect to the Socket of a server, and when the connection is established, the server creates a Socket object, so that the client and the server can communicate by writing and reading the Socket object.

Further, the data upload request may be uploaded to the top-layer Actor model shown in fig. 9 through a network layer, that is, a network protocol, so that the top-layer Actor model may respond to the data upload request and acquire the to-be-transmitted data of the time type or the file type carried in the data upload request.

In step S102, according to the type of the target data, performing fragmentation processing on the data to be transmitted to obtain K pieces of fragmented data and an actual storage amount corresponding to each piece of fragmented data, where each piece of fragmented data corresponds to one data tag, and K is an integer greater than 1;

in this embodiment, after obtaining the data to be transmitted, the server may transmit the obtained data to be transmitted to a Cluster Sharding (Akka Cluster sharing) module through the Actor model to perform Sharding processing, so as to obtain K pieces of sharded data and an actual storage amount corresponding to each piece of sharded data, assign a data tag to each piece of sharded data, and divide the data to be transmitted with a large data size into K pieces of sharded data with a small data size to perform distributed transmission, thereby reducing a data storage load to a certain extent.

The fragmentation processing is a mechanism that data is dispersed to nodes, that is, the fragmentation is to disperse data to different nodes as uniformly as possible to achieve the purpose of balancing.

The fragmentation data is based on a mechanism that data are scattered on nodes, and the acquired data to be transmitted are divided into K data with large data volume and small data volume according to a certain uniform dividing mode.

The actual storage amount is a storage space resource amount actually required for storing each piece of fragmented data, and the actual storage amount may be similar to the actual data size of each piece of fragmented data, that is, the actual storage amount may be slightly larger than or equal to the actual data size of each piece of fragmented data, so that the server may create a storage space resource matching the actual storage amount to store the piece of fragmented data according to the actual storage amount corresponding to each piece of fragmented data.

The Sharding is a fragmentation addressing mapping framework realized aiming at the Actor model, is not traditional for fragmenting data, and can be understood as a fragmentation of pairwise computation.

The principle of fragmentation processing of data to be transmitted by Actor sharing is roughly as follows:

1. regarding each piece of fragment data as an entity (entity), creating an Actor, such as Actor1 to Actor n shown in fig. 9, where each Actor has an Actor address as a unique identifier, and all messages of each piece of fragment data are processed by the actors corresponding to the same;

2. when the Actor does not work for a period of time, the Actor will timeout and automatically destroy (kill self);

3. when a new node, i.e., new fragment data, is added to a storage service cluster, a new Actor will be automatically created on the new node (node), or an old Actor will be load balanced and will migrate to the new node, and similarly, when a node is suspended, a suspended Actor will migrate to the new node.

For convenience of understanding, the explanation of step S102 is provided with reference to fig. 9, in the embodiment of the present application, an Actor shading module is used to perform fragmentation processing on data to be transmitted, and data to be transmitted is weather change data of a week, when data to be transmitted of weather change data of a week is obtained, a server may transmit the obtained weather change data of a week to the Actor shading module through the Actor model to perform fragmentation processing, and may divide the weather change data of a week into pieces of data respectively corresponding to monday, such as monday weather data, tuesday weather data, and the like, and assign a data tag to each piece of data, such as monday weather or monday weather, and meanwhile, according to a data amount of each piece of data, an actual storage amount corresponding to each piece of data may be calculated through the Actor shading module, and an actual storage amount of the weather data of monday is calculated to be 48k, where the weather data storage amount of the meteorological data of monday is 48k.

In step S103, allocating a preset storage amount to each fragmented data according to the actual storage amount corresponding to each fragmented data, wherein the preset storage amount is greater than or equal to the corresponding actual storage amount;

in this embodiment, after the actual storage amount corresponding to each piece of fragment data is obtained, the preset storage amount corresponding to each piece of fragment data may be calculated according to the actual storage amount corresponding to each piece of fragment data and through the Actor sharing module.

The preset storage amount corresponding to each fragment data is a storage space resource amount which can be allocated by the server and used for storing the fragment data, and in order to enable the fragment data to be smoothly stored in the server, the preset storage amount is larger than a corresponding actual storage amount generally, and can also be equal to the actual storage amount.

For convenience of understanding, step S103 is explained with reference to fig. 9, in the embodiment of the present application, the piece data is monday meteorological data, and the actual storage volume of the meteorological data of which the actual storage volume corresponding to each piece data is monday meteorological data is 48k, for example, when the actual storage volume of the acquired monday meteorological data is 48k, the server may estimate, by using an Actor Sharding module, the storage space resource amount required for the acquired actual storage volume of the meteorological data of monday meteorological data to be 48k, and if the upper limit of the preset storage volume that is greater than the corresponding actual storage volume is +10k, the preset storage volume corresponding to each piece data may be calculated by using the Actor Sharding module according to the actual storage volume of the meteorological data of monday meteorological data being 48k, and if the actual storage volume of the meteorological data of monday meteorological data is 48k, the preset storage volume of the meteorological data of monday meteorological data that is 58k is calculated, smooth storage of the meteorological data of monday meteorological data to the server can be realized, and the storage space resource amount allocated by the server is prevented from being improperly allocated, thereby avoiding resource waste to a certain extent.

In step S104, if the data tag of each piece of fragmented data exists in the history storage record, determining the storage address of each piece of fragmented data, where the history storage record includes the mapping relationship between the data tag and the storage address in the history time;

in this embodiment, after obtaining each piece of fragment data and the data tag of each piece of fragment data, the Actor shading module may search, according to the data tag of each piece of fragment data, whether a data tag in a historical time consistent with the data tag of the piece of fragment data exists in the historical storage record, if the data tag exists, it may be understood that the Actor shading module finds that a data feature consistent with the piece of fragment data exists in the historical data, a storage address may be provided for the piece of fragment data according to a mapping relationship between the data tag and the storage address in the historical time, so that the Actor shading module may allocate each piece of fragment data to a corresponding Actor according to the storage address of each piece of fragment data, and transmit each piece of fragment data to the server through the Actor, so that the server creates a storage resource matching a preset storage amount for each piece of fragment data, to store K pieces of fragment data, and avoid resource waste.

The storage address is an Actor address, and is used for accurately allocating the fragment data to a corresponding node Actor through the Actor address for data transmission or processing.

For convenience of understanding, in the embodiment of the present application, the piece data is monday weather data, and the data tag of monday weather data is monday weather, for example, when the acquired monday weather data and the data tag of monday weather data are monday weather, the server may search, in the history storage record, through the Actor Sharding module, whether a historical time data tag that is consistent with the data tag of monday weather data exists in the history storage record, and when monday weather exists in the historical time data tag and is consistent with the data tag of monday weather data, a storage address that has a mapping relationship with the monday weather of the data tag in the history time may be used as a storage address of monday weather data, so that the Actor may subsequently allocate the monday weather data to a corresponding Actor, such as Actor1, according to the storage address of monday weather data, and transmit each piece data to the server through Actor1, so that the server creates a storage resource that matches a preset storage space for each piece data, so as to store the monday weather data, thereby avoiding waste of monday resources to some extent.

In step S105, according to the storage address of each piece of fragmented data, sending a resource creation request to the server, so that the server creates a storage resource matching a preset storage amount for each piece of fragmented data, where the resource creation request carries the preset storage amount of the piece of fragmented data;

in this embodiment, after the storage address of each piece of fragmented data is obtained, a resource creation request carrying a preset storage amount of the piece of fragmented data may be sent to the server, so that the server creates a storage resource matching the preset storage amount for each piece of fragmented data, and can allocate a storage resource adapted to each piece of fragmented data to accurately store each piece of fragmented data, thereby avoiding resource waste to a certain extent.

Specifically, as shown in fig. 9, when a storage address of each piece of fragment data is obtained, a resource creation request carrying a preset storage amount of the piece of fragment data may reach a bottom-layer data server through an internal gateway, after the data server receives the resource creation request, the server may check a current available storage resource, create a storage resource matching the preset storage amount, and generate a corresponding Actor of the piece of fragment data, where the Actor is configured to receive the piece of fragment data and transmit the piece of fragment data to the storage resource created by the server for storage, and when the server creates the storage resource matching the preset storage amount and generates the corresponding Actor, it may be understood that the server is already in a state of ready to receive the piece of fragment data, so that a ready state may be notified to an Actor Sharding module in the form of a message structure, and notify the Actor Sharding module of the piece of fragment data to be stored, it may be understood that, if the server checks that the current available storage resource is insufficient after the data server receives the resource creation request, a developer may notify the Actor of expanding the resource, and notify the Actor of the waiting state to notify the Actor module of insufficient fragment data.

For convenience of understanding, as shown in fig. 10 (c) and fig. 11, in the embodiment of the present application, the fragmented data is meteorological data of monday, the preset storage amount of the fragmented data is 58k, and the storage address of the meteorological data of monday is Actor1, when the storage address of the acquired meteorological data of monday is Actor1, as in the message structure illustrated in fig. 10 (c), a resource creation request carrying the preset storage amount of the fragmented data may be in the form of a message structure, and as shown in fig. 11, a start socket is used to send the resource creation request to the server, when the resource creation request carrying the preset storage amount of the fragmented data is received by the server, a 58k storage resource matching the preset storage amount may be created, and an Actor1 corresponding to the storage address of the fragmented data is generated at the same time, and then, the ready state may be notified to the Actor in the memory module by the Actor socket illustrated in fig. 10 (c), and the Actor socket may notify the Sharding module of the ready state to issue the storage data.

The StartSocket socket is a message structure body initiated by the client, and can be used for sending a data uploading request or a data acquiring request, and also can be used for establishing long connection between the client and the data server. In addition, the slave data server also returns a StartSocket result message structure body for informing the client that the client is ready for transmission or has no data.

The middlebocket socket is that the data server can return data to the client through a middlebocket message structure according to the progress of loading or storing the data by the server, and the middlebocket socket can contain fragmented data, information of the current progress and the total progress, and the like.

In step S106, K pieces of shard data are sent to the server, so that the server stores the K pieces of shard data.

In this embodiment, after receiving the notification of the ready state sent by the data server, the Actor sharing module may issue the K pieces of fragmented data to the Actor corresponding to the storage address through the storage address corresponding to each piece of fragmented data, so that the Actor may transmit the pieces of fragmented data to the storage resource created by the server to store the pieces of fragmented data, and may accurately and smoothly store each piece of fragmented data, thereby avoiding waste of resources to a certain extent.

For convenience of understanding, as shown in fig. 10 (c) and fig. 11, in the embodiment of the present application, the piece of meteorological data is used as monday meteorological data, the preset storage amount of the piece of meteorological data is 58K, and the storage address of the monday meteorological data is Actor1, when receiving a notification of a ready state sent by the data server, the Actor sharing module may send the monday meteorological data to the Actor1 corresponding to the storage address by using the storage address of the monday meteorological data as Actor1, and then the Actor1 may transmit the monday meteorological data to a 58K storage resource created by the server for storage, and when the data server completes storage of K pieces of sliced data, a message structure shown in fig. 10 (c) may be used, the data storage completion request may be in the form of a message structure, and as shown in fig. 11, the data storage completion request is sent to the client by using a finhsocket socket.

The finshsocket socket is used for sending a Finishsocket message structure to the client after the data server processes all data, wherein the Finishsocket message structure can comprise the current progress and the total progress, and the client disconnects the data server after receiving the Finishsocket message structure.

It is understood that the data to be transmitted may be time type data or file type data, but some special file data, such as important encrypted file data, is only stored in one server, and the time sequence data may be stored in different data servers according to the time characteristics of the data.

It should be noted that, when the storage server cluster is in an idle state, the storage server cluster may use full-node backup or (node number/2 + 1) to backup the data stored on the node according to the backup policy of the storage server cluster, and different data servers may communicate with each other through the actor to backup the data and return the data to the backup state of both communication parties, so that when any data server fails or is abnormal and the stored data cannot be used, the backed-up data may be obtained from other data servers that communicate with the failed data server to backup the data through the actor to continue to be used, so as to maintain normal use of the data.

Optionally, on the basis of the embodiment corresponding to fig. 2, in another optional embodiment of the method for storing data provided in the embodiment of the present application, as shown in fig. 3, the method further includes:

in step S301, if the data tag of the fragmented data does not exist in the history storage record, sending an address allocation request to the server, so that the server allocates a corresponding storage address for the fragmented data according to the address allocation request;

in step S302, an address allocation response sent by the server is received, where the address allocation response carries a storage address allocated for the fragmented data.

In this embodiment, after obtaining each piece of fragment data and the data tag of each piece of fragment data, the Actor sharing module may search, according to the data tag of each piece of fragment data, whether a data tag in a history time that is consistent with the data tag of the piece of fragment data exists in the history storage record, if the data tag does not exist, it may be understood that the Actor sharing module does not find that a data feature that is consistent with the piece of fragment data exists in the history data, and the piece of fragment data is a new piece of data that has not been processed before, an address allocation request is sent to the server, so that the server allocates a corresponding storage address to the piece of fragment data according to the address allocation request, and then an address allocation response that is sent by the server and carries the storage address allocated to the piece of fragment data is received, so that the subsequent Actor sharing module may allocate each piece of fragment data to a corresponding Actor according to the storage address of each piece of fragment data, and transmit each piece of fragment data to the server through the Actor, so that the server creates a storage resource that matches a preset storage amount for each piece of fragment data, so as to store K pieces of fragment data, and avoid resource waste.

For convenience of understanding, as shown in fig. 10 (c) and fig. 11, in the embodiment of the present application, it is described that segmented data is meteorological data on a day of the week, and a data tag of the meteorological data on the day of the week is sunday meteorological data, when the meteorological data on the day of the week and the data tag of the meteorological data on the monday are acquired as sunday meteorological data, the server may search, by using an Actor Sharding module, in a history storage record, whether a historical time data tag that is consistent with the data tag of the meteorological data on the day of the week exists in the server, and when no sunday meteorological data is found in the historical time data tag, an address allocation request may be sent to the server through a start socket as shown in fig. 11, so that the server allocates a corresponding storage address to the segmented data according to the address allocation request, such as Actor7, and then, an address allocation response carrying a storage address of the Actor shard data allocated to the segmented data sent by the receiving server may be sent to the Actor shard data through a socket as shown in fig. 11, so that the effector may create a storage resource allocation response to the meteorological data that the meteorological data on the weedy data may be transmitted to the Actor shard data by the Actor Sharding module, and the Actor Sharding module, so that the meteorological data may be transmitted to the meteorological data on the week.

Optionally, on the basis of the embodiment corresponding to fig. 2, in another optional embodiment of the method for storing data provided in the embodiment of the present application, as shown in fig. 4, sending K pieces of sharded data to a server includes:

in step S401, determining a transmission identifier of each fragmented data according to a data tag of each fragmented data, where the transmission identifier is used to indicate a transmission order of the fragmented data;

in step S402, arranging the transmission identifiers of each piece of sliced data to obtain a data transmission sequence, where the data transmission sequence includes K transmission identifiers arranged in a descending order or an ascending order;

in step S403, the K pieces of fragmented data are sequentially sent to the server according to the data transmission sequence.

In this embodiment, after the data tag of each piece of fragment data is obtained, the transmission identifier carried in each data tag may be obtained as the transmission identifier of each piece of fragment data according to the data tag of each piece of fragment data, and then the obtained transmission identifiers may be arranged, so that a data transmission sequence including K transmission identifiers arranged in a descending order or an ascending order may be obtained, and then, the data transmission sequence may sequentially pass through the storage addresses corresponding to the K piece of fragment data according to the data transmission sequence, and the K piece of fragment data is issued to an Actor corresponding to the storage addresses, so that the Actor may transmit the K piece of fragment data to a storage resource created by a server for storage, and may accurately and smoothly store the K piece of fragment data, thereby avoiding resource waste to a certain extent.

The transmission Identifier (ID) is used to indicate a transmission order of the fragment data, and may be embodied as an integer (int) -type numeric string, or may be embodied as a character string, or the like.

Specifically, when the data tag of each piece of sliced data is obtained, it may be determined whether the data tag of each piece of sliced data carries a transmission identifier, if yes, the transmission identifier of each piece of sliced data may be determined, for example, the data tag of monday meteorological data is monday meteorological data, which carries a transmission identifier "01", the data tag of tuesday meteorological data is tuesday meteorological data, which carries a transmission identifier "02", and the data tag of wednesday meteorological data is wednesday meteorological data, which carries a transmission identifier "03", and similarly, the data tag of sunday meteorological data is sunday meteorological data, which carries a transmission identifier "07", so that the transmission identifier of monday meteorological data to sunday meteorological data may be determined, and then, the transmission identifiers may be arranged in an ascending order or a descending order according to actual application requirements, for example, the transmission identifiers of the monday meteorological data to sunday meteorological data may be arranged in an ascending order, which may obtain "01"; 02;03;04;05, carrying out a reaction; 06;07", then, according to the transmission identifiers arranged in a descending order of 7 pieces in the data transmission sequence, firstly, the meteorological data of monday is issued to the Actor1 corresponding to the storage address through the storage address Actor1 corresponding to the transmission identifier" 01", so that the Actor1 can transmit the meteorological data of monday to the storage resource created by the server for storage, when the server completes the storage of the meteorological data of monday, a notification that the meteorological data of monday is completely stored can be sent to the Actor sharing module, then, the Actor sharing module issues the meteorological data of monday in a waiting state to the meteorological data of monday, according to the transmission identifiers arranged in a descending order of 7 pieces in the data transmission sequence, the meteorological data of monday is issued to the Actor2 corresponding to the storage address through the storage address Actor2 corresponding to the transmission identifier" 02", so that the Actor2 can transmit the meteorological data of monday to the storage resource created by the server for storage, and similarly, 5 pieces of the meteorological data of the data of monday under the transmission identifiers can be successively transmitted to the server, thereby avoiding the waste of the meteorological data in the transmission resources.

Further, if there is no transmission identifier in the data tag of each piece of fragmented data, the piece of fragmented data may be randomly sent to the server for storage, for example, when the Actor sharing module receives the notification of the ready state sent by the server, the piece of fragmented data may be sent to the corresponding server for storage according to the sequence of the received notification of the ready state sent by the server, and in actual application, the sending manner of sending K pieces of fragmented data to the server may be set according to the actual application requirements, and is not particularly limited.

Optionally, on the basis of the embodiment corresponding to fig. 2, in another optional embodiment of the method for storing data provided in the embodiment of the present application, as shown in fig. 5, the method further includes:

in step S501, K segments of storage schedules sequentially sent by the server are received, where one segment of fragmented data corresponds to one segment of storage schedule;

in step S502, the K segments of storage progress are sequentially sent to the client according to the data transmission sequence, so that the client displays the K segments of storage progress.

In this embodiment, after the server completes the storage of the K classified data in sequence, K segments of storage schedules sequentially sent by the server may be received, wherein each segment of storage schedule is used to indicate the current storage schedule of each classified data, then, the Actor sharing module may send the K segments of storage schedules to the client in sequence according to the data transmission sequence, so that the client displays the K segments of storage schedules, the storage state of each classified data can be visually displayed on the application interface with the client through the K segments of storage schedules, and a user can conveniently and accurately perceive which classified data completes the storage in real time, and perceive the current storage schedule of the classified data.

Specifically, when the server stores the K classified data in sequence, each time one classified data is stored, the current storage progress and the total storage progress of the classified data may be sent to the client through the middlesecket socket in the form of a message structure as illustrated in fig. 10 (c), so as to display the current storage progress and the total storage progress of the classified data on the display interface of the client, and similarly, the current storage progress and the total storage progress of the other K-1 classified data may be sent to the client in sequence, so as to display the current storage progress and the total storage progress of the K classified data on the display interface of the client.

For convenience of understanding, as shown in fig. 10 (c) and fig. 11, the embodiment of the present application is described by taking, as an example, a weather data from a piece of K pieces of sliced data to a day of the week, when the storage of the weather data of the day of the week is completed, the current storage progress of the weather data of the day of the week is "14.3%" or "1/7" and the total storage progress is "14.3%" or "1/7", the message structure shown in fig. 10 (c) is converted into a form of a message structure, the event name "weather data of the day of the week", the current progress of the weather data "14.3%", the total progress of the weather data "14.3%", and the transmission data is empty, and the current storage progress of the weather data of the day of the week is "14.3%" and the total storage progress of the weather data "14.3%" can be transmitted to the client through the middlemocke socket, in order to display the Monday meteorological data on the display interface of the client, the current storage progress of the data is 14.3% and the total storage progress is 14.3%, similarly, the current storage progress and the total storage progress of other 6 meteorological data can be sequentially sent to the client, so that the current storage progress of the Monday meteorological data is 14.3% and the total storage progress is 14.3%, the current storage progress of the Tuesday meteorological data is 14.3% and the total storage progress is 28.6% and the like are sequentially displayed on the display interface of the client, until the current storage progress of the Monday meteorological data is 14.3% and the total storage progress is 100%.

Optionally, on the basis of the embodiment corresponding to fig. 2, in another optional embodiment of the method for storing data provided in the embodiment of the present application, as shown in fig. 6, the obtaining data to be transmitted includes:

in step S601, a data upload request sent by a client is received, where the data upload request carries a target data type;

in step S602, the data to be processed is acquired in response to the data upload request;

in step S603, data conversion is performed on the data to be processed according to the type of the target data, so as to obtain data to be transmitted.

In this embodiment, when a user wants to upload data to a server for storage, the user may perform data upload operation through an application interface of a client to generate a data upload request, when an Actor model of the server receives a data upload request sent by the client, it may be determined first whether data to be processed carried in the data upload request is data supported by the Actor model, if so, the Actor model responds to the data upload request and obtains the data to be processed, and then the data to be processed may be converted into data to be transmitted according to a target data type, so that the data to be transmitted with a large data size may be subsequently fragmented, and the data to be transmitted with a large data size may be divided into K fragmented data with a small data size for decentralized transmission, thereby reducing a data storage load to a certain extent.

Specifically, as shown in fig. 9, a user may perform a data upload operation through an application interface of a client, generate a data upload request according to a message structure body shown in fig. 10 (c), and before the data upload request is transmitted through a network layer, the network layer may authenticate the data upload request, where the authentication is to determine whether the user or data to be processed carried in the data upload request has a right to access a server, specifically may be authentication techniques such as a password or a pass, or other authentication techniques, and is not limited herein.

Further, after the data upload request is authenticated by the network layer, the data upload request may be uploaded to the top layer Actor model shown in fig. 9, so that the top layer Actor model may respond to the data upload request, acquire the to-be-processed data of the time type or the file type carried in the data upload request, and then convert the to-be-processed data into to-be-transmitted data according to the data upload structure shown in fig. 10 (a).

Optionally, on the basis of the embodiment corresponding to fig. 2, in another optional embodiment of the method for storing data provided in the embodiment of the present application, as shown in fig. 7, performing data conversion on data to be processed according to a type of target data to obtain data to be transmitted includes:

in step S701, if the target data type is a time type, converting the data to be processed into data to be transmitted in a time-data value format;

in step S702, if the target data type is a file type, the data to be processed is converted into data to be transmitted in a file name-file object format.

In this embodiment, after the data to be processed is acquired, the type of the data uploading structure can be determined according to the target data type of the data to be processed to perform data conversion on the data to be processed, so that the Actor model can better and faster identify or respond to the data to be transmitted, and the Actor shading module can better and faster process the data to be transmitted.

Specifically, as shown in fig. 10 (a), when the data to be processed is acquired, if the target data type of the data to be processed is the time type, the data to be processed is converted into the data to be transmitted of the time data sequence-data value sequence according to the time data structure illustrated in fig. 10 (a) and according to the time-data value format; if the target data type of the data to be processed is a file type, converting the data to be processed into data to be transmitted of a file-file object according to a file data structure as illustrated in fig. 10 (a) and a file name-file object format.

For convenience of understanding, as shown in fig. 10 (a), in the embodiment of the present application, the case where the meteorological data from the meteorological data to the sunday whose data to be processed is monday is taken as an example is described, and when the meteorological data from the meteorological data to the sunday is acquired, the meteorological data from the meteorological data to the sunday of monday may be the data to be transmitted of the time data series-data value series according to the time data structure shown in fig. 10 (a) and according to the time-data value format, for example, "monday" - "where the highest air temperature is 32 degrees, the lowest air temperature is 21 degrees, and the average air temperature is 27 degrees.

Optionally, on the basis of the embodiment corresponding to fig. 2, in another optional embodiment of the method for storing data provided in the embodiment of the present application, as shown in fig. 8, the method further includes:

in step S801, a first read request sent by a client is received, where the first read request carries a data read range;

in step S802, in response to the first read request, T data tags are determined according to a data read range, where T is an integer greater than or equal to 1 and less than or equal to K;

in step S803, a storage address corresponding to each of the T data tags is determined according to the history storage record;

in step S804, a second reading request is sent to the server according to the storage address corresponding to each data tag, so that the server responds to the second reading request to obtain T fragmented data;

in step S805, T pieces of shard data transmitted by the server are received.

In this embodiment, when a user wants to obtain stored data from a server, the user may perform a data reading operation through an application interface of the client to generate a first reading request, and when an Actor model of the server receives the first reading request carrying a data reading range sent by the client, it may first determine whether the user has an authority to access the server, and if so, the Actor model responds to the first reading request and determines T data tags according to the data reading range, and then may determine a storage address corresponding to each data tag in the T data tags according to a mapping relationship between the data tags and the storage addresses recorded in a historical storage, and further may send a second reading request to a corresponding server according to the storage address corresponding to each data tag, so that the server responds to the second reading request to obtain T fragmented data, receives the T fragmented data sent by the server through a corresponding Actor, and then sends the T fragmented data to the client.

Specifically, as shown in fig. 9, a user may perform a data obtaining operation through an application interface of a client, generate a first read request according to a message structure shown in fig. 10 (c), before the first read request is transmitted through a network layer, the network layer may authenticate a data uploading request, after the first read request is authenticated through the network layer, upload the first read request to a top-level Actor model shown in fig. 9, so that the top-level Actor model may respond to the first read request and obtain a data read range carried in the first request, then transmit the data read range to an Actor Sharding module according to a data obtaining structure shown in fig. 10 (b), for example, weather data with a data read range from monday to monday, convert the data read range into a data read range in a format of a data start time-data end time according to a time data structure shown in fig. 10 (b), and convert the data read range into a data read range in a format of a file name-file identification type unique to file format according to a file data structure shown in fig. 10 (b) if the data read range is a sales report of a product.

Furthermore, the Actor shading module can perform fragmentation processing on the data reading range, then assign T data tags according to the fragmentation result, further find the data tags consistent with the T data tags in the historical storage record of the server, and determine the storage address corresponding to each data tag in the T data tags according to the mapping relation between the data tags and the storage addresses.

Further, a second read request may be sent to a corresponding server according to a storage address corresponding to each data tag, so that the server can obtain T pieces of fragment data from the storage resource respectively in response to the second read request, receive T pieces of fragment data sent by the server through an Actor corresponding to the storage address of each data tag, and then send the T pieces of fragment data to the client.

For convenience of understanding, as shown in fig. 10 (b) and fig. 10 (c), in the embodiment of the present application, taking the meteorological data from the meteorological data with T pieces of fragmented data as monday to sunday as an example for explanation, when acquiring the first read request sent by the client, the top-level Actor model may respond to the first read request and acquire the data read range carried in the first request, and then convert the data read range into the "data start time" in the format of data start time-data end time according to the time data structure illustrated in fig. 10 (b): monday-data end time: sunday ", and transmitted to the Actor sharing module so that the Actor sharing module can compare the" data start time: monday-data end time: the sunday is subjected to fragmentation processing, so that 7 data tags, such as monday weather or sunday weather, can be obtained, and then the data tags consistent with the 7 data tags can be found in the historical storage record of the server, and then the storage address corresponding to each data tag in the 7 data tags is determined according to the mapping relationship between the data tags and the storage addresses, where the storage address corresponding to the monday weather is Actor1, for example.

Further, a second reading request in the form of a message structure as illustrated in fig. 10 (c) may be sent to the server according to the storage address corresponding to each data tag, for example, according to the fact that the storage address corresponding to the monday weather is Actor1, so that the server can respectively obtain 7 pieces of fragmented data from the storage resources in response to the second reading request, for example, the monday weather data sent by the server is received by the Actor1 corresponding to the storage address of the monday weather, and then the monday weather data is sent to the client.

Further, when the server responds to the second read request, the server may sequentially acquire T fragment data from the storage resource according to the sequence of the transmission identifiers carried by the T data tags, and sequentially transmit and load the current read progress and the total read progress of the acquired fragment data to the client by using the message structure body as illustrated in fig. 10 (c), so that the read fragment data, the current read progress and the total read progress of the fragment data are displayed on the display interface of the client; the fragmented data may also be acquired from the storage resource according to the sequence in which the server responds to the second read request, and in actual application, the acquisition mode of the fragmented data may be set according to the actual application requirements, and is not particularly limited.

Referring to fig. 12, fig. 12 is a schematic view of an embodiment of a data storage device in an embodiment of the present application, and the data storage device 20 includes:

an obtaining unit 201, configured to obtain data to be transmitted, where the data to be transmitted carries a target data type, and the target data type is a time type or a file type;

a processing unit 202, configured to perform fragmentation processing on data to be transmitted according to a target data type to obtain K pieces of fragmentation data and an actual storage amount corresponding to each piece of fragmentation data, where each piece of fragmentation data corresponds to one data tag, and K is an integer greater than 1;

the processing unit 202 is further configured to allocate a preset storage amount to each piece of fragmented data according to the actual storage amount corresponding to each piece of fragmented data, where the preset storage amount is greater than or equal to the corresponding actual storage amount;

a determining unit 203, configured to determine a storage address of each piece of sliced data if a data tag of each piece of sliced data exists in a history storage record, where the history storage record includes a mapping relationship between the data tag and the storage address in a history time;

the processing unit 202 is further configured to send a resource creation request to the server according to the storage address of each piece of fragmented data, so that the server creates a storage resource matching a preset storage amount for each piece of fragmented data, where the resource creation request carries the preset storage amount of the piece of fragmented data;

the processing unit 202 is further configured to send the K pieces of sliced data to the server, so that the server stores the K pieces of sliced data.

Alternatively, on the basis of the embodiment corresponding to fig. 12, in another embodiment of the data storage device provided in the embodiment of the present application,

the processing unit 202 is further configured to send an address allocation request to the server if the data tag of the fragmented data does not exist in the history storage record, so that the server allocates a corresponding storage address for the fragmented data according to the address allocation request;

a receiving unit 204, configured to receive an address allocation response sent by the server, where the address allocation response carries a storage address allocated for the fragmented data.

Optionally, on the basis of the embodiment corresponding to fig. 12, in another embodiment of the data storage device provided in the embodiment of the present application, the processing unit 202 may specifically be configured to:

the receiving unit 204 is further configured to receive K segments of storage schedules sequentially sent by the server, where one segment of fragmented data corresponds to one segment of storage schedule;

the processing unit 202 is further configured to sequentially send the K segments of storage progress to the client according to the data transmission sequence, so that the client displays the K segments of storage progress.

Optionally, on the basis of the embodiment corresponding to fig. 12, in another embodiment of the apparatus for storing data provided in the embodiment of the present application, the obtaining unit 201 may specifically be configured to:

responding to the data uploading request, and acquiring data to be processed;

Optionally, on the basis of the embodiment corresponding to fig. 12, in another embodiment of the data storage device provided in the embodiment of the present application, the obtaining unit 201 may specifically be configured to:

Optionally, on the basis of the embodiment corresponding to fig. 12, in another embodiment of the data storage device provided in the embodiment of the present application, the receiving unit is further configured to receive a first read request sent by the client, where the first read request carries a data read range;

the determining unit 203 is further configured to determine T data tags according to the data reading range in response to the first read request, where T is an integer greater than or equal to 1 and less than or equal to K;

the determining unit 203 is further configured to determine, according to the history storage record, a storage address corresponding to each data tag in the T data tags;

the processing unit 202 is further configured to send a second reading request to the server according to the storage address corresponding to each data tag, so that the server responds to the second reading request to obtain T pieces of fragmented data;

the receiving unit 204 is further configured to receive T pieces of sliced data sent by the server.

Another exemplary computer device is provided, as shown in fig. 13, fig. 13 is a schematic structural diagram of a computer device provided in this embodiment, and the computer device 300 may have a relatively large difference due to different configurations or performances, and may include one or more Central Processing Units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) storing an application 331 or data 332. Memory 320 and storage media 330 may be, among other things, transient storage or persistent storage. The program stored on the storage medium 330 may include one or more modules (not shown), each of which may include a sequence of instructions operating on the computer device 300. Still further, the central processor 310 may be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the computer device 300.

The computer device 300 may also include one or more power supplies 340, one or moreAn upper wired or wireless network interface 350, one or more input-output interfaces 360, and/or one or more operating systems 333, such as a Windows Server ^TM ，Mac OS X ^TM ，Unix ^TM ,Linux ^TM ，FreeBSD ^TM And so on.

The computer device 300 described above is also used to perform the steps in the embodiments corresponding to fig. 2 to 8.

Another aspect of the present application provides a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to perform the steps in the method as described in the embodiments shown in fig. 2 to 8.

Another aspect of the application provides a computer program product comprising instructions which, when run on a computer or processor, cause the computer or processor to perform the steps of the method as described in the embodiments shown in fig. 2 to 8.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, which are essential or part of the technical solutions contributing to the prior art, or all or part of the technical solutions, may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a portable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.

Claims

1. A method of data storage, comprising:

according to the target data type, carrying out fragmentation processing on the data to be transmitted to obtain K fragmentation data and an actual storage amount corresponding to each fragmentation data, wherein each fragmentation data corresponds to a data tag, and K is an integer greater than 1;

distributing a preset storage amount to each fragment data according to the actual storage amount corresponding to each fragment data, wherein the preset storage amount is greater than or equal to the corresponding actual storage amount;

if the data label of each fragment data exists in a historical storage record, determining the storage address of each fragment data, wherein the historical storage record comprises the mapping relation between the data label and the storage address in the historical time;

according to the storage address of each piece of fragmented data, a resource creation request is sent to a server, so that the server creates a storage resource matched with a preset storage amount for each piece of fragmented data, wherein the resource creation request carries the preset storage amount of the fragmented data;

2. The method of claim 1, further comprising:

if the data label of the fragment data does not exist in the history storage record, sending an address allocation request to the server so that the server allocates a corresponding storage address for the fragment data according to the address allocation request;

and receiving an address allocation response sent by the server, wherein the address allocation response carries a storage address allocated for the fragmented data.

3. The method according to claim 1, wherein the sending the K pieces of sharded data to the server comprises:

determining a transmission identifier of each piece of sliced data according to the data tag of each piece of sliced data, wherein the transmission identifier is used for indicating the transmission order of the piece of sliced data;

and sequentially sending the K pieces of fragment data to the server according to the data transmission sequence.

4. The method of claim 3, further comprising:

receiving K sections of storage progress sent by the server in sequence, wherein one section of fragment data corresponds to one section of storage progress;

and sequentially sending the K segments of storage progress to a client according to the data transmission sequence so that the client displays the K segments of storage progress.

5. The method of claim 1, wherein the obtaining data to be transmitted comprises:

receiving a data uploading request sent by a client, wherein the data uploading request carries the target data type;

responding to the data uploading request, and acquiring data to be processed;

6. The method according to claim 5, wherein the performing data conversion on the data to be processed according to the target data type to obtain the data to be transmitted includes:

if the target data type is the time type, converting the data to be processed into the data to be transmitted in a time-data value format;

and if the target data type is the file type, converting the data to be processed into the data to be transmitted in a file name-file object format.

7. The method according to claim 1, wherein after the sending the K pieces of sharded data to the server, the method further comprises:

receiving a first reading request sent by a client, wherein the first reading request carries a data reading range;

responding to the first reading request, and determining T data tags according to the data reading range, wherein T is an integer which is greater than or equal to 1 and less than or equal to K;

determining a storage address corresponding to each data tag in the T data tags according to the historical storage record;

sending a second reading request to the server according to the storage address corresponding to each data tag, so that the server responds to the second reading request to obtain T fragmented data;

and receiving the T pieces of fragment data sent by the server.

8. An apparatus for data storage, comprising:

the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring data to be transmitted, the data to be transmitted carries a target data type, and the target data type is a time type or a file type;

the processing unit is used for carrying out fragmentation processing on the data to be transmitted according to the target data type to obtain K fragmentation data and an actual storage amount corresponding to each fragmentation data, wherein each fragmentation data corresponds to one data label, and K is an integer larger than 1;

the processing unit is further configured to allocate a preset storage amount to each piece of fragmented data according to an actual storage amount corresponding to each piece of fragmented data, where the preset storage amount is greater than or equal to the corresponding actual storage amount;

the determining unit is used for determining the storage address of each piece of fragmented data if the data tag of each piece of fragmented data exists in a historical storage record, wherein the historical storage record comprises the mapping relation between the data tag and the storage address in the historical time;

the processing unit is further configured to send a resource creation request to a server according to the storage address of each piece of fragmented data, so that the server creates a storage resource matching a preset storage amount for each piece of fragmented data, where the resource creation request carries the preset storage amount of the piece of fragmented data;

the processing unit is further configured to send the K pieces of sliced data to the server, so that the server stores the K pieces of sliced data.

9. A computer device, comprising: a memory, a transceiver, a processor, and a bus system;

wherein the memory is used for storing programs;

the processor when executing the program in the memory implementing the method of any one of claims 1 to 7;

10. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 7.