CN110543279B - Data storage and processing method, device and system - Google Patents
Data storage and processing method, device and system Download PDFInfo
- Publication number
- CN110543279B CN110543279B CN201810531026.7A CN201810531026A CN110543279B CN 110543279 B CN110543279 B CN 110543279B CN 201810531026 A CN201810531026 A CN 201810531026A CN 110543279 B CN110543279 B CN 110543279B
- Authority
- CN
- China
- Prior art keywords
- data
- stored
- computing device
- processing chip
- preset threshold
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0605—Improving or facilitating administration, e.g. storage management by facilitating the interaction with a user or administrator
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the application provides a data storage and processing method, device and system, in the scheme, if data to be stored is larger than the capacity of a processing chip, the data to be stored is divided into hot data and cold data, the hot data is respectively stored in a plurality of processing chips, the cold data is respectively stored in storage spaces of a plurality of computing devices, that is, the hot data and the cold data are respectively stored in a plurality of parts, so that after a comparison request is received, the comparison request can be processed in parallel by using the hot data and the cold data in the plurality of parts.
Description
Technical Field
The present disclosure relates to the field of intelligent analysis technologies, and in particular, to a data storage method, a data processing method, a data storage device, a data processing device, and a data storage system.
Background
The intelligent analysis system generally comprises a scheduling device and a computing device, wherein the scheduling device can receive the comparison request and specify the computing device to process the comparison request, and the specified computing device compares the data stored by itself with the data to be compared contained in the comparison request and feeds back the comparison result to the scheduling device.
For example, a plurality of face images may be stored in the computing device, the scheduling device receives the face image to be identified and transmits the face image to be identified to the selected computing device, and the selected computing device compares the face image stored by itself with the face image to be identified and feeds back the comparison result to the scheduling device.
Generally, an intelligent analysis system includes a plurality of computing devices, each of which includes one or more processing chips (such as GPU: graphics Processing Unit, graphics processor), and a scheduling device respectively designates different processing chips for processing a plurality of received comparison requests, so that parallel processing of the comparison requests is realized.
The storage scheme based on intelligent analysis systems is typically to store the same data into each processing chip to support parallel processing of the compare requests. Taking the above face image as an example, assuming that the face image needs to be compared with a face database of one city, the face database of one city needs to be stored in each GPU. While the storage resources in a GPU are limited, if the storage scheme is used, when more data needs to be stored, parallel processing of the comparison request cannot be realized.
Disclosure of Invention
An objective of the embodiments of the present application is to provide a method, an apparatus, and a system for storing and processing data, so as to solve the above technical problems.
In order to achieve the above objective, an embodiment of the present application provides a data storage method, which is applied to a scheduling device in an intelligent analysis system, where the system further includes a plurality of computing devices, each computing device includes a storage space and at least one processing chip; the method comprises the following steps:
Acquiring data to be stored; judging whether the data volume of the data to be stored is larger than a first preset threshold value or not, wherein the first preset threshold value is set according to the capacity of a processing chip in the system;
if the data to be stored is larger than the first preset threshold, dividing the data to be stored into hot data and cold data, wherein the data quantity of the hot data is not larger than the first preset threshold;
the hot data is stored to a plurality of processing chips in the system, respectively, and the cold data is stored to storage spaces of a plurality of computing devices, respectively.
Optionally, in the case that the data amount of the data to be stored is determined to be greater than a first preset threshold, the method may further include:
calculating a difference value between the data quantity of the data to be stored and the first preset threshold value;
judging whether the difference value is larger than the first preset threshold value or not;
if the difference value is larger than the first preset threshold value, executing the step of dividing the data to be stored into hot data and cold data;
if the difference value is not greater than the first preset threshold value, dividing the data to be stored into two parts of data, wherein the two parts of data comprise a first part of data and a second part of data, and the data amount of the first part of data is equal to the first preset threshold value;
And respectively storing the first data and the second data to different processing chips.
Optionally, the storing the first portion of data and the second portion of data in different processing chips may include:
determining a first number of processing chips storing the first data and a second number of processing chips storing the second data according to the ratio of the data amount of the first data to the data amount of the second data;
storing the first data to the first number of processing chips, respectively;
and storing the second data to the second number of processing chips respectively.
Optionally, the processing chip is a GPU, and the storage space is a memory.
In order to achieve the above objective, an embodiment of the present application further provides a data processing method, which is applied to a computing device in an intelligent analysis system, where the computing device includes a storage space and at least one processing chip; the method comprises the following steps:
receiving data to be processed;
judging whether target data compared with the data to be processed is divided into hot data and cold data or not;
if so, comparing the data to be processed with hot data stored in a processing chip of the computing device and cold data stored in a storage space of the computing device.
Optionally, the comparing the data to be processed with hot data stored in a processing chip of the computing device and cold data stored in a storage space of the computing device may include:
comparing the data to be processed with thermal data stored in a processing chip of the computing device;
moving the hot data to a storage space of the computing device, and loading cold data stored in the storage space of the computing device to a processing chip of the computing device;
and comparing the data to be processed with cold data loaded to the processing chip.
Optionally, the loading the cold data stored in the storage space of the computing device to the processing chip of the computing device may include:
judging whether the data volume of the cold data stored in the storage space is larger than a second preset threshold value or not, wherein the second preset threshold value is set according to the capacity of the processing chip;
if not, loading cold data stored in a memory space of the computing device to a processing chip of the computing device;
if the data is larger than the second preset threshold, dividing the cold data stored in the storage space of the computing device into a plurality of pieces of data to be switched, determining each piece of data to be switched as current switching data in turn, and loading the current switching data to the processing chip after the data in the processing chip is moved to the storage space.
In order to achieve the above objective, an embodiment of the present application further provides a data storage device, which is applied to a scheduling device in an intelligent analysis system, where the system further includes a plurality of computing devices, each computing device includes a storage space and at least one processing chip; the device comprises:
the acquisition module is used for acquiring data to be stored;
the first judging module is used for judging whether the data volume of the data to be stored is larger than a first preset threshold value or not, and the first preset threshold value is set according to the capacity of a processing chip in the system; if the number is greater than the threshold value, triggering a dividing module;
the first dividing module is used for dividing the data to be stored into hot data and cold data, and the data volume of the hot data is not more than the first preset threshold value;
and the first storage module is used for respectively storing the hot data to a plurality of processing chips in the system and respectively storing the cold data to storage spaces of a plurality of computing devices.
Optionally, the apparatus may further include:
the calculating module is used for calculating the difference value between the data volume of the data to be stored and the first preset threshold value under the condition that the data volume of the data to be stored is judged to be larger than the first preset threshold value;
The second judging module is used for judging whether the difference value is larger than the first preset threshold value or not; if the difference value is larger than the first preset threshold value, triggering the first dividing module; if the difference value is not greater than the first preset threshold value, triggering a second dividing module;
the second dividing module is used for dividing the data to be stored into two parts of data, the two parts of data comprise a first part of data and a second part of data, and the data amount of the first part of data is equal to the first preset threshold value;
the second storage module is used for respectively storing the first data and the second data to different processing chips.
Optionally, the second storage module may be specifically configured to:
determining a first number of processing chips storing the first data and a second number of processing chips storing the second data according to the ratio of the data amount of the first data to the data amount of the second data;
storing the first data to the first number of processing chips, respectively;
and storing the second data to the second number of processing chips respectively.
Optionally, the processing chip is a GPU, and the storage space is a memory.
In order to achieve the above object, an embodiment of the present application further provides a data processing apparatus, which is applied to a computing device in an intelligent analysis system, where the computing device includes a storage space and at least one processing chip; the device comprises:
the receiving module is used for receiving the data to be processed;
a third judging module for judging whether the target data compared with the data to be processed has been divided into hot data and cold data; if yes, triggering a comparison module;
and the comparison module is used for comparing the data to be processed with hot data stored in a processing chip of the computing device and cold data stored in a storage space of the computing device.
Optionally, the comparing module may include:
a first comparison sub-module for comparing the data to be processed with thermal data stored in a processing chip of the computing device;
a movement submodule to move the thermal data to a memory space of the computing device;
a loading sub-module for loading cold data stored in a memory space of the computing device to a processing chip of the computing device;
and the second comparison sub-module is used for comparing the data to be processed with the cold data loaded to the processing chip.
Optionally, the loading submodule may be specifically configured to:
judging whether the data volume of the cold data stored in the storage space is larger than a second preset threshold value or not, wherein the second preset threshold value is set according to the capacity of the processing chip;
if not, loading cold data stored in a memory space of the computing device to a processing chip of the computing device;
if the data is larger than the second preset threshold, dividing the cold data stored in the storage space of the computing device into a plurality of pieces of data to be switched, determining each piece of data to be switched as current switching data in turn, and loading the current switching data to the processing chip after the data in the processing chip is moved to the storage space.
In order to achieve the above object, an embodiment of the present application further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing any data storage and processing method when executing the program stored in the memory.
In order to achieve the above object, an embodiment of the present application further provides an intelligent analysis system, where the system includes a scheduling device and a plurality of computing devices, each computing device including a storage space and at least one processing chip;
the scheduling device is used for acquiring data to be stored, judging whether the data volume of the data to be stored is larger than a first preset threshold value or not, wherein the first preset threshold value is set according to the capacity of each processing chip in the system; if the data to be stored is not greater than the first storage instruction, sending a first storage instruction to each computing device, wherein the first storage instruction carries the data to be stored; if the data to be stored is larger than the first storage instruction, dividing the data to be stored into hot data and cold data, and sending a second storage instruction to each computing device, wherein the second storage instruction carries the hot data and the cold data;
the computing device is used for storing the data to be stored to each processing chip of the computing device after receiving the first storage instruction; after receiving the second storage instruction, storing the hot data to each processing chip of the computing device and storing the cold data to a storage space of the computing device.
Optionally, the scheduling device is further configured to obtain data to be processed; determining metadata corresponding to target data to be compared with the data to be processed; judging whether the target data is divided into hot data and cold data according to the metadata; if yes, sending a first processing instruction to the computing equipment and the data to be processed; if not, sending a second processing instruction and the data to be processed to the computing equipment;
the computing device is further configured to compare the data to be processed with hot data stored in a processing chip of the computing device and cold data stored in a storage space of the computing device after receiving the first processing instruction and the data to be processed; and after the second processing instruction and the data to be processed are received, comparing the data to be processed with target data stored in a processing chip of the computing device.
By applying the embodiment of the application, if the data to be stored is larger than the capacity of the processing chip, the data to be stored is divided into hot data and cold data, the hot data is respectively stored in a plurality of processing chips, the cold data is respectively stored in storage spaces of a plurality of computing devices, that is, the hot data and the cold data are respectively stored in a plurality of parts, so that after receiving the comparison request, the comparison request can be processed in parallel by using the plurality of parts of hot data and the cold data.
Of course, not all of the above-described advantages need be achieved simultaneously in practicing any one of the products or methods of the present application.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic structural diagram of an intelligent analysis system according to an embodiment of the present application;
fig. 2 is a flow chart of a data storage method applied to a scheduling device according to an embodiment of the present application;
FIG. 3 is a flowchart of a data processing method applied to a computing device according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data storage device applied to a scheduling apparatus according to an embodiment of the present application;
FIG. 5 is a schematic structural diagram of a data processing apparatus applied to a computing device according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
In order to solve the technical problems, embodiments of the present application provide a data storage method and apparatus applied to a scheduling device, a data processing method and apparatus applied to a computing device, an electronic device, and an intelligent analysis system.
The intelligent analysis system provided in the embodiment of the application may include a scheduling device and a plurality of computing devices, where each computing device includes a storage space and at least one processing chip, as shown in fig. 1. The intelligent analysis system may comprise a plurality of scheduling devices, and fig. 1 is only for illustration and is not limited to the specific architecture of the intelligent analysis system. A data storage method applied to a scheduling device provided in the embodiment of the present application is first described in detail below.
Fig. 2 is a flow chart of a data storage method applied to a scheduling device according to an embodiment of the present application, including:
s201: and acquiring data to be stored.
For example, a face database may be stored in a computing device of the intelligent analysis system, and the data to be stored may be the face database.
S202: judging whether the data volume of the data to be stored is larger than a first preset threshold value or not, wherein the first preset threshold value is set according to the capacity of a processing chip in the system. If not, S203 is performed, and if so, S204 is performed.
For example, the processing chip may be a GPU, and the capacity of each GPU in the system may be the same or different. If the first preset threshold value is the same, the first preset threshold value can be equal to the capacity of the GPU or slightly smaller than the capacity of the GPU; if the first preset threshold is different, the first preset threshold can be equal to the minimum capacity of the GPU in the system or slightly smaller than the minimum capacity of the GPU in the system.
S203: storing the data to be stored to a plurality of processing chips in the system.
Continuing the above example, if the data size of the face database is smaller than the capacity of the GPU, the face database is stored in the GPU entirely, so that when there is a face image to be recognized, the face image can be compared with the face database in the GPU, and the GPU processing speed is fast and the comparison efficiency is high.
As an embodiment, S203 may include: the data to be stored are stored to each processing chip in the system respectively. Continuing the above example, each GPU in the system stores a face database so that each GPU can process the compare requests in parallel after receiving the compare requests.
As another implementation manner, computing devices in the intelligent analysis system may be divided into different task groups, the same data may be stored in computing devices belonging to the same task group, and one scheduling device may manage one or more task groups. For example, the intelligent analysis system comprises three task groups, wherein the computing equipment in the first task group stores a face database in an area A, the computing equipment in the second task group stores a face database in an area B, and the computing equipment in the third task group stores a face database in an area C; in this case, if the face database of the a region is acquired in S201, the data to be stored is stored in S203 to each processing chip in the first task group, respectively.
In this way, each computing device belonging to the same task group stores hot data and cold data, that is, each computing device belonging to the same task group stores a complete face database, so that after receiving a comparison request, each computing device of one task group can process the comparison request in parallel.
S204: dividing the data to be stored into hot data and cold data, wherein the data volume of the hot data is not more than the first preset threshold value.
Continuing the above example, the data to be stored is a face database, and the face database may be divided according to age information corresponding to the face data. For example, face data of 0-50 years old may be divided into hot data, and face data of other ages may be divided into cold data. Alternatively, the face database may be partitioned according to other attributes of the face data, and the partitioning manners are various and are not listed one by one.
Or, before executing the embodiment, the data to be stored may be divided by using multiple dividing modes, and under each dividing mode, the computing device records the dividing mode with higher processing speed for the comparison request. In this way, the data to be stored can be separated into hot data and cold data by selecting the recorded division manner in S204.
As an embodiment, the data amount of the thermal data may be equal to or slightly smaller than the first preset threshold. That is, if the capacity of each GPU in the system is the same, the amount of hot data may be equal to or slightly less than the capacity of the GPU; if the capacity of each GPU in the system is different, the data size of the hot data can be equal to or slightly smaller than the minimum capacity of the GPUs in the system.
S205: the hot data is stored to a plurality of processing chips in the system, respectively, and the cold data is stored to storage spaces of a plurality of computing devices, respectively.
As one implementation, the hot data may be stored separately to each processing chip in the system and the cold data may be stored separately to the storage space of each computing device. Each computing device in the system stores hot data and cold data, continuing the above example, that is, each computing device stores a complete face database, so that after receiving the comparison request, each computing device can process the comparison request in parallel.
As another implementation, computing devices in the intelligent analysis system may be divided into different task groups, with the same data stored in computing devices belonging to the same task group. For example, the intelligent analysis system comprises three task groups, wherein the computing equipment in the first task group stores a face database in an area A, the computing equipment in the second task group stores a face database in an area B, and the computing equipment in the third task group stores a face database in an area C; in this case, if the face database of the region a is acquired in S201, the hot data is stored in each processing chip in the first task group in S205, and the cold data is stored in the storage space of each computing device in the first task group in S205.
Each computing device belonging to the same task group stores hot data and cold data, that is, each computing device belonging to the same task group stores a complete face database, so that after receiving a comparison request, each computing device of one task group can process the comparison request in parallel.
In the following, an example is described in which one intelligent analysis system stores the same data, or in which one intelligent analysis system includes only one task group.
For example, assuming that the capacity of the GPU in the system is 4G, the first preset threshold is 4G, and the data size of the data to be stored is 6G, the data to be stored may be divided into two parts, one part is hot data, the data size of the hot data is 4G, the other part is cold data, and the data size of the cold data is 2G.
Assume that 20 computing devices are included in the system, where 10 computing devices include one GPU and one share of memory, and another 10 computing devices include two GPUs and one share of memory, that is, a total of 30 GPUs and 20 shares of memory are included in the system. The hot data of 4G are respectively stored in the 30 GPUs, and the cold data of 2G are respectively stored in the 20 storage spaces. That is, the hot data of 4G is stored in each of the 30 GPUs in the system, and the cold data of 2G is stored in each of the 20 storage spaces in the system.
If the processing chip of the computing device is a GPU, the storage space of the computing device may be a memory; if the processing chip of the computing device is a CPU (Central Processing Unit ), the storage space of the computing device may be a disk. If multiple disks are included in a computing device, the multiple disks may be provided as a single piece of storage space. In the following, a processing chip will be described as an example of a GPU.
As an embodiment, in the case that it is determined in S202 that the data amount of the data to be stored is greater than the first preset threshold, a difference between the data amount of the data to be stored and the first preset threshold may be calculated first; judging whether the difference value is larger than the first preset threshold value or not; if the difference is greater than the first preset threshold, the step of dividing the data to be stored into hot data and cold data is executed, and the data amount of the hot data is equal to the first preset threshold; if the difference value is not greater than the first preset threshold value, dividing the data to be stored into two parts of data, wherein the two parts of data comprise a first part of data and a second part of data, and the data amount of the first data is equal to the first preset threshold value; and respectively storing the first data and the second data to different processing chips.
For example, assuming that the first preset threshold is 4G and the data size of the data to be stored is 6G, calculating that the difference between the data size of the data to be stored and the first preset threshold is 2G, where 2G is smaller than the first preset threshold; in this case, S204 may not be performed, but the data to be stored may be divided into two pieces of data including a first piece of data of which the data amount is 4G and a second piece of data of which the data amount is 2G. The first and second data are stored to different processing chips, respectively. Assuming that the system includes 30 GPUs, the first data may be stored to 15 GPUs of the 30 GPUs, respectively, and the second data may be stored to the remaining 15 GPUs, respectively.
Alternatively, a first number of processing chips storing the first portion of data and a second number of processing chips storing the second portion of data may be determined according to a ratio of the data amount of the first portion of data to the data amount of the second portion of data; storing the first data to the first number of processing chips; and storing the second data to the second number of processing chips.
Continuing the above example, the data amount of the first data is 4G, the data amount of the second data is 2G, and the ratio of the two data amounts is 2:1, the ratio of the first number to the second number may be 2:1, i.e. a first number of 20 and a second number of 10, respectively stores the first data to 20 GPUs of the 30 GPUs and the second data to the remaining 10 GPUs.
As another example, assuming that the first preset threshold is 4G and the data amount of the data to be stored is 9G, calculating that the difference between the data amount of the data to be stored and the first preset threshold is 5G, and that 5G is greater than the first preset threshold; in this case, S204 may be performed: the data to be stored is divided into hot data having a data amount of 4G and cold data having a data amount of 5G. And respectively storing the 4G hot data into each GPU of each computing device, and respectively storing the 5G cold data into the storage space of each computing device.
Continuing the example above, assuming that 30 GPUs are included in the system, 20 memory locations, each of the 30 GPUs stores 4G hot data and each of the 20 memory locations stores 5G cold data.
By applying the embodiment shown in fig. 2 of the present application, if the data to be stored is greater than the capacity of the processing chip, the data to be stored is divided into hot data and cold data, the hot data is respectively stored in a plurality of processing chips, the cold data is respectively stored in storage spaces of a plurality of computing devices, that is, the hot data and the cold data are both stored in a plurality of portions, so that after receiving the comparison request, the comparison request can be processed in parallel by using the plurality of portions of hot data and cold data.
Fig. 3 is a flowchart of a data processing method applied to a computing device according to an embodiment of the present application, including:
s301: and receiving data to be processed.
For example, a face database is stored in a computing device in the intelligent analysis system, and the data to be processed may be face images to be compared. For example, the scheduling device may receive a comparison request sent by the user device, obtain data to be processed from the comparison request, and send the data to be processed to the computing device.
S302: it is judged whether or not the target data to be compared with the data to be processed has been divided into hot data and cold data, and if so, S303 is executed, and if not, S304 is executed.
For convenience of description, data to be compared with the data to be processed is referred to as target data. Continuing the above example, the data to be processed is a face image, and the target data is a face database.
For example, metadata corresponding to the face database may be stored in the scheduling device, where the metadata may include whether the face database is divided into hot data and cold data, or may include other, which is not limited specifically. The scheduling device may send the metadata to the computing device so that the computing device may determine whether the target data compared to the data to be processed has been separated into hot data and cold data.
Alternatively, when the computing device stores the face database, the computing device may generate metadata for the face database, so that the computing device may determine whether the target data compared with the data to be processed has been separated into hot data and cold data from the metadata stored by itself.
Alternatively, the computing device may determine whether the target data compared to the data to be processed has been separated into hot data and cold data by other means, such as: if the computing device receives the first type of processing instructions sent by the scheduling device, the target data is divided into hot data and cold data, and if the computing device receives the second type of processing instructions sent by the scheduling device.
The judging modes are various and are not listed one by one.
S303: the data to be processed is compared with hot data stored in a processing chip of the computing device and cold data stored in a memory space of the computing device.
S304: the data to be processed is compared with target data stored in a processing chip of the computing device.
In the present embodiment, when storing target data, if the target data is not divided into hot data and cold data, the target data is stored into each processing chip of the computing device, respectively; if the target data is split into hot data and cold data, the hot data is stored separately in each processing chip and the cold data is stored in the memory space of the computing device.
For example, if the processing chip of the computing device is a GPU, the storage space of the computing device may be memory; if the processing chip of the computing device is a CPU (Central Processing Unit ), the storage space of the computing device may be a disk. If multiple disks are included in a computing device, the multiple disks may be provided as a single piece of storage space. In the following, a processing chip will be described as an example of a GPU.
Therefore, when the judgment result of S302 is yes, the data to be processed is compared with the hot data in each GPU and the cold data in the storage space, and when the judgment result of S302 is no, the data to be processed is compared with the target data in each GPU.
As an embodiment, S303 may include: comparing the data to be processed with thermal data stored in a processing chip of the computing device; moving the hot data to a storage space of the computing device, and loading cold data stored in the storage space of the computing device to a processing chip of the computing device; and comparing the data to be processed with cold data loaded to the processing chip.
For example, assume that the computing device includes a GPU having 4G hot data stored therein and 2G cold data stored in a memory space of the computing device; and comparing the data to be processed with the 4G hot data in the GPU, moving the 4G hot data in the GPU to a storage space of the computing device after the comparison is completed, loading the 2G cold data in the storage space into the GPU, and comparing the data to be processed with the 2G cold data loaded into the GPU.
Therefore, in this embodiment, the comparison process is performed in the processing chip, so that the comparison efficiency can be improved.
As an embodiment, loading cold data stored in a memory space of the computing device to a processing chip of the computing device may include:
judging whether the data volume of the cold data stored in the storage space is larger than a second preset threshold value or not, wherein the second preset threshold value is set according to the capacity of the processing chip;
if not, loading cold data stored in a memory space of the computing device to a processing chip of the computing device;
if the data is larger than the second preset threshold, dividing the cold data stored in the storage space of the computing device into a plurality of pieces of data to be switched, determining each piece of data to be switched as current switching data in turn, and loading the current switching data to the processing chip after the data in the processing chip is moved to the storage space.
The second preset threshold may be equal to the capacity of the GPU or may be slightly smaller than the capacity of the GPU, assuming that the second preset threshold is 4GB. Assuming that the computing device includes a GPU, where 4G hot data is stored, and 6G cold data is stored in the storage space of the computing device, that is, the amount of cold data is greater than a second preset threshold, in this case:
Comparing the data to be processed with the 4G thermal data in the GPU, and moving the 4G thermal data in the GPU to a storage space of the computing equipment after the comparison is completed; dividing 6G cold data in the storage space into two parts of data to be switched, for example, wherein the first part of data to be switched is 4GB, the second part of data to be switched is 2GB, determining the first part of data to be switched as current switching data, loading the current switching data into the GPU, comparing the data to be processed with the first part of data to be switched loaded into the GPU, and after the comparison is completed, moving the 4G first part of data to be switched in the GPU to the storage space of the computing equipment; and determining the second data to be switched as current switching data, loading the current switching data into the GPU, comparing the data to be processed with the second data to be switched loaded into the GPU, and after the comparison, moving the second data to be switched of the 2G in the GPU to the storage space of the computing equipment. Finally, the 4GB of hot data may be reloaded into the GPU.
Alternatively, in the above example, the 6GB of cold data may be divided into two 3GB of data to be switched, where the data size of the data to be switched is not greater than the second preset threshold, and the specific data size is not limited.
By applying the embodiment shown in fig. 3 of the present application, hot data is stored in a processing chip of a computing device, cold data is stored in a storage space of the computing device, after the computing device receives data to be processed, the data to be processed is compared with the hot data stored in the processing chip and the cold data stored in the storage space, that is, each computing device can compare the hot data and the cold data stored in itself with the data to be processed, so that multiple computing devices can implement parallel processing of the data to be processed.
Corresponding to the method embodiment, the embodiment of the application also provides a data storage device which is applied to the scheduling equipment in the intelligent analysis system, wherein the system also comprises a plurality of computing equipment, and each computing equipment comprises a storage space and at least one processing chip; the apparatus may, as shown in fig. 4, include:
an acquisition module 401, configured to acquire data to be stored;
a first judging module 402, configured to judge whether a data amount of the data to be stored is greater than a first preset threshold, where the first preset threshold is set according to a capacity of a processing chip in the system; if the number is greater than the threshold value, triggering a dividing module;
A first dividing module 403, configured to divide the data to be stored into hot data and cold data, where a data amount of the hot data is not greater than the first preset threshold;
a first storage module 404, configured to store the hot data to a plurality of processing chips in the system, and store the cold data to storage spaces of a plurality of computing devices, respectively.
As an embodiment, the apparatus may further include: a calculation module, a second judgment module, a second division module and a second storage module (not shown in the figure), wherein,
the calculating module is used for calculating the difference value between the data volume of the data to be stored and the first preset threshold value under the condition that the data volume of the data to be stored is judged to be larger than the first preset threshold value;
the second judging module is used for judging whether the difference value is larger than the first preset threshold value or not; if the difference value is larger than the first preset threshold value, triggering the first dividing module; if the difference value is not greater than the first preset threshold value, triggering a second dividing module;
the second dividing module is used for dividing the data to be stored into two parts of data, the two parts of data comprise a first part of data and a second part of data, and the data amount of the first part of data is equal to the first preset threshold value;
And the second storage module is used for respectively storing the first data and the second data to different processing chips.
As an embodiment, the second storage module may specifically be configured to:
determining a first number of processing chips storing the first data and a second number of processing chips storing the second data according to the ratio of the data amount of the first data to the data amount of the second data;
storing the first data to the first number of processing chips, respectively;
and storing the second data to the second number of processing chips respectively.
As an implementation manner, the processing chip is a GPU, and the storage space is a memory.
By applying the embodiment shown in fig. 4 of the present application, if the data to be stored is greater than the capacity of the processing chip, the data to be stored is divided into hot data and cold data, the hot data is respectively stored in a plurality of processing chips, the cold data is respectively stored in storage spaces of a plurality of computing devices, that is, the hot data and the cold data are both stored in a plurality of portions, so that after receiving the comparison request, the comparison request can be processed in parallel by using the plurality of portions of hot data and cold data.
Fig. 5 is a schematic structural diagram of a data processing apparatus applied to a computing device according to an embodiment of the present application, including:
a receiving module 501, configured to receive data to be processed;
a third judging module 502, configured to judge whether target data compared with the data to be processed has been separated into hot data and cold data; if yes, triggering a comparison module;
a comparing module 503, configured to compare the data to be processed with hot data stored in a processing chip of the computing device and cold data stored in a storage space of the computing device.
As an embodiment, the comparing module 503 may include: a first contrast sub-module, a move sub-module, a load sub-module, and a second contrast sub-module (not shown), wherein,
a first comparison sub-module for comparing the data to be processed with thermal data stored in a processing chip of the computing device;
a movement submodule to move the thermal data to a memory space of the computing device;
a loading sub-module for loading cold data stored in a memory space of the computing device to a processing chip of the computing device;
And the second comparison sub-module is used for comparing the data to be processed with the cold data loaded to the processing chip.
As an embodiment, the loading sub-module may be specifically configured to:
judging whether the data volume of the cold data stored in the storage space is larger than a second preset threshold value or not, wherein the second preset threshold value is set according to the capacity of the processing chip;
if not, loading cold data stored in a memory space of the computing device to a processing chip of the computing device;
if the data is larger than the second preset threshold, dividing the cold data stored in the storage space of the computing device into a plurality of pieces of data to be switched, determining each piece of data to be switched as current switching data in turn, and loading the current switching data to the processing chip after the data in the processing chip is moved to the storage space.
By applying the embodiment shown in fig. 5 of the present application, hot data is stored in a processing chip of a computing device, cold data is stored in a storage space of the computing device, after the computing device receives data to be processed, the data to be processed is compared with the hot data stored in the processing chip and the cold data stored in the storage space, that is, each computing device can compare the hot data and the cold data stored in itself with the data to be processed, so that multiple computing devices can implement parallel processing of the data to be processed.
The embodiment of the present application further provides an electronic device, as shown in fig. 6, including a processor 601, a communication interface 602, a memory 603, and a communication bus 604, where the processor 601, the communication interface 602, and the memory 603 perform communication with each other through the communication bus 604,
a memory 603 for storing a computer program;
the processor 601 is configured to implement any one of the above-described data storage methods and any one of the above-described data processing methods when executing the program stored in the memory 603.
The electronic device can be a scheduling device in the intelligent analysis system, and can also be a computing device in the intelligent analysis system.
The communication bus mentioned above for the electronic devices may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The Memory may include random access Memory (Random Access Memory, RAM) or may include Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processing, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
Embodiments of the present application also provide a computer readable storage medium having a computer program stored therein, which when executed by a processor, implements any one of the data storage methods described above, and any one of the data processing methods described above.
The embodiment of the application also provides an intelligent analysis system, as shown in fig. 1, which comprises a scheduling device and a plurality of computing devices, wherein each computing device comprises a storage space and at least one processing chip;
the scheduling device is used for acquiring data to be stored, judging whether the data volume of the data to be stored is larger than a first preset threshold value or not, wherein the first preset threshold value is set according to the capacity of each processing chip in the system; if the data to be stored is not greater than the first storage instruction, sending a first storage instruction to each computing device, wherein the first storage instruction carries the data to be stored; if the data to be stored is larger than the first storage instruction, dividing the data to be stored into hot data and cold data, and sending a second storage instruction to each computing device, wherein the second storage instruction carries the hot data and the cold data;
The computing device is used for storing the data to be stored to each processing chip of the computing device after receiving the first storage instruction; after receiving the second storage instruction, storing the hot data to each processing chip of the computing device and storing the cold data to a storage space of the computing device.
The intelligent analysis system may comprise a plurality of scheduling devices, and fig. 1 is only for illustration and is not limited to the specific architecture of the intelligent analysis system.
As an implementation manner, the scheduling device is further configured to obtain data to be processed; determining metadata corresponding to target data to be compared with the data to be processed; judging whether the target data is divided into hot data and cold data according to the metadata; if yes, sending a first processing instruction to the computing equipment and the data to be processed; if not, sending a second processing instruction and the data to be processed to the computing equipment;
the computing device is further configured to compare the data to be processed with hot data stored in a processing chip of the computing device and cold data stored in a storage space of the computing device after receiving the first processing instruction and the data to be processed; and after the second processing instruction and the data to be processed are received, comparing the data to be processed with target data stored in a processing chip of the computing device.
The scheduling device in this embodiment may further perform any of the data storage methods in the embodiment of fig. 2, and the computing device in this embodiment may further perform any of the data processing methods in the embodiment of fig. 3.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, with respect to the intelligent analysis system embodiment shown in fig. 1, the data storage apparatus embodiment shown in fig. 4, the electronic device embodiment performing the data storage method, and the computer-readable storage medium embodiment performing the data storage method, since they are substantially similar to the data storage method embodiment shown in fig. 2, the description is relatively simple, and the relevant points are referred to in the partial explanation of the data storage method embodiment shown in fig. 2. The intelligent analysis system embodiment shown in fig. 1, the data processing apparatus embodiment shown in fig. 5, the electronic device embodiment for performing the data processing method, and the computer-readable storage medium embodiment for performing the data processing method are relatively simple in description, since they are substantially similar to the data processing method embodiment shown in fig. 3, and the relevant points are referred to as part of the description of the data processing method embodiment shown in fig. 3.
The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the scope of the present application. Any modifications, equivalent substitutions, improvements, etc. that are within the spirit and principles of the present application are intended to be included within the scope of the present application.
Claims (16)
1. The data storage method is characterized by being applied to scheduling equipment in an intelligent analysis system, wherein the system further comprises a plurality of computing equipment, and each computing equipment comprises a storage space and at least one processing chip; the method comprises the following steps:
acquiring data to be stored; judging whether the data volume of the data to be stored is larger than a first preset threshold value or not, wherein the first preset threshold value is set according to the capacity of a processing chip in the system;
if the data to be stored is larger than the first preset threshold, dividing the data to be stored into hot data and cold data, wherein the data quantity of the hot data is not larger than the first preset threshold;
the hot data is stored to a plurality of processing chips in the system, respectively, and the cold data is stored to storage spaces of a plurality of computing devices, respectively.
2. The method according to claim 1, wherein in case it is determined that the data amount of the data to be stored is larger than a first preset threshold, the method further comprises:
Calculating a difference value between the data quantity of the data to be stored and the first preset threshold value;
judging whether the difference value is larger than the first preset threshold value or not;
if the difference value is larger than the first preset threshold value, executing the step of dividing the data to be stored into hot data and cold data;
if the difference value is not greater than the first preset threshold value, dividing the data to be stored into two parts of data, wherein the two parts of data comprise a first part of data and a second part of data, and the data amount of the first part of data is equal to the first preset threshold value;
and respectively storing the first data and the second data to different processing chips.
3. The method of claim 2, wherein storing the first and second portions of data, respectively, to different processing chips comprises:
determining a first number of processing chips storing the first data and a second number of processing chips storing the second data according to the ratio of the data amount of the first data to the data amount of the second data;
storing the first data to the first number of processing chips, respectively;
And storing the second data to the second number of processing chips respectively.
4. The method of claim 1, wherein the processing chip is a GPU and the storage space is a memory.
5. A data processing method, characterized in that the method is applied to a computing device in an intelligent analysis system, wherein the computing device comprises a storage space and at least one processing chip; the method comprises the following steps:
receiving data to be processed;
judging whether target data compared with the data to be processed is divided into hot data and cold data or not;
if so, comparing the data to be processed with hot data stored in a processing chip of the computing device and cold data stored in a storage space of the computing device.
6. The method of claim 5, wherein comparing the data to be processed with hot data stored in a processing chip of the computing device and cold data stored in a memory space of the computing device comprises:
comparing the data to be processed with thermal data stored in a processing chip of the computing device;
moving the hot data to a storage space of the computing device, and loading cold data stored in the storage space of the computing device to a processing chip of the computing device;
And comparing the data to be processed with cold data loaded to the processing chip.
7. The method of claim 6, wherein loading the cold data stored in the memory space of the computing device to the processing chip of the computing device comprises:
judging whether the data volume of the cold data stored in the storage space is larger than a second preset threshold value or not, wherein the second preset threshold value is set according to the capacity of the processing chip;
if not, loading cold data stored in a memory space of the computing device to a processing chip of the computing device;
if the data is larger than the second preset threshold, dividing the cold data stored in the storage space of the computing device into a plurality of pieces of data to be switched, determining each piece of data to be switched as current switching data in turn, and loading the current switching data to the processing chip after the data in the processing chip is moved to the storage space.
8. A data storage device, characterized in that it is applied to a scheduling device in an intelligent analysis system, the system further comprises a plurality of computing devices, each computing device comprising a storage space and at least one processing chip; the device comprises:
The acquisition module is used for acquiring data to be stored;
the first judging module is used for judging whether the data volume of the data to be stored is larger than a first preset threshold value or not, and the first preset threshold value is set according to the capacity of a processing chip in the system; if the number is greater than the threshold value, triggering a dividing module;
the first dividing module is used for dividing the data to be stored into hot data and cold data, and the data volume of the hot data is not more than the first preset threshold value;
and the first storage module is used for respectively storing the hot data to a plurality of processing chips in the system and respectively storing the cold data to storage spaces of a plurality of computing devices.
9. The apparatus of claim 8, wherein the apparatus further comprises:
the calculating module is used for calculating the difference value between the data volume of the data to be stored and the first preset threshold value under the condition that the data volume of the data to be stored is judged to be larger than the first preset threshold value;
the second judging module is used for judging whether the difference value is larger than the first preset threshold value or not; if the difference value is larger than the first preset threshold value, triggering the first dividing module; if the difference value is not greater than the first preset threshold value, triggering a second dividing module;
The second dividing module is used for dividing the data to be stored into two parts of data, the two parts of data comprise a first part of data and a second part of data, and the data amount of the first part of data is equal to the first preset threshold value;
the second storage module is used for respectively storing the first data and the second data to different processing chips.
10. The apparatus of claim 9, wherein the second storage module is specifically configured to:
determining a first number of processing chips storing the first data and a second number of processing chips storing the second data according to the ratio of the data amount of the first data to the data amount of the second data;
storing the first data to the first number of processing chips, respectively;
and storing the second data to the second number of processing chips respectively.
11. The apparatus of claim 8, wherein the processing chip is a GPU and the storage space is a memory.
12. A data processing apparatus, characterized by being applied to a computing device in an intelligent analysis system, the computing device comprising a memory space and at least one processing chip; the device comprises:
The receiving module is used for receiving the data to be processed;
a third judging module for judging whether the target data compared with the data to be processed has been divided into hot data and cold data; if yes, triggering a comparison module;
and the comparison module is used for comparing the data to be processed with hot data stored in a processing chip of the computing device and cold data stored in a storage space of the computing device.
13. The apparatus of claim 12, wherein the contrast module comprises:
a first comparison sub-module for comparing the data to be processed with thermal data stored in a processing chip of the computing device;
a movement submodule to move the thermal data to a memory space of the computing device;
a loading sub-module for loading cold data stored in a memory space of the computing device to a processing chip of the computing device;
and the second comparison sub-module is used for comparing the data to be processed with the cold data loaded to the processing chip.
14. The apparatus of claim 13, wherein the loading submodule is specifically configured to:
Judging whether the data volume of the cold data stored in the storage space is larger than a second preset threshold value or not, wherein the second preset threshold value is set according to the capacity of the processing chip;
if not, loading cold data stored in a memory space of the computing device to a processing chip of the computing device;
if the data is larger than the second preset threshold, dividing the cold data stored in the storage space of the computing device into a plurality of pieces of data to be switched, determining each piece of data to be switched as current switching data in turn, and loading the current switching data to the processing chip after the data in the processing chip is moved to the storage space.
15. An intelligent analysis system, comprising a scheduling device and a plurality of computing devices, each computing device comprising a memory space and at least one processing chip;
the scheduling device is used for acquiring data to be stored, judging whether the data volume of the data to be stored is larger than a first preset threshold value or not, wherein the first preset threshold value is set according to the capacity of each processing chip in the system; if the data to be stored is not greater than the first storage instruction, sending a first storage instruction to each computing device, wherein the first storage instruction carries the data to be stored; if the data to be stored is larger than the first storage instruction, dividing the data to be stored into hot data and cold data, and sending a second storage instruction to each computing device, wherein the second storage instruction carries the hot data and the cold data;
The computing device is used for storing the data to be stored to each processing chip of the computing device after receiving the first storage instruction; after receiving the second storage instruction, storing the hot data to each processing chip of the computing device and storing the cold data to a storage space of the computing device.
16. The system of claim 15, wherein the system further comprises a controller configured to control the controller,
the scheduling equipment is also used for acquiring data to be processed; determining metadata corresponding to target data to be compared with the data to be processed; judging whether the target data is divided into hot data and cold data according to the metadata; if yes, sending a first processing instruction to the computing equipment and the data to be processed; if not, sending a second processing instruction and the data to be processed to the computing equipment;
the computing device is further configured to compare the data to be processed with hot data stored in a processing chip of the computing device and cold data stored in a storage space of the computing device after receiving the first processing instruction and the data to be processed; and after the second processing instruction and the data to be processed are received, comparing the data to be processed with target data stored in a processing chip of the computing device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810531026.7A CN110543279B (en) | 2018-05-29 | 2018-05-29 | Data storage and processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810531026.7A CN110543279B (en) | 2018-05-29 | 2018-05-29 | Data storage and processing method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110543279A CN110543279A (en) | 2019-12-06 |
| CN110543279B true CN110543279B (en) | 2023-08-04 |
Family
ID=68701475
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810531026.7A Active CN110543279B (en) | 2018-05-29 | 2018-05-29 | Data storage and processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110543279B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111459939B (en) * | 2020-03-31 | 2023-09-19 | 中国银行股份有限公司 | Data processing method and device |
| CN112214447A (en) * | 2020-10-10 | 2021-01-12 | 中科声龙科技发展（北京）有限公司 | Dynamic reconstruction method, system and device for workload certification operation chip cluster data |
| CN112988083B (en) * | 2021-05-19 | 2021-08-03 | 四川省商投信息技术有限责任公司 | Cold data storage method and system |
| CN114185487A (en) * | 2021-11-25 | 2022-03-15 | 深圳市德明利技术股份有限公司 | Cold and hot data identification method and device and computer equipment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731794B (en) * | 2013-12-19 | 2018-07-06 | 北京华易互动科技有限公司 | A kind of cold and hot data fragmentation excavates storage method |
| US9424129B2 (en) * | 2014-04-24 | 2016-08-23 | Seagate Technology Llc | Methods and systems including at least two types of non-volatile cells |
| CN105701028B (en) * | 2014-11-28 | 2018-10-09 | 国际商业机器公司 | Disk management method in distributed memory system and equipment |
| US11126544B2 (en) * | 2016-12-14 | 2021-09-21 | Via Technologies, Inc. | Method and apparatus for efficient garbage collection based on access probability of data |
| CN107193500A (en) * | 2017-05-26 | 2017-09-22 | 郑州云海信息技术有限公司 | A kind of distributed file system Bedding storage method and system |
| CN107908367A (en) * | 2017-11-16 | 2018-04-13 | 郑州云海信息技术有限公司 | Method, apparatus, equipment and the storage medium that data store in storage system |
-
2018
- 2018-05-29 CN CN201810531026.7A patent/CN110543279B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110543279A (en) | 2019-12-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110543279B (en) | Data storage and processing method, device and system | |
| US10455003B2 (en) | Method, server, and system for sharing resource data | |
| US20120266175A1 (en) | Method and device for balancing load of multiprocessor system | |
| CN105589661A (en) | Data Storage Device, Data Processing System And Method Of Operation | |
| CN107621959B (en) | Electronic device and software training method and computing system thereof | |
| US20220035544A1 (en) | Memory allocation method and device, and electronic apparatus | |
| CN108363553A (en) | A kind of data processing method, apparatus and system | |
| US20150095587A1 (en) | Removing cached data | |
| KR102034659B1 (en) | Method for convolution operation redution and system for performing the same | |
| CN105608490A (en) | Cellular array computing system and communication method thereof | |
| CN111679900A (en) | Task processing method and device | |
| US11194619B2 (en) | Information processing system and non-transitory computer readable medium storing program for multitenant service | |
| CN111459937A (en) | Data table association method, device, server and storage medium | |
| CN105868002B (en) | Method and device for processing retransmission request in distributed computing | |
| EP3296878B1 (en) | Electronic device and page merging method therefor | |
| CN114816322A (en) | External sorting method and device of SSD and SSD memory | |
| CN116820324A (en) | Storage capacity expansion method and device, storage medium and electronic equipment | |
| US10169271B1 (en) | Direct memory access descriptor | |
| CN111625358B (en) | Resource allocation method and device, electronic equipment and storage medium | |
| CN114764372A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113568733A (en) | Resource allocation method, device, electronic equipment and storage medium | |
| CN112684972B (en) | Data storage method and device based on distributed block storage | |
| CN112650441A (en) | Stripe cache allocation method and device, electronic equipment and storage medium | |
| US10277912B2 (en) | Methods and apparatus for storing data related to video decoding | |
| US11954339B2 (en) | Memory allocation device and method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |