CN108108384B - Data storage method and device - Google Patents
Data storage method and device Download PDFInfo
- Publication number
- CN108108384B CN108108384B CN201611070058.9A CN201611070058A CN108108384B CN 108108384 B CN108108384 B CN 108108384B CN 201611070058 A CN201611070058 A CN 201611070058A CN 108108384 B CN108108384 B CN 108108384B
- Authority
- CN
- China
- Prior art keywords
- database
- stored
- data
- test
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage method and a device, wherein the method comprises the following steps: aiming at the current data to be stored, extracting a set amount of data from the data to be stored as test data; determining a test type for testing a database to be stored according to a preset performance requirement; testing each database to be stored according to the test type and the test data, and determining a test parameter corresponding to each database to be stored; and determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirements, and storing the data to be stored into the target database. The method and the device are used for solving the problems that a selected database for storing the data to be stored in the prior art is unreasonable, the requirements of users on the performance of the database for storing the data to be stored cannot be met, and the user experience is influenced.
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a data storage method and apparatus.
Background
With the rapid development of information technology, data presents an explosion situation, the amount of data to be stored is larger and larger, and the data type is more and more complex. In the prior art, when data storage is performed, a plurality of databases are generally arranged, and when the database for currently performing data storage is full of data, the next database is started to perform data storage.
However, in the prior art, after the data to be stored is received, the data is stored according to the storage sequence of the database, and the current storage environment requirement and the performance requirement of the user when the data to be stored is stored are not considered, so that the data storage method in the prior art causes unreasonable database selection, cannot meet the performance requirement of the user and the storage environment on the database for storing the data to be stored, and affects the user experience.
Disclosure of Invention
The invention provides a data storage method and a data storage device, which are used for solving the problems that a database for storing data to be stored, which is selected in the prior art, is unreasonable, the performance requirements of a user on the database for storing the data to be stored cannot be met, and the user experience is influenced.
In order to achieve the above object, an embodiment of the present invention discloses a data storage method applied to an electronic device, where the method includes:
aiming at the current data to be stored, extracting a set amount of data from the data to be stored as test data;
determining a test type for testing a database to be stored according to a preset performance requirement;
testing each database to be stored according to the test type and the test data, and determining a test parameter corresponding to each database to be stored;
and determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirements, and storing the data to be stored into the target database.
Further, before determining the test type for testing the database to be stored according to the preset performance requirement, the method further includes:
and selecting a database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type aimed at by each storage database.
Further, the performance requirements include:
the method has the advantages of priority of loading performance, priority of storage efficiency and priority of query performance.
Further, the determining the test type for testing the database to be stored according to the preset performance requirement includes:
if the preset performance requirement is that loading performance is prior or storage efficiency is prior, determining to carry out loading test on the database to be stored;
and if the preset performance requirement is that the query performance is prior, determining to perform query test on the database to be stored.
Further, if the test type is a load test, the testing each database to be stored according to the test type and the test data, and determining the test parameters corresponding to each database to be stored includes:
and controlling each database to be stored to load the test data, and determining the loading time, occupied storage space and the average utilization rate of the CPU for each database to be stored to load the test data.
Further, if the test type is an inquiry test, the testing each database to be stored according to the test type and the test data, and determining the test parameters corresponding to each database to be stored includes:
and inquiring each piece of data in the test data in each database to be stored, and determining the average response time, the average network data transmission quantity and the average CPU utilization rate of each database to be stored for inquiring each piece of data in the test data.
Further, if the preset performance requirement is that loading performance is prioritized, determining a target database according to the test parameters corresponding to each to-be-stored database and the preset performance requirement includes:
determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and average utilization rate of a CPU (central processing unit) for each database to be stored;
sorting the first product results in a descending order, and selecting a first set number of databases to be stored with smaller first product results as first candidate databases;
and determining the database to be stored with the minimum loading time in the first candidate database as a target database.
Further, if the preset performance requirement is storage efficiency priority, determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirement includes:
determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and average utilization rate of a CPU (central processing unit) for each database to be stored;
sorting the first product results in a descending order, and selecting a second set number of databases to be stored with smaller first product results as second candidate databases;
and determining the database to be stored with the minimum storage space occupied in the second candidate database as a target database.
Further, if the preset performance requirement is that the query performance is prior, the determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirement includes:
determining a second product result of average response time, average network data transmission quantity and CPU average utilization rate corresponding to each piece of data in the test data queried in each to-be-stored database;
sorting the second product results in a descending order, and selecting a third set number of databases to be stored with smaller second product results as a third candidate database;
and determining the database to be stored with the minimum average response time in the third candidate databases as a target database.
The embodiment of the invention discloses a data storage device, which is applied to electronic equipment and comprises:
the extraction module is used for extracting a set amount of data from the data to be stored as test data aiming at the current data to be stored;
the first determining module is used for determining the test type for testing the database to be stored according to the preset performance requirement;
the second determining module is used for testing each database to be stored according to the test type and the test data and determining the test parameters corresponding to each database to be stored;
and the third determining module is used for determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirements, and storing the data to be stored into the target database.
Further, the apparatus further comprises:
and the selection module is used for selecting the database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type aimed at by each storage database.
Further, the first determining module is specifically configured to determine to perform a load test on the database to be stored if the preset performance requirement is a load performance priority or a storage efficiency priority; and if the preset performance requirement is that the query performance is prior, determining to perform query test on the database to be stored.
Further, the second determining module is specifically configured to, if the test type is a load test, control each to-be-stored database to load the test data, and determine a load time, an occupied storage space, and an average utilization rate of the CPU for each to-be-stored database to load the test data.
Further, the second determining module is specifically configured to, if the test type is a query test, query each piece of data in the test data in each to-be-stored database, and determine an average response time, an average network data transmission amount, and an average CPU utilization rate of each to-be-stored database for querying each piece of data in the test data.
Further, the third determining module is specifically configured to determine, for each to-be-stored database, a first product result of loading time, occupied storage space, and average utilization rate of the CPU, where the loading time corresponds to the test data loaded by the to-be-stored database, if the preset performance requirement is that loading performance is prioritized; sorting the first product results in a descending order, and selecting a first set number of databases to be stored with smaller first product results as first candidate databases; and determining the database to be stored with the minimum loading time in the first candidate database as a target database.
Further, the third determining module is specifically configured to determine, for each to-be-stored database, a first product result of loading time, occupied storage space, and average utilization rate of the CPU, where the loading time corresponds to the test data loaded by the to-be-stored database, if the preset performance requirement is that storage efficiency is first; sorting the first product results in a descending order, and selecting a second set number of databases to be stored with smaller first product results as second candidate databases; and determining the database to be stored with the minimum storage space occupied in the second candidate database as a target database.
Further, the third determining module is specifically configured to determine, for each to-be-stored database, a second product result of an average response time, an average network data transmission amount, and an average CPU utilization rate, which correspond to each piece of data in the test data to be queried in the to-be-stored database, if the preset performance requirement is that query performance is prioritized; sorting the second product results in a descending order, and selecting a third set number of databases to be stored with smaller second product results as a third candidate database; and determining the database to be stored with the minimum average response time in the third candidate databases as a target database.
The embodiment of the invention discloses a data storage method and a data storage device, wherein the method comprises the following steps: aiming at the current data to be stored, extracting a set amount of data from the data to be stored as test data; determining a test type for testing a database to be stored according to a preset performance requirement; testing each database to be stored according to the test type and the test data, and determining a test parameter corresponding to each database to be stored; and determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirements, and storing the data to be stored into the target database. In the embodiment of the invention, the set amount of data is extracted from the data to be stored as the test data, and each database to be stored is tested by adopting the corresponding test type according to the preset performance requirement, so that the target database is determined according to the test parameters corresponding to each database to be stored, and the selected target database is the database meeting the current performance requirement, thereby the data storage is more reasonable, and the user experience is improved.
Drawings
Fig. 1 is a schematic diagram of a data storage process provided in embodiment 1 of the present invention;
fig. 2 is a schematic diagram of a data storage process according to embodiment 2 of the present invention;
fig. 3 is a schematic diagram of a data storage process according to embodiment 4 of the present invention;
fig. 4 is a schematic diagram of a data storage process provided in embodiment 5 of the present invention;
fig. 5 is a schematic diagram of a data storage process according to embodiment 6 of the present invention;
fig. 6 is a schematic structural diagram of a data storage device according to embodiment 7 of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1:
fig. 1 is a schematic diagram of a data storage process provided in an embodiment of the present invention, where the process includes:
s101: and aiming at the current data to be stored, extracting a set number of data from the data to be stored as test data.
The data storage method provided by the embodiment of the invention is applied to electronic equipment, and the electronic equipment can be a personal computer, a tablet personal computer, a server and other equipment.
Specifically, for the data to be currently stored, a set number of data is extracted from the data to be stored as the test data, where the set number may be 5, 10, and so on, for example: the current data to be stored contains 100 pieces of data in total, the set number is 10 pieces, and 10 pieces of data are randomly extracted from the data to be stored as test data, but the set number may also be a proportional value for extracting the data to be stored, for example: the received data to be stored contains 100 pieces of data in total, the data with the quantity set to be 10% of the data to be stored is set, and 10% of the data is randomly extracted from the data to be stored, namely 10 pieces of data are randomly extracted to serve as test data.
In the embodiment of the present invention, randomly extracting a set number of data from the data to be stored as test data is the prior art, and details are not repeated in the embodiment of the present invention.
S102: and determining the test type for testing the database to be stored according to the preset performance requirement.
A database is a repository that organizes, stores, and manages data according to a data structure. Different types or types of databases with different architectures are different in architecture and different in data storage structure, and when different databases load or read the same data, parameters such as loading time, reading time, and memory occupancy rate may also be different.
In the embodiment of the present invention, a plurality of performance requirements are stored in the electronic device, and when receiving data to be stored, the electronic device may select the performance requirements according to the current storage environment and the number of data pieces included in the data to be stored, for example: if the storage space of the database to be stored is nearly full, the selected performance requirement is that the storage efficiency is prior, so that the purpose of reducing the storage space of the data as much as possible is achieved, and if the number of the data to be stored exceeds a set threshold value, the selected performance requirement is that the query performance is prior, so that the aim of improving the data query efficiency is achieved, and certainly, the user can select the performance requirement according to the requirement. Each performance requirement corresponds to a test type, for example: the performance requirement is that the query performance corresponds to the query test preferentially, and the performance requirement is that the memory performance corresponds to the read test preferentially when reading.
Specifically, according to a preset performance requirement, a test type for testing the database to be stored is determined, for example: the preset performance requirement is that the memory performance is prior during reading, and the reading test is determined to be performed on the database to be stored.
S103: and testing each database to be stored according to the test type and the test data, and determining the test parameters corresponding to each database to be stored.
Specifically, the test parameter is an operation parameter of the electronic device where each to-be-stored database is located when each to-be-stored database is tested according to the test type and the test data, and the operation parameter can reflect an influence of the database on performance of the electronic device when the database performs corresponding operation on the test data.
The test parameters may be, for example: the average utilization rate of a Central Processing Unit (CPU), the average occupied amount of a memory and other parameters. If the test type is a read test, the databases to be stored comprise a database A, a database B, a database C and a database D, the test data are respectively stored in the database A, the database B, the database C and the database D, and parameters such as reading time, memory occupation amount, CPU average usage amount and the like corresponding to the test data read by the database A, the database B, the database C and the database D are recorded.
S104: and determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirements, and storing the data to be stored into the target database.
Specifically, specific parameters in the test parameters corresponding to each database to be stored may be compared according to a preset performance requirement to determine a target database, for example: the database to be stored comprises a database A, a database B and a database C, wherein the parameters corresponding to the database A are reading time 1 second, CPU average utilization rate 10% and memory occupation amount 30M, the parameters corresponding to the database B are reading time 2 second, CPU average utilization rate 15% and memory occupation amount 30M, the parameters corresponding to the database C are reading time 2.5 second, CPU average utilization rate 10% and memory occupation amount 30M, the preset performance requirement is reading time priority, the reading times corresponding to the database A, the database B and the database C are selected for comparison, and the database A with the minimum reading time is determined to be used as a target database.
Of course, the test parameters corresponding to each database to be stored may also be comprehensively compared according to the preset performance requirement to determine the target database, for example: the database to be stored comprises a database A, a database B and a database C, wherein the parameters corresponding to the database A are reading time 1 second, CPU average utilization rate 10% and memory occupation amount 30M, the parameters corresponding to the database B are reading time 2 seconds, CPU average utilization rate 15% and memory occupation amount 30M, the parameters corresponding to the database C are reading time 2.5 seconds, CPU average utilization rate 10% and memory occupation amount 30M, the preset performance requirement is reading time priority, products of test parameters corresponding to the database A, the database B and the database C are respectively compared, and the database A with the minimum product of the test parameters is determined to be used as a target database.
In the embodiment of the invention, the set amount of data is extracted from the data to be stored as the test data, and each database to be stored is tested by adopting the corresponding test type according to the preset performance requirement, so that the target storage database is determined according to the test parameters corresponding to each database to be stored, and therefore, the selected target storage database is the database meeting the current performance requirement, the data storage is more reasonable, and the user experience is improved.
Example 2:
in order to accurately determine a target database for data storage, on the basis of the above embodiment, in an embodiment of the present invention, before determining a test type for testing a database to be stored according to a preset performance requirement, the method further includes:
and selecting a database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type aimed at by each storage database.
According to the different architectures of the database, the database comprises: key-value store databases, document-type databases, column store databases, and graph databases, wherein a key-value store database mainly uses a hash table having a specific key and a pointer pointing to specific data, and such databases have advantages of simplicity and easy deployment, and typical key-value store databases include: redis, Memcached, Dynamo, Voldemout, LevelDB, the type of data mainly aiming at the database comprises structured configuration type data, log type data and the like; the column store database is typically used to handle the massive amount of data stored in a distributed manner, keys still exist, but the column store database is characterized by pointing to a plurality of columns, and a typical column store database includes: HBase and Cassandra, wherein the data types mainly aimed by the database of the type comprise account form data, detail form data and the like; graph databases, unlike other row-column and rigid-structured databases, use flexible graphical models and can be extended to multiple servers, typical Graph databases include Neo4J, Infinite Graph, OrientDB, and the types of data that the databases of this type are primarily directed to include social data, relational data, and the like; the data structure of the document type database is not strict, the table structure is variable, the table structure does not need to be predefined like a relational database, a typical document type database comprises MongoDB and CouchDB, and the data types mainly aimed at by the database comprise website type data, cache type data and the like.
Because the data types of each database are different, the selected data type of the database to be stored corresponds to the data type to be stored, so that the requirement of a user on the storage performance during the storage of the data to be stored can be better met, and the performance of the database to be stored is fully exerted. The identification of the type of the data to be stored is the prior art and is not described in detail in the embodiment of the present invention.
Specifically, according to the type of the data to be stored and the data type targeted by each storage database, the database corresponding to the type of the data to be stored is selected as the database to be stored. For example: and selecting the databases HBase and Cassandra aiming at the bill type data as the databases to be stored.
Fig. 2 is a schematic diagram of a data storage process provided in an embodiment of the present invention, where the process includes:
s201: and aiming at the current data to be stored, extracting a set number of data from the data to be stored as test data.
S202: and selecting a database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type aimed at by each storage database.
S203: and determining the test type for testing the database to be stored according to the preset performance requirement.
S204: and testing each database to be stored according to the test type and the test data, and determining the test parameters corresponding to each database to be stored.
S205: and determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirements, and storing the data to be stored into the target database.
In the above steps, in the processes of extracting the set amount of data from the data to be stored as the test data in S202 and S201, the sequence may not be limited, and a database corresponding to the type of the data to be stored may be selected as the database to be stored first, or the set amount of data may be extracted from the data to be stored first as the test data.
Example 3:
in an embodiment of the present invention, the performance requirements include:
the method has the advantages of priority of loading performance, priority of storage efficiency and priority of query performance.
The loading performance priority means that the loading performance of the database needs to be considered preferentially when the target database is selected; the storage efficiency priority means that the storage efficiency of the database needs to be considered preferentially when the target database is selected; the query performance priority means that the query performance of the database needs to be considered when selecting the target database.
In order to understand the corresponding performance requirements of each database to be stored, the database to be tested needs to be tested correspondingly, because each performance requirement is different, the test types of the test performed on the database to be tested are also different. Specifically, in the embodiment of the present invention, the determining, according to the preset performance requirement, the test type for testing the database to be stored includes:
if the preset performance requirement is that loading performance is prior or storage efficiency is prior, determining to carry out loading test on the database to be stored;
and if the preset performance requirement is that the query performance is prior, determining to perform query test on the database to be stored.
Specifically, if the preset performance requirement is that the loading performance is first or the storage efficiency is first, it indicates that the loading performance or the storage performance requirement for the user to load the data to be stored in the database to be stored is higher, and in order to determine the loading performance and the storage performance of the database to be stored, a loading test needs to be performed on the database to be stored.
If the preset performance requirement is that the query performance is prior, the query performance requirement that the user queries the data to be stored in the database to be stored is higher, and the query test needs to be carried out on the database to be stored.
Example 4:
in order to accurately select a target database meeting a preset performance requirement, on the basis of the above embodiments, in an embodiment of the present invention, if the preset performance requirement is a loading performance priority, a determined test type is a loading test, the testing of each database to be stored according to the test type and the test data, and determining a test parameter corresponding to each database to be stored includes:
and controlling each database to be stored to load the test data, and determining the loading time, occupied storage space and the average utilization rate of the CPU for each database to be stored to load the test data.
Specifically, each database to be stored is controlled to load the test data, because the test data may include multiple pieces of data, for each database to be stored, the loading time of the database to be stored in the process of loading the test data and the average utilization rate of the CPU are counted, and the size of the storage space occupied by the test data in the database to be stored is determined.
If the preset performance requirement is that the loading performance is prior, the loading test is carried out on each database to be stored by adopting the test data, after the test data of the database to be tested is obtained, the step of determining the target database according to the test parameters corresponding to each database to be stored and the preset performance requirement comprises the following steps:
determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and average utilization rate of a CPU (central processing unit) for each database to be stored;
sorting the first product results in a descending order, and selecting a first set number of databases to be stored with smaller first product results as first candidate databases;
and determining the database to be stored with the minimum loading time in the first candidate database as a target database.
Specifically, in the embodiment of the present invention, when the target database is determined, for each to-be-stored database, a first product result of loading time, occupied storage space, and average utilization rate of the CPU corresponding to the test data loaded by the to-be-stored database is determined, the first product results are sorted from large to small, and a first set number of to-be-stored databases with smaller first product results are selected as first candidate databases, that is, a first set number of to-be-stored databases with smaller test data consumption resources are selected as first candidate databases, where the set number may be 2, 3, 5, and so on. And selecting the database to be stored with the minimum loading time as a target database from the first candidate databases with smaller test data loading consumption resources. For example: the first product result corresponding to the database A to be stored is 2, the first product result corresponding to the database B to be stored is 2.1, the first product result corresponding to the database C to be stored is 2.5, the first product result corresponding to the database D to be stored is 3, the first set number is 2, the first product results corresponding to the database A to be stored, the database B to be stored, the database C to be stored and the database D to be stored are sorted from large to small, the database A to be stored and the database B to be stored with the smaller first product result are selected as first candidate databases, the loading time corresponding to the database A to be stored and the database B to be stored is respectively 0.1 second and 0.11 second, and the database A to be stored with the smallest loading time is selected as a target database.
Fig. 3 is a schematic diagram of a data storage process provided in an embodiment of the present invention, where the process includes:
s301: and aiming at the current data to be stored, extracting a set number of data from the data to be stored as test data.
S302: and selecting a database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type aimed at by each storage database.
S303: and if the preset performance requirement is that the loading performance is prior, controlling each database to be stored to load the test data, and determining the loading time, the occupied storage space and the average utilization rate of the CPU for each database to be stored to load the test data.
S304: and determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and the average utilization rate of the CPU for each database to be stored.
S305: and sorting the first product results according to the descending order, and selecting a first set number of databases to be stored with smaller first product results as a first candidate database.
S306: and determining the database to be stored with the minimum loading time in the first candidate database as a target database, and storing the data to be stored in the target database.
In the above steps, in the processes of extracting the set amount of data from the data to be stored as the test data in S302 and S301, the sequence may not be limited, and a database corresponding to the type of the data to be stored may be selected as the database to be stored, or the set amount of data may be extracted from the data to be stored as the test data.
Example 5:
in order to accurately select a target database meeting a preset performance requirement, on the basis of the above embodiments, in an embodiment of the present invention, if the preset performance requirement is storage efficiency, determining that a test type is a loading test, testing each to-be-stored database according to the test type and the test data, and determining a test parameter corresponding to each to-be-stored database includes:
and controlling each database to be stored to load the test data, and determining the loading time, occupied storage space and the average utilization rate of the CPU for each database to be stored to load the test data.
Specifically, each database to be stored is controlled to load the test data, because the test data may include multiple pieces of data, for each database to be stored, the loading time of the database to be stored in the process of loading the test data and the average utilization rate of the CPU are counted, and the size of the storage space occupied by the test data in the database to be stored is determined.
If the preset performance requirement is that the loading performance is prior, the loading test is carried out on each database to be stored by adopting the test data, after the test data of the database to be tested is obtained, the step of determining the target database according to the test parameters corresponding to each database to be stored and the preset performance requirement comprises the following steps:
determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and average utilization rate of a CPU (central processing unit) for each database to be stored;
sorting the first product results in a descending order, and selecting a second set number of databases to be stored with smaller first product results as second candidate databases;
and determining the database to be stored with the minimum storage space occupied in the second candidate database as a target database.
Specifically, in the embodiment of the present invention, for each to-be-stored database when determining the target database, a first product result of loading time corresponding to the test data, occupied storage space, and average utilization rate of the CPU for the to-be-stored database is determined, the first product result is sorted from large to small, a second set number of to-be-stored databases with smaller first product results are selected as second candidate databases, that is, a second set number of to-be-stored databases with smaller test data consumption resources are selected as the second candidate databases, where the set number may be 2, 3, 5, and so on. And selecting a database to be stored with the minimum occupied storage space from a second candidate database with smaller consumption resources for loading the test data as a target database. For example: the first product result corresponding to the database A to be stored is 2, the first product result corresponding to the database B to be stored is 2.1, the first product result corresponding to the database C to be stored is 2.5, the first product result corresponding to the database D to be stored is 3, the second set number is 2, the first product results corresponding to the database A to be stored, the database B to be stored, the database C to be stored and the database D to be stored are sorted from large to small, the database A to be stored and the database B to be stored with the smaller first product result are selected as second candidate databases, occupied storage spaces corresponding to the database A to be stored and the database B to be stored are respectively 600 bytes and 550 bytes, and the database B to be stored with the smallest occupied storage space is selected as a target database.
Fig. 4 is a schematic diagram of a data storage process provided in an embodiment of the present invention, where the process includes:
s401: and aiming at the current data to be stored, extracting a set number of data from the data to be stored as test data.
S402: and selecting a database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type aimed at by each storage database.
S403: and if the preset performance requirement is storage efficiency, controlling each database to be stored to load the test data, and determining the loading time, occupied storage space and the average utilization rate of the CPU for each database to be stored to load the test data.
S404: and determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and the average utilization rate of the CPU for each database to be stored.
S405: and sorting the first product results in a descending order, and selecting a second set number of databases to be stored with smaller first product results as second candidate databases.
S406: and determining the database to be stored with the minimum storage space occupied in the second candidate database as a target database, and storing the data to be stored into the target database.
In the above steps, in the processes of extracting the set amount of data from the data to be stored as the test data in S402 and S401, the sequence may not be limited, a database corresponding to the type of the data to be stored may be selected as the database to be stored, or the set amount of data may be extracted from the data to be stored as the test data.
Example 6:
in order to accurately select a target database meeting a preset performance requirement, on the basis of the above embodiments, in an embodiment of the present invention, if the preset performance requirement is a query performance priority, determining a test type as a query test, testing each to-be-stored database according to the test type and the test data, and determining a test parameter corresponding to each to-be-stored database includes:
and inquiring each piece of data in the test data in each database to be stored, and determining the average response time, the average network data transmission quantity and the average CPU utilization rate of each database to be stored for inquiring each piece of data in the test data.
Specifically, because the test data may include a plurality of pieces of data, each piece of data in the test data is queried in each database to be stored, and the average response time, the average network data transmission amount and the average CPU utilization rate in the process of querying each piece of data in the test data in the database to be stored are counted.
If the preset performance requirement is that the query performance is prior, performing query test on each database to be stored by adopting test data, and after the test data of the database to be tested is obtained, determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirement comprises the following steps:
determining a second product result of average response time, average network data transmission quantity and CPU average utilization rate corresponding to each piece of data in the test data queried in each to-be-stored database;
sorting the second product results in a descending order, and selecting a third set number of databases to be stored with smaller second product results as a third candidate database;
and determining the database to be stored with the minimum average response time in the third candidate databases as a target database.
Specifically, in the embodiment of the present invention, when the target database is determined, for each to-be-stored database, a second product result of an average response time, an average network data transmission amount, and an average CPU utilization rate corresponding to each piece of data in the test data in the to-be-stored database is determined, the second product results are sorted from large to small, and a third set number of to-be-stored databases with smaller second product results are selected as a third candidate database, that is, a third set number of to-be-stored databases with smaller average resource consumption for querying each piece of data in the test data are selected as the third candidate database, where the set number may be 2, 3, 5, and the like. And selecting the database with the minimum average response time as a target database from third candidate databases with smaller average resource consumption of each piece of data in the test data. For example: the second product result corresponding to the database A to be stored is 3, the second product result corresponding to the database B to be stored is 2.8, the second product result corresponding to the database C to be stored is 2.6, the second product result corresponding to the database D to be stored is 2.5, the third set number is 2, the second product results corresponding to the database A to be stored, the database B to be stored, the database C to be stored and the database D to be stored are sorted from large to small, the database C to be stored and the database D to be stored with smaller second product results are selected as third candidate databases, the average response time corresponding to the database C to be stored and the database D to be stored is respectively 0.11 second and 0.1 second, and the database D to be stored with the smallest average response time is selected as a target database.
Fig. 5 is a schematic diagram of a data storage process provided in an embodiment of the present invention, where the process includes:
s501: and aiming at the current data to be stored, extracting a set number of data from the data to be stored as test data.
S502: and selecting a database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type aimed at by each storage database.
S503: and if the preset performance requirement is that the query performance is prior, querying each piece of data in the test data in each database to be stored, and determining the average response time, the average network data transmission quantity and the average CPU utilization rate of each database to be stored for querying each piece of data in the test data.
S504: and determining a second product result of the average response time, the average network data transmission quantity and the CPU average utilization rate corresponding to the query of each piece of data in the test data in each database to be stored.
S505: and sorting the second product results in a descending order, and selecting a third set number of databases to be stored with smaller second product results as a third candidate database.
S506: and determining the database to be stored with the minimum average response time in the third candidate databases as a target database, and storing the data to be stored into the target database.
In the above steps, in the processes of extracting the set amount of data from the data to be stored as the test data in S502 and S501, the sequence may not be limited, and a database corresponding to the type of the data to be stored may be selected as the database to be stored first, or the set amount of data may be extracted from the data to be stored first as the test data.
Example 7:
fig. 6 is a schematic structural diagram of a data storage device according to an embodiment of the present invention, where the data storage device includes:
the extracting module 61 is configured to, for current data to be stored, extract a set number of data from the data to be stored as test data;
the first determining module 63 is configured to determine a test type for testing the database to be stored according to a preset performance requirement;
a second determining module 64, configured to test each to-be-stored database according to the test type and the test data, and determine a test parameter corresponding to each to-be-stored database;
a third determining module 65, configured to determine a target database according to the test parameter corresponding to each to-be-stored database and the preset performance requirement, and store the to-be-stored data in the target database.
The device further comprises:
and a selecting module 62, configured to select, according to the type of the data to be stored and the data type targeted by each storage database, a database corresponding to the type of the data to be stored as a database to be stored.
The first determining module 63 is specifically configured to determine to perform a load test on the database to be stored if the preset performance requirement is a load performance priority or a storage efficiency priority; and if the preset performance requirement is that the query performance is prior, determining to perform query test on the database to be stored.
The second determining module 64 is specifically configured to, if the test type is a load test, control each to-be-stored database to load the test data, and determine a loading time, an occupied storage space, and an average utilization rate of the CPU for each to-be-stored database to load the test data.
The second determining module 64 is specifically configured to, if the test type is a query test, query each piece of data in the test data in each to-be-stored database, and determine an average response time, an average network data transmission amount, and an average CPU utilization rate of each to-be-stored database for querying each piece of data in the test data.
The third determining module 65 is specifically configured to determine, for each to-be-stored database, a first product result of loading time, occupied storage space, and average utilization rate of the CPU, where the loading time, occupied storage space, and average utilization rate correspond to loading of the test data by the to-be-stored database, if the preset performance requirement is that loading performance is prioritized; sorting the first product results in a descending order, and selecting a first set number of databases to be stored with smaller first product results as first candidate databases; and determining the database to be stored with the minimum loading time in the first candidate database as a target database.
The third determining module 65 is specifically configured to determine, for each to-be-stored database, a first product result of loading time, occupied storage space, and average utilization rate of the CPU, where the loading time corresponds to loading the test data in the to-be-stored database, if the preset performance requirement is that storage efficiency is first; sorting the first product results in a descending order, and selecting a second set number of databases to be stored with smaller first product results as second candidate databases; and determining the database to be stored with the minimum storage space occupied in the second candidate database as a target database.
The third determining module 65 is specifically configured to determine, for each to-be-stored database, a second product result of an average response time, an average network data transmission amount, and an average CPU utilization rate, which correspond to the query of each piece of data in the test data in the to-be-stored database, if the preset performance requirement is that the query performance is prioritized; sorting the second product results in a descending order, and selecting a third set number of databases to be stored with smaller second product results as a third candidate database; and determining the database to be stored with the minimum average response time in the third candidate databases as a target database.
The embodiment of the invention discloses a data storage method and a data storage device, which are applied to electronic equipment, wherein the method comprises the following steps of; aiming at the current data to be stored, extracting a set amount of data from the data to be stored as test data; determining a test type for testing a database to be stored according to a preset performance requirement; testing each database to be stored according to the test type and the test data, and determining a test parameter corresponding to each database to be stored; and determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirements, and storing the data to be stored into the target database. In the embodiment of the invention, the set amount of data is extracted from the data to be stored as the test data, and each database to be stored is tested by adopting the corresponding test type according to the preset performance requirement, so that the target database is determined according to the test parameters corresponding to each database to be stored, and the selected target database is the database meeting the current performance requirement, thereby the data storage is more reasonable, and the user experience is improved.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (11)
1. A data storage method is applied to an electronic device, and the method comprises the following steps:
aiming at the current data to be stored, extracting a set amount of data from the data to be stored as test data;
determining a test type for testing a database to be stored according to a preset performance requirement;
testing each database to be stored according to the test type and the test data, and determining a test parameter corresponding to each database to be stored;
determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirements, and storing the data to be stored into the target database;
wherein, if the preset performance requirement is that the loading performance is prior, the determining the target database according to the test parameters corresponding to each database to be stored and the preset performance requirement comprises:
determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and average utilization rate of a CPU (central processing unit) for each database to be stored;
sorting the first product results in a descending order, and selecting a first set number of databases to be stored with smaller first product results as first candidate databases;
determining a database to be stored with the minimum loading time in the first candidate database as a target database;
if the preset performance requirement is that storage efficiency is first, determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirement comprises:
determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and average utilization rate of a CPU (central processing unit) for each database to be stored;
sorting the first product results in a descending order, and selecting a second set number of databases to be stored with smaller first product results as second candidate databases;
determining the database to be stored with the minimum storage space occupied in the second candidate database as a target database;
if the preset performance requirement is that the query performance is prior, determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirement comprises:
determining a second product result of average response time, average network data transmission quantity and CPU average utilization rate corresponding to each piece of data in the test data queried in each to-be-stored database;
sorting the second product results in a descending order, and selecting a third set number of databases to be stored with smaller second product results as a third candidate database;
and determining the database to be stored with the minimum average response time in the third candidate databases as a target database.
2. The method of claim 1, wherein before determining the type of test to test the database to be stored according to the predetermined performance requirement, the method further comprises:
and selecting a database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type aimed at by each storage database.
3. The method of claim 1, wherein the performance requirements comprise:
the method has the advantages of priority of loading performance, priority of storage efficiency and priority of query performance.
4. The method of claim 3, wherein determining the type of test to test the database to be stored according to the predetermined performance requirement comprises:
if the preset performance requirement is that loading performance is prior or storage efficiency is prior, determining to carry out loading test on the database to be stored;
and if the preset performance requirement is that the query performance is prior, determining to perform query test on the database to be stored.
5. The method of claim 4, wherein if the test type is a load test, the testing each database to be stored according to the test type and the test data, and determining the test parameters corresponding to each database to be stored comprises:
and controlling each database to be stored to load the test data, and determining the loading time, occupied storage space and the average utilization rate of the CPU for each database to be stored to load the test data.
6. The method of claim 4, wherein if the test type is query test, the testing each database to be stored according to the test type and the test data, and determining the test parameters corresponding to each database to be stored comprises:
and inquiring each piece of data in the test data in each database to be stored, and determining the average response time, the average network data transmission quantity and the average CPU utilization rate of each database to be stored for inquiring each piece of data in the test data.
7. A data storage device, for use in an electronic device, the device comprising:
the extraction module is used for extracting a set amount of data from the data to be stored as test data aiming at the current data to be stored;
the first determining module is used for determining the test type for testing the database to be stored according to the preset performance requirement;
the second determining module is used for testing each database to be stored according to the test type and the test data and determining the test parameters corresponding to each database to be stored;
a third determining module, configured to determine a target database according to the test parameters corresponding to each to-be-stored database and the preset performance requirement, and store the to-be-stored data in the target database;
the third determining module is specifically configured to determine, for each to-be-stored database, a first product result of loading time, occupied storage space, and average utilization rate of the CPU, where the loading time, occupied storage space, and average utilization rate correspond to loading of the test data by the to-be-stored database, if the preset performance requirement is that loading performance is prioritized; sorting the first product results in a descending order, and selecting a first set number of databases to be stored with smaller first product results as first candidate databases; determining a database to be stored with the minimum loading time in the first candidate database as a target database;
if the preset performance requirement is that the storage efficiency is prior, determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and the average utilization rate of the CPU for each database to be stored; sorting the first product results in a descending order, and selecting a second set number of databases to be stored with smaller first product results as second candidate databases; determining the database to be stored with the minimum storage space occupied in the second candidate database as a target database;
if the preset performance requirement is that the query performance is prior, determining a second product result of average response time, average network data transmission quantity and CPU average utilization rate corresponding to the query of each piece of data in the test data in each database to be stored aiming at each database to be stored; sorting the second product results in a descending order, and selecting a third set number of databases to be stored with smaller second product results as a third candidate database; and determining the database to be stored with the minimum average response time in the third candidate databases as a target database.
8. The apparatus of claim 7, wherein the apparatus further comprises:
and the selection module is used for selecting the database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type aimed at by each storage database.
9. The apparatus according to claim 7, wherein the first determining module is specifically configured to determine to perform a load test on the database to be stored if the preset performance requirement is a load performance priority or a storage efficiency priority; and if the preset performance requirement is that the query performance is prior, determining to perform query test on the database to be stored.
10. The apparatus according to claim 9, wherein the second determining module is specifically configured to, if the test type is a load test, control each of the databases to be stored to load the test data, and determine a load time, an occupied storage space, and an average utilization rate of the CPU for each of the databases to be stored to load the test data.
11. The apparatus according to claim 9, wherein the second determining module is specifically configured to, if the test type is a query test, query each piece of data in the test data in each of the databases to be stored, and determine an average response time, an average network data transmission amount, and an average CPU utilization rate of each database to be stored for querying each piece of data in the test data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611070058.9A CN108108384B (en) | 2016-11-25 | 2016-11-25 | Data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611070058.9A CN108108384B (en) | 2016-11-25 | 2016-11-25 | Data storage method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108108384A CN108108384A (en) | 2018-06-01 |
| CN108108384B true CN108108384B (en) | 2020-10-09 |
Family
ID=62206105
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611070058.9A Active CN108108384B (en) | 2016-11-25 | 2016-11-25 | Data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108108384B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109189785A (en) * | 2018-08-10 | 2019-01-11 | 平安科技（深圳）有限公司 | Date storage method, device, computer equipment and storage medium |
| CN109256173A (en) * | 2018-08-13 | 2019-01-22 | 中国航空无线电电子研究所 | On-board memory devices test macro |
| CN110895450A (en) * | 2019-11-01 | 2020-03-20 | 中电和瑞科技有限公司 | Data storage method and device, storage medium and computer equipment |
| CN110990606B (en) * | 2019-12-11 | 2023-10-03 | Tcl移动通信科技（宁波）有限公司 | Picture storage method and device, storage medium and electronic equipment |
| US11645424B2 (en) * | 2020-04-27 | 2023-05-09 | International Business Machines Corporation | Integrity verification in cloud key-value stores |
| CN113760889B (en) * | 2020-11-09 | 2024-05-17 | 北京沃东天骏信息技术有限公司 | Data processing method, device, computer system and computer readable storage medium |
| CN113032402B (en) * | 2021-04-09 | 2024-01-12 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for storing data and acquiring data |
| CN113268681A (en) * | 2021-07-05 | 2021-08-17 | 数字广东网络建设有限公司 | User home location identification method and device |
| CN113449042B (en) * | 2021-07-14 | 2024-06-21 | 中国银行股份有限公司 | Automatic data warehouse separation method and device |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7761420B2 (en) * | 2007-10-16 | 2010-07-20 | International Business Machines Corporation | Method and system for replicating objects |
| CN101742531B (en) * | 2008-11-13 | 2013-03-20 | 中国移动通信集团设计院有限公司 | Method and device for storing test data |
| DE102010033512A1 (en) * | 2010-09-28 | 2012-03-29 | Siemens Aktiengesellschaft | Method for increasing performance of memory system for permanent storage of medical image data, involves storing defragmented data in sequence and calling data of respective applications |
| CN103905517A (en) * | 2012-12-28 | 2014-07-02 | 中国移动通信集团公司 | Data storage method and equipment |
| US9465715B2 (en) * | 2014-06-12 | 2016-10-11 | Oracle International Corporation | Optimizing the number of shared processes executing in a computer system |
| CN105550274B (en) * | 2015-12-10 | 2019-01-25 | 曙光信息产业(北京)有限公司 | The querying method and device of this parallel database of two-pack |
| CN105608138B (en) * | 2015-12-18 | 2019-03-12 | 贵州大学 | A kind of system of optimization array data base concurrency data loading performance |
-
2016
- 2016-11-25 CN CN201611070058.9A patent/CN108108384B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN108108384A (en) | 2018-06-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108108384B (en) | Data storage method and device | |
| US11526799B2 (en) | Identification and application of hyperparameters for machine learning | |
| US11055360B2 (en) | Data write-in method and apparatus in a distributed file system | |
| CN109035013B (en) | Apparatus and method for detecting abnormal financial transaction and computer readable storage medium | |
| KR102104193B1 (en) | How to select service parameters and related devices | |
| US10579616B2 (en) | Data search system, data search method, and program product | |
| CN104572717B (en) | Information searching method and device | |
| JP2020074193A (en) | Search method, device, facility, and non-volatile computer memory | |
| CN106789837B (en) | Network anomalous behaviors detection method and detection device | |
| US9760836B2 (en) | Data typing with probabilistic maps having imbalanced error costs | |
| GB2581458A (en) | Authentication of users at multiple terminals | |
| EP3301603A1 (en) | Improved search for data loss prevention | |
| CN106598649A (en) | Application program installation method and device, and electronic equipment | |
| CN105989152B (en) | Method, device and system for monitoring service quality of search engine | |
| CN109213972B (en) | Method, device, equipment and computer storage medium for determining document similarity | |
| CN113763034A (en) | Advertisement data processing method and device | |
| CN110688223B (en) | Data processing method and related product | |
| CN106257449A (en) | A kind of information determines method and apparatus | |
| CN111027065B (en) | Leucavirus identification method and device, electronic equipment and storage medium | |
| CN112101024A (en) | Target object identification system based on app information | |
| CN104077272A (en) | Method and device for compressing dictionary | |
| CN105589683B (en) | Sample extraction method and device | |
| US10241708B2 (en) | Storage of a data chunk with a colliding fingerprint | |
| CN105740210A (en) | Method and device for determining information similarity | |
| CN105144139A (en) | Generating a feature set |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |