CN110008272B - NoSQL database evaluation system for sensor data and construction method thereof - Google Patents
NoSQL database evaluation system for sensor data and construction method thereof Download PDFInfo
- Publication number
- CN110008272B CN110008272B CN201910284813.0A CN201910284813A CN110008272B CN 110008272 B CN110008272 B CN 110008272B CN 201910284813 A CN201910284813 A CN 201910284813A CN 110008272 B CN110008272 B CN 110008272B
- Authority
- CN
- China
- Prior art keywords
- nosql
- database
- data
- kafka
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45595—Network integration; Enabling network access in virtual machine instances
Abstract
The invention discloses NoSQL database evaluation system facing sensor data and a construction method thereof, wherein the system comprises a NoSQL database layer, a middleware layer and a user interface layer, the NoSQL database layer comprises a plurality of NoSQL databases, the middleware layer adopts a micro-service architecture middleware layer, the NoSQL databases are deployed on virtual machines constructed in a Cloud computing environment by using Spring Cloud technology, the configuration of the virtual machines is completely , the user interface layer provides an operation area and a result display area for the operation of the MySQL databases for users on the user interface, the transmission mode of the sensor data under the real condition is simulated by Kafka, the sensor data is provided for different NoSQL databases in a mode, and the message transmission among modules in the micro-service is realized by Kafka.
Description
Technical Field
The invention relates to NoSQL database evaluation systems for sensor data and a construction method thereof, belonging to the technical field of software performance evaluation and measurement analysis.
Background
With the rapid development of the internet of things and sensor technologies, the world has become more instrumented and interconnected, and we have also witnessed the unprecedented speed increase of massive sensor data generated by various sensors, which have important research and application values, and have widely served the business scenarios of flood protection, environmental protection, industrial manufacturing, etc. however, conventional storage approaches, represented by relational databases, have difficulty in efficiently storing large-scale sensor stream data, in the face of large unstructured data sets or sensor stream data, the shortcomings of conventional databases are mainly manifested in (1) increasing data in a single table, low processing efficiency, (2) high maintenance costs of relational databases due to complex data transformation/migration, (3) high parallelization of data due to their heterogeneity, and difficulty in improving database performance by expanding machine clusters.
Currently, although there are typical tools or research works such as YCSB, YCSB + +, benchfountain for evaluation of different nosqls, the evaluation work is far from sufficient compared to many nosqls, and is more likely to be digressive than conventional evaluation systems and tools for relational databases.
Disclosure of Invention
The invention aims to solve the problems and the defects in the prior art, the invention provides NoSQL database evaluation systems for sensor data and a construction method thereof according to the characteristics of the sensor data, provides a selection reference for a storage mechanism of the sensor data from the aspect of the performance of executing query, update and deletion operations on the sensor data, realizes the connection of different NoSQL database systems by researching and developing middleware layers based on micro-services, and realizes different load tasks (condition query, deletion and update operations) on the basis of the connection, so that a user can fairly and quickly compare the performance differences of different NoSQL databases through the execution performance of different load tasks, and provides a quantifiable comparison scheme for the selection of the storage mechanism of the sensor data, namely the user can easily judge which NoSQL database is selected for query speed or which NoSQL database is selected for update operation with the highest execution speed.
According to the technical scheme, NoSQL database evaluation systems for sensor data enable users to fairly and quickly compare performance differences of storage mechanisms of the NoSQL databases according to the evaluation systems, and therefore a storage mode most suitable for self requirements is selected.
The system comprises a NoSQL database layer, a middleware layer and a user interface layer, wherein sensor data are greatly changed in scale and structure in the past along with the development of the technology, the data volume is increased in geometric quantity due to the continuous improvement of the acquisition precision, meanwhile, the data are not limited to types of relational type, and various non-relational data are generated continuously, wherein the NoSQL database layer comprises a plurality of NoSQL databases, the middleware layer is used as a core and integrates different NoSQL databases, so that not only is the perfect support provided for performance test service, but also effective screening is performed on the database to be tested before testing aiming at the data characteristics of the sensor, reasonable evaluation schemes are provided, the testing efficiency is improved, the layer adopts a micro-business architecture, aims to realize the solution by decomposing various operations of different databases into various discrete services to realize the solution, realizes the realization of each database by using a Spring Noud software technology, namely 34 independent services, and provides various kinds of simple and complicated configuration through a Cloud-Cloud service registration, namely a Spud software, namely a Spud software, a software;
the user interface layer adopts a WYSIWYG style, and provides an operation area and a result display area for the user to operate the MySQL database on the interface.
The middleware layer deploys the NoSQL databases on virtual machines constructed in a cloud computing environment, and the configuration of these virtual machines is completely such that virtual machines correspond to each NoSQL database.
The Kafka simulates the transmission mode of the sensor data under the real condition, the sensor data is provided to different NoSQL databases in an -based mode so as to carry out load pressure test, and the message transmission among modules in the micro service is realized through the Kafka.
The NoSQL database layer comprises 4 NoSQL databases which are respectively as follows: a document database MongoDB, a wide-column database HBase, a key-value database Memcached and a key-value database Redis.
The specific operation of the operation area comprises the following steps: selecting operation types, setting retrieval conditions, connecting and disconnecting the NoSQL database, resetting data and displaying results. The result display area can display whether the database is connected and the load execution result.
The selection of operation types includes query, delete, and update operations.
The user presets a search condition which can be a key search or a condition search, and simultaneously, the user can select or a plurality of databases to simultaneously evaluate, the evaluations are concurrent and can simultaneously search the plurality of databases.
And connecting and disconnecting the NoSQL database. Before the test starts, the database is guaranteed to be available by testing the connection, and when the test is terminated, the database connection can be disconnected.
And data resetting, namely, when testing, ensuring that each NoSQL database is in the same state, and resetting the NoSQL database to ensure that data in the database is samples.
A construction method of a NoSQL database evaluation system for sensor data specifically comprises the following steps:
Step 2: installation of the open source cloud computing solution Apache cloudstock. Apache cloudstock is used as software to manage the virtual cluster.
And step 3: selecting a virtual machine group with the same scale, wherein the configuration standard is not lower than the lowest configuration capable of running of the database, and then respectively installing and deploying HBase, Redis, MongoDB, Memcached and MySQL databases.
And 4, step 4: based on Kafka, connection and sensor data transmission of different NoSQL databases are achieved, and the sensor data transmission refers to that data are inserted into each NoSQL database through Kafka. The algorithm is as follows:
inputting: nosql type, Nosql data, Kafka topics, Kafka partitions, query language, Nosql connectivity information
And (3) outputting: kafka stream data
Step , judging the database type and connecting the database;
step two: when the Nosql database is connected, continuously reading data from the database according to the query language, initializing the data, and outputting the data according to a specified format;
step three: judging the number of Kafka themes, when the number of the Kafka themes is not empty, continuously judging the number of partitions in the themes, when the number of the partitions is larger than 0 and the number of the required partitions is not larger than the default number of the partitions in the Kafka, distributing information to different partitions according to Key (Key) value calculation, sending the information through a Kafka producer, if the Kafka consumer does not exist, caching the information in the Kafka into the Kafka, if the Kafka consumer does exist, conducting consumer-side consumption on the cached information in the Kafka, splitting sensor data through consumer-side logic, and outputting the split sensor data in a stream form;
and 5: implementing a corresponding load operation, including: the method comprises the steps of condition query, deletion operation, updating operation, maximum value calculation, minimum value calculation, average value calculation, log clearing and data resetting, and operation results are displayed on an evaluation system interface for a user to refer to.
And (3) condition query: and the user inputs related query conditions, executes corresponding query operations by the NoSQL database, returns the execution time of each NoSQL database and the data volume of the query operation result, and displays the execution time and the data volume in the result display for the user to evaluate the execution performance of each NoSQL database query operation.
And (3) deleting operation: deleting the data meeting the conditions, executing corresponding deleting operation by NoSQL, returning the executing time and the deleted data volume of each NoSQL database, and displaying in result display for the user to evaluate the executing performance of the deleting operation of each NoSQL database.
Updating operation: and updating data according to the main key, covering the original data or deleting the original data and inserting new data, executing corresponding updating operation by the NoSQL database, returning the execution time of each NoSQL and the data volume of the updating operation, and displaying in result display for a user to evaluate the execution performance of each NoSQL database updating operation.
Calculating the maximum value: calculating the maximum value of all the numerical values which accord with the input calculation condition according to the input calculation condition;
calculating a minimum value: according to the input calculation condition, calculating the minimum value of all numerical values meeting the condition;
calculating the average value: calculating the average value of all values meeting the input calculation condition according to the input calculation condition;
step 6: and designing and realizing an interface of the evaluation system, and providing an interface for operating the NoSQL database for a user.
(1) Section is a global query option the user needs to first determine the type of profile database based on the existing data set.
(2) The second column is the state of the database service, and only when the state of the database service is open, the subsequent operations, such as retrieval, addition, update, etc., can be performed, the third column is a clear data operation, which refers to the deletion of all the data in the test database, the fourth column is a reset data operation, when the data is reset, the clear table operation is performed first, then the sensor data is read from the MySQL and finally inserted into the corresponding NoSQL database, the fifth column is a statistical quantity operation, which counts the number of all the data of the selected NoSQL database table.
(3) And the third part is result display, and all operations of the NoSQL database are recorded by logs and displayed in the result display part. The contents displayed are: the operation is time-consuming and the data size of the operation result is large.
And (3) condition query: the user inputs relevant query conditions and clicks a condition query button. The log records that the operation is a condition query operation, a query condition, execution time of each database and queried data volume.
And (3) deleting operation: and deleting the sensor data meeting the conditions, and clicking a delete button. The log records that the operation is a deletion operation, deletion conditions, execution time of each NoSQL database and deleted data volume.
Updating operation: and updating the sensor data according to the main key, covering the original data or deleting the original data and inserting new data, and clicking an update button. The log records that the operation is an updating operation, the updating condition, the execution time of each NoSQL database and the updating data volume.
Calculating the maximum value: and calculating the maximum value of the inquired result. The log records that the operation is the operation of calculating the maximum value, and the execution time and the maximum value of each NoSQL database.
Calculating a minimum value: and calculating the minimum value of the inquired result. The log records that the operation is the operation of calculating the minimum value, and the execution time and the minimum value of each NoSQL database.
Calculating the average value: and calculating an average value of the inquired results. The log records that the operation is an average value calculation operation, and the execution time and the average value of each NoSQL database.
Drawings
FIG. 1 is an architectural diagram of an embodiment of the present invention;
FIG. 2 is a diagram of a system user interface of an embodiment of the present invention.
Detailed Description
the invention is further illustrated by reference to specific examples, which are intended to be illustrative only and not to be limiting of the scope of the invention, and various equivalent modifications will occur to those skilled in the art upon reading the disclosure and fall within the scope of the invention as defined in the appended claims.
FIG. 1 is a block diagram of an architecture of NoSQL database evaluation systems for sensor data according to the present invention, wherein the evaluation systems are divided into three layers, i.e., a NoSQL database layer, a middleware layer and a user interface layer from bottom to top.
The NoSQL database layer comprises 4 NoSQL databases, namely a document database MongoDB, a wide-column database HBase, a key value database Memcached and a key value database Redis, and is also connected with a MySQL database, in order to fairly evaluate different NoSQL databases, the NoSQL database is deployed on virtual machines constructed in a cloud computing environment, and the configuration of the virtual machines is completely .
A flow data technology of Kafka is selected for fairly evaluating different NoSQL databases and ensuring that sensor data can be provided to the different NoSQL databases in a -based mode, a transmission mode of the sensor data under a real condition is simulated through the Kafka and is provided to the different NoSQL databases in a -based mode so as to carry out load pressure test, message transmission among modules (4 Nosql databases and Kafka modules) in micro service is realized through the Kafka, although a REST design mode is very popular, the REST design mode is synchronous and can cause blockage, and therefore a Kafka message queue is used for replacing REST requests/responses in load test and function request selection.
The middleware layer is a core, integrates different NoSQL databases, not only provides complete support on performance test services, but also provides reasonable evaluation schemes for the databases to be tested before testing according to the data characteristics of the sensors, and improves the testing efficiency.
And the user interface layer adopts a WYSIWYG style and provides an operation area and a result display area for the operation of the MySQL database for the user on the interface. The operation area specifically comprises the following operations: selecting operation types, setting operation conditions, connecting and disconnecting the NoSQL database, resetting data and displaying results. The result display area can display whether the NoSQL database is connected and load the execution result.
The selection of operation types includes query, delete, and update operations.
The search condition setting is that the user sets the search condition in advance, which can be the primary key search or the condition search, and at the same time, the user can select or a plurality of databases to evaluate simultaneously, the evaluations are concurrent, and the operations can be performed on the plurality of databases simultaneously.
Connecting and disconnecting NoSQL database: before the test starts, the database is guaranteed to be available by testing the connection, and when the test is terminated, the database connection can be disconnected.
And (3) resetting data, namely, when testing is carried out, the NoSQL databases are required to be ensured to be in the same state, the NoSQL databases are required to be reset, and the data volume in the databases is ensured to be .
And (3) displaying the result: and the method is used for returning the evaluation result of the test to be performed for visual display.
The method comprises the steps that before a database is evaluated, sensor data need to be imported into a NoSQL database in advance, the sensor data are provided by a user, the user needs to provide information such as corresponding data relation of the data and imported data quantity, a system can automatically create corresponding scripts according to the information, different NoSQL databases have different import mechanisms, and the scripts can automatically judge and select proper import methods to execute corresponding import operations.
The invention also discloses a construction method of the NoSQL database evaluation systems for the sensor data, which comprises the following steps:
the method comprises the following steps of 1, virtualization of infrastructure, integration and shared utilization of a host and storage resources by adopting a virtualization technology, wherein virtualization is mainly carried out from two aspects, two virtualization pools, namely a computing virtualization pool and a storage virtualization pool, are established, the computing virtualization pool mainly realizes application virtualization and application middleware virtualization at the computing resource level, the server virtualization refers to the abstraction of physical resources of the server into logical resources, servers are changed into several or even hundreds of mutually isolated virtual servers, the storage virtualization pool mainly realizes data storage virtualization and comprises storage hardware architecture virtualization and storage software virtualization at the storage level, the invention builds the hardware such as the host, a management node, a plurality of computing nodes and network equipment according to the thought, evaluates different NoSQL databases to provide necessary infrastructure bases, caused by the evaluation, concretely operates as follows, rents or purchases a plurality of servers, utilizes the server virtualization technology to expand a plurality of virtual servers, ensures that the configuration of the virtual servers is completely , installs an operating system on the virtual servers, plans a network node, and installs a data base, and evaluates the required NoSQL database.
Step 2: installation of the open source cloud computing solution Apache cloudstock. The Apache cloud Stack is used as software for managing the virtual machine cluster, so that a user can quickly and conveniently create a private cloud computing platform on the existing infrastructure and manage the virtual machine cluster, and the installation process mainly comprises the following steps:
(1) configuring the installation source (both the management and the compute nodes need to be configured);
$ sudo sed -i s/us.archive.ubuntu.com/mirrors.163.com/g /etc/apt/sources.list
$ sudo sed -i s/security.ubuntu.com/mirrors.163.com/g /etc/apt/sources.list
configuring cloudstock official sources
$ sudo vim /etc/apt/sources.list.d/cloudstack.list
deb http://cloudstack.apt-get.eu/ubuntu precise 4.0
Configuring cloudstock official source certificates
$ wget -O - http://cloudstack.apt-get.eu/release.asc| sudo apt-keyadd -
Installation source for updating system
$ sudo apt-get update
(2) Installing a CloudStack Management Server;
(3) installing a HOST HOST;
after the operations (2) and (3) are performed, the mutual communication among the plurality of virtual servers can be realized.
(4) Configuration security policies, bridges, firewalls, NFS sharing, etc.
And configuring a security policy.
$ sudo dpkg --list "apparmor"
And configuring a network bridge, setting the physical network card as a manual, and then setting an IP on the virtual network card and bridging to the physical network card.
$ sudo vim /etc/network/interfaces
And configuring a firewall, and closing the firewall to prevent the communication from being disabled.
$ sudo ufw disable
NFS sharing, cloudstock requires places to store Primar and Secondary Storage, and authorities recommend using NFS sharing.
$ sudo apt-get install nfs-common nfs-kernel-server
And step 3: selecting a virtual machine group with the same scale, wherein the configuration standard is not lower than the lowest configuration capable of running of the database, and then respectively installing and deploying HBase, Redis, MongoDB, Memcached and MySQL databases.
And 4, step 4: based on Kafka, Java programming is utilized to realize connection of different NoSQL databases and sensor data transmission, wherein the sensor data transmission refers to the insertion of data into each NoSQL database through Kafka. The algorithm is as follows.
V. NoSQL System interface
Input:nosql data x,Kafka_topic t,Kafka_Pation p,QuerySql q,NosqlConnection c;
Output:Kafka Stream z;
GetConnect(Nosql){
switch(Nosql){
case nosqltype:
return c;
}
}
//Determine types of NoSQL and Get different connections
while(c is connected){
x<-GetData(q,c)
//Continuously read messages from the current NoSQL database
x'<-Serialization(x); //Serialized message
if(x'.next is true){
message=format(x');
}
if(t is none){
throw Error;
p=None; //Judge whether kafka topic exists or not
}
if(p not none){
if(p<0 or partion>=numPartions){
throw Error; // Determine partitioning
}
else(){
p<-hash(key)%numPartions;
// If the partition is normal, the partition is
// determined bypassing in a message key value
KafkaProducer<-message;
//Messages are passed into kafkaProducer to form kafka Stream
if(kafkaConsumer is none){
Kafka<-kafkaProducer.message;
//If there is no consumer, the message is cached to Kafka
}
else(){
kafkaConsumer<-kafka.message;
// send the kakfa message to the consumer
message'=split(kafkaConsumer.message);
z=message';
//Message splitting and form data stream
}
}
}
}
Inputting: nosql type, Nosql data, Kafka topics, Kafka partitions, query language, Nosql connectivity information
And (3) outputting: kafka stream data
Step , judging the database type and connecting the database;
step two: when the Nosql database is connected, continuously reading data from the database according to the query language, initializing the data, and outputting the data according to a specified format;
step three: judging the number of Kafka themes, when the number of the themes in the Kafka is not empty, continuously judging the number of partitions in the themes, when the number of the partitions is larger than 0 and the number of the required partitions is not larger than the default number of the partitions in the Kafka, distributing information to different partitions according to Key value calculation, sending the information through a Kafka producer, if the Kafka consumer does not exist, caching the information in the Kafka into the Kafka, if the Kafka consumer does exist, conducting consumer-side consumption on the cached information in the Kafka, splitting sensor data through consumer-side logic, and outputting the split sensor data in a stream form;
and 5: based on Java programming development, corresponding load operation is realized, and the method comprises the following steps: condition query, deletion operation, updating operation, maximum calculation, minimum calculation, average calculation, log clearing and data resetting.
And (3) condition query: and the user inputs related query conditions, executes corresponding query operations by the NoSQL database, returns the execution time of each NoSQL database and the data volume of the query operation result, and displays the execution time and the data volume in the result display for the user to evaluate the execution performance of each NoSQL database query operation.
And (3) deleting operation: deleting the data meeting the conditions, executing corresponding deleting operation by NoSQL, returning the executing time and the deleted data volume of each NoSQL database, and displaying in result display for the user to evaluate the executing performance of the deleting operation of each NoSQL database.
Updating operation: and updating data according to the main key, covering the original data or deleting the original data and inserting new data, executing corresponding updating operation by the NoSQL database, returning the execution time of each NoSQL and the data volume of the updating operation, and displaying in result display for a user to evaluate the execution performance of each NoSQL database updating operation.
Calculating the maximum value: calculating the maximum value of all the numerical values which accord with the input calculation condition according to the input calculation condition;
calculating a minimum value: according to the input calculation condition, calculating the minimum value of all numerical values meeting the condition;
calculating the average value: calculating the average value of all values meeting the input calculation condition according to the input calculation condition;
and 5: based on Java programming development, corresponding load operation is realized, and the method comprises the following steps: condition query, deletion operation, updating operation, maximum calculation, minimum calculation, average calculation, log clearing and data resetting.
Step 6: the design and implementation of the evaluation system interface is shown in fig. 2.
(1) The method comprises the steps of firstly, determining the type of an evaluation database according to an existing data set, wherein the data set of a sensor is of multiple types, such as key value type, document type, time sequence type and the like, the user inputs the data set type, and an evaluation system recommends several corresponding databases according to the type.
(2) The second part is the operation of all databases, column is the database name, the second column is the state of the database service, only if the state of the database service is open, the following operations, such as retrieval, addition, update, etc., can be performed, the third column is the operation of clearing data, the fourth column is the operation of resetting data, when resetting data, the operation of clearing the table is performed first, then the sensor data is read from MySQL, and finally inserted into the corresponding NoSQL database, the fifth column is the operation of counting number, which can count the number of all data of the selected NoSQL database table, the sixth column is the operation type, which can select the query operation, update operation or delete operation, the following two columns are the original value and the updated value, if the query operation, only the original value needs to be written, if the update operation, the original value needs to be written and the updated value, the last column is the operation performed.
(3) And the third part is result display, and each operation of the NoSQL database is recorded by a log and displayed in the result display part. The contents displayed are: the operation is time-consuming and the data size of the operation result is large.
And (3) condition query: the user inputs relevant query conditions and clicks a condition query button. The log records that the operation is a condition query operation, a query condition, execution time of each database and queried data volume.
And (3) deleting operation: and deleting the sensor data meeting the conditions, and clicking a delete button. The log records that the operation is a deletion operation, deletion conditions, execution time of each NoSQL database and deleted data volume.
Updating operation: and updating the sensor data according to the main key, covering the original data or deleting the original data and inserting new data, and clicking an update button. The log records that the operation is an updating operation, the updating condition, the execution time of each NoSQL database and the updating data volume.
Calculating the maximum value: and calculating the maximum value of the inquired result. The log records that the operation is the operation of calculating the maximum value, and the execution time and the maximum value of each NoSQL database.
Calculating a minimum value: and calculating the minimum value of the inquired result. The log records that the operation is the operation of calculating the minimum value, and the execution time and the minimum value of each NoSQL database.
Calculating the average value: and calculating an average value of the inquired results. The log records that the operation is an average value calculation operation, and the execution time and the average value of each NoSQL database.
Claims (7)
1. A NoSQL database evaluation system facing sensor data is characterized in that the system comprises a NoSQL database layer, a middleware layer and a user interface layer, wherein the NoSQL database layer comprises a plurality of NoSQL databases, the NoSQL database layer comprises 4 NoSQL databases, namely a document database MongoDB, a wide-column database HBase, a key value database Memcached and a key value database Redis, the middleware layer adopts a micro service architecture middleware layer, various operations of different databases are decomposed into various discrete services to realize the decoupling of a solution, the solution is realized by using a Spring Cloud technology, the NoSQL database is deployed on virtual machines constructed in a Cloud computing environment, and the configuration of the virtual machines is completely , each NoSQL database is independent services, the registration and discovery of related services are carried out by using Spring Cloud Eureka, various configuration problems of the NoSQL databases exist before, and the user operation result is provided for a user management API through a Congwash management system, and the user interface layer provides a user interface API for the user through a Spring Cloud API 36;
the Kafka simulates the transmission mode of the sensor data under the real condition, and the sensor data is provided to different NoSQL databases in an mode so as to carry out load pressure test;
firstly, NoSQL database interfaces are unified, and NoSQL type, NoSQL data, Kafka theme, Kafka partition, query language and NoSQL connection information are input, Kafka stream data:
step , judging the database type and connecting the database;
step two: when the NoSQL database is connected, continuously reading data from the database according to the query language, initializing the data and outputting the data according to a specified format;
step three: judging the number of Kafka themes, when the number of the themes in the Kafka is not empty, continuously judging the number of partitions in the themes, when the number of the partitions is larger than 0 and the number of the required partitions is not larger than the default number of the partitions in the Kafka, distributing information to different partitions according to Key value calculation, sending the information through a Kafka producer, if the Kafka consumer does not exist, caching the information in the Kafka, if the Kafka consumer does not exist, performing consumer-side consumption on the information cached in the Kafka, splitting the sensor data through consumer-side logic, and outputting the split sensor data in a stream form.
2. The NoSQL database evaluation system for sensor data according to claim 1, wherein: the specific operation of the operation area comprises the following steps: selecting operation types, setting retrieval conditions, connecting and disconnecting a NoSQL database and resetting data; and the result display area displays whether the database is connected and the operation result.
3. The NoSQL database evaluation system for the sensor data according to claim 1, wherein the search condition is set by a user, the search condition is preset by the user, the search condition can be a key search or a condition search, the user can select or more databases to evaluate simultaneously, the evaluations are concurrent and can search a plurality of NoSQL databases simultaneously, and the connection and disconnection of the NoSQL databases are realized by testing the connection before the test starts and selecting the disconnection of the databases when the test is terminated.
4. The NoSQL database evaluation system according to claim 1, wherein the data resetting is performed to ensure that each NoSQL database is in the same state, to ensure that the data in the database is sample.
5, method for constructing NoSQL database evaluation system for sensor data, which is characterized by comprising the following steps:
the method comprises the following steps of 1, virtualizing infrastructure, virtualizing from two aspects by utilizing a server virtualization technology, establishing two virtualization pools, namely a calculation virtualization pool and a storage virtualization pool, wherein the calculation virtualization pool is used for realizing application virtualization and comprises server virtualization and application middleware virtualization at a calculation resource level;
step 2: installing an open source cloud computing solution Apache cloud Stack; using Apache cloud as software for managing the virtual machine group;
and step 3: selecting a virtual cluster with the same scale, and then respectively installing and deploying different NoSQL databases and MySQL databases; different NoSQL databases include HBase, Redis, MongoDB, Memcached;
step 4, based on Kafka, realizing connection of different NoSQL databases and sending sensor data, wherein the sending of the sensor data refers to inserting data into each NoSQL database through the Kafka, firstly, summarizing NoSQL database interfaces, inputting NoSQL type, NoSQL data, Kafka theme, Kafka partition, query language and Nosql connection information, and outputting Kafka flow data:
step , judging the database type and connecting the database;
step two: when the NoSQL database is connected, continuously reading data from the database according to the query language, initializing the data and outputting the data according to a specified format;
step three: judging the number of Kafka themes, when the number of the themes in the Kafka is not empty, continuously judging the number of partitions in the themes, when the number of the partitions is larger than 0 and the number of the required partitions is not larger than the default number of the partitions in the Kafka, distributing information to different partitions according to Key value calculation, sending the information through a Kafka producer, if the Kafka consumer does not exist, caching the information in the Kafka into the Kafka, if the Kafka consumer does exist, conducting consumer-side consumption on the cached information in the Kafka, splitting sensor data through consumer-side logic, and outputting the split sensor data in a stream form;
and 5: realizing corresponding load operation of the NoSQL database, and displaying an operation result on an interface of an evaluation system for a user to refer;
step 6: designing and realizing an evaluation system interface, and providing an interface for operating the NoSQL database for a user;
(1) the part is the top global query option, the user needs to determine the type of the evaluation database according to the existing data set, because the sensor data set has multiple types, the user inputs the data set type, the evaluation system recommends several corresponding databases according to the type, for the hydrological sensor data, the user needs to provide the query condition, the query condition comprises the site, the site number and the time interval, then when inquiring MongoDB and HBase, the input values of the three options are added into the query condition, because Memcached and Redis do not support the complex query, the option does not support Memcached and Redis, if the site number is not input, the default query is performed on all the sensor sites, if the input is performed, the query is performed according to the input value, the time range can be either a closed interval or an open interval;
(2) the second part is the operation of all databases, the th column is the name of the database, the second column is the state of the database service, only when the state of the database service is open, the subsequent operation can be carried out, the operation comprises retrieval, addition and update, the third column is the operation of emptying data, the fourth column is the operation of resetting data, when the data is reset, the operation of clearing a table is firstly carried out, then the sensor data is read from MySQL and finally inserted into a corresponding NoSQL database, the fifth column is the operation of counting quantity, the number of all data of the selected NoSQL database table can be counted, the sixth column is the operation type, and the operation can be selected as query operation, update operation or delete operation, the subsequent two columns are respectively the original value and the updated value, if the query operation is carried out, only the original value needs to be written, if the update operation is carried out, the last column is the operation, the second part is the lowest value counting, and only when the query is carried out, the average value counting is carried out;
(3) the third part is result display, and each operation of the NoSQL database is recorded by a log and displayed in the result display part; the contents displayed are: the executed operation, the time consumption and the data volume of the operation result;
and (3) condition query: a user inputs related query conditions and clicks a condition query button; the log records that the operation is a condition query operation, a query condition, execution time of each database and queried data volume;
and (3) deleting operation: deleting the sensor data meeting the conditions, and clicking a delete button; the log records that the operation is a deleting operation, deleting conditions, execution time of each NoSQL database and deleted data volume;
updating operation: updating sensor data according to the main key, covering original data or deleting the original data and inserting new data, and clicking an update button; the log records that the operation is an updating operation, updating conditions, execution time of each NoSQL database and updating data volume;
calculating the maximum value: calculating the maximum value of the inquired result; the log records that the operation is maximum value calculation operation, and the execution time and the maximum value of each NoSQL database;
calculating a minimum value: calculating the minimum value of the inquired result; the log records that the operation is the minimum value calculation operation, and the execution time and the minimum value of each NoSQL database;
calculating the average value: calculating an average value of the inquired results; the log records that the operation is an average value calculation operation, and the execution time and the average value of each NoSQL database.
6. The method for constructing the NoSQL database evaluation system for sensor data according to claim 5, wherein the corresponding load operation of the NoSQL database includes: condition query, deletion operation, updating operation, maximum value calculation, minimum value calculation, average value calculation and data resetting;
and (3) condition query: inputting related query conditions by a user, executing corresponding query operations by the NoSQL database, returning the execution time of each NoSQL database and the data volume of the query operation result, and displaying in result display for the user to evaluate the execution performance of each NoSQL database query operation;
and (3) deleting operation: deleting the data meeting the conditions, executing corresponding deleting operation by NoSQL, returning the execution time and the deleted data volume of each NoSQL database, and displaying in result display for the user to evaluate the execution performance of the deleting operation of each NoSQL database;
updating operation: updating data according to the main key, covering the original data or deleting the original data and inserting new data, executing corresponding updating operation by the NoSQL database, returning the execution time of each NoSQL and the data volume of the updating operation, and displaying in result display for a user to evaluate the execution performance of each NoSQL database updating operation;
calculating the maximum value: calculating the maximum value of all the numerical values which accord with the input calculation condition according to the input calculation condition;
calculating a minimum value: according to the input calculation condition, calculating the minimum value of all numerical values meeting the condition;
calculating the average value: and calculating the average value of all the numerical values meeting the input calculation condition according to the input calculation condition.
7. The method for constructing the NoSQL database evaluation system for the sensor data according to claim 5, wherein in the step 3, a virtual cluster with the same scale is selected, the configuration standard is not lower than the lowest configuration capable of being operated by the database, and then HBase, Redis, MongoDB, Memcached and MySQL databases are respectively installed and deployed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910284813.0A CN110008272B (en) | 2019-04-10 | 2019-04-10 | NoSQL database evaluation system for sensor data and construction method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910284813.0A CN110008272B (en) | 2019-04-10 | 2019-04-10 | NoSQL database evaluation system for sensor data and construction method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110008272A CN110008272A (en) | 2019-07-12 |
| CN110008272B true CN110008272B (en) | 2020-01-31 |
Family
ID=67170771
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910284813.0A Active CN110008272B (en) | 2019-04-10 | 2019-04-10 | NoSQL database evaluation system for sensor data and construction method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110008272B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110908956B (en) * | 2019-11-19 | 2023-06-02 | 许继集团有限公司 | Information protection master station system and fault information archiving method thereof |
| CN112948228B (en) * | 2021-03-15 | 2023-07-21 | 河海大学 | Multi-mode database evaluation benchmark system for stream data and construction method thereof |
| CN112948357B (en) * | 2021-03-15 | 2023-04-25 | 河海大学 | Multimode database OrientDB-oriented tuning system and construction method thereof |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103049556A (en) * | 2012-12-28 | 2013-04-17 | 中国科学院深圳先进技术研究院 | Fast statistical query method for mass medical data |
| CN104317789A (en) * | 2014-04-24 | 2015-01-28 | 科技谷（厦门）信息技术有限公司 | Method for building passenger social network |
| CN105138685A (en) * | 2015-09-17 | 2015-12-09 | 福建新大陆软件工程有限公司 | HBase-oriented performance testing system |
| CN105404581A (en) * | 2015-12-25 | 2016-03-16 | 北京奇虎科技有限公司 | Database evaluation method and device |
| CN106682148A (en) * | 2016-12-22 | 2017-05-17 | 北京锐安科技有限公司 | Method and device based on Solr data search |
| CN108768826A (en) * | 2018-05-16 | 2018-11-06 | 华南理工大学 | Based on the message route method under MQTT and Kafka high concurrent scenes |
| CN109144485A (en) * | 2018-09-10 | 2019-01-04 | 南方电网科学研究院有限责任公司 | A kind of dispositions method of micro services, device, equipment and readable storage medium storing program for executing |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109542639B (en) * | 2018-11-06 | 2021-05-11 | 用友网络科技股份有限公司 | Processing method and processing device for guaranteeing consistency of microservice calling data |
-
2019
- 2019-04-10 CN CN201910284813.0A patent/CN110008272B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103049556A (en) * | 2012-12-28 | 2013-04-17 | 中国科学院深圳先进技术研究院 | Fast statistical query method for mass medical data |
| CN104317789A (en) * | 2014-04-24 | 2015-01-28 | 科技谷（厦门）信息技术有限公司 | Method for building passenger social network |
| CN105138685A (en) * | 2015-09-17 | 2015-12-09 | 福建新大陆软件工程有限公司 | HBase-oriented performance testing system |
| CN105404581A (en) * | 2015-12-25 | 2016-03-16 | 北京奇虎科技有限公司 | Database evaluation method and device |
| CN106682148A (en) * | 2016-12-22 | 2017-05-17 | 北京锐安科技有限公司 | Method and device based on Solr data search |
| CN108768826A (en) * | 2018-05-16 | 2018-11-06 | 华南理工大学 | Based on the message route method under MQTT and Kafka high concurrent scenes |
| CN109144485A (en) * | 2018-09-10 | 2019-01-04 | 南方电网科学研究院有限责任公司 | A kind of dispositions method of micro services, device, equipment and readable storage medium storing program for executing |
Non-Patent Citations (1)
| Title |
|---|
| 虚拟化环境下的NoSQL数据库评测;段虎一 等;《空军预警学院学报》;20131215;第27卷(第6期);摘要,第1.2,2.2,3.1-3.2节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110008272A (en) | 2019-07-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112685385B (en) | Big data platform for smart city construction | |
| CN110543537B (en) | Intelligent planning space-time cloud GIS platform based on Docker container and micro-service architecture | |
| CN104331477B (en) | The cloud platform concurrency performance method of testing retrieved based on federal style | |
| CN1713179B (en) | Impact analysis in an object model | |
| CN110008272B (en) | NoSQL database evaluation system for sensor data and construction method thereof | |
| EP3077917B1 (en) | Distributing data on distributed storage systems | |
| CN111523003A (en) | Data application method and platform with time sequence dynamic map as core | |
| CN102402583B (en) | Real-load tuning of database applications | |
| CN106534291A (en) | Voltage monitoring method based on big data processing | |
| CN101741614B (en) | Equivalent type node manager and equivalent type node management method | |
| CN111639114A (en) | Distributed data fusion management system based on Internet of things platform | |
| CN103473696A (en) | Method and system for collecting, analyzing and distributing internet business information | |
| CN108268614B (en) | Distributed management method for forest resource spatial data | |
| CN110516076B (en) | Knowledge graph-based cloud computing management method and system | |
| CN107122238B (en) | Efficient iterative Mechanism Design method based on Hadoop cloud Computational frame | |
| CN114791846B (en) | Method for realizing observability aiming at cloud-originated chaos engineering experiment | |
| CN112148578A (en) | IT fault defect prediction method based on machine learning | |
| CN103473247B (en) | Geological data information cluster mechanism and interface aggregation system | |
| CN113127526A (en) | Distributed data storage and retrieval system based on Kubernetes | |
| CN102868601B (en) | Routing system related to network topology based on graphic configuration database businesses | |
| Xiong et al. | Data vitalization's perspective towards smart city: a reference model for data service oriented architecture | |
| CN110083529A (en) | Automated testing method, device, medium and electronic equipment | |
| CN111131449B (en) | Method for constructing service clustering framework of water resource management system | |
| WO2012062515A1 (en) | Method and system for visualising a system model | |
| Niu | Optimization of teaching management system based on association rules algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |