CN115586869A - Ad hoc network system and stream data processing method thereof - Google Patents
Ad hoc network system and stream data processing method thereof Download PDFInfo
- Publication number
- CN115586869A CN115586869A CN202211193073.8A CN202211193073A CN115586869A CN 115586869 A CN115586869 A CN 115586869A CN 202211193073 A CN202211193073 A CN 202211193073A CN 115586869 A CN115586869 A CN 115586869A
- Authority
- CN
- China
- Prior art keywords
- data
- read
- write
- hot
- cold
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 238000000034 method Methods 0.000 claims description 11
- 238000013500 data storage Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000004891 communication Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to an ad hoc network system and a stream data processing method thereof, belonging to the technical field of data processing; the problems that in the prior art, the hardware storage cost of an ad hoc network system is high, and the processing speed of the existing streaming data processing method is low are solved; the system comprises at least one data acquisition device, a client and a server; the system comprises at least one data acquisition device, a server and a data processing device, wherein the at least one data acquisition device is used for acquiring streaming data and sending a data writing request to the server through a wireless link; the client is used for sending a data reading request to the server; the server comprises a memory and a processor; the processor is used for respectively storing the received streaming data in the memory according to cold data and hot data when receiving a data writing request, and reading the data stored in the memory to the client according to a preset flow when receiving a data reading request; and the cold data and the hot data are obtained by counting the read-write frequency of the data stored in the memory within preset time.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to an ad hoc network system and a stream data processing method for the ad hoc network system.
Background
The self-organizing network is a network combining mobile communication and a computer network, and for a self-organizing network system applied to a battlefield environment, voices, messages, geographic positions and images collected by a sensor can be used as streaming data to be stored through carrying a backpack self-organizing network communication device by an individual soldier, additionally arranging a backpack self-organizing network communication device on a combat vehicle, additionally arranging an airborne self-organizing network communication device on an unmanned aerial vehicle or a helicopter and the like. The ad hoc network system in the battlefield environment needs to store or read data quickly, accurately and efficiently, due to the diversity of the ad hoc network devices, the number of the ad hoc network devices needs to be adjusted at any time according to the battlefield requirements, data stream information from different devices aiming at the same regional target needs to be received at the same time, and the stream data is a group of data sequences which arrive sequentially, massively, quickly and continuously, generally, the stream data can be regarded as a dynamic data set which grows infinitely along with the time duration. In order to realize rapid and accurate reading of information acquired by sensor acquisition equipment and meet the requirements of hardware storage cost of different users, it is increasingly important to classify and store stream data of an ad hoc network system and to plan a reading process comprehensively.
Disclosure of Invention
In view of the foregoing analysis, the present invention aims to provide an ad hoc network system and a stream data processing method of the ad hoc network system; the problems that in the prior art, the hardware storage cost of an ad hoc network system is high, and the processing speed of an existing streaming data processing method is low are solved.
The purpose of the invention is mainly realized by the following technical scheme:
in one aspect, the present invention provides an ad hoc network system, comprising at least one data acquisition device, a client and a server; wherein,
the at least one data acquisition device is used for acquiring streaming data and sending a data writing request to the server through a wireless link;
the client is used for sending a read data request to the server;
the server comprises a memory and a processor; the processor is used for respectively storing the received streaming data in the memory according to cold data and hot data when receiving the data writing request, and reading the data stored in the memory to a client according to a preset flow when receiving the data reading request; and the cold data and the hot data are obtained by counting the read-write frequency of the data stored in the memory within preset time.
Further, the memory comprises a preset memory space, an SSD hard disk and an HDD hard disk which are applied from the memory of the server; the preset memory space comprises a read-write temporary cache region, a read-write cold data temporary cache region and a read-write hot data temporary cache region; wherein,
the read-write temporary buffer area is used for storing data written by the data acquisition equipment;
the read-write cold data temporary cache region and the read-write hot data temporary cache region are respectively used for storing pre-identification cold data and pre-identification hot data obtained after pre-identification of cold data and hot data of the data in the read-write temporary cache region;
and the SSD hard disk and the HDD hard disk are respectively used for storing hot data and cold data obtained by performing cold and hot data identification on the pre-identified cold data and the pre-identified hot data.
Further, the processor stores the received streaming data in the memory according to the cold data and the hot data, respectively, and includes:
receiving a data writing request of at least one data acquisition device, and writing stream data requested to be written by the data writing request into a read-write temporary buffer area of the memory;
counting the read-write frequency of all data in a first time threshold value in the read-write temporary cache area, dividing the data into pre-identification cold data and pre-identification hot data based on a preset read-write frequency threshold value, and respectively storing the pre-identification cold data and the pre-identification hot data in the read-write cold data temporary cache area and the read-write hot data temporary cache area of the memory;
and counting the read-write frequency of all data in the read-write cold data temporary cache region and the read-write hot data temporary cache region within a second time threshold, dividing the data into cold data and hot data based on the read-write frequency threshold, and respectively storing the cold data and the hot data in the HDD hard disk and the SSD hard disk.
Further, the reading the data stored in the memory to the client according to the preset flow includes:
the processor receives a data reading request of a client;
on the basis of the set priority, sequentially inquiring whether the read-write temporary cache region, the read-write hot data temporary cache region and the read-write cold data temporary cache region have the data requested to be read by the read-data request;
if yes, reading the data requested to be read to the client;
and if the data does not exist, sequentially inquiring the data in the SDD hard disk and the HDD hard disk, writing the inquired data requested to be read into the data read-write cache region from the corresponding SSD hard disk or HDD hard disk through the page cache of the operating system of the server, and reading the data requested to be read to the client through the data read-write cache region.
Further, after the data is divided into cold data and hot data, the method further comprises the steps of calculating the storage probability of the hot data according to the read-write frequency of each hot data and the data amount in the temporary cache of the cold data and the hot data, storing the hot data with the storage probability larger than the threshold value in the SSD hard disk according to the storage probability threshold value, and storing the hot data and the cold data with the storage probability smaller than the threshold value in the HDD hard disk.
Further, the thermal data storage probability P is calculated by the following formula:
wherein, P e Probability of storage as thermal data e, t e The time is the read-write frequency of the hot data e in the preset time, w is the total read-write times of all the hot data in the preset time, R is the total data volume of the cold data and the hot data, and R is the data volume of the hot data.
Further, the first time threshold and the second time threshold are set to 12 hours and 24 hours, respectively.
On the other hand, the method for processing the stream data of the ad hoc network system is also disclosed, and comprises the following steps:
receiving a read-write data request through a server of the ad hoc network system;
when a data writing request is received, writing data requested to be written by the data writing request into a read-write temporary cache region of the server; respectively storing the data in a read-write hot data temporary cache region, a read-write cold data temporary cache region, an SSD hard disk and an HDD hard disk through the statistics of the read-write frequency of the data in a preset time and the calculation of the storage probability;
and when a data reading request is received, reading data through the read-write temporary cache, the read-write hot data temporary cache, the read-write cold data temporary cache, the SSD hard disk and the HDD hard disk of the server based on the set priority.
Further, the step of respectively storing the data in the read-write hot data temporary cache area, the read-write cold data temporary cache area, the SSD hard disk and the HDD hard disk by counting the read-write frequency and calculating the storage probability of the data in the read-write temporary cache area within a preset time includes:
counting the read-write frequency of all data in a first time threshold value in the read-write temporary cache area, dividing the data into pre-identification cold data and pre-identification hot data based on a preset read-write frequency threshold value, and respectively storing the pre-identification cold data and the pre-identification hot data in the read-write cold data temporary cache area and the read-write hot data temporary cache area;
and counting the read-write frequency of all data in a second time threshold in the read-write cold data temporary cache region and the read-write hot data temporary cache region to obtain cold data and hot data, calculating the storage probability of the hot data, and storing the data in the HDD hard disk and the SSD hard disk respectively based on the storage probability threshold.
Further, the thermal data storage probability is calculated by the following formula:
wherein, P e Probability of storage as thermal data e, t e The time is the read-write frequency of the hot data e in the preset time, w is the total read-write times of all the hot data in the preset time, R is the total data volume of the cold data and the hot data, and R is the data volume of the hot data.
The beneficial effects of the technical scheme are as follows:
1. the invention divides the flow data collected by the ad hoc network system into cold data and hot data for storage respectively, optimizes the data reading flow by the set priority, and improves the data reading response speed of the ad hoc network system.
2. According to the invention, cold data and hot data are accurately distinguished through twice recognition, and the decision is made on the data storage mode based on the storage probability, so that the cold data and the hot data are respectively and correspondingly stored according to the user requirements, the relation between the data reading speed and the storage cost is balanced, and the application range of the scheme is widened.
3. The invention respectively sets a read-write data cache region, a read-write cold data cache region and a read-write hot data cache region, which are respectively used for storing data acquired by sensor acquisition equipment and cold data and hot data acquired after cold and hot pre-identification; and when receiving a data reading request, reading the read-write data cache region, the read-write hot data cache region and the read-write cold data cache region in sequence according to the set priority and the possibility of the data which is requested to be read, thereby improving the data reading speed. And the reading times of the magnetic disk are reduced, the operating pressure of the magnetic disk is reduced, in addition, the effective reading times of the magnetic disk are improved, and the service life of the magnetic disk can be prolonged.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, wherein like reference numerals are used to designate like parts throughout.
FIG. 1 is a schematic diagram of an ad hoc network system according to an embodiment of the present invention;
Detailed Description
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and together with the description, serve to explain the principles of the invention and not to limit the scope of the invention.
An embodiment of the present invention discloses an ad hoc network system, as shown in fig. 1, the ad hoc network system includes: at least one data acquisition device, a client and a server; wherein,
the data acquisition equipment is used for acquiring the streaming data and sending a data writing request to the server through a wireless link; the data acquisition equipment can be sensor acquisition equipment of backpack type ad hoc network communication equipment, unmanned aerial vehicles, sensor acquisition equipment of onboard type ad hoc network communication equipment additionally arranged on a helicopter, and the like; the collected flow data may include voice, message, geographical location, image or video data, etc.
The client is used for sending a data reading request to the server; preferably, the client needs to perform key pairing with the server before communicating with the server, and the client can send a read data request to the server after successful pairing. In addition, the client can also be used for sending a control instruction to the server, and controlling the server to start or stop receiving the streaming data from the networking equipment; carrying out parameter configuration on the server; when the server fails, a corresponding scheme is started, or the process of receiving the streaming data by the server is directly repaired and controlled through the instruction information.
The server comprises a memory and a processor; the processor is used for respectively storing the received streaming data in the memory according to cold data and hot data when receiving a data writing request, and reading the data stored in the memory to the client according to a preset flow when receiving a data reading request; and the cold data and the hot data are obtained by counting the read-write frequency of the data stored in the memory within preset time.
Preferably, the storage in the server comprises a preset memory space, an SSD hard disk and an HDD hard disk, which are applied from the server memory; the preset memory space comprises a read-write temporary cache region, a read-write cold data temporary cache region and a read-write hot data temporary cache region; wherein,
the read-write temporary buffer area is used for storing data written by the data acquisition equipment; the method comprises the steps that the flow data acquired by the additionally-installed data acquisition equipment such as the unmanned aerial vehicle and the helicopter are firstly stored in a read-write temporary buffer area of a storage, a user can send a read data request through a client, and the stored data is read and checked through the read-write temporary buffer area.
The read-write cold data temporary cache region and the read-write hot data temporary cache region are respectively used for storing pre-identification cold data and pre-identification hot data obtained after pre-identification of cold data and hot data in the read-write temporary cache region; specifically, the embodiment sets the pre-recognition time to be 12 hours, and performs cold and hot pre-recognition on the data in the read-write data cache region every 12 hours, and stores the pre-recognized cold data and the pre-recognized hot data obtained by the pre-recognition in the read-write cold data temporary cache region and the read-write hot data temporary cache region, respectively.
The SSD hard disk and the HDD hard disk are respectively used for storing hot data and cold data obtained by performing cold and hot data identification on the pre-identified cold data and the pre-identified hot data.
More specifically, the processor stores the received streaming data in the memory according to the cold data and the hot data, respectively, and includes:
receiving a data writing request of at least one data acquisition device, and writing the stream data requested to be written in by the data writing request into a read-write temporary cache region of a memory;
counting the read-write frequency of all data in a first time threshold in the read-write temporary cache region, dividing the data into pre-identified cold data and pre-identified hot data based on a preset read-write frequency threshold, and respectively storing the pre-identified cold data and the pre-identified hot data in the read-write cold data temporary cache region and the read-write hot data temporary cache region of the memory; the first time threshold of the present embodiment is set to 12 hours.
And counting the read-write frequency of all data in the read-write cold data temporary cache region and the read-write hot data temporary cache region within a second time threshold, dividing the data into cold data and hot data based on the read-write frequency threshold, and storing the cold data and the hot data in the HDD hard disk and the SSD hard disk respectively. The second time threshold of the present embodiment is set to 24 hours.
The reading and writing frequency refers to the number of times of reading and writing data within a preset time, and is obtained through statistics of a reading and writing frequency statistical module of the processor.
As a specific embodiment, an individual soldier carries the backpack self-networking communication equipment, and a combat vehicle is additionally provided with the backpack self-networking communication equipment; an unmanned aerial vehicle and a helicopter store voices, messages (which can contain state data), geographic positions and images collected by a sensor as streaming data by additionally arranging an airborne ad hoc network communication device and a front finger by compiling a tactical cloud node device; according to the cold and hot data identification processing method, aiming at the cold and hot data identification processing of the streaming data acquired by the battlefield sensor, under the sensor data monitoring scene of battlefield equipment, a large amount of data acquired by the sensor is used as the streaming data, and the streaming processing method is used for data processing and analysis, so that the streaming processing process of the sensor data has the characteristics of low delay and high throughput, and the real-time performance of data processing is guaranteed. The ad hoc network equipment can adjust the number of the ad hoc network equipment at any time according to battlefield requirements in a battlefield environment, and can also receive streaming data information from different equipment in the same area. The data volume is huge, and the requirements on the real-time performance and the accuracy of the data are high. Based on the application scenario of the embodiment, after the data acquisition device acquires and stores the streaming data, the user usually reads and checks the stored data in real time, so that the embodiment stores the acquired streaming data in the read-write temporary buffer area, and stores the streaming data for 12 hours, and in a time period when the user has the maximum probability of reading new acquired data, the user can directly read data from the read-write temporary buffer area, thereby improving the reading speed and reducing the reading operation of the processor on a disk. After 12 hours and 24 hours, cold data and hot data are distinguished and stored twice through statistics of data read-write frequency, so that the accuracy of cold data and hot data identification and the data read speed are improved to a great extent.
It should be noted that the streaming data acquired by the data acquisition device of the ad hoc network system is a group of sequential, massive, fast, and continuous data sequences, and in general, the streaming data can be regarded as a dynamic data set that grows infinitely along with the time duration. The pre-identified cold data and hot data may have access frequency change and cold-heat exchange or may have hot data increase and cold data decrease or may have cold data increase and hot data decrease; in the embodiment, based on the characteristics that the probability that a user views streaming data within 12 hours is the maximum, and the probability that data with fewer viewing times (namely cold data) within 24 hours is read and viewed in subsequent processing is also lower, cold and hot data pre-recognition is performed within 12 hours, cold and hot data re-recognition is performed within 24 hours, and cold and hot data in existing data are timely distinguished by recognizing the cold and hot data twice, so that the data reading speed is ensured, and meanwhile, the lower real-time overhead of a system is realized. And through secondary identification, cold and hot data can be accurately distinguished, the accuracy of cold and hot data identification is improved, the data reading speed is improved, and the storage space is reasonably allocated.
Preferably, in order to increase the data reading speed, the process of reading the data stored in the memory to the client includes:
the processor receives a data reading request of a client; the read data request may be sent by the user through the client; preferably, after the user passes the identity authentication, the user sends a control instruction through the client, so that reading and classification processing of streaming data can be realized, classification results and stored branch information can be checked, or stored parameters can be adjusted.
And inquiring whether the read-write temporary cache region, the read-write hot data temporary cache region and the read-write cold data temporary cache region have the data requested to be read by the read-write request or not in sequence based on the set priority.
The set priority refers to that the processor preferentially inquires the read-write temporary cache region after receiving a read data request, and if the read-request data does not exist, the read-write hot data temporary cache region and the read-write cold data temporary cache region are sequentially inquired according to the priority. Through priority setting, the cache region with the highest probability of requesting to read data is inquired preferentially, the data requested to be read is read in the shortest time, and the reading efficiency is improved.
If yes, reading the data requested to be read to the client;
and if the data does not exist, sequentially inquiring the data in the SDD hard disk and the HDD hard disk, writing the inquired data requested to be read into a data read-write cache region from the corresponding SSD hard disk or HDD hard disk through an operating system page cache of the server, and reading the data requested to be read from the data read-write cache region to the client.
According to one embodiment of the invention, after the data is divided into cold data and hot data, the hot data storage probability can be calculated according to the read-write frequency of each hot data and the data amount in the temporary cache of the cold data and the hot data, the hot data with the storage probability larger than the threshold value is stored in the SSD hard disk according to the storage probability threshold value, and the hot data and the cold data with the storage probability smaller than the threshold value are stored in the HDD hard disk.
Preferably, the thermal data storage probability P is calculated by the following formula:
wherein, P e Probability of storage as thermal data e, t e The read-write frequency of the hot data e in the preset time, w is the total read-write times of all the hot data in the preset time, R is the total data volume of the cold data and the hot data, and R is the data volume of the hot data.
Specifically, for some users with limited hardware storage device cost, the cost of storing all data in the memory is too high, and since the HDD hard disk is low in cost but low in read-write speed and the SSD hard disk is high in cost but high in read-write speed, in order to meet the requirements of different users, the data stored in the HDD and the SSD hard disk are distinguished by the storage probability, so that selective storage of the data is realized, and the data reading speed is increased under the condition that the existing hardware resources are fully utilized.
One embodiment of the invention discloses a stream data processing method of an ad hoc network system, which comprises the following steps:
receiving a read-write data request through a server of the ad hoc network system;
when a data writing request is received, writing data requested to be written by the data writing request into a read-write temporary cache region of a server; through the statistics of the read-write frequency and the calculation of the storage probability of the data in the read-write temporary cache area within the preset time, the data are respectively stored in the read-write hot data temporary cache area, the read-write cold data temporary cache area, the SSD hard disk and the HDD hard disk, and the method specifically comprises the following steps:
counting the read-write frequency of all data in a first time threshold in the read-write temporary cache region, dividing the data into pre-identified cold data and pre-identified hot data based on a preset read-write frequency threshold, and respectively storing the pre-identified cold data and the pre-identified hot data in the read-write cold data temporary cache region and the read-write hot data temporary cache region;
and counting the read-write frequency of all data in a second time threshold in the read-write cold data temporary cache region and the read-write hot data temporary cache region to obtain cold data and hot data, calculating the storage probability of the hot data, and storing the data in the HDD hard disk and the SSD hard disk respectively based on the storage probability threshold.
Wherein, the thermal data storage probability is calculated by the following formula:
wherein, P e Probability of storage as thermal data e, t e The time is the read-write frequency of the hot data e in the preset time, w is the total read-write times of all the hot data in the preset time, R is the total data volume of the cold data and the hot data, and R is the data volume of the hot data.
When a data reading request is received, based on the set priority, data is read through a read-write temporary cache, a read-write hot data temporary cache, a read-write cold data temporary cache, an SSD hard disk and an HDD hard disk of the server.
In summary, in the ad hoc network system and the stream data processing method of the ad hoc network system provided by the embodiments of the present invention, the ad hoc network system of the present invention realizes temporary storage of data through a read-write temporary buffer area of a server, and is provided with a read-write cold data temporary buffer area and a read-write hot data temporary buffer area, and stores data pre-identified by cold data and hot data respectively by using the read-write cold data temporary buffer area and the read-write hot data temporary buffer area through secondary identification of the cold data and the hot data; the secondary identification is adopted to realize the accurate identification of the cold and hot data after the pre-identification, and the accurate writing of the subsequent cold and hot data is ensured. According to the invention, cold and hot data pre-identification operation is carried out on the data stored in the read-write temporary buffer area every 12 hours, and the data are respectively stored in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area, so that the space occupation of the read-write temporary buffer area is reduced, and the sufficient space for temporary storage of subsequent data is ensured.
The initial set value of the read-write cache cold and hot identification timing module is 24 hours, so that the cold and hot data accurate identification operation is carried out on the data stored in the read-write cold data temporary cache region and the read-write hot data temporary cache region every 24 hours, the data re-identified and defined as hot data and cold data is respectively written into the SSD hard disk module and the HDD hard disk module through the operating system page buffer module by combining the storage probability, the accurate writing of the cold and hot data is realized, the space occupation of the read-write cold data temporary cache region and the read-write hot data temporary cache region is reduced, the enough space for the subsequent data pre-identified by the cold and hot data to be temporarily stored is ensured, and the storage requirements of different users can be met.
Those skilled in the art will appreciate that all or part of the processes for implementing the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, for instructing the relevant hardware. The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.
Claims (10)
1. An ad hoc network system, comprising at least one data acquisition device, a client and a server; wherein,
the at least one data acquisition device is used for acquiring streaming data and sending a data writing request to the server through a wireless link;
the client is used for sending a read data request to the server;
the server comprises a memory and a processor; the processor is used for respectively storing the received streaming data in the memory according to cold data and hot data when receiving the data writing request, and reading the data stored in the memory to a client according to a preset flow when receiving the data reading request; and the cold data and the hot data are obtained by counting the read-write frequency of the data stored in the memory within preset time.
2. The ad hoc network system according to claim 1, wherein the storage includes a preset memory space, an SSD hard disk and an HDD hard disk, which are applied from a server memory; the preset memory space comprises a read-write temporary cache region, a read-write cold data temporary cache region and a read-write hot data temporary cache region; wherein,
the read-write temporary buffer area is used for storing data written by the data acquisition equipment;
the read-write cold data temporary cache region and the read-write hot data temporary cache region are respectively used for storing pre-identification cold data and pre-identification hot data obtained by pre-identifying cold data and hot data in the read-write temporary cache region;
and the SSD hard disk and the HDD hard disk are respectively used for storing hot data and cold data obtained by performing cold and hot data identification on the pre-identified cold data and the pre-identified hot data.
3. The ad hoc network system according to claim 1, wherein the processor stores the received streaming data in the memory as cold data and hot data, respectively, comprising:
receiving a data writing request of at least one data acquisition device, and writing stream data requested to be written by the data writing request into a read-write temporary buffer area of the memory;
counting the read-write frequency of all data in a first time threshold value in the read-write temporary cache area, dividing the data into pre-identification cold data and pre-identification hot data based on a preset read-write frequency threshold value, and respectively storing the pre-identification cold data and the pre-identification hot data in the read-write cold data temporary cache area and the read-write hot data temporary cache area of the memory;
and counting the read-write frequency of all data in the read-write cold data temporary cache region and the read-write hot data temporary cache region within a second time threshold, dividing the data into cold data and hot data based on the read-write frequency threshold, and respectively storing the cold data and the hot data in the HDD hard disk and the SSD hard disk.
4. The ad hoc network system according to claim 1, wherein the reading the data stored in the memory to the client according to the preset flow comprises:
the processor receives a data reading request of a client;
whether the data requested to be read by the read data request exists in the read-write temporary buffer area, the read-write hot data temporary buffer area and the read-write cold data temporary buffer area is sequentially inquired based on the set priority;
if yes, reading the data requested to be read to the client;
and if the data does not exist, sequentially inquiring the data in the SDD hard disk and the HDD hard disk, writing the inquired data requested to be read into the data read-write cache region from the corresponding SSD hard disk or HDD hard disk through the page cache of the operating system of the server, and reading the data requested to be read to the client through the data read-write cache region.
5. The ad-hoc network system according to claim 2, wherein after the data is divided into cold data and hot data, further comprising calculating a hot data storage probability according to a read-write frequency of each hot data and a data amount in the temporary cache of the cold and hot data, and storing the hot data having the storage probability greater than a threshold value in the SSD hard disk according to a storage probability threshold value, and storing the hot data and the cold data having the storage probability less than the threshold value in the HDD hard disk.
6. A method for reading and writing data according to claim 4, wherein the hot data storage probability P is calculated by the following formula:
wherein, P e Is the storage probability of the thermal data e, t e The time is the read-write frequency of the hot data e in the preset time, w is the total read-write times of all the hot data in the preset time, R is the total data volume of the cold data and the hot data, and R is the data volume of the hot data.
7. The ad-hoc network system according to claim 3, wherein the first time threshold and the second time threshold are set to 12 hours and 24 hours, respectively.
8. A stream data processing method of an ad hoc network system is characterized by comprising the following steps:
receiving a read-write data request through a server of the ad hoc network system;
when a data writing request is received, writing data requested to be written by the data writing request into a read-write temporary cache region of the server; respectively storing the data in a read-write hot data temporary cache region, a read-write cold data temporary cache region, an SSD hard disk and an HDD hard disk through the read-write frequency statistics and the storage probability calculation of the data in a preset time;
and when a data reading request is received, reading data through the read-write temporary cache, the read-write hot data temporary cache, the read-write cold data temporary cache, the SSD hard disk and the HDD hard disk of the server based on the set priority.
9. The stream data processing method according to claim 8, wherein the step of storing the data in the read-write hot data temporary buffer area, the read-write cold data temporary buffer area, the SSD hard disk, and the HDD hard disk respectively by performing statistics on the read-write frequency and calculation on the storage probability of the data within a preset time includes:
counting the read-write frequency of all data in a first time threshold value in the read-write temporary cache area, dividing the data into pre-identification cold data and pre-identification hot data based on a preset read-write frequency threshold value, and respectively storing the pre-identification cold data and the pre-identification hot data in the read-write cold data temporary cache area and the read-write hot data temporary cache area;
counting the read-write frequency of all data in a second time threshold in the read-write cold data temporary cache region and the read-write hot data temporary cache region to obtain cold data and hot data, calculating the storage probability of the hot data, storing the data with the storage probability smaller than the threshold and the cold data in the HDD hard disk based on the storage probability threshold, and storing the data with the storage probability larger than the threshold in the SSD hard disk.
10. The streaming data processing method according to claim 9, wherein the hot data storage probability is calculated by the following formula:
wherein, P e Is the storage probability of the thermal data e, t e The time is the read-write frequency of the hot data e in the preset time, w is the total read-write times of all the hot data in the preset time, R is the total data volume of the cold data and the hot data, and R is the data volume of the hot data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211193073.8A CN115586869B (en) | 2022-09-28 | 2022-09-28 | Ad hoc network system and stream data processing method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211193073.8A CN115586869B (en) | 2022-09-28 | 2022-09-28 | Ad hoc network system and stream data processing method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115586869A true CN115586869A (en) | 2023-01-10 |
| CN115586869B CN115586869B (en) | 2023-06-06 |
Family
ID=84778637
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211193073.8A Active CN115586869B (en) | 2022-09-28 | 2022-09-28 | Ad hoc network system and stream data processing method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115586869B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118467797A (en) * | 2024-07-12 | 2024-08-09 | 成都赛力斯科技有限公司 | Method, device, equipment and storage medium for storing and exporting vehicle-end data |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103984644A (en) * | 2014-05-07 | 2014-08-13 | 华为技术有限公司 | Data management method and device |
| CN105892947A (en) * | 2016-03-31 | 2016-08-24 | 华中科技大学 | SSD and HDD hybrid caching management method and system of energy-saving storage system |
| CN108073527A (en) * | 2016-11-07 | 2018-05-25 | 华为技术有限公司 | It is a kind of to cache the method and apparatus replaced |
| CN111159066A (en) * | 2020-01-07 | 2020-05-15 | 杭州电子科技大学 | Dynamically-adjusted cache data management and elimination method |
| CN111309646A (en) * | 2020-03-27 | 2020-06-19 | 西安紫光国芯半导体有限公司 | Storage device and data writing, reading, backup and recovery method thereof |
| CN111666260A (en) * | 2019-03-08 | 2020-09-15 | 杭州海康威视数字技术股份有限公司 | Data processing method and device |
| CN112860185A (en) * | 2021-01-29 | 2021-05-28 | 西藏宁算科技集团有限公司 | High-availability caching method based on LRU algorithm, storage device and electronic equipment |
| CN113672169A (en) * | 2021-07-19 | 2021-11-19 | 浙江大华技术股份有限公司 | Data reading and writing method of stream processing system and stream processing system |
-
2022
- 2022-09-28 CN CN202211193073.8A patent/CN115586869B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103984644A (en) * | 2014-05-07 | 2014-08-13 | 华为技术有限公司 | Data management method and device |
| CN105892947A (en) * | 2016-03-31 | 2016-08-24 | 华中科技大学 | SSD and HDD hybrid caching management method and system of energy-saving storage system |
| CN108073527A (en) * | 2016-11-07 | 2018-05-25 | 华为技术有限公司 | It is a kind of to cache the method and apparatus replaced |
| CN111666260A (en) * | 2019-03-08 | 2020-09-15 | 杭州海康威视数字技术股份有限公司 | Data processing method and device |
| CN111159066A (en) * | 2020-01-07 | 2020-05-15 | 杭州电子科技大学 | Dynamically-adjusted cache data management and elimination method |
| CN111309646A (en) * | 2020-03-27 | 2020-06-19 | 西安紫光国芯半导体有限公司 | Storage device and data writing, reading, backup and recovery method thereof |
| CN112860185A (en) * | 2021-01-29 | 2021-05-28 | 西藏宁算科技集团有限公司 | High-availability caching method based on LRU algorithm, storage device and electronic equipment |
| CN113672169A (en) * | 2021-07-19 | 2021-11-19 | 浙江大华技术股份有限公司 | Data reading and writing method of stream processing system and stream processing system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118467797A (en) * | 2024-07-12 | 2024-08-09 | 成都赛力斯科技有限公司 | Method, device, equipment and storage medium for storing and exporting vehicle-end data |
| CN118467797B (en) * | 2024-07-12 | 2024-09-20 | 成都赛力斯科技有限公司 | Method, device, equipment and storage medium for storing and exporting vehicle-end data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115586869B (en) | 2023-06-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TW202131661A (en) | Device and method for network optimization and non-transitory computer-readable medium | |
| CN110213627A (en) | Flow medium buffer distributor and its working method based on multiple cell user mobility | |
| CN107592656B (en) | Caching method based on base station clustering | |
| CN108093435B (en) | Cellular downlink network energy efficiency optimization system and method based on cached popular content | |
| JP2015526051A (en) | Terminal access method, system, and terminal | |
| CN105095451A (en) | Police unmanned aerial vehicle big data acquisition system and crime spatial database construction method | |
| CN105070017A (en) | Wireless communication method and system for unmanned aerial vehicle | |
| CN115586869A (en) | Ad hoc network system and stream data processing method thereof | |
| CN109802868B (en) | Mobile application real-time identification method based on cloud computing | |
| CN101917345A (en) | Method and device for controlling flows in wireless local area network | |
| CN110913239A (en) | Video cache updating method for refined mobile edge calculation | |
| CN112887943B (en) | Cache resource allocation method and system based on centrality | |
| US12078509B1 (en) | Location clustering and routing for 5G drive testing | |
| Li et al. | A smart content caching and replacement scheme for UAV-assisted fog computing network | |
| Wang et al. | Deep reinforcement learning for caching in d2d-enabled uav-relaying networks | |
| Gu et al. | Grouping-based consistency protocol design for end-edge-cloud hierarchical storage system | |
| CN109688213B (en) | Protocol stack data regulating system and method thereof | |
| WO2021057289A1 (en) | Abnormal terminal identifying method and apparatus, base station and storage medium | |
| KR101885144B1 (en) | Hybrid Content Caching Method and System | |
| CN117640554B (en) | Cache data scheduling method and system of road side equipment | |
| CN113630742B (en) | Mobile edge cache replacement method adopting request rate and dynamic property of information source issued content | |
| CN110501985A (en) | Signal transmission control method and device for regional control system and regional control system | |
| CN111124298A (en) | Mist computing network content cache replacement method based on value function | |
| WO2023169100A1 (en) | User clustering method and apparatus, and computer storage medium | |
| CN116390165B (en) | Dynamic management method and system for data-driven cellular base station |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |