CN115586869B - Ad hoc network system and stream data processing method thereof - Google Patents
Ad hoc network system and stream data processing method thereof Download PDFInfo
- Publication number
- CN115586869B CN115586869B CN202211193073.8A CN202211193073A CN115586869B CN 115586869 B CN115586869 B CN 115586869B CN 202211193073 A CN202211193073 A CN 202211193073A CN 115586869 B CN115586869 B CN 115586869B
- Authority
- CN
- China
- Prior art keywords
- data
- read
- write
- hot
- cold
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Abstract
The invention relates to an ad hoc network system and a stream data processing method of the ad hoc network system, belonging to the technical field of data processing; the method solves the problems of high hardware storage cost of the ad hoc network system and low processing speed of the existing stream data processing method in the prior art; the system comprises at least one data acquisition device, a client and a server; the data acquisition equipment is used for acquiring stream data and sending a data writing request to the server through a wireless link; the client is used for sending a reading data request to the server; the server includes a memory and a processor; the processor is used for respectively storing the received stream data in the memory according to the cold data and the hot data when the data writing request is received, and reading the data stored in the memory to the client according to a preset flow when the data reading request is received; the cold data and the hot data are obtained through statistics of read-write frequency of data stored in a memory in preset time.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to an ad hoc network system and a stream data processing method of the ad hoc network system.
Background
The ad hoc network is a network combining mobile communication and a computer network, and for an ad hoc network system applied in a battlefield environment, voice, messages, geographic positions and images acquired by a sensor can be stored as stream data by carrying a knapsack ad hoc network communication device by an individual soldier, installing the knapsack ad hoc network communication device on a combat vehicle, installing a loading type ad hoc network communication device on an unmanned plane or a helicopter, and the like. The ad hoc network system of the battlefield environment needs to store or read data quickly, accurately and efficiently, and due to the diversity of the ad hoc network devices, the number of the ad hoc network devices needs to be adjusted at any time according to the battlefield requirement, data stream information from different devices needs to be received simultaneously aiming at the same area target, and the stream data is a group of data sequences which arrive in sequence, in large quantity, quickly and continuously, and can be regarded as a dynamic data set which continues with time and grows infinitely in general. In order to realize rapid and accurate reading of information acquired by the sensor acquisition equipment, and simultaneously to meet the requirements of hardware storage cost of different users, it is increasingly important to store stream data of an ad hoc network system in a classified manner and to comprehensively plan a reading flow.
Disclosure of Invention
In view of the above analysis, the present invention is directed to an ad hoc network system and a stream data processing method of the ad hoc network system; the method solves the problems of high hardware storage cost of the ad hoc network system and low processing speed of the existing stream data processing method in the prior art.
The aim of the invention is mainly realized by the following technical scheme:
in one aspect, the present invention provides an ad hoc network system, the system comprising at least one data acquisition device, a client, and a server; wherein, the liquid crystal display device comprises a liquid crystal display device,
at least one data acquisition device is used for acquiring stream data and sending a data writing request to the server through a wireless link;
the client is used for sending a read data request to the server;
the server includes a memory and a processor; the processor is used for respectively storing the received stream data in the memory according to cold data and hot data when the data writing request is received, and reading the data stored in the memory to the client according to a preset flow when the data reading request is received; and the cold data and the hot data are obtained by counting the read-write frequency of the data stored in the memory within preset time.
Further, the storage comprises a preset memory space, an SSD hard disk and an HDD hard disk, which are applied from the memory of the server; the preset memory space comprises a read-write temporary buffer area, a read-write cold data temporary buffer area and a read-write hot data temporary buffer area; wherein, the liquid crystal display device comprises a liquid crystal display device,
the read-write temporary buffer area is used for storing data written in by the data acquisition equipment;
the temporary read-write cold data buffer area and the temporary read-write hot data buffer area are respectively used for storing pre-identification cold data and pre-identification hot data obtained after pre-identification of cold and hot data in the temporary read-write buffer area;
and the SSD hard disk and the HDD hard disk are respectively used for storing hot data and cold data obtained after cold and hot data identification is carried out on the pre-identification cold data and the pre-identification hot data.
Further, the processor stores the received stream data in the memory according to cold data and hot data, respectively, including:
receiving a data writing request of at least one data acquisition device, and writing stream data requested to be written by the data writing request into a read-write temporary buffer area of the memory;
counting the read-write frequency of all data in a first time threshold value in a read-write temporary buffer area, dividing the data into pre-identification cold data and pre-identification hot data based on a preset read-write frequency threshold value, and respectively storing the pre-identification cold data and the pre-identification hot data in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area of the memory;
and counting the read-write frequency of all data in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area in a second time threshold, dividing the data into cold data and hot data based on the read-write frequency threshold, and respectively storing the cold data and the hot data in an HDD hard disk and an SSD hard disk.
Further, the reading the data stored in the memory to the client according to the preset flow includes:
the processor receives a read data request of a client;
sequentially inquiring whether the read-write temporary buffer area, the read-write hot data temporary buffer area and the read-write cold data temporary buffer area have the data requested to be read by the read data request or not based on the set priority;
if yes, reading the data which is requested to be read to the client;
and if the data in the SDD hard disk and the HDD hard disk do not exist, sequentially inquiring the data in the SDD hard disk and the HDD hard disk, writing the inquired data which are requested to be read into the data read-write cache area from the corresponding SSD hard disk or HDD hard disk through the page cache of the operating system of the server, and reading the requested data to be read to the client through the data read-write cache area.
Further, after dividing the data into cold data and hot data, calculating the storage probability of the hot data according to the read-write frequency of each hot data and the data amount in the temporary cache of the cold data and the hot data, and storing the hot data with the storage probability larger than the threshold in the SSD hard disk according to the threshold of the storage probability, and storing the hot data and the cold data with the storage probability smaller than the threshold in the HDD hard disk.
Further, the thermal data storage probability P is calculated by the following formula:
wherein P is e Storage probability for thermal data e, t e For the read-write frequency of the hot data e in the preset time, w is the total read-write frequency of all the hot data in the preset time, R is the total data quantity of the cold and hot data, and R is the data quantity of the hot data.
Further, the first time threshold and the second time threshold are set to 12 hours and 24 hours, respectively.
On the other hand, the invention also discloses a stream data processing method of the ad hoc network system, which comprises the following steps:
receiving a read-write data request through a server of an ad hoc network system;
when a data writing request is received, writing the data which is requested to be written by the data writing request into a read-write temporary buffer area of the server; respectively storing the data in a read-write hot data temporary cache area, a read-write cold data temporary cache area, an SSD hard disk and an HDD hard disk through the statistics of the read-write frequency and the calculation of the storage probability of the data in a preset time;
when a data reading request is received, data is read through the read-write temporary buffer memory, the read-write thermal data temporary buffer memory, the read-write cold data temporary buffer memory, the SSD hard disk and the HDD hard disk of the server based on the set priority.
Further, the step of respectively storing the data in the temporary read-write hot data buffer area, the temporary read-write cold data buffer area, the SSD hard disk and the HDD hard disk by counting the read-write frequency and calculating the storage probability of the data in the temporary read-write buffer area within a preset time includes:
counting the read-write frequency of all data in a first time threshold value in the read-write temporary buffer area, dividing the data into pre-identification cold data and pre-identification hot data based on a preset read-write frequency threshold value, and respectively storing the pre-identification cold data and the pre-identification hot data in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area;
and counting the read-write frequency of all data in the second time threshold value in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area to obtain cold data and hot data, calculating the storage probability of the hot data, and respectively storing the data in the HDD hard disk and the SSD hard disk based on the storage probability threshold value.
Further, the thermal data storage probability is calculated by the following formula:
wherein P is e Storage probability for thermal data e, t e For the read-write frequency of the hot data e in the preset time, w is the total read-write frequency of all the hot data in the preset time, R is the total data quantity of the cold and hot data, and R is the data quantity of the hot data.
The beneficial effects of this technical scheme:
1. according to the invention, the stream data collected by the self-organizing network system is divided into the cold data and the hot data to be stored respectively, and the data reading flow is optimized through the set priority, so that the data reading response speed of the self-organizing network system is improved.
2. According to the method, the cold data and the hot data are accurately distinguished through twice identification, and the storage mode of the data is decided based on the storage probability, so that the cold data and the hot data are respectively and correspondingly stored according to the user demand, the relation between the data reading speed and the storage cost is balanced, and the application range of the scheme is improved.
3. The invention is provided with a read-write data buffer area, a read-write cold data buffer area and a read-write hot data buffer area respectively, which are used for storing the data acquired by the sensor acquisition equipment and cold data and hot data obtained after cold and hot pre-identification; when a data reading request is received, the data reading buffer area, the data reading hot data buffer area and the data reading cold data buffer area are sequentially read according to the set priority and the possibility of the data which is requested to be read, so that the data reading speed is improved. And the number of times of reading the magnetic disk is reduced, the operating pressure of the magnetic disk is reduced, in addition, the effective number of times of reading the magnetic disk is improved, and the service life of the magnetic disk can be prolonged.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, like reference numerals being used to refer to like parts throughout the several views.
Fig. 1 is a schematic diagram of an ad hoc network system according to an embodiment of the present invention;
Detailed Description
Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form a part hereof, and together with the description serve to explain the principles of the invention, and are not intended to limit the scope of the invention.
In one embodiment of the present invention, an ad hoc network system is disclosed, as shown in fig. 1, comprising: at least one data acquisition device, a client and a server; wherein, the liquid crystal display device comprises a liquid crystal display device,
the data acquisition equipment is used for acquiring stream data and sending a data writing request to the server through a wireless link; the data acquisition equipment can be sensor acquisition equipment of backpack ad hoc network communication equipment, unmanned aerial vehicle, sensor acquisition equipment of airborne ad hoc network communication equipment additionally arranged on a helicopter and the like; the stream data collected by the device can comprise voice, message, geographic position, image or video data and the like.
The client is used for sending a reading data request to the server; preferably, the client needs to be in key pairing with the server before communicating with the server, and after the pairing is successful, the client can send a read data request to the server. In addition, the client can also be used for sending a control instruction to the server, and controlling the server to start or stop receiving the stream data of the ad hoc network device; performing parameter configuration of a server; when the server fails, a corresponding scheme is started, or the process of receiving stream data by the server is directly repaired and controlled through instruction information.
The server includes a memory and a processor; the processor is used for respectively storing the received stream data in the memory according to the cold data and the hot data when the data writing request is received, and reading the data stored in the memory to the client according to a preset flow when the data reading request is received; the cold data and the hot data are obtained through statistics of read-write frequency of data stored in a memory in preset time.
Preferably, the memory in the server comprises a preset memory space, an SSD hard disk and an HDD hard disk, which are applied from the memory of the server; the preset memory space comprises a read-write temporary buffer area, a read-write cold data temporary buffer area and a read-write hot data temporary buffer area; wherein, the liquid crystal display device comprises a liquid crystal display device,
the temporary read-write buffer area is used for storing the data written in by the data acquisition equipment; the data acquisition device comprises a storage, a temporary buffer memory, a data acquisition device, a client, a data storage device and a data storage device.
The read-write cold data temporary buffer area and the read-write hot data temporary buffer area are respectively used for storing pre-identification cold data and pre-identification hot data obtained after cold and hot data pre-identification is carried out on the data in the read-write temporary buffer area; specifically, the pre-recognition time is set to be 12 hours, and each time 12 hours passes, cold and hot pre-recognition is performed on the data in the read-write data buffer area, and pre-recognition cold data and pre-recognition hot data obtained by pre-recognition are respectively stored in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area.
The SSD hard disk and the HDD hard disk are respectively used for storing hot data and cold data obtained after cold and hot data identification is carried out on the pre-identification cold data and the pre-identification hot data.
More specifically, the processor stores received stream data in the memory in terms of cold data and hot data, respectively, including:
receiving a data writing request of at least one data acquisition device, and writing stream data requested to be written by the data writing request into a read-write temporary buffer area of a memory;
counting the read-write frequency of all data in a first time threshold value in a read-write temporary buffer area, dividing the data into pre-identification cold data and pre-identification hot data based on a preset read-write frequency threshold value, and respectively storing the pre-identification cold data and the pre-identification hot data in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area of a memory; the first time threshold of this embodiment is set to 12 hours.
And counting the read-write frequency of all data in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area in a second time threshold, dividing the data into cold data and hot data based on the read-write frequency threshold, and respectively storing the cold data and the hot data in an HDD hard disk and an SSD hard disk. The second time threshold of this embodiment is set to 24 hours.
The read-write frequency refers to the number of times of reading and writing data in a preset time, and is obtained through statistics of a read-write frequency statistics module of the processor.
As a specific embodiment, the individual soldiers carry the backpack type self-networking communication equipment, and the combat vehicles are additionally provided with the backpack type self-networking communication equipment; the unmanned plane and the helicopter store the voice, the message (which can contain state data), the geographic position and the image acquired by the sensor as stream data through loading the loaded ad hoc network communication equipment and the forefinger through the martial cloud node equipment; according to the method, the cold data and the hot data of the stream data acquired by the battlefield sensor are identified, a large amount of data acquired by the sensor are used as the stream data in a sensor data monitoring scene of battlefield equipment, and the stream processing method is used for data processing and analysis, so that the stream processing process of the sensor data has the characteristics of low delay and high throughput, and the real-time performance of the data processing is ensured. The number of the ad hoc network devices can be adjusted at any time according to the battlefield requirements in the battlefield environment, and stream data information from different devices can be received for the same area at the same time. The data volume is huge, and the requirements on the real-time performance and the accuracy of the data are high. Based on the application scenario of the embodiment, after the data acquisition device acquires and stores the stream data, the user usually reads and views the stored data in real time, so that the embodiment stores the acquired stream data in the temporary read-write buffer area and stores the stream data for 12 hours, and in the period of time when the user has the maximum probability of reading new acquired data, the user can directly read the data from the temporary read-write buffer area, thereby improving the reading speed and reducing the reading operation of the processor on the magnetic disk. After 12 hours and 24 hours, the data are distinguished and stored respectively by carrying out statistics on the data read-write frequency, so that the accuracy of cold and hot data identification and the speed of data reading are greatly improved.
It should be noted that, the stream data acquired by the data acquisition device of the ad hoc network system is a group of sequential, massive, rapid and continuous arriving data sequences, and in general, the stream data can be regarded as a dynamic data set which continues with time and grows infinitely. Because the pre-identified cold and hot data can change the access frequency, cold and hot alternation can occur, or hot data increase, cold data decrease, cold data increase and hot data decrease can occur; based on the characteristics that the probability of viewing stream data within 12 hours is maximum and the probability of reading and viewing data (namely cold data) with fewer times within 24 hours in subsequent processing is also smaller, the embodiment is based on the characteristics that the 12-hour pre-recognition of cold and hot data and the 24-hour re-recognition of cold and hot data are set, and the cold and hot data in the existing data are timely distinguished through twice-recognition of the cold and hot data, so that the lower real-time overhead of the system is realized while the data reading speed is ensured. And through the secondary identification, cold and hot data can be accurately distinguished, the accuracy of cold and hot data identification is improved, the data reading speed is improved, and the storage space is reasonably distributed.
Preferably, in order to increase the data reading speed, the process of reading the data stored in the memory to the client according to the present embodiment includes:
the processor receives a read data request of the client; the read data request may be sent by a user through a client; preferably, after the user passes the identity verification, a control instruction is sent out by the client, so that the reading and classification processing of the stream data can be realized, and the classification result and the stored subsection information can be checked or the stored parameters can be adjusted.
And based on the set priority, sequentially inquiring whether the read-write temporary buffer area, the read-write hot data temporary buffer area and the read-write cold data temporary buffer area have the data requested to be read by the read data request.
The set priority refers to that after the processor receives a read data request, the processor queries the read-write temporary buffer area preferentially, and if the read data request does not exist, the processor queries the read-write hot data temporary buffer area and the read-write cold data temporary buffer area sequentially according to the priority order. By setting the priority, the buffer area with the highest probability of requesting to read the data is queried preferentially, so that the data requested to be read is read in the shortest time, and the reading efficiency is improved.
If yes, reading the data read by the request to the client;
if the data does not exist, sequentially inquiring the data in the SDD hard disk and the HDD hard disk, writing the inquired data which are requested to be read into the data read-write cache area from the corresponding SSD hard disk or HDD hard disk through the page cache of the operating system of the server, and reading the requested data to the client through the data read-write cache area.
According to one embodiment of the invention, after data is divided into cold data and hot data, the storage probability of the hot data can be calculated according to the read-write frequency of each hot data and the data quantity in the temporary cache of the cold and hot data, the hot data with the storage probability larger than the threshold value is stored in an SSD hard disk according to the threshold value of the storage probability, and the hot data and the cold data with the storage probability smaller than the threshold value are stored in the HDD hard disk.
Preferably, the thermal data storage probability P is calculated by the following formula:
wherein P is e Storage probability for thermal data e, t e For the read-write frequency of the hot data e in the preset time, w is the total read-write frequency of all the hot data in the preset time, R is the total data quantity of the cold and hot data, and R is the data quantity of the hot data.
Specifically, for some users with limited cost of storage hardware equipment, the cost of storing all data in the memory is too high, the cost of the HDD hard disk is low, the read-write speed is low, the cost of the SSD hard disk is high, but the read-write speed is high, in order to adapt to the requirements of different users, the data stored in the HDD and the SSD hard disk are distinguished through the storage probability, the selective storage of the data is realized, and the data reading speed is improved under the condition that the existing hardware resources are fully utilized.
The invention discloses a stream data processing method of an ad hoc network system, which comprises the following steps:
receiving a read-write data request through a server of an ad hoc network system;
when a data writing request is received, writing the data which is requested to be written by the data writing request into a read-write temporary buffer area of a server; the method comprises the steps of respectively storing data in a read-write hot data temporary cache area, a read-write cold data temporary cache area, an SSD hard disk and an HDD hard disk by counting the read-write frequency and calculating the storage probability of the data in the read-write temporary cache area in a preset time, and specifically comprises the following steps:
counting the read-write frequency of all data in a first time threshold value in a read-write temporary buffer area, dividing the data into pre-identification cold data and pre-identification hot data based on a preset read-write frequency threshold value, and respectively storing the pre-identification cold data and the pre-identification hot data in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area;
and counting the read-write frequency of all data in the second time threshold value in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area to obtain cold data and hot data, calculating the storage probability of the hot data, and respectively storing the data in the HDD hard disk and the SSD hard disk based on the storage probability threshold value.
Wherein the thermal data storage probability is calculated by the following formula:
wherein P is e Storage probability for thermal data e, t e For the read-write frequency of the hot data e in the preset time, w is the total read-write frequency of all the hot data in the preset time, R is the total data quantity of the cold and hot data, and R is the data quantity of the hot data.
When a data reading request is received, data is read through the read-write temporary buffer memory, the read-write thermal data temporary buffer memory, the read-write cold data temporary buffer memory, the SSD hard disk and the HDD hard disk of the server based on the set priority.
In summary, according to the ad hoc network system and the stream data processing method thereof provided by the embodiments of the present invention, the ad hoc network system realizes temporary storage of data through the read-write temporary buffer area of the server, and is provided with the read-write cold data temporary buffer area and the read-write hot data temporary buffer area, and the data pre-identified by the cold data and the hot data are respectively stored by the read-write cold data temporary buffer area and the read-write hot data temporary buffer area through secondary identification of the cold data and the hot data; the data after the pre-identification of the cold and hot data is accurately identified again through the secondary identification, so that the accurate writing of the subsequent cold and hot data is ensured. The invention performs the pre-identification operation of the cold and hot data stored in the temporary read-write buffer area every 12 hours, and stores the data into the temporary read-write cold data buffer area and the temporary read-write hot data buffer area respectively, thereby reducing the space occupation of the temporary read-write buffer area and ensuring that the temporary read-write temporary buffer area has enough space for the temporary storage of the subsequent data.
The initial setting value of the read-write cache cold-hot identification timing module is 24 hours, so that the data stored in the read-write cold data temporary cache area and the read-write hot data temporary cache area are accurately identified every 24 hours, the data which are re-identified as hot data and cold data are respectively written into the SSD hard disk module and the HDD hard disk module through the operating system page buffer module by combining the storage probability, the accurate writing of the cold data and the hot data is realized, the space occupation of the read-write cold data temporary cache area and the read-write hot data temporary cache area is reduced, the sufficient space for the subsequent temporary storage of the data which is pre-identified by the cold data and the hot data is ensured, and the storage requirements of different users can be met.
Those skilled in the art will appreciate that all or part of the flow of the methods of the embodiments described above may be accomplished by way of a computer program to instruct associated hardware, where the program may be stored on a computer readable storage medium. Wherein the computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory, etc.
The present invention is not limited to the above-mentioned embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the technical scope of the present invention are intended to be included in the scope of the present invention.
Claims (5)
1. An ad hoc network system, wherein the system comprises at least one data acquisition device, a client and a server; wherein, the liquid crystal display device comprises a liquid crystal display device,
at least one data acquisition device is used for acquiring stream data and sending a data writing request to the server through a wireless link;
the client is used for sending a reading data request to the server;
the server includes a memory and a processor; the storage comprises a preset memory space, an SSD hard disk and an HDD hard disk, wherein the preset memory space, the SSD hard disk and the HDD hard disk are applied from a server memory; the preset memory space comprises a read-write temporary buffer area, a read-write cold data temporary buffer area and a read-write hot data temporary buffer area; the processor is configured to store, when the write data request is received, the received stream data in the memory according to cold data and hot data, respectively, and includes:
receiving a data writing request of at least one data acquisition device, and writing stream data requested to be written by the data writing request into a read-write temporary buffer area of the memory;
counting the read-write frequency of all data in a first time threshold value in a read-write temporary buffer area, dividing the data into pre-identification cold data and pre-identification hot data based on a preset read-write frequency threshold value, and respectively storing the pre-identification cold data and the pre-identification hot data in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area of the memory;
counting the read-write frequency of all data in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area in a second time threshold, and dividing the data into cold data and hot data based on the read-write frequency threshold; after dividing the data into cold data and hot data, calculating the storage probability of the hot data according to the read-write frequency of each hot data and the data quantity in the temporary cache of the cold and hot data, and storing the hot data with the storage probability larger than the threshold in an SSD hard disk according to the threshold of the storage probability, wherein the hot data and the cold data with the storage probability smaller than the threshold in the HDD hard disk; the thermal data storage probability P is calculated by the following formula:
wherein P is e Storage probability for thermal data e, t e For the read-write frequency of the hot data e in the preset time, w is the total read-write times of all the hot data in the preset time, R is the total data quantity of the cold and hot data, and R is the data quantity of the hot data; and when the data reading request is received, reading the data stored in the memory to the client according to a preset flow.
2. The ad hoc network system according to claim 1, wherein said reading the data stored in the memory to the client according to the preset flow includes:
the processor receives a read data request of a client;
sequentially inquiring whether the read-write temporary buffer area, the read-write hot data temporary buffer area and the read-write cold data temporary buffer area have the data requested to be read by the read data request or not based on the set priority;
if yes, reading the data which is requested to be read to the client;
and if the data in the SSD hard disk and the HDD hard disk do not exist, sequentially inquiring the data in the SSD hard disk and the HDD hard disk, writing the inquired data which are requested to be read into the data read-write cache area from the corresponding SSD hard disk or HDD hard disk through the page cache of the operating system of the server, and reading the requested data to be read to the client through the data read-write cache area.
3. The ad hoc network system of claim 1, wherein said first and second time thresholds are set to 12 hours and 24 hours, respectively.
4. A stream data processing method of an ad hoc network system, comprising the steps of:
receiving a read-write data request through a server of an ad hoc network system;
when a data writing request is received, writing the data which is requested to be written by the data writing request into a read-write temporary buffer area of the server; respectively storing the data in a read-write hot data temporary cache area, a read-write cold data temporary cache area, an SSD hard disk and an HDD hard disk through the statistics of the read-write frequency and the calculation of the storage probability of the data in a preset time; the thermal data storage probability is calculated by the following formula:
wherein P is e Storage probability for thermal data e, t e For the read-write frequency of the hot data e in the preset time, w is the total read-write times of all the hot data in the preset time, R is the total data quantity of the cold and hot data, and R is the data quantity of the hot data;
when a data reading request is received, data is read through the read-write temporary buffer memory, the read-write thermal data temporary buffer memory, the read-write cold data temporary buffer memory, the SSD hard disk and the HDD hard disk of the server based on the set priority.
5. The stream data processing method according to claim 4, wherein the storing the data in the read-write hot data temporary buffer area, the read-write cold data temporary buffer area, the SSD hard disk and the HDD hard disk respectively by counting the read-write frequency and calculating the storage probability of the data in a preset time, comprises:
counting the read-write frequency of all data in a first time threshold value in the read-write temporary buffer area, dividing the data into pre-identification cold data and pre-identification hot data based on a preset read-write frequency threshold value, and respectively storing the pre-identification cold data and the pre-identification hot data in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area;
and counting the read-write frequency of all data in the second time threshold value in the read-write cold data temporary buffer area and the read-write hot data temporary buffer area to obtain cold data and hot data, calculating the storage probability of the hot data, storing the data with the storage probability smaller than the threshold value and the cold data in the HDD hard disk based on the storage probability threshold value, and storing the data with the storage probability larger than the threshold value in the SSD hard disk.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211193073.8A CN115586869B (en) | 2022-09-28 | 2022-09-28 | Ad hoc network system and stream data processing method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211193073.8A CN115586869B (en) | 2022-09-28 | 2022-09-28 | Ad hoc network system and stream data processing method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115586869A CN115586869A (en) | 2023-01-10 |
| CN115586869B true CN115586869B (en) | 2023-06-06 |
Family
ID=84778637
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211193073.8A Active CN115586869B (en) | 2022-09-28 | 2022-09-28 | Ad hoc network system and stream data processing method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115586869B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105892947A (en) * | 2016-03-31 | 2016-08-24 | 华中科技大学 | SSD and HDD hybrid caching management method and system of energy-saving storage system |
| CN108073527A (en) * | 2016-11-07 | 2018-05-25 | 华为技术有限公司 | It is a kind of to cache the method and apparatus replaced |
| CN111309646A (en) * | 2020-03-27 | 2020-06-19 | 西安紫光国芯半导体有限公司 | Storage device and data writing, reading, backup and recovery method thereof |
| CN112860185A (en) * | 2021-01-29 | 2021-05-28 | 西藏宁算科技集团有限公司 | High-availability caching method based on LRU algorithm, storage device and electronic equipment |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103984644B (en) * | 2014-05-07 | 2017-12-05 | 华为技术有限公司 | A kind of data managing method and device |
| CN111666260A (en) * | 2019-03-08 | 2020-09-15 | 杭州海康威视数字技术股份有限公司 | Data processing method and device |
| CN111159066A (en) * | 2020-01-07 | 2020-05-15 | 杭州电子科技大学 | Dynamically-adjusted cache data management and elimination method |
| CN113672169A (en) * | 2021-07-19 | 2021-11-19 | 浙江大华技术股份有限公司 | Data reading and writing method of stream processing system and stream processing system |
-
2022
- 2022-09-28 CN CN202211193073.8A patent/CN115586869B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105892947A (en) * | 2016-03-31 | 2016-08-24 | 华中科技大学 | SSD and HDD hybrid caching management method and system of energy-saving storage system |
| CN108073527A (en) * | 2016-11-07 | 2018-05-25 | 华为技术有限公司 | It is a kind of to cache the method and apparatus replaced |
| CN111309646A (en) * | 2020-03-27 | 2020-06-19 | 西安紫光国芯半导体有限公司 | Storage device and data writing, reading, backup and recovery method thereof |
| CN112860185A (en) * | 2021-01-29 | 2021-05-28 | 西藏宁算科技集团有限公司 | High-availability caching method based on LRU algorithm, storage device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115586869A (en) | 2023-01-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TW202131661A (en) | Device and method for network optimization and non-transitory computer-readable medium | |
| TWI400617B (en) | Schedule and data caching for wireless transmission | |
| CN105338297B (en) | A kind of storage of video data and playback system, device and method | |
| CN111427844B (en) | Data migration system and method for file hierarchical storage | |
| CN107204198B (en) | Control method and device for high-speed access double-rate synchronous dynamic random access memory | |
| CN103530388A (en) | Performance improving data processing method in cloud storage system | |
| CN104536701A (en) | Realizing method and system for NVME protocol multi-command queues | |
| CN103581242A (en) | Method, client-side equipment, server and system for uploading files | |
| CN103491152A (en) | Metadata obtaining method, device and system in distributed file system | |
| CN104092670A (en) | Method for utilizing network cache server to process files and device for processing cache files | |
| CN104811646B (en) | The storage method of the modulation of multiple video strems Data Concurrent and buffering based on Coutinuous store model | |
| CN109348454A (en) | A kind of D2D Cache Communication content sharing method | |
| CN112417350A (en) | Data storage adjusting method and device and computer equipment | |
| CN115586869B (en) | Ad hoc network system and stream data processing method thereof | |
| US8549274B2 (en) | Distributive cache accessing device and method for accelerating to boot remote diskless computers | |
| US10003551B2 (en) | Packet memory system, method and device for preventing underrun | |
| CN104424189A (en) | Positioning resolving method and positioning resolving system based on cloud platform | |
| CN106453663B (en) | Improved storage expansion method and device based on cloud service | |
| CN105227665A (en) | A kind of caching replacement method for cache node | |
| US9317678B2 (en) | System and method for managing logins in a network interface | |
| US9762696B2 (en) | Data storage communication apparatus, and data transmission and management methods using the same | |
| CN109240574B (en) | Method and system for prefetching files in mobile internet based on operation heat | |
| US20090132765A1 (en) | Dual controller storage apparatus and cache memory mirror method thereof | |
| US10009265B2 (en) | Communication control apparatus, communication control method, communication system, and recording medium | |
| CN115242729B (en) | Cache query system based on multiple priorities |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |