CN115086301B - Data analysis system and method for compression uploading equalization - Google Patents
Data analysis system and method for compression uploading equalization Download PDFInfo
- Publication number
- CN115086301B CN115086301B CN202210690330.2A CN202210690330A CN115086301B CN 115086301 B CN115086301 B CN 115086301B CN 202210690330 A CN202210690330 A CN 202210690330A CN 115086301 B CN115086301 B CN 115086301B
- Authority
- CN
- China
- Prior art keywords
- compression
- uploading
- file
- compressed
- processing device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007906 compression Methods 0.000 title claims abstract description 261
- 230000006835 compression Effects 0.000 title claims abstract description 261
- 238000007405 data analysis Methods 0.000 title claims abstract description 48
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000012545 processing Methods 0.000 claims abstract description 94
- 230000007246 mechanism Effects 0.000 claims abstract description 74
- 238000012360 testing method Methods 0.000 claims abstract description 55
- 238000006243 chemical reaction Methods 0.000 claims abstract description 24
- 238000002360 preparation method Methods 0.000 claims abstract description 24
- 238000013144 data compression Methods 0.000 claims abstract description 12
- 238000004458 analytical method Methods 0.000 claims description 84
- 238000000605 extraction Methods 0.000 claims description 20
- 230000009471 action Effects 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 10
- 230000008094 contradictory effect Effects 0.000 description 4
- 230000009467 reduction Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/38—Flow control; Congestion control by adapting coding or compression rate
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The invention relates to a data analysis system for compression uploading equalization, which comprises: the test preparation mechanism is used for acquiring the file data volume of the file for test and setting the compression uploading duration, and the value of the set compression uploading duration is obtained by dividing the file data volume by the lowest compression uploading data volume threshold of the unit duration; the number conversion device is used for obtaining an effective compression mode corresponding to the current file to be compressed and uploaded; a compression processing means for performing compression processing with an effective compression mode; the minimum compression uploading data volume threshold value of the unit time length is the minimum data volume which is allowed by a data operator and is compressed and uploaded per unit time length. The invention also relates to a data analysis method for the compression uploading equalization. By the method and the device, the compression mode meeting the requirement of a data operator for acquiring the data speed can be analyzed for each data file based on the performance of the data compression equipment, the network speed of the data uploading equipment and the data volume of the data file.
Description
Technical Field
The invention relates to the field of data analysis, in particular to a data analysis system and method for compression uploading equalization.
Background
Offline data analysis is used for more complex and time-consuming data analysis and processing, and is generally built on a cloud computing platform, such as an open-source HDFS file system and a MapReduce operation framework. The Hadoop cluster comprises hundreds or even thousands of servers, stores PB or even tens of PB data, runs thousands of offline data analysis jobs every day, processes hundreds of MB to hundreds of TB or even more data for each job, and has a running time of several minutes, hours, days or even longer.
Online data analysis, also known as online analytical processing, is used to process a user's online requests and has a relatively high demand for response time (typically no more than a few seconds). In contrast to offline data analysis, online data analysis can process a user's request in real time, allowing the user to change the constraints and limitations of the analysis at any time. Online data analysis can handle much smaller amounts of data than offline data analysis, but with advances in technology, current online analysis systems have been able to handle tens or even hundreds of millions of records in real time. The traditional online data analysis system is built on a data warehouse taking a relational database as a core, and the online big data analysis system is built on a NoSQL system of a cloud computing platform. If online analysis and processing of big data are not available, huge internet web pages cannot be stored and indexed, so that an existing efficient search engine cannot be provided, and the vigorous development of microblogs, blogs, social networks and the like built on the basis of big data processing cannot be realized.
At present, for a data file to be compressed and uploaded to a data operator server, a compression mode directly determines a compression time length and an uploading time length, however, the reduction of the compression time length represents the reduction of a compression ratio, which leads to the rise of uploaded data, and further causes the rise of the uploading time length, therefore, it can be seen that the compression time length and the uploading time length are contradictory, and a data operator only pays attention to the total time length of the two, therefore, a deep data analysis mechanism is needed to realize the relation discussion of the compression time length and the uploading time length so as to analyze the compression mode meeting the requirement of the data operator for obtaining the data speed.
Disclosure of Invention
In order to solve the technical problems in the related field, the invention provides a data analysis system and method for compression uploading balance, which can analyze a compression mode meeting the requirement of a data operator on data acquisition speed for each data file based on the performance of data compression equipment, the network speed of the data uploading equipment and the data volume of the data file, thereby realizing the dynamic balance of the original contradictory compression time length and uploading time length.
According to an aspect of the present invention, there is provided a data analysis system for compressive upload equalization, the system comprising:
the test preparation mechanism is used for acquiring the file data volume of each test file and setting the compression uploading duration, and the value of the set compression uploading duration is obtained by dividing the file data volume by the lowest compression uploading data volume threshold of unit duration;
the type extraction mechanism is connected with the test preparation mechanism and is used for compressing and uploading compressed files by adopting different types of data compression modes aiming at each test file, counting the total time consumed by completing compression operation and uploading operation by adopting each type of compression mode to be used as the reference total time corresponding to the type of compression mode, and using the compression mode of the type corresponding to the reference total time which is less than the set compression uploading time as an effective compression mode and outputting a number corresponding to the effective compression mode;
the model reconstruction mechanism is connected with the type extraction mechanism and is used for executing a learning action on the intelligent analysis model which executes the compression mode analysis by adopting each test file so as to complete the reconstruction of the intelligent analysis model;
the number conversion device is connected with the model reconstruction mechanism and used for taking the file data volume of the current file to be compressed and uploaded, the unit time calculation amount of a processor chip of the compression processing device executing the compression operation and the current uploading rate of the uploading processing device executing the uploading operation as the input of the intelligent analysis model after repeated reconstruction, and operating the intelligent analysis model after repeated reconstruction to obtain the number corresponding to the effective compression mode corresponding to the current file to be compressed and uploaded;
the compression processing device is connected with the number conversion device and is used for executing compression processing on the current file to be compressed and uploaded by adopting an effective compression mode corresponding to the number output by the number conversion device so as to obtain a compressed file corresponding to the current file to be compressed and uploaded;
wherein, the value of the set compressed uploading time length obtained by dividing the file data amount by the lowest compressed uploading data amount threshold value of the unit time length comprises: the minimum compression uploading data volume threshold value per unit time length is the minimum data volume which is allowed by a data operator and is compressed and uploaded per unit time length.
According to another aspect of the present invention, there is also provided a data analysis method for compression upload equalization, the method including:
the device comprises a test preparation mechanism, a compression uploading mechanism and a compression uploading mechanism, wherein the test preparation mechanism is used for acquiring the file data volume of each test file and setting the compression uploading duration, and the value of the set compression uploading duration is obtained by dividing the file data volume by the lowest compression uploading data volume threshold of unit duration;
the usage type extraction mechanism is connected with the test preparation mechanism and used for compressing and uploading compressed files by adopting different types of data compression modes aiming at each test file, counting the total time consumed by completing compression operation and uploading operation by adopting each type of compression mode as the reference total time corresponding to the type of compression mode, and taking the compression mode of the type corresponding to the reference total time which is less than the set compression uploading time as an effective compression mode and outputting a number corresponding to the effective compression mode;
the model reconstruction mechanism is connected with the type extraction mechanism and used for executing a learning action on the intelligent analysis model executing the compression mode analysis by adopting each test file so as to complete the reconstruction of the intelligent analysis model;
the number conversion device is connected with the model reconstruction mechanism and used for taking the file data volume of the current file to be compressed and uploaded, the unit time calculation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of the intelligent analysis model after repeated reconstruction, and the intelligent analysis model after repeated reconstruction is operated to obtain the number corresponding to the effective compression mode corresponding to the current file to be compressed and uploaded;
the number conversion device is used for outputting an effective compression mode corresponding to the number to be compressed to the current file to be uploaded;
wherein, the value of the set compressed uploading time length obtained by dividing the file data amount by the lowest compressed uploading data amount threshold value of the unit time length comprises: the minimum compression uploading data volume threshold value of the unit time length is the minimum data volume which is allowed by a data operator and is compressed and uploaded per unit time length.
Therefore, the invention has at least two prominent substantive characteristics: firstly, intelligently analyzing a compression coding type meeting the speed requirement of the minimum data volume compressed and uploaded in each unit time length allowed by a data operator based on the file data volume of a current file to be compressed and uploaded, the unit time operation amount of a processor chip of a compression processing device executing compression operation and the current uploading speed of an uploading processing device executing uploading operation, so as to achieve dynamic balance between the consumed time lengths of data compression and data uploading and further ensure the total compression uploading time; and secondly, executing a learning action on the intelligent analysis model executing the compression mode analysis by adopting each test file to complete one reconstruction of the intelligent analysis model, and realizing multiple reconstruction of the intelligent analysis model to be used, wherein the reconstruction times are reversely associated with the value of the lowest compression uploading data volume threshold value in unit time length, so that the precision of the intelligent analysis result is improved.
Drawings
Embodiments of the invention will now be described with reference to the accompanying drawings, in which:
fig. 1 is a block diagram illustrating a data analysis system for compression upload equalization according to an embodiment of the present invention.
Fig. 2 is a flow chart illustrating steps of a data analysis method for compression upload equalization according to an embodiment of the present invention.
Detailed Description
An embodiment of the data analysis method for compression upload equalization of the present invention will be described in detail below with reference to the accompanying drawings.
A compression coding technique of multimedia data is developed on the basis of the c.e. shannon information theory. Coding methods can be divided into three categories: (1) According to the statistical characteristics of the information source, methods (first generation coding methods) such as predictive coding, transform coding, vector quantization coding, subband coding, neural network coding, and the like are employed. (2) According to the visual characteristics of human eyes, methods (second generation coding methods) such as image coding based on directional filtering, image contour-ethic coding based on image contour, coding based on wavelet analysis and the like are adopted. (3) according to the transferred scene features: fractal coding, model-based coding, and the like (second generation coding methods) are employed.
At present, for a data file to be compressed and uploaded to a data operator server, a compression mode directly determines a compression time length and an uploading time length, however, the reduction of the compression time length represents the reduction of a compression ratio, which leads to the rise of uploaded data, and further causes the rise of the uploading time length, therefore, it can be seen that the compression time length and the uploading time length are contradictory, and a data operator only pays attention to the total time length of the two, therefore, a deep data analysis mechanism is needed to realize the relation discussion of the compression time length and the uploading time length so as to analyze the compression mode meeting the requirement of the data operator for obtaining the data speed.
In order to overcome the defects, the invention builds a data analysis system and a data analysis method for compression uploading balance, and can effectively solve the corresponding technical problems.
The invention has at least the following two prominent substantive characteristics:
firstly, intelligently analyzing the compression coding type meeting the speed requirement of minimum data volume compression and uploading per unit time allowed by a data operator based on the file data volume of a current file to be compressed and uploaded, the unit time operation amount of a processor chip of a compression processing device executing compression operation and the current uploading speed of an uploading processing device executing uploading operation so as to achieve dynamic balance between the consumed time of data compression and data uploading and further ensure the total compression uploading time;
and secondly, executing a learning action on the intelligent analysis model executing the compression mode analysis by adopting each test file to complete one reconstruction of the intelligent analysis model, and realizing multiple reconstruction of the intelligent analysis model to be used, wherein the reconstruction times are reversely associated with the value of the lowest compression upload data volume threshold value in unit time length, so that the precision of the intelligent analysis result is improved.
Fig. 1 is a block diagram illustrating a data analysis system for compressed upload equalization according to an embodiment of the present invention, the system including:
the test preparation mechanism is used for acquiring the file data volume of each test file and setting the compression uploading time length, and the value of the set compression uploading time length is obtained by dividing the file data volume by the lowest compression uploading data volume threshold of the unit time length;
the type extraction mechanism is connected with the test preparation mechanism and is used for compressing and uploading compressed files by adopting different types of data compression modes aiming at each test file, counting the total time consumed by completing compression operation and uploading operation by adopting each type of compression mode to be used as the reference total time corresponding to the type of compression mode, and using the compression mode of the type corresponding to the reference total time which is less than the set compression uploading time as an effective compression mode and outputting a number corresponding to the effective compression mode;
the model reconstruction mechanism is connected with the type extraction mechanism and is used for executing a learning action on the intelligent analysis model executing the compression mode analysis by adopting each test file so as to complete the reconstruction of the intelligent analysis model;
the number conversion device is connected with the model reconstruction mechanism and used for taking the file data volume of the current file to be compressed and uploaded, the unit time calculation amount of a processor chip of the compression processing device executing the compression operation and the current uploading rate of the uploading processing device executing the uploading operation as the input of the intelligent analysis model after repeated reconstruction, and operating the intelligent analysis model after repeated reconstruction to obtain the number corresponding to the effective compression mode corresponding to the current file to be compressed and uploaded;
the compression processing device is connected with the number conversion device and is used for executing compression processing on the current file to be compressed and uploaded by adopting an effective compression mode corresponding to the number output by the number conversion device so as to obtain a compressed file corresponding to the current file to be compressed and uploaded;
wherein, the value of the set compressed uploading time length obtained by dividing the file data amount by the lowest compressed uploading data amount threshold value of the unit time length comprises: the minimum compression uploading data volume threshold value of the unit time length is the minimum data volume which is allowed by a data operator and is compressed and uploaded per unit time length;
illustratively, the minimum compressed upload data amount threshold per unit time length is 10M per second when the minimum data amount compressed and uploaded per unit time length allowed by the data operator is 10M per second.
Next, a further description will be made of a specific structure of the data analysis system for compressive upload equalization of the present invention.
The data analysis system for compression upload equalization may further include:
the uploading processing device is connected with the compression processing device and is used for uploading the compressed file corresponding to the current file to be compressed to a network storage node occupied by a data operator;
the method comprises the following steps of taking the file data volume of a current file to be compressed and uploaded, the unit time calculation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of an intelligent analysis model which is repeatedly reconstructed, and operating the intelligent analysis model which is repeatedly reconstructed to obtain the number corresponding to an effective compression mode corresponding to the current file to be compressed and uploaded comprises the following steps: the number of times of reconstruction is inversely related to the value of the lowest compression uploading data volume threshold value in unit time length.
In the data analysis system for compressive upload equalization:
the step of executing a learning action on the intelligent analysis model executing the compression mode analysis by adopting each test file to complete a reconstruction of the intelligent analysis model comprises the following steps: taking the file data volume of the test file, the unit time operation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of the intelligent analysis model, taking the number corresponding to the effective compression mode corresponding to the test file as the output of the intelligent analysis model, and executing a learning action on the intelligent analysis model to finish a reconstruction of the intelligent analysis model;
wherein, taking the compression mode of the type corresponding to the reference total duration less than the set compression uploading duration as the effective compression mode and outputting the number corresponding to the effective compression mode comprises: and when more than two reference total time lengths exist, the compression mode of the type corresponding to the reference total time length with the minimum value is taken as an effective compression mode, and a number corresponding to the effective compression mode is output.
In the data analysis system for compression upload equalization:
the test preparation mechanism, the type extraction mechanism, the model reconstruction mechanism, the number conversion device, the compression processing device and the uploading processing device are arranged in the same data processing terminal;
the test preparation mechanism, the type extraction mechanism, the model reconstruction mechanism, the number conversion device, the compression processing device and the uploading processing device share the same power supply in the data processing terminal.
In the data analysis system for compression upload equalization:
the method comprises the following steps of taking the file data volume of a current file to be compressed and uploaded, the unit time operation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of an intelligent analysis model which is reconstructed for multiple times, wherein the method comprises the following steps: and when the file data volume of the current file to be compressed and uploaded exceeds the maximum set threshold, stopping executing the file data volume of the current file to be compressed and uploaded, the unit time operation amount of a processor chip of a compression processing device executing the compression operation and the current uploading rate of the uploading processing device executing the uploading operation as the input of the multi-time reconstructed intelligent analysis model.
Fig. 2 is a flow chart illustrating steps of a data analysis method for compressed upload equalization according to an embodiment of the present invention, the method comprising:
the device comprises a test preparation mechanism, a compression uploading mechanism and a compression uploading mechanism, wherein the test preparation mechanism is used for acquiring the file data volume of each test file and setting the compression uploading duration, and the value of the set compression uploading duration is obtained by dividing the file data volume by the lowest compression uploading data volume threshold of unit duration;
the usage type extraction mechanism is connected with the test preparation mechanism and used for compressing and uploading compressed files by adopting different types of data compression modes aiming at each test file, counting the total time consumed by completing compression operation and uploading operation by adopting each type of compression mode as the reference total time corresponding to the type of compression mode, and taking the compression mode of the type corresponding to the reference total time which is less than the set compression uploading time as an effective compression mode and outputting a number corresponding to the effective compression mode;
the model reconstruction mechanism is connected with the type extraction mechanism and used for executing a learning action on the intelligent analysis model executing the compression mode analysis by adopting each test file so as to complete the reconstruction of the intelligent analysis model;
the number conversion device is connected with the model reconstruction mechanism and used for taking the file data volume of the current file to be compressed and uploaded, the unit time calculation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of the intelligent analysis model after repeated reconstruction, and the intelligent analysis model after repeated reconstruction is operated to obtain the number corresponding to the effective compression mode corresponding to the current file to be compressed and uploaded;
the number conversion device is used for outputting an effective compression mode corresponding to the number to be compressed to the file to be uploaded so as to obtain a compressed file corresponding to the file to be compressed to be uploaded;
wherein, the value of the set compressed uploading time length obtained by dividing the file data amount by the lowest compressed uploading data amount threshold value of the unit time length comprises: the minimum compression uploading data volume threshold value of the unit time length is the minimum data volume which is allowed by a data operator and is compressed and uploaded per unit time length.
Next, the detailed steps of the data analysis method for compression upload equalization according to the present invention will be further described.
The data analysis method for compression upload equalization may further include:
the uploading processing device is connected with the compression processing device and used for uploading the compressed file corresponding to the current file to be compressed to a network storage node occupied by a data operator;
the method comprises the following steps of taking the file data volume of a current file to be compressed and uploaded, the unit time calculation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of an intelligent analysis model which is repeatedly reconstructed, and operating the intelligent analysis model which is repeatedly reconstructed to obtain the number corresponding to an effective compression mode corresponding to the current file to be compressed and uploaded comprises the following steps: the number of times of reconstruction is inversely related to the value of the lowest compression uploading data volume threshold value in unit time length.
The data analysis method for the compression uploading equalization comprises the following steps:
the step of executing a learning action on the intelligent analysis model executing the compression mode analysis by adopting each test file to complete a reconstruction of the intelligent analysis model comprises the following steps: taking the file data volume of the test file, the unit time operation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of the intelligent analysis model, taking the number corresponding to the effective compression mode corresponding to the test file as the output of the intelligent analysis model, and executing a learning action on the intelligent analysis model to finish the reconstruction of the intelligent analysis model;
the step of taking the compression mode of the type corresponding to the reference total duration less than the set compression uploading duration as the effective compression mode and outputting the number corresponding to the effective compression mode comprises the following steps: and when more than two reference total time lengths exist, the compression mode of the type corresponding to the reference total time length with the minimum value is taken as an effective compression mode, and a number corresponding to the effective compression mode is output.
The data analysis method for the compression uploading equalization comprises the following steps:
the test preparation mechanism, the type extraction mechanism, the model reconstruction mechanism, the number conversion device, the compression processing device and the uploading processing device are arranged in the same data processing terminal;
the test preparation mechanism, the type extraction mechanism, the model reconstruction mechanism, the number conversion device, the compression processing device and the uploading processing device share the same power supply in the data processing terminal.
The data analysis method for the compression uploading equalization comprises the following steps:
the method comprises the following steps of taking the file data volume of a current file to be compressed and uploaded, the unit time operation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of an intelligent analysis model which is reconstructed for multiple times, wherein the method comprises the following steps: and when the file data volume of the current file to be compressed and uploaded exceeds the maximum set threshold, stopping executing the file data volume of the current file to be compressed and uploaded, the unit time operation amount of a processor chip of a compression processing device executing the compression operation and the current uploading rate of the uploading processing device executing the uploading operation as the input of the multi-time reconstructed intelligent analysis model.
In addition, in the data analysis system and method for compression upload equalization, the file data volume of the current file to be compressed and uploaded, the unit time computation of the processor chip of the compression processing device executing the compression operation, and the current upload rate of the upload processing device executing the upload operation are used as the input of the multi-reconstructed intelligent analysis model, and the operation of the multi-reconstructed intelligent analysis model to obtain the number corresponding to the effective compression mode corresponding to the current file to be compressed and uploaded includes: the intelligent analytical model is based on a feedforward neural network.
By adopting the data analysis system and method for compression uploading balance, aiming at the technical problem of contradiction between the requirements of data compression duration and data uploading duration in the prior art, the compression mode meeting the requirement of a data operator for acquiring data speed can be analyzed for each data file based on the performance of the data compression equipment, the network speed of the data uploading equipment and the data volume of the data file, so that the dynamic balance between the original contradictory compression duration and uploading duration is realized.
While the above description refers to a preferred design, the preferred design may be further modified within the spirit and scope of this disclosure. This application is therefore intended to cover any variations, uses, or adaptations of the preferred design using the general principles disclosed herein. Further, this application is intended to cover such departures from the present disclosure as come within known or customary practice in the art to which this invention pertains and which fall within the limits of the appended claims.
Claims (8)
1. A data analysis system for compressed upload equalization, the system comprising:
the test preparation mechanism is used for acquiring the file data volume of each test file and setting the compression uploading duration, and the value of the set compression uploading duration is obtained by dividing the file data volume by the lowest compression uploading data volume threshold of unit duration;
the type extraction mechanism is connected with the test preparation mechanism and used for compressing and uploading compressed files by adopting different types of data compression modes aiming at each test file, counting the total time consumed by completing compression operation and uploading operation by adopting each type of compression mode as the reference total time corresponding to the type compression mode, taking the compression mode of the type corresponding to the reference total time less than the set compression uploading time as an effective compression mode and outputting the number corresponding to the effective compression mode, wherein when the reference total time less than the set compression uploading time is more than two, the compression mode of the type corresponding to the reference total time with the minimum value is taken as the effective compression mode and the number corresponding to the effective compression mode is output;
the model reconstruction mechanism is connected with the type extraction mechanism and used for executing a learning action on an intelligent analysis model which executes compression mode analysis by adopting each test file so as to finish a reconstruction on the intelligent analysis model, and the model reconstruction mechanism takes the file data volume of the test file, the unit time operation amount of a processor chip of a compression processing device which executes compression operation and the current uploading rate of an uploading processing device which executes uploading operation as the input of the intelligent analysis model, takes the number corresponding to the effective compression mode corresponding to the test file as the output of the intelligent analysis model, and executes a learning action on the intelligent analysis model so as to finish a reconstruction on the intelligent analysis model; the number conversion device is connected with the model reconstruction mechanism and used for taking the file data volume of the current file to be compressed and uploaded, the unit time calculation amount of a processor chip of the compression processing device executing the compression operation and the current uploading rate of the uploading processing device executing the uploading operation as the input of the intelligent analysis model after repeated reconstruction, and operating the intelligent analysis model after repeated reconstruction to obtain the number corresponding to the effective compression mode corresponding to the current file to be compressed and uploaded;
the compression processing device is connected with the number conversion device and is used for executing compression processing on the current file to be compressed and uploaded by adopting an effective compression mode corresponding to the number output by the number conversion device so as to obtain a compressed file corresponding to the current file to be compressed and uploaded;
wherein, the value of the set compressed uploading time length obtained by dividing the file data amount by the lowest compressed uploading data amount threshold value of the unit time length comprises: the minimum compression uploading data volume threshold value per unit time length is the minimum data volume which is allowed by a data operator and is compressed and uploaded per unit time length.
2. The data analysis system for compression upload equalization of claim 1 further comprising:
the uploading processing device is connected with the compression processing device and is used for uploading the compressed file corresponding to the current file to be compressed to a network storage node occupied by a data operator;
the method comprises the following steps of taking the file data volume of a current file to be compressed and uploaded, the unit time calculation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of an intelligent analysis model which is repeatedly reconstructed, and operating the intelligent analysis model which is repeatedly reconstructed to obtain the number corresponding to an effective compression mode corresponding to the current file to be compressed and uploaded comprises the following steps: the number of times of reconstruction is inversely related to the value of the lowest compression uploading data volume threshold value in unit time length.
3. A data analysis system for compression upload equalization as claimed in any of claims 1-2 wherein:
the test preparation mechanism, the type extraction mechanism, the model reconstruction mechanism, the number conversion device, the compression processing device and the uploading processing device are arranged in the same data processing terminal;
the test preparation mechanism, the type extraction mechanism, the model reconstruction mechanism, the number conversion device, the compression processing device and the uploading processing device share the same power supply in the data processing terminal.
4. A data analysis system for compression upload equalization according to any of claims 1-2, characterized by:
the method comprises the following steps of taking the file data volume of a current file to be compressed and uploaded, the unit time operation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of an intelligent analysis model which is reconstructed for multiple times, wherein the method comprises the following steps: and when the file data volume of the current file to be compressed and uploaded exceeds a maximum set threshold, stopping executing the file data volume of the current file to be compressed and uploaded, the unit time operation amount of a processor chip of a compression processing device executing the compression operation and the current uploading rate of the uploading processing device executing the uploading operation as the input of the intelligent analysis model after repeated reconstruction.
5. A method of data analysis for compressed upload equalization, the method comprising:
the device comprises a test preparation mechanism, a compression uploading mechanism and a compression uploading mechanism, wherein the test preparation mechanism is used for acquiring the file data volume of each test file and setting the compression uploading duration, and the value of the set compression uploading duration is obtained by dividing the file data volume by the lowest compression uploading data volume threshold of unit duration;
the usage type extraction mechanism is connected with the test preparation mechanism and used for compressing and uploading compressed files by adopting different types of data compression modes aiming at each test file, counting the total time consumed by completing compression operation and uploading operation by adopting each type of compression mode as the reference total time corresponding to the type of compression mode, taking the compression mode of the type corresponding to the reference total time less than the set compression uploading time as an effective compression mode and outputting the number corresponding to the effective compression mode, wherein when the reference total time less than the set compression uploading time is more than two, the compression mode of the type corresponding to the reference total time with the minimum value is taken as the effective compression mode and the number corresponding to the effective compression mode is output;
the model reconstruction mechanism is connected with the type extraction mechanism and used for executing a learning action on an intelligent analysis model which executes compression mode analysis by adopting each test file so as to finish a reconstruction on the intelligent analysis model, and the model reconstruction mechanism comprises the steps of taking the file data volume of the test file, the unit time operation quantity of a processor chip of a compression processing device which executes compression operation and the current uploading rate of an uploading processing device which executes uploading operation as the input of the intelligent analysis model, taking the number corresponding to the effective compression mode corresponding to the test file as the output of the intelligent analysis model, and executing a learning action on the intelligent analysis model so as to finish a reconstruction on the intelligent analysis model;
the number conversion device is connected with the model reconstruction mechanism and used for taking the file data volume of the current file to be compressed and uploaded, the unit time calculation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of the intelligent analysis model after repeated reconstruction, and the intelligent analysis model after repeated reconstruction is operated to obtain the number corresponding to the effective compression mode corresponding to the current file to be compressed and uploaded;
the number conversion device is used for outputting an effective compression mode corresponding to the number to be compressed to the file to be uploaded so as to obtain a compressed file corresponding to the file to be compressed to be uploaded;
the dividing of the file data volume by the lowest compression uploading data volume threshold value of the unit time length to obtain the value of the set compression uploading time length comprises the following steps: the minimum compression uploading data volume threshold value of the unit time length is the minimum data volume which is allowed by a data operator and is compressed and uploaded per unit time length.
6. The data analysis method for compression upload equalization of claim 5 further comprising:
the uploading processing device is connected with the compression processing device and used for uploading the compressed file corresponding to the current file to be compressed to a network storage node occupied by a data operator;
the method comprises the following steps of taking the file data volume of a current file to be compressed and uploaded, the unit time calculation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of an intelligent analysis model which is repeatedly reconstructed, and operating the intelligent analysis model which is repeatedly reconstructed to obtain the number corresponding to an effective compression mode corresponding to the current file to be compressed and uploaded comprises the following steps: the times of reconstruction are inversely related to the value of the lowest compression uploading data volume threshold value in unit time length.
7. A method of data analysis for compression upload equalization according to any of claims 5-6, characterized by:
the test preparation mechanism, the type extraction mechanism, the model reconstruction mechanism, the number conversion device, the compression processing device and the uploading processing device are arranged in the same data processing terminal;
the test preparation mechanism, the type extraction mechanism, the model reconstruction mechanism, the number conversion device, the compression processing device and the uploading processing device share the same power supply in the data processing terminal.
8. A method of data analysis for compression upload equalization according to any of claims 5-6, characterized by:
the method comprises the following steps of taking the file data volume of a current file to be compressed and uploaded, the unit time operation amount of a processor chip of a compression processing device executing compression operation and the current uploading rate of an uploading processing device executing uploading operation as the input of an intelligent analysis model which is reconstructed for multiple times, wherein the method comprises the following steps: and when the file data volume of the current file to be compressed and uploaded exceeds the maximum set threshold, stopping executing the file data volume of the current file to be compressed and uploaded, the unit time operation amount of a processor chip of a compression processing device executing the compression operation and the current uploading rate of the uploading processing device executing the uploading operation as the input of the multi-time reconstructed intelligent analysis model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210690330.2A CN115086301B (en) | 2022-06-17 | 2022-06-17 | Data analysis system and method for compression uploading equalization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210690330.2A CN115086301B (en) | 2022-06-17 | 2022-06-17 | Data analysis system and method for compression uploading equalization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115086301A CN115086301A (en) | 2022-09-20 |
| CN115086301B true CN115086301B (en) | 2022-12-27 |
Family
ID=83254330
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210690330.2A Active CN115086301B (en) | 2022-06-17 | 2022-06-17 | Data analysis system and method for compression uploading equalization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115086301B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116233438B (en) * | 2023-03-09 | 2023-08-29 | 上海华期信息技术有限责任公司 | Data prediction acquisition system using weighting algorithm |
| CN116347080B (en) * | 2023-03-27 | 2023-10-31 | 苏州利博特信息科技有限公司 | Intelligent algorithm application system and method based on downsampling processing |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110727643A (en) * | 2019-09-16 | 2020-01-24 | 北京明朝万达科技股份有限公司 | File classification management method and system based on machine learning |
| CN111476002A (en) * | 2020-04-07 | 2020-07-31 | 北京东方金信科技有限公司 | Data file coding format conversion method and system |
| CN112398481A (en) * | 2020-11-20 | 2021-02-23 | 南京轨道交通系统工程有限公司 | Feedback type matching prediction multistage real-time compression system and method |
| CN112637598A (en) * | 2020-12-18 | 2021-04-09 | 国家电网有限公司信息通信分公司 | Video compression and decompression method, device, equipment and readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10313256B2 (en) * | 2015-05-21 | 2019-06-04 | Intel Corporation | Apparatus and methods for adaptive data compression |
-
2022
- 2022-06-17 CN CN202210690330.2A patent/CN115086301B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110727643A (en) * | 2019-09-16 | 2020-01-24 | 北京明朝万达科技股份有限公司 | File classification management method and system based on machine learning |
| CN111476002A (en) * | 2020-04-07 | 2020-07-31 | 北京东方金信科技有限公司 | Data file coding format conversion method and system |
| CN112398481A (en) * | 2020-11-20 | 2021-02-23 | 南京轨道交通系统工程有限公司 | Feedback type matching prediction multistage real-time compression system and method |
| CN112637598A (en) * | 2020-12-18 | 2021-04-09 | 国家电网有限公司信息通信分公司 | Video compression and decompression method, device, equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115086301A (en) | 2022-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115086301B (en) | Data analysis system and method for compression uploading equalization | |
| CN110389820B (en) | Private cloud task scheduling method for resource prediction based on v-TGRU model | |
| CN112000636A (en) | User behavior statistical analysis method based on Flink streaming processing | |
| CN111339433B (en) | Information recommendation method and device based on artificial intelligence and electronic equipment | |
| CN111274256B (en) | Resource management and control method, device, equipment and storage medium based on time sequence database | |
| CN112418482A (en) | Cloud computing energy consumption prediction method based on time series clustering | |
| CN110083518B (en) | AdaBoost-Elman-based virtual machine software aging prediction method | |
| DE112018005205T5 (en) | Compression of completely connected / recurring layers from one or more deep networks by enforcing spatial locality for weight matrices and effecting frequency compression | |
| CN111738435B (en) | Online sparse training method and system based on mobile equipment | |
| CN110730099A (en) | Flow prediction method based on historical flow data time sequence | |
| CN113568740B (en) | Model aggregation method, system, equipment and medium based on federal learning | |
| CN114154646A (en) | Efficiency optimization method for federal learning in mobile edge network | |
| DE102021127244A1 (en) | Artificial intelligence optimization platform | |
| CN111163314A (en) | Image compression method and system | |
| Chen et al. | FedOBD: Opportunistic block dropout for efficiently training large-scale neural networks through federated learning | |
| CN111209105A (en) | Capacity expansion processing method, capacity expansion processing device, capacity expansion processing equipment and readable storage medium | |
| CN110263917B (en) | Neural network compression method and device | |
| DE102022204244A1 (en) | Generate digital recommendations using collaborative filtering, reinforcement learning, and inclusive sets of negative feedback | |
| CN114861907A (en) | Data calculation method, device, storage medium and equipment | |
| CN117651144A (en) | Deep learning-based building point cloud compression method and system | |
| CN116149848A (en) | Load prediction method and device, electronic equipment and storage medium | |
| CN116957043A (en) | Model quantization method, device, equipment and medium | |
| CN114610234B (en) | Storage system parameter recommendation method and related device | |
| CN116187387A (en) | Neural network model quantization method, device, computer equipment and storage medium | |
| CN110647448A (en) | Mobile application operation log data real-time analysis method, server and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20221208 Address after: Room 102-2, 1st floor, building 42, Zhifu center, 966 Xiuyuan Road, Xiuzhou District, Jiaxing City, Zhejiang Province, 314031 Applicant after: JIAXING YUNQIE SUPPLY CHAIN MANAGEMENT CO.,LTD. Address before: 710121 School of computer science, Xi'an University of Posts and telecommunications, Chang'an District, Xi'an City, Shaanxi Province Applicant before: Yan Xue |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: Room 102-2, 1st floor, building 42, Zhifu center, 966 Xiuyuan Road, Xiuzhou District, Jiaxing City, Zhejiang Province, 314031 Patentee after: Jiaxing Yunche Online Technology Co.,Ltd. Country or region after: China Address before: Room 102-2, 1st floor, building 42, Zhifu center, 966 Xiuyuan Road, Xiuzhou District, Jiaxing City, Zhejiang Province, 314031 Patentee before: JIAXING YUNQIE SUPPLY CHAIN MANAGEMENT CO.,LTD. Country or region before: China |