CN112765109B - Queue type data storage analysis method and system - Google Patents
Queue type data storage analysis method and system Download PDFInfo
- Publication number
- CN112765109B CN112765109B CN202110072351.3A CN202110072351A CN112765109B CN 112765109 B CN112765109 B CN 112765109B CN 202110072351 A CN202110072351 A CN 202110072351A CN 112765109 B CN112765109 B CN 112765109B
- Authority
- CN
- China
- Prior art keywords
- data
- processing center
- platform
- transmitting
- service processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 26
- 238000004458 analytical method Methods 0.000 title claims abstract description 13
- 238000007405 data analysis Methods 0.000 claims abstract description 44
- 230000005540 biological transmission Effects 0.000 claims abstract description 34
- 238000000034 method Methods 0.000 claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 17
- 239000011159 matrix material Substances 0.000 claims description 28
- 238000012795 verification Methods 0.000 claims description 20
- 238000012546 transfer Methods 0.000 claims description 6
- 238000012935 Averaging Methods 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 3
- 238000013486 operation strategy Methods 0.000 claims description 3
- 238000013468 resource allocation Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 6
- 238000011084 recovery Methods 0.000 description 3
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 206010033799 Paralysis Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1435—Saving, restoring, recovering or retrying at system level using file system or storage system metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Bioethics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Library & Information Science (AREA)
- Quality & Reliability (AREA)
- Telephonic Communication Services (AREA)
Abstract
The invention provides a queue type data storage analysis method and a device, wherein the method comprises the following steps: acquiring a plurality of landing platform data; transmitting the plurality of landing platform data to the MQ server through a transmission control protocol; acquiring data in the MQ server through a data collection center, and transmitting the data to a central database; the central database performs data backup on the data and transmits the data to a data processing center; the data processing center analyzes the data through the big data analysis engine and transmits the analyzed data to the service processing center; the service processing center performs service processing according to the analyzed data, so that the connection and processing pressure of the server is effectively relieved through a data storage mode of a queue mechanism, whether the resource allocation is reasonable or not is effectively analyzed, the cost is saved, the potential risk of operation is timely found, the user experience is improved, a small amount of data analysis is avoided, and the actual condition of the whole platform cannot be reflected.
Description
Technical Field
The present invention relates to the field of data storage, and in particular, to a method and system for analyzing queue-type data storage.
Background
At present, data storage technology is getting more and more attention, and it is common practice to transplant a file system on a single chip microcomputer, and manage data to be stored through the file system. A file system is a method and a data structure for an operating system to definitely store files on a device or a partition, and common storage devices include a magnetic disk, a solid state disk based on NANDF L ASH, and the like. The file system consists of three parts: an interface of a file system, a software set for controlling and managing objects, objects and attributes. From a system perspective, a file system is a system that organizes and allocates space for file storage devices, is responsible for storing files, and protects and retrieves stored files. Specifically, it is responsible for creating files for users, storing, reading out, modifying, dumping files, controlling access to files, revoking files when users are no longer using, etc.
The file system is used for storing and managing the data, so that the universality is high, the types of the data which can be stored and managed are wide, but the resource requirements on the singlechip are high, including the requirements on the CPU and the storage memory in the singlechip are relatively high. And the complexity of the file system is higher, the corresponding storage position of the data is not easy to find, the reliability is poor, when a user adds new storage data in the file system or modifies the stored data, the index of the data (namely key data in the file system) in the file system can be automatically modified, and at the moment, the sudden electrode breaking can possibly cause the damage of the key data of the file system, so that the whole file system is paralyzed, and the stored data can possibly not be acquired in the file system.
Disclosure of Invention
Because the service call list data of the my department 400 are distributed on different operator landing platforms in various places of the country, the service call data of a single platform cannot actually reflect the service condition of the my department, and in order to solve the problems, the invention provides a queue type data storage analysis method and a system, wherein the method comprises the following steps:
acquiring a plurality of landing platform data;
transmitting the plurality of landing platform data to the MQ server through a transmission control protocol;
Acquiring data in the MQ server through a data collection center, and transmitting the data to a central database;
The central database performs data backup on the data and transmits the data to a data processing center;
The data processing center analyzes the data through the big data analysis engine and transmits the analyzed data to the service processing center;
And carrying out service processing according to the analyzed data by a service processing center.
Further, the transmitting the plurality of landing platform data to the MQ server via the transmission control protocol includes:
Generating a working key for the acquired plurality of land platform data;
The operation key is used for compressing and encrypting the acquired plurality of landing platform data to obtain a plurality of encrypted data, each encrypted data corresponds to one landing platform data, the plurality of encrypted data are randomly ordered, and the plurality of encrypted data are sequentially transmitted to the MQ server through a transmission control protocol;
And transmitting the working key to the MQ server through a transmission control protocol, wherein the MQ server utilizes the working key to decrypt the plurality of encrypted data, acquires a plurality of decrypted data, and sequentially transmits the plurality of decrypted data to a data processing center according to the sequence of transmitting the corresponding encrypted data to the MQ server.
Further, the generating a working key for the file data to be transmitted includes:
randomly generating a Random character string containing letters and data by utilizing a Random function;
and taking the randomly generated character string as a working key.
Further, the acquiring, by the data collection center, the data in the MQ server includes:
The data collection center sends a remote login request to the MQ server end;
The MQ server side sends a verification information request to the data collection center according to the remote login request;
The data collection center sends verification information to the MQ server side according to the verification information request;
The MQ server judges whether the verification information sent by the data collection center is correct or not according to the pre-stored verification information, if yes, the MQ server transmits a plurality of decrypted data to the data collection center; if not, waiting for a new telnet request.
Further, the central database performs data backup on the data and transmits the data to a data processing center, and the method comprises the following steps:
Creating a backup catalog and a script catalog in a central database and giving permission;
editing a script program according to the backup catalog and the script catalog;
Editing a command script according to the script program so as to backup data according to the command script, and transmitting the data to a data processing center.
Further, the data processing center analyzes the data through the big data analysis engine and transmits the analyzed data to the service processing center, and the method comprises the following steps:
according to the customer traffic usage ranking, finding the industry to which the corresponding customer company belongs, and then carrying out industry classification ranking to obtain the front-ranked industry data information, and transmitting the data to a service processing center;
Averaging again according to the time difference between the calling incoming time and the called answering time of all the connected calls, so as to obtain average switching duration data, and transmitting the data to a service processing center;
and counting the line usage amount in the platform per minute, comparing the peak value with the usage amount threshold value configured by the platform, thereby acquiring the data information of the platform frequency and peak period, and transmitting the data to the service processing center.
Further, the ranking according to the traffic usage of the client finds the industry to which the corresponding client company belongs, and then performs the industry classification ranking to obtain the top-ranked industry data information, and transmits the data to the service processing center, including:
Extracting the customer traffic usage data in the sorted data, and sorting the customer traffic usage data through a sort function;
the ordered customer business usage data are respectively found out to correspond to industries to which the customer formulas belong, and the industries are classified and combined;
and ranking websites for each type of industry name according to the classified and combined customer service usage data, so as to obtain the top-ranked industry data information, and transmitting the data to a service processing center.
Further, the counting of the line usage amount per minute in the platform, comparing the peak value with the usage amount threshold value configured by the platform, thereby obtaining the data information of the platform frequency and peak period, and transmitting the data to the service processing center, including:
Step A1, acquiring each data analysis matrix, wherein the data analysis matrix comprises current line usage data per minute and current line peak value taking data, and calculating current parameters of each data analysis matrix according to the current line usage data per minute and the current line peak value taking data by using the following formula:
Wherein J K represents the current parameters of the data analysis matrix, Representing current line usage data information per minute,The method comprises the steps that peak data information is taken from a current line, k represents iteration times, k epsilon [1, T ], T represents preset iteration times, alpha and beta represent parameter adjustment coefficients, the value range of alpha and beta is 0-1, i represents the number of rows of a data analysis matrix, i=1, 2,..m, j represents the number of columns of the data analysis matrix, j=1, 2,..n;
Step A2, analyzing the current parameter J K of the matrix according to the following data, and calculating the frequency of the landing platform by using the following formula:
wherein F represents the frequency of the landing platform, pi represents a natural constant, and tan represents a tangent function;
and step A3, comparing the frequency F of the landing platform with a preset threshold, and when the frequency F of the landing platform is higher than the preset threshold, acquiring peak data information under the frequency F and transmitting the data information to a service processing center.
Further, the service processing by the service processing center according to the sorted data includes:
Positioning customer groups for industry customers according to the industry data information in the analyzed data, and adjusting the operation strategy;
According to the transfer duration data in the analyzed data, optimizing the communication line of an operator and improving the user experience for the length of transfer seat time and the call completing rate;
according to the platform frequency and peak period data information in the analyzed data, the formula investment of the platform with less resource consumption is reduced, the busy platform is used for early warning and timely energy expansion in advance, and the stable operation of the platform is ensured.
Further, the system comprises:
the acquisition module is used for acquiring a plurality of landing platform data;
The transmission module is used for transmitting the data of the plurality of landing platforms to the MQ server through a transmission control protocol;
The data collection center is used for acquiring the data in the MQ server and transmitting the data to the central database;
the central database is used for carrying out data backup on the data and transmitting the data to the data processing center;
The data processing center is used for analyzing the data through the big data analysis engine and transmitting the analyzed data to the service processing center;
And the service processing center is used for carrying out service processing according to the analyzed data.
Compared with the prior art, the invention has the beneficial effects that: the invention provides a method and a system for storing and analyzing queue type data, wherein the method comprises the following steps: acquiring a plurality of landing platform data; transmitting the plurality of landing platform data to the MQ server through a transmission control protocol; acquiring data in the MQ server through a data collection center, and transmitting the data to a central database; the central database performs data backup on the data and transmits the data to a data processing center; the data processing center analyzes the data through the big data analysis engine and transmits the analyzed data to the service processing center; the service processing center performs service processing according to the analyzed data, so that the connection and processing pressure of the server is effectively relieved through a data storage mode of a queue mechanism, whether the resource allocation is reasonable or not is effectively analyzed, the cost is saved, the potential risk of operation is timely found, the user experience is improved, a small amount of data analysis is avoided, and the actual condition of the whole platform cannot be reflected.
Drawings
In order to more clearly illustrate the technical solution of the embodiments of the present invention, the following description will briefly explain the drawings of the embodiments of the present invention. Wherein the showings are for the purpose of illustrating some embodiments of the invention only and not for the purpose of limiting the same.
FIG. 1 is a flow chart of a method for analyzing queue data storage according to the present invention;
FIG. 2 is a flow chart of a queued data storage analysis system according to the present invention;
fig. 3 is a block diagram of a queue data storage analysis system according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1-3, the technical problem to be solved by the present invention is to provide a method for analyzing queue type data storage, which includes:
SI, obtaining a plurality of landing platform data;
S2, transmitting the data of the plurality of landing platforms to an MQ server through a transmission control protocol;
S3, acquiring data in the MQ server through a data collection center, and transmitting the data to a central database;
s4, the central database performs data backup on the data and transmits the data to a data processing center;
S5, the data processing center analyzes the data through the big data analysis engine and transmits the analyzed data to the service processing center;
S6, business processing is carried out through the business processing center according to the analyzed data.
Firstly, acquiring a plurality of landing platform data, wherein the landing platform data comprises a calling number, a called number, a home zone, etc. (character type); secondly, transmitting the data of the plurality of landing platforms to an MQ server through a transmission control protocol, wherein the MQ server is based on cloud storage and performs data storage by adopting a first-in first-out storage principle, so that the sequence between the data is ensured; then, acquiring data in the MQ server through a data collection center, and transmitting the data to a central database; then, the central database performs data backup on the data and transmits the data to a data processing center, so that the safety of the data is ensured; the data processing center analyzes the data through the big data analysis engine and transmits the analyzed data to the service processing center; finally, business processing is carried out through a business processing center according to the analyzed data; by adopting a queue mechanism data storage mode, the connection and processing pressure of the link server are effective, whether the resource configuration is reasonable or not is effectively analyzed through data analysis, the cost is saved, the potential risk of operation is timely found, the user experience is improved, a small amount of data analysis is avoided, and the actual condition of the whole platform cannot be reflected.
In one embodiment of the present invention, the transmitting the plurality of landing platform data to the MQ server via the transmission control protocol includes:
Generating a working key for the acquired plurality of land platform data;
The operation key is used for compressing and encrypting the acquired plurality of landing platform data to obtain a plurality of encrypted data, each encrypted data corresponds to one landing platform data, the plurality of encrypted data are randomly ordered, and the plurality of encrypted data are sequentially transmitted to the MQ server through a transmission control protocol;
And transmitting the working key to the MQ server through a transmission control protocol, wherein the MQ server utilizes the working key to decrypt the plurality of encrypted data, acquires a plurality of decrypted data, and sequentially transmits the plurality of decrypted data to a data processing center according to the sequence of transmitting the corresponding encrypted data to the MQ server.
In the above technical solution, first, a working key is generated for the acquired plurality of land platform data, and the generating the working key for the file data to be transmitted includes: randomly generating a Random character string containing letters and data by utilizing a Random function, and taking the randomly generated character string as a working key; secondly, the working key is used for compressing and encrypting the acquired plurality of landing platform data to obtain a plurality of encrypted data, each encrypted data corresponds to one landing platform data, the plurality of encrypted data are randomly ordered, and the plurality of encrypted data are sequentially transmitted to the MQ server through a transmission control protocol; and finally, the working key is transmitted to the MQ server through a transmission control protocol, the MQ server decrypts the encrypted data by using the working key to obtain decrypted data, the decrypted data are sequentially transmitted to a data processing center according to the sequence of the transmission of the encrypted data to the MQ server, the transmission data are compressed and encrypted by using a big data technology, so that cluster hardware resources are recycled, the compression encryption efficiency and the transmission efficiency are greatly improved, the network transmission safety is improved by compressing the encrypted data, the data volume is greatly reduced after compression, the network transmission rate is improved, and the transmission efficiency is ensured.
In one embodiment of the present invention, the obtaining, by the data collection center, the data of the plurality of landing platforms in the MQ server includes:
The data collection center sends a remote login request to the MQ server end;
The MQ server side sends a verification information request to the data collection center according to the remote login request;
The data collection center sends verification information to the MQ server side according to the verification information request;
The MQ server judges whether the verification information sent by the data collection center is correct or not according to the pre-stored verification information, if yes, the MQ server transmits a plurality of decrypted data to the data collection center; if not, waiting for a new telnet request.
In the above technical scheme, firstly, the data collection center sends a remote login request to the MQ server end; secondly, the MQ server side sends a verification information request to the data collection center according to the remote login request; then, the data collection center sends verification information to the MQ server side according to the verification information request; finally, the MQ server judges whether the verification information sent by the data collection center is correct or not according to the pre-stored verification information, if yes, the MQ server transmits a plurality of decrypted data to the data collection center; if not, waiting for a new remote login request; therefore, the safety of data transmission is ensured, the MQ server can realize connection with the data collection center, the transmission object is ensured to be confirmed when the data is transmitted, the transmission correctness is ensured, and the data is prevented from being lost.
In one embodiment of the present invention, the central database performs data backup on a plurality of land platform data, and transmits the data to a data processing center, including:
Creating a backup catalog and a script catalog in a central database and giving permission;
editing a script program according to the backup catalog and the script catalog;
Editing a command script according to the script program so as to backup data according to the command script, and transmitting the data to a data processing center.
Firstly, creating a backup catalog and a script catalog in a central database and giving permission; secondly, editing a script program according to the backup catalog and the script catalog; finally, editing a command script according to the script program to backup data according to the command script, and transmitting the data to a data processing center; by backing up the data, the loss of the data is avoided, the high availability and disaster recovery of the system are improved, the data can be found by backing up the database when the database system crashes, the data is restored by backing up the database, which is the optimal scheme for providing the minimum cost of data recovery when the system crashes, and the recovery of the data loss is ensured.
In one embodiment of the present invention, the data processing center analyzes data through a big data analysis engine, sorts the analyzed data, and transmits the sorted data to a service processing center, including:
according to the customer traffic usage ranking, finding the industry to which the corresponding customer company belongs, and then carrying out industry classification ranking to obtain the front-ranked industry data information, and transmitting the data to a service processing center;
Averaging again according to the time difference between the calling incoming time and the called answering time of all the connected calls, so as to obtain average switching duration data, and transmitting the data to a service processing center;
and counting the line usage amount in the platform per minute, comparing the peak value with the usage amount threshold value configured by the platform, thereby acquiring the data information of the platform frequency and peak period, and transmitting the data to the service processing center.
According to the technical scheme, firstly, according to the ranking of the traffic usage amount of the client, the industry of the corresponding client company is found, and then the industry classification ranking is carried out, so that the industry data information with the top ranking is obtained, and the data is transmitted to the service processing center, wherein the method comprises the following steps: extracting customer traffic usage data in the sorted data, sorting the customer traffic usage data through a sort function, respectively finding the industries to which the corresponding customer formulas of the sorted customer traffic usage data belong, carrying out industry classification and combination, and carrying out website ranking on each type of industry names according to the classified and combined customer traffic usage data, thereby acquiring the top-ranked industry data information, transmitting the data to a business processing center, thereby better acquiring the top-ranked industry information data, carrying out customer group targeted advertisement delivery according to the industry quantity, and ensuring the integrity of a system; secondly, according to the time difference value of all calling incoming call time and called answering time of the call, averaging again, so as to obtain average switching duration data, and transmitting the data to a service processing center; finally, counting the line usage amount in each minute in the platform, comparing the peak value with the usage amount threshold value configured by the platform, thereby obtaining the data information of the platform frequency and peak period, and transmitting the data to a service processing center, wherein the method comprises the following steps: step A1, acquiring each data analysis matrix, wherein the data analysis matrix comprises current line usage data per minute and current line peak value taking data, and calculating current parameters of each data analysis matrix according to the current line usage data per minute and the current line peak value taking data by using the following formula:
Wherein J K represents the current parameters of the data analysis matrix, Representing current line usage data information per minute,The method comprises the steps that peak data information is taken from a current line, k represents iteration times, k epsilon [1, T ], T represents preset iteration times, alpha and beta represent parameter adjustment coefficients, the value range of alpha and beta is 0-1, i represents the number of rows of a data analysis matrix, i=1, 2,..m, j represents the number of columns of the data analysis matrix, j=1, 2,..n;
Step A2, analyzing the current parameter J K of the matrix according to the following data, and calculating the frequency of the landing platform by using the following formula:
wherein F represents the frequency of the landing platform, pi represents a natural constant, and tan represents a tangent function;
Step A3, comparing the frequency F of the landing platform with a preset threshold, and when the frequency F of the landing platform is higher than the preset threshold, acquiring peak data information under the frequency F, transmitting the data information to a service processing center, so as to analyze the data, and performing service processing according to the analyzed data by the service processing center, wherein the step A3 comprises the following steps:
Positioning customer groups for industry customers according to the industry data information in the analyzed data, and adjusting the operation strategy;
According to the transfer duration data in the analyzed data, optimizing the communication line of an operator and improving the user experience for the length of transfer seat time and the call completing rate;
According to the platform frequency and peak period data information in the analyzed data, the formula input is reduced, the busy platform is used for early warning and timely energy expansion in advance, and the stable operation of the platform is ensured.
A queued data storage analysis system, the system comprising:
the acquisition module is used for acquiring a plurality of landing platform data;
The transmission module is used for transmitting the data of the plurality of landing platforms to the MQ server through a transmission control protocol;
The data collection center is used for acquiring the data in the MQ server and transmitting the data to the central database;
the central database is used for carrying out data backup on the data and transmitting the data to the data processing center;
The data processing center is used for analyzing the data through the big data analysis engine and transmitting the analyzed data to the service processing center;
And the service processing center is used for carrying out service processing according to the analyzed data.
The technical scheme includes that firstly, an acquisition module acquires a plurality of landing platform data, wherein the landing platform data comprise calling and called numbers, home areas, and the like (character types); secondly, the transmission module transmits the data of the plurality of landing platforms to the MQ server through a transmission control protocol, the MQ server is based on cloud storage, and the data storage is carried out by adopting a first-in first-out storage principle, so that the sequence between the data is ensured; then, the data collection center acquires the data in the MQ server and transmits the data to the central database; then, the central database performs data backup on the data and transmits the data to a data processing center, so that the safety of the data is ensured; the data processing center analyzes the data through the big data analysis engine and transmits the analyzed data to the service processing center; finally, the service processing center carries out service processing according to the analyzed data; by adopting a queue mechanism data storage mode, the connection and processing pressure of the link server are effective, whether the resource configuration is reasonable or not is effectively analyzed through data analysis, the cost is saved, the potential risk of operation is timely found, the user experience is improved, a small amount of data analysis is avoided, and the actual condition of the whole platform cannot be reflected.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A method of queue data storage analysis, the method comprising:
acquiring a plurality of landing platform data;
transmitting the plurality of landing platform data to the MQ server through a transmission control protocol;
Acquiring data in the MQ server through a data collection center, and transmitting the data to a central database;
The central database performs data backup on the data and transmits the data to a data processing center;
The data processing center analyzes the data through the big data analysis engine and transmits the analyzed data to the service processing center;
Carrying out service processing according to the analyzed data by a service processing center;
The data processing center analyzes the data through the big data analysis engine and transmits the analyzed data to the service processing center, and the method comprises the following steps:
according to the customer traffic usage ranking, finding the industry to which the corresponding customer company belongs, and then carrying out industry classification ranking to obtain the front-ranked industry data information, and transmitting the data to a service processing center;
Averaging again according to the time difference between the calling incoming time and the called answering time of all the connected calls, so as to obtain average switching duration data, and transmitting the data to a service processing center;
Counting the line usage amount in each minute in the platform, comparing the peak value with the usage amount threshold value configured by the platform, thereby acquiring the data information of the platform frequency and peak period, and transmitting the data to a service processing center;
the statistics of the line usage amount per minute in the platform, the comparison of the peak value and the usage amount threshold value configured by the platform, thereby obtaining the data information of the platform frequency and peak period, and transmitting the data to the service processing center, comprises the following steps:
Step A1, acquiring each data analysis matrix, wherein the data analysis matrix comprises current line usage data per minute and current line peak value taking data, and calculating current parameters of each data analysis matrix according to the current line usage data per minute and the current line peak value taking data by using the following formula:
Wherein J K represents the current parameters of the data analysis matrix, Data information representing current line usage per minute,/>The method comprises the steps of representing peak data information of a current line, wherein k represents iteration times, k epsilon [1, T ], T represents preset iteration times, alpha and beta represent parameter adjustment coefficients, the value range of alpha and beta is 0-1, i represents the number of rows of a data analysis matrix, i=1, 2, … m, j represents the number of columns of the data analysis matrix, and j=1, 2, … n;
Step A2, analyzing the current parameter J K of the matrix according to the following data, and calculating the frequency of the landing platform by using the following formula:
wherein F represents the frequency of the landing platform, pi represents a natural constant, and tan represents a tangent function;
and step A3, comparing the frequency F of the landing platform with a preset threshold, and when the frequency F of the landing platform is higher than the preset threshold, acquiring peak data information under the frequency F and transmitting the data information to a service processing center.
2. The method for queue data storage analysis of claim 1, wherein transmitting the plurality of landing platform data to the MQ server via the transmission control protocol comprises:
Generating a working key for the acquired plurality of land platform data;
The operation key is used for compressing and encrypting the acquired plurality of landing platform data to obtain a plurality of encrypted data, each encrypted data corresponds to one landing platform data, the plurality of encrypted data are randomly ordered, and the plurality of encrypted data are sequentially transmitted to the MQ server through a transmission control protocol;
And transmitting the working key to the MQ server through a transmission control protocol, wherein the MQ server utilizes the working key to decrypt the plurality of encrypted data, acquires a plurality of decrypted data, and sequentially transmits the plurality of decrypted data to a data processing center according to the sequence of transmitting the corresponding encrypted data to the MQ server.
3. The method for queue data storage analysis of claim 2, wherein generating the working key for the acquired plurality of landing platform data comprises:
randomly generating a Random character string containing letters and data by utilizing a Random function;
and taking the randomly generated character string as a working key.
4. A method of queued data storage analysis as claimed in claim 1, wherein the obtaining data in the MQ server by the data collection center comprises:
the data collection center sends a remote login request to the MQ server side;
The MQ server side sends a verification information request to the data collection center according to the remote login request;
The data collection center sends verification information to the MQ server side according to the verification information request;
The MQ server side judges whether the verification information sent by the data collection center is correct or not according to the verification information, and if yes, the MQ server transmits a plurality of decrypted data to the data collection center; if not, waiting for a new telnet request.
5. A method of queued data storage analysis as claimed in claim 1, wherein the central database performs data backup of the data and transmits the data to a data processing center, comprising:
Creating a backup catalog and a script catalog in a central database and giving permission;
editing a script program according to the backup catalog and the script catalog;
Editing a command script according to the script program so as to backup data according to the command script, and transmitting the data to a data processing center.
6. The method for analyzing the queue type data storage of claim 1, wherein the steps of ranking according to the traffic usage of the client, finding the industry to which the corresponding client company belongs, ranking the industries in a classification manner, obtaining the top-ranked industry data information, and transmitting the data to the service processing center, include:
Extracting the customer traffic usage data in the sorted data, and sorting the customer traffic usage data through a sort function;
the ordered customer business usage data are respectively found out to correspond to industries to which the customer formulas belong, and the industries are classified and combined;
and ranking websites for each type of industry name according to the classified and combined customer service usage data, so as to obtain the top-ranked industry data information, and transmitting the data to a service processing center.
7. The method for analyzing the queue type data storage of claim 1, wherein the service processing by the service processing center according to the sorted data comprises the following steps:
Positioning customer groups for industry customers according to the industry data information in the analyzed data, and adjusting the operation strategy;
According to the transfer duration data in the analyzed data, optimizing the communication line of an operator and improving the user experience for the length of transfer seat time and the call completing rate;
according to the platform frequency and peak period data information in the analyzed data, the formula investment of the platform with less resource consumption is reduced, the busy platform is used for early warning and timely energy expansion in advance, and the stable operation of the platform is ensured.
8. A queued data storage analysis system, the system comprising:
the acquisition module is used for acquiring a plurality of landing platform data;
The transmission module is used for transmitting the data of the plurality of landing platforms to the MQ server through a transmission control protocol;
The data collection center is used for acquiring the data in the MQ server and transmitting the data to the central database;
the central database is used for carrying out data backup on the data and transmitting the data to the data processing center;
The data processing center is used for analyzing the data through the big data analysis engine and transmitting the analyzed data to the service processing center;
the service processing center is used for carrying out service processing according to the analyzed data;
The data processing center analyzes the data through the big data analysis engine and transmits the analyzed data to the service processing center, and the method comprises the following steps:
according to the customer traffic usage ranking, finding the industry to which the corresponding customer company belongs, and then carrying out industry classification ranking to obtain the front-ranked industry data information, and transmitting the data to a service processing center;
Averaging again according to the time difference between the calling incoming time and the called answering time of all the connected calls, so as to obtain average switching duration data, and transmitting the data to a service processing center;
Counting the line usage amount in each minute in the platform, comparing the peak value with the usage amount threshold value configured by the platform, thereby acquiring the data information of the platform frequency and peak period, and transmitting the data to a service processing center;
the statistics of the line usage amount per minute in the platform, the comparison of the peak value and the usage amount threshold value configured by the platform, thereby obtaining the data information of the platform frequency and peak period, and transmitting the data to the service processing center, comprises the following steps:
Step A1, acquiring each data analysis matrix, wherein the data analysis matrix comprises current line usage data per minute and current line peak value taking data, and calculating current parameters of each data analysis matrix according to the current line usage data per minute and the current line peak value taking data by using the following formula:
Wherein J K represents the current parameters of the data analysis matrix, Data information representing current line usage per minute,/>The method comprises the steps of representing peak data information of a current line, wherein k represents iteration times, k epsilon [1, T ], T represents preset iteration times, alpha and beta represent parameter adjustment coefficients, the value range of alpha and beta is 0-1, i represents the number of rows of a data analysis matrix, i=1, 2, … m, j represents the number of columns of the data analysis matrix, and j=1, 2, … n;
Step A2, analyzing the current parameter J K of the matrix according to the following data, and calculating the frequency of the landing platform by using the following formula:
wherein F represents the frequency of the landing platform, pi represents a natural constant, and tan represents a tangent function;
and step A3, comparing the frequency F of the landing platform with a preset threshold, and when the frequency F of the landing platform is higher than the preset threshold, acquiring peak data information under the frequency F and transmitting the data information to a service processing center.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110072351.3A CN112765109B (en) | 2021-01-20 | 2021-01-20 | Queue type data storage analysis method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110072351.3A CN112765109B (en) | 2021-01-20 | 2021-01-20 | Queue type data storage analysis method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112765109A CN112765109A (en) | 2021-05-07 |
| CN112765109B true CN112765109B (en) | 2024-05-28 |
Family
ID=75703449
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110072351.3A Active CN112765109B (en) | 2021-01-20 | 2021-01-20 | Queue type data storage analysis method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112765109B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115297163A (en) * | 2022-07-04 | 2022-11-04 | 上海群之脉信息科技有限公司 | Shopping guide background operation optimization system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016054908A1 (en) * | 2014-10-10 | 2016-04-14 | 中兴通讯股份有限公司 | Internet of things big data platform-based intelligent user profiling method and apparatus |
| CN109376149A (en) * | 2018-08-22 | 2019-02-22 | 中国平安人寿保险股份有限公司 | By the method, equipment and storage medium of data landing to data platform |
| CN110210705A (en) * | 2019-04-29 | 2019-09-06 | 德邦物流股份有限公司 | A kind of data analysing method and system |
| CN110457153A (en) * | 2019-07-18 | 2019-11-15 | 北京顺丰同城科技有限公司 | Data check processing method and processing device |
| CN111221831A (en) * | 2019-12-26 | 2020-06-02 | 杭州顺网科技股份有限公司 | Computing system for real-time processing of advertisement effect data |
| CN111770106A (en) * | 2020-07-07 | 2020-10-13 | 杭州安恒信息技术股份有限公司 | Method, device, system, electronic device and storage medium for data threat analysis |
-
2021
- 2021-01-20 CN CN202110072351.3A patent/CN112765109B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016054908A1 (en) * | 2014-10-10 | 2016-04-14 | 中兴通讯股份有限公司 | Internet of things big data platform-based intelligent user profiling method and apparatus |
| CN109376149A (en) * | 2018-08-22 | 2019-02-22 | 中国平安人寿保险股份有限公司 | By the method, equipment and storage medium of data landing to data platform |
| CN110210705A (en) * | 2019-04-29 | 2019-09-06 | 德邦物流股份有限公司 | A kind of data analysing method and system |
| CN110457153A (en) * | 2019-07-18 | 2019-11-15 | 北京顺丰同城科技有限公司 | Data check processing method and processing device |
| CN111221831A (en) * | 2019-12-26 | 2020-06-02 | 杭州顺网科技股份有限公司 | Computing system for real-time processing of advertisement effect data |
| CN111770106A (en) * | 2020-07-07 | 2020-10-13 | 杭州安恒信息技术股份有限公司 | Method, device, system, electronic device and storage medium for data threat analysis |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112765109A (en) | 2021-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9830111B1 (en) | Data storage space management | |
| US8239535B2 (en) | Network architecture with load balancing, fault tolerance and distributed querying | |
| CN101621541A (en) | Method and apparatus for distributed application context-aware transaction processing | |
| CN111382985B (en) | Integrated pushing system and working method for message to be handled | |
| CN107798037A (en) | The acquisition methods and server of user characteristic data | |
| CN103999077A (en) | System and method for monitoring and managing data center resources in real time incorporating manageability subsystem | |
| CN109039817B (en) | Information processing method, device, equipment and medium for flow monitoring | |
| CN109558294A (en) | Application Monitoring management method, device, computer equipment and storage medium | |
| US11356535B2 (en) | System and method for asset management and integration | |
| CN112765109B (en) | Queue type data storage analysis method and system | |
| CN103490978A (en) | Terminal, server and message monitoring method | |
| CN101330431A (en) | Method and system for storing instant information | |
| CN110222054A (en) | A kind of method, apparatus, terminal device and storage medium improving retrieval rate | |
| CN108628954B (en) | Mass data self-service query method and device | |
| CN113448926A (en) | Block chaining operation and maintenance management system and method | |
| CN103297477A (en) | Data collecting and reporting system, data processing method and proxy server | |
| CN115629880A (en) | Log desensitization method, device, equipment and storage medium | |
| CN111401819B (en) | Intersystem data pushing method and system | |
| KR102093764B1 (en) | Managment server for managing the server and storage | |
| CN104506424B (en) | The notification method of linkman state and notice device in instant messaging | |
| KR20210050827A (en) | An extraction-system using dispersion deep learning information analysis management based cloud and method of it | |
| CN112631991B (en) | File migration method and device | |
| US8606813B1 (en) | System and method for function selection in analytic processing | |
| US20220382775A1 (en) | Employee compensation manager | |
| CN103347061B (en) | Based on the strange land electronic data recovery system of corporate intranet |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |