TW201738776A - Real-time streaming record data analysis system and method using a computed result obtained by a computing process as the system and method for increasing the future computing efficiency - Google Patents

Abstract

This invention relates to a real-time streaming record data analysis system and method. The real-time streaming record data analysis system is composed of a plurality of user devices, a plurality of online webpage servers, a plurality of online database servers, a plurality of manager devices, a recorded data collecting device, a plurality of distributed databases, a recorded data analysis module, a data mining master module, a distributed computing device, a rapid access database and a combined node device, wherein the recorded data collecting device, the plurality of distributed databases, the recorded data analysis module, the data mining master module, the distributed computing device, the rapid access database and the combined node device are arranged in the system. By using the real-time streaming record data analysis system, recorded streaming data can be distributively subjected to synchronous computing by different mining modules after being encrypted or not encrypted, and a decrypted analysis result is produced and known by a manager, and even more, a computed result obtained by a computing process can be used as the system and method for increasing the future computing efficiency.

Description

Instant streaming record data analysis system and method

The invention relates to an instant stream recording data analysis system and method.

There are a number of conventional techniques for recording or analyzing systems and methods for streaming data, but each has its own strengths and weaknesses.

First, there is a cross-layer log record tracking system and method that can obtain different log data sources, record a large amount of log data and access trajectories, and serve as a crime tracing. However, although this method can record log data, it cannot analyze the records, and the production analysis results are for the administrator's reference.

The log data recording method is a method for instantly compressing and storing log data for a specified image; however, although it is possible to record log data, it is impossible to analyze the record and produce the analysis result.

Finally, there is a method for measuring communication network traffic, which can record packets in a side-by-side manner and adapt to different types of packets for recording. However, the same analysis and production analysis results are also not available to the administrator.

Therefore, it should be known that providing an instant streaming data system that can analyze and produce results, not just records, is a technology that is urgently needed in this field.

The present invention provides an instant stream record data analysis system consisting of a plurality of user devices external to the system, a plurality of online web server servers, a plurality of online database servers, and a plurality of manager devices, and the system of the present invention. A record data collecting device, a plurality of distributed data bases, a record data analysis module, a data exploration main module, a distributed computing device, a cache database, and a combined node device.

The plurality of external user devices can be operated by the user to connect to the online web server to request the web service and related information and applications from the online web server.

The plurality of external online web servers mainly provide network services and related information to external user devices by accessing relevant data to the online database server according to the network service requirements transmitted by the user equipment, and are also applicable to all network devices. The service requires a record and the network service usage record is transmitted to the record data processing device for resolution and storage.

Multiple external online database server devices: Online database server devices can receive database operation requirements for online web server devices and respond to relevant information as required. And for each database operation requirements record, and the database operation record can be transmitted to the record data collection device for analysis and storage.

A record data collection device for parsing records from online web server and online database server and parsing them according to the format of the records and storing them in a decentralized database. The data collection device can also be selectively provided. The encryption function includes at least one private key, at least one public key, and at least one arbitrary integer value, that is, after parsing and then transmitting the above three pairs The data is encrypted and stored in a decentralized database.

The plurality of decentralized databases can store the parsed network service usage records and database operation records. When the distributed computing device performs decentralized operations and record analysis, the record data is provided for the decentralized computing device operations.

A plurality of external manager devices, the manager operation manager device is connected to the record data analysis module, and the record data analysis module is connected to the data exploration main module to select a suitable exploration sub-module, and then assigned to the distributed model. The arithmetic unit performs calculations.

A record data analysis module is manually or automatically connected to the data master exploration module and selects the appropriate exploration sub-module, and then assigned to the distributed computing device for calculation, recording the data analysis module and The combined node device obtains the operation result.

A data master exploration module may include a plurality of exploration sub-modules for operation and analysis by a distributed computing device.

A plurality of distributed computing devices can obtain record data from the distributed database, and assign tasks to a plurality of node devices and distributed computing modules according to the selected data mining module, respectively performing operations and analysis, and decentralized computing devices The calculation result can be temporarily stored in the cache database, and the distributed computing device can also selectively perform the function of ciphertext calculation for calculating the ciphertext data.

A plurality of cache databases are configured to store the required analysis results or related parameters of the discrete computing device for temporarily storing the respective record data for later acceleration calculation applications.

A combined node device for extracting each operation result of the distributed computing device for integration and analysis, and then transmitting the analysis result back to the record data The analysis module, in the same way, the combined node device may additionally be provided with a decryption function, which is provided with each of the private key corresponding to the record data collecting device, the public key and the arbitrary integer value, so as to be distributed The result obtained by the ciphertext calculation of the computing device is decrypted, and the plaintext data is sent back to the record data analysis module after decryption.

The method for analyzing the real-time streaming record data of the present invention comprises the following six steps.

Recording online data steps: A record data collection device will collect and store network service requests and reply records from external online users from external online web servers and external online database servers.

The process of depositing into a decentralized database: The record data collection device stores the aforementioned network service requirements and reply records to a plurality of decentralized databases.

Step of selecting the data exploration module: The external manager is connected with a record data analysis module to select and use the data search module from the plurality of data search modules through the record data analysis module.

The assignment of the work to the distributed computing device step: the data exploration and exploration master module will operate the user's network service request and the reply record according to the exploration sub-module selected by the external manager to assign the plurality of distributed computing devices.

The process of temporarily storing to the cache database device: the calculation results produced by each of the distributed computing devices will be temporarily stored in a cache database for future analysis and use.

The result of the return and display operation: one of the combined node devices connected to each of the distributed computing devices integrates the operation result into an analysis result, and transmits the result to the external manager device through the record data analysis module to present the analysis result to external use. By.

The method for analyzing real-time streaming data of the present invention also has The function of encrypting and decrypting, wherein the record data collecting device encrypts the network service request and the reply record data by using at least one private key, a public key and an arbitrary integer value, and each of the distributed computing devices Directly calculating the network service request and reply record data of the ciphertext state, and generating the operation result, the combined node device having the corresponding private key, the public key and the arbitrary integer value to decrypt the analysis result Provided to external managers.

In summary, it can be seen that the real-time streaming data analysis system and method of the present invention can instantly record and analyze online network service request record data and database operation record data, and can also select different data exploration sub-modules. Record data analysis.

The invention further combines a distributed computing device and a distributed database to perform distributed computing in an instant, and an additional cache database provides calculation results and related parameters of the temporary distributed computing device to enhance the subsequent analysis. effectiveness.

Moreover, the present invention can selectively utilize the encryption and decryption method to encrypt the data and store it in the distributed database, so that the data security can be ensured, and the data can be operated in the ciphertext state, so that the operation efficiency is improved. .

101‧‧‧User equipment

102‧‧‧Online web server

103‧‧‧Online database server

104‧‧‧Manager equipment

105‧‧‧Record data collection device

106‧‧‧Distributed database

107‧‧‧Record data analysis module

108‧‧‧Data exploration main module

109‧‧‧Distributed computing device

110‧‧‧Cache Database

111‧‧‧Combined node device

1081‧‧‧Recent neighbor exploration sub-module

1082‧‧‧Multivariate weighted linear regression exploration sub-module

S201~S208‧‧‧ method steps

1 is a structural diagram of an instant stream recording data analysis system of the present invention.

FIG. 1 is a schematic diagram showing the steps of a method for encrypting and analyzing an instant stream record data according to the present invention.

The present invention will be further described in the following embodiments in conjunction with the drawings. In view of the increasing demand for data processing and analysis in real time, the present invention provides an instant stream recording data analysis system, and the overall system required for the operation of the present invention. The architecture is as follows, as shown in FIG. 1: a plurality of user devices 101 outside the system, a plurality of online web server 102, a plurality of online database servers 103, and a plurality of manager devices 104, and the system of the present invention. a record data collecting device 105, a plurality of distributed data banks 106, a record data analysis module 107, a data search main module 108, a plurality of distributed computing devices 109, a cache database 110, and a combined node device The data exploration main module 108 further includes a nearest neighbor exploration sub-module 1081 and a multi-weighted linear regression exploration sub-module 1082.

The record data collecting device of the present invention can be implemented by using tools such as Splunk or Logstash, which are used to collect the recorded data transmitted by the online web server and the online database server.

The decentralized database of the present invention can be developed in combination with NoBase-based HBase or MongoDB, and its function is to record related data from a permanent record storage device.

The distributed computing device of the present invention uses the MapReduce development model in Hadoop or MongoDB to efficiently segment and merge data, which can greatly improve the speed of computation and analysis.

In addition, the present invention further provides a cache database for collecting the heterogeneous data sources and performing real-time analysis and calculation in response to the load demand of the instantaneous large amount of data operations, and avoiding repeated operations and feeding back the data to the external management. By.

More specifically, the foregoing instant stream recording data of the present invention is divided into The various parts of the system can be explained as follows: external plural user equipment: the user equipment can be an electronic device such as a personal computer, a tablet, a smart phone, a personal digital assistant, an in-vehicle device, etc., and the devices can be used on the device. Browser components (such as Internet Explorer, Chrome, Firefox, Safari, etc.) or other web-connected applications to connect to online web servers and request web services via online web servers and Information application needs.

External multiple online web server: These online web servers can be developed through web server components such as Microsoft Internet Information Services (IIS) and Apache to provide external web users with web services that provide multiple web services. The web server can obtain the requested data from the online database server according to the network service requirements and information application requirements transmitted by the external user equipment, and then provide the corresponding network service and information to the external user equipment. The online web server also records each network service request, and can store the network service usage record data (such as IIS Log or Apache Log) according to the server component and transmit the network service usage record data to the record data collecting device. Analyze and store.

Multiple online database servers: Online database server devices use Microsoft SQL Server, MySQL, Oracle DB, IBM DB2, PostgreSQL and other database server components for implementation development, and provide a variety of database operations (including at least There are new, modified, deleted, inquired, etc.) to enable the online web server to access; can receive the database operation requirements of the online web server device, and reply to relevant information according to the requirements. And for each database operation requirements record, and can generate database operation records according to different database components, and transfer the database operation records to the record data processing device for analysis and storage.

A record data collection device: the record data collection device must use Splunk, Logstash and other records and analysis components for development, and provide various record data analysis modules (including at least the network service use record data analysis module and database operation) The record data analysis module is used by the user; wherein the network service use record data analysis module has at least IIS Log or Apache Log parsing function to parse the record from the online web server; in addition, the database operation record data analysis The module also includes an analysis function such as Microsoft SQL Server Log, which can parse the records from the online database server; when the record data collection device is parsed, it is parsed according to its record format and stored in a distributed database; For example, the example shown in the following Table 1 is an IIS Log record data, and the record data collecting device parses the data to obtain the record date of the record data as 2015-08-18 and the record time is 09:12: 15. The client IP is 10.144.198.130, the server IP is 10.144.192.1, the link nickname is 80, and the required network. The service system is "/index.html", the response status code is 200, the client browser is Mozilla/4.0+ (compatible; MSIE+5.5; +Windows+2000+Server), and the record data collection device is also optional. The encryption function includes at least one private key, at least one public key, and at least one arbitrary integer value, that is, the data is parsed and then encrypted by the above three, and then stored in the distributed database.

Multiple decentralized databases: Decentralized databases can be developed using decentralized database elements such as HBase and MongoDB for storing and manipulating huge amounts of data and with clusters to support backup data analysis and processing; The distributed database mainly stores the network service usage records and database operation records analyzed by the recorded data collection device, and continuously provides the record data to maintain the calculation process when the distributed computing device performs the operation analysis.

A plurality of manager devices: the manager device is a device such as a personal computer, a tablet, a smart phone, a personal digital assistant, etc., and the device has a browser component (a browser such as Internet Explorer, Chrome, Firefox, Safari, etc.) or the like. The connectable application is connected to the record data collection device and connected to the data search main module via the record data collection device to select a suitable exploration sub-module to be assigned to the distributed computing device for calculation, and finally, The combined node device integrates the operation result and returns the data analysis module, and the data analysis module replies to the external manager device.

A record data analysis module: the record data analysis module is a server with network service, which can interface with external manager equipment, data exploration main module, and combined node device through a network service interface. And receiving data; the record data analysis module may be manually or automatically connected to the data exploration main module by the manager to select a suitable exploration sub-module and assign it to the distributed computing device for calculation, and to the combined node The device obtains the result of the operation.

A data exploration module: the main module of data exploration is also The server of the network service can interface with the exploration sub-module and the distributed computing device via the network service interface and transmit and receive the data, and can include a plurality of exploration sub-modules for operation and analysis by the distributed computing device; Among them, at least the nearest neighbor exploration sub-module, a decentralized computing module calculated by the logic of k nearest neighbor method (k-Nearest Neighbors Method), and a multiple linear regression exploration sub-module, which is multivariate linear The logic of the Multi Factor Line Regression Method is used to calculate the decentralized computing module. Therefore, the data exploration module can be assigned to the distributed computing device for computational analysis according to the selected exploration sub-module.

At least one distributed computing device: the distributed computing device uses a decentralized computing unit such as Hadoop or MongoDB for development, and includes at least a plurality of node devices and a plurality of distributed computing modules to analyze huge amounts of data. The node device may generate a plurality of distributed computing modules according to the data exploration module selected by the record data analysis device, and obtain the record data from the distributed database device, and assign the data to the distributed computing module for analysis; The module performs calculation and analysis of the record data according to the selected exploration sub-module. For example, the MapReduce distributed computing module provided by Hadoop or MongoDB respectively performs distributed operation according to the task of the assigned sub-module, and then integrates the operation result into the combined node device, and each distributed computing device is also selected. The succinct setting has the function of ciphertext calculation, which can be used to calculate the data in the ciphertext state.

At least one cache database: the cache database is implemented by using an associated database or a non-associated database component, and is used for storing the distributed computing device to temporarily store the results of each record data analysis and related parameters to accelerate the operation. For example, after the decentralized operation of the nearest neighbor exploration sub-module is performed by the distributed computing device, the plurality of similarly recorded data will be obtained. And the record data is transmitted to the cache database for storage. In the subsequent calculation, the most similar plurality of record data can be obtained from the cache database for comparison analysis. For another example, if the distributed computing device performs the decentralized operation of the multiple linear regression exploration sub-module, the linear regression model parameters (including the slope or intercept) can be generated and stored in the cache database, and the real-time computing process is followed. The linear regression model parameters stored in the cache database can be utilized, and the cache database is further written into the new record data or the old record data is deleted to avoid redundant time calculation, which can greatly improve the overall operation efficiency.

A combined node device: the combined node device is also a server with a network service, and is connected to a record data collecting device and a distributed computing device via a network service interface to transmit and receive data, which is generated by a distributed computing device. The calculation results are integrated and analyzed, and the analysis results are transmitted back to the record data analysis module. In addition, the combined node device can also have a decryption function, and stores a private key and a public key corresponding to the record data collection device. And any integer value, so that the result obtained by the ciphertext calculation of the distributed computing device is decrypted, and the plaintext data is sent back to the record data analysis module after decryption.

Another embodiment disclosed below is also related to the instant stream recording data analysis system of the present invention. The system architecture includes at least one record data collecting device, a plurality of distributed data bases, a data exploration main module, and a plurality of distributed The computing device, a combined node device, and a plurality of cache databases; wherein the data master exploration module includes a nearest neighbor exploration sub-module, which can perform a network record data analysis calculation using the k nearest neighbor method to generate positioning information. The system operates as described below.

The system includes a record data collecting device for collecting the latitude and longitude coordinate data of the smart phone (ie, the training position, in the embodiment, m positions) and the base station signal strength set data, recording the data collecting device, and recording and analyzing the above. Data, which records each training position ( L = { l ₁ , l ₂ , ..., l _m }) and the base station signal strength set data corresponding to the training position ( c _i = { c ₁ ⁱ , c ₂ ⁱ , ..., c _n ⁱ }) in a decentralized database; where c _j ⁱ represents the signal strength of the jth base station in the set, j = 1 ,..., n (in the embodiment there are n bases) Taiwan); then, when the smart phone moves, the smart phone can measure and report the base signal strength set ( r ={ r ₁ , r ₂ ,..., r _n }) nearby, and will be The data master exploration module, the distributed computing device, and the cache database use the nearest neighbor exploration sub-module to calculate the base signal intensity set r and all the positions in the distributed database and their signal intensity sets to cross-match to estimate the wisdom. The possible location of the phone at the time.

The system includes a plurality of decentralized database devices for storing each training position ( L = { l ₁ , l ₂ , ..., l _m }) and its corresponding base station signal strength set data ( c _i = { c ₁ ⁱ , c ₂ ⁱ ,..., c _n ⁱ }); when the decentralized computing device performs calculations and record analysis, the decentralized database provides its record data.

The system further comprises a data master exploration module, which has at least one nearest neighbor exploration sub-module for evaluating the position loc(r) of each signal intensity set r ; in this embodiment, applying the Euclidean distance (Euclidean Distance) operation method, using the following formula (1) to combine the signal strength ( r = { r ₁ , r ₂ , ..., r _n }) with each position l _i and its signal in the database The intensity set ( c _i ={ c ₁ ⁱ , c ₂ ⁱ ,..., c _n ⁱ }) is calculated by the distance (), and then the Euclidean distance operation is also performed for each training position, and then the formula is passed ( 2) Find the closest position h _{1 of the} signal strength and the other closest k positions (ie { h ₁ , h ₂ ,..., h _k }), and the data master exploration module will explore the nearest neighbors. The sub-modules are assigned to the distributed computing device for execution.

The plurality of distributed computing devices may include a plurality of node devices, and each node device shall correspond to at least one distributed computing module, and the distributed computing module may perform operations according to the exploration sub-module selected by the data mining module. In this embodiment, the distributed database has a total of m positions (that is, there are m pen data to be compared), so the m pen data can be evenly distributed to each node device and then dispersed by each node device. The operation module respectively executes the nearest neighbor exploration sub-module, and obtains the closest total k positions (ie, { h ₁ , h ₂ , . . . , h _k }), and the closest k position information is obtained by multiplexing. It will then be transmitted to the combined node device for the combined node device to compute the final location information.

As described above, the combined node device can receive the information from the operation of the distributed computing device for integration and analysis. In this embodiment, the combined node device receives the distributed operation under a plurality of node devices. The module calculates the obtained k positions separately, and then compares them from the set to obtain k absolute close positions, and uses the following formula (3) to generate a signal intensity set ( r = { h ₁ , h ₂ ,..., h _k }) corresponding position information l(r) ;

The plurality of cache databases are mainly used to store the results of the operations performed by the distributed computing device and related parameters for subsequent analysis to quickly obtain efficiency; in this embodiment, the cache database will be obtained by each node device. The closest q × k position information (where q × k is less than m and q is a positive integer) and its corresponding base station signal set is stored and stored. If necessary, the base station signal strength of the same smart phone report must be analyzed. When collecting, the closest q × k position information in the cache database and its corresponding base station signal set can be analyzed without re-matching the original m pen data. In addition, the data can be used to analyze the speed of smart phone movement. For example, when the smart phone moves slowly or at rest, the q value can be set to a minimum value (such as 1), and when the smart phone moves quickly, The q value can be set to a larger value.

Another embodiment disclosed in the following is also an instant stream record data analysis system of the present invention. The system architecture includes at least one record data collection device, a plurality of distributed data bases, a data exploration main module, and a plurality of distributed An arithmetic device, a combined node device, and a plurality of cache databases; wherein the data master exploration module includes a recent multiple linear regression exploration sub-module, which is capable of performing a traffic log data analysis calculation using a multiple linear regression module The traffic forecasting information is as follows: The data exploration module device includes at least a multiple linear regression module, and the multivariate linear regression module is used for traffic record data analysis, and the traffic prediction information is produced. As described below.

The system comprises a record data collecting device for collecting the arrival time information of the in-vehicle device set on the cleaning vehicle, and parsing the station time information by the record data collecting device to calculate the clean car route between the station and the station. travel time, for example: the first is intended to indicate the information of the pen r th inj removal station to the travel time between the removal of the stations in a T _{inj ^r, in;} device history data collection system to travel to each of the preceding calculation Time collections are stored in a decentralized database for subsequent analysis.

The system includes a plurality of distributed database devices. In this embodiment, the distributed database is developed using distributed database components such as HBase and MongoDB to store travel time between each station and the station.

The system further comprises a data master exploration module, which has at least a multiple linear regression exploration sub-module for calculating the correlation (such as slope, intercept, etc.) of the travel time between the stations and the stations; In this embodiment, k weighted linear regression models are generated by analyzing the m- pen data in the historical data. ( t ^r _inj,in ) is an example; the predicted travel time from the in- th cleaning point to the i- th clearing point ( ) Can be weighted using multiple linear regression model (equation (4)) is obtained for operation, you will reach the main point of removal in accordance with a first k removal in a removal point of the point in the execution stage Travel time (i.e., _{{t in -1, in, t} in- 2, in, ..., t ink, in}) weighted multivariate linear regression model of cooperative have been trained to predict the removal of a point in the first The predicted travel time of the i clearing points (as shown in equation (5)).

The plurality of distributed computing devices may include a plurality of node devices, and each node device shall correspond to at least one distributed computing module, and the distributed computing module may perform operations according to the exploration sub-module selected by the data mining module. In this embodiment, the models of the multivariate weighted linear regression exploration sub-module mostly use the addition and multiplication factors and have the characteristics of the combination law. For example, the task can be evenly distributed to each node according to the number of historical data m pens. The device, respectively, performs a multivariate weighted linear regression on the distributed computing module in each node device, or distributes the k weighted linear regression models to be distributed on each node device, and the distributed computing mode in each node device. Each group will perform each multivariate weighted linear regression exploration sub-module; and after the operation is completed, the distributed computing device will slope the multi-weighted linear regression models (eg ), intercept (such as ), as well as weights (such as ) Stored separately in the cache database for later analysis.

As described above, the combined node device can receive the information obtained from the operation of the distributed computing device and integrate it to generate the analysis result. In this embodiment, the combined node device can receive the plurality of node devices and calculate the calculated k weighted linear regression model and its parameters (i.e. slope, intercept and weight), and then in a second operation to generate the predicted travel time to the point of removal of the i-th point through the removal equation (5) of the embodiment.

In this embodiment, the plurality of cache databases mainly store the slopes, intercepts, and weights of the respective multivariate weighted linear regression models calculated by the respective node devices in each of the cache databases for use as analysis data. In addition, when the data input subsequently has a change, the multivariate weighted linear regression model should be mainly additive and multiplication and may have mathematical characteristics such as combination law. Therefore, if it is used with the historical data temporarily stored in the cache database, only Adding new data or subtracting deleted data can quickly adjust the slope, intercept or weight used by the multivariate weighted linear regression model instead of recalculating the original m- pen data to increase efficiency.

Another embodiment disclosed in the following is another embodiment of the method for analyzing real-time streaming data of the present invention, which mainly adds a live string with ciphertext computing function on the original real-time streaming data analysis system. The flow record data analysis method, as shown in FIG. 2, the method mainly includes eight steps, respectively step S201: record online data step, step S202: data encryption step, step S203: deposit into the distributed database step, step S204: selecting a data exploration sub-module step, step S205: assigning a work to the distributed computing device and performing a ciphertext calculation step, step S206: temporarily storing the operation result to the cache database step, step S207: returning and decrypting steps, and Step S208: eight steps of displaying the result step, etc., the detailed implementation of each step will be described in the following embodiments.

Step S201: Recording online data step: The record data collecting device collects and stores the service request and response record of the external online web server and the external online database server into a distributed database; for example, the vehicle-mounted device 1 (external user) Equipment) arrive at Site 1, Site 2, Site 3 at 09:00:00, 09:03:20, 09:07:00, etc.; Onboard Device 2 (External User Equipment) at 10:00 :00, 10:04:00, 10:08:10, etc. arrive at Site 1, Site 2, Site 3 respectively; in addition, the in-vehicle device 3 (external user equipment) at 11:00:00, 11: 03:30, 11:07:20 and other three points arrive at the same site 1, site 2, site 3; and, on-board machine 4 (external user equipment) at 12:00:00, 12:03: At 40 o'clock, the two arrive at Site 1, Site 2, as shown in Table 2 below.

The following is Table 2, which indicates the arrival time of the vehicle-mounted machine:

When the above four in-vehicle devices (external user devices) arrive at each site, the location information and time information of the in-vehicle device are transmitted to the external online web server and the external online database server via the intermediary software (for example, RESTful API). And the record data collection device will store and analyze these location information and time information records to calculate the travel time between the station arrival time, for example: the vehicle-mounted device 1 from site 1 to site 2 The travel time ( t _{1 , 2} ) is 200 seconds, and the travel time ( t _{2 , 3} ) from station 2 to station 3 is 220 seconds, as shown in Table 3 below.

The following is Table 3, which is used to indicate the travel time between stations (in seconds):

Step S202: Data encryption step: after the record data collecting device collects the location information and the time information record of the external online web server and the external online database server, the data is encrypted by an encryption algorithm; the record data collecting device The travel time between the station and the station should be calculated, and then t _{1 , 2} multiplied by the value of t _{2 , 3 and} the value of t _{1 , 2} square, respectively, to generate the relevant parameter values of the arrival time, as shown in Table 4 Shown.

The following is Table 4, showing the relevant parameter values for the arrival time:

In the data encryption step, the record data collecting device may then use a predetermined one of the private key p, a public key q, an arbitrary integer value z, and the like, and then perform the relevant parameter values by the following formula (6). Encryption, in this embodiment, it is assumed that the value of the private key p is 39,916,801, the value of the public key q is 112,909, and the value of any integer value z is 7, and the result of the calculation is as follows: the value of the original plaintext data 44,000 is encrypted by parameter values and formulas to obtain ciphertext data 279, 461, 607, and other example results are organized as shown in Table 5 below.

Formula (6): f(x)=(x+p×z) mod(p×q), where x is the original correlation parameter value, and mod operation is the divisor of the value of the previous term as the divisor of the value of the previous term. Take the remainder of the modulus division operation.

The following is Table 5, showing the relevant parameter values after encryption:

Step S203: depositing into the distributed database step: the record data collecting device of the present invention can selectively store the data in a plain text or ciphertext manner into a plurality of distributed database, and in the embodiment, the record data collecting device The encrypted parameter values presented in Table 5 are stored in each of the distributed In the database, the function of the present invention for storing ciphertext in the database is a method for effectively preventing the risk of the data being immediately leaked when the database is intruded or the data is stolen.

Step S204: Step of selecting the data exploration sub-module: the external manager can connect to the record data analysis device through the external manager device, and access the data exploration main module through the record data analysis device to select the external manager to use. The exploration sub-module, in this embodiment, the external manager chooses to use the multiple linear regression sub-module, so the subsequent examples will continue to use the multiple linear regression sub-module as the main tool for analysis and calculation.

Step S205: assigning the work to the distributed computing device and performing the ciphertext computing step: the data exploration main module may perform the operation analysis according to the exploration sub-module selected by the external manager to assign the plurality of distributed computing devices, and each of the A plurality of distributed computing modules under the distributed computing device calculate the allocated recorded data, and the distributed computing device processes the recorded data in the ciphertext state by directly processing the ciphertext.

Wherein, each of the distributed computing devices can use the multiple linear regression sub-modules selected by the external manager to pass the operation requirements of the aforementioned formula (4) and formula (5), etc., and the plurality of subordinates The decentralized computing module performs operations to add the required parameter values, and the summed results are shown in Table 6 below; in this embodiment, the parameters of a set of regression sub-modules a and b are calculated, but The application of the present invention is not limited to this example. Therefore, each of the distributed computing devices can perform a large number of regression sub-module parameter operations in parallel using each of the distributed computing modules.

The following is the sixth table, showing the sum of the relevant parameter values after encryption:

Step S206: Temporarily storing the operation result to the cache database step: each of the distributed computing device operation results are temporarily stored in the cache database, and the cache database may also be plural, and the purpose of the temporary storage system calculation result is mainly It is used to improve the efficiency of data analysis in the future; in this embodiment, the data of the in-vehicle device 1, the in-vehicle device 2, and the in-vehicle device 3 have been added, and the total result will be temporarily stored in the cache database device. If the follow-up needs to be used, the total result can be directly applied, and it is not necessary to re-add the data of the in-vehicle device 1, the in-vehicle device 2, and the in-vehicle device 3, which can save a lot of time.

Step S207: Backhaul and decryption steps: each of the distributed computing devices will return the calculated result to the combined node device, and the combined node device decrypts and integrates the ciphertext data into the analysis with the same parameter value as the encryption method. result. After the combined node device receives the operation result of the distributed computing device, it may use the same private key p, a public key q, an arbitrary integer value z as the record data processing device, and decrypt using the formula (7). In this embodiment, the private key p is set to 39,916,801, the public key q is 112,909, and the arbitrary integer value z is 7. For example, the ciphertext 838, 405, 121 after the summation results in the plaintext is 152,300 after decryption, as shown in Table 7.

Equation (7): g(x) = (x) mod(p), where mod is also a modular division operation.

The following is Table VII, showing the sum of the relevant parameter values after decryption:

Through the decrypted aggregate data shown in Table 7 above and the known number of data sheets (m=3), the a and b parameters can be calculated using the above formula (4), as shown in the following formula (8). ), and then calculate the travel time required for the in-vehicle device 4 from the station 2 to the station 3 by the following calculation formula (9), and the result It is estimated that it is about 236 seconds, so it is inferred that the predicted arrival time of the in-vehicle device 4 arriving at the station 3 should be 12:07:36.

Step S208: displaying the result step: the combination node device transmits the operation result to the record data analysis module, and then the record data analysis module notifies the external manager device to present the analysis result on the external manager device; After the combined node device calculates the prediction result, the prediction result is transmitted to the record data analysis module, and the record data analysis module notifies the external manager device to present the onboard device 4 to the external manager device. Site 3's predicted arrival time is 12:07:36 for external managers.

In summary, the present invention is innovative in terms of technical ideas, and also has various functions that are not in the prior art, and has fully complied with the statutory invention patent requirements of novelty and progressiveness, and has filed a patent application according to law, and invites you to approve the invention. The patent application was inspired to invent, and it was a matter of feeling.

101‧‧‧User equipment

102‧‧‧Online web server

103‧‧‧Online database server

104‧‧‧Manager equipment

105‧‧‧Record data collection device

106‧‧‧Distributed database

107‧‧‧Record data analysis module

108‧‧‧Data exploration main module

109‧‧‧Distributed computing device

110‧‧‧Cache Database

111‧‧‧Combined node device

1081‧‧‧Recent neighbor exploration sub-module

1082‧‧‧Multivariate weighted linear regression exploration sub-module

Claims (8)

An instant stream record data analysis system, comprising: a record data collection device, connected to an external online web server or an external online database server, and stored in an external online web server or an external online database according to format analysis The external network users in the server use the network service requirements and the reply records sent by the user equipment; the plurality of distributed databases are used to store the user network service requirements analyzed by the record data collection device. Reply record; a cache database is used to provide fast preemption to accelerate the computing process; a data exploration main module, including multiple exploration sub-modules and individually providing modular different algorithms And the calculation logic is used for calculation and analysis; a record data analysis module is connected with the external manager device, and the record data analysis module is controlled by the administrator or independently connected to the data exploration module device and selected appropriate Exploring sub-modules and assigning equipment for performing computing tasks; at least one decentralized computing device, each of the decentralized The computing device obtains the user network service request and the reply record from each of the distributed databases, and performs the task according to the exploration sub-module selected by the record data analysis module and the assigned operation task to be respectively assigned to the internal computing module. And temporarily storing the obtained operation and analysis calculation result in the cache database; and a combination node device, extracting and integrating from each node device in the distributed operation device and each of the distributed operation modules An analysis result is generated, and the analysis result is returned to the record data analysis module. For example, the analysis system of real-time streaming records as described in item 1 of the patent application scope The data collection device encrypts the network service request and the reply record data by using at least one private key, a public key, and an arbitrary integer value, and each of the distributed computing devices directly directly accesses the ciphertext The state network service request and the reply record data are calculated and the operation result is generated, and the combined node device has the corresponding private key, the public key and the arbitrary integer value to decrypt the analysis result. The real-time streaming record data analysis system of claim 1, wherein the data exploration main module further comprises: a nearest neighbor exploration sub-module, which is a k-Nearest Neighbors Method. A decentralized computing module for logic calculation; and a multivariate linear regression exploration sub-module, which is a decentralized computing module that is calculated by the logic of a Multi Factor Line Regression Method. The real-time streaming record data analysis system of claim 1, wherein each of the distributed computing devices further comprises: a plurality of node devices, and the plurality of exploration sub-modules selected according to the record data analysis module are assigned to the plurality of The distributed computing module, the plurality of node devices acquires and records the records data to each of the distributed data bases, wherein each of the distributed computing modules performs operations and analyzes the recorded data according to the operation process of the exploration sub-module. An instant stream recording data analysis method, the steps comprising: recording online data steps: a record data collecting device collects and stores network services sent by external plural users from an external online web server and an external online database server Requirements and response records; deposit in a decentralized database step: record data collection device will be the aforementioned network service Requirement and reply record storage to a plurality of decentralized databases; selection of data exploration module steps: an external manager is linked to a record data analysis module to request a data exploration main module via the record data analysis module The plurality of exploration sub-modules are selected for use; the assignment work to the distributed computing device step: the data exploration and exploration main module will assign a plurality of distributed computing devices to the user network according to the exploration sub-module selected by the external manager Requires operation with the reply record; temporary storage to the cache database device step: each of the results of the distributed computing device output will be temporarily stored in a cache database for future analysis; and backhaul and display The result of the operation: the combination node device connected to each of the distributed computing devices integrates the calculation result into an analysis result, and transmits the result to the external manager device through the record data analysis module to present the analysis result to the external user. The method for analyzing real-time streaming data as described in claim 5, further comprising the step of: the record data collecting device serving the network by using at least one private key, a public key, and an arbitrary integer value. Requiring and replying to the record data for encryption; each of the distributed computing devices directly calculates the network service request of the ciphertext state and the reply record data and generates a calculation result; and the combined node device has corresponding corresponding private keys The public key and the arbitrary integer value decrypt the analysis result. For example, the real-time streaming record data analysis method described in claim 5, wherein the data exploration main module further comprises: a nearest neighbor exploration sub-module, which is a k-nearest neighbor method (k-Nearest). The logic of the Neighbors Method) is a decentralized computing module; and a multivariate linear regression exploration sub-module is a decentralized computing module that is calculated by the logic of the Multi Factor Line Regression Method. The method for analyzing real-time streaming data according to claim 5, wherein each of the distributed computing devices further comprises: a plurality of node devices, and the plurality of exploration sub-modules selected according to the record data analysis module are assigned to the plurality of The distributed computing module, the plurality of node devices acquires and records the records data to each of the distributed data bases, wherein each of the distributed computing modules performs operations and analyzes the recorded data according to the operation process of the exploration sub-module.