TW202009733A - Method of timely processing and scheduling big data - Google Patents

Abstract

The present invention relates to a method of timely processing and scheduling big data. It comprises the following steps of calculating a data growth rate of a plurality of big data sources by a processing module for determining whether the plurality of big data sources achieve a data growth threshold value; scheduling the plurality of big data sources conforming to the data growth threshold value according to an estimated processing time; and arranging the plurality of big data sources into a plurality of memories with low memory usage to analyze. Therefore, the present invention only processes and analyzes the plurality of big data sources conforming to the standard instead of duplicated big data sources after screening the plurality of big data sources so as to improve the processing efficiency of the plurality of big data sources.

Description

Method for processing and scheduling huge amounts of data in time

The present invention relates to a method for processing and scheduling huge amounts of data in a timely manner, in particular to a method that can eliminate the huge amount of data sources that need not be processed, which is based on the speed of data growth to determine the distance before Whether the new data volume has been added after the processing time of the huge data source.

According to this, huge amount of data is also called big data, and huge amount of data represents a large amount of complex and unstructured data. The size of a single huge amount of data set ranges from terabytes (TB) to tens of megabytes Gigabytes (PB) vary, and huge amounts of data are increasing in number and complexity. However, the analysis of huge amounts of data is already the trend of future technological development. The huge amounts of data have different meanings for different industries. Examples include big science, RFID, sensor network, astronomy, atmospherics, transportation, genomics, Biological or large social data analysis can be applied to the processing and analysis of huge amounts of data. Due to the huge characteristics of data, the data storage system for huge amounts of data must be able to deal with the diversity and complexity of huge amounts of data. Features such as huge capacity and timely processing are required so that consumers and analysts can view huge amounts of data in a timely manner.

However, although front-end and mobile devices provide analysis results that can view huge amounts of data; however, most of the current massive data processing methods use timed batch methods, allowing the system to process and analyze huge amounts of data according to time settings. The user is informed of the timely situation. Therefore, the huge amount of data after analysis often loses its timeliness, and also has the disadvantage of consuming system resources and time; and some of the ideas in the traditional method are to use the instant method, Although it can respond immediately, it still consumes a lot of system resources, and the relationship between scheduling and processing methods is not considered in the analysis. Republic of China Patent Announcement No. TW I522827 "Real-time storage and real-time reading method for huge amounts of data in non-associated databases" provides a method for real-time storage and reading of huge amounts of data, based on real-time computing processing performance requirements. Provide decentralized multiplexing access data mechanism, real-time reading program module depends on the number of current data receiving interfaces, allowing multiple client programs to read at the same time, each reading non-repetitive data fragments, to increase data reading Immediacy.

However, the processing and analysis of huge amounts of data now does not first filter the huge amounts of data, but analyzes all the data. This not only causes a huge performance burden on the system, but also increases the processing and analysis time; How to provide a method for processing huge amounts of data in time, and consider the relevance of scheduling and processing, so as to achieve the purpose of reducing the load of system processing and analysis, which is the direction of the inventor.

Today, the inventor is in view of the fact that the above-mentioned existing huge amount of data processing and scheduling methods still have many deficiencies in actual implementation and use, so it is a tireless spirit, and through its rich professional knowledge and years of practice With the help of experience and improvement, the invention was developed accordingly.

The main purpose of the present invention is to provide a method for processing and scheduling huge amounts of data in a timely manner, which is to determine whether a large amount of data source needs to be processed and analyzed through the data growth rate. If the large amount of data source is old data that is repeated before, It is a huge data source that is unnecessary to process, and timely analysis of other new huge data sources and the correlation of corresponding analysis; thereby, reducing the load of analysis by the processing module to improve the processing efficiency of the huge data source .

In order to achieve the above-mentioned implementation purpose, the present invention provides a method for processing and scheduling huge amounts of data in a timely manner. The method includes a processing module to calculate whether the data growth rate of a plurality of huge data sources reaches the data growth threshold; the data growth threshold is reached The multiple huge data sources will be imported into the processing module; the multiple huge data sources that meet the data growth threshold will be scheduled according to the estimated processing time from small to large; the processing module will store the multiple huge data The source is discharged into a plurality of memories with low memory usage and analyzed; remove the plurality of memories occupied by the plurality of huge data sources after analysis, or retain the plurality of huge data sources after analysis The multiple memories are provided for use by other unanalyzed multiple sources.

In an embodiment of the invention, each huge data source has at least one corresponding analysis result.

In one embodiment of the present invention, the relationship between a plurality of huge data sources and at least one analysis result is divided into one-to-one, one-to-many, and many-to-one, with a total of three data structures.

In one embodiment of the present invention, the data growth rate is a new data amount divided by a total data amount of each huge data source, and the total data amount is the sum of the new data amount and the old data amount.

In one embodiment of the invention, the plurality of huge data sources that meet the data growth threshold cover the old huge data sources in the original processing module.

In one embodiment of the present invention, the estimated processing time of the plurality of huge data sources is the time required to load one of the huge data sources into the plurality of memories plus the processing module to process one of the huge data sources The time required.

In one embodiment of the present invention, if the memory space occupied by one of the huge data sources is larger than the available space of each memory, the huge data source is divided, and the processing module is analyzed in batches.

In one embodiment of the present invention, when one of the huge data sources is divided, it is divided into 2 to the nth power.

The purpose of the present invention and its structural and functional advantages will be explained based on the structure shown in the following drawings and in conjunction with specific embodiments, so that the reviewing committee can have a more in-depth and specific understanding of the present invention.

Referring to the first figure, a method for processing and scheduling huge amounts of data in time according to the present invention includes a processing module to calculate whether the data growth rate of a plurality of huge data sources reaches the data growth threshold. The data growth rate is based on One new data volume of each huge data source is divided by a total data volume, and the total data volume is the sum of the new data volume and the old data volume of the huge data source; a plurality of huge data reaching the data growth threshold The source will be imported into the processing module, and each huge data source that reaches the data growth threshold has at least one corresponding analysis result, which can be divided into one-to-one, one-to-many, and many-to-one data structure;

Multiple huge data sources that meet the data growth threshold may be all huge data sources or some huge data sources, which will cover the old huge data sources that were originally in the processing module, and according to The estimated processing time is scheduled from small to large, where the estimated processing time is the time required to load one of the huge data sources to a plurality of memories plus the processing module to process one of the huge data sources Time; the processing module arranges a plurality of huge data sources into a plurality of memories with low memory usage, and analyzes it. If one of the huge data sources takes up more memory space than the available space of each memory , Then divide the huge data source to the nth power of 2, so that the processing module can be analyzed in batches; remove the multiple memories occupied by the multiple huge data sources after analysis, or keep the post-analysis memory The plurality of memory occupied by a plurality of huge data sources is provided to other unanalyzed plural data sources for use.

In this way, the method for timely processing and scheduling of huge amounts of data of the present invention allows the screened multiple huge amounts of data sources to be imported into the processing module for processing and analysis, eliminating the huge amounts of data sources that do not need to be analyzed this time, and Scheduling according to the processing performance required by the massive data source not only reduces the load of the processing module, but also enables the massive data source to be processed and analyzed in time and order, so that the data that the user sees is Reflect the content of the situation at that time.

In addition, through the following specific embodiments, the scope of the present invention can be further proved to be practical, but it is not intended to limit the scope of the present invention in any form.

Please continue to refer to the first figure. A method for processing and scheduling huge amounts of data in time according to the present invention is mainly to select a large amount of data sources that can be processed and analyzed from a plurality of huge data sources. In the embodiment, for example, there may be The five huge data sources are S1, S2, S3, S4, and S5. After the huge data source is processed and analyzed by the processing module, at least one analysis result will be generated correspondingly, and its form is basically divided into a huge amount Three data structures: one data source for one analysis result, one huge data source for multiple analysis results, and multiple huge data sources for one analysis result; in this embodiment, an S5 huge data source can be used to analyze one Analysis result T4, two huge data sources S1 and S2 can be correspondingly analyzed to analyze an analysis result T1, and two huge data sources S3 and S4 can be correspondingly analyzed to analyze an analysis result T3, and the huge data source of S2 can be taken Two analysis results T1 and T2 can be analyzed correspondingly, as shown in the second figure; such a huge amount of data sources are related to the analysis results and can be called "related" items, which are related to the above four items For the project, there will be four thresholds C1, C2, C3, and C4 respectively, which is the threshold for data growth. This related item is a structure that organizes, arranges, and stores data in computer memory.

Therefore, the processing module is to calculate whether the data growth rate i of S1, S2, S3, S4, S5 has reached the corresponding data growth threshold C1, C2, C3, C4, and each data growth threshold is different, it is used for The value is set according to the past huge data source growth and experience accumulation experience; for example, whether the data growth rate i of the S5 huge data source has reached the corresponding data growth threshold C4, and this data growth rate i is the S5 giant The amount of new data of the data source is divided by the total amount of data. The total amount of data is the sum of the amount of new data and the amount of old data. The amount of new data is defined as the time point after the processing and analysis of the previous S5 huge data source. Increased data volume; if the data growth rate of a huge data source is greater than or equal to the data growth threshold, it means that this huge data source can be used, if the data growth rate of a huge data source is less than the data growth threshold, it means this The huge data source does not need to be processed and analyzed; among them, the processing module only needs a very short time to calculate the result when calculating whether the huge data source reaches the data growth threshold, which is equivalent to the performance burden of the processing module If this step is not performed to screen out huge data sources that do not need to be processed and analyzed, and all of the huge data sources are analyzed, it will consume huge performance and resources of the processing module.

Successively, import a huge amount of data sources that meet the data growth threshold into the processing module, and all data sources that are analysis items related to the huge data source will be imported, such as huge amounts of data When source S2 reaches the data growth threshold, huge data sources S1 and S2 will be imported into the processing module at the same time, and huge data source S3 reaches the data growth threshold, and huge data sources S3 and S4 will be imported into the processing module at the same time. Within the group, the huge data source after the import will overwrite the old huge data source, and wait for processing analysis on the waiting queue.

At this time, the processing module calculates the estimated processing time of each huge data source. The estimated processing time = (the time required to load a huge data source into the memory + the processing module to process the huge data source Time required), after the estimated processing time of all huge data sources is calculated, the huge data sources with less estimated processing time are ranked first, and the huge data with longer estimated processing time When the source is in front, it is in the back; when scheduling, you also need to take into account the memory usage. A large number of data sources usually use multiple memories when processing. In order to increase the efficiency of processing analysis, the system Large data sources will be prioritized in the memory with low memory usage, this memory usage = (a memory has been occupied space + memory space required by a large data source) / the memory Total space, calculated memory usage rate is usually less than 1, so you can prioritize the huge amount of data sources with less estimated processing time into the memory for processing and analysis. Among them, each large amount of scheduled work The data source will correspond to the data structure of related items.

However, if there is an analysis item for a huge amount of data sources, the calculated memory usage rate for each memory is greater than 1, which means that the memory space required by the huge data source is greater than the current time of each memory. Available space, so you can divide the huge data source by the power of 2 to n; if n is 1, the huge data source is divided into the first half and the second half, allowing the memory to be stored in batches and processed The module will first process and analyze the first half of the huge data source, and then process and analyze the second half of the huge data source; if n is 2, the huge data source is divided into 4 items, and the memory will divide it 4 batches of storage, the processing module will also be processed in batches in sequence.

In addition, when some huge data sources have finished processing and analysis, they can be removed from the occupied memory, or the memory occupied by the huge data sources can be retained to provide other unanalyzed huge data sources. Data source to save the time for other huge data sources to be repeatedly downloaded from the hard disk of the processing module to the memory.

It can be seen from the above implementation description that the present invention has the following advantages compared with the prior art:

1. The method for timely processing and scheduling of huge amounts of data according to the present invention can determine whether there is new data in the huge amount of data sources according to the data growth rate of the huge amount of data sources, and eliminate the need to analyze the huge amount of data sources to reduce Processing the load of the module to improve the efficiency of the processing and analysis of huge data sources.

2. The method for timely processing and scheduling of huge amounts of data according to the present invention is based on the estimated processing time of the huge amount of data sources, scheduling them from small to large, and arranging them into a memory with a low memory usage rate to increase The speed of the analysis of huge data sources allows the results after analysis to reflect the situation at that time.

3. The method for processing and scheduling large amounts of data in time according to the present invention uses the data structure idea of "relevant" items, whereby the idea is to cope with the processing of large amounts of data sources and wait for scheduling, and integrate the entire huge data processing system , To save the use of different data structures before and after, so that the processing module generates additional performance requirements, to avoid the memory burden on the space.

In summary, the huge amount of data real-time processing and scheduling method of the present invention can indeed achieve the expected use effect by the embodiments disclosed above, and the present invention has not been disclosed before the application, and it has been fully in line with The provisions and requirements of the Patent Law. I filed an application for a patent for invention in accordance with the law, pleaded for the review, and granted the patent.

However, the illustrations and descriptions disclosed above are only preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention; those who are familiar with this skill, according to the characteristic scope of the present invention, do other things Equivalent changes or modifications should be regarded as not departing from the design scope of the present invention.

no

Figure 1: Flow chart of the preferred embodiment of the present invention.

Figure 2: Schematic diagram of the data structure of the preferred embodiment of the present invention.

Claims (8)

A method for processing and scheduling huge amounts of data in a timely manner, the method includes a processing module to calculate whether the data growth rate of a plurality of huge data sources reaches the data growth threshold; the plurality of huge data sources that reach the data growth threshold Will be imported into the processing module; the multiple huge data sources that meet the data growth threshold will be scheduled according to the estimated processing time from small to large; the processing module will sort the multiple huge data sources into Multiple memory with low memory usage rate and analysis; remove the multiple memory occupied by the multiple huge data sources after analysis, or keep the multiple huge data sources after analysis The plurality of memories are provided for use by other unanalyzed plurality of huge data sources. The method for processing and scheduling huge amounts of data as described in item 1 of the scope of patent application, wherein each of the multiple huge data sources has at least one corresponding analysis result. The method for timely processing and scheduling of huge amounts of data as described in item 2 of the patent application scope, wherein the relationship between the multiple huge amounts of data sources and the at least one analysis result is divided into one-to-one, one-to-many and many-to-one, There are three data structures. The method of timely processing and scheduling of huge amounts of data as described in item 1 of the patent scope, where the growth rate of the data is divided by the amount of new data for each of the huge amounts of data sources divided by a total amount of data The data volume is the sum of the new data volume and the old data volume. The method for timely processing and scheduling of huge amounts of data as described in item 1 of the scope of the patent application, wherein the plurality of huge amounts of data sources that meet the data growth threshold cover the old huge amounts of data sources originally in the processing module. The method for timely processing and scheduling of huge amounts of data as described in item 1 of the scope of patent application, wherein the estimated processing time of the multiple huge data sources is to load one of the multiple huge data sources to the multiple memories The time required, plus the time required by the processing module to process one or more of the massive data sources, is added. For example, the method for processing and scheduling huge amounts of data as described in item 1 of the scope of patent application, where if the memory space occupied by one of the plurality of huge data sources is greater than the available space of each of the plurality of memories, then Divide one or more of these huge data sources to analyze the processing module in batches. The method for timely processing and scheduling of huge amounts of data as described in item 7 of the scope of patent application, in which one of the multiple huge amounts of data sources is divided into 2 to the nth power.

Images