KR101645396B1 - Method of processing time-series big data and system thereof - Google Patents

Abstract

Provided are a method and a system for processing big data which can quickly extract and provide various levels of information scattered in time-series big data to a manager or a user. The big data processing method comprises the following steps: generating event data of a lower layer composed of a key and a value related to an obtaining data of raw data by using the raw data of big data; generating lists having values aligned in a time order on each key by sorting the event data of the lower layer based on the key; and generating event data of an upper layer by using the lists.

Description

METHOD OF PROCESSING TIME-SERIES BIG DATA AND SYSTEM THEREOF BACKGROUND OF THE INVENTION [0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a big data processing method and system, and more particularly, to a big data processing method and system capable of extracting and analyzing meaningful information at various levels scattered in time series big data.

Big data is a vast amount of data that is difficult to collect, store, and analyze with a typical database system. Examples of big data include social data generated from social network services, network traffic logs, web logs of web servers and application programs, sensing data and logs obtained by sensor equipment, vehicle driving data, , Weather data, and the like.

Big data including time-series data such as sensing data, vehicle driving data, weather data, etc., explosively increase the size of data to be handled over time, and various patterns Formation, and repetition of various levels of meaningful information.

Therefore, it is necessary to develop a big data processing method and system capable of rapidly extracting various levels of information from the vast amount of time series big data and providing it to the manager or the user.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to provide a large data processing method and system capable of rapidly extracting various levels of information scattered in time series big data and providing the same to an administrator or a user.

A big data processing method according to an aspect of the present invention is a method for processing a big data by using raw data of big data and generating a key data and a value associated with an acquisition date and time of the raw data, Generating event data of the first data stream; Sorting the event data of the lower layer based on the key and generating lists having values arranged in order of acquisition date and time for each of the keys; And generating event data of an upper layer using the lists.

According to another aspect of the present invention, there is provided a big data processing system for processing and analyzing time series big data, comprising: a big data storage unit storing the time series big data; And generating event data of a lower layer related to the meaning indicated by the row data using the row data of the time series big data, and using the event data of the lower layer, And an event data generator for generating event data of the event data.

The method and system for processing big data according to the technical idea of the present invention is characterized in that basic information contained in each row data constituting time series big data, upper information inherent in extracted basic information, It is possible to hierarchically and quickly extract each of the event data related to the upper level information of the higher level and analyze the extracted event data or event data to provide information to the manager or the user, Data usage, and accurate decision making.

BRIEF DESCRIPTION OF THE DRAWINGS A brief description of each drawing is provided to more fully understand the drawings recited in the description of the invention.
FIG. 1 is a diagram for explaining a big data processing system according to an embodiment of the present invention.
2 is a flowchart illustrating a method of processing big data performed in an event data generating unit of a big data processing system according to an embodiment of the present invention.
FIGS. 3 to 6 are flowcharts for explaining each step of the big data processing method of FIG. 2 in more detail; FIGS. 7 and 8 are diagrams illustrating states of some data in each step of the big data processing method of FIG. FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same reference numerals are used for the same constituent elements in the drawings, and a duplicate description thereof will be omitted.

Embodiments of the present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown. These embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. The present invention is not limited to the following embodiments. Rather, these embodiments are provided so that this disclosure will be more thorough and complete, and will fully convey the concept of the invention to those skilled in the art.

In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. In addition, numerals (e.g., first, second, etc.) used in the description of the present invention are merely an identifier for distinguishing one component from another. Also, in this specification, when an element is referred to as being "connected" or "connected" with another element, the element may be directly connected or directly connected to the other element, It should be understood that, unless an opposite description is present, it may be connected or connected via another element in the middle.

The terminologies used herein are terms used to properly represent embodiments of the present invention, which may vary depending on the user, intent of the operator, or custom in the field to which the present invention belongs. Therefore, the definitions of these terms should be based on the contents throughout this specification. Like reference symbols in the drawings denote like elements.

FIG. 1 is a diagram for explaining a big data processing system according to an embodiment of the present invention.

Referring to FIG. 1, the big data processing system 100 may include a big data storage unit 110, an event data generation unit 130, and an analysis unit 150. Although not shown in FIG. 1, the big data processing system 100 includes a preprocessor for processing the time series data input from the previous stage of the big data storage 100 to fit the big data storage 100, 150), and a post-processing unit for processing data processed and analyzed at a later stage to be suitable for use by an administrator or a user. Hereinafter, the detailed description of the pre-processing unit and the post-processing unit will be omitted for convenience of explanation.

The big data storage unit 100 may store the input time series data. Each of the time series data includes an acquisition date and time field for the time, date and the like when the data is acquired, an identifier field for identifying the device associated with the data, a user of the device, and the like, . The time series data may have a primary meaning at each acquired instant, and may have a higher level, that is, a higher level, than the primary meaning in terms of a longer term than the acquisition period.

In some embodiments, the time series data may be vehicle driving data obtained at a predetermined cycle. The vehicle driving data may include vehicle driving related information such as speed, acceleration, and RPM directly obtained through equipment mounted on the vehicle such as a VDR (Vehicle Driving Recorder), a sensor, and a black box. Alternatively, the vehicle operation data may include vehicle operation related information (for example, a GPS sensor mounted on the smartphone, acceleration (acceleration), etc.) indirectly obtained through a user terminal (e.g., smart phone or the like) Position, velocity, acceleration, etc., obtainable via sensors, etc.). Here, the vehicle driving data may have a primary meaning indicating whether the driver is overspeed or rapidly accelerated at each acquisition period, and the vehicle speed data may have a primary meaning indicating whether the driver is overspeed, driving habit / Meanings can be inherent.

In another embodiment, the time series data may be sensed data acquired from each sensor node of the sensor network in a predetermined period to collect surrounding information and / or object recognition information in a specific region or space. The sensor network may be a network for managing and operating a building, a parking lot, a large market, a factory, a power plant, etc., and the sensing data may include predetermined information obtained from a corresponding sensor. Here, if the sensor network is a network implemented for management and operation of a power plant, and the sensing data is data obtained from a sensor for sensing the state of the power generation facility, the sensing data may include an operation state , And it can include the meaning of the upper layer indicating whether the power generation facility is failed or not from the viewpoint of the long term rather than the acquisition cycle.

In another embodiment, the time-series data includes at least one of economic related data related to economic activities such as rainfall, snowfall, number of earthquake occurrences, temperature change, weather related data related to fine dust concentration, total gross production value, Related data such as demographic related data related to the sales amount of the commodity of the specific company, the marketing related data related to the product advertisement amount, the total population, the population growth rate, the traffic volume, the traffic accident number, The primary meaning for each acquisition cycle and the higher level of significance from a longer-term perspective than the acquisition cycle can be implied.

The big data storage unit 100 may transmit the stored time series data to the event data generating unit 130 in a raw form and may store event data of the lower, Lt; / RTI >

The big data storage unit 110 may be configured with a distributed file system including a storage unit (not shown) and a management unit (not shown), for example, a Hadoop distribution file system. For example, the storage means may comprise a plurality of physically separated storages, and each of the storages may share and store input time series data and hierarchical event data. The management means may store the time series data in the storage means, output the time series data from the storage means to the event data generation unit 130, and input hierarchical event data from the event data generation unit 130 to the storage means. And so on. However, the technical idea of the present invention is not limited thereto, and the big data storage unit 110 may include a cloud file storage system for storing big data, a network storage system, and the like.

The event data generation unit 130 may receive time series data from the big data storage unit 110 in a low format (hereinafter, referred to as low data). The event data generation unit 130 may generate event data in various layers from lower layer data to higher layer data in order to extract and analyze various levels of semantics contained in the big data.

In more detail, the event data generator 130 may generate low-level event data related to the meaning indicated by each of the row data using the row data, and may use event data of a lower layer to generate event data The event data of the upper layer related to the meaning represented by the event data. In addition, the event data generating unit 130 may generate event data of the next higher layer based on the generated event data of the upper layer. In this manner, the event data generating unit 130 can continuously generate higher-level event data based on the generated low-level event data.

The event data generation unit 130 may use, for example, a map-reduction mechanism in hierarchically generating event data.

The hierarchical event data generation process of the event data generation unit 130 will be described in more detail with reference to FIG. 2 to FIG. 8 below.

The analysis unit 150 may generate analysis result information on the meaning indicated by the big data using at least one of the row data and the layer event data stored in the big data storage unit 110. [ The analyzer 150 may analyze at least one of the row data and the layer event data through various data mining analysis techniques to generate the analysis result information.

The analysis unit 150 may transmit the analysis result information to an administrator or a user to assist the administrator or the user in using big data and making a decision. Although not shown in FIG. 1, according to an embodiment, the analyzer 150 may transmit the raw data and the layer event data themselves to an administrator or a user to assist the administrator or the user in using the big data, have.

FIG. 2 is a flow chart for explaining a big data processing method performed in an event data generation unit of a big data processing system according to an embodiment of the present invention; FIGS. 3 to 6 are flowcharts And FIGS. 7 and 8 are diagrams illustrating states of some data in each step of the method of processing large data of FIG. 2. Referring to FIG. Each step of the big data processing method shown in FIG. 2 can be performed in the event data generation unit 130 through at least one software that implements the corresponding algorithm. Hereinafter, the steps of the big data processing method shown in FIG. 2 will be described with reference to FIGS. 3 to 8. FIG.

Event data generation process of lower layer ( S210 )

Referring to FIG. 2, the event data generating unit 130 may generate event data of a lower layer composed of keys and values associated with the acquisition date and time of the row data using the input row data (S210). Step S210 will be described in further detail with reference to FIG. 3, which shows an example of generation of lower layer event data for any one of the row data.

First, the event data generation unit 130 may parse the row data by field (S310). That is, the event data generation unit 130 may parse the row data by the acquisition date / time field, the identifier field, and the numeric field.

Here, the acquisition date / time field may be a field for the time, date, etc., at which the row data is acquired. The identifier field may be a field for identifying a device, a user, or the like corresponding to the row data. The numerical field may be, for example, a field for a vehicle speed, an acceleration, etc. when the low data is vehicle driving data. In another example, if the raw data is sensing data related to the power generation facility, the numerical field may be a field for the power generation amount of the power generation facility. As another example, when the raw data is weather data, the numerical field may be a field for rainfall or the like.

The event data generation unit 130 may determine whether a record exists in the numeric field of the row data (S320). When there is a record in the numeric field of the row data, the event data generation unit 130 performs the following steps, and if there is no record in the numeric field of the row data, the event data generation unit 130 generates a process Can be terminated.

If there is a record in the numeric field of the row data, the event data generation unit 130 may determine whether the record of the numeric field meets a predetermined first condition (S330).

The first condition may be set by a user as a judgment criterion for extracting a meaning. The first condition may be variously set according to, for example, a determination method, whether the record of the numeric field is larger than a predetermined threshold, whether it is smaller than a predetermined threshold, whether it is within a predetermined threshold range have.

For example, when the row data is vehicle driving data and the numeric field is a speed field, the first condition is set as a criterion for determining whether the speed is overspeed or not, .

For example, when the row data is sensing data related to the power generation facility and the numerical field is a field for the power generation amount, the first condition is a criterion for determining whether the abnormal operation is abnormal, Range can be set.

For example, when the row data is weather data and the numerical field is a field for rainfall, the first condition is set as a criterion for determining whether or not the rainfall amount is the amount of dangerous rainfall required for the announcement of a heavy rainfall warning, .

The event data generation unit 130 may generate event data of the lower layer if the numeric field record of the row data satisfies the first condition (S340). At this time, the event data generating unit 130 can generate a lower layer event name field related to an event type of a lower layer, generate a key using a record of the 'lower layer event name, acquisition date and time, identifier' field, (See LD1 to LDw in FIG. 7). [0051] In this case, as shown in FIG. Alternatively, the event data generation unit 130 may generate a key with a record of 'date and time of acquisition, identifier' and generate a value with a record of 'acquisition date and time, lower hierarchy event name' to generate event data of the lower hierarchy (See LD1 to LDw in Fig. 8).

For example, if the first condition is set as a criterion for determining overspeed of the vehicle, the event data generation unit 130 may generate a lower layer event name field The event data of the lower layer can be generated by generating an 'overspeed' as a record of the lower layer hierarchy and including a record of the generated lower hierarchy event name field in the key or value.

For example, if the first condition is set as a criterion for determining whether the power plant is abnormal, the event data generation unit 130 may generate a power generation error error range for normal operation The event data of the lower layer can be generated by generating an 'abnormal' as a record of the lower layer event name field and including a record of the generated lower layer event name field in the key or value.

For example, if the first condition is set as a criterion for determining whether the first condition is a critical amount of rainfall, the event data generation unit 130 may generate a lower layer event when the numerical field record of the raw data exceeds a predetermined amount of rainfall, The event data of the lower layer can be generated by generating a 'dangerous rainfall amount' as a record of the name field and including a record of the generated lower layer event name field in the key or value.

The event data generation unit 130 may generate the plurality of lower layer event data by performing the above-described steps S310 to S340 for each of the row data.

List creation process ( S220 )

Referring to FIG. 2, the event data generator 130 may sort the event data of the lower layer based on the key to generate lists having values sorted in chronological order for each of the keys (S230).

Referring to FIG. 4, when sorting the lower-level event data based on a key having each of the field records as a partial key, the event data generating unit 130 first extracts a portion The event data of the lower layer may be sorted based on the key, and the values may be sorted for each of the partial keys (S410). In operation S410, the event data generator 130 generates the lists by rearranging the sorted values based on the partial keys of the records of the acquisition date and time field in chronological order.

For example, when the event data of the lower layer has each of the records of the 'acquisition date and time, identifier, lower layer event name' field as partial keys (see LD1 to LDw in FIG. 7), the event data generating unit 130 The values are sorted on the basis of partial keys related to the records of the 'identifier' and the 'lower layer event name' fields, and then the lists are sorted by chronologically rearranging the values sorted on the basis of the record of the 'date and time of acquisition' field (LT1 to LTx in Fig. 7). If a record of the 'lower layer event name' field is included as a partial key, an event of a higher layer related to one kind of lower layer event may be extracted through an event data generating process of an upper layer (S230) . On the other hand, when sorting the values using the partial keys related to the records of the 'identifier' and the 'lower layer event name' fields, it is not a problem to arrange the values based on which partial key is used first.

For example, when the event data of the lower layer has each of the records of the 'acquisition date and time, identifier' field as a partial key (see LD1 to LDw in FIG. 8), the event data generation unit 130 first generates the ' (LT1 to LTx of FIG. 8) by rearranging the values based on the record of the " date and time of acquisition " If the record of the 'lower layer event name' field is not included in the partial key but is included in the value, the event data generation process of the upper layer (S230) Event data can be extracted.

Event data generation process of upper layer ( S230 )

2, 5 and 6, the event data generating unit 130 may generate event data of an upper layer using the generated lists (S230). 5 and 6 show the steps performed on any one of the lists in an embodiment of the event data generation process of the upper layer shown in FIG. The event data generation unit 130 may perform only one of the implementations shown in FIG. 5 and FIG. 6 in parallel for a plurality of lists, but the technical idea of the present invention is limited thereto It is not. The event data generation unit 130 may perform one of the implementations shown in FIG. 5 and FIG. 6 in parallel for some lists and the other for the other lists in parallel.

2 and 5, the event data generation unit 130 determines whether the n-th value is the first value in any one of the lists (S510). If the nth value is the first value, the event data generating unit 130 may generate the event data of the upper layer by performing step S540, and may perform subsequent steps if the nth value is not the first value.

If the nth value is not the first value, the event data generation unit 130 may parse the nth value by field (S520). On the other hand, the step S520 may be omitted if the value includes only one field. Hereinafter, the case where the value includes the acquisition date / time field will be described as an example (FIG. 6 is also the same).

The event data generation unit 130 may determine whether the record difference between the acquisition date and time field of the n-th value and the acquisition date and time field of the (n-1) th value meets the second condition (S530). Here, the second condition may be set as to whether or not it corresponds to the above-described acquisition period of the row data, and the event data generator 130 may generate event data for the low- It can be judged whether or not it is.

If the record difference between the acquisition date and time field of the n-th value and the acquisition date and time field of the (n-1) th value does not match the second condition, the event data generation unit 130 generates an event data Field to generate event data of an upper layer (refer to S540, HD1 to HDy in FIGS. 7 and 8). If the record difference between the acquisition date and time field of the n-th value and the acquisition date and time field of the (n-1) th value meets the second condition, the event data generation unit 130 generates the event data of the n + Step S510 may be performed for the first value (S560).

For example, when the lower layer event name is related to the 'overspeed' of the vehicle operator, the event data generation unit 130 generates the event data of the record of the record between the acquisition date and time field of the n-th value and the acquisition date and time field of the n- If the difference does not match the second condition, it can be determined that the overspeed continues for a period corresponding to the record difference. Accordingly, the event data generation unit 130 can generate the record of the upper layer event name field as 'continuous overspeed' in a short term viewpoint, and generate the upper layer event date field The event data of the upper layer can be generated.

In another example, when the lower layer event name is related to the 'abnormal' of the power generation facility, the event data generation unit 130 generates the event data of the n-th value and the n-1th value If the record difference does not match the second condition, it can be determined that the period corresponding to the record difference is a malfunction in terms of a short term. Accordingly, the event data generating unit 130 can generate a record of the upper layer event name field as a 'continuous malfunction'.

In another example, when the lower layer event name is related to the 'dangerous rainfall amount', the event data generation unit 130 generates the event data by using the record between the acquisition date and time field of the n-th value and the acquisition date and time field of the n- If the difference does not match the second condition, it can be determined that a heavy rain has occurred during a period corresponding to the record difference, and a record of the upper layer event name field is generated as a 'continuous risk rainfall amount' can do.

The event data generation unit 130 may determine whether the nth value is the last value in any of the lists (S550). If the nth value is not the last value, the event data generating unit 130 may perform step S510 for the n + 1th value (S560).

Next, referring to FIG. 2 and FIG. 6, the event data generation unit 130 determines whether the nth value is the first value in any one of the lists. (S610). If the nth value is the first value, the event data generating unit 130 may generate the event data of the upper layer by performing step S650. If the nth value is not the first value, the event data generating unit 130 may perform the following steps .

If the n-th value is not the first value, the event data generation unit 130 may parse the n-th value by field (S620). On the other hand, the step S620 may be omitted if the values include only one field as described with reference to FIG.

The event data generation unit 130 may determine whether the record difference between the acquisition date and time field of the n-th value and the acquisition date and time field of the (n-1) th value meets the second condition (S630). Here, the second condition may be set as to whether the record difference corresponds to the acquisition period of the row data as described above with reference to Fig.

The event data generation unit 130 may perform step S640 if the record difference between the acquisition date and time field of the n-th value and the acquisition date and time field of the (n-1) th value does not meet the second condition, If the record difference between the acquisition date and time field of the (n-1) th value and the acquisition date and time field of the (n-1) th data satisfies the second condition, step S610 may be performed for the (S670).

The event data generating unit 130 may determine whether the record of the acquisition date / time field of the n-th value is included in the preset time range without the record difference satisfying the second condition (S640). The time range may be a time range arbitrarily set by an administrator or a user so as to prevent repetitive generation of event data of a higher layer within a predetermined time interval and to analyze data by applying a flexible temporal viewpoint.

If the record of the acquisition date and time field of the n-th value is not included in the preset time range, the event data generation unit 130 generates the upper layer event name field and the upper layer event date field to generate event data of the upper layer If the record of the acquisition date / time field of the n-th value is included in the preset time range, event data of the upper layer is not generated and the (n + 1) Step S610 may be performed (S670).

For example, when the lower layer event name is related to the 'overspeed' of the vehicle operator, if the record of the acquisition date / time field of the n-th value does not belong to the preset time range, It can be determined that the overspeed during the period corresponding to the predetermined time is not the continuous overspeed within the predetermined time range. Accordingly, the event data generation unit 130 can generate a record of the upper layer event name field as 'continuous overspeed', generate a record of the upper layer event date / time field based on each of the acquisition date / time field records and the record difference And event data of an upper layer can be generated.

For example, if the lower layer event name is related to the 'abnormal' of the power generation facility, the event data generation unit 130, if the record of the acquisition date and time field of the n-th value does not belong to the predetermined time range, It can be determined that the malfunction is in a state of malfunction for a period corresponding to the difference, but it can be judged that the malfunction is not a continuous malfunction within a predetermined time range. Accordingly, the event data generating unit 130 can generate a record of the upper layer event name field as a 'continuous malfunction'.

In another example, when the lower layer event name is related to the 'dangerous rainfall amount', the event data generation unit 130, if the record of the acquisition date and time field of the n-th value does not belong to the preset time range, It is determined that the occurrence of the heavy rainfall during the corresponding period is not continuous within the predetermined time range, and a record of the upper layer event name field is generated as the 'continuous risk rainfall amount'.

The event data generation unit 130 may determine whether the n-th value is the last value in any of the lists (S660). If the nth value is not the last value, the event data generating unit 130 may perform step S610 on the (n + 1) th value in step S670.

Event data generation process of the next higher layer ( S240 )

Referring to FIG. 2, the event data generation unit 130 may generate event data of a next higher layer using event data of an upper layer (S240). That is, the event data generation unit 130 can extract event data of the next higher layer having the meaning of a longer-term viewpoint than the event data of the upper layer.

Similar to the above-described steps S220 and S230, the event data generating unit 130 arranges the event data of the upper layer in chronological order, and then determines whether the upper layer event data is continuous (or whether it is continuous or within a predetermined time range The event data of the next higher layer can be generated. More specifically, the event data generating unit 130 may define a key and a value based on an identifier field, an upper layer event name field, and an upper layer event date / time field included in event data of an upper layer, The event data of the next higher layer may be generated using the lists by generating lists having values sorted in chronological order with respect to each of the keys by sorting the event data of the lower layer by a common key.

The event data generating unit 130 may generate higher level event data using event data of the next higher layer, similar to the event data generating process of the next higher layer using event data of the upper layer.

Accordingly, in connection with the big data composed of the vehicle driving data, the event data generating unit 130 extracts, from the event data related to the 'continuous speeding' in the short-term viewpoint, 'long-term speeding' Trends, patterns, and so on.

In the case of the big data constituted by the sensing data, the event data generation unit 130 may generate the event data related to the 'continuous malfunction' from the short-term point of view and generate a 'long-term malfunction' It is possible to generate related higher level event data.

In the case of the big data composed of the weather data, the event data generating unit 130 generates the event data related to the 'continuous risk precipitation amount' from the short-term viewpoint, the 'announcement of the storm warning' It is possible to generate higher-level event data related to a repetitive pattern or the like.

In this way, the big data processing system 100 can extract the event data hierarchically from the event data generation unit 130, and the analysis unit 150 can generate a pattern in which the analysis data is repeated in the big data based on the hierarchical event data. The present invention can provide the information to the administrator or the user as a result of analyzing the event, the event, and the like, thereby making it possible to make an accurate decision of the administrator or the user, and to prepare an effective countermeasure.

In addition, the event data generation unit 130 extracts data related to a high-level event from a basic event and related data, and can process data in parallel, thereby extracting meaningful information more quickly, Or the convenience of the user.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, This is possible.

100: Big data processing system
110: Big data storage unit
130: Event data generation unit
150: Analytical Department

Claims (17)

delete Generating low-level event data composed of a key and a value associated with the acquisition date and time of the raw data using raw data of the big data;
Sorting the event data of the lower layer based on the key and generating lists having chronologically sorted values for each of the keys; And
Generating event data of an upper layer using the lists;
, ≪ / RTI &
Wherein the raw data includes an acquisition date and time field, an identifier field, and a numeric field, the key including: an acquisition date / time field, an identifier field and a lower layer event name field generated under a predetermined condition, Wherein the value comprises the acquisition date and time field or the acquisition date and time field and a lower layer event name field,
The step of generating event data of an upper layer using the lists may include:
Parsing the n-th value of the sorted values by field;
Determining whether a record difference between the acquisition date and time field of the n-th value and the acquisition date and time field of the (n-1) th value meets a second preset condition; And
If the record difference does not match the second condition, generates an upper layer event name field and generates an upper layer event date field based on the record of the acquisition date / time field of the nth value to generate event data of the upper layer The method comprising the steps < RTI ID = 0.0 > of: < / RTI >
3. The method of claim 2,
Wherein the generating of the lower layer event data comprises:
Parsing each of the row data by fields;
Determining whether a numeric field record of the corresponding row data satisfies a first condition set in advance if a record exists in the numeric field of each of the row data; And
And generating the lower layer event name field to generate event data of the lower layer composed of the key and value if the numeric field record of the corresponding row data satisfies the first condition. Data processing method.
3. The method of claim 2,
Wherein the generating the lists comprises:
Sorting the event data of the lower layer based on at least one partial key associated with a field other than the acquisition date and time field, and sorting the values for each of the partial keys; And
And rearranging the sorted values based on a partial key associated with the acquisition date and time field to generate the lists having chronologically sorted values for each of the keys.
delete Generating low-level event data composed of a key and a value associated with the acquisition date and time of the raw data using raw data of the big data;
Sorting the event data of the lower layer based on the key and generating lists having chronologically sorted values for each of the keys; And
Generating event data of an upper layer using the lists;
, ≪ / RTI &
Wherein the raw data includes an acquisition date and time field, an identifier field, and a numeric field, the key including: an acquisition date / time field, an identifier field and a lower layer event name field generated under a predetermined condition, Wherein the value comprises the acquisition date and time field or the acquisition date and time field and a lower layer event name field,
The step of generating event data of an upper layer using the lists may include:
Parsing the n-th value of the sorted values by field;
Determining whether a record difference between the acquisition date and time field of the n-th value and the acquisition date and time field of the (n-1) th value meets a second preset condition; And
Determining whether the record of the acquisition date / time field of the n-th value falls within a predetermined time range if the record difference does not match the second condition; And
If the record of the acquisition date / time field of the n-th value does not fall within the time range, generates an upper layer event name field and generates an upper layer event date / time field based on the record of the acquisition date / And generating event data of the second data.
Generating low-level event data composed of a key and a value associated with the acquisition date and time of the raw data using raw data of the big data;
Sorting the event data of the lower layer based on the key and generating lists having chronologically sorted values for each of the keys; And
Generating event data of an upper layer using the lists;
, ≪ / RTI &
Wherein the raw data includes an acquisition date and time field, an identifier field, and a numeric field, the key including: an acquisition date / time field, an identifier field and a lower layer event name field generated under a predetermined condition, Wherein the value comprises the acquisition date and time field or the acquisition date and time field and a lower layer event name field,
The event data of the upper layer includes:
A key and a value associated with the upper layer event date and time,
After generating the event data of the upper layer,
Sorting the event data of the upper layer based on the key to generate lists having the values sorted in chronological order for each of the keys; And
And generating event data of a next higher layer using the lists.
delete It is a system for processing and analyzing time series big data,
A big data storage unit for storing the time series big data; And
Generating event data of a lower layer related to the meaning indicated by the row data using the row data of the time series big data, and generating event data of a lower layer related to the meaning represented by the event data of the lower layer using the event data of the lower layer An event data generation unit for generating event data;
, ≪ / RTI &
Wherein the event data generating unit comprises:
Generating event data of the lower layer composed of keys and values associated with the acquisition date and time of the row data using the row data,
Wherein the raw data includes an acquisition date and time field, an identifier field, and a numeric field, the key including: an acquisition date / time field, an identifier field and a lower layer event name field generated under a predetermined condition, Wherein the value comprises the acquisition date and time field or the acquisition date and time field and a lower layer event name field,
Wherein the event data generating unit comprises:
Sorting event data of the lower layer on the basis of the key, generating lists having values sorted in chronological order with respect to each of the keys, generating event data of the upper layer using the lists,
Wherein the event data generating unit comprises:
Determining whether a record difference between the acquisition date and time field of the n-th value and the acquisition date and time field of the (n-1) -th value satisfies a preset second condition, If the record difference does not match the second condition, generates an upper layer event name field and generates an upper layer event date field based on the record of the acquisition date / time field of the nth value to generate event data of the upper layer Big data processing system.
10. The method of claim 9,
Wherein the event data generating unit comprises:
Parsing each of the row data by field,
Determining whether a numeric field record of the corresponding row data satisfies a first condition set in advance if a record is present in the numeric field of each of the row data,
And if the numeric field record of the corresponding row data meets the first condition, generates the lower layer event name field to generate event data of the lower layer composed of the key and the value.
delete 10. The method of claim 9,
Wherein the event data generating unit comprises:
Sort the event data of the lower layer based on at least one partial key associated with a field other than the acquisition date and time field, sort the values for each of the partial keys,
And rearranging the sorted values based on a partial key associated with the acquisition date and time field to generate the lists having chronologically sorted values for each of the keys.
delete It is a system for processing and analyzing time series big data,
A big data storage unit for storing the time series big data; And
Generating event data of a lower layer related to the meaning indicated by the row data using the row data of the time series big data, and generating event data of a lower layer related to the meaning represented by the event data of the lower layer using the event data of the lower layer An event data generation unit for generating event data;
, ≪ / RTI &
Wherein the event data generating unit comprises:
Generating event data of the lower layer composed of keys and values associated with the acquisition date and time of the row data using the row data,
Wherein the raw data includes an acquisition date and time field, an identifier field, and a numeric field, the key including: an acquisition date / time field, an identifier field and a lower layer event name field generated under a predetermined condition, Wherein the value comprises the acquisition date and time field or the acquisition date and time field and a lower layer event name field,
Wherein the event data generating unit comprises:
Sorting event data of the lower layer on the basis of the key, generating lists having values sorted in chronological order with respect to each of the keys, generating event data of the upper layer using the lists,
Wherein the event data generating unit comprises:
Determining whether a record difference between the acquisition date and time field of the n-th value and the acquisition date and time field of the (n-1) -th value satisfies a preset second condition, Determining whether the record of the acquisition date / time field of the n-th value belongs to a predetermined time range if the record difference does not match the second condition, and if the record of the acquisition date / time field of the n- Generates an upper layer event name field and generates an upper layer event date field based on the record of the acquisition date and time field of the nth value to generate event data of the upper layer.
It is a system for processing and analyzing time series big data,
A big data storage unit for storing the time series big data; And
Generating event data of a lower layer related to the meaning indicated by the row data using the row data of the time series big data, and generating event data of a lower layer related to the meaning represented by the event data of the lower layer using the event data of the lower layer An event data generation unit for generating event data;
, ≪ / RTI &
The event data of the upper layer includes:
A key and a value associated with the upper layer event date and time,
Wherein the event data generating unit comprises:
The event data of the upper layer is sorted by the key, the lists having the values sorted in chronological order are generated for each of the keys, and the event data of the next higher layer is further generated using the lists system.
10. The method of claim 9,
The big data storage unit stores,
A big data processing system, consisting of a distributed file system.
10. The method of claim 9,
Wherein the big data storage further stores event data of the lower and upper layers generated by the event data generator,
And an analysis unit for generating analysis result information on the meaning indicated by the big data using at least one of the low data and the lower and upper layer event data.

Images