CN114461199A - Method and device for visual data normalization editing - Google Patents

Abstract

The invention discloses a method and a device for visual data normalization editing, and particularly relates to a data normalization method for simplifying a target data step by progressive segmentation and conversion through an operator and a method for simplifying a normalization process by performing visual interactive operation on the normalization process through the method. Firstly, cutting and trimming a data sample to obtain target data and determine a normalized model; then, segmenting target data and extracting data fields; and converting the obtained fields corresponding to the normalization model, sequencing the fields according to the model, and finally generating the normalization data. Has the advantages that: the segmentation, conversion and extraction operations in the normalization process are completed by constructing different operators, so that the normalization process is simplified; through visual interactive interface, the standardization of data is operated, so that the data in each stage can be conveniently checked, and the standardization operation is simple.

Description

Method and device for visual data normalization editing

Technical Field

The invention belongs to the technical field of data normalization, relates to a data normalization method for segmenting data source data and dividing a data normalization process into a plurality of execution sections, and particularly relates to a method and a device for visual data normalization editing.

Background

With the frequent occurrence of network security events, collecting and analyzing logs of various security devices becomes an important means for dealing with network security threats for each important government and enterprise unit. Due to different log formats of different brands, different models of safety equipment and systems, logs of different formats need to be normalized (mapped into the same data model) according to types after collection is completed. At present, the normalization is mainly carried out by manually writing regular expressions, codes and the like, and the use threshold is higher, so that the personal capability requirement on safety service personnel is high, and the use of the safety service personnel is not facilitated. Moreover, the regular expression is manually compiled, the errors need to be manually checked, and the error analysis, arrangement and correction process is time-consuming and labor-consuming and is quite difficult.

Therefore, the manual normalization process is low in efficiency and difficult to process, the process cannot be visualized, and the whole writing process is difficult to debug and correct.

Disclosure of Invention

The invention aims to overcome the defects in the prior art and provides a data normalization method for simplifying the steps of stepwise progressive segmentation and conversion of target data by using an operator and a method for simplifying the normalization process by performing visual interactive operation on the normalization process by using the method.

In order to achieve the purpose, the invention adopts the following technical scheme to realize the purpose:

a method for visual data normalization editing,

the method comprises a data source component and an operator, and comprises the following steps when data normalization is carried out:

s1, importing the data sample into a data source component, and cutting and trimming the data sample to obtain target data;

s2, determining a normalized model of the target data to be generated;

s3, firstly, dividing the target data into a plurality of data units according to different data types through an operator, and then continuously dividing the obtained data units through the operator until the data units are divided into a plurality of data fields;

s4, converting the data fields corresponding to the normalized model in the data fields obtained in the step S3 into normalized fields with the consistent data format through operators;

and S5, arranging the acquired normalized fields according to the sequence of the normalized model, generating normalized data and storing the normalized data in a database.

Furthermore, the operator is a plurality of minimum units for realizing different computing functions and is divided into a data segmentation operator and a data conversion operator; wherein the content of the first and second substances,

the data segmentation operators comprise operators with different segmentation functions and are used for segmenting the target data and the unit data;

and the data conversion operator comprises operators for converting different formats and carries out format conversion on the data field.

Further, the normalized model is a group of fields which define a data format and are arranged in a certain order.

Furthermore, the operator is constructed and configured in a regular expression or JSON object analysis or XML format analysis or Script code analysis mode;

the operator includes: the system comprises a regular segmentation operator, a JSON field extraction operator, a script code extraction operator, an XML format extraction operator, a conversion operator, a desensitization operator, a mapping operator and a cleaning operator.

Further, the operators adopted in steps S3 to S5 are combined according to the logic, operation, and arrangement order of each step, so as to obtain a normalized expression for converting the target data into a corresponding normalized model.

Further, in the normalized expression, the same sub-formula or operation units are combined.

And further, combining units operated by regular expressions in the normalized expressions.

Further, the step processes of segmenting and converting the target data by using the operator in the steps S3 to S5 are checked by using the directed acyclic graph, so that no dead loop exists in the step processes, and the steps can be executed.

Further, the data source component, the operator, the normalized model, the normalized stored graphic plug-in and various intermediate data obtained in the steps S1 to S5 are visually displayed and operated through an interactive interface;

the operation steps are as follows:

displaying the data of the data sample in an interactive interface by operating a data source component graphic plug-in, and cutting, trimming and displaying the data to obtain target data;

then, by adding a data segmentation operator, performing primary segmentation on the target data, and displaying the result into a plurality of data units;

adding a data segmentation operator, performing secondary segmentation on the obtained data units to obtain smaller data units or fields, and continuing to segment the data units until the fields are obtained; meanwhile, the data of each division is displayed.

Adding a data conversion operator to a data field corresponding to the normalized model, and corresponding and displaying a result and the field of the normalized model;

and finally generating and displaying operating and normalizing data by operating and normalizing the stored graphic plug-in.

The invention also provides a device for visual data normalization editing, which comprises:

the data source processing module is used for importing the data sample into the data source component, cutting and trimming the data sample and obtaining target data;

the normalization model module is used for determining a normalization model which needs to be generated by normalization of the target data;

the data segmentation module is used for segmenting target data into a plurality of data units through an operator according to different data types, and then continuously segmenting the obtained data units through the operator until the data units are segmented into a plurality of data fields;

a data conversion module, configured to convert, through an operator, a data field corresponding to the normalized model in the data field obtained in step S3 into a normalized field with a consistent data format;

and the normalization generation module is used for arranging the obtained normalized fields according to the sequence of the normalization model, generating normalization data and storing the normalization data into the database.

The invention also provides a visual data normalization editing system, which comprises one or more memories, wherein the one or more memories store executable instructions; and one or more processors executing the executable instructions to implement a method for centrally managing and controlling services by the device and the WEB application system.

The invention also provides a computer readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform a method for centrally managing and controlling services for a device and a WEB application system.

The invention also provides a computer program which comprises computer executable instructions, and the instructions are used for realizing a method for centralized management and control of services by equipment and a WEB application system when being executed.

The working principle is as follows:

when the data is normalized, firstly, target data is segmented into operable data units for segmentation and extraction, then the data units are further analyzed and then segmented and extracted, finally, the data is segmented into a plurality of fields through segmentation and extraction of operators layer by layer, and finally, the obtained fields corresponding to the normalized model are converted into fields with the same format as that of the model; the operation steps of segmentation, extraction and conversion in the whole normalization process are constructed into a plurality of operators capable of carrying out corresponding operation, and each step is directly finished by adopting the operators.

Furthermore, the whole normalization process is simplified by the Achilles tendon through the display and operation of an interactive interface.

Has the advantages that:

1. the segmentation, conversion and extraction operations in the normalization process are completed by constructing different operators, so that the normalization process is simplified;

2. the whole normalization process is subjected to sectional operation, intermediate data of each stage are generated, and the whole normalization process can be checked and checked in a sectional mode, so that the checking of the normalization process is facilitated;

3. the normalization process is split through step-by-step processing of the normalization process, and after the splitting, the data processed each time is reduced, so that the whole normalization process is simplified;

4. through visual interactive interface, the standardization of data is operated, so that the data in each stage can be conveniently checked, and the standardization operation is simple.

Drawings

Fig. 1 is an architecture diagram implemented by a system according to a method for visual data normalization editing provided by an embodiment of the present invention;

fig. 2 is a flowchart illustrating normalized editing of visual data according to an embodiment of the present invention;

fig. 3 is a flowchart of a method for generating a normalized expression and optimizing according to an embodiment of the present invention on the basis of embodiment 2;

FIG. 4 is a graphical editing schematic of a method for performing visual editing using the method described in embodiment 2

Fig. 5 provides an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Example 1

As shown in fig. 1, an architecture diagram implemented by a system is adopted for a method for visualization data normalization editing according to an embodiment of the present invention. The method comprises a data source component and an operator.

The architecture comprises:

the system comprises a data source processing module 101, a normalization model module 104, a data segmentation module 102, a data conversion module 103, a normalization generation module 105 and a storage module 106.

The data source processing module 101 is configured to import a data sample into a data source component, and perform cutting and trimming on the data sample to obtain target data;

the normalization model module 104 is configured to determine a normalization model that the target data needs to be generated in a normalization manner;

the data segmentation module 102 is configured to segment target data into a plurality of data units according to different data types through an operator, and then segment the obtained plurality of data units continuously through the operator until the data units are segmented into a plurality of data fields;

the data conversion module 103 is configured to convert a data field corresponding to the normalized model in the data field obtained in step S3 into a normalized field with a consistent data format through an operator;

the normalization generation module 105 is configured to arrange the obtained normalization fields according to the order of the normalization model, generate normalization data, and store the normalization data in the database;

the storage module 106 is configured to store the intermediate data, the normalized model, and the normalized generated data and normalized expression generated by each module in the normalization process.

The data sample can be a group of data containing a plurality of different types of data formats and data types, such as a log file, a data fragment crawled by a crawler tool, a section of record file and the like.

Example 2

Fig. 2 is a flowchart illustrating normalized editing of visual data according to an embodiment of the present invention. The method can be applied in a system architecture as described in implementation 1.

The method comprises a data source component and an operator, and comprises the following steps when data normalization is carried out:

s201, importing a data sample into a data source component, and cutting and trimming the data sample to obtain target data;

s202, determining a normalized model of the target data to be normalized;

s203, firstly, dividing target data into a plurality of data units according to different data types through an operator, and then continuously dividing the obtained data units through the operator until the data units are divided into a plurality of data fields;

s204, converting the data fields corresponding to the normalized model in the data fields obtained in the step S203 into normalized fields with consistent data formats through operators;

s205, arranging the obtained normalized fields according to the sequence of the normalized model, generating normalized data, and storing the normalized data in a database.

The operator is a plurality of minimum units for realizing different computing functions and is divided into a data segmentation operator and a data conversion operator; the operator is constructed and configured in a regular expression or JSON object analysis or XML format analysis or Script code analysis mode.

Wherein the content of the first and second substances,

the data segmentation operators comprise operators with different segmentation functions and are used for segmenting the target data and the unit data; the method specifically comprises the following steps: the system comprises a regular segmentation operator, a JSON field extraction operator, a script code extraction operator, an XML format extraction operator and the like. Data of

The regular segmentation operator is used for segmenting the data according to a regular expression;

extracting an operator from the JSON field, and extracting a corresponding JSON field;

extracting an operator from the script code, and extracting a corresponding script code;

XML format extraction operator: extracting a corresponding XML format code;

and the data conversion operator comprises operators for converting different formats and carries out format conversion on the data field. The method specifically comprises the following steps: a conversion operator, a desensitization operator, a mapping operator and a cleaning operator.

The conversion operator is used for converting the field format type;

a desensitization operator for performing decryption processing on the obtained field;

the mapping operator is used for mapping the acquired field into a required data field type;

and the cleaning operator is used for cleaning the obtained field to obtain an effective and required field type.

Example 3

Fig. 3 is a flowchart of a method for generating and optimizing a normalized expression according to an embodiment of the present invention on the basis of embodiment 2.

The provided method for generating and optimizing the normalized expression extracts the expressions generated when the operator is adopted for operation in each step in the visual data normalized editing method, and sorts the expressions according to the data arrangement sequence of the normalized model, so that the normalized expression for the target data is generated.

The method comprises the following specific steps:

s301, combining the operators adopted in steps S203 to S205 in embodiment 2 according to the logic, operation, and arrangement order of each step, to obtain a normalized expression for converting the target data into a corresponding normalized model.

And S302, merging the same sub-formula or operation units in the normalized expression.

And S303, merging the units which are operated by the regular expression in the normalized expression.

And S304, generating a final normalized expression.

Further, the step processes of segmenting and converting the target data by using the operator in the steps S201 to S205 are checked by using the directed acyclic graph, so that no dead loop exists in the step processes, and the steps can be executed.

Example 4

Fig. 4 is a schematic diagram illustrating a method for performing visual editing by using the method described in embodiment 2. On the visual interactive interface 4, the data source component 401, the operator 404, the normalized model 402, the graphical plug-in of the normalized storage 405 and the various intermediate data 403 obtained in the steps S101 to S105 are visually displayed and operated through the interactive interface;

the specific editing steps are as follows:

displaying the data of the data sample in the visual interactive interface 4 by operating the graphic plug-in of the data source component 401, and cutting and trimming the data to obtain and display the target data 406;

then, the target data is divided once through an adding operator 404, and the result is displayed as a plurality of data units; the data unit at this time generates intermediate data 403 for the intermediate process; the operator 404 employed at this time is used for data division into data division operators, and the division can be performed by a regular operator.

Adding an operator 404, segmenting the obtained data units again to obtain intermediate data 403 of smaller data units or fields, and if the data units still exist, continuing to segment the data units until the fields are obtained; at the same time, the intermediate data 403, such as data units, fields, of each division is displayed. In this case, the operator 404 for performing segmentation is a data segmentation operator, and may be a JSON field extraction operator, a script code extraction operator, or an XML format extraction operator.

Adding an operator 404 to a data field corresponding to the normalized model 402 for conversion under the condition that the field formats are inconsistent, and corresponding and displaying the result and the field of the normalized model; the operator 404 used at this time is a data conversion operator, which may be a conversion operator, a desensitization operator, a mapping operator, or a cleaning operator.

Finally, the graphical plug-in of the operation normalization storage 405 generates and displays the operation normalization data.

After the visual interactive interface 4 finishes the canonicalization flow construction, a constructed graph is judged whether to be a directed acyclic graph or not at the moment, so as to judge whether a closed loop exists in the construction process or not and judge whether the constructed graph can be executed or not; when the directed acyclic graph is confirmed to be yes, the construction rule is correct, and a forming result can be executed; if a closed loop is formed, the method cannot be executed, and a normalization result cannot be formed; the detection is carried out through the program in the construction process, so that the correctness of the construction flow is ensured, and the convenience is provided for the whole simplification operation process.

Specifically, the judging process is as follows: the constructed graph first finds an element with an in-degree of 0, which is the "root" element. Then, this element is removed, and the in-degree of the adjacent node is correspondingly reduced, and the new element with the in-degree of 0 is added into the queue. After such a loop operation, if a loop exists, it is certain that the in-degree of the element is always not 0, and the element has never been accessed, so that it is possible to determine whether a loop exists.

Example 5

As shown in fig. 5, an embodiment of the present invention provides an electronic device. The electronic device includes a Central Processing Unit (CPU) that can perform various appropriate actions and processes according to computer program instructions stored in a Read Only Memory (ROM) or computer program instructions loaded from a storage unit into a Random Access Memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input/output (I/O) interface is also connected to the bus.

A plurality of components in an electronic device are connected to an I/O interface, including: an input unit such as a keyboard, a mouse, etc.; an output unit such as various types of displays, speakers, and the like; storage units such as magnetic disks, optical disks, and the like; and a communication unit such as a network card, modem, wireless communication transceiver, etc. The communication unit allows the electronic device to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The processing unit executes the respective methods and processes described above, such as the methods S201 to S205, S301 to S304. For example, in some embodiments, the methods S201-S205, S301-S304 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device via ROM and/or the communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more of the steps of methods S201-S205, S301-S304 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to perform methods S201-S205, S301-S304 in any other suitable manner (e.g., by means of firmware).

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a load programmable logic device (CPLD), and the like.

Program code for implementing the methods of the present invention may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present invention, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Further, while operations are depicted in a particular order, this should be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the invention. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination.

While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims (10)

1. A method for visualization data normalization editing is characterized in that,

the method comprises a data source component and an operator, and comprises the following steps when data normalization is carried out:

s1, importing the data sample into a data source component, and cutting and trimming the data sample to obtain target data;

s2, determining a normalized model of the target data to be generated;

s3, firstly, dividing the target data into a plurality of data units according to different data types through an operator, and then continuously dividing the obtained data units through the operator until the data units are divided into a plurality of data fields;

s4, converting the data fields corresponding to the normalized model in the data fields obtained in the step S3 into normalized fields with the consistent data format through operators;

and S5, arranging the acquired normalized fields according to the sequence of the normalized model, generating normalized data and storing the normalized data in a database.

2. The method of claim 1,

the operator is a plurality of minimum units for realizing different computing functions and is divided into a data segmentation operator and a data conversion operator; wherein the content of the first and second substances,

the data segmentation operators comprise operators with different segmentation functions and are used for segmenting the target data and the unit data;

and the data conversion operator comprises operators for converting different formats and carries out format conversion on the data field.

3. The method of claim 1, wherein the canonicalization model is a plurality of fields that define a data format and are arranged in a sequence.

4. The method of claim 2,

the operator is constructed and configured in a regular expression or JSON object analysis or XML format analysis or Script code analysis mode;

the operator includes: the system comprises a regular segmentation operator, a JSON field extraction operator, a script code extraction operator, an XML format extraction operator, a conversion operator, a desensitization operator, a mapping operator and a cleaning operator.

5. The method of claim 2,

and combining the operators adopted in the steps S3 to S5 according to the logic, operation and arrangement sequence of each step to obtain a normalized expression for converting the target data into a corresponding normalized model.

6. The method of claim 5,

in the normalized expression, the same sub-formula or operation units are combined.

7. The method of claim 6,

and merging the units operated by the regular expression in the normalized expression.

8. The method of claim 2,

and checking the step processes of segmenting and converting the target data by using the operator in the steps S3 to S5 by using the directed acyclic graph, and ensuring that no dead loop exists in the step processes and the steps can be executed.

9. The method of claim 2,

the data source component, the operator, the normalized model, the normalized stored graphic plug-in and various intermediate data obtained in the steps S1 to S5 are visually displayed and operated through an interactive interface;

the operation steps are as follows:

displaying the data of the data sample in an interactive interface by operating a data source component graphic plug-in, and cutting, trimming and displaying the data to obtain target data;

then, by adding a data segmentation operator, performing primary segmentation on the target data, and displaying the result into a plurality of data units;

adding a data segmentation operator, performing secondary segmentation on the obtained data units to obtain smaller data units or fields, and continuing to segment the data units until the fields are obtained; meanwhile, the data of each division is displayed.

Adding a data conversion operator to a data field corresponding to the normalized model, and corresponding and displaying a result and the field of the normalized model;

and finally generating and displaying operating and normalizing data by operating and normalizing the stored graphic plug-in.

10. A device for visualized data normalization editing is characterized in that,

the method comprises the following steps:

the data source processing module is used for importing the data sample into the data source component, cutting and trimming the data sample and obtaining target data;

the normalization model module is used for determining a normalization model which needs to be generated by normalization of the target data;

the data segmentation module is used for segmenting target data into a plurality of data units through an operator according to different data types, and then continuously segmenting the obtained data units through the operator until the data units are segmented into a plurality of data fields;

a data conversion module, configured to convert, through an operator, a data field corresponding to the normalized model in the data field obtained in step S3 into a normalized field with a consistent data format;

and the normalization generation module is used for arranging the obtained normalized fields according to the sequence of the normalization model, generating normalization data and storing the normalization data into the database.

Images