CN110245208B - A retrieval and analysis method, device and medium based on big data storage - Google Patents

Abstract

The invention discloses a retrieval and analysis method, device and medium based on big data storage, and relates to the technical field of data storage analysis, including receiving field query information; searching for corresponding on-site searchable events in stored data according to the field query information, and the field-searchable event is an active event in terms of security or performance of one or more information technology systems; generating a corresponding subset of field-searchable events; identifying fields in one or more events in the subset, and determining that such fields exist The number of unique values; display the selected field name and the number of unique values, the field name is the field name of the field that can be referenced in the query information; Massive big data, query a set of searchable events based on specific query criteria, display query field results and unique value numbers, thereby greatly improving the retrieval speed and improving the experience of large database retrieval.

Description

A retrieval and analysis method, device and medium based on big data storage

technical field

The invention relates to the technical field of data storage and analysis, in particular to a retrieval and analysis method, device and medium based on big data storage.

Background technique

In the big data environment, quickly and accurately retrieving the information that the user cares about and is interested in according to the conditions provided by the user is the foundation and an important part of the big data application. With the continuous development of computer technology and the continuous improvement of the degree of informatization, the amount of data has grown rapidly, and the storage and application of massive data has also flourished, and the application of big data has become more and more extensive. For example, in network security, big data technology is used to analyze network attacks; in e-commerce, big data technology is used to analyze users' shopping preferences or the most popular commodities; in urban construction, big data technology is used to build smart cities, which is convenient for People travel. In this way, big data technology has played a positive role in building a conservation-minded society and improving production efficiency.

However, with the continuous increase of data volume and the continuous development of big data applications, more and more data centers are used to store data by business or province. Some industries or institutions often set up multiple data centers to store business data, so in massive data analysis applications, data can only be extracted from a single data center, and all data in each data center can be grouped and counted as a whole data set. , sorting and other simple analysis requirements are also increasingly obvious. In big data applications, it is one of the necessary means to analyze the massive data stored in each data center as a whole.

SUMMARY OF THE INVENTION

In view of the problems of the background technology, the present invention provides a retrieval and analysis method, device and medium based on big data storage, so as to increase the retrieval speed and improve the use experience of retrieval of large databases.

In order to achieve the above purpose, the present invention proposes a retrieval and analysis method based on big data storage, the method comprising:

Receive field query information;

Searching the stored data for a corresponding on-site searchable event according to the field query information, wherein the on-site searchable event is an activity event in terms of security or performance of one or more information technology systems;

generating a subset of the corresponding field searchable events;

identifying fields in one or more events in the subset, and determining the number of unique values that exist for the fields;

The selected field name and the number of the unique values are displayed, wherein the field name is the field name of the field that can be referenced in the query information.

Preferably, the field query information includes but is not limited to: field value criteria, and/or keyword criteria, and/or time range.

Preferably, the stored data includes: multiple groups of on-site searchable events, wherein,

Each event in each set of field searchable events is associated with a timestamp;

Each event in each set of field-searchable events includes machine data reflecting activity in one or more information technology systems;

At least one event in each set of field-searchable events includes log data reflecting activity in one or more information technology systems;

At least one event in each set of field searchable events includes unstructured data.

Preferably, it also includes:

Displays information for two or more events having fields and present in a subset of events, wherein the two or more events are displayed in an order corresponding to the ordering of the fields.

Preferably, the value criteria of the field, wherein the field exists in one or more events in a set of field searchable events.

Preferably, the criterion of the keyword is specifically: a criterion that requires the matching event to have a specific keyword.

The present invention also proposes a retrieval and analysis device based on big data storage, including:

Query receiver, used to receive field query information;

a query executor, configured to search the stored data for a corresponding field searchable event according to the field query information, wherein the field searchable event is an activity event in terms of security or performance of one or more information technology systems; generating a subset of the corresponding on-site searchable events; identifying fields in one or more events in the subset, and determining the number of unique values that exist for the fields;

Data memory for storing multiple sets of field searchable events;

The field selector is used to select and display the field name and the number of the unique values, wherein the field name is the field name of the field that can be referenced in the query information.

Preferably, it also includes: a data storage for storing a plurality of sets of field searchable events.

Preferably, the field selector is also used to display information about one or more events in the subset of live searchable events.

The present invention also provides a computer-readable storage medium on which computer program instructions are stored, and when executed by a processor, the computer program instructions implement the retrieval and analysis method based on big data storage.

The present invention provides a retrieval and analysis method, device and medium based on big data storage. According to the conditions input by the user, facing the massive big data stored in each data center, a set of searchable events is queried based on specific query criteria, and the display is displayed. Query field results and unique value numbers, thereby greatly improving the retrieval rate and improving the experience of large database retrieval.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained according to the structures shown in these drawings without creative efforts.

1 is a flowchart of a retrieval and analysis method based on big data storage in an embodiment of the present invention;

FIG. 2 is a schematic diagram of an on-site searchable event in an embodiment of the present invention;

3 is a schematic structural diagram of a retrieval and analysis device based on big data storage in an embodiment of the present invention;

4 is a schematic structural diagram of a computer storage medium in an embodiment of the present invention;

The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that if there are directional indications (such as up, down, left, right, front, back, etc.) involved in the embodiments of the present invention, the directional indications are only used to explain a certain posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly.

In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for the purpose of description, and should not be construed as indicating or implying Its relative importance or implicitly indicates the number of technical features indicated. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In addition, the technical solutions between the various embodiments can be combined with each other, but must be based on the realization by those of ordinary skill in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that the combination of such technical solutions does not exist. , is not within the scope of protection required by the present invention.

System architecture: The execution device of the present invention can be a device in a local area network, or a device on the Internet, a personal computer, a server, a workstation, etc.;

A retrieval and analysis method based on big data storage proposed by the present invention;

In the first preferred embodiment of the present invention, as shown in FIG. 1 , the method includes:

S100, receiving field query information;

In this embodiment of the present invention, the information about the target field query input by the user is received by the execution device, and the field query information includes but is not limited to: field value criteria, and/or keyword criteria, and/or time range; specifically For: the field is present in one or more of a set of field-searchable events, criteria that requires matching events to have a specific keyword, the query used to search the set of field-searchable events must fall within the matching event associated with the time frame entered therein;

S200. Search the stored data for a corresponding on-site searchable event according to the field query information, wherein the on-site searchable event is an activity event in terms of security or performance of one or more information technology systems;

In the embodiment of the present invention, the retrieval of events is performed based on the big data stored in each data center,

In this embodiment of the present invention, a plurality of groups of on-site searchable events are stored, wherein each event in each group of on-site searchable events is associated with a timestamp; each event in each group of on-site searchable events includes a Machine data of activity in multiple information technology systems; at least one event in each group of field-searchable events includes log data reflecting activity in one or more information technology systems; at least one event in each group of field-searchable events includes unstructured data;

Example: As shown in Figure 2, the stored searchable event set includes event 1 to event N, where N times are associated with timestamps, and event 1 includes: fields, machine data, log data and unstructured data ; Event N includes: fields, machine data and unstructured data; the time range is set through the event associated timestamp and query, and the key information of the event is extracted from each subset to achieve targeted query search to improve retrieval speed;

S300, generating a subset of the corresponding on-site searchable events;

S400, identifying the fields in one or more events in the subset, and determining the number of unique values existing in the above fields;

S500. Display the selected field name and the number of the unique values, where the field name is a field name of a field that can be referenced in the query information.

In this embodiment of the present invention, the information of two or more events having fields and existing in a subset of events is displayed, wherein the two or more events are displayed in order in the order corresponding to the fields; The number of unique values that exist for the field and the field name by which the field can be referenced in queries (used to specify field criteria to further filter the subset of events), and also display information about one or more events in the subset of events; and receive User selection of a field name, hides at least one previously displayed event that does not contain the selected field.

In this embodiment of the present invention, the display of the histogram is made to indicate how many events in the event subset are associated with timestamps that fall within each of the multiple time ranges.

In this embodiment of the present invention, the display of at least the second number corresponding to how many unique values exist in the event subset corresponding to the second field existing in one or more events of the event subset;

The present invention also provides a retrieval and analysis device based on big data storage, the hardware structure of the device includes a display, a processor and a storage;

In the second preferred embodiment of the present invention, as shown in FIG. 3 , it includes:

Query receiver, used to receive field query information;

a query executor, configured to search the stored data for a corresponding field searchable event according to the field query information, wherein the field searchable event is an activity event in terms of security or performance of one or more information technology systems; generating a subset of the corresponding on-site searchable events; identifying fields in one or more events in the subset, and determining the number of unique values that exist for the fields;

Data memory for storing multiple sets of field searchable events;

The field selector is used to select and display the field name and the number of the unique values, wherein the field name is the field name of the field that can be referenced in the query information.

In the embodiment of the present invention, the technical details of the specific components have been described in the first preferred embodiment, and will not be repeated here;

The present invention also provides a computer-readable storage medium;

In the third preferred embodiment of the present invention, as shown in FIG. 4 , computer program instructions are stored thereon, and when the computer program instructions are executed by the processor, the retrieval and analysis method based on large data storage is implemented, for example:

S100, receiving field query information;

S200. Search the stored data for a corresponding on-site searchable event according to the field query information, wherein the on-site searchable event is an activity event in terms of security or performance of one or more information technology systems;

S300, generating a subset of the corresponding on-site searchable events;

S400, identifying the fields in one or more events in the subset, and determining the number of unique values existing in the above fields;

S500. Display the selected field name and the number of the unique values, where the field name is a field name of a field that can be referenced in the query information.

In the embodiment of the present invention, the technical details of the specific components have been described in the first preferred embodiment, and will not be repeated here;

In the description of the embodiments of the present invention, it should be noted that any process or method description in the flow chart or otherwise described herein may be understood to mean that it includes one or more functions for implementing a specific logic function or modules, segments, or portions of code of executable instructions for the steps of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations, which may not be in the order shown or discussed, including on a rudimentary basis depending on the functionality involved The functions are performed in a concurrent manner or in the reverse order, as would be understood by those skilled in the art to which embodiments of the invention pertain.

The logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use by an instruction execution system, apparatus or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus or apparatus), or in conjunction with such instruction execution system, apparatus or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in conjunction with an instruction execution system, apparatus, or apparatus . More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wiring (electronic devices), portable computer disk cartridges (magnetic devices), random access memory, read only memory , Erasable Editable ROM, Fiber Optic Devices, and Portable Optical ROM. In addition, the computer readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, and then edited, interpreted or otherwise rewritten as necessary. Process in a suitable manner to obtain the program electronically and then store it in computer memory.

The above descriptions are only the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Under the inventive concept of the present invention, the equivalent structural transformations made by the contents of the description and drawings of the present invention, or the direct/indirect application Other related technical fields are included in the scope of patent protection of the present invention.

Claims (9)

1. a retrieval analysis method based on big data storage, is characterized in that, method comprises: Receive field query information; Searching the stored data for a corresponding on-site searchable event according to the field query information, wherein the on-site searchable event is an activity event in terms of security or performance of one or more information technology systems; generating a subset of the corresponding field searchable events; identifying fields in one or more events in the subset, and determining the number of unique values that exist for the fields; Display the selected field name and the number of the unique values, wherein the field name is the field name of the field that can be referenced in the query information; The stored data includes: a plurality of groups of on-site searchable events, wherein, Each event in each set of field searchable events is associated with a timestamp; Each event in each set of field-searchable events includes machine data reflecting activity in one or more information technology systems; At least one event in each set of field-searchable events includes log data reflecting activity in one or more information technology systems; At least one event in each set of field searchable events includes unstructured data. 2. The retrieval analysis method based on big data storage according to claim 1, wherein the field query information includes but is not limited to: field value criteria, and/or keyword criteria, and/or time scope. 3. The retrieval analysis method based on big data storage according to claim 1, is characterized in that, also comprises: Displays information for two or more events having fields and present in a subset of events, wherein the two or more events are displayed in an order corresponding to the ordering of the fields. 4. The retrieval and analysis method based on big data storage according to claim 2, wherein the value standard of the field, wherein the field exists in one or more events in a set of field searchable events . 5 . The retrieval and analysis method based on big data storage according to claim 2 , wherein the criterion of the keyword is specifically: a criterion that requires a matching event to have a specific keyword. 6 . 6. A retrieval analysis device based on big data storage, applied to the retrieval analysis method according to any one of claims 1-5, characterized in that, comprising: Query receiver, used to receive field query information; a query executor, configured to search the stored data for a corresponding field searchable event according to the field query information, wherein the field searchable event is an activity event in terms of security or performance of one or more information technology systems; generating a subset of the corresponding on-site searchable events; identifying fields in one or more events in the subset, and determining the number of unique values that exist for the fields; The field selector is used to select and display the field name and the number of the unique values, wherein the field name is the field name of the field that can be referenced in the query information. 7 . The retrieval and analysis device based on big data storage according to claim 6 , further comprising: a data storage for storing multiple groups of on-site searchable events. 8 . 8 . The retrieval and analysis device based on big data storage according to claim 6 , wherein the field selector is further configured to display information of one or more events in the subset of on-site searchable events. 9 . 9. A computer-readable storage medium on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the method according to any one of claims 1-5.

Images