KR102671436B1 - Device, method and program for evaluating security reports based on artificial intelligence - Google Patents

Abstract

This disclosure relates to an artificial intelligence-based security report evaluation device, which receives a security report through a communication unit, inspects the received security report using an artificial intelligence model using at least one preset inspection method, and inspects the security report. Based on the results, it can be determined whether the received security reports are duplicates.

Description

Device, method and program for evaluating security reports based on artificial intelligence}

This disclosure relates to a security report evaluation device, and more specifically, to a device that can evaluate a security report using an artificial intelligence model.

The bug bounty platform receives security vulnerability reports from various participants, evaluates their effectiveness, and provides rewards based on the severity of the vulnerability.

At this time, participants writing security vulnerability reports are not aware of the content reported by other participants, so they frequently submit duplicate reports with the same content.

The bug bounty platform provides preferential rewards to participants who first report security vulnerabilities, but a lot of manpower and time are consumed in reviewing reports to determine whether there are duplicates.

Since this is a problem not only for the bug bounty platform but also for the participants, a technology that can automatically check for duplication of security reports is needed to solve this problem. However, at present, such technology is not publicly available. The situation is not there.

Republic of Korea Patent Publication No. 10-2023-0125653, (August 29, 2023)

Embodiments disclosed in this disclosure seek to provide an artificial intelligence-based security report evaluation device.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

An artificial intelligence-based security report evaluation device according to an embodiment of the present disclosure to solve the above-described problem includes: a communication unit; a memory storing at least one instruction; and a processor, wherein the processor executes the at least one instruction, receives a security report through the communication unit, and uses an artificial intelligence model to check the received security report using at least one preset inspection method. It is possible to inspect and determine whether the received security report is duplicated based on the inspection result.

In addition, the processor summarizes the contents of the received security report using an artificial intelligence model, calculates semantic similarity for the summarized contents, and reports the received security report based on the result of calculating the similarity. The degree of overlap can be calculated.

In addition, the processor calculates similarity by performing an STS (Semantic Textual Similarity) task based on an artificial intelligence model, applies additional weight according to the calculated similarity using a loss function, and then calculates the degree of overlap. .

Additionally, the processor may use data augmentation techniques to increase the amount and variety of data in the received security report.

Additionally, the processor translates the security report written in a first language into at least one other language, translates the security report translated into the other language into the first language, and generates a re-translated security report, Using an artificial intelligence model, the re-translated security report can be inspected using at least one preset inspection method.

In addition, the processor extracts a plurality of keywords from the security report based on the report target, report purpose, and report contents of the security report, and uses an explainable artificial intelligence model to explain the security report. Obtain the report target, report purpose, and report content, and select at least one data column to be used to determine whether the security report is duplicated among a plurality of data columns based on the obtained report target, report purpose, and report content. You can.

In addition, the plurality of data columns include report title, security vulnerability discovery location, scope, attack point, payload, attack type, attack impact, evaluation of the user who wrote the security report, vulnerability description, action plan, company name, and program. Includes name and submission date.

In addition, a summary description of the security report is requested to include the report target, report purpose, and report content of the obtained security report using the explainable artificial intelligence model, and the summary obtained from the explainable artificial intelligence model is requested. The provided description can be inspected using at least one preset inspection method.

In addition, an artificial intelligence-based security report evaluation method according to an embodiment of the present disclosure to solve the above-described problem is performed by a device, and includes the steps of receiving a security report through a communication unit; inspecting the received security report using at least one preset inspection method using an artificial intelligence model; and determining whether the received security report is a duplicate based on the inspection result.

In addition to this, a computer program stored in a computer-readable recording medium for execution to implement the present disclosure may be further provided.

In addition to this, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

According to the means for solving the above-described problem of the present disclosure, security reports can be automatically evaluated based on artificial intelligence.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below.

1 is a schematic diagram of an artificial intelligence-based security report evaluation system according to an embodiment of the present disclosure.
Figure 2 is a block diagram of an artificial intelligence-based security report evaluation device according to an embodiment of the present disclosure.
Figure 3 is a flowchart of an artificial intelligence-based security report evaluation method according to an embodiment of the present disclosure.
Figures 4 and 5 are example diagrams for explaining an embodiment of the present disclosure.
Figure 6 is a diagram illustrating translating a security report into another language, then translating it back to the original language, and checking for duplication using the re-translated security report.
Figure 7 is a diagram illustrating obtaining information in a security report using an explainable artificial intelligence model and determining a data column to be used in security report analysis through this.
Figure 8 is a diagram illustrating each item included in a plurality of data columns.
Figure 9 is a diagram illustrating summarizing a security report using an explainable artificial intelligence model and checking for duplicates using this.

Like reference numerals refer to like elements throughout this disclosure. This disclosure does not describe all elements of the embodiments, and general content or overlapping content between embodiments in the technical field to which this disclosure pertains is omitted. The term 'part, module, member, block' used in the specification may be implemented as software or hardware, and depending on the embodiment, a plurality of 'part, module, member, block' may be implemented as a single component, or It is also possible for one 'part, module, member, or block' to include multiple components.

Throughout the specification, when a part is said to be “connected” to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection through a wireless communication network. do.

Additionally, when a part is said to “include” a certain component, this means that it may further include other components, rather than excluding other components, unless specifically stated to the contrary.

Throughout the specification, when a member is said to be located “on” another member, this includes not only cases where a member is in contact with another member, but also cases where another member exists between the two members.

Terms such as first and second are used to distinguish one component from another component, and the components are not limited by the above-mentioned terms.

Singular expressions include plural expressions unless the context clearly makes an exception.

The identification code for each step is used for convenience of explanation. The identification code does not explain the order of each step, and each step may be performed differently from the specified order unless a specific order is clearly stated in the context. there is.

Hereinafter, the operating principle and embodiments of the present disclosure will be described with reference to the attached drawings.

In this specification, the 'security report evaluation device 100 according to the present disclosure' includes all various devices 100 that can perform computational processing and provide results to the user. For example, the security report evaluation device 100 according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may take the form of any one.

In an embodiment of the present disclosure, a user refers to an evaluator who reports a security vulnerability.

Here, the computer may include, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

The server device is a server that processes information by communicating with external devices and may include an application server, computing server, database server, file server, game server, mail server, proxy server, and web server.

The portable terminal is, for example, a wireless communication device that guarantees portability and mobility, such as PCS, GSM, PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication) -2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, all types of handhelds such as smart phones, etc. It may include wireless communication devices and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-device (HMD).

Functions related to artificial intelligence according to the present disclosure are operated through the processor 110 and the storage unit. The processor 110 may be comprised of one or multiple processors 110 . At this time, one or more processors 110 may be a general-purpose processor 110 such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-specific processor 110 such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence processor such as an NPU. It may be a processor 110 dedicated to intelligence. One or more processors 110 control input data to be processed according to predefined operation rules or artificial intelligence models stored in the storage unit. Alternatively, when one or more processors 110 are dedicated artificial intelligence processors 110, the artificial intelligence dedicated processors 110 may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

Predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned using a large number of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform the desired characteristics (or purpose). It means burden. This learning may be performed in the device itself that performs the artificial intelligence according to the present disclosure, or may be performed through a separate server and/or system 10. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

An artificial intelligence model may be composed of multiple neural network layers. Each of the plurality of neural network layers has a plurality of weights, and neural network calculation is performed through calculation between the calculation result of the previous layer and the plurality of weights. Multiple weights of multiple neural network layers can be optimized by the learning results of the artificial intelligence model. For example, during the learning process, a plurality of weights may be updated so that the loss or cost value obtained from the artificial intelligence model is reduced or minimized. Artificial neural networks may include deep neural networks (DNN), such as Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or deep Q-network, etc., but are not limited to the examples described above.

According to an exemplary embodiment of the present disclosure, the processor 110 may implement artificial intelligence. Artificial intelligence refers to a machine learning method based on artificial neural networks that allows machines to learn by imitating human biological neurons. Methodology of artificial intelligence includes supervised learning, in which the answer (output data) to the problem (input data) is determined by providing input data and output data together as training data according to the learning method, and only input data is provided without output data. Unsupervised learning, in which the solution (output data) to the problem (input data) is not determined, and rewards are given from the external environment whenever an action is taken in the current state, learning is directed to maximizing these rewards. It can be divided into reinforcement learning. In addition, artificial intelligence methodologies can be divided according to the architecture, which is the structure of the learning model. The architecture of widely used deep learning technology can be divided into convolutional neural networks, recurrent neural networks, transformers, and generative adversarial neural networks.

The device 100 may include an artificial intelligence model. An artificial intelligence model may be a single artificial intelligence model or may be implemented as multiple artificial intelligence models. Artificial intelligence models may be composed of neural networks (or artificial neural networks) and may include statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network can refer to an overall model in which artificial neurons (nodes), which form a network through the combination of synapses, change the strength of the synapse connection through learning and have problem-solving capabilities. Neurons in a neural network can contain combinations of weights or biases. A neural network may include one or more layers consisting of one or more neurons or nodes. By way of example, the device 100 may include an input layer, a hidden layer, and an output layer. The neural network constituting the device 100 can infer the result to be predicted from arbitrary input by changing the weight of the neuron through learning.

The processor 110 generates a neural network, trains or learns a neural network, performs an operation based on received input data, generates an information signal based on the performance result, or generates a neural network. Neural network models can be retrained such as CNN, R-CNN, RPN, RNN, S-DNN, S-SDNN, Deconvolution Network, DBN, RBM, Fully Convolutional Network, LSTM, etc. The processor 110 may include various types of models such as Network and Classification Network, but is not limited to this. The processor 110 may include one or more processors 110 to perform operations according to neural network models. For example, a neural network may include a deep neural network.

Neural networks include CNN, RNN, perceptron, multi-layer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE ( Auto Encoder), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), Depp Belief Network (DBN), Deep Convolutional Network (DCN), Deconvolutional Network (DN), Deep Convolutional Inverse Graphics Network (DCIGN), Generative Adversarial Network (GAN), Liquid State Machine (LSM), Extreme Learning Machine (ELM), It may include Echo State Network (ESN), Deep Residual Network (DRN), Differential Neural Computer (DNC), Neural Turning Machine (NTM), Capsule Network (CN), Kohonen Network (KN), and Attention Network (AN). Those skilled in the art will understand that it is not limited to this and may include any neural network.

According to an exemplary embodiment of the present disclosure, the processor 110 may be configured to operate a Convolution Neural Network (CNN), Region with Convolution Neural Network (R-CNN), Region Proposal Network (RPN), or RNN, such as GoogleNet, AlexNet, VGG Network, etc. Recurrent Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT for natural language processing, SP-BERT, MRC/QA, Text Analysis, Dialog System, Various artificial intelligence structures and algorithms such as GPT-3, GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, Anomaly Detection, Prediction, Time-Series Forecasting, Optimization, Recommendation, and Data Creation for ResNet data intelligence. It can be used, but is not limited to this. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

Figure 1 is a schematic diagram of an artificial intelligence-based security report evaluation system 10 according to an embodiment of the present disclosure.

Referring to FIG. 1, the artificial intelligence-based security report evaluation system 10 according to an embodiment of the present disclosure may include a plurality of evaluator terminals 200 and a device 100.

However, in some embodiments, system 10 may include fewer or more components than those shown in FIG. 1 .

In an embodiment of the present disclosure, the device 100 operates a bug bounty platform and provides a bug bounty service. That is, the device 100 may operate a bug bounty service for services it provides itself, or may provide a bug bounty service for various service platforms.

Participants write reports on errors, security vulnerabilities, problems, etc. that occur in various service platforms and provide/report them to the device 100.

Then, the device 100 reviews the report received from the evaluator terminal 200 and provides compensation corresponding to the review result.

At this time, reports may be received from multiple evaluators regarding the same security vulnerability, and the device 100 reviews them and provides compensation to the evaluator who first reported the security vulnerability.

In the past, this process involved manually reviewing all reports to check for duplicates, which consumed a lot of manpower and time, and there was a problem from the evaluator's perspective that it took a lot of time to receive compensation.

The artificial intelligence-based security report evaluation device 100, method, and program according to an embodiment of the present disclosure seek to solve this problem by automatically evaluating security reports based on an artificial intelligence model to determine whether there is overlap.

Below, the artificial intelligence-based security report evaluation device 100, method, and program according to an embodiment of the present disclosure will be described in more detail with reference to other drawings.

Figure 2 is a block diagram of an artificial intelligence-based security report evaluation device 100 according to an embodiment of the present disclosure.

Referring to FIG. 2, the artificial intelligence-based security report evaluation device 100 according to an embodiment of the present disclosure includes a processor 110, a communication unit 120, and a memory 130.

However, in some embodiments, the server may include fewer or more components than those shown in FIG. 2.

The processor 110 includes a storage unit that stores data for an algorithm for controlling the operation of components within the device 100 or a program that reproduces the algorithm, and a storage unit that performs the above-described operations using the data stored in the storage unit. It may be implemented with at least one processor 110. At this time, the storage unit and the processor 110 may each be implemented as separate chips. Alternatively, the storage unit and processor 110 may be implemented as a single chip.

In addition, the processor 110 may control any one or a combination of the components described above in order to implement various embodiments according to the present disclosure described in the drawings below on the device 100.

In addition to operations related to the application program, the processor 110 may generally control the overall operation of the device 100. The processor 110 can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above, or by running an application program stored in the storage unit.

Additionally, the processor 110 may control at least some of the components of the device 100 to run an application program stored in the storage unit. Furthermore, the processor 110 may operate in combination with at least two or more of the components included in the device 100 in order to run the application program.

Processor 110 may be implemented as one or more. Hereinafter, the processor 110 may be considered plural even if it is expressed in the singular. The processor 110 may control the configurations of the security report evaluation device 100. The processor 110 may refer to a data processing device 100 built into hardware that has a physically structured circuit to perform functions expressed by codes or instructions included in a program. As such, the processor 110 is an example of the data processing device 100 built into hardware, and includes a microprocessor, a central processing unit (CPU), a processor core, It may encompass processing devices 100 such as multiprocessors, application-specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs), but the scope of the present invention is not limited thereto. The processor 110 may be separately provided with a learning processor 110 for performing artificial intelligence calculations, or may be provided with its own learning processor 110.

The communication unit 120 may include one or more modules that connect the security report evaluation device 100 to one or more networks.

The communication unit 120 may include one or more components that enable communication with an external device, for example, at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module. It can be included.

Wired communication modules include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as USB (Universal Serial Bus) modules. ), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, wireless communication modules include GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and UMTS (universal mobile telecommunications system). ), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, 6G, etc. may include a wireless communication module that supports various wireless communication methods.

The wireless communication module may include a wireless communication interface including an antenna and a transmitter that transmits communication signals. In addition, the wireless communication module may further include a signal conversion module that modulates a digital control signal output from the processor 110 through a wireless communication interface into an analog wireless signal under the control of the processor 110.

The short-range communication module is for short-range communication and includes Bluetooth®, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, and NFC ( Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology can be used to support short-distance communication.

The memory 130 can store data supporting various functions of the device 100. The memory 130 may store a number of application programs (application programs or applications) running on the device 100, data for operating the device 100, and commands. At least some of these application programs may exist for the basic functions of the device 100. Meanwhile, the application program may be stored in the memory 130, installed in the device 100, and driven to perform an operation (or function) by the processor 110.

The memory 130 can store data supporting various functions of the device 100 and a program for the operation of the processor 110, and can store input/output data (e.g., music files, still images, video, etc.) can be stored, a number of application programs (application programs or applications) running on the device 100, data for operation of the device 100, and commands can be stored. At least some of these applications may be downloaded from an external server via wireless communication.

The memory 130 is of a flash memory type, hard disk type, solid state disk type, SDD type (Silicon Disk Drive type), and multimedia card micro type. (multimedia card micro type), card type memory (e.g. SD or XD memory 130, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM) ), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory 130, magnetic disk, and optical disk. Additionally, the memory 130 is separate from the device 100, but may be a database connected wired or wirelessly.

The memory 130 may be electrically connected to the processor 110 and may store at least one code executed by the processor 110. Memory 130 may refer to various types of storage devices. The memory 130 can store information necessary to perform calculations using artificial intelligence, machine learning, and artificial neural networks.

Memory 130 can store various learning models. Learning models stored in the memory 130 can infer result values for new input data other than learning data, and the inferred values can be used as a basis for judgment to perform certain operations. The learning models stored in the memory 130 may be learned based on label information, and in order to increase the accuracy of learning, various backpropagation algorithms may be applied so that the loss function has a target value.

Additionally, the storage unit may be provided with a plurality of processes for the security report evaluation device 100.

The input unit is for inputting image information (or signal), audio information (or signal), data, or information input from a user, and may include at least one of at least one camera, at least one microphone, and a user input unit. . Voice data or image data collected from the input unit can be analyzed and processed as a user's control command.

The input unit is for receiving information from the user. When information is input through the input unit, the processor 110 can control the operation of the device 100 to correspond to the input information. These input units include hardware-type physical keys (e.g., buttons, dome switches, jog wheels, jog switches, etc. located on at least one of the front, back, and sides of the device 100) and software-type touch. Can contain keys. As an example, the touch key consists of a virtual key, soft key, or visual key displayed on a touch screen-type display unit through software processing, or is displayed on the touch screen. It may be composed of touch keys placed in other parts. Meanwhile, the virtual key or visual key can be displayed on the touch screen in various forms, for example, graphic, text, icon, video or these. It can be made up of a combination of .

The output unit is intended to generate output related to vision, hearing, or tactile sensation, and may include at least one of a display unit, an audio output unit, a haptic module, and an optical output unit. A touch screen can be implemented by forming a layered structure with the touch sensor or being integrated with the display unit. This touch screen functions as a user input unit that provides an input interface between the device 100 and the user, and can simultaneously provide an output interface between the device 100 and the user.

Figure 3 is a flowchart of an artificial intelligence-based security report evaluation method according to an embodiment of the present disclosure.

4 and 5 are exemplary drawings for explaining an embodiment of the present disclosure.

An artificial intelligence-based security report evaluation method according to an embodiment of the present disclosure will be described with reference to FIGS. 3 to 5.

Processor 110 receives a security report. (S100)

More specifically, the processor 110 receives a security report prepared from the evaluator's terminal 200 through the communication unit 120.

In one embodiment, the security report evaluation device 100 may provide a service through the web or an app, and may provide a security report form to the terminal 200 connected to the device 100, and may provide a security report form based on a question and answer format. You can also input and receive the report contents.

Additionally, the processor 110 may store the received security report in the memory 130 or a cloud server.

Processor 110 inspects the security report received from S100. (S200)

The memory 130 stores various commands and algorithms for inspecting security reports, and may store at least one artificial intelligence model that has learned how to inspect security reports.

The processor 110 uses at least one artificial intelligence model to inspect the security report received from S100 using at least one preset inspection method.

First, let's explain the algorithms and terms that will be explained below.

NLP (Natural Language Processing): A field of research that enables computers to understand and process human language. It includes various subfields such as text analysis, language modeling, translation, and sentiment analysis.

Language Model: A language model is used in the field of natural language processing (NLP) and is a model that predicts the probability of a sentence or word sequence. It can estimate how likely the next word or word sequence will appear for a given context. .

Feature: A variable or attribute that represents a specific aspect of input data in a machine learning model. These features are used for learning and prediction of the model.

Classification method: Classification is a subfield of machine learning that divides given data into predefined classes.

Semantic Textual Similarity (STS): A method of measuring semantic similarity between two texts. You can determine how semantically close two texts are by mapping the text to a vector space and calculating the similarity in that space.

Semantic similarity: This refers to measuring the semantic correlation between text or word sentences. Semantic similarity can be used in natural language processing, information retrieval, data analysis, etc.

Summarization: This refers to condensing a long text or document into a shorter form without losing its core content.

Text Augmentation: Text augmentation is a method of expanding the dataset by modifying existing text data. It is used to improve the performance of machine learning models and includes methods such as changing the order of words or using synonyms.

Back Translation: A method of translating text from one language to another and then back to the original language. It can be used in text augmentation to create new text that is different from the original text but has similar meaning.

Pre-training: Pre-training is the process of first training a machine learning model for a general task. After pre-training the model using a large amount of text data in a language model, it can be fine-tuned for a specific task.

Fine tuning: Fine tuning is the process of performing additional training on a pre-trained model to optimize it for a specific task. It typically involves updating the model's weights using a small amount of labeled data.

Loss function: A function that measures the performance of the model. The smaller the value of this function, the closer the model's prediction is to the actual value, so you can aim to minimize this value during the learning process.

Referring to Figures 4 and 5, the processor 110 may inspect the security report using at least one method among Pre-processing, Augmentation, Summarization, Semantic Textual Similarity, and Classifier.

The processor 110 selects necessary columns from the data in the security report through pre-processing, normalizes and transforms the data, processes missing values, undergoes categorical data conversion, processes URLs and file paths, and processes the data into categories. Unify type data, process text data, and convert into categorical data key values.

The processor 110 can perform data augmentation in the security report through augmentation.

More specifically, the processor 110 can perform text augmentation, augmenting text using back traslation methodology, synonym substitution, random word insertion, random word swap, and random word deletion techniques, and targeting 30% of the data. Data augmentation can be performed by randomly performing multiple times.

For example, in an embodiment of the present invention, the processor 110 may perform data augmentation using at least one method among back translation, random swap, random deletion, synonym replacement, and random insertion.

The processor 110 can perform summarization using an artificial intelligence model and can use pre-train data and fine-tuning data. In detail, the processor 110 can summarize the text data in the security report and compress the semantic features of the text, thereby improving performance in STS, Classification, etc. At this time, text data such as detailed explanations and action plans can be applied to the text data in the security report.

The processor 110 can exert a comprehensive data preprocessing effect by reducing text that exceeds a preset length according to preset conditions.

The processor 110 can perform Semantic Textual Similarity using an artificial intelligence model, calculate a similarity optimized for a sentence, and use the calculated similarity as a feature to input it into a Classifier (classification model).

In an embodiment of the present invention, the processor 110 can fine-tune a deep learning-based language model with a semantic textual similarity task and calculate document-level similarity using the model.

The processor 110 can perform a classifier using an artificial intelligence model: a classification model, and can use a Custom Loss Function that gives weight according to the calculated value of STS Fearture.

The processor 110 determines whether the security report is duplicated based on the inspection result of S200. (S300)

When the process of the preset inspection method as shown in FIGS. 4 and 5 is completed, the processor 110 can calculate/output the degree of duplication of the security report.

As an example, the processor 110 may use an artificial intelligence model to summarize the contents of the security report, calculate semantic similarity for the summarized contents, and calculate the degree of duplication of the security report based on the similarity calculation result. . At this time, the processor 110 calculates the similarity by performing an STS (Semantic Textual Similarity) task based on the artificial intelligence model, uses a loss function to assign additional weight according to the calculated similarity, and then calculates the degree of overlap. And, the data amount and diversity of the received security report can be increased using data augmentation techniques.

The artificial intelligence-based security report evaluation device 100 according to an embodiment of the present disclosure may further perform a document summarization process before calculating STS-based similarity.

As an example, the processor 110 can perform a deep learning-based summary of a security report by fine-tuning the Language Model with data suitable for the text summarization task.

This summary can achieve the following effects:

① By calculating the similarity for the summarized document instead of the original document, the size of the target for similarity search calculation is reduced, which has the effect of shortening the time required for similarity testing.

② During the summarization process, important content in the security report is emphasized and unnecessary content is excluded, which has the effect of enabling more accurate calculations when calculating similarity.

③ Unnecessary or redundant information is removed during the summarization process, which reduces noise when calculating similarity, which has the effect of calculating similarity between documents more accurately.

④ Summarized documents are generally smaller than the original documents, which has the effect of reducing memory usage.

The processor 110 can augment the amount of data in the security report, calculate the consistency of the augmented security report, and check the degree of duplication using the augmented security report only when the preset consistency is satisfied. there is.

In one embodiment, the processor 110 measures the consistency of the augmented security report based on the summary of the security report and the augmented security report obtained using the artificial intelligence model, the report target, report purpose, and report contents. can do. At this time, the processor 110 may determine that data augmentation has failed if the object and purpose of the security report and the augmented security report do not match.

Figure 6 is a diagram illustrating translating a security report into another language, then translating it back to the original language, and checking for duplication using the re-translated security report.

Referring to FIG. 6, the processor 110 translates a security report written in the first language into at least one other language. Additionally, the processor 110 may translate the security report translated into another language back into the first language and input the security report re-translated into the first language into an artificial intelligence model to determine whether there is a duplicate.

At this time, the processor 110 may inspect the security report re-translated using the artificial intelligence model using at least one preset inspection method and determine whether the security report is duplicated based on the inspection result.

To this end, the security report evaluation device 100 may further include a translation module or translation model.

In one embodiment, the processor 110 translates a security report written in the first language into a plurality of different languages. Then, the processor 110 translates each translated security report back into the first language to generate a plurality of re-translated security reports.

The processor 110 generates a first summary so that the security report can be explained using an explainable artificial intelligence model.

The processor 110 generates a plurality of second summaries so that each of the plurality of re-translated security reports can be explained using an explainable artificial intelligence model.

The summary may include the report subject, report purpose, and report content of the security report.

The processor 110 measures the degree of consistency between the first summary and each of the plurality of second summaries, and selects the retranslated security report corresponding to the second summary with the highest degree of agreement as the document to be inspected.

Through this, the security report evaluation device 100 can prevent the contents of the security report from changing due to re-translation.

In addition, the processor 110 stores the degree of agreement measured between the report object, report purpose, and report contents of the security report and the first summary and the second summary and trains the artificial intelligence model.

And, when a new security report is received, the processor 110 generates a summary of the security report (report object, purpose, and contents) using an explainable artificial intelligence model, and obtains a re-translated security report based on this. You can select the language to translate the report into.

In other words, there is an optimized translation language for inspection that matches the report target, purpose, and content of the security report, and this means improving efficiency.

Figure 7 is a diagram illustrating obtaining information in a security report using an explainable artificial intelligence model and determining a data column to be used in security report analysis through this.

Referring to FIG. 7, the processor 110 extracts a plurality of keywords from the security report based on the report target, report purpose, and report contents of the security report.

A plurality of keywords may be selected that can represent/express the report target, purpose of the report, and contents of the security report.

The processor 110 can obtain the report target, report purpose, and report contents of the security report that explains the security report using an explainable artificial intelligence model.

Additionally, the processor 110 may select at least one data column to be used to determine whether a security report is duplicated from among a plurality of data columns based on the obtained report target, report purpose, and report content.

Figure 8 is a diagram illustrating each item included in a plurality of data columns.

At this time, multiple data columns include report title, security vulnerability discovery location, scope, attack point, payload, attack type, attack impact, evaluation of the evaluator who wrote the security report, vulnerability description, action plan, company name, program name, and At least one of the submission date and time may be included.

Referring to FIG. 8, examples of each of the plurality of data columns mentioned above are shown.

Figure 9 is a diagram illustrating summarizing a security report using an explainable artificial intelligence model and checking for duplicates using this.

Referring to FIG. 9, the processor 110 requests a summary explanation of the security report to include the report target, report purpose, and report content of the security report using an explainable artificial intelligence model, and generates a summary explanation from the explainable artificial intelligence model. Obtain the completed summary.

Additionally, the processor 110 may inspect the summary obtained from the explainable artificial intelligence model using at least one preset inspection method. The processor 110 may use an artificial intelligence model to inspect the obtained summary using at least one preset inspection method and determine whether the security report is duplicated based on the inspection result.

If the processor 110 determines that the security report is duplicate, it generates a command prompt to generate an answer to be provided to the evaluator based on the duplicate elements and degree of overlap.

Then, the processor 110 inputs the generated command prompt into the generative model and provides answer data obtained from the generative model to the evaluator to explain the reason for the duplication.

If it is determined that the security report is duplicated, the processor 110 may extract at least one duplicate element from the document whose content overlaps with the security report.

Additionally, the processor 110 can calculate the degree of redundancy, which means the degree of overlap, for each extracted redundant element. At this time, if there is no overlap at all, the redundancy can be set to 0, and if they are completely identical, the redundancy can be set to 100.

Accordingly, the processor 110 generates a command prompt to generate a response to explain to the evaluator that the security report is duplicated based on the report object of the security report, the purpose of the report, the report content, and the duplicate elements and the degree of redundancy for each duplicate element. Create and input it into the generative model to obtain an answer.

The processor 110 provides the obtained answer to the evaluator terminal 200 through the communication unit 120.

In an embodiment of the present disclosure, the process of the processor 110 checking whether a security report is duplicated may mean checking whether a document with similar content to the security report exists in a previous document.

The security report evaluation device 100 can calculate duplication and similarity using the following methods. In addition, any algorithm suitable for determining duplication of security reports and calculating similarity can be applied.

In one embodiment, the processor 110 obtains a plurality of hash values by applying n hash functions in the security report, and has a rule for selecting one of the plurality of hash values for each of the n hash functions as a feature value. A feature vector consisting of n feature values is derived using a function.

The processor 110 stores a detection database consisting of n tables in which the calculated n feature values for each of a plurality of previously collected documents have feature values that do not overlap within each table. At this time, the plurality of documents already collected may correspond to existing security reports for which a prior report has been completed.

The processor 110 may compare the n feature values derived from the security report with the feature values of the n tables corresponding to the detection database, and calculate the similarity of the security report according to the ratio of overlapping feature values.

The processor 110 extracts a plurality of keywords from the security report based on the report target, report purpose, and report contents of the security report.

A plurality of keywords may be selected that can represent/express the report target, purpose of the report, and contents of the security report.

The processor 110 may extract a plurality of first keywords related to the report target, report purpose, and report content of the security report. Then, the processor 110 may extract the first keyword and the first keyword within the security report based on the first keyword. A plurality of second keywords having a similarity greater than or equal to a preset similarity are extracted.

The processor 110 assigns a first weight to the first keyword and a second weight to the second keyword based on similarity.

The processor 110 searches a database in which previously reported documents (security reports) are stored and searches for documents that overlap with the first keyword and the second keyword or have a similarity level higher than a preset level.

When it is searched that a target document exists, the processor 110 selects a keyword that overlaps with the target document or has a similarity higher than a preset level from among the first keyword and the second keyword. The processor 110 calculates the final similarity based on the weight assigned to each selected keyword.

The processor 110 determines whether the security report is duplicated based on the calculated final similarity.

The method according to an embodiment of the present disclosure described above may be implemented as a program (or application) and stored in a medium in order to be executed in combination with a server, which is hardware.

The above-mentioned program is C, C++, JAVA that the processor (110, (CPU)) of the computer can read through the device interface of the computer in order for the computer to read the program and execute the methods implemented in the program. , This code may include code coded in a computer language such as machine language, and may include functional code related to functions that define the necessary functions for executing the methods. In addition, these codes may include control codes related to execution procedures necessary for the computer's processor 110 to execute the functions according to predetermined procedures, and may include additional information or media required for the computer's processor 110 to execute the functions. Additionally, the computer's processor 110 may further include a memory 130 reference-related code indicating which location (address address) of the computer's internal or external memory 130 should be referenced. If communication with any other remote computer or server is required for execution, the code must determine how to communicate with any other remote computer or server using the computer's communication module, and what information or media is used during communication. It may further include communication-related codes such as whether to transmit or receive.

The storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory 130. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers that the computer can access or on various recording media on the evaluator's computer. Additionally, the medium may be distributed in a computer system 10 connected to a network, and computer-readable code may be stored in a distributed manner.

The steps of the method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which this disclosure pertains.

Above, embodiments of the present disclosure have been described with reference to the attached drawings, but those skilled in the art will understand that the present disclosure can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

10: Artificial intelligence-based security report evaluation system
100: Security report evaluation device
110: processor
120: Department of Communications
130: memory
200: Evaluator terminal

Claims (10)

Ministry of Communications;
a memory storing at least one instruction;
Contains a processor,
The processor executes the at least one instruction,
Receive a security report through the communication unit,
Using an artificial intelligence model, the received security report is inspected using at least one preset inspection method,
Based on the inspection results, determine whether the received security report is duplicated,
Extracting a plurality of keywords from the security report based on the report target, report purpose, and report contents of the security report,
Obtaining the report target, report purpose, and report content of the security report that explains the security report using an explainable artificial intelligence model,
Characterized in selecting at least one data column to be used in determining whether the security report is duplicated from among a plurality of data columns based on the obtained report target, report purpose, and report content.
Artificial intelligence-based security report evaluation device.
According to paragraph 1,
The processor,
Summarize the contents of the received security report using an artificial intelligence model,
Calculate the semantic similarity of the above summarized contents,
Characterized in calculating the degree of duplication of the received security report based on the result of calculating the similarity,
Artificial intelligence-based security report evaluation device.
According to paragraph 1,
The processor,
Characterized by calculating similarity by performing an STS (Semantic Textual Similarity) task based on an artificial intelligence model, applying additional weight according to the calculated similarity using a loss function, and then calculating the degree of overlap.
Artificial intelligence-based security report evaluation device.
According to paragraph 1,
The processor,
Characterized by increasing the data amount and diversity of the received security report using data augmentation techniques,
Artificial intelligence-based security report evaluation device.
According to paragraph 1,
The processor,
Translating the security report written in the first language into at least one other language,
Translating the security report translated into the other language into the first language to generate a re-translated security report,
Characterized in that the re-translated security report is inspected using at least one preset inspection method using the artificial intelligence model,
Artificial intelligence-based security report evaluation device.
delete According to paragraph 1,
The plurality of data columns are:
Includes report title, security vulnerability discovery location, scope, attack point, payload, attack type, attack impact, evaluation of the user who created the security report, vulnerability description, action plan, company name, program name, and submission date.
Artificial intelligence-based security report evaluation device.
According to paragraph 1,
Requesting a summary explanation of the security report using the explainable artificial intelligence model to include the report target, report purpose, and report content of the obtained security report,
Characterized in that the summarized description obtained from the explainable artificial intelligence model is inspected using the at least one preset inspection method,
Artificial intelligence-based security report evaluation device.
In a manner performed by the device,
Receiving a security report through a communication unit;
inspecting the received security report using at least one preset inspection method using an artificial intelligence model; and
A step of determining whether the received security report is a duplicate based on the result of the inspection,
The device is,
Extracting a plurality of keywords from the security report based on the report target, report purpose, and report contents of the security report,
Obtaining the report target, report purpose, and report content of the security report that explains the security report using an explainable artificial intelligence model,
Characterized in selecting at least one data column to be used in determining whether the security report is duplicated from among a plurality of data columns based on the obtained report target, report purpose, and report content,
Artificial intelligence-based security report evaluation method.
A computer-readable recording medium combined with a hardware computer and storing a program for executing the method of claim 9.

Images