WO2022066039A1

WO2022066039A1 - Assessing the likelihood of critical cybersecurity defects

Info

Publication number: WO2022066039A1
Application number: PCT/RU2020/000713
Authority: WO
Inventors: Дмитрий Сергеевич КУДИЯРОВ; Виталий Оттович БИФЕРТ; Елена Анатольевна ДЕМЬЯНОВА; Геннадий Геннадьевич ГЛОТОВ; Максим Андреевич ЧЕРНЯЕВ
Original assignee: Публичное Акционерное Общество "Сбербанк России"
Priority date: 2020-09-24
Filing date: 2020-12-16
Publication date: 2022-03-31
Also published as: EA202092870A1; EA039867B1; RU2745369C1

Abstract

The inventions relate to an automated method and system for assessing with the aid of machine learning algorithms the likelihood of critical cybersecurity defects occurring during the acceptance testing of product releases. By means of a processor, data is obtained which contains information about software product releases that are in development statuses preceding acceptance testing. The data obtained is processed using a machine learning model. This processing step entails: separating the imported data into categorical and numerical variables; transforming the obtained variables, with vectorization of the categorical variables and normalization of the numerical variables; concatenating the transformed variables and constructing on the basis thereof a vector that corresponds to releases for assessment which are in development statuses preceding acceptance testing; classifying each release with the aid of said vector while indicating the degree of likelihood of critical cybersecurity defects occurring during the acceptance testing of the product releases in order to increase the speed and accuracy of assessing the likelihood of critical cybersecurity defects occurring during the acceptance testing of the product releases.

Description

ASSESSMENT OF THE PROBABILITY OF CRITICAL DEFECTS IN CYBER SECURITY

FIELD OF TECHNOLOGY

[0001] The claimed technical solution generally relates to the field of computer technology, and in particular to an automated method and system for assessing the probability of occurrence of critical cybersecurity defects in acceptance tests of product releases using machine learning algorithms.

BACKGROUND OF THE INVENTION

[0002] Software development for large financial institutions (eg, banks) is always time-consuming and painstaking work. In addition, when developing a software product, it is necessary to take into account all the risks of cybersecurity defects. For these checks, cybersecurity experts are involved, who manually check for critical defects in the developed software product. Cybersecurity experts working with development teams (Agile teams) developing banking products can be involved in the preparation of a release at different stages: idea, design, coding, testing, acceptance testing. The sooner an expert is involved in the preparation of the release, the less time consuming it is for the team to meet the cybersecurity requirements and the lower the cybersecurity risks. However, it is not always possible to attract experts at an early stage. In the event that serious violations of cybersecurity requirements are found during testing, the cybersecurity expert indicates the corresponding critical remark in the test report, which prohibits the team from implementing the release in a production environment. Situations like this increase product development time.

[0003] Engaging a cybersecurity expert exclusively for acceptance tests in the face of a large number of synchronous agile sprints for different teams and the expert’s ignorance about the implementation of cybersecurity functions in releases entail significant peaks in the load on experts, which leads to an increase in verification time and, as a result, there is a shift release of the software product. [0004] The prior art patent US 8631384B2 “Creating a test progression plan”, patent holder: IBM, published: 12/01/2011. This solution describes an automated process for compiling test plans for software products. The known solution provides for the automatic creation of a software test execution plan by calculating for each unit of the testing period x the efforts to complete the test blocks ATTX and the efforts to complete the execution of the test block CCx. The calculation introduces three variables that characterize the testing strategy: efficiency, which represents the effectiveness of the testing group, the defect density coefficient, and the value of the verification coefficient. When choosing a test strategy, the test manager determines the values of three variables that affect the development plan. During test execution, the cumulative "attempt" curve of the ATTX values and the cumulative "completion" curve of the CCx values allow the test manager to compare the efforts already made with the expected efforts made for the test blocks that have been taken and for the test units that have been completed, i.e., when the defects found in the code were fixed.

[0005] The disadvantage of the known solution in this field of technology is the lack of the possibility of automated assessment of the probability of occurrence of critical cybersecurity defects on acceptance tests of product releases.

DISCLOSURE OF THE INVENTION

[0006] The claimed technical solution proposes a new approach to assessing the probability of occurrence of critical cybersecurity defects during acceptance testing of product releases. This solution uses a machine learning algorithm that automates the process of checking releases for critical cybersecurity defects and assesses their occurrence with high accuracy.

[0007] Thus, the technical problem of automated assessment of the probability of occurrence of critical cybersecurity defects during acceptance tests of product releases is solved.

[0008] The technical result achieved in solving this problem is to increase the speed and accuracy of estimating the probability of occurrence critical cybersecurity defects during product release acceptance tests.

[0009] The specified technical result is achieved due to the implementation of a computer-implemented method for assessing the probability of occurrence of critical cybersecurity defects in acceptance tests of product releases, performed using at least one processor and containing the steps at which:

- receive data containing information, at least, about releases of software products that are in development statuses preceding acceptance tests;

- carry out the processing of the received data using a machine learning (ML) model, and in the course of this processing, the following is performed: o separation of the imported data into categorical and numerical variables; o transformation of the obtained variables, which performs vectorization of categorical variables and normalization of numerical variables; o concatenation of the transformed variables and construction on their basis of a vector corresponding to the evaluated releases that are in the statuses preceding the acceptance tests; o classification using the obtained vector of each release with the assignment of the degree of probability of occurrence of critical cybersecurity defects during acceptance tests of product releases.

[0010] In one of the private embodiments of the method, the processing of the received data is carried out using a machine learning model based on a random forest classifier.

[OOP] In another particular embodiment of the method, the MO model is pre-trained on historical data on the receipt of critical cybersecurity defects on acceptance tests of software product releases.

[0012] In another particular embodiment of the method, the processing of the received data is carried out using an ensemble of neural networks.

[0013] In another particular embodiment of the method, the received data additionally contains information about records about tasks of the release category in the system task management for development, including: number, type, importance, status, creation time, time taken to work, time marked as solved, list of dependent task records and dependency types, list of dependent task records and dependency types.

[0014] In another particular embodiment of the method, the data obtained additionally contains information on the number of critical cybersecurity defects identified during acceptance tests of all previous releases of software products.

[0015] In addition, the claimed technical result is achieved through a system for assessing the probability of occurrence of critical cybersecurity defects during acceptance tests of product releases containing:

- at least one processor;

- at least one memory coupled to the processor, which contains machine-readable instructions, which, when executed by at least one processor, provide a method for assessing the probability of occurrence of critical cybersecurity defects on acceptance tests of product releases.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The features and advantages of the present invention will become apparent from the following detailed description of the invention and the accompanying drawings.

[0017] FIG. 1 illustrates a block diagram of the claimed method.

[0018] FIG. 2 illustrates the ROC curve (error curve) for a release classifier based on a random forest.

[0019] FIG. 3 illustrates the error matrix (without normalization) for a release classifier based on a random forest.

[0020] FIG. 4 illustrates an example of a general view of a computing system that provides the implementation of the claimed solution.

IMPLEMENTATION OF THE INVENTION

[0001] Concepts and terms necessary for understanding this technical solution will be described below. [0002] A model in machine learning (ML) is a set of artificial intelligence methods, the characteristic feature of which is not the direct solution of a problem, but learning in the process of applying solutions to many similar problems.

[0003] The F-1 measure is a joint score of precision and recall.

[0004] ROC-curve - a graphical characteristic of the quality of a binary classifier, reflecting the dependence of the proportion of true-positive classifications on the proportion of false-positive classifications when the threshold of the decision rule is varied.

[0005] The error matrix is a way to break classified objects into four categories depending on the combination of the actual class and classifier response.

[0006] Connectors are software components that collect data from information sources (Task Management System / Release Collaboration System / Version Control System / Project Management System / Enterprise Service Management System / etc.) and bring the data to the required structure and format.

[0007] Storage - a system for storing large amounts of data collected and processed by connectors, as well as generated by other components of the system.

[0008] This technical solution can be implemented on a computer, in the form of an automated information system (AIS) or a machine-readable medium containing instructions for performing the above method.

[0009] The technical solution can be implemented as a distributed computer system.

[0010] In this solution, a system means a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, well-defined sequence of computational operations (actions, instructions).

[OOP] A command processing device is an electronic unit or an integrated circuit (microprocessor) that executes machine instructions (programs) / [0012] The instruction processor reads and executes machine instructions (programs) from one or more storage devices, such as random access memory (RAM) and/or read only memory (ROM). ROM can be, but not limited to, hard disk drives (HDD), flash memory, solid state drives (SSD), optical storage media (CD, DVD, BD, MD, etc.), etc.

[0013] A program is a sequence of instructions intended to be executed by a computer control device or command processing device.

[0014] Training of the MO model is performed on pre-labeled data. A total of 431 releases were available at the time the model was created, created within a given time range, for example, 5-6 months. To assess the quality of the model, the data set was divided into 2 parts: training and control samples. The splitting took place as follows (sorting and selection for the test of the latest hits (tickets) by the date of creation is due to the peculiarities of the task): within the data set, release tickets were grouped by projects in a task management system, for example, Jira (htps ://ru. wikipedia.org/ wiki/Jira) and sorted by creation date; the last 20% within each group (for each project) were set aside for the control sample if there were more than 2 tickets in the group; one ticket was added to the control and test samples if there were 2 tickets.

[0015] The weighted f-1 score for the classifier is about 0.8, the accuracy is about 0.8.

[0016] In FIG. Figure 2 shows the ROC curve (error curve) for a release classifier based on a random forest.

[0017] In FIG. 3 shows the error matrix (without normalization) for a release classifier based on a random forest

[0018] Connectors obtain the necessary source information (by downloading files, queries to the database, to the API, parsing web pages, reading event logs, etc.), store it in storage. [0019] Connectors extract significant parameters from the downloaded data for further calculations, perform their preprocessing and form in the storage a table of values of the specified parameters (or features) by releases with the following columns:

The number of Bug tickets in the release;

The number of Feature type tickets in the release;

Number of minor priority tickets in the release;

The number of major priority tickets in the release;

The number of tickets with priority critical in the release;

Number of communications between team members and the cybersecurity expert;

Average time from creating a Release type ticket to marking it as resolved;

Average time from creating a Feature ticket to marking it as resolved;

Average time from creating a Bug ticket to marking it as resolved;

The number of releases released by the product team;

The number of any tickets in one Epic;

A set of parameters (or features) that are products of the values of all pairs of the above parameters (or features);

The number of critical comments on all product releases in the past.

[0020] Based on the values of the release parameters contained in the table, the machine learning algorithm marks the releases present in it as carrying high, medium, and low cybersecurity risks. Marking results are stored as a table in the repository.

[0021] The method for assessing the probability of occurrence of critical cybersecurity defects on acceptance tests of product releases (100) consists of several stages performed by at least one processor.

[0022] At step (101), the machine learning model is inputted with data containing information on at least releases of software products that are in development status prior to acceptance testing. The data may also contain information about: - release tickets in the development task management system (number, type, importance, status, creation time, time taken to work, time marked as resolved, list of dependent tickets and types of dependencies, list of dependent tickets and types of dependencies, responsible agile team, responsible team member)

- the number of critical non-cybersecurity defects identified during acceptance testing of all previous releases of software products;

- data about agile teams and their members developing releases: teams, team members, their roles in the team, positions, training completed, exams passed and their results, data on previous transitions of employees between agile teams and changes in positions, a list releases that employees have worked on;

- the number of critical cybersecurity defects identified during acceptance testing of all previous releases of software products;

- data on documentation for releases (its volume and page hierarchy, number of attempts and dates of its approval by cybersecurity experts and other employees);

- data on the source code of releases (languages used, number of modules, amount of code, number of functions, methods, classes, variables, files);

- release coding data (number of build attempts, number of errors and warnings that occurred during build attempts, the amount of code sent to the build, the number of functions, methods, classes, variables, files);

- data on testing releases (the number of attempts to pass autotests, load and functional testing, the amount of code sent for testing, the number of functions, methods, classes, variables, files);

- data on passing checks by the system of static and dynamic analysis for the presence of vulnerabilities in releases (the number and types of detected vulnerabilities, the results of their marking by the developers of releases in the system as true-positive/false-positive, the amount of code sent for assembly, the number of functions, methods , classes, variables, files); data on vulnerabilities discovered after the release into commercial operation in previous releases of software products (release number, date detection, the developer who created the vulnerable code, the type of vulnerability, the severity of the vulnerability, who discovered the vulnerability).

[0023] Next, at step (102), the received data is processed using a machine learning (ML) model, for example, but not limited to, using a machine learning algorithm based on a random forest classifier (English, random forest).

[0024] During processing, the machine learning algorithm performs:

- in step (103) separating the imported data into categorical and numerical variables

- at step (104) the transformation of the obtained variables, which performs the vectorization of categorical variables and the normalization of numerical variables;

- at step (105) the concatenation of the transformed variables and the construction on their basis of a vector corresponding to the evaluated releases that are in statuses preceding the acceptance tests;

- at step (106) classification using the obtained vector of each release with the assignment of the degree of probability of occurrence of critical cybersecurity defects in the acceptance tests of product releases. For the vector corresponding to the release, a numerical estimate of the probability of a defect is calculated (a number from 0 to 1). Further, a qualitative assessment is added to the numerical assessment: high/medium/low probability (comparison occurs in accordance with the ranges, for example, 0.8-1=high).

[0025] The claimed technical solution provides a new opportunity for automated assessment of cybersecurity risk levels generated by the activities of product agile teams, and their classification into those that comply with high, medium and low cybersecurity requirements when developing software products, allows you to automatically generate a list of releases, sorted by the degree of cybersecurity risk they generate.

[0026] The use of such a list by a cybersecurity expert to prioritize their tasks leads to the identification of risky releases at an earlier stage of their creation, saves labor costs for cybersecurity experts and members of agile teams, while reducing the level of enterprise cybersecurity risks, generated by the activities of product agile teams, reducing the development time of software products.

[0027] In FIG. 4 shows an example of a general view of a computing system (300), which provides the implementation of the claimed method or is a part of a computer system, for example, a server, a personal computer, a part of a computing cluster that processes the necessary data to implement the claimed technical solution.

[0028] In general, the system (300) comprises one or more processors (301), memory facilities such as RAM (302) and ROM (303), input/output interfaces (304), input devices connected by a common information exchange bus / output (1105), and a device for networking (306).

[0029] The processor (301) (or multiple processors, multi-core processor, etc.) may be selected from a range of devices currently widely used, for example, manufacturers such as: Intel™, AMD™, Apple™, Samsung Exynos ™, MediaTEK™, Qualcomm Snapdragon™, etc. Under the processor or one of the processors used in the system (300), it is also necessary to take into account the graphics processor, for example, NVIDIA GPU or Graphcore, the type of which is also suitable for full or partial execution of the method, and can also be used to train and apply machine learning models in various information systems.

[0030] RAM (302) is a random access memory and is designed to store machine-readable instructions executable by the processor (301) to perform the necessary data logical processing operations. The RAM (302) typically contains the executable instructions of the operating system and associated software components (applications, program modules, etc.). In this case, the RAM (302) may be the available memory of the graphics card or graphics processor.

[0031] A ROM (303) is one or more persistent storage devices such as a hard disk drive (HDD), a solid state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R/RW, DVD-R/RW, BlueRay Disc, MD), etc.

[0032] Various types of I/O interfaces (304) are used to organize the operation of system components (300) and organize the operation of external connected devices. The choice of appropriate interfaces depends on the particular design of the computing device, which can be, but not limited to: PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0033] To ensure user interaction with the computer system (300), various means (305) of I/O information are used, for example, a keyboard, a display (monitor), a touch screen, a touchpad, a joystick, a mouse, a light pen, a stylus, touch panel, trackball, speakers, microphone, augmented reality, optical sensors, tablet, indicator lights, projector, camera, biometric identification tools (retinal scanner, fingerprint scanner, voice recognition module), etc.

[0034] The network communication means (306) provides data transmission via an internal or external computer network, for example, an Intranet, Internet, LAN, and the like. As one or more means (306) can be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module and others

[0035] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be construed as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims

FORMULA

1. A computer-implemented method for assessing the probability of occurrence of critical cybersecurity defects during acceptance tests of product releases, performed using at least one processor and containing the steps at which:

2. The method according to claim 1, characterized in that the processing of the received data is carried out using a machine learning model based on a random forest classifier (random forest).

3. The method according to claim 1, characterized in that the ML model is pre-trained on historical data on the receipt of critical cybersecurity defects during acceptance tests of software product releases.

4. The method according to claim 1, characterized in that the processing of the received data is carried out using an ensemble of neural networks. The method according to claim 1, characterized in that the received data additionally contains information about the records of tasks of the release category in the development task management system, including: number, type, importance, status, creation time, time taken to work, time marked as resolved, list of dependent task entries and dependency types, list of dependent task entries and dependency types. The method according to claim 1, characterized in that the data received additionally contain information on the number of critical cybersecurity defects identified during acceptance tests of all previous releases of software products. A system for assessing the probability of occurrence of critical cybersecurity defects during acceptance testing of product releases, which contains:

- at least one processor; at least one memory coupled to the processor, which contains machine-readable instructions, which, when executed by at least one processor, perform the method according to any one of paragraphs. 1-6.