KR20200084441A - Automatic build apparatus and method of application for generating training data set of machine learning - Google Patents

Abstract

The present invention relates to an automation build device which automatically generates a data set of a sample application for machine learning to perform malware detection, classification, and the like, and automatically solves a problem which occurs in a build process when generating the data set. The automation build device comprises: a storage part which stores a plurality of open source data for machine learning; a setting part which performs system setting including identification of the build system type of open source data stored in the storage part, setting of appropriate environment or option information, and application of necessary libraries; and a construction part which generates an application data set by building the open source data in an environment or options set by the system setting part.

Description

Application automation build device and method for creating machine learning learning dataset {AUTOMATIC BUILD APPARATUS AND METHOD OF APPLICATION FOR GENERATING TRAINING DATA SET OF MACHINE LEARNING}

The present invention relates to an application automation build apparatus and method for generating a learning dataset of machine learning, and more specifically, to automatically generate a dataset of a sample application for learning of machine learning to perform malware detection, classification, etc. , This is a technology for an automated build device and method that automatically solves problems that occur during the build process when creating a data set.

With the advancement of computing devices, automation systems using machine learning are emerging as next-generation technologies.

Meanwhile, with the emergence of various types of malware and concealment-based malware, efforts to utilize machine learning to detect and classify malware are continuing. Types of malware include viruses, worms, Trojan horses, backdoors, logical bombs, bots, adware, spyware, and ransomware.

In order for machine learning to perform its functions, learning must be performed in advance. Machine learning requires a vast learning dataset. The factor that determines the performance of machine learning is the training data set, so the importance of the training data set is the greatest in the use of machine learning.

Machine learning for malware detection or classification requires that applications that are not infected with malware and applications that are infected with malware are provided.

Administrators must be able to identify malware-infected applications in the training dataset in order to conduct proper training and testing. However, in general, among applications that can be obtained online, such as an app store, there is a possibility that there is a malware infection, so it may not be perfect to classify a conventional application as being infected with a malware or not. If an application infected with malware is learned to machine learning as an application that is not infected, the machine learning continuously has a problem that the detection rate of the related malware may be lowered.

Therefore, a preferred application as a learning dataset is an open source application that can review source content. However, in the case of open source, it is coded in various build system environments, and each application has different options or calling libraries. Therefore, when you build open source manually, you need to solve dozens of errors during the build to obtain a single executable application.

However, in order to implement a reliable machine learning algorithm, thousands to millions of training data are required. Therefore, it is virtually impossible to manually create a training dataset for machine learning that performs malware detection and classification.

Registered Patent Publication No. 10-1880628

Accordingly, the present invention is proposed to solve the above-mentioned problems, and the object of the present invention is to automatically generate a dataset of a sample application for learning machine learning to perform malware detection, classification, etc. The task is to provide an automated build device and method that automatically solves problems that occur during the build process when creating a set.

In order to achieve the above object, the application automation build apparatus for generating a learning dataset of machine learning according to the technical idea of the present invention includes a storage unit in which a plurality of open source data for learning machine learning are stored; A setting unit for identifying a build system type of open source data stored in the storage unit, setting a suitable environment or option information, and setting a system including application of a necessary library; It characterized in that it comprises a building unit for generating an application data set by building the open source data in an environment or option set by the setting unit system.

In addition, it may be characterized in that it further comprises a concealment unit for concealing by applying an obfuscation technique or a packing technique to the open source data.

In addition, the building unit may further include a signature unit that adds an electronic signature when building the open source data.

In addition, it may be characterized in that it further comprises a malware insertion unit for inserting the learning malware code necessary for machine learning learning in the open source data.

In addition, the setting unit, the build analysis unit for identifying the build system type of the open source data, and generates a missing build required file; An environment adjustment unit for setting environment or option information suitable for the open source data; A library configuration unit to apply a library called by the open source data; And a correction unit for correcting grammatical errors and updating grammar of the open source data.

In addition, the machine learning is an algorithm for malware detection or classification of the Android application, the open source data is the open source of the Android application, and the build analysis unit includes the build system of the open source data is Eclipse, Android Studio ( Android studio), Gradle (Gradle), it can be characterized by identifying the corresponding.

In addition, the library configuration unit may include a library database for searching the required library of the open source data.

In addition, the library configuration unit may be characterized in that when the open source data includes connection information of an invalid Android server, the connection information is updated to the latest Android server information.

In addition, the proofing unit may be characterized by performing grammatical error correction and grammar updating for build-related files including local.properties, build.xml, and gradle-wrapper.properties.

On the other hand, in order to achieve the above object, the application automation build method for generating a learning dataset of machine learning according to the technical idea of the present invention comprises: (A) storing a plurality of open source data for machine learning learning; ; (B) the setting unit performs a system setting including identification of a build system type of open source data stored in the storage unit, setting of suitable environment or option information, and application of necessary libraries; And (C) characterized in that it comprises the step of building an application data set by the build unit to build the open source data in an environment or option set by the setting unit system.

In addition, before the step (C), the hiding portion may further include the step of concealing by applying an obfuscation technique or a packing technique to the open source data.

In addition, before the step (C), the signature unit may further include the step of adding an electronic signature to the open source data.

In addition, before the step (C), the malware insertion unit may further include the step of inserting the learning malware code necessary for machine learning learning into the open source data.

According to an application automation build apparatus and method for generating a learning dataset of machine learning according to the present invention,

First, because it automatically creates an application dataset for learning machine learning that performs malware detection, classification, etc., it is possible to support sufficient learning of machine learning.

Second, the configuration and library application necessary for the build of open source are automatically executed, and the code with errors is automatically corrected, so that an application dataset can be quickly generated.

Third, open source can be concealed in various ways, and it is possible to support the learning of machine learning to detect concealed malware by efficiently generating concealed malware.

Fourth, it is easy to prepare a sufficient dataset of malware-infected applications because it can be built by embedding learning malware code in open source.

1 is a configuration diagram of an application automation build device for generating a learning dataset of machine learning according to an embodiment of the present invention.
2 is a flow chart of an application automation build method for generating a learning dataset of machine learning according to an embodiment of the present invention.

With reference to the accompanying drawings will be described in detail with respect to the application automation build apparatus and method for generating a learning dataset of machine learning according to embodiments of the present invention. The present invention can be applied to various changes and may have various forms, and specific embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to a specific disclosure form, it should be understood to include all modifications, equivalents, or substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals are used for similar components.

In addition, unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains. Terms, such as those defined in a commonly used dictionary, should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. Does not.

The present invention can be used without limitation in various platforms such as Windows environment, Android environment, Linux environment, and iOS environment. To illustrate the invention, an Android environment is used as an example, but this does not limit the application environment of the present invention to Android.

The application automation build device 100 for generating a learning dataset of machine learning according to an embodiment of the present invention is installed and operated in one computer device, or distributedly installed in a plurality of computer devices, and each computer device is wired or wireless to each other It may be implemented by being connected and interlocked.

Referring to FIG. 1, the application automation build apparatus 100 for generating a learning dataset of machine learning according to an embodiment of the present invention includes: a storage unit 110 in which a plurality of open source data for learning machine learning are stored; A setting unit 140 for identifying a build system type of open source data stored in the storage unit 110, setting a suitable environment and option information, and setting a system including application of necessary libraries; And a building unit 180 that builds the open source data in an environment or option set by the setting unit 140 and creates an application data set.

Open source data includes applications for which source code has been released.

The storage unit 110 may be operated by a database management application program or by creating a directory for each open source.

The list generating unit 120 lists the unique names, classifications in the App Store, included functions, used APIs, and included permissions in the open source data stored in the storage unit 110.

Hereinafter, it is assumed that the open source data of this embodiment is an open source of an Android application, and machine learning is an algorithm for detecting or classifying malware in an Android application.

In the storage unit 110, open source of an Android application published online or offline, such as F-droid, is collected and stored.

The setting unit 140 includes: a build analysis unit 142 for identifying a build system type of open source data and generating a missing build required file; An environment adjustment unit 143 for setting environment or option information suitable for open source data; A library construction unit 145 that applies a library called by open source data; And a correction unit 144 for correcting grammatical errors and updating grammar of open source data.

When the open source data is an Android application, the build analysis unit 142 identifies whether the build system corresponds to Eclipse, Android studio, or Gradle.

In addition, the build analysis unit 142 automatically generates missing build required files such as build.xml and local.properties.

The environment adjustment unit 143 sets environment or option information suitable for open source data. The environment or options include the Android operating system version (Android 4.1~4.3 Jelly Bean, Android 4.4 KitKat, Android 5.0~5.1 Lollipop, Android 6.0 Marshmallow, Android 7 Nougat, Android 8 Oreo, Android 9 Pie), compiler version information, and obfuscation applied. Tools or packing tools, compilation optimization or obfuscation options. For example, as of Android 6.0 Marshmallow, the permission mode is applied with runtime permission, so the application is built differently from the previous version.

The library configuration unit 145 applies a library required for open source data. Examples of libraries are Facebook, Gson, Flurry Analytics, Bolts, Crashlytics, OKHttp, Nine Old Androids, Picasso, Retrofit, etc.

When the build analysis unit 142 determines that the build system of the open source data is Eclipse or Android Studio, the necessary libraries are automatically downloaded from the Google library and installed.

Meanwhile, if the build system is a gradle, the library configuration unit 145 automatically downloads and installs the library included in the Google library. If the user library is not included in the Google library, and there are no repositories in the open source data, the library configuration unit 145 searches the library in the built-in library database. To this end, the library configuration unit 145 includes a library database for searching for a library of open source data. Various known user libraries are stored in the library database.

Also, old open source data may contain invalid Android server connection information. The library configuration unit 145 updates the connection information to the latest Android server information when the open source data includes invalid Android server connection information.

The correction unit 144 corrects grammar errors in the open source data and updates grammar. The correction unit 144 performs grammatical error correction and grammar updating for build-related files including local.properties, build.xml, and gradle-wrapper.properties in the Android environment.

The correction unit 144 automatically corrects errors and problems that may occur during build.

For example, check the directory structure and file configuration of the Android project. Also, if the repositories of open source data are out of date, change them to the latest version and fix related errors. Also, if the build system is a Gradle, some source code that may be wrong is corrected. If it is difficult to build because the used gradle is an old version, fix it to the latest gradle version.

The building unit 180 builds open source data with a build system of the type identified by the build analysis unit 142. To this end, the building unit 180 may rearrange the build structure of the open source data corresponding to the build system.

In addition, this embodiment further includes a concealment unit 150 to conceal by applying a code obfuscation technique or a packing technique to open source data.

Code obfuscation is a technique that makes it difficult to interpret source code or executable code. Depending on the object, obfuscation can be divided into source code obfuscation and binary obfuscation. Source code obfuscation is a technique that converts source code into a form that is difficult to recognize, and binary obfuscation is a technique that converts binary generated after compilation into difficult to analyze by reverse engineering. The obfuscation techniques include layout obfuscation, control obfuscation (control flow obfuscation = control flow obfuscation), data obfuscation, and preventive transformations.

The packing technique includes transformation techniques such as encryption, encoding, and compression. Packing includes an operation to encrypt or encode or compress the source code so that it is difficult to analyze the source code, and an operation to artificially insert unnecessary code. In the case of encryption, the application is encrypted and distributed or stored using a specific key, and if necessary, decrypted using the key so that the application can be normally executed. A powerful packing tool packs dozens to hundreds of times, and includes anti-debugging techniques in the middle of the code, making it difficult to interpret.

When malware is obfuscated or packed and included in an application in a concealed state, machine learning learned with an application data set that is not concealed may not detect or classify the malware. However, when the concealment unit 150 obfuscates or packs the open source data, it is possible to prepare the concealed application dataset from a single open source data in various ways. Machine learning learned with obfuscated application datasets or packed application datasets improves malware detection and classification for concealed applications.

In addition, this embodiment further includes a signature unit 160 in which the building unit 180 adds an electronic signature when building the open source data.

In addition, this embodiment may further include a malware insertion unit 170 that inserts learning malware code necessary for machine learning learning into the open source data. The malware insertion unit 170 includes previously released source code of malware.

In order to learn malware detection and classification machine learning, a dataset of malware-infected applications is also required. However, collecting malware-infected applications or their open source data is more difficult than collecting non-infected applications or open source data.

Therefore, the malware insertion unit 170 may randomly generate an application infected with malware by additionally inserting the source code of the previously released malware into a plurality of open source data. There are viruses, worms, Trojan horses, backdoors, logical bombs, bots, adware, spyware, ransomware, and the like, and there are tens to thousands of malwares for each type. By inserting the source codes of these various malware into one open source, it is possible to acquire applications infected with at least one thousand malware. Since a plurality of open source data are stored in the storage unit 110 of this embodiment, application of a malware source code to each open source data by type enables provision of an application data set infected with hundreds of millions of malware.

The malware inserting unit 170 inserts the malware source code into the location corresponding to the execution region in the open source data.

The malware insertion unit 170 labels what malware is applied to the open source data in which the malware source code is arbitrarily inserted. The label may include information such as the name of the malware and the type of malware. As a result, the administrator can select only the applications infected with the malware from among the applications built in this embodiment, and can also select the type of the required malware.

In particular, when the malware insertion unit 170 inserts malware code into the open source data, and the hiding unit 150 conceals the corresponding open source data, it becomes possible to generate an application dataset of a type close to infinity.

Next, an application automation build method for generating a learning dataset of machine learning according to an embodiment of the present invention will be described.

Referring to FIG. 2, this embodiment includes the step of storing, in the storage unit 110, a plurality of open source data for machine learning (S110); The setting unit 140 identifies the build system type of the open source data stored in the storage unit 110, sets a suitable environment or option information, and sets a system including application of necessary libraries (S120); And a step in which the building unit 180 builds the open source data in an environment or option set by the setting unit 140 in a system setting to generate an application data set (S160).

In addition, this embodiment further includes a step S130 of inserting the malware for learning necessary for learning machine learning into the open source data by the malware inserter 170 before step S160.

In addition, this embodiment further includes a step S140 of concealing by applying an obfuscation technique or a packing technique to the open source data by the concealment unit 150 before the step S160. Step S140 is preferably executed after step S130 so that the malware code can also be concealed.

In addition, this embodiment further includes a step (S150) of adding the digital signature to the open source data by the signing unit 160 before the step S160.

After the step S160, the output unit 190 is provided to the learning device 200 for learning the machine learning included in the malware diagnosis classification device 300, or the application data set generated in the separate storage for use in a suitable environment in the future Save (S170).

Although the preferred embodiments of the present invention have been described above, the present invention can use various changes, modifications, and equivalents. It is clear that the present invention can be equally applied by appropriately modifying the above embodiments. Therefore, the above description is not intended to limit the scope of the present invention as defined by the following claims.

100: application automation build device
110: storage unit 120: list generation unit
140: setting unit 142: build analysis unit
143: environmental adjustment unit 144: calibration unit
145: library component 150: concealed
160: signature unit 170: malware insertion unit
180: construction unit 190: output unit
200: machine learning learning device 300: malware detection classification device

Claims (13)

A storage unit storing a plurality of open source data for machine learning learning;
A setting unit for identifying a build system type of open source data stored in the storage unit, setting a suitable environment or option information, and setting a system including application of necessary libraries; And
An application automation build device for creating a learning dataset for machine learning, comprising a building unit that builds the open source data in an environment or option set by the setting unit in a system setting to generate an application data set. According to claim 1,
Application automation build device for machine learning learning data set generation, characterized in that it further comprises a concealment unit for concealing by applying an obfuscation technique or a packing technique to the open source data. According to claim 1,
An application automation build device for generating a learning dataset of machine learning, characterized in that the building unit further comprises a signature unit that adds an electronic signature when building the open source data. According to claim 1,
An application automation build device for creating a learning dataset for machine learning, further comprising a malware inserter for inserting learning malware code necessary for learning machine learning into the open source data. According to claim 1, The setting unit,
A build analysis unit for identifying the build system type of the open source data and generating a missing build required file;
An environment adjustment unit for setting environment or option information suitable for the open source data;
A library configuration unit to apply a library called by the open source data; And
An application automation build device for creating a learning dataset of machine learning, comprising a correction unit for correcting grammar errors and updating grammar of the open source data. The method of claim 5,
The machine learning is an algorithm for malware detection or classification of Android applications,
The open source data is the open source of the Android application,
The build analysis unit is an application for generating a learning dataset of machine learning, characterized by identifying whether the build system of the open source data corresponds to Eclipse, Android studio, or Gradle. Automated build device. The method of claim 6,
The library constructing unit includes a library database for searching the required library of the open source data. The application automation build device for creating a learning dataset of machine learning. The method of claim 6,
When the open source data includes connection information of an invalid Android server, the library configuration unit updates the connection information to the latest Android server related information, thereby automating an application for creating a learning dataset of machine learning. Build device. The method of claim 6,
The proofing unit performs grammatical error correction and grammar updating for build-related files including local.properties, build.xml, and gradle-wrapper.properties to automate the application for generating a learning dataset of machine learning. Build device. (A) the storage unit storing a plurality of open source data for machine learning learning;
(B) the setting unit performs a system setting including identifying a build system type of open source data stored in the storage unit, setting an appropriate environment or option information, and applying a necessary library; And
(C) Application automation build method for creating a learning dataset of machine learning, characterized in that the building unit comprises the steps of creating an application dataset by building the open source data in an environment or option set by the setting unit. The method of claim 10, before step (C),
A method of building an application automation for generating a learning dataset of machine learning, further comprising the step of concealing by applying an obfuscation technique or a packing technique to the open source data. The method of claim 10, before step (C),
The signing unit further comprises the step of adding an electronic signature to the open source data Application automation build method for creating a learning dataset of machine learning. The method of claim 10, before step (C),
A method for automatically building an application for generating a learning dataset of machine learning, characterized in that the malware insertion unit further includes inserting learning malware code necessary for learning machine learning into the open source data.

Images