RU2758358C2 - Method for generating signature for spam detection - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: method for generating a signature for spam detection contains stages, at which a set of electronic messages is formed to check for spam from a stream of electronic messages; based on attribute values of each electronic message from the generated set of electronic messages, at least one feature is found for spam detection; a decision tree is formed for spam detection using all found features; using the decision tree and the generated set of electronic messages, a list of found signs for detecting spam that have worked more than once is identified; a signature for detecting spam is formed based on the identified list of signs.

EFFECT: ensuring information security in conditions of mass mailing of electronic messages.

14 cl, 4 dwg

Description

Technology area

The invention relates to the field of information security, and more specifically to systems and methods for creating signatures to combat spam.

State of the art

Online advertising is one of the cheapest ways to advertise. Spam messages, as the main and most massive form of advertising in the modern world, account for 70% of the total volume of mail traffic.

Spam - mass mailing of advertisements or other types of information to persons who did not express a desire to receive them. Spam includes messages sent by email, instant messaging protocols, social networks, blogs, dating sites, forums, and SMS and MMS messages.

Due to the constant growth in the volume of spam mailing, problems of a technical, economic and criminal nature arise. The load on equipment and data transmission channels, user time spent on processing messages, changing the direction of messages towards fraud and theft - these and other aspects show the urgent need for a continuous fight against spam.

There are many ways to counter spam. One effective way is to use trained machine learning models to identify spam emails.

For example, publication US 8180834 B2 describes a system in which additional training of classifiers is periodically carried out, which are used to detect spam. Additional training can be performed upon detection of errors or at the initiative of the user.

This solution classifies e-mails using machine learning tools, but does not effectively solve the problem of creating a signature to detect spam in sets of messages sent by e-mail.

Disclosure of invention

The invention relates to systems and methods for generating anti-spam signatures.

The technical result of the present invention is to ensure information security in conditions of mass mailing of electronic messages. The specified technical result is achieved by generating a signature for detecting spam based on a decision tree and a set of electronic messages

In one embodiment, a method for generating a signature for detecting spam is provided, comprising the steps of: generating a set of electronic messages; calculating at least one feature for detecting spam based on the generated set of electronic messages; generating a decision tree for detecting spam using all the calculated features; generating a signature for spam detection based on the generated decision tree and the generated set of electronic messages.

In another embodiment of the method, a feature for detecting spam is understood as a feature calculated based on the values of an attribute of an electronic message that characterize the presence of spam.

In yet another embodiment of the method, a signature for spam detection is understood as a list of features for detecting spam and their values characteristic of an electronic message containing spam.

In another embodiment of the method, the set of electronic messages consists of a set of electronic messages for forming a decision tree, which contains at least two electronic messages, and a test set of electronic messages, which contains at least two electronic messages.

In another embodiment of the method, the set of email messages for generating the decision tree does not include email messages from the test set of email messages.

In yet another embodiment of the method, the attribute values of each email from the set of email messages are identified to generate a decision tree.

In another embodiment of the method, features for spam detection are calculated based on the detected attribute values.

In another embodiment of the method, the generated spam detection decision tree is used to analyze each electronic message from the test set of messages.

In another embodiment of the method, based on the analysis results, a list of features for detecting spam that has been triggered more than once is identified.

In yet another embodiment of the method, a signature for detecting spam is generated based on the identified list of features for detecting spam.

In another embodiment of the method, in the absence of a list of features that have been triggered more than once, the generated decision tree is retrained to detect spam.

Brief Description of Drawings

FIG. 1 depicts the structure of a decision tree for spam detection and signatures for spam detection.

FIG. 2 illustrates a block diagram of a signature generation system for spam detection.

FIG. 3 illustrates a flow chart of a signature generation system for spam detection.

FIG. 4 shows an example of a general purpose computer system.

Although the invention may take various modifications and alternative forms, the characteristic features shown by way of example in the drawings will be described in detail. It should be understood, however, that the purpose of the description is not to limit the invention to a specific embodiment. On the contrary, the purpose of the description is to cover all changes, modifications falling within the scope of this invention, as defined by the appended claims.

Description of embodiments of the invention

The objects and features of the present invention, methods for achieving these objects and features will become apparent by reference to exemplary embodiments. However, the present invention is not limited to the exemplary embodiments disclosed below, but may be embodied in various forms. The essence recited in the description is nothing more than the specific details necessary to assist a person skilled in the art in a thorough understanding of the invention, and the present invention is defined within the scope of the appended claims.

Let's introduce a number of definitions and concepts that will be used in describing the embodiments of the invention.

Electronic mail (e-mail, e-mail) - a set of computer network services for sending messages between its users. It is a means of fast delivery of letters, program texts, documents and other similar correspondence. When transmitting a message by e-mail, the transmitting and receiving computers do not necessarily interact with each other directly (Dorot V.L., Novikov F.A. .: BHV-Petersburg, 2004 .-- 608 p.: Ill.).

Email or message - According to RFC 5322, it is a sequence of characters. Messages conforming to this specification include characters with decimal codes 1 through 127, interpreted as US-ASCII. A message consists of header fields (collectively referred to as the message header section), which may be followed by the main body of the message. A header section is a sequence of character strings, the syntax for which is described in this specification. The body of a message is a sequence of characters that follows the header section and is separated from it by a blank line (a line containing only CRLF). Hereinafter and in the text, an electronic message is understood to mean an electronic message transmitted by electronic mail.

An electronic message attribute is a necessary, essential, inherent property of an electronic message.

Spam detection attribute is a attribute calculated based on the value of an email message attribute that characterizes the presence of spam and is used when using machine learning technologies.

Message flow is a collection of electronic messages transmitted by email that one or more users receive. Set of e-mail messages - a fixed number of messages selected from the message flow to perform a spam check.

The classic virus signature is a contiguous sequence of bytes typical of a particular malicious application. Signature for spam detection - a list of signs for detecting spam and their specific values, characteristic of a particular e-mail message containing spam.

Decision tree is a data analysis method for building classification and regression models, it is both an extraction method and at the same time a data presentation method. A decision tree is a way of representing rules in a hierarchical, sequential structure, where each object has a single node that gives a solution.

The creator of bulk email messages containing spam typically uses a template to generate the text and content of the referenced email messages. To make spam detection more difficult, it can also use many tools, such as obfuscation, anonymization, etc. Decision trees can be generated to identify patterns and similar algorithms for generating spam messages. FIG. 1 illustrates the structure of a decision tree for spam detection and signatures for spam detection. Decision tree 110 is used to classify groups of email messages. Computed features are used as tree nodes. Specific values or ranges of feature values can be used as transitions between nodes. Based on several nodes and transitions, a signature 120 can be generated.

Spam detection signature generation is performed by a spam detection signature generation system. FIG. 2 depicts a block diagram of a signature generation system for detecting spam, which includes a set of electronic messages 210, recognition means 220, calculator 230, and generation means 240.

Recognizer 220 is configured to generate a set of electronic messages and transmit the generated set of electronic messages to calculator 230. Set of electronic messages 210 consists of a set of electronic messages for generating a decision tree and a test set of electronic messages. The set of e-mails for forming a decision tree contains at least two e-mails, the said set is used at the stage of building or changing the decision tree. The verification set of e-mails contains at least two e-mails, the set is used at the stage of building signatures for detecting spam. However, the set of e-mails for forming the decision tree does not contain e-mails from the test set of e-mails.

In one implementation, the formation of a set of electronic messages 210 is performed by adding messages received by different users over a given period of time. In another implementation, the formation of a set of electronic messages 210 is performed by adding messages received by one user over a certain period of time. In yet another implementation, the generation of the set of electronic messages 210 is performed by adding a fixed number of messages. The optimal size of an email set 210 depends on the frequency of receiving messages. The size is determined empirically by incrementally increasing, for example, the time interval over which a set of emails 210 is generated.

The calculator 230 is for calculating features for detecting spam based on the generated set of email messages 210, generating a decision tree for detecting spam using the computed features, and transmitting data about the generated decision tree for detecting spam to the generating means 240.

In one implementation, the spam detection features are calculated based on the attributes of the email message. The determination of the attributes of an electronic message is performed by analyzing the process of sending and receiving the message. Examples of email attributes are: the sender's IP address, the size of the email, the language of the email text, the number of characters in the email header, the size of the email, and so on.

Examples of computed features for detecting spam are the following features: the presence of a dynamic PTR record for an IP address (from the English pointer) connects the IP address of a host with its canonical name); checksum from HTML layout without variable attributes; msgid - unique message number; msgid_type - heuristically determined through the appearance of the header; msgid agent that sent the message; checksum from the sequence of MIME headers; type of email content, etc.

Each of the calculated features is empirically given a weight w _i that has been calculated based on predetermined statistics. The weight value, to one degree or another, characterizes the presence of spam. After calculating the features, a decision tree is generated for detecting spam. For example, the start of the tree can be mailer_name (the application with which the email was sent), which can take 3 values. If the value 1 is accepted, then the transition to the attribute max_url_length (the maximum value of the URL length in the letter) occurs, if the value 2 is accepted, then the transition to the attribute msgid_type occurs, etc.

Generator 240 is for generating a signature for spam detection based on the generated decision tree and the generated set of email messages.

The generated decision tree is used to parse the email messages from the test set of email messages from the email set 210. The analysis determines the message attributes, computes the spam detection features, and applies the decision tree to spam detection. After applying the tree, the e-mail falls into a group that has a total weight calculated in accordance with the weights of the calculated features according to the formula W _sum = Σw _i . The limit value of the total weight of the groups is determined empirically. Groups of e-mails with an aggregate weight above the limit are considered spam, and those below the limit are considered non-spam. Generation of a signature for spam detection is performed by identifying a list of features for detecting spam that have been triggered more than once in a group of spam e-mail messages.

If the list of signs for detecting spam that did not work more than once in any of the groups was not identified, retraining of the generated decision tree for detecting spam is started. During retraining, the decision tree is modified using machine learning technologies, for example, gradient boosting, where alternative tree variants, their depth and width are formed, and indicators of efficiency, reliability, and false positives are calculated.

Using signatures allows you to process a large number of emails much faster than performing analysis using the entire decision tree. The signature is included in a set of updates for the spam filter, which subsequently checks for spam in order to ensure information security of the e-mail transmission process.

For example, a user received four emails. The first and second messages are sent to a set of emails to form a decision tree. Revealed attributes of the first message:

• text of the message: "BUY FLOWERS", instead of spaces, random symbols of white color are used;

• IP address: 191.157.1.1;

• message size: 1 Kb;

• attached file: 200 Kb, 32.jpg.

Revealed attributes of the second message:

• text of the message: "BUY FLOWERS", instead of spaces, random symbols of white color are used;

• IP address: 181.147.2.2;

• message size: 1.5 Kb;

• attached file: 300 Kb, 32.bmp.

Computed features from the attributes of the first message:

- Sign 1 (unknown IP address) - yes, weight 0.05;

- Sign 2 (message size <10 kB) - yes, weight 0.1;

- Sign 3 (hidden text) - yes, weight 0.3;

- Sign 4 (graphic format file) - yes, weight 0.1.

Computed features from the attributes of the second message:

- Sign 1 (unknown IP address) - yes, weight 0.05;

- Sign 2 (message size <10 kB) - yes, weight 0.1;

- Sign 3 (hidden text) - yes, weight 0.3;

- Sign 4 (graphic format file) - yes, weight 0.1.

One of the variants of the decision tree for detecting spam, generated on the basis of a set of e-mail messages for forming a decision tree, will be the following decision tree: feature 1 with branches "yes" or "no", if feature 1 has the value "yes", then a transition to feature 2 with branches "is" or "no", if feature 2 has the value "is", then there is a transition to feature 4 with branches "is" or "no", if feature 4 has the value "is", then there is a transition to feature 3 with "yes" or "no" branches, if feature 3 has the value "yes", then the email message will fall into group 1, if feature 3 has the value "no", then the email message will fall into group 2, and so on. .d. The total weight of group 1 will be 0.55. The total weight of group 2 will be 0.25.

To apply the generated decision tree, a test set of e-mail messages is used. The third and fourth emails fall into the test set of emails. Revealed attributes of the third message:

• text of the message: "BUY FLOWERS", instead of spaces, random symbols of white color are used;

• IP address: 193.153.1.1;

• message size: 1.7 Kb;

• attached file: 250 Kb, 32.png.

Revealed attributes of the fourth message:

• message text: "Happy Birthday !!!", no white characters;

• IP address: 192.161.7.2;

• message size: 0.5 Kb;

• attached file: 250 Kb, postcard.jpg.

Computed features from the attributes of the third message:

- Sign 1 (unknown IP address) - yes;

- Sign 2 (message size <10 kB) - yes;

- Sign 3 (hidden text) - yes;

- Sign 4 (graphic format file) - yes.

Calculated features from the attributes of the fourth message:

- Sign 1 (unknown IP address) - yes;

- Sign 2 (message size <10 kB) - yes;

- Sign 3 (hidden text) - no;

- Sign 4 (graphic format file) - yes.

As a result of applying the generated decision tree, the third message will fall into group 1, the fourth message - into group 2. The limit value of the total weight is set to 0.5. Thus, the third message is a spam email. A list of features that have triggered more than once with a weight greater than 0.08 is identified. A signature is formed, where the list of features consists of feature 2 associated with feature 4 through the value "is", feature 4 associated with feature 3 with the value "is", and feature 3, which has the value "is".

FIG. 3 illustrates an algorithm for generating a signature for spam detection. In step 311, the recognition means 220 generates a set of electronic messages and transmits the generated set of electronic messages to the calculator 230. In step 312, the calculator 230 calculates features for detecting spam based on the generated set of electronic messages. At step 313, the calculator 230 generates a decision tree for spam detection using the calculated features, and sends data about the generated decision tree for detecting spam to the generating means 240. At step 314, the generating means 240 generates a signature for spam detection based on the generated a decision tree and a generated set of emails.

FIG. 4 shows an example of a general-purpose computer system, a personal computer or server 20, comprising a central processing unit 21, a system memory 22, and a system bus 23 that contains various system components, including memory associated with the central processing unit 21. The system bus 23 is implemented as any bus structure known from the prior art, containing in turn a bus memory or a bus memory controller, a peripheral bus and a local bus that is capable of interfacing with any other bus architecture. System memory contains read-only memory (ROM) 24, random access memory (RAM) 25. The main input / output system (BIOS) 26 contains basic procedures that transfer information between the elements of the personal computer 20, for example, at the time of loading the operating room. systems using ROM 24.

The personal computer 20, in turn, contains a hard disk 27 for reading and writing data, a magnetic disk drive 28 for reading and writing to removable magnetic disks 29 and an optical drive 30 for reading and writing to removable optical disks 31, such as CD-ROM, DVD -ROM and other optical media. The hard disk 27, the magnetic disk drive 28, and the optical drive 30 are connected to the system bus 23 via the hard disk interface 32, the magnetic disk interface 33, and the optical drive interface 34, respectively. Drives and corresponding computer storage media are non-volatile storage media for computer instructions, data structures, program modules and other data of a personal computer 20.

The present description discloses an implementation of a system that uses a hard disk 27, a removable magnetic disk 29 and a removable optical disk 31, but it should be understood that other types of computer storage media 56 that are capable of storing data in a computer readable form (solid state drives, flash memory cards, digital disks, random access memory (RAM), etc.), which are connected to the system bus 23 through the controller 55.

Computer 20 has a file system 36, where the recorded operating system 35 is stored, as well as additional software applications 37, other program modules 38 and program data 39. The user has the ability to enter commands and information into the personal computer 20 through input devices (keyboard 40, manipulator " mouse "42). Other input devices may be used (not shown): microphone, joystick, game console, scanner, etc. Such input devices are conventionally connected to the computer system 20 through a serial port 46, which in turn is connected to the system bus, but can be connected in another way, for example, using a parallel port, a game port, or a universal serial bus (USB). A monitor 47 or other type of display device is also connected to the system bus 23 via an interface such as a video adapter 48. In addition to the monitor 47, the personal computer may be equipped with other peripheral output devices (not displayed), such as speakers, a printer, etc. ...

The personal computer 20 is capable of operating in a networked environment using a network connection with other or more remote computers 49. The remote computer (or computers) 49 are the same personal computers or servers that have most or all of the elements mentioned earlier in the description of the entity. the personal computer 20 shown in FIG. 4. In a computer network, there may also be other devices, such as routers, network stations, peer-to-peer devices, or other network nodes.

Network connections can form a local area network (LAN) 50 and a wide area network (WAN). Such networks are used in corporate computer networks, internal networks of companies and, as a rule, have access to the Internet. In LAN or WAN networks, personal computer 20 is connected to local network 50 via a network adapter or network interface 51. When using networks, personal computer 20 may use a modem 54 or other means of providing communication with a wide area network, such as the Internet. Modem 54, which is an internal or external device, is connected to the system bus 23 via a serial port 46. It should be noted that the network connections are only exemplary and are not required to reflect the exact configuration of the network, i. E. in fact, there are other ways of establishing a connection by technical means of communication of one computer with another.

In conclusion, it should be noted that the information given in the description are examples, which do not limit the scope of the present invention defined by the claims.

Claims (19)

1. A method for generating a signature for spam detection, comprising the steps of: a) generate a set of e-mail messages to check for spam from the e-mail stream; b) based on the values of the attributes of each electronic message from the generated set of electronic messages, calculates at least one feature for detecting spam; c) form a decision tree for spam detection using all calculated features; d) using a decision tree and a generated set of e-mail messages, a list of computed features for detecting spam that has been triggered more than once is identified; e) generate a signature for detecting spam based on the identified list of features. 2. The method according to claim 1, according to which a feature for detecting spam is understood as a feature calculated based on the values of an attribute of an electronic message that characterize the presence of spam. 3. The method according to claim 1, according to which a signature for spam detection is understood as a list of features for detecting spam and their values characteristic of an electronic message containing spam. 4. The method according to claim 1, wherein the set of electronic messages consists of a set of electronic messages for forming a decision tree, which contains at least two electronic messages, and a test set of electronic messages, which contains at least two electronic messages. 5. The method according to claim 4, wherein the set of electronic messages for forming the decision tree does not contain electronic messages from the test set of electronic messages. 6. The method according to claim 5, according to which the attribute values of each electronic message from the set of electronic messages are identified to form a decision tree. 7. The method according to claim 6, in which features for detecting spam are calculated based on the detected attribute values. 8. The method according to claim 1, wherein the generated spam detection decision tree is used to analyze each electronic message from the test set of messages. 9. The method according to claim 8, according to which, based on the analysis results, a list of signs for detecting spam that have been triggered more than once is identified. 10. The method according to claim 9, according to which a signature for detecting spam is generated based on the identified list of features for detecting spam. 11. The method according to claim 9, according to which, in the absence of a list of features that have been triggered more than once, the generated decision tree is retrained to detect spam. 12. The method according to claim 4, according to which the formation of a set of electronic messages is performed by adding messages received by different users for a certain period of time. 13. The method according to claim 4, according to which the formation of a set of electronic messages is performed by adding messages received by one user for a certain period of time. 14. The method according to claim 4, according to which the formation of the set of electronic messages is performed by adding a fixed number of messages.

Images