RU2796490C2 - Parental control systems and methods for detecting the disclosure of confidential information - Google Patents

Abstract

FIELD: computer security.

SUBSTANCE: invention relates to computer security. The technical result is achieved due to the fact that when analyzing the transmitted images, distinctive visual features of a physical object are identified, and each indicator from a set of indicators indicates whether the image shows a distinctive the visual feature of the physical object, including the type of physical object, and transmit a parental notification indicating the disclosure of confidential information to minors.

EFFECT: technical result is to increase the accuracy of detecting a network threat.

15 cl, 14 dwg

Description

RELATED APPLICATIONS

[0001] This application claims priority over the filing date of U.S. Provisional Application No. 62/794,856, filed Jan. 21, 2019, titled Parental Control Systems and Methods, the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

[0002] The present invention relates to computer security, and more particularly to systems and methods for protecting vulnerable Internet users (e.g., children) from online threats such as cyberbullying, online abuse, grooming, sexual exploitation, and identity theft, among others.

[0003] Bullying is commonly defined as repetitive aggressive behavior intended to harm another person physically, mentally, or emotionally. Bullying can manifest itself in many ways, such as verbally, physically, etc. When bullying occurs through modern means of communication such as email and social media posting, it is commonly referred to as cyberbullying. Successful bullying usually requires an imbalance of power and/or peer pressure, with the weak side taking the abuse. Bullying is known to cause severe suffering and in some cases even lead to suicide. Some social categories (children, young people, members of racial or sexual minorities) may be more susceptible to such threats than others.

[0004] With the rapid growth of the Internet, children and adolescents spend a significant amount of time browsing and interacting online at a stage in their physical and emotional development when they are particularly vulnerable to threats such as bullying, sexual exploitation, and identity theft. The problem is exacerbated by the fact that the online culture of social media and instant messaging is beyond the control of traditional authorities (parents, teachers, etc.), either because young users are often more technologically savvy than their guardians, or because that the communication platforms themselves do not provide for the possibility of tracking.

[0005] In recent years, security software has been successfully used to protect computer users from computer threats such as malware (malware) and intrusion (hacking). There is currently considerable interest in developing software that can protect users from other emerging threats such as cyberbullying, grooming, sexual exploitation and online harassment, ideally while maintaining the privacy of their email communications.

DISCLOSURE OF THE INVENTION

[0006] According to one aspect, a parental control method includes using at least one computer system hardware processor to analyze an image transmitted as part of an electronic dialogue between a minor and another party to determine if the image shows a selected physical item from the group consisting of a banking cards, social security cards and identity cards. The method further includes, in response, when the image shows a physical item, using at least one hardware processor to send a parent notification to a parent reporting device identified from a plurality of devices according to the minor, the parent notification indicating disclosure of confidential information to the minor.

[0007] According to another aspect, the computer system includes at least one hardware processor configured to execute an image processor and a parent notification manager. The image processor is configured to analyze an image transmitted as part of an electronic dialogue between the minor and the other party to determine if the image shows a selected physical item from the group consisting of a bank card, a social security card, and an identity card. The parent notification manager is configured, in response to the image processor's determination that the image shows a physical item, to send a parent notification to a parent reporting device identified from a plurality of devices according to the minor, the parent notification indicating disclosure of confidential information to the minor.

[0008] According to another aspect, the non-volatile computer-readable medium stores instructions that, when executed by at least one hardware processor of a computer system, cause the computer system to generate an image processor and a parent notification manager. The image processor is configured to analyze an image transmitted as part of an electronic dialogue between the minor and the other party to determine if the image shows a selected physical item from the group consisting of a bank card, a social security card, and an identity card. The parent notification manager is configured, in response to the image processor's determination that the image shows a physical item, to send a parent notification to a parent reporting device identified from a plurality of devices according to the minor, the parent notification indicating disclosure of confidential information to the minor.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above aspects and advantages of the present invention will become more clear after reading the following detailed description, given with reference to the drawings, which depict the following:

[0010] FIG. 1 illustrates an exemplary parental control system in which a monitored device participating in an email exchange is protected from online threats, in accordance with some embodiments of the present invention.

[0011] FIG. 2-A shows an exemplary communication between a monitored device, a security server, and a reporting device, in accordance with some embodiments of the present invention.

[0012] FIG. 2-B shows an alternative communication between a monitored device, a security server, and a reporting device, in accordance with some embodiments of the present invention.

[0013] FIG. 3 shows exemplary software components executing on a monitored device in accordance with some embodiments of the present invention.

[0014] FIG. 4 illustrates the operation of an exemplary parental control application running on a monitored device in accordance with some embodiments of the present invention.

[0015] FIG. 5 illustrates an exemplary dialog indicator in accordance with some embodiments of the present invention.

[0016] FIG. 6 shows an exemplary sequence of steps performed by a parental control application in accordance with some embodiments of the present invention.

[0017] FIG. 7 shows an exemplary sequence of steps performed by a message aggregator to build a set of conversations in accordance with some embodiments of the present invention.

[0018] FIG. 8 shows exemplary software components running on a security server, in accordance with some embodiments of the present invention.

[0019] FIG. 9 shows an exemplary operation of the software components shown in FIG.

[0020] FIG. 10 shows an exemplary sequence of steps performed by a security server in accordance with some embodiments of the present invention.

[0021] FIG. 11 illustrates a set of exemplary word processors in accordance with some embodiments of the present invention.

[0022] FIG. 12 shows a set of exemplary image processors in accordance with some embodiments of the present invention.

[0023] FIG. 13 illustrates exemplary body parts that an image processor is trained to detect in an image, in accordance with some embodiments of the present invention.

[0024] FIG. 14 shows an exemplary hardware configuration of a computing device configured to perform parental control operations in accordance with some embodiments of the present invention.

IMPLEMENTATION OF THE INVENTION

[0025] It should be understood that in the following description, all listed connections between structures can be direct functional connections or indirect functional connections through intermediate structures. An element set includes one or more elements. Any reference to an element is considered to refer to at least one element. The set of elements includes at least two elements. Unless otherwise noted, any use of "OR" refers to a non-exclusive or. Unless otherwise required, any of the method steps described need not be performed in the specific illustrated order. The first element (eg, data) derived from the second element includes the first element equal to the second element, as well as the first element generated by processing the second element and optionally other data. Making a determination or decision in accordance with a parameter includes making a determination or decision in accordance with the parameter and, optionally, in accordance with other data. Unless otherwise indicated, an indicator of a quantity/data may be the quantity/data itself or an indicator other than the quantity/data itself. A minor is a person who has not reached the age of full legal responsibility. A computer program is a sequence of processor instructions that perform a task. The computer programs described in some embodiments of the present invention may be stand-alone program objects or sub-objects (eg, routines, libraries) of other computer programs. Computer-readable media includes non-durable media such as magnetic, optical, and semiconductor storage media (eg, hard disks, optical disks, flash memory, DRAM) as well as communication channels such as conductive cables and fiber optic lines. In accordance with some embodiments, the present invention provides, among other things, computer systems comprising hardware (e.g., one or more processors) programmed to perform the methods described herein, as well as machine-readable media encoding instructions for performing the methods described herein.

[0026] The following description illustrates embodiments of the invention by way of example, and not necessarily as a limitation.

[0027] FIG. 1 shows an exemplary parental control system that protects the user of a monitored device from online threats such as cyberbullying, sexual exploitation, and privacy theft, among others. In a typical scenario in accordance with some embodiments of the present invention, a protected user (for example, a minor) uses messaging software running on a monitored device 10 (for example, a smartphone) to exchange electronic messages with users of other partner devices 12a-b over messaging. In some embodiments, security software running on monitored device 10 and/or remote security server 18 may be used to monitor such conversations, typically without the knowledge of the respective user. The dialogues are then analyzed for content. When the security software determines, according to the contents of the dialog, that the user is exposed to an online threat, some embodiments send a notification to the other party (e.g., parent, teacher, manager, etc.) via a reporting device 14 such as a smartphone or personal computer .

[0028] Monitored device 10 may include any electronic device having a processor and memory and capable of connecting to a communications network to exchange electronic messages with messaging partner devices 12a-b. Examples of monitored devices 10 include personal computers, laptop computers, tablet computers, smartphones, gaming consoles, virtual assistant devices, home appliances (eg, smart TVs, media players, refrigerators), and wearable computing devices (eg, smart watches).

[0029] An electronic message includes communication between two electronic devices, the communication including at least the encoding of a text message between two human users of the respective devices. Electronic messaging is usually carried out using an instant messaging platform such as FACEBOOK® Messenger®, Instagram® Direct®, Snapchat®, WhatsApp®, etc., via email (email) and/or via a telephone messaging service such as as short message service (SMS). Messaging platforms contain software configured to allow a user to send and receive electronic messages to other users. Messages may vary in format depending on the respective platform/service, but in general, an e-mail message includes text part encoding and/or multimedia file encoding (eg, image, clip, sound, etc.). The text portion may contain natural language text (eg, English, Chinese, etc.) as well as other alphanumeric and/or special characters such as emoticons, among others. In a typical configuration, messages are coordinated, centralized and sent by the messaging server 16 in the sense that electronic messages between the monitored device 10 and partner devices 12a-b are routed through the server 16 (client-server protocol). In alternative embodiments, electronic messaging uses a decentralized peer-to-peer network of connections between monitored devices and their respective messaging partner devices, monitored device 10, partner device or messaging partner devices 12a-b, and messaging server 16 are interconnected by a communication network. 15, such as the Internet. Portions of network 15 may include a local area network (LAN) and a telecommunications network (eg, mobile telephony).

[0030] Threat detection operations may be divided between the monitored device 10 and the security server 18 in various ways, as detailed below. Server 18 is generally a collection of interconnected computers that may or may not be in physical proximity to each other. Fig. 2-A-B show exemplary communications between monitored device 10 and security server 18 in accordance with some embodiments of the present invention. In various embodiments, the monitored device 10 may transmit conversation data (represented by the conversation indicator 20 in FIG. 2-A) and/or information indicative of a threat (represented by the risk assessment indicator 22 in FIG. 2-B) to the security server 18. At least part of the threat analysis/detection related to the dialog can then be performed by components running on the security server 18 . When analysis indicates a potential threat to the user of the monitored device 10, some embodiments of the security server 18 send a parent notification 24 to the reporting device 14 (e.g., mobile phone, personal computer, etc.) associated with the respective monitored device, thereby informing the user reporting device 14 about the corresponding threat. The term "parent" is used here for simplicity only and is not intended to be limiting in the sense that the recipient of the corresponding notification is necessarily the parent, or that the protected user is necessarily the child. While exemplary applications of some embodiments are under parental control, one skilled in the art will appreciate that they can be adapted to track and/or protect other categories of users/devices. In child tracking apps, notification 24 can be sent to a teacher, guardian, or any other person assigned to look after the child in question. In other exemplary applications aimed at protecting employees from bullying and/or sexual harassment, notification 24 may be delivered to, for example, a manager, supervisor, or human resources officer. Exemplary notification formats and content 24 are shown below.

[0031] FIG. 3 shows exemplary software components executing on a monitored device 10 in accordance with some embodiments of the present invention. Operating system 46a may include any widely available operating system such as Microsoft Windows®, MacOS®, Linux®, iOS®, or Android®, among others. The operating system (OS) 46a provides an interface between other computer programs (represented by applications 48 and 50) and the hardware devices of the monitored device 10.

[0032] Messaging application 48 is generally any software configured to enable a user of device 10 to exchange electronic messages with other users. Examples of messaging applications 48 include Yahoo® Messenger®, FACEBOOK®, Instagram®, and Snapchat® client applications, among others. Another example of a messaging application 48 includes an email client. Another example of a messaging application 48 includes software for running a short message service (SMS) on a mobile phone. Application 48 may display the content of each email message on an output device (eg, screen) of monitored device 10 and may further organize messages according to sender, recipient, time, subject, or other criteria. Application 48 may further receive text input from a user of device 10 (e.g., from a keyboard, touch screen, or dictation interface), formulate electronic messages in accordance with the received text input, and transmit electronic messages to the messaging server 16 and/or directly to the partner device. or the messaging partner devices 12a-b. The message format and encoding may vary depending on the messaging platform. Sending a message may include, for example, adding an encoding of the corresponding message to the outgoing queue of the communication interface of the monitored device 10.

[0033] In some embodiments, the parental control application 50 includes software configured to access, collect, and/or analyze the content of the exchange of messages between the monitored device 10 and the partner device or devices 12a-b. The parental control application 50 may be part of a larger computer security software including, but not limited to, anti-malware and intrusion detection tools. Fig. 4 shows exemplary components of a parental control application 50 in accordance with some embodiments of the present invention.

[0034] The data captor 52 is configured to extract the content of the message generated and/or received by the messaging application 48. Message content retrieval may include identifying individual electronic messages and determining message-specific characteristics such as the sender and/or recipient, time of transmission (e.g., timestamp), text of the corresponding message, and possibly other content data such as an image attached to the corresponding message. Retrieval of the content may be carried out by any method known in the art. In some embodiments, the data invader 52 covertly modifies a component of the messaging application 48 (eg, by connecting via an address) to install a software agent that notifies the data invader when the application 48 performs some specific operation, such as receiving a message or user input, and allows the data invader 52 to retrieve message information. Some embodiments retrieve the content of the message using built-in functions of the OS 46a, such as an accessibility application program interface (API). Accessibility APIs include software typically configured to collect information currently displayed on an output device (eg, screen) of monitored device 10 in order to make such information accessible to people with disabilities. One exemplary application of such accessibility APIs includes translating text on the screen into sound (voice text) so that visually impaired people can use the computer. Some embodiments of the data captor 52 are configured to invoke certain accessibility API functions to parse data structures, such as user interface trees, while the device 10 displays content generated by the messaging application 48 and thus extracts information such as names/aliases of the interlocutors of the message and the content of individual messages. Yet another embodiment of data hijacker 52 may extract message content directly from intercepted network traffic entering the messaging application 48 and/or passing through the network adapter or network adapters of the monitored device 10. Such communication interceptors may implement communication protocols such as HTTP, WebSocket and MQTT, among other things, for parsing messages and extracting structured message data. When instant messages are encrypted, some embodiments use techniques such as man-in-the-middle (MITM) to decrypt the traffic to extract the content of the message.

[0035] Some embodiments of the present invention are based on the observation that threats such as bullying and sexual grooming are typically associated with complex social dynamics and are therefore more accurately inferred from extended dialogue rather than individual messages. Thus, in some embodiments, message aggregator 54 may combine individual messages into conversations consisting of multiple messages exchanged by the same pair of interlocutors (in the case of a mutual exchange) or within the same group (for example, in the case of a group chat). The message aggregator 54 may work in conjunction with the data captor 52 to identify the sender and/or recipient of each intercepted message, organize the flow of messages into separate conversations, and output a conversation indicator 20. The operation of the message aggregator 54 is described in more detail below.

[0036] The exemplary conversation indicator 20 shown in FIG. 5 includes "User_ID" identifying the monitored device 10 and/or an individual user of the corresponding device, and "Pair_ID" uniquely identifying a pair of conversation partners. In some embodiments, the conversation indicator 20 further includes a plurality of message indicators Message_1 ... Message_n, each of which corresponds to a separate message being exchanged by the respective interlocutors. The individual message indicators may in turn include a sender and/or recipient identifier, the text content of each message (represented as MessageText_i in FIG. 5), and a timestamp indicating the point in time when the corresponding message was sent and/or received. In an alternative embodiment, the dialog indicator 20 includes concatenating the text content of all messages in the respective dialog, the individual messages, arranged in transmission order according to their respective timestamp.

[0037] The dialog indicator 20 may further include a set of media indicators (represented as MediaFile_j in FIG. 5), such as copies of image/video/audio files attached to messages belonging to the respective dialog, or network addresses/uniform resource locator (URL ) where the corresponding media file is located. Other exemplary media indicators may include a media format (encoding protocol) indicator, and so on. It will be understood by the skilled artisan that the actual data format for encoding the dialog indicator 20 may vary in different embodiments; exemplary formats include, but are not limited to, Extensible Markup Language (XML) version and Javascript Object Notation (JSON).

[0038] FIG. 6 shows an exemplary sequence of steps performed by a parental control application 50 in accordance with some embodiments of the present invention. Fig. 7 further illustrates an exemplary algorithm for constructing conversations from individual messages (step 204 in FIG. 6).

[0039] The parental control application 50 may represent each dialog as a separate data structure (eg, an object with multiple data fields). The conversations may be defined according to various criteria such as length (eg, total messages, total words) and/or time (eg, messages exchanged in a predetermined time interval). In some embodiments, the dialogue is considered active until the number of messages exceeds a predetermined value; alternatively, the conversation may be considered active as long as the time elapsed since its first message does not exceed a predetermined time threshold, and/or until the time elapsed since its last message does not exceed another predetermined threshold. time. Conversations that are no longer active are considered to have ended. In one example, illustrated in FIG. 7, the parental control application 50 keeps track of multiple active conversations, with each conversation being identified by a unique conversation ID. At 212, the amount of time that has elapsed since the last message of each active conversation is determined. When the specified amount of time exceeds a predetermined threshold (eg, one hour), the message aggregator 54 may consider the corresponding conversation closed/ended and remove it from the set of active conversations. In the next step 230, a dialogue indicator 20 for the corresponding dialogue can be formulated and the corresponding data can be passed for further analysis. Such a flowchart may describe the operation of an alternative message aggregator that considers a conversation to be completed when the number of messages exceeds a predetermined threshold (eg, 500).

[0040] In the meantime, data captor 52 may intercept new messages (block 216). When a message is detected, step 220 may identify the interlocutors of the corresponding message, for example, by parsing the message data or by parsing the user interface of the messaging application 48 (see above with respect to using the Accessibility API). When there is currently at least an active conversation with the respective buddies, at step 226 aggregator 54 may add data indicative of the current message to the conversation object identified by the current buddies (eg, pair ID). When there is currently no active conversation between the buddies of the current message, at step 224 a new conversation object identified by the current buddy/pair ID can be initialized and the message data can be added to the newly initialized object. Application 50 may then return to intercepting new messages and/or determining if any current conversation has ended.

[0041] Figures 8-9 show exemplary software components executing on the security server 18 and, accordingly, exemplary operation of such components in accordance with some embodiments of the present invention. Fig. 10 further details the operation of said components as an exemplary sequence of steps.

[0042] In some embodiments, dialog data is received from message aggregator 54 in the form of dialog indicator(s) 20 . Each indicator 20 may represent one conversation, which in turn may contain multiple messages exchanged by the same interlocutors over a period of time. In some embodiments, the dialogue indicators 20 accumulate in a queue pending further processing. Such processing may include selecting a dialog and removing it from the queue (steps 302-204-306 in FIG. 10). The selected indicator 20 is then loaded into the dialogue analyzer 51, which analyzes the content of the corresponding dialogue to determine a plurality of evaluation indicators (eg, numerical or logical scores, category labels, etc.) and output them to the decision block 53 . In a further step 312, the decision block 53 may aggregate the analysis results obtained from the dialogue analyzer 51 and apply a set of decision criteria to determine whether the user of the monitored device 10 is exposed to an online threat such as bullying, sexual harassment, grooming, etc. . In some embodiments, the decision block 53 transmits the single risk assessment indicator 22 to the notification manager 59 . At 314, the notification manager may determine if the notification condition is satisfied in accordance with the received evaluation indicator. If so, the dispatcher 59 may formulate and transmit the parent notification 24 to the reporting device 14. Some of the above steps will be detailed below.

[0043] In some embodiments, the dialogue parser 51 comprises a set of text processors 56 configured to parse the text content of the dialogue and/or a set of image processors 58 configured to parse the image and/or video content of the dialogue. Each processor 56-58 may analyze each dialogue according to a different aspect of the corresponding dialogue and/or according to a different algorithm. For example, each processor 56-58 may determine if the user is exposed to a different type of threat (bullying, sexual harassment, grooming, etc.). In another example, there may be multiple processors detecting the same type of threat, but each processor may use a different criteria or algorithm, or may consider different aspects of the conversation being parsed. For example, some word processors may search parsed dialogue for specific keywords, while others may use a neural network to obtain a metric or label that characterizes the corresponding message or dialogue, and so on. Other exemplary aspects of the dialogue include aggressiveness, friendliness, and sexual content, among others.

[0044] In some embodiments, the text content of the dialog is normalized in preparation for serving to at least some of the text processors 56 (block 308 in FIG. 10). Such normalization may include spell checking, abbreviation expansion, detection and interpretation of emoticons, URLs, names of people and/or places. The normalization may include looking up the appropriate natural language vocabulary (eg, English) augmented with slang elements and various expressions/acronyms commonly used in instant messaging.

[0045] Some exemplary text and image processors 56-58 are illustrated in FIGS. 11-12, respectively. Each word processor 56 may output a text evaluation indicator 26 . Similarly, image processors 58 may output a set of image evaluation indicators 28. One exemplary text evaluation indicator 26 includes a numeric score indicating the likelihood that at least one interlocutor is the target of an online threat (eg, bullying) according to the text content of the respective dialogue. The exemplary image rating indicator 28 may indicate whether the current dialog contains at least one image belonging to a particular category (e.g., a nude or partially nude image, a credit card photo, etc.)

[0046] Some embodiments of the present invention are based on the observation that due to the complexity of social interactions, which are especially emotionally charged during childhood and adolescence, a single analysis algorithm/protocol is unlikely to successfully detect threats such as bullying. For example, children and teenagers often refer to each other using derogatory nicknames, insults and derogatory language, even if they do not mean any harm. Such "posing" is simply considered cool or fun. Therefore, a text analysis algorithm that simply looks for insults and/or expressions that indicate conflict may misclassify a cheeky exchange between close friends as a verbal altercation or bullying. To avoid such false positives, some embodiments use multiple natural language processing algorithms to analyze various aspects of each dialogue and extract various evaluation indicators. Some embodiments for this increase the reliability of threat detection by aggregating information provided by multiple individual evaluation indicators. Image evaluation indicators can be combined with text evaluation indicators. For example, an image of a nudity can give an additional clue to the suspicion of sexting, etc.

[0047] The exemplary text processors 56 shown in FIG. 11 include, but are not limited to, an aggressiveness estimator, a friendliness estimator, a sexual content estimator, a mood estimator, and a text confidentiality estimator. Each word processor 56 may output a set of metrics, labels, and so on. Such metrics/labels may be defined for each individual dialog message, or may be defined for the corresponding dialog as a whole.

[0048] An exemplary aggressiveness estimator calculates a metric for each dialog message, with the metric indicating the level of aggression indicated by the language of the corresponding message. The aggressiveness index can be expressed as a binary number (1/0, YES/NO) or as a non-binary number that can take on any value between predetermined limits. Aggressiveness evaluators may use techniques such as detecting the presence of certain keywords indicative of aggression, or any other method known in the art. The preferred embodiment trains a recurrent neural network (RNN) using a vector representation of each word in the dictionary. Exemplary vector representations may be generated using a version of the word-2-vec and/or Glove family of algorithms. Each dialog message can be represented as a sequence of vectors. The aggressiveness estimator architecture may include, among other things, a long short term memory (LSTM) placed on top of a closed recurrent unit (GRU) layer. Certain rules may apply in teaching, for example, only insults articulated against a second person may be labeled as positive/aggressive. In one such example, "you're so stupid" might score higher in aggressiveness than "he's so stupid." The output of such a neural network may contain a metric/label defined for each individual message, or a metric/label defined for the entire conversation.

[0049] The architecture of the exemplary sexual content evaluator may be similar to the architecture described for the aggressiveness evaluator. However, the sexual content evaluator may be specifically trained to output an indicator indicating whether each dialogue and/or message contains sexual language. Sometimes sexual and aggressive language coexist in the dialogue, so this is an example of how having independent experts on each aspect of the dialogue can give a finer and perhaps more accurate classification of the relevant dialogue. Some embodiments may be further trained to identify other text patterns that may not be sexually explicit but may nonetheless indicate grooming or sexual predation. For example, some embodiments may determine if the meeting message requests a personal address, and so on. Some embodiments of the sexual content evaluator may be trained to distinguish between multiple scenarios and/or categories of sexual content (eg, grooming, sexting, etc.). In one such example, the sexual content evaluator may output a vector of scores, each score corresponding to a different category/scenario and indicating the likelihood that the analyzed dialogue falls into the corresponding category/scenario.

[0050] An exemplary friendliness scorer aims to detect phrases that demonstrate affection and friendliness towards a particular interlocutor. Since friends often tease each other with profanity, the Friendliness indicator can help distinguish genuine insults from behavior that may seem aggressive but is actually playful and friendly. An exemplary friendliness rater may use a rule-based system to determine a direct positive phrase towards the interlocutor (for example, "I like you") and / or indirect language, in which the current message is used to positively respond to the previous one (for example, "I like you"). do you like it?", "God, you're stupid. Of course! You're the best." This is another example where analysis of textual content is correlated between multiple messages of the same conversation, as opposed to analyzing each message individually.

[0051] An exemplary mood estimator may use any method known in the art to determine a numerical or categorical sentiment/mood indicator of the corresponding dialogue. An exemplary indicator might have positive values when the dialogue is considered happy/relaxed, and negative values when the dialogue indicates stress, depression, anger, etc. The value of the respective evaluation indicator may indicate the intensity of the respective mood. The exemplary sentiment estimator uses a sentiment-subtext and justification dictionary (VADER) methodology in which each message token (e.g., each word or phrase) is labeled positive or negative according to its semantic orientation, and the aggregate score/label is computed by combining the individual labels. tokens. The aggregate score can be calculated at the level of detail of individual messages or for the conversation as a whole. In some embodiments, aggressive dialogue in which only one party feels bad/frustrated is a strong indication that bullying is taking place. Thus, such a situation may receive a relatively high aggregate indicator of bullying for the corresponding dialogue.

[0052] An exemplary text privacy evaluator may determine whether the dialogue conveys sensitive information that the appropriate user (eg, child, employee) should not share with others. Some examples of such information are, among others, credit card numbers, social security numbers, and home addresses. One exemplary text privacy evaluator may use character pattern matching (eg, regular expressions) to identify data such as credit card numbers and addresses. Other embodiments may train a neural network to detect text patterns that look like credit card information, social security numbers, and so on. The text sensitivity evaluator can output a vector of scores, each score indicating whether the text of the current conversation contains a particular category of sensitive data. Such indicators of text confidentiality can be defined for the dialogue as a whole.

[0053] In some embodiments, the image processor 58 (see FIGS. 5 and 10) uses various methods/algorithms to detect various characteristics of the image and/or video data exchanged within the dialogue. One exemplary image processor 58 includes a nudity estimator configured to return a metric indicative of the likelihood that an image contains nudity. In an alternative embodiment, the nudity evaluator may return a plurality of scores, each of which indicates the likelihood that an image shows a particular body part (e.g., face, chest, nipple, leg) and/or that the corresponding image is likely to be of a certain body part. type of images (sexual activity, sunbathing, etc.). In some embodiments, the nudity evaluator is further configured to return an indicator of whether each visible body part is nude or covered.

[0054] FIG. 13 shows several exemplary body parts 60a-c that an exemplary nudity evaluator is trained to detect in an image file transmitted as part of a conversation, in accordance with some embodiments of the present invention. Each body part 60a-c includes a human body part such as head, face, hair, chest, cleavage, chest, nipple, inframammary region, abdomen, navel, lower waist, perineum, genitals, anus, buttock, sacrum, lower part back, mid back, shoulder blade, neck, occiput, upper arm, lower arm, hand, thigh, upper leg, shin, knee and foot, among others. Some of these body parts may overlap. Some embodiments are further trained to determine whether a body part found in an image is male or female.

[0055] In a preferred embodiment, the nudity estimator may comprise a set of interconnected artificial neural networks, such as a stack of convolutional neural networks, further fed to a fully connected layer. The corresponding nudity estimator may take the analyzed image as input and may be configured to output a set of ratings and/or labels. Neural networks can be trained on a collection of annotated images. Training a neural network may include iteratively tuning a set of functional parameters (eg, connection weights) of the corresponding neural network in an attempt to reduce the mismatch between the network's actual output and a desired output, such as that provided by an annotation.

[0056] Another exemplary image processor 58 includes an image privacy estimator configured to return a metric indicative of the likelihood that the corresponding image contains sensitive information. Examples of sensitive image data include an image of a bank card, an image of an official identification document such as a driver's license, social security card or passport, an image of a car license plate, an image of the user's home/school, and so on. Bank cards include, but are not limited to, credit and debit cards.

[0057] In some embodiments, the image privacy estimator comprises a set of interconnected artificial neural networks (e.g., convolutional neural networks) trained to input an image and output a set of metrics and/or labels indicating the likelihood that the image falls within a specialized category (e.g., , the image shows a certain type of physical object, such as a bank card). The corresponding neural networks can be trained on an annotated collection containing images of different types of documents in different contexts, for example, bank cards issued by different banks and having different designs, passports and/or driver's licenses issued by different countries, etc.

[0058] Some embodiments detect the presence of a physical object in an image according to the characteristics of the corresponding physical object. For example, to detect the presence of a bank card, an exemplary image privacy evaluator can be trained to detect a magnetic stripe image, a handwritten signature image adjacent to a magnetic stripe, an embedded microchip image, an image of 16 digits aligned and divided into groups of four (i.e., . card number), image of the VISA ® or MASTERCARD® logo, etc. In the case of a social security card, an image privacy evaluator can be trained to determine if an analyzed image contains a social security logo and/or a set of 11 digits aligned and divided into three groups (ie, a social security number). Driver's licenses and passports can also be identified by distinctive features, such as a photograph of a human head, and the specific placement of various data in the respective document.

[0059] In some embodiments, an image privacy estimator (e.g., a neural network) can be trained to output a plurality of metrics, with each metric indicating the likelihood that the analyzed image shows a feature of a bank card, social security card, etc. For example, one metric may indicate the likelihood that the image shows the card's embedded microchip, another metric may indicate the likelihood that the image shows the VISA® logo, and so on. Such individual scores can then be aggregated by decision block 53, for example using a weighted average or a decision algorithm.

[0060] Some embodiments of an image privacy evaluator can be further trained to extract structured data from analyzed images. For example, in addition to determining that an image shows a credit card, some embodiments may determine the type of card (eg, VISA®), issuing bank, and so on. Similarly, in addition to detecting a driver's license image, some embodiments may automatically detect the driver's name, and so on.

[0061] In some embodiments, decision block 53 (FIG. 9) inputs individual score indicators 26-28 received from text and/or image processors 56-58, respectively, and outputs an aggregated risk score indicator 22 determined according to with individual risk assessment indicators. An exemplary aggregate risk indicator 22 is defined for the dialogue as a whole and contains a set of indicators, where each indicator indicates the likelihood of a particular type of threat or scenario (for example, fighting, bullying, depression, sexual disclosure, grooming, loss of confidential data, etc.). Aggregate indicators 22 or indicators can be calculated using any method known in the art. One example involves calculating a weighted average of individual performance indicators/indicators. In another example, the aggregate score is determined according to the decision algorithm: if the x score is YES and the y score is below 0.4, then the aggregate score is 0.8.

[0062] The bullying aggregate score can be determined according to the following observations. Bullying usually happens in scattered outbursts rather than evenly over dialogue. There are usually several such outbursts in offensive dialogue. In some embodiments, to qualify as bullying, the aggressive language must be consistent within a single outbreak (ie, a single offensive message does not indicate bullying). Most often, children use profanity without the intention of causing harm. Typically, these types of interactions (i.e., non-bullying) show friendly language, and the overall tone of the dialogue is more positive or neutral. Finally, in many cases, aggressive language and sexual language overlap.

[0063] Using the above observations, some embodiments scan each conversation with a fixed length window and a fixed pitch (ie, a predetermined number of consecutive messages at a time). For each such dialogue segment and each interlocutor, decision block 53 may create a vector in which each element represents a combined score determined for the individual message of the corresponding dialogue. The individual text evaluation indicators can be combined as follows:

[0064] where S _i denotes the combined score for a particular message, A _i and F _i denote the aggressiveness and friendliness score of the respective message, and X _i denotes the sexual content (eg, sexting) score of the respective message. For example, some of the following situations may occur: if the message is only aggressive, the corresponding combined score, S _i , is 1; if the message is detected to be both aggressive and sexual, the combined score S _i is 0 (sexual language overrides aggressive language); if the message is detected to be both aggressive and friendly, the combined score S _i will also be 0 (friendly language overrides aggressive language).

[0065] In the next step, you can calculate the aggressiveness concentration score for the current dialogue segment, for example, using the formula:

[0066] where N denotes the total number of messages in the corresponding segment of the dialogue, S _i is the combined score of each message of the corresponding segment, ad _i denotes the interval (for example, the number of messages) between the current message and the nearest aggressive message (for example, with the combined score S _i = 1). Formula [2] gives a relatively higher value for dialogue segments that contain closely spaced aggressive messages compared to other dialogue segments. In the next step, the C value can be compared with a predetermined threshold value (eg 0.5). A value above the threshold indicates an aggressive user for the corresponding conversation segment. The calculation can be performed separately for each interlocutor and each segment of the dialogue. Some embodiments may then determine the type of relationship between the interlocutors, for example: normal - none of the interlocutors is overly aggressive; bullying - one of the interlocutors is significantly more aggressive than the other; quarrel - both interlocutors are essentially and equally aggressive. To reach a bullying verdict, some embodiments may further determine whether the bully is a user of the monitored device 10 or not so that the notification manager 59 can include such information in the parent notification 24.

[0067] In some embodiments, the dialogue-specific risk assessment indicators/metrics 22 are compared to a set of predetermined thresholds specific to each type of threat. An indicator above the corresponding threshold may indicate the presence of the corresponding threat/scenario. When the score exceeds an appropriate threshold, some embodiments of the notification manager 59 may formulate and send the parent notification 24 to the reporting device 14.

[0068] Several exemplary dialogue fragments and associated evaluation are shown below.

[0069] Example 1: Bullying

[0070] This example is characterized by a significant imbalance of aggression: user A insults user B, and B does not respond in kind. The aggressiveness concentration indicator determined according to the formula [2] gives 2.0 for user A and 0.0 for user B. Some embodiments compare the difference in aggressiveness between two users with a threshold value (for example, 0.5), and since the difference exceeds the threshold, it is determined that user A is significantly more aggressive than B. Therefore, A is bullying B.

[0071] Example 2: Not Bullying

[0072] In this example, aggressive language coexists with friendly language. However, the indicator of friendliness exceeds the indicator of aggressiveness, and the formula [2] gives negative values for both interlocutors. Therefore, the dialogue is not classified as bullying.

[0073] Example 3: Sexting

[0074] In this example, aggressive language coexists with sexting, and hence the aggressiveness score is cancelled. The dialogue does not qualify as bullying, even though only one side of the dialogue is aggressive.

[0075] Thresholds and/or other scoring parameters (e.g., weights assigned to specific metrics) can be adapted and/or adjusted for the monitored device, user, and/or user category, e.g., according to subscription type or service agreement , which provides a configurable degree of vigilance. Some embodiments are based on the observation that what is considered "acceptable behavior" can vary widely between countries, cultures, and even individuals. For example, in some countries and cultures, women are required to cover their hair in public, so depicting a woman with her head uncovered may seem unacceptably revealing, while in other cultures it is perfectly normal. The same is true for other parts of the body, such as the ankle or shoulder. Even in Western societies, conservative families are stricter about the behavior of children and adolescents than more liberal ones. For example, for some, a short skirt may seem the norm, while for others it may be too revealing. Therefore, in some embodiments of the present invention, thresholds and/or score aggregation strategies may be adjusted according to personal choice, cultural criteria, and/or according to the geographic location of devices 10 and/or 14. In one such example, when installing and/ or configuring the software on the monitored device 10 and/or reporting device 14, the user may be shown a configuration interface and prompted to configure a set of criteria for receiving parental notifications. For example, the user may be prompted to select a general vigilance level (eg, on a sliding scale from 0 to 10) and/or select a subset of scenarios that should trigger notifications from a wider set of exemplary scenarios. In some embodiments, the software may automatically select a set of parameter values (eg, notification scripts, vigilance level, etc.) according to the country and/or culture of the respective user. Such automatic selection may include, for example, automatically detecting the geolocation of reporting device 14 and/or tracked device 10 and searching the database for location-specific settings. Such settings can then be automatically converted to specific thresholds and/or other metric aggregation options.

[0076] When determining the aggregate evaluation indicator 22, the decision block 53 may combine the text evaluation indicators with the image evaluation indicators defined for the same dialog. In one such example, the output of a sexual content evaluator (text) may be combined with the output of a nudity evaluator (image) to generate an aggregate measure of sexual content. When the text of the dialogue includes sexual content, the relevant content may simply be vulgar language used in a verbal altercation. Thus, the score assigned to the relevant dialogue by the sexual content evaluator may not be high enough to qualify the dialogue as sexually threatening. However, when the corresponding dialog also includes a pop-up image, the score given by the nudity evaluator may be combined with the score returned by the word processor to obtain an aggregated score that exceeds the corresponding threshold. The scores can be combined, eg as a weighted average, with each individual weight reflecting the relevance of the respective score to a particular threat/situation. For the example of sexual content, the score generated by the image processor may have a higher weight than the score generated by the word processor.

[0077] In preparation for sending the parental notification 24, the dispatcher 59 may identify the appropriate recipient of the corresponding notification, i.e. reporting device 14. In some embodiments, parental control services are provided in accordance with a subscription and/or service agreement (SLA). In order to provide such services, some embodiments maintain a database of subscription or account records, in which each record can be attached to the reporting device 14 so that notifications generated for the corresponding subscription/account are delivered to the corresponding reporting device. Reporting device 14 may be identified, for example, by a network address or by a unique identifier generated by a software agent running on device 14 and configured to communicate with server 18 when notifications are delivered. The corresponding subscription/account entry may further specify a set of monitored devices 10 as targets for conversation data collection and analysis. A single subscription/account may serve multiple monitored devices 10. The subscription/account entry may additionally identify the monitored device user 10, for example, as a username, alias and/or avatar used by the monitored user (e.g., a minor) in the application 48 messaging. Communication between monitored devices 10, users, and reporting device 14 allows dispatcher 59 to selectively identify reporting device 14 according to the ID of the monitored device 10 and/or according to the user ID of the device 10.

[0078] Delivery of notification 24 may occur in accordance with any method known in the art, for example, by sending notification 24 to a software agent/application running on reporting device 14, including notification 24 in the form of an email or SMS message, etc. d.

[0079] Parent notification 24 may contain a notification formulated in natural language, such as English. Notification messages may include an indicator of a detected incident/scenario/threat, such as bullying a child, a child sent confidential information, etc. To preserve the privacy of the monitored user, some embodiments do not disclose the actual content of the message to parents/guardians/administrators. Some embodiments further include parental advice/suggestions on how to respond to the corresponding detected scenario or threat and/or a set of psychological resources (hyperlinks, literary references, etc.) related to the corresponding detected incident/threat. Notices may be worded to the greatest extent possible so as not to disturb the parent/guardian or reveal the identities of the parties involved in the relevant dialogues. When bullying is detected, the notification message may indicate whether the user of the monitored device 10 is the source or recipient of the abuse. Below are some examples of notifications.

Cyberbullying and anti-predation

[0080] Examples of notifications in response to scenarios/threats detected using text analysis:

[0081] The child engages in a one-on-one verbal altercation: "Jenny is involved in a verbal altercation in a chat called [dialog name]. When talking about online behavior, remind your child that aggression is never "just a joke." Even if it seems like everyone is doing it, it's not normal.

[0082] A child is being bullied one-on-one: "Jenny is being cyber-bullied in a chat room called [dialog name]. Listen to your child. Find out exactly what happened, how she felt, and why. Here are some tips on how to start a conversation: [link] »

[0083] The child is participating in a group chat and is the only one aggressive: "Jenny is participating in a verbal argument in a group chat called [dialog name]. Her behavior seems aggressive towards the interlocutors. When talking about online behavior, remind your child that aggression is never "just a joke." Even if it seems like everyone is doing it, it's not normal.

[0084] The child engages in a group verbal argument but is not aggressive: "Jenny engages in a verbal argument in a group chat called [conversation name]. Her behavior does not seem aggressive towards the interlocutors. When talking about online behavior, remind your child that aggression is never "just a joke." Even if it seems like everyone is doing it, it's not normal.” Child safety, sexual predation and grooming

[0085] Examples of notifications in response to scenarios/threats detected using text analysis:

[0086] The child receives a personal address: "Jenny received a personal address in a dialogue called [dialog name]. Remain calm and talk face to face with your child about the importance of privacy. Here are some tips on how to start a conversation: [link]"

[0087] Child is asked to meet in person: "Jenny received a meeting request on a chat called [dialog name]. Calmly talk to the child about the meeting request, preferably in person.”

[0088] The child agrees to a face-to-face meeting: "Jenny agreed to a meeting in dialogues called [dialogue name] at 19:00. This does not necessarily mean that something is wrong. Maybe it's just good that you know that."

[0089] The child is asked for images: “Jenny received a request for photos in a chat called [dialog name]. Remind your children that sharing everything will make them vulnerable. Both online and offline, some things must remain private.”

[0090] The child has a new contact on social networks: “Jenny started chatting online with a new contact in a chat called [dialog name]. Keep a good balance. New contact does not necessarily mean trouble, and you should not interrogate the child. However, you can check the new contact's profile and note any changes in your child's behavior."

[0091] Examples of notifications in response to scenarios/threats detected using image analysis or a combination of text and images:

[0092] The child receives a too open picture. “Jenny received a nude photo in a chat called [dialog name]. If an image or video of a sexual nature has been sent to your child, advise him/her to remove it immediately and not share it with anyone. Talk about it later."

[0093] The child sends an image that is too open: "Jenny sent an image containing nudity to a chat called [dialog name]. Remain calm and start a dialogue with your child by asking, "If you had a fight with this person, would you like them to send you this photo?"

[0094] The child has an open photo stored on her device: "Inappropriate media content detected on Jenny's portable device. If an image or video of a sexual nature has been sent to your child, advise him/her to remove it immediately and not share it with anyone. Talk about it later."

Privacy, identity theft and family safety

[0095] Examples of notifications in response to incidents/threats detected using text analysis:

[0096] The child is asked for her personal address: "Someone asked Jenny for her personal address in a chat called [dialog name]. Remind your child that your address should only be shared with your consent to certain people.”

[0097] The child sends his personal address: "Jenny sent a personal address in the conversation called [dialog name]. Remain calm and talk face to face with your child about the importance of privacy.”

[0098] A child is asked for credit card numbers: "Jenny was asked for a credit card number in a chat called [dialog name]. Calmly but firmly advise your child not to disclose such information. Discuss the consequences in more detail, face to face."

[0099] A child is asked for a social security number: "Jenny was asked for a social security number in a chat called [dialog name]. Calmly but firmly advise your child not to disclose such information. Discuss the consequences in more detail, face to face."

[0100] The child sends a credit card number in a dialog: "Jenny sent a credit card number in a dialog called [dialog name]. Remain calm and talk face to face with your child about the risks of financial disclosure.”

[0101] The child sends the social security number in the dialogue: "Jenny sent the social security number in the dialogue called [dialog name]. Remain calm and talk face to face with your child about the risks of disclosing personal information.”

[0102] Examples of notifications in response to events/threats detected using image analysis or a combination of image and text:

[0103] The child sends a photo of a credit card: "Jenny sent a picture of a credit card in a dialog called [tax name]. It appears to be a Mastercard issued by Chase. The details of this credit card are now available online and are in the possession of another person. You must secure the respective card in accordance with the banking procedure. After that, calmly but firmly talk about it with your child.

[0104] The child sends a photo of the social security card: “Jenny sent a photo of the social security card in a conversation called [dialog name]. The social security number is now owned by someone else. Various sensitive data is attached to your social security number. Here are a few steps you can take to check if the information received is being used for malicious purposes such as identity theft [link]. Calmly but firmly talk to your child about this.”

[0105] The child sends a photo of an identity document: "Jenny sent a photo of an identity document in a conversation called [dialog name]. The document is similar to an American passport. Calmly but firmly talk to your child about identity theft and/or the risks of sharing personal information online.”

[0106] A photograph is stored on the child's device, which shows a credit card. “An image of a credit card has been found on Jennie's portable device. Remain calm and talk face to face with your child about the risks of financial disclosure.”

[0107] The child has a photo stored on her device, the photo shows an ID: “The ID photo was found on Jenny's portable device. Remain calm and talk face to face with your child about the risks of disclosing personal information.”

[0108] While the above description refers to the configuration described in Figures 3 and 8, the skilled artisan will appreciate that alternative embodiments may use a different distribution of software components. For example, in some embodiments, the dialog parser 51 and decision block 53 may run on the monitored device 10 instead of the server 18, as shown in FIG. In such configurations, a typical communication between device 10 and server 18 is shown in Figure 2-B. Such configurations may have the advantage that the entire content of the message remains on the monitored device 10, thus ensuring the privacy of the respective user. The disadvantage is that the operations of the dialog analyzer 51 and/or the decision block 53 are usually computationally intensive and can place an unacceptable burden on a relatively limited device such as a mobile phone or tablet computer. Another potential disadvantage of performing text and/or image processing on a monitored device 10 is the need to distribute software updates to all such devices. In contrast, when conversation analysis is performed on the security server 18, a single machine may process conversation data received from multiple (perhaps thousands) of monitored devices.

[0109] In yet another alternative embodiment, the message aggregator 54 may run on the security server 18 instead of the monitored device 10, as shown in FIG. In such configurations, aggregator 54 may receive message indicators from multiple devices. Such configurations may allow conversations to be aggregated across multiple devices, such as when a user initiates FACEBOOK® messaging on a smartphone but continues later from a personal computer.

[0110] FIG. 14 shows an exemplary hardware configuration of a computing device 70 programmed to perform some of the methods described herein. The device 70 may be any of the monitored device 10, the security server 18, and the reporting device 14 of FIG. The configuration shown is that of a personal computer; other computing devices such as mobile phones, tablet computers, and wearable devices may have slightly different hardware. Processor(s) 72 comprise a physical device (eg, a microprocessor, a multi-core integrated circuit formed on a semiconductor substrate) configured to perform computational and/or logic operations on a set of signals and/or data. Such signals or data may be encoded and delivered to processor(s) 72 in the form of processor instructions, such as machine code. Processor(s) 72 may include a central processing unit (CPU) and/or an array of graphics processing units (GPU).

[0111] Memory unit 74 may comprise a volatile computer-readable medium (eg, DRAM) that stores data/signal/instruction encodings that are accessed or generated by the processor or processors 72 during operations. Input devices 76 may include computer keyboards, mice, and microphones, including, among other things, appropriate hardware interfaces and/or adapters that allow a user to enter data and/or instructions into computing device 70. Output devices 78 may include display devices. such as monitors and speakers, among others, as well as hardware interfaces/adapters such as graphics cards that allow the computing device 70 to communicate data to a user. In some embodiments, input and output devices 76-78 share common hardware (eg, a touch screen). Storage devices 82 include computer-readable media for non-volatile storage, reading, and writing of program instructions and/or data. Exemplary storage devices include magnetic and optical disks and flash memory devices, as well as removable media such as CDs and/or DVDs and drives. The network adapter or network adapters 84 allow computing device 70 to connect to an electronic communications network (eg, network 15 in FIG. 1) and/or to other devices/computer systems.

[0112] Controller hub 80 generally represents a plurality of system, peripheral, and/or chipset buses and/or all other circuitry that provides communication between the processor or processors 22 and the rest of the hardware components of device 70. For example, controller hub 80 may include memory controller, input / output (I / O) controller and interrupt controller. Depending on the equipment manufacturer, some of these controllers may be included on a single integrated circuit and/or may be integrated with the processor or processors 72. In another example, the hub controller 80 may include a northbridge-to-memory connection processor 72 and/or a connection processor 72 southbridge to devices 76, 78, 82 and 84.

[0113] It will also be apparent to those of ordinary skill in the art that aspects of the invention as described above may be implemented in various forms of software, firmware, and hardware, or combinations thereof. For example, some parts of the invention may be described as dedicated hardware logic that performs one or more functions. This dedicated logic may include an application specific integrated circuit (ASIC) or field programmable gate array (FPGA). The actual program code or specialized control hardware used to implement aspects consistent with the principles of the invention does not limit the present invention. Thus, the operation and behavior of aspects of the invention have been described without reference to specific program code - it being understood that a person skilled in the art would be able to develop software and hardware to implement the aspects based on the present description.

[0114] The exemplary systems and methods described herein allow the protection of vulnerable Internet users (e.g., minors) from online threats such as cyberbullying, online abuse, grooming, sexual harassment or exploitation, and theft of confidential information, among other things. Such systems and methods are usually categorized as parental controls. However, some of the systems and methods described here may go beyond classic parental control applications, such as detecting online harassment such as racist, sexist, or homophobic attacks against adults using online messaging services.

[0115] In some embodiments, the electronic messages exchanged by the monitored user are automatically and secretly collected from the electronic device (eg, smartphone, tablet computer) that the respective user uses to communicate. The messages are then selectively combined into conversations, consisting of messages exchanged by the same interlocutors. The dialogue data is then analyzed according to various criteria. When the analysis concludes that the monitored user is exposed to an online threat, some embodiments send a notification message to the respective user's caretaker (eg, parent, teacher, manager, etc.).

[0116] Some embodiments are based on the observation that the social dynamics involved in dangerous scenarios such as bullying are relatively complex. Therefore, in determining whether a user is exposed to such an online threat, some embodiments aggregate information from multiple messages and/or multiple aspects of a conversation. For example, some embodiments combine the parsing result of the dialog text with the parsing result of an image transmitted as part of the corresponding dialog. In turn, the analysis of the text part can also be multifaceted: in some variants, assessments of aggressiveness, friendliness and sexual content of the dialogue are combined.

[0117] Other exemplary embodiments combine image analysis with text analysis to determine if a tracked user is engaging in risky behavior of disclosing sensitive information such as credit card details and social security numbers, among other things. In one such example, images collected from email messages are analyzed to determine if they contain a photo of a bank card, social security card, driver's license, and so on. The discovery of such an image may trigger a notification to parents.

[0118] Those skilled in the art will appreciate that the above embodiments may be varied in various ways without departing from the scope of the present invention. Accordingly, the scope of protection of the present invention is defined by the following claims and their legal equivalents.

Claims (21)

1. A method of protecting Internet users from network threats, including the use of at least one hardware processor of a computer system for: analyzing an image transmitted as part of an electronic dialogue between the minor and the other party to determine if the image shows a physical item and identifying the type of that physical item, the image analysis comprising using a neural network to determine a plurality of indicators, the neural network being trained to identify distinctive visual features a physical item, each indicator of the plurality of indicators indicating whether the image shows a distinctive visual feature of the physical item, and the types of physical items identified by the neural network include a bank card, a social security card, and an identity card; And in response, when the image shows a physical item, sending a parent notification to a parent reporting device identified from the plurality of devices according to the minor, the parent notification indicating disclosure of confidential information to the minor and indicating the type of the physical item. 2. The method of claim 1, wherein determining whether the image shows a physical item includes determining the probability that the image shows a credit card magnetic stripe. 3. The method of claim 1, wherein determining whether the image shows a physical item includes determining the probability that the image shows an embedded microchip of a bank card. 4. The method of claim 1, wherein determining whether the image shows a physical item includes determining the probability that the image shows a handwritten signature. 5. The method of claim 1, wherein determining whether the image shows a physical item includes determining the probability that the image shows the logo of the credit card company. 6. The method of claim 1, wherein determining whether the image shows an item includes determining the probability that the image shows a bank card number. 7. The method of claim 1, wherein the parent notice further includes a description of the risk associated with the disclosure of confidential information and recommendations for mitigating the risk, the recommendations being determined according to the type of physical item. 8. A system for protecting Internet users from network threats, comprising at least one hardware processor configured to perform the functions of an image processor and a parental notification manager, wherein: the image processor is configured to analyze an image transmitted as part of an electronic dialogue between the minor and the other party to determine if the image shows a physical item and identify the type of that physical item, the image analysis comprising using a neural network to determine a plurality of indicators, the neural network trained to identify distinctive visual features of a physical item, each indicator of the plurality of indicators indicating whether the image shows a distinctive visual feature of the physical item, and the types of physical items identified by the neural network include a bank card, a social security card, and an identity card; And the parent notification manager is configured, in response to the determination by the image processor that the image shows a physical item, to send a parent notification to the parent reporting device identified from the plurality of devices according to the minor, the parent notification indicating disclosure of confidential information to the minor and indicating the type of physical subject. 9. The system of claim 8, wherein determining whether the image shows a physical item includes determining the probability that the image shows a credit card magnetic stripe. 10. The system of claim 8, wherein determining whether the image shows a physical item includes determining the probability that the image shows an embedded microchip of a bank card. 11. The system of claim 8, wherein determining whether the image shows a physical item includes determining the probability that the image shows a handwritten signature. 12. The system of claim 8, wherein determining whether the image shows a physical item includes determining the probability that the image shows the logo of the credit card company. 13. The system of claim 8, wherein determining whether the image shows a physical item includes determining the probability that the image shows a bank card number. 14. The system of claim 8, wherein the parent notice further includes a description of the risk associated with the disclosure of confidential information and recommendations for risk mitigation, the recommendations being determined according to the type of physical item. 15. A non-volatile computer-readable medium that stores instructions that, when executed by at least one hardware processor of a computer system, cause the computer system to perform the functions of an image processor and a parental control notification manager, wherein: the image processor is configured to analyze an image transmitted as part of an electronic dialogue between the minor and the other party to determine if the image shows a physical item and identify the type of that physical item, the image analysis comprising using a neural network to determine a plurality of indicators, the neural network trained to identify distinctive visual features of a physical item, each indicator of the plurality of indicators indicating whether the image shows a distinctive visual feature of a physical item, and the types of physical items identified by the neural network include a bank card, a social security card, and an identity card; And the parent notification manager is configured, in response to the determination by the image processor that the image shows a physical item, to send a parent notification to the parent reporting device identified from the plurality of devices according to the minor, the parent notification indicating disclosure of confidential information to the minor and indicating the type of physical subject.

Images