KR20180008717A - Automatic extraction of commit and request from communication and content - Google Patents

Abstract

A system for analyzing the content of an electronic communication may automatically extract a request or commitment from the electronic communication. In an exemplary process, the processing component analyzes the content to determine one or more meanings of the content; Query the content of one or more data sources associated with electronic communication; At least in part, automatically identifies and extracts a request or commitment from the content based on (i) one or more meanings of the content and (ii) the content of the one or more data sources. After initial recognition and extraction, a request or commitment can be identified and improved, and a request or commitment can be tracked to help one or more of the sender, sender, or other person, including creation of additional messages, reminders, appointments or to-do lists Followed by a plurality of actions including an action to resolve.

Description

Automatic extraction of commit and request from communication and content

Electronic communication has become an important form of social and business interaction. Such electronic communications include, by way of example, email, calendars, SMS text messages, voice mail, images, video and other digital communications and content. Electronic communication is automatically or manually generated by a user in any one of several computing devices.

This disclosure describes techniques and architectures for identifying requests and commitments in electronic communications such as messages between users. For example, the exchange of e-mails between two people may include text of a first person sending a request to perform a task to a second person and a second person responding with a message indicating a commitment to perform the task . E-mail exchanges are used to automatically determine the existence of a request to cause the system to perform an action and / or a commitment to perform an action by the recipient, and also to perform the person and task that originated the request, or to contribute to the task completion. And may convey enough information to determine the identity of the person or persons responding to the commitment. If the email exchange does not convey enough information to determine the existence of the request and / or commitment, the system may query other sources of information that may be associated with one or more portions of the email exchange. For example, the system may examine longer message history such as those contained in the retained "thread " of the email, or query the calendar or database of one or both of the email exchange writers for additional information. The system may also seek confirmation from one or more users involved in the communication about the existence of a potential request or commitment to perform a task if there is some level of uncertainty about either side.

The present invention is provided in order to provide a simplified description of selected embodiments of the present invention. The present invention is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter. For example, the term " technique "may be used interchangeably with system (s), method (s), computer readable instructions, module (s), algorithms, hardware logic (e.g., Field- (ASIC), an Application-Specific Standard Product (ASSP), a System-on-a-Chip system (SOC), a Complex Programmable Logic Device (CPLD)), and / Other techniques may be referred to.

The detailed description will be made with reference to the accompanying drawings. In the drawings, the left-most digit (s) of a reference number refers to the figure in which the reference number first appears. The same reference numbers in different drawings indicate similar or identical items.
Figure 1 is a block diagram illustrating an exemplary environment in which the techniques described herein may be implemented.
2 is a block diagram illustrating electronic communication applied to an exemplary task extraction process.
3 is a block diagram illustrating an electronic communication including an exemplary text thread of a request and commitment and a task extraction process.
4 is a table showing exemplary relationships between messages, commitments, and requests.
5 is a block diagram of multiple information sources capable of communicating with an exemplary extraction module.
Figure 6 is a block diagram of an exemplary extraction module operating on non-text communication.
7 is a block diagram of an exemplary machine learning system.
8 is a block diagram of an exemplary machine learning model.
Figure 9 is a block diagram illustrating an example of online and offline processes for commit and request extraction.
Figure 10 is a flow diagram of an exemplary job extraction process.

Various examples include, for example, a technique for a system to perform an extraction of a task from an electronic communication, such as a message between one or more users (e.g., a single user may send a message to itself or one or more other users) Describe the architecture. For example, an e-mail exchange between two people could include the text of a first person sending a request to perform a task to a second person and a second person making a commitment to perform the task (eg, agreeing) have. The e-mail exchange can convey information sufficient for the system to automatically determine the presence of a task execution request and / or task execution commitment. In some implementations, the email exchange does not convey enough information to determine the existence of the request and / or the commit. In this case, the system can query other sources of information that may be associated with one or more parts of the email exchange. For example, the system may check for other messages exchanged by one or both of the authors of the email exchange or by another person. The system may also check for larger emails and other messages. In addition to other messages, the system can query the calendar or database of one or both authors of the email exchange for additional information. In some implementations, the system may query traffic or weather conditions at each location of one or both of the authors.

As used herein, "extract" is used to describe the determination of a request or commitment in a communication. For example, the system may extract a request or commitment from a series of text messages. Here, the system determines or identifies a request or commitment from a series of text messages, but does not necessarily remove the request or commitment from the series of text messages. In other words, "extraction" in the context of the present application does not mean "removal"

In the present application, the process of extracting a request and / or commitment from a communication may be described as a process of extracting "task content ". That is, the term "work content" as used herein refers to one or more requests, one or more commitments and / or projects, including combinations of requests and commitments conveyed in the sense of a communication. In various implementations, the interaction between commit and request can be identified and extracted. This interaction may be, for example, when the commit to the requestor creates one or more requests to the requestor and / or a third party (e.g., an individual, a group, a processing component, etc.). For example, a commitment to request completion of a production yield analysis from an engineering manager may generate a secondary request to the manufacturing team for production data.

In various implementations, the process may extract a piece of text that includes the commit or request. For example, a paragraph may contain a commitment or a request in the second sentence of the paragraph. The process may also extract text fragments, sentences, or paragraphs that contain commit or request, such as the third sentence of the paragraph or various word phrases.

In various implementations, the process may augment the extracted work content (e.g., request or commitment) to one or more locations associated with the identity of the person and the extracted work content. For example, the extracted request may be stored or processed with additional information such as the identity of the requester and / or "subject (s) ", related location (s), time /

Once identified and extracted by the computing system, the operational content of the communication (e.g., the proposal or confirmation of the commitment or request) can be further processed or analyzed to identify or infer the semantics of the commitment or request, This is the primary owner of the request or commitment (e.g., not the communication party); The nature of the work content and its attributes (e.g., description or summary); Relevant dates specified or inferred (e.g., deadlines for commit or request completion); Related responses such as initial responses or follow-up messages, and their expected timing (e.g., at each time an greeting is expected or during an effective communication or per organization to complete a task between people); And identifying information resources used to satisfy the request. Such information resources may provide information, for example, time, person, location, and so on. The inference for the identified task content and task content can be automated (e.g., reminder), reminder, modification (e.g., display) of to-do lists, appointments, meeting requests, Computer-generated) services. In some instances, such automated services may be applied at other times, such as during message creation (e.g., typing e-mail or text), reading a message, or offline processing of e-mail on a server or client device. The initial extraction and inference of a request or commitment may also be made in cooperation with one or more participants to provide a current understanding of the request or commitment based on the identification of at least partially missing information or uncertainty in one or more attributes detected or inferred from the communication Or invoke an automated service that verifies or improves the state of the inference and request or commitment.

In some instances, the work content may include digital content that captures interpersonal communications (e.g., email, SMS text, instant messaging, phone calls, posting on social media, etc.), and content created (e.g., e-mail, A note-taking and organization tool such as OneNote®, a word processing document, etc.).

As described below, some exemplary techniques for identifying and extracting work content from various types of electronic communications may include language analysis of the content of an electronic communication, wherein a human annotation is committed or requested And can be annotated as including. Human annotations can be used in the process of generating corpus of training data used to build and test various attributes for commit or request, and automatic extraction of commit or request. The technique may also include a proxy for a human-generated label (e.g., based on e-mail participation data or a relatively sophisticated extraction method). For real-time use of methods for developing methods used in extraction systems or for identifying and / or reasoning work, commitments and characteristics, the analysis may include natural language processing (NLP) analysis at different points along a sophisticated spectrum. For example, analysis with a relatively low level of sophistication may involve identifying keywords based on simple word separation and stemming. Analysis with a relatively moderate degree of sophistication may involve consideration of analysis of a larger set of words (a "bag of words"). Analysis with a relatively high degree of sophistication may involve elaborate parsing of sentences into a parse tree and a logical form in communication. Techniques for identifying and extracting work content may include identifying attributes or "features" of a message's components and messages. These techniques can use these traits in the training and testing paradigm to build a statistical model for classifying message components. For example, these components may include sentences or entire messages that include requests and / or commitments, and may also identify and / or summarize text that best describes the request and / or commitment.

In some instances, the extraction technique may include an analytic hierarchy that includes using a sentence centered approach, consideration of multiple sentences in the message, and global analysis of relatively long communication threads. In some implementations, such relatively long communication threads may include a set of messages for a period of time, and a set of threads and long-term communications (e.g., for a few days, weeks, months, or years). A plurality of content sources associated with a particular communication may be considered. Such a source may include the history and / or relationship between people / people associated with a particular communication, the location of people over a period of time, people's calendar information, and details of the organizational structure associated with various aspects of the organization and people .

In some instances, the technique may directly consider a request or commitment identified from a component of the content as a representation of a request or commitment, or may be further summarized. The technique may extract, in a sentence or a longer message, other information including the relevant date (e.g., request or commitment deadline), location, urgency, time requirement, task topic (e.g., project) and people. The characteristics of the extracted work content in some implementations are determined by entrusting commitments and / or requests to specific creators of the message. This may be particularly useful in the case of multi-party email, for example, with multiple recipients.

In addition to the message text, the technique may include, for example, image and other graphical content, structure of the message, title header, message length, location of the sentence or phrase in the message, date / time the message was sent, And other information about the abstract and summarization. The scheme may consider the characteristics of the message itself (e.g., number of recipients, number of responses, total length, etc.) and context (e.g., day of the week). In some implementations, one technique may further refine or prioritize an initial analysis of the candidate message / content or result extraction based at least in part on the history of the originator or recipient (s) and communication and / or organizational structure.

In some examples, the technique may include analyzing features of various communications besides current communications (e.g., e-mail, text, etc.). For example, a technique may be used to determine whether an initial portion of a community thread includes commit or request, interaction between commit and request, such as the number of commitments and / or requests previously made between two (or more) Can be considered.

In some examples, the technique may include analyzing various communication features, including conditional task content commitments or requests. For example, a conditional commitment can be "If I see him, I'll let him know". The conditional request could be "If the weather is clear tomorrow, please paint the house."

In some instances, the technique may include the use of work content (e.g., commitments and / or requests) extracted with additional information such as deadlines, identifiers of people associated with the work content (e.g., name, ID number, ≪ / RTI >

In some instances, the computing system constructs a prediction model for identifying and extracting requests and commitments and related information using machine learning procedures that manipulate training sets of annotated corpus of sentences or messages (e.g., machine learning features) can do. In another example, a computing system may use a relatively simple rule-based approach to perform extraction and summarization.

In some instances, the computing system may explicitly display the task content extracted from the message in the message itself. In various implementations, the computing system may be publicly available through products and services offered by Microsoft's Windows®, Cortana®, Outlook®, Outlook Web App® (OWA), Xbox®, Skype®, Lync® and Band® You may flag messages that include requests and commitments from multiple electronic services and experiences that may include products or services, and those other services and experience provided by other companies. In various implementations, the computing system may extract requests and commitments from an audio feed, such as, for example, a telephone call or voice mail message, an SMS image, an instant messaging stream, and an oral request for a digital personal assistant.

In some instances, the computing system may learn to enhance prediction models and summaries used to extract working content by implicit and explicit feedback by a user. For example, such feedback may include user input as to whether the extracted content is correct or incorrect (e.g., in response to the displayed extracted work content). Such feedback may be quantified and / or stored by a computer system and then applied to, for example, a prediction model.

Various examples are further described with reference to Figs.

The environment described below constitutes one example, but is not intended to limit the claim to any one particular operating environment. Other environments may be used without departing from the spirit and scope of the claimed subject matter.

FIG. 1 illustrates an exemplary environment 100 in which an exemplary process, including the task extraction, as described herein may operate. In some instances, various devices and / or components of environment 100 include various computing devices 102. By way of example, and not limitation, computing device 102 may include devices 102a-102e. Although depicted as a variety of device types, the computing device 102 may be of other device types and is not limited to the device types shown. Computing device 102 may be any type of processor having one or more processors 104 operatively connected to input / output interface 106 and computer readable medium 108 via, for example, Lt; / RTI > device. The computing device 102 may be any type of computing device that may be used to perform the functions of the present invention such as a desktop computer 102a, a laptop computer 102b, a tablet computer 102c, a remote communication device 102d, a PDA 102e, , Personal health tracking accessories, etc.), automotive computers, game machines, and the like. Computing device 102 may also include, for example, a server computer, a thin client, a terminal, and / or a workstation. In some instances, the computing device 102 may include components for integration into a computing device, appliance, or other type of device.

In some instances, some or all of the functionality described as being performed by the computing device 102 may be implemented by one or more remote peer computing devices, remote servers or servers, or distributed computing resources, e. G., Via cloud computing . In some instances, the computing device 102 may include an input port for receiving electronic communications. The computing device 102 may further include one or more processors 104 for accessing various information sources associated with or associated with a particular electronic communication. Such sources may include electronic calendars and databases of history or personal information about the creators of the messages included in the electronic communication, to name a few examples. In some implementations, the author must "opt-in" or take other positive steps before any of the processors 104 can access the creator's personal information. In some instances, one or more processors 104 may be configured to extract work content from electronic communications. One or more of the processors 104 may be a hardware processor or a software processor. As used herein, a processing unit represents a hardware processor.

In some instances, in connection with device 102d, the computer-readable medium 108 is loaded by an operating system (OS) 112, a machine learning module 114, an extraction module 116, And store executable instructions by the processor 104, including executable programs or applications 118. [ The one or more processors 104 may include one or more central processing units (CPUs), graphics processing units (GPUs), video buffer processing units, and the like. In some implementations, the machine learning module 114 includes executable code stored in a computer-readable medium 108 and can be communicated locally or remotely by the computing device 102 via the input / (Not shown). The information may be associated with one or more applications 118. The machine learning module 114 may optionally include any of a number of machine learning decision models stored in the computer-readable medium 108 (or more specifically, stored in the machine learning module 114) for application to the input data Can be applied.

In some implementations, the extraction module 116 may include executable code stored in the computer-readable medium 108 and may store information locally or remotely by the computing device 102 via the input / (Not shown). The information may be associated with one or more applications 118. Extraction module 116 may selectively apply any of a number of statistical or prediction models stored in computer readable medium 108 (e.g., via machine learning module 114) for application to input data have.

Although a particular module is described as performing various operations, the module is merely an example, and the same or similar functions may be performed by more or fewer modules. Also, the functions performed by the depicted module need not necessarily be performed locally by a single device. Rather, some operations may be performed by a remote device (e.g., peer, server, cloud, etc.).

Alternatively, or in addition, some or all of the functions described herein may be performed, at least in part, by one or more hardware logic components. Exemplary types of hardware logic components that may be used, for example and without limitation, include field-programmable gate arrays (FPGAs), program-specific integrated circuits (ASICs), program-specific standard products (ASSPs) a-chip system, a complex programmable logic device (CPLD), and the like.

In some instances, the computing device 102 may be associated with a microphone capable of capturing images and / or video and / or a camera capable of capturing audio. For example, input / output module 106 may include such a camera and / or microphone. For example, an image of an object or text may be transformed into text corresponding to the content and / or semantics of the image and analyzed for the working content. Speech audio can be converted to text and analyzed for work content.

Computer-readable media include computer storage media and / or communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, a phase change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), EEPROM, Memory technology, CD-ROM, DVD or any other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transferable device that may be used to store information for access by a computing device But are not limited to, media.

On the other hand, communication media embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism. As defined herein, computer storage media do not include a communication medium. In various examples, memory 108 is an example of a computer storage medium storing computer executable instructions. When executed by the processor (s) 104, the computer-executable instructions cause the processor (s) to, among other things, analyze the content of an individual electronic message, wherein the electronic message is (i) received (ii) entered by a user via a user interface, or (iii) retrieved from a memory, - extracting text corresponding to a request or commitment from an electronic message based at least in part on the analysis of the content .

In various examples, an input device of or in connection with the input / output (I / O) interface 106 may be a direct touch input device (e.g., a touch screen), an indirect touch device (e.g., a touch pad), an indirect input device Mouse, keyboard, camera or camera array, etc.), or other type of non-contact device, such as an audio input device.

The computing device (s) 102 may also include one or more input / output (I / O) interfaces 106 that extract computing devices 102 and working content via the network 111 One or more communication interfaces that enable wired or wireless communication between other participating networked computing devices or other computing devices. Such a communication interface may include a network interface controller (NIC), such as one or more transceiver devices, for example, an Ethernet NIC or other type of transceiver device, for transmitting and receiving communications over a network. The processor 104 (e. G., A processing unit) may exchange data over each communication interface. In some instances, the communication interface may be a PCIe transceiver and the network 111 may be a PCIe bus. In some instances, the communication interface may include, but is not limited to, a cellular (3G, 4G, etc.), WI-FI, Ultra-wideband (UWB), Bluetooth or transceiver for satellite transmission. The communication interface may include a wired I / O interface such as an Ethernet interface, a serial interface, a universal serial bus (USB) interface, an INFINIBAND interface, or other wired interface. To simplify the description, these and other components are omitted from the illustrated computing device 102. The input / output (I / O) interface 106 is configured to allow the device 102 to communicate with a user input peripheral device (e.g., a keyboard, mouse, pen, game controller, voice input device, touch input device, gesture input device, Output peripheral device (e.g., display, printer, audio speaker, haptic output, etc.).

FIG. 2 is a block diagram illustrating an electronic communication 202 via an exemplary task extraction processor 204. FIG. For example, process 204 may include any of a number of techniques for detecting whether commit 206 or request 208 has been made in an incoming or outgoing communication. The process 204 may also include techniques for automatically marking, annotating, or otherwise identifying a message as including a commitment or request. In some instances, the process 204 may include techniques for extracting a summary (not shown) of a commitment or request for presentation and subsequent tracking and analysis. Commitment 206 or request 208 may be extracted from the multiple types of content of electronic communication 202. Such content may include interpersonal communications such as e-mail, SMS text or images, instant messaging, posts in social media, meeting notes, and the like. Such content may also include, among other things, content created using an email application or a word processing application.

In some instances, the job extraction process 204 may extract work content associated with a third party. For example, electronic communication 202, such as e-mail, may include a commitment by a first person to respond to e-mail. This commitment may be the commitment of the first person. However, this commitment may be a commitment by a third person, which may be a commitment by a first person (in response to an e-mail) on behalf of another person. For example, a first person may be a supervisor who establishes a commitment to a vice president for a subordinate performing a task. As a specific example, the commitment of the third person may be " My assistant john will report to you later today ".

In some instances, the task extraction process 204 may be based, at least in part, on personal and / or professional relationships between the creators of a message (e.g., an email thread) or people associated with the message content And may extract the work content from the electronic communication 202, such as a message. The task extraction process 204 may also extract the task content from the message based at least in part on previous communications between the creators of the message or people associated with the message content.

In some instances, the job extraction process 204 may (i) analyze the content of the electronic communication 202 and (ii) automatically extract the request or commitment from the content of the electronic communication in real time. For example, during the job extraction process 204, the system performing the job extraction process may prompt the user for confirmation by notifying the user of a possible time conflict or other commitment so that the user does not make unreasonable commitments And / or provide real-time support to the user. In a specific example, the system helps time management and "You can not do this and already have promised too much work this week (you probably can not do this too many things this week ) "Message to notify the user of the overload.

FIG. 3 is a block diagram illustrating an electronic communication 302 that includes an example text thread of a request or commitment and a task extraction process 304. For example, the communication 302 may be a text message from another user received on the user's computing device, including text 306 from another user and text 308 from the user. The job extraction process 304 may include analyzing the content of the communication 302 (e.g., text 306 and text 308) and (i) committing the user or other user and / or (ii) Lt; / RTI > In the example shown in Figure 3, the text 306 by other users (help a user named Alexis to help prepare for a birthday party on May 9, set up Alexis ' birthday party on May 9 ^th ) < / RTI > Text 308 by the user is the user and would help to support the Alexis May 9 3:00 pm Birthday Party (help set up Alexis ' birthday party on May 9 ^th , by 3pm ) < / RTI > The task extraction process 304 may determine the request and commitment by any of a number of techniques, including analyzing the text 306 and the text 308. In some implementations, the task extraction process 304 may query any of a plurality of data sources if the text is insufficient to determine a request or commitment (e.g., information "missing" or highly uncertain information). For example, if the text 306 did not include the date of the Alexis birthday party (e.g., another user might think that the user remembers the date), then the job extraction process 304 may determine that the user You can view the date of birth on your calendar.

In various examples, the job extraction process 304 may be used to determine whether a received or originated message (e.g., email, text, etc.) contains a probability (e.g., an inferred probability) Can be determined. Such reliability or probability may be determined, at least in part, from the calculated probability of a valid request or commitment date of a summary of one or more components or components of a message.

In some instances, the job extraction process 304 may determine the reliability of a commitment, where a low confidence commitment is not likely the user will fulfill the commitment, and a trustworthy commitment is one in which the user is likely to fulfill the commitment will be. The probability (e.g., likelihood) or other reliability can be used to capture how a job extraction process 304 is related to the extracted commitment based at least in part on, for example, the use of the statistical classifier. Commitment reliability can be useful for follow-up services such as reminders, to-do list corrections, appointments, meeting requests, and other time management activities. Determination of the trustworthiness of the commitment may be based, at least in part, on the history of the user's events (e.g., execution of past commitments) and / or the history of other user's events and / or the personal information of the user or other user (e.g. age, People who travel frequently, etc.). For example, the job extraction process 304 may query this history. In some implementations, one or both of the users must "opt-in" or take other positive action before the job extraction process 304 queries the user's personal information. For example, if this history proves that the user has attended several previous birthdays of Alexis, tends to attend many other people's birthdays, and is relatively close to Alexis and / or other users, Lt; RTI ID = 0.0 > 304 < / RTI > can assign a relatively high trust to the commit by that user. Determining the confidence of the commitment may also be based, at least in part, on the keywords or terms of the text 306 and / or the text 308. For example, a "birthday party" generally has a positive and favorable meaning (e.g., a party in contrast to a work task), and therefore the commitment may be relatively strong. On the other hand, in another example involving a commitment to prepare an accounting report, such activities are generally unacceptable, and therefore such credits may be given relatively low credibility. However, if such a commitment to make an accounting report relates to a user's job (eg, job), then such commitment may be given a relatively high degree of confidence. The job extraction process 304 may weight a number of such scenarios and factors to determine the reliability of the commitment. For example, the job extraction process 304 may determine the reliability (e.g., importance) of a request or commitment of a message based at least in part on the content associated with the electronic message.

4 is a table 400 illustrating exemplary relationships between messages and working content. Specifically, such work content may be generated by a user of a computing device or by a "different user entity ", which may be one or more people in one or more computing devices (e.g., Quot;) < / RTI > commit and / or request. In some instances, other user entities may be users who can send messages to the user. In another example, a user and / or other user entity may be any person (e.g., agent, secretary, administrator, etc.) or machine (e.g., a processor-based system configured to receive and execute commands). Table 400 shows origination messages generated by a user of a computing device and transmitted to another user entity and received messages generated by other user entities and received by a user of the computing device.

An example of a commitment that can be extracted from an outgoing message or an incoming message is "I will prepare the document and send it to you on Monday.", I'll send you a check. ","I'll do it. ","I'll be back to you. back to you. "and" Will do. " The latter example indicates that the commitment (or its statement) does not need to include a time or a deadline. An example of a request that can be extracted from an incoming or outgoing message is "Can you leave the key under the mat?", "Let me know if you can do it early I do not know if you can make it earlier for dinner. "," Can you get the budget analysis done by end of month? " The request does not have to be a form of direct question. For example, " Do not forget to get a report by 5:00 pm ( Do not forget to get your report in by 5pm ) "is not a direct question, but this statement is in the form of a request.

Table 400 includes four specific case tasks included in the message. One case is an outgoing message that contains a commitment the user sent to another user entity. Another case is an outgoing message that contains a request from a user to another user entity. Another case is an incoming message that contains a commitment sent to the user from another user entity. Another case is an incoming message containing a request sent to the user from another user entity. The process of extracting the action content from the message may differ from each other at least in part depending on which of the particular cases is handled. Such a process may be performed by a computing system (e.g., a server) in communication with a computing device or computing device of a user. For example, a process applied when an incoming message includes a commitment sent to a user from another user entity may be referred to various data sources to determine the trustworthiness (e.g., authenticity, reliability, value) of the other user entity Lt; / RTI > These various data sources may include personal data or the history of other user entities. In some examples, the data source may be a memory associated with a processing component of a device, such as a memory device electrically coupled to the processor via a bus. In some instances, the history of operations by other user entities (cancellation of a conference or failure to perform an operation) may indicate confidence in the commitment of other user entities. In some implementations, the user and / or other user entity must "opt-in" or take other positive action before the process can access the personal information of the user and / or other user entity.

As another example, the process applied in the case where the originating message includes a request for another user entity by the user is that the other user entity has a strong (e.g., true, reliable, valuable) commitment to the user's request And querying the various data sources to determine the probability of responding. These various data sources (which need not be outside the device (s) performing the process) may include personal data or history of other user entities. For example, an activity history (cancellation of a conference or failure to perform an operation) by another user entity may indicate the possibility (or lack thereof) that another user entity may accept or perform a commitment to the user's request.

On the other hand, the process applied in the case where the incoming message includes a request sent to the user from another user entity includes querying the various data sources to determine the importance of the request (at the same time, the importance of the commitment to the request) can do. For example, if the other user entity is the user's manager, this request may be relatively important. Thus, the process can query various data sources including personal and / or expert data of other user entities to determine whether other user entities are supervisors, subordinates, colleagues, friends, family members, and so on.

In another example, a process that is applied when an originating message includes a commitment sent by a user to another user entity may include querying various data sources to determine the importance of the commitment. For example, commitments can be relatively important if the other user entity is the user's supervisor. Thus, the process can query various data sources including personal and / or expert data of other user entities to determine whether other user entities are supervisors, subordinates, colleagues, friends, family members, and so on.

5 is a block diagram of an exemplary system 500 that includes an extraction module 502 in communication with a plurality of entities 504-518. Such entities may include, for example, a host application (e.g., an Internet browser, an SMS text editor, an email application, an electronic calendar function, etc.), a database or an information source A third-party data aggregator, etc.) that can provide such information. The extraction module 502 may be, for example, similar to the extraction module 116 of the computing device 102 shown in FIG. For example, some entities 504-518, such as data collected from training data 510, calendars 512, and social media 516, may be stored in a memory device associated with extraction module 502. [ For example, the memory device may be directly connected (e. G., Wired) to extraction module 502 (which may be a processing component, for example). In another example, the memory device may be wirelessly and / or remotely connected to the extraction module 502 (e.g., by one or more remote peer computing devices, remote servers or servers, or by distributed computing resources, e.g., via cloud computing) .

The extraction module 502 may be configured to analyze the communication content and / or data or information provided by the entities 504-518 by applying any of a number of language analysis techniques. For example, the extraction module 502 may be configured to analyze the content of the communication provided by the email entity 504, the SMS text message entity 506, and so on. The extraction module 502 may also be configured to analyze data or information provided by the Internet entity 508, the machine learning entity providing the training data 510, the email entity 504, the calendar entity 512, . The extraction module 502 may analyze the content by applying language analysis to information or data collected from any of the entities 504-518.

A bi-directional arrow in FIG. 5 indicates that data or information may flow in either direction between entity 504-518 and extraction module 502. For example, data or information flowing from extraction module 502 to any entity 504-518 may be part of the query generated by the extraction module to query the entity. This query may be used by the extraction module 502 to determine one or more meanings of the content provided by any entity.

In some instances, the extraction module 502 may receive content of an email exchange (e.g., communication) among a plurality of users from an email entity 504. The extraction module may analyze the content to determine one or more meanings of the content. Content analysis can be performed through one of several techniques for determining the meaning of an element of content, such as words, phrases, sentences, metadata (e.g., size of email, date of creation, etc.) . The "meaning" of the content may be a method of interpreting the content in a natural language. For example, the meaning of the content may include a request by an individual to perform an action. In another example, the meaning of the content may include a description of the job, a time at which the job should be completed, background information about the job, and the like.

In an optional implementation, the extraction module may query the content of one or more data sources, e.g., social media entity 516. [ Such content of one or more data sources may be associated with the content of the email exchange (e.g., by topic, author, date, time, location, etc.). Based on at least one of (i) one or more meanings of the content of the email exchange and (ii) the content of the one or more data sources, the extraction module 502 may automatically extract the request or commitment from the content of the email exchange.

In some instances, the extraction module 502 may extract the working content using the predictive model learned from the real-time ongoing communication between the training data 510 and / or between the extraction module and any of the entities 504-518. can do. This predictive model may infer that an originating or receiving communication (e.g., a message) or communication content includes a request. Likewise, outgoing or incoming communication or communication content may include commitments to perform tasks. The identification and extraction of commitments and requests from the receiving or originating communication may provide a number of functions to support the originator and the recipient of the communication for commit and request.

In some embodiments, the extraction module 502 may extract the working content using the statistical model and send a request to the e-mail entity 504, for example, from the e-mail received from the e-mail entity 504 or from the SMS text message received from the SMS text message entity 506, Can be identified and extracted. The statistical model may be based, at least in part, on data or information from one or a combination of the entities 504-518.

In some instances, the extraction module 502 may extract the task content while the message creator is composing a message. For example, such authoring may include typing an email or text message using any type of text editor or application. In another example, the extraction module 502 may extract the work content while the person is reading the received message. For example, when a person reads a message, the extraction module 502 may annotate a portion of the message by highlighting or highlighting the request or commitment in the text of the message. In some implementations, the extraction module may add relevant information to the message during the reading (or display) of the message. For example, this pertinent information may be deduced from additional data or information sources, such as entities 504-518. In a particular example, the computer system may display a message that includes a request by the reader to attend a class of classes. The extraction module 502 may query the Internet 508 to determine that a number of such classes are provided at various locations at various times in the location of the reader (e.g., which may be deduced from the personal data about the reader) . Therefore, the extraction module can generate a selection or proposal list and provide it to the reader. This list may be displayed near the text of the relevant portion of the text in response to a mouse-over, or may be "permanently" displayed, for example, to another portion of the display. In some implementations, the list may include a list such that the request may include a time selected by the reader (this time may replace the "suggested" time by the requester and the requester may be automatically notified of the time selected by the reader) You can include an item that the reader can select (by mouse click).

6 is a block diagram of an exemplary extraction module 602 that is capable of performing a task extraction for non-textual content 604, such as audio recording, image or video recording. The extraction module 602 may be the same as or similar to the extraction module 502 shown in FIG. For example, the extraction module 602 may communicate with any one or all of the entities 504-518.

The non-text content 604 may be translated into a corresponding text 606 describing the elements of the non-text content. For example, any of a number of image recognition techniques may be used to translate images (or stills of video recordings) into text. Similarly, one of a number of audio-text techniques can be used to convert audio recordings to text. The corresponding text 606 may be provided to the extraction module 602 and the extraction module 602 may then extract the working content from the corresponding text. Such extracted work content may include, for example, commit 608 and / or request 610. [

Specific illustrative examples of how extraction module 602 can extract work content from non-text content 604 include messages containing images of balloons and streamers. This image may be translated into text 606 by image recognition techniques that recognize images of balloons and confetti and produce text "balloons" and "streamers. &Quot; Additional text may be included in the image to describe the juxtaposition between the balloon and the flag. Extraction module 602 may query any of a number of entities (e.g., 504-518) to determine the context of the balloons and confetti for the originator of the message. In one example, the extraction module 602 may determine that the message is an advertisement for the party item (e.g., by searching for a match between the sender and the Internet site). As a result, the extraction module can conclude that the message does not contain commit or request. In another example, the extraction module 602 may retrieve (e. G., An individual about a sender and a message recipient) an extractor module 602 (e.g., if the birthday of the sender or any family member is approaching or if the recipient has attended the birthday for the past several years) It may determine that the message is a notification to the birthday party). In this case, the extraction module 602 may consider this image to be a request by the recipient to attend the birthday party. The extracting module can also deduce the party date, thus creating a complete request that includes the time to perform the task and the task.

In some instances, the job extraction process performed by the extraction module 602 (or 502) may be associated with a message originator and / or recipient to confirm the accuracy of the commitment or request extracted by the extraction module. In particular, if the extraction module 602 performs reasoning with relatively low confidence (e.g., reasoning based on opaque or loosely interrelated information), the extraction module may provide the sender and / or recipient with additional information about the in- Or may require confirmation. On the other hand, if the extraction module 602 performs reasoning with a relatively high degree of confidence (e.g., reasoning based on faithful or closely interrelated information), then the extraction module may add to the sender and / No information or confirmation is required.

In some examples, extraction module 602 may be configured to perform conversion of non-textual content into corresponding text. In another example, the extraction module 602 may be configured to simply receive corresponding text that has already been translated from the non-text content.

7 is a block diagram of a machine learning system 700 in accordance with various examples. The machine learning system 700 includes a machine learning model 702, a training module 704, and an extraction module 502 (which may be similar or identical to the machine learning module 114 shown in FIG. 1) And an extraction module 706 that is the same as or similar to the extraction module 706. Although shown as a separate block, in some embodiments the extraction module 706 may include a machine learning model 702. [ The machine learning model 702 may receive training data from the offline training module 704. For example, the training data may include data from a memory of a computing system, including the machine learning system 700, or data from any combination of the entities 502-518 shown in FIG. The memory may store a history of requests and commitments received and / or transmitted to a computing system or to a particular user by a computing system or a particular user. Data from the memory or entity may be used to train the machine learning model 702. Following this training, the machine learning model 702 may be utilized by the extraction module 706. Thus, training using data from, for example, requests for offline training and / or the history of commitments can serve as an initial condition for the machine learning model. Other training techniques may be used, including the featurization described below.

8 is a block diagram of a machine learning model 800 in accordance with various examples. The machine learning model 800 may be the same as or similar to the machine learning model 702 shown in FIG. The machine learning model 800 includes any of a number of functional blocks, such as a random forest block 802, a support vector machine block 804, and a graphics model block 806. The random forest block 802 may include an ensemble learning method for classification that operates by constructing a decision tree at training time. The random forest block 802 may, for example, output a class that is a mode of a class output by an individual tree. The random forest block 802 may function as a framework that includes several interchangeable parts that can be mixed and matched to create multiple specific models. In this framework, constructing a machine learning model determines the direction of decisions used by each node, determines the type of predictor to use in each leaf, determines the partitioning goal to optimize at each node, ), And so on.

The support vector machine block 804 classifies the data for the machine learning model 800. The support vector machine block 804 can serve as a supervised learning model with associated learning algorithms that analyze data and recognize patterns used in classification and regression analysis. For example, given a set of training data respectively labeled as belonging to one of the two categories, the support vector machine training algorithm builds a machine learning model that assigns the new training data to one category or another category.

Graphic model block 806 functions as a probability model, which is a probabilistic graphical model in which graphs represent conditional dependencies and independence between random variables. The probabilistic graphical model represents a joint probability distribution for a set of variables of interest. Probabilistic inference algorithms operate to perform inference based on specific evidence in these graphical models. Inference provides an update to a probability of interest, such as the probability that a message or a particular sentence will contain a commitment or request. Learning procedures can construct such probability models from data through the process of finding systems from a set of unstructured information. The learning procedure may also construct such a probability model from explicit feedback from the user (e.g., checking whether the extracted task information is correct). Applications of the graphical model that can be used to infer task content from non-textual content may include, for example, information extraction, speech recognition, image recognition, decoding of computer vision and low density parity check codes, and the like.

FIG. 9 is a block diagram illustrating an exemplary online and offline process 900 related to commit and request extraction. Such a process may be performed by a computing device, such as a processor (e. G., A processing unit) or the computing device 102 described above. "Offline" refers to a training state in which machine learning algorithms are trained using supervised / labeled training data (e.g., an email set with commit and request sentences displayed). "Online" refers to the application of a model trained to extract commitments and requests from new (invisible) emails. The characterization process 902 and the model learning process 904 may be performed by an offline or on-line computing device. On the other hand, the process 908 of receiving a new message 906 and applying the model may occur online.

In some examples, any or all of the characterizing process 902, the model learning process 904, and the process 908 applying the model may be performed by an extraction module, such as extraction module 116 or 502 . In another example, characterization process 902 and / or model learning process 904 may be performed by a machine learning module (e.g., machine learning module 114 shown in FIG. 1) (908) may be performed by an extraction module.

In some examples, characterization process 902 may receive training data 910 and data 912 from various sources, such as any of the entities 504-518 shown in FIG. The characterization process 902 may generate a feature set of a text fragment that is useful for classification. A piece of text may include a portion of the content of one or more communications (e.g., communications of a generally relatively large number of training data 910). For example, the text segment may be a word, a term, a phrase, or a combination thereof. The model learning process 904 is a machine learning process that creates and iteratively refines the model used in the process 908 to extract work content, such as requests and commitments, from the communication. For example, the model may be applied to new messages 906 (e.g., e-mail, text, etc.). The computing device may perform the model learning process 904 asynchronously, continually, occasionally, or periodically with the process 908 of applying the model to the new message 906. Thus, for example, the model learning process 904 may update or enhance the model offline, independent of the online process, such as applying the model (or the current version of the model) to the message 906.

The process 908 of applying the model to the new message 906 may include consideration of other information 914 that may be received, for example, from the entities 504-518 described above. In some implementations, at least some of the data 912 from other sources may be the same as the other information 914. The process applying the model 908 may result in extracting the work content contained in the new message 906. Such work content may include commitments and / or requests.

10 is a flow diagram of an exemplary job extraction process 1000 that may be performed by an extraction module or processor (e.g., a processing unit). For example, process 1000 may be performed by computing device 102 (e.g., extraction module 116) shown in FIG. 1, or more particularly in another example, extraction module 502 ). ≪ / RTI >

At block 1002, the extraction module may analyze the content of the electronic communication to determine one or more meanings of the content. For example, such electronic communications may include email, text messages, non-text content, social media posts, and the like. At block 1004, the extraction module may query the content of one or more data sources associated with the electronic communication. For example, one or more data sources may include any of the entities 504-518 described in the example of FIG. In another example, where the extraction module is an extraction module 116, one or more data sources may include any portion of the computer-readable medium 108 described in the example of FIG. The one or more data sources may be associated with electronic communication by, for example, a subject, a creator of an electronic communication, a person associated with the creator, a time, a date, an event history, and an organization.

At block 1006, the extraction module may automatically extract a request or commitment from the content. Such extraction may be based, at least in part, on (i) one or more meanings of the content and (ii) the content of one or more data sources.

In some implementations, the electronic communication includes audio, image or video. The conversion module may be used to convert audio, image, or video into corresponding text to generate the content of the electronic communication. The content of the electronic communication may be provided to the extraction module.

In some implementations, the extraction module may perform the process 1000 in real time.

The flow of operations depicted in FIG. 10 is illustrated as a set of blocks and / or arrows representing an operational sequence that may be implemented in hardware, software, firmware, or a combination thereof. The block description order should not be construed as a restriction, and any number of the described operations may be combined in any order to implement one or more methods or other methods. Further, the individual operations may be omitted in the operational flow without departing from the spirit and scope of the claimed subject matter described herein. In the context of software, a block represents computer readable instructions that, when executed by one or more processors, constitute the processor (s) to perform the enumerated operation. With regard to hardware, a block may represent one or more circuits (e.g., an FPGA, an application specific integrated circuit (ASIC), etc.) configured to perform the enumerated operation.

Any ordinary description, element, or block of operations depicted in FIG. 10 may represent a module, segment, or portion of code, including one or more executable instructions for implementing a particular logical function or element of a routine.

example

A system, comprising: a receiver port for receiving content of an electronic communication;

Wherein the extraction module analyzes the content of the electronic communication to determine one or more semantics of the content and queries the content of one or more data sources associated with the electronic communication, (Ii) automatically extracts a request or commitment from the content based at least in part on the content of the one or more data sources.

B. The system of claim A wherein the content of the one or more data sources comprises personal data of one or more creators of the content of the electronic communication.

A method according to claim 1, wherein said electronic communication comprises audio, image or video, said system further comprising a conversion module, said conversion module converting said audio, said image or said video into corresponding text, A system for generating content of an electronic communication and providing the content of the electronic communication to the extraction module.

D. A system according to claim 1, wherein the extraction module is configured to analyze the content of the electronic communication by applying a statistical model to the content of the electronic communication.

The system of claim A wherein the extraction module is configured to augment the extracted request or commitment to the identity and one or more locations of the person associated with the extracted request or commitment.

A system according to F. A, further comprising a machine learning module configured to use the content of the electronic communication and / or the content of the one or more data sources as training data.

G. The system of claim A wherein the extraction module is configured to (i) analyze the content of the electronic communication and (ii) automatically extract the request or commitment from the content of the electronic communication in real time.

A method comprising: receiving a message; applying a language analysis to the message to automatically convert the message to a machine language feature; searching a data source for information associated with the message; Receiving information related to the message from the data source; and automatically identifying a request or commitment from among the machine language features based at least in part on the received information.

I. In claim H, the message comprises audio, image, or video,

Wherein applying the language analysis to the message comprises: determining the audio, the image, or the text corresponding to the video; and applying the language analysis to the audio, the image, or the text corresponding to the video Further comprising:

The method according to claim 19, wherein the data source associated with the message comprises another message.

K. A method according to H, wherein the data source associated with the message comprises one or more characteristics of the message creator.

The method of claim L. H, wherein receiving the message further comprises: sequentially receiving a portion of the message for a period of time; and applying a language analysis to the received portion of the message for the period of time How to.

The method of claim 9, further comprising flagging / annotating and annotating the message as comprising the request or the commit.

N. A computing device comprising: a transceiver port for receiving and transmitting data;

A processor for analyzing an electronic message entered by a user via a user interface, retrieving data for content associated with the electronic message, and based at least in part on the content associated with the electronic message, And extracts text corresponding to the request or commitment from the electronic message.

O. The apparatus of claim 22, wherein the processor is configured to determine an importance of the request or the commitment based at least in part on the content associated with the electronic message.

[0040] [0040] The system of claim N., wherein the processor is configured to apply the electronic message or the data as training data for a machine learning process.

Q. The computing device of claim 1, wherein analyzing the electronic message is performed by a machine learning process.

R. The apparatus of claim 12, further comprising an electronic display, wherein the processor is further configured to generate an image to be displayed on the electronic display, wherein the image indicates whether the text corresponding to the request or the commitment is correct or true And prompting the user to confirm.

The system of claim 9, wherein the processor is further configured to analyze parameters of the electronic message, the parameters including at least one of a number of recipients, a length, a date and time, Device.

10. The computer-readable medium of claim 10, wherein the processor is further configured to analyze information about the user while the user is entering the electronic message.

Although the techniques are described in language specific to structural features and / or methodological acts, it should be understood that the appended claims are not necessarily limited to the features or acts described. Rather, the features and acts are described as exemplary implementations of such techniques.

Unless stated otherwise, all of the above-described methods and processes may be implemented in whole or in part by one or more general purpose computers or software code modules executed by a processor. The code module may be stored in any type of computer readable storage medium or other computer storage device. Some or all of the methods may alternatively be implemented in whole or in part by special computer hardware such as FPGAs, ASICs, and the like.

Unless otherwise stated, conditional languages such as " can ", " could ", or "may" Lt; / RTI > is used to indicate that some examples include but not other examples. Thus, unless expressly stated otherwise, such conditional language is intended to encompass within the spirit and / or scope of the following claims such that certain features, elements, and / or steps may be required by one or more examples in any way, It is not intended to indicate that the invention includes logic to determine whether such features, elements and / or steps are included or are performed in any particular instance.

Combinations such as "at least one of X, Y, or Z" should be understood to indicate that an item, term, etc. may be X or Y or Z, or a combination thereof, unless specifically stated otherwise.

Many variations and modifications may be made to the above examples, which elements should be understood as other permissible examples. All such variations and modifications are intended to be included within the scope of this disclosure.

Claims (15)

In the system,
A receiver port for receiving content of electronic communication; And
Extraction module
Wherein the extraction module comprises:
Analyze the content of the electronic communication to determine one or more meanings of the content,
Query the content of one or more data sources associated with the electronic communication,
wherein the system automatically extracts a request or commitment from the content based at least in part on (i) the one or more meanings of the content and (ii) the content of the one or more data sources. The method according to claim 1,
Wherein the content of the one or more data sources comprises personal data of one or more creators of the content of the electronic communication. The method according to claim 1,
The electronic communication includes audio, image or video,
Wherein the system further comprises a conversion module,
Converting the audio, the image or the video into a corresponding text to generate content of the electronic communication,
And provide the content of the electronic communication to the extraction module. The method according to claim 1,
Wherein the extraction module is configured to augment the extracted request or commitment to an identity of the person and one or more locations associated with the extracted request or commitment. The method according to claim 1,
A machine learning module configured to use the content of the electronic communication and / or the content of the one or more data sources as training data
The system further comprising: In the method,
Receiving a message;
Applying a language analysis to the message to automatically convert the message into machine language features;
Retrieving a data source for information associated with the message;
Receiving information associated with the message from the data source; And
Automatically identifying a request or commitment from among the machine language features based at least in part on the received information
≪ / RTI > The method according to claim 6,
The message comprising audio, image, or video,
Wherein applying the language analysis to the message comprises:
Determining a text corresponding to the audio, the image or the video, and
Applying the language analysis to the audio, the image, or text corresponding to the video. The method according to claim 6,
Wherein the receiving the message comprises:
Sequentially receiving portions of the message for a period of time, and
Applying the language analysis to the received portion of the message for the period of time. The method according to claim 6,
Flagging / annotating / annotating the message as comprising the request or the commit
≪ / RTI > A computing device comprising:
A transceiver port for receiving and transmitting data; And
Processor
The processor comprising:
Analyzing the electronic message entered by the user through the user interface,
Retrieving data about content associated with the electronic message,
And extracts text corresponding to a request or commitment from the electronic message based at least in part on content associated with the electronic message. 11. The method of claim 10,
Wherein the processor is configured to determine the importance of the request or the commitment based at least in part on the content associated with the electronic message. 11. The method of claim 10,
Wherein the processor is configured to apply the electronic message or the data as training data for a machine learning process. 11. The method of claim 10,
Electronic display
Wherein the processor is further configured to generate an image to be displayed on the electronic display, the image being prompted by the user to confirm whether the text corresponding to the request or the commitment is correct or true And a computing device. 11. The method of claim 10,
Wherein the processor is further configured to analyze parameters of the electronic message, wherein the parameters include at least one of a number of recipients, a length, a date and time, and a title header of the individual electronic message. 11. The method of claim 10,
Wherein the processor is further configured to analyze information about the user while the user is entering the electronic message.

Images