JP6904449B2 - Generate a meeting review document that contains links to one or more documents that have been reviewed - Google Patents

Description

[Related application]
This application applies to US Patent Application No. 14 / 992,273 (Agent Control Number: 49896-0859), Name "Electronic Meeting Intelligence", filed January 11, 2016; US Patent Application No. 14 / 992,278 ( Agent Management Number: 49896-0860), Name "Electronic Meeting Intelligence", filed January 11, 2016; US Patent Application No. 15 / 290.855 (Agent Management Number: 49896-0888), Name "Managing Electronic" Meetings Using Artificial Intelligence and Meeting Rules Templates, filed October 11, 2016; US Patent Application No. 15 / 290,856 (Agent Control Number: 49896-0889), Named "Creating Agendas for Electronic Meetings Using Artificial Intelligence" , Filed October 11, 2016; US Patent Application No. 15 / 290,858 (Agent Control Number: 49896-0890), Named "Selecting Meeting Participants for Electronic Meetings Using Artificial Intelligence", filed October 11, 2016 US Patent Application No. 15 / 290,860 (Agent Control Number: 49896-0891), Name "Real-Time (Intra-Meeting) Processing Using Artificial Intelligence", filed October 11, 2016; US Patent Application Number 15 / 290,861 (agent management number: 49896-0892), name "Post-Meeting Processing Using Artificial Intelligence", filed October 11, 2016; US patent application number 15 / 477,240 (agent management) Number: 49986-0902), name "Electronic Meeting Intelligence", filed April 3, 2017; US Patent Application No. 15 / 477,276 (agent control number: 49896-0903), name "Electronic Meeting Intelligence", Filed April 3, 2017; US Patent Application No. 15 / 728,360 (Agent Control Number: 49896-0906), Name "Person Detection, Person Identification and Meeting Start for Interactive Whiteboard Appliances", October 9, 2017 Japanese application; US patent application number 15 / 728,367 (agent control number: 49896-0907), name "Speech-to-Text Conversion for Interactive Whiteboard Appliances Using Multiple Services", filed October 9, 2017; US Patent application number 15 / 728,368 (agent control number: 49896-0910), name "Speech-to-Text Conversion for Interactive Whiteboard Appliances in Multi-Language Electronic Meetings", filed October 9, 2017; US patent Application No. 15 / 728,372 (Agent Management Number: 49896-0911), Name "Attendance Tracking, Presentation Files, Meeting Services and Agenda Extraction for Interactive Whiteboard Appliances", filed October 9, 2017; US Patent Application Number 15 / 728,376 (agent control number: 49896-0912), name "Interactive Whiteboard Appliances With Learning Capabilities", filed October 9, 2017; US patent application number 15 / 910,878 (agent control number) : 49986-0919), named "Conducting Electronic Meetings over Computer Networks using Interactive Whiteboard Appliances and Mobile Devices", filed March 2, 2018; US Patent Application No. 16 / 112,361 (Agent Control Number: 49896-0935) ), Name "Electronic Meeting Intelli gence ”, filed August 24, 2018; US Patent Application No. XX / XXX, XXX (agent control number: 49896-0937), name“ Artificial Intelligence Assisted Review of Physical Documents ”, filed March 15, 2019 US Patent Application No. XX / XXX, XXX (Agent Control Number: 49896-0938), Name "Artificial Intelligence Assisted Review of Electronic Documents", filed March 15, 2019; US Patent Application No. XX / XXX, XXX (agent management number: 49896-0941), name "Generating Document Edit Requests for Electronic Documents Managed by a Third Party Document Management Service using Artificial Intelligence", filed March 15, 2019; US Patent Application No. XX / XXX , XXX (agent control number: 49896-0943), name "Updating Existing Content Suggestions to include Suggestions from Recorded Media using Artificial Intelligence", filed March 15, 2019; US Patent Application No. XX / XXX, XXX (proxy) Personnel Control Number: 49896-0944), in connection with the name "Generating Suggested Document Edits from Recorded Media using Artificial Intelligence", filed March 15, 2019; the full contents of these applications are hereby incorporated by reference. The whole is incorporated as if it were fully described.

[Technical field]
Embodiments generally relate to the processing of electronic documents, and more specifically to computer software that analyzes suggestions for content in electronic documents and produces proposed edits. Proposed technical group unit: 2657. Proposed classification: 704.

The approaches described in this chapter are approaches that may have been attempted, but not necessarily those that were previously devised or achieved. Therefore, unless otherwise indicated, the approaches described in this chapter are not prior art to the claims of the present application and are not considered prior art by inclusion in this chapter.

One of the most common activities in the professional and / or academic environment is the review of the information presented in the document. For example, in a business environment, an employee may generate a document containing content that needs to be reviewed by one or more peers. Document review meetings may be held to discuss the content of the document and provide feedback on the content.

For example, during a document review meeting, participants may communicate suggested edits and comments to the document author. When receiving input from other participants, the author may listen and take notes during the meeting to ensure that all feedback is written down and that the feedback applies to the revision of the document. In addition, some participants may provide feedback on a printed version of the document or an electronic version of the document. The result of the review meeting can be a collection of comments and suggested edits in the form of captured notes, handwritten comments, and electronic comments. However, integrating the collected comments can be challenging when trying to ensure that all comments discussed during the meeting are accurately captured.

In view of the above, there is a need for an approach that automatically receives and identifies content proposals from conference participants and incorporates such proposals into the document.

According to another embodiment, the device includes one or more processors and one or more memories that store instructions. The processing of instructions by one or more processors results in the step of receiving a physical document containing the original content for review and markup provided by a first user in order to generate an electronic markup document. From the electronic markup document, a user code that identifies the first user who provided the markup to the physical document is identified. The first user ID associated with the first user is determined based on the identified user code. One or more electronic markups representing the content proposals proposed by the first user are identified from the generated electronic markup document. For each markup of one or more identified markups, the document portion of the original content of the electronic markup document corresponding to the markup is identified. Annotations containing the markup and the first user ID are generated for the markup. The original content with one or more annotations generated from the one or more markup that associates the annotation with the identified document portion is displayed electronically in the display window.

According to another embodiment, the device includes one or more processors and one or more memories that store instructions. The processing of instructions by one or more processors results in the step of receiving an electronic document containing the original content from the original electronic document for review, and an electronic markup provided by a first user. One or more electronic markups representing content proposals proposed by the first user are identified from the electronic document. For each electronic markup of one or more identified electronic markups, the document portion of the original content corresponding to the electronic markup is identified and includes the electronic markup and the first user ID of the first user. Electronic markup annotations are generated and associated with the identified document portion. The original content with one or more annotations generated from one or more electronic markup is displayed in electronic form in the display window.

According to another embodiment, the device includes one or more processors and one or more memories that store instructions. The processing of instructions by one or more processors results in the step of receiving one or more media content items that represent the content captured from the discussion of one or more electronic documents by one or more users. The media content portion corresponding to the content proposal for one or more electronic documents is identified from the one or more media content items. For each media content portion of the media content portion, the document portion is identified from one or more electronic documents corresponding to the media content portion, and a comment presenting the media content portion is generated. Annotations are associated with positions within a particular electronic document that correspond to a document portion of one or more electronic documents. One or more electronic documents with one or more annotations generated from the media content portion are displayed in electronic form in the display window.

According to another embodiment, the device includes one or more processors and one or more memories that store instructions. The processing of instructions by one or more processors results in the step of receiving one or more media content items that represent the content captured from the discussion of an electronic document by one or more users. The media content portion corresponding to the content proposal for the electronic document is identified from the media content item. For each media content portion of the media content portion, the document portion of the electronic document corresponding to the media content portion is identified and a determination is made as to whether the document portion of the electronic document is associated with an existing annotation. If it is determined that a document portion of an electronic document is associated with an existing annotation, the existing annotation is updated based on the media content portion. An electronic document with one or more updated annotations associated with one media content portion is displayed in electronic form within a display window.

According to another embodiment, the device includes one or more processors and one or more memories that store instructions. The processing of instructions by one or more processors results in the step of receiving content inputs that represent content suggestions for content in electronic documents managed by a third-party management system. One or more content proposals are identified from the content input. For each content proposal out of one or more content proposals from the content input, the document portion of the electronic document corresponding to the content proposal is identified and a document proposal request for the content proposal is generated. The document proposal request includes a content proposal, a content proposal type, a user's associated user ID associated with the content proposal, and an electronic document location based on the identified document portion. One or more generated document proposal requests are sent to a third party management system.

According to one embodiment, the device includes one or more processors and one or more memories that store instructions. The processing of instructions by one or more processors results in the step of receiving one or more media content items representing the content captured from the review meeting of one or more electronic documents. From one or more media content items, one or more agenda items discussed during the review meeting are identified. The media content portion corresponding to the content proposal for one or more electronic documents discussed during the review meeting is then identified from the one or more media content items. For each media content portion of the media content portion, the agenda of one or more agendas is associated with the media content portion, and the document portion corresponding to the media content portion is identified from the one or more electronic documents. .. To generate a topic set of media content parts, the media content parts are aggregated based on their associated agenda. A conference summary document is generated that includes a first area containing a media content item player configured to play one or more media content items, and a second area containing a list of one or more agenda items. Each agenda item listed contains a corresponding topic set for the media content portion. The meeting summary is presented in the display window.

In the accompanying drawings, similar reference numerals represent similar elements.

An exemplary computer architecture in which embodiments can be implemented is shown.

It is a flow diagram which shows the approach of generating an annotation based on the change proposed from the markup document and displaying the document together with the generated annotation.

An exemplary embodiment of a printed document containing a review user identifier is shown.

Shown is an exemplary embodiment of a graphical user interface that displays the original content of an electronic document and the identified markup superimposed on the original content.

Shown is an exemplary embodiment of an electronic document displayed with annotations representing the identified markup.

It is a flow diagram which shows the approach of generating an annotation based on a suggestion received from a markup electronic document and displaying the document together with the generated annotation.

Shown is an exemplary embodiment of an electronic document displayed with annotations representing the identified markup.

It is a flow diagram showing an approach to generate an annotation based on a proposed change from a media content item and generate an updated electronic document containing the corresponding proposed change.

Shown is an exemplary embodiment of an electronic document displayed with annotations representing the identified media content portion.

It is a flow diagram showing an approach of updating an existing annotation based on a proposed change from a media content item and generating an updated electronic document containing the corresponding proposed change.

It is a flow diagram which shows the approach to generate the annotation request based on the suggestion received from the content input.

Document Review A flow diagram showing an approach to generate a meeting summary document based on the media content items of the discussion captured during the meeting.

An exemplary embodiment of the generated conference summary document is shown.

It is a block diagram of the computer system which embodiment of this invention can be implemented.

In the following description, many details will be described in order to provide a complete understanding of the embodiments of the present invention for purposes of explanation. However, one of ordinary skill in the art will appreciate that the present invention can be practiced regardless of such particular details. In another example, well-known structures and devices are shown in block diagrams so as not to unnecessarily obscure the invention. Various aspects of the invention are described in the following chapters.
I. Outline II. Architecture A. Document management B. Intellectual content recognition model 1. Content recognition service 2. Markup recognition service 3. Conversation recognition service 4. Content correlation service C. Input / Output Service III. Outline of procedure A. Processing of marked-up physical documents B. Processing of marked-up electronic documents C. Processing of media content items, including suggestions D. Update of annotations with additional suggestions E. Generation of third party annotation request F. Generation of meeting summary document IV. Implementation mechanism

I. Summary Artificial intelligence has been introduced into document processing to identify content suggestions related to an electronic document under review from a wide variety of different types of sources. Content suggestions may represent comments and suggested edits to the content in the document under review. Content suggestions may be identified from physical notes and markup on the printed version of the document, or from changes identified in the electronic version of the document. In addition, content suggestions may be identified from the captured audio and video recordings of the document review meeting. Content suggestions may be received in the form of screenshots or other media from conference presentation devices such as the Interactive Whiteboard Appliance or other media devices. Artificial intelligence may analyze received content inputs, such as physical or electronic documents and media content, and identify content proposals that correspond to the document portion of the document being reviewed from the received content inputs. Artificial intelligence may determine whether the content proposal is a comment or proposed edit to the original content of the document. Artificial intelligence may generate appropriate annotations from the analyzed content suggestions and associate the annotations with their position in the original content of the document. A document with new annotations may then be presented to a user, such as the author of the document, for future review. The approach described herein improves the efficiency of correlating external content proposals with the original content in the document. Therefore, it provides a more efficient way to receive content suggestions and correlate the received suggestions with their corresponding document parts.

II. Architecture Figure 1 shows an exemplary computer architecture in which embodiments can be implemented. FIG. 1 includes various examples of a document management system 100 that supports receiving external content inputs and integrating the document under review into an electronic representation. The document management system 100 includes a document management service 105, a content reception service 110, a user identification management service 115, an annotation generation service 120, an intellectual content recognition model 130, a multifunction printing service 140, a document presentation service 150, a network 160, and data. Includes repository 170. Some elements shown in FIG. 1 may not be shown in other figures for illustration purposes. However, these elements may be included in the elements of other figures.

The document review meeting supported by the document management system 100 may include any kind of electronic or direct meeting in which one or more documents are reviewed by one or more participants. The document management system 100 may manage the content of the document under review and may automatically apply content suggestions and edits discussed during the meeting to the document under review.

In one embodiment, the document management system 100 represents a network of computers. A "computer" may be one or more physical computers, virtual computers, and / or computing devices. The computer may be a client and / or a server. Reference to "computer" herein may mean one or more computers unless expressly stated otherwise. Each of the modules / services shown in FIG. 1 causes the document management system 100 to refer to the module and perform a function or operation described in the present specification when an executable instruction is loaded and executed. It may include a set of one or more pages of main memory, such as RAM, in a computer. In addition, the document management service 105 may include a set of pages in RAM that, when executed, include instructions for performing electronic document management and storage as described herein. The instructions may be in machine executable code in the CPU's instruction set, in JAVA®, C, C ++, OBJECTIVE-C, or any other human-readable programming language or environment. , Alone, or in combination with scripts in JAVASCRIPT®, other scripting languages and other programming source texts, may be compiled based on the written source code. The term "page" is intended to broadly represent any area in main memory. Certain terms used in the system may vary depending on the memory architecture or processor architecture. In another embodiment, each component in the document management system 100 is also one of the source codes digitally stored in a mass storage device such as a non-volatile RAM or disk storage device in one or more computers. It may represent one or more files or projects, which, when compiled or interpreted, result in the generation of executable instructions, which, when executed, provide the document management system 100 with the modules described above. Perform a function or operation described herein with reference to it. In other words, the drawing organizes the source code for the programmer or software developer to later compile into an executable format or to interpret into bytecode or equivalent for execution by the document management system 100. A method of configuration may be presented.

A. Document Management In one embodiment, the content receiving service 110 may include instructions for receiving content related to the document under review. The content receiving service 110 may receive proposed changes or comments related to the original content, the marked content, and / or the document under review. For example, the content receiving service 110 may receive an electronic file containing the original document for review. In another example, the content receiving service 110 may receive an electronic file containing both the original content and markup for the original content. In yet another example, the content receiving service 110 may receive content media items, such as audio files, video files, or image files, that contain proposed changes or comments related to the document under review.

The content receiving service 110 may be configured to transmit the received electronic document to the document management service 105. In one embodiment, the document management service 105 identifies the original content for the received electronic document, identifies the content as a markup to the original document, associates the user with their corresponding markup, and / or It may include instructions for managing content storage and communication, and content proposals for electronic documents. For example, the content receiving service 110 may send an electronic document representing the original version of the document to the document management service 105. The document management service 105 may identify from the electronic file the original content representing the original version of the document for review. In another example, the content receiving service 110 may send a version of the document, including the markup, to the document management service 105. The document management service 105 may compare the content from the markup version of a document to the original content of a previously received document in order to identify one or more markups in the markup version of the document. ..

In one embodiment, the document management service 105 may identify one or more codes and / or marks in a markup version of a document that indicate the user who created the markup. For example, the markup version of a document received by the content receiving service 110 may be due to a printed version of the document, including a custom barcode or QR code® that identifies a particular user. A printed version of the document may include a custom ID code that can be used to identify a particular user as a reviewer who has handwritten the mark. The document management service 105 may identify the code in the markup version of the document and may send the identified code to the user ID management service 115 to identify a particular user associated with the code.

In one embodiment, the user ID management service 115 may include an instruction to identify the user associated with the custom ID code in the markup document. The data repository 170 may be configured to store the association between the user and the custom ID code. The user ID management service 115 may communicate with the data repository 170 to read the user-ID code association to determine the user associated with the custom ID code in the received markup document. For example, the content receiving service 110 may receive a markup document including a specific ID code. The content receiving service 110 may transmit the document to the document management service 105. The document management service 105 may identify the specific ID code in the document, and may transmit the specific ID code to the user ID management service 115. The user ID management service 115 may receive the specific ID code and may query the data repository 170 to determine the specific user associated with the specific ID code. Upon determining the specific user associated with the specific ID code, the user ID management service 115 may transmit the identifier of the specific user to the document management service 105. As a result, the document management service 105 may associate the markup from the received electronic document with the appropriate user who created the markup.

In one embodiment, the annotation generation service 120 may include instructions for generating one or more annotations and associating the annotations with positions in the original content of the document. For example, the document management service 105 may manage the original content of the document as well as the markup identified for the document. The document management service 105 may send a request to the annotation generation service 120. The annotation generation service 120 may receive the request and generate a new annotation containing the identified markup, with the newly generated annotation being the original content corresponding to the document portion corresponding to the identified markup. It may be associated with a position within. In another example, the annotation generation service 120 may receive a request to update an existing annotation to include additional markup. The annotation generation service 120 may identify existing annotations and insert additional markup into the existing annotations.

In one embodiment, generating new annotations and / or updating existing annotations is not limited to generating and / or updating comments. Annotations may represent proposed edits to the content in the document. For example, if the markup indicates to change the word "impossible" to "highly unlikely", the annotation generation service 120 will change the word "impossible" from the original content to "impossible". You may receive a request to update to "hard to happen". The annotation generation service 120 may generate proposed edit annotations representing markup and may also include the proposed edits and comments indicating the user who proposed the edits.

FIG. 3B shows an exemplary embodiment of a graphical user interface that displays the original content of an electronic document and the identified markup superimposed on the original content. Annotations 330A and 340A represent annotations that are proposed edits generated by annotation generation service 120. Note 330A and 340A are based on the markup interpreted as the proposed edit. Note 330A is based on mark 330, which represents a strikethrough drawn on the word "this". The strikethrough may be interpreted as a suggestion to remove the word "this". Note 340A is based on the mark 340, which represents a handwritten circle surrounding the word "holigram". The word "holigram" is a typographical error in the word "hologram" and the mark 340 may be interpreted as a suggestion to correct the spelling of "holigram". Note 335A is based on mark 335, which represents a highlighted area of a word with a line to a handwritten note "do we want to show the 22" model? ". Mark 335 may be interpreted as a comment.

In one embodiment, the generated annotation may be customized to reflect the annotation type. For example, annotation 335A may be colored with one particular color, such as yellow, to represent comment annotations. On the other hand, annotations 330A and 340A may be colored with another color, such as red, to represent the proposed editorial annotation. In one embodiment, the annotation may include the user ID of the user associated with the annotation. The annotation may include selectable action buttons that may be used to accept or reject the proposed edit, such as the check button 341 and the reject button 342 in the annotation 340A. The annotation may also include an electronic link to the content input associated with the generated annotation. For example, the annotation may include an electronic link to the audio file of the captured discussion associated with the annotation. The electronic link may be a screenshot, or other image of markup, or a link to a suggested comment or change. The electronic link may be a file, web page, object, or any kind of link, hyperlink, or shortcut to any other destination specified by the electronic link.

In one embodiment, network 160 may represent a communication medium or mechanism that provides data exchange between A and B. Examples of network 160 include networks such as LAN (Local Area Network), WAN (Wide Area Network), Ethernet (registered trademark) or the Internet, or one or more terrestrial, satellite or wireless networks. Not limited. In one embodiment, the Data Repository 170 stores any kind of data related to documents, markup, annotations, associated media content items, and any other data object associated with documents and annotation management. It may be configured as follows.

B. Intellectual Content Recognition Model In one embodiment, the intellectual content recognition model 130 identifies the document type and document portion of the received electronic document, determines the content proposed from the handwritten memo, and proposes the content from the media content item. It may include instructions for implementing one or more services that are configured to determine and determine the correlation between the content proposal and the document portion. The intellectual content recognition model 130 may be composed of a content recognition service 131, a markup recognition service 132, a conversation recognition service 133, and a content correlation service 134.

In one embodiment, the intellectual content recognition model 130 trains one or more machine learning models that can be used by each of the content recognition service 131, the markup recognition service 132, the conversation recognition service 133, and the content correlation service 134. And instructions to execute. The intelligent content recognition model 130 may implement one or more machine learning models using various machine learning algorithms and structures. Exemplary machine learning algorithms include artificial neural networks, deep neural networks, convolutional neural networks, recurrent neural networks, classifiers, and other supervised or unsupervised machine learning algorithms. The proposed methods and systems may be implemented using any suitable machine learning algorithm or architecture.

1. 1. Content Recognition Service In one embodiment, the content recognition service 131 determines the document type of the original electronic document based on the content portion in the document, the format and style of the document, and other directives in the content of the original electronic document. It may include instructions configured to identify. For example, the content recognition service 131 may receive the original electronic document, which is a draft of the patent application document, as input. Content recognition service 131 may analyze the content of the original electronic document, including titles, chapter titles, formats, keywords within the content, and any other property that may indicate the document type of the original electronic document. For example, when the original electronic document is a draft of a patent application document, the content recognition service 131 includes "background technology", "brief description of drawings", "form for carrying out the invention", and "claims". , And chapter names such as "summary" may be identified and analyzed, and based on the chapter title, the original electronic document may be determined to be the document type for the patent application.

In one embodiment, the content recognition service 131 may be configured to implement a machine learning model configured to determine the document type of an electronic document. The machine learning model takes an electronic document as input and produces an output that specifies the document type of the electronic document based on the title, chapter title, format, keywords in the content, and any other characteristics of the electronic document. It may be configured. In addition, the machine learning model may output a dataset of context-specific characteristics of the determined document type. For example, context-specific characteristics may define keywords, phrases, formats, and any other characteristics that can be used to help determine the proposal type and location of received content proposals.

In one embodiment, the machine learning model is trained to determine document types using a training dataset that includes a set of documents that are classified as specific document types based on the identified document characteristics that define each document type. You can do it. For example, the training dataset may include documents and record identified document characteristics classified as any of patent application document types, contract document types, proposal document types, and technical specification document types.

In one embodiment, the machine learning model may be configured to identify a document portion from the original content that can be used when correlating the markup with the original content. Document parts may represent specific keywords and specific phrases such as titles, chapters, paragraphs, sentences, product descriptions and names. For example, if the document type is a technical specification document, the chapters and subdivisions within the document may be identified as parts. In addition, specific product keywords such as "Interactive Assistance Service" and "IFDP" may be identified as keyword portions.

2. Markup Recognition Service In one embodiment, the markup recognition service 132 may include instructions configured to interpret a handwritten memo identified from a physical document and generate a Transcription of the handwritten memo. For example, the markup recognition service 132 may receive, as input, a handwritten note from the markup document, i.e. an image of "can we expand on this idea." The markup recognition service 132 may analyze the image of the handwritten memo, determine the meaning of the handwritten memo, and generate text representing the interpretation of the memo. The markup recognition service 132 may determine the type of content proposal, such as a comment or proposed edit, based on the text transcription and meaning of the handwritten memo. Using this example, the memo "can we expand on this idea" may be interpreted as a comment by the markup recognition service 132. On the other hand, a handwritten note containing a note stating "change to'highly unlikely'" following the strikethrough of the word "impossible" may be interpreted as a proposed edit.

In one embodiment, the markup recognition service 132 may be configured to implement a machine learning model configured to convert handwritten notes into text. The machine learning model may be configured to receive an image of a handwritten memo as input and may generate a converted character character set as output. In one embodiment, the machine learning model may be trained to determine the character transfer of a handwritten memo using a training image set of the handwritten memo and a training data set containing a corresponding character transfer from one or more users. .. For example, a training image set may include handwritten words, letters, shorthand, and other symbols that may be used by one or more users. In one embodiment, the machine learning model may be further configured to input a document type into the model. The input document type may be used to improve the character transfer accuracy by specifying the document context of each handwritten memo. For example, if the document type is academic papers related to chemical, machine learning model identifies the handwritten memo as "CO2" as compounds, trained to convert the note to "CO _2" or "carbon dioxide" May be done.

In one embodiment, the machine learning model implemented by the markup recognition service 132 may be configured to assign a confidence score to the markup character transcription. The confidence score may represent a value that determines how accurate the output of the machine learning model is based on the model training data. For example, machine learning models are highly confident in interpreting the cancellation of the word "this" based on the position of the line above the word, the angle of the line, the color of the line, and whether the cancellation is associated with other marks. You may assign a score. Conversely, if the undo of the word "this" overlaps below the word and it is unclear whether the mark is undo or underlined, the machine learning model may assign a low confidence score.

3. 3. Conversation Recognition Service In one embodiment, the conversation recognition service 133 may include instructions configured to translate a conversation captured from audio and / or video into translated characters. For example, the conversation recognition service 133 may receive a portion of the recorded audio file from the document review meeting as input. The conversation recognition service 133 may analyze the audio content and determine the translated characters that represent the spoken words in the audio content.

In one embodiment, the conversation recognition service 133 may be configured to implement a machine learning model configured to translate the captured conversation into translated characters. The machine learning model may be configured to receive input containing media content items that represent captured conversations to be transformed. The machine learning model may generate a transcription of the captured conversation as output. In one embodiment, the machine learning model may be trained with a training dataset that includes parts of the conversation and the corresponding transcription. For example, training data may include phrases spoken by a plurality of different users, as well as corresponding meanings. The spoken phrase in the training data may also include attributes such as accents, themes, and identified background noise that can affect how the conversation is transformed.

In one embodiment, the machine learning model of the conversation recognition service 133 may be configured to identify a media content portion that indicates a content suggestion for either a comment or a proposed edit. Detected conversations within media content items may be converted from conversations to text. The letters may then be analyzed for specific words or phrases that may indicate either the proposed edits or comments of the reviewed document. For example, remarks such as "the word ... is duplicated", "we need more examples", "the introduction is too short", "the word ... is misspelled", "could you update figure X" It may be shown that a particular media content portion is relevant to the proposed edits or comments on the document. The machine learning model may also identify statements that may indicate a position in the document, such as "on the second page," "the final paragraph," "the sentence starting with," and so on. The output of the machine learning model may include a set of media content parts identified as possible proposed edits for the reviewed document.

In one embodiment, the machine learning model implemented by the conversation recognition service 133 may be configured to assign a confidence score to the character transcription of a set of identified media content portions. The confidence score may represent a value that determines how accurate the output of the machine learning model is based on the model training data. Determining the confidence score depends on the clarity of the conversation within the media content portion and how well the machine learning model can match the translated words and phrases into the spoken conversation within the media content portion. May be based. For example, if the conversation is ambiguous or the speaker speaks very fast, the machine learning model may assign a low confidence score to the translated characters.

4. Content Correlation Service In one embodiment, the Content Correlation Service 134 identifies a match between a document portion of an electronic document and converted characters from handwritten notes, captured conversations, screenshot images, and any other media. May include instructions configured to do so. For example, the content correlation service 134 may receive, as input, a text transcription of the original content of the electronic document and a content proposal that is from a handwritten markup or from a captured conversation. The content correlation service 134 may identify document parts within the original content and may determine whether one or more document parts match the text transcription of the content proposal. Alternatively, the content correlation service 134 may receive input specifying the document type. The input may be further used to determine a match between the transcription and the document portion of the original content.

In one embodiment, the content correlation service 134 may be configured to implement a machine learning model configured to determine the match between the character transcription and the document portion. Inputs to the machine learning model may include transcriptions representing the original content of the document, the document type, and the content proposed from handwritten notes or from captured conversations. The machine learning model may output the contextual position in the document part and / or the original content corresponding to the character transcription. For example, if the electronic document type is a research paper and the character transcription is equal to "reference 49 is incorrect", the machine learning model indicates that the character transcription represents a reference footnote in the electronic document, that the document type of the document containing the footnote. And may be determined based on the identified document part. The machine learning model may output the position in the original content corresponding to footnote 49. The machine learning model may be trained using a plurality of document parts of an electronic document and corresponding character transcriptions of content proposals from multiple document parts of an electronic document.

C. Input / Output Services The document management system 100 may include one or more input and output services that receive the document and present an electronic document containing the interpreted annotations. In one embodiment, the multifunction printing service 140 may represent one or more multifunction printer devices configured to scan a physical document. For example, a multifunctional printing service may be used to scan the original document or markup physical document. The multifunction printing service 140 may transmit the scanned document as a data object to the content receiving service 110 for processing. In another embodiment, the multifunction printing service 140 may receive a request to print a physical version of an electronic document for distribution to reviewers and other users.

In one embodiment, the document presentation service 150 may include instructions for implementing a graphical user interface that presents one or more electronic documents containing the corresponding annotations. For example, the document presentation service 150 includes one or more content window frames for displaying content from an electronic document, one or more annotations corresponding to the displayed content, and annotations and others based on the reviewer or other user. You may generate a graphical user interface that includes display option controls for filtering your content.

III. Procedure Overview The following chapters describe embodiments for implementing document reviews, including receiving and merging document proposals from multiple different input services.

A. Processing Marked-up Physical Documents In one embodiment, the document management system 100 receives document markup in the form of physical paper, identifies content proposals, and generates annotations about the content proposals of the document under review. By doing so, it may be used to assist in document review. Illustrative features include receiving one or more markup documents containing one or more markups for content proposals, identifying the type of content proposal, and generating annotations for each content proposal, without limitation. Includes that, associating the generated annotation with the document portion of the document under review, and displaying the document in the generated annotation in a display window.

The document management system 100 may be configured to receive the original document, including the unmarked content, and determine the original content for review. The original content may represent content that is reviewed by one or more users. In one embodiment, the multifunction printing service 140 may represent a scanner that receives and scans a paper document. The multifunction printing service 140 may generate a scanned image such as a PDF (Portable Document Format) file. The multifunction printing service 140 may transmit the PDF file of the original document to the content receiving service 110. The content receiving service 110 may transmit the PDF file to the document management service 105. The document management service 105 may analyze the PDF file and identify the original content to be marked for review. The document management service 105 may store the original content in the data repository 170.

In another embodiment, the multifunction printing service 140 may represent a multifunction printer capable of identifying the original content from the scanned physical document and storing the original content in the data repository 170. For example, the multifunctional printing service 140 may be configured by plug-in software or may use an API (application programming interface) in order to communicate with various services in the document management system 100. The API may include a function call that performs identification of the original content and storage of the original content in the data repository 170.

Identifying the original content from the scanned physical document, the document management system 100 may generate one or more copies of the physical document that may be distributed to one or more review users. In one embodiment, the document management service 105 may send a print request to generate one or more physical documents to the multifunction print service 140. The print request may include the user identifier of the review user. For example, if a document review meeting is scheduled by users A to D, the print request may specify users A to D. As a result, four copies of the document are printed, and each printed document contains an identifier that identifies one of users A to D.

In one embodiment, the document management system 100 may determine potential reviewers based on the name identified by the original content of the document. For example, if the document is a research treatise by multiple authors, the document management system 100 may identify the author's name and generate one or more copies of the physical document, each copy of each of the authors. Includes identifier.

In one embodiment, the multifunction printing service 140 may print a document having a printed name of the review user, a machine-readable code that identifies the review user, or both. FIG. 3A shows an exemplary embodiment of a printed document that includes a review user identifier. Document 305 represents a physical document printed by the multifunction printing service 140. Title 310 represents a document review title indicating the purpose of the physical document. For example, title 310 may include a document review meeting, including the time and place of the meeting. In another example, title 310 may have "For Review" added before the document title. In yet another example, title 310 may be customized to display any title for the review user.

The reviewer portion 315 may indicate a printed portion that includes the name of the reviewer. For example, the reviewer portion 315 may display the name of a review user, such as Jane Doe. Each printed physical copy may contain a different reviewer's name for each review user. If an additional or unidentified user reviews the printed document, the multifunction printing service 140 may print a copy of the document with "Reviewer: ________". Here, the additional user may write his or her name.

The user code 320 may indicate any printed code, such as a QR code, barcode, or any other machine-readable code. It may be used by the document management system 100 to identify review users when a marked copy of a document is scanned and analyzed for proposed edits and comments. For example, the multifunction printing service 140 may print a QR code that identifies the reviewer Jane Doe. In an example where an additional document copy is printed without an identified user, the multifunction printing service 140 can be used to flag the scanned marked-up document to be reviewed by an ad hoc reviewer. The guest user code 320 may be printed. In one example, when the document management system 100 identifies an ad hoc reviewer, the document management system 100 may analyze the handwritten name on the marked-up document to determine the user's identification information. If the user's identification information is not determined, the document management system 100 may associate any markup by the reviewer as a "guest reviewer". The "guest reviewer" may then be manually assigned to the appropriate reviewer. Content 325 in document 305 may represent the original content to be reviewed.

The document management system 100 is not limited to printing a copy of a document using the multifunction printing service 140. In one embodiment, the document management service 105 may generate, for each reviewer, a document file containing each of the title 310, the reviewer portion 315, the user code 320, and the content 325, with the document files being separate. It may be sent to each reviewer for printing on the printing device.

After printing the physical document and distributing the document to each reviewer, each reviewer may generate the proposed edits and comments directly on the physical document. FIG. 2 is a flow diagram showing an approach of generating annotations based on the changes proposed from the markup document and displaying the document with the generated annotations. Process 200 may be executed by a single program or a plurality of programs. The operation of the processing as shown in FIG. 2 may be implemented by using the processor executable instruction stored in the computer memory. For the purpose of providing a clearer example, the operation of FIG. 2 is described as being performed by the service of the document management system 100. For clarity purposes, process 200 is described in terms of a single entity.

At block 202, the document management system 100 may receive a physical document containing the original content for review and markup provided by the first user to generate the electronic markup document. In one embodiment, the multifunction printing service 140 may scan a physical document containing markups for the original content and send a scan called an electronic markup document to the content receiving service 110. The content receiving service 110 may receive the electronic markup document and transmit the electronic markup document to the document management service 105.

FIG. 3B shows an exemplary embodiment of the marked up physical document received by the document management system 100. In one embodiment, the marked-up physical document is handwritten or typed, such as written words, strikethroughs, underlines, enclosed words, highlights, and any other type of physical mark. May include markups. Referring to FIG. 3B, document 305 represents the same printed document as shown in FIG. 3A and includes a title 310, a reviewer portion 315, a user code 320, and content 325. Markup is indicated by marks 330, 335, and 340. The mark 330 represents a strikethrough above the word "this". The mark 335 represents a highlighted portion of the word in the content 325 and a line to the handwritten note "do we want to show the 22" model? ". The mark 340 represents a handwritten circle around the word "holigram".

Referring to FIG. 2, at block 204, the document management system 100 identifies the first user who provided the markup on the physical document. In one embodiment, the document management service 105 may identify the first user by identifying the user code 320 from the electronic markup document, and manages the user ID to determine the user associated with the user code 320. A request may be sent to service 115.

In another embodiment, if the physical document does not include the user code 320, the document management service 105 may identify the first user by identifying the mark in the reviewer portion 315 that corresponds to the handwritten name. The document management service 105 may communicate with the markup recognition service 132 to identify the first user based on the handwritten name. For example, the markup recognition service 132 may receive an image crop of a handwritten name from the reviewer portion 315 and may analyze the image crop to determine a character set representing the name of the first user.

The markup recognition service 132 is not limited to identifying the first user based on the mark corresponding to the handwritten name. In another embodiment, if the physical document does not include a user code 320, the document management service 105 may identify the first user by identifying the machine-printed name within the reviewer portion 315. If the physical document contains a name printed within the reviewer portion 315, or a name printed near or around the reviewer portion 315, the document management service 105 communicates with the markup recognition service 132 and is printed. The first user may be identified based on the name. For example, the markup recognition service 132 may receive an image crop of the printed name and implements Optical Character Recognition (OCR) technology to represent the name of the first user from the image crop. The character set may be identified.

At block 206, the document management system 100 determines a first user ID associated with the identified first user. In one embodiment, if the identified first user is based on user code 320, user ID management service 115 queries the data repository 170 to read the associated first user ID associated with user code 320. The first user ID may be determined accordingly. The data repository 170 may store the association between the user, the user code, and the user ID, such as user code 320 and user Jane Doe. The user ID management service 115 may receive the first user ID of Jane Doe from the data repository 170. The first user ID may represent any kind of user identifier, such as an employee ID, an email-based ID, or any other identifier configured to be used to identify the user.

In another embodiment, if the identified first user is based on a handwritten name transcribed from the reviewer portion 315, the markup recognition service 132 accesses the data repository 170 and the user's name is derived from the handwritten name. It may be determined whether or not it matches the transcribed character set. For example, if the character set "Jane Doe" is identified by the markup recognition service 132, the markup recognition service 132 queries the data repository 170 to see if the user matches the "Jane Doe" character set. May be decided. If the user matches "Jane Doe", the markup recognition service 132 may query the data repository 170 to read the first user ID associated with the user "Jane Doe".

In yet another embodiment, the markup recognition service 132 cannot determine at a certain confidence level the character set that constitutes the user's name, based on the handwritten name transcribed from the reviewer portion 315 by the identified first user. , The document management service 105 may assign a first user ID corresponding to a "guest user" or an "unidentified user". If the markup recognition service 132 fails to find a matching user in the data repository 170, the first user ID may correspond to the "guest user" or "unidentified user" label. In one embodiment, the user may later manually assign a user ID to replace the "guest user" or "unidentified user" label.

At block 208, the document management system 100 may identify one or more markups that represent content proposals proposed by the first user. In one embodiment, the document management service 105 compares the electronic markup document with the original content from the original document to identify the original content in the electronic markup document to identify the mark in the electronic markup. The up may be identified. The document management service 105 may then identify the markup in the electronic markup document by identifying content that is separate from the original identified content. For example, with reference to FIG. 3B, the original content may be identified as the printed content within the content 325. The document management service 105 may then identify marks 330, 335, 340 as one or more markup that is separate from content 325. In one embodiment, the document management service 105 ignores additional printed content, such as title 310, reviewer portion 315, and user code 320, because the content is not considered as markup or original content. It may be configured to be treated as content that should be treated.

In one embodiment, the document management service 105 may use the markup recognition service 132 to analyze each of the identified markups to determine the meaning of the markup. For example, the markup may include handwritten comments that can be translated into text, and other lines or arrows within the markup are analyzed and their position on the document is referenced to determine the meaning of the markup. You can do it. For example, markup may be identified as a line that removes duplicate words, adds punctuation, corrects errors, inserts words, and so on. In one embodiment, the markup recognition service 132 may implement a machine learning model configured to convert handwritten notes into characters and determine the meaning of the converted characters. The machine learning model may receive an image of handwritten markup as input and may generate an output that includes a transcription of the handwritten markup. For markups that include lines, arrows, circles, or any other non-character writing, the machine learning model outputs the transformed characters that describe the markup, and the decision whether the markup is a comment or a proposed edit. You can do it. For example, if the markup is a word strikethrough, the output may be characters indicating that the canceled word should be deleted and that the markup is the proposed edit. Referring to FIG. 3B, the mark 330 is a strikethrough above the word "this" and may be translated into a character describing "delete'this'?".

In one embodiment, the markup recognition service 132 may be configured to determine the proposed type of markup analyzed. Proposal types may include, but are not limited to, comments and proposed edits. The proposed edit may represent a handwritten note or mark indicating a proposal to change the text in the original content. For example, if a word has a strikethrough mark above the word, the strikethrough mark may be interpreted as a proposed edit to remove the word. Other examples may include marks and handwritten characters. For example, the word "car" may be circled and may have the handwritten character "change to automobile" next to the circle. The handwritten letter "change to automobile", along with the circle surrounding the word "car", may be interpreted as a proposed edit to change the word car to the word automobile.

Comments may represent notes, questions, or other comments related to one or more words or phrases in the original content. For example, the text in the original content may be underlined or circled and may include a handwritten note "Should we expand upon this?". This handwritten note may be determined to be a comment as it does not convey the explicitly proposed edits. Conversely, if the handwritten note says "please remove" next to the underlined / circled sentence, the markup recognition service 132 has the note "please remove" underlined. It may be determined that the edit was proposed to remove the / circled sentence.

In one embodiment, the markup recognition service 132 may transmit output to the document management service 105. The output includes the converted characters of the handwritten markup, the determined markup type such as the proposed edit or comment, the image of the markup, and the confidence score that describes the confidence level assigned to the converted characters. May include a data object that contains. For example, a data object that represents a markup for an underlined sentence containing the note "please remove" could be a translated character with the phrase "please remove", a markup type assigned as the proposed edit, and an underline. It may include image crops of drawn sentences and handwritten notes, and confidence scores associated with the converted characters.

At decision block 210, the document management system 100 may determine if there is any remaining markup to analyze. In one embodiment, the document management service 105 may be configured to iteratively analyze each of the identified markups. For example, marks 330, 335, and 340 may be identified and the document management service 105 may iteratively analyze each markup to determine the electronic document portion corresponding to the markup. In one embodiment, if the document management service 105 determines that there is remaining markup to analyze, the document management service 105 may proceed to block 212.

At block 212, the document management system 100 identifies a document portion corresponding to a particular markup from the original content. In one embodiment, the document management system 100 implements one or more machine learning models to identify the document type of the original content and address sections, phrases, specific keywords, and any other content-specific parts. The specific document portion in the original content may be identified and the correlation between the specific markup and the specific document portion in the original content may be determined.

In one embodiment, the content recognition service 131 may analyze the original content from the document to determine the document type. Different document types may have their own terms, phrases, and formats. These may be used to help determine how a particular markup is interpreted and correlates with the document portion. For example, if the original content is of a patent application document type, the proposed edit containing a phrase such as "move limitation to a dependent" will include the phrase highlighted by the claims limitation or the proposed edit. It may be interpreted as a proposal to move from an independent claim to a new dependent claim. In one embodiment, the content recognition service 131 may implement a machine learning model configured to determine the document type of the electronic document and the specific document portion that constitutes the original content of the electronic document. The machine learning model may receive the original content from the electronic document as input and may output the document type of the electronic document and the document portion identified from the original content.

The document type determination can improve the accuracy of determining the correlation between the markup and the content portion, thereby improving the quality of the automatically generated annotations. In one embodiment, the document management service 105 may be configured to store the output of the content recognition service 131, including the document type and the document portion from the original content, in memory. As a result, the determination of the document type and the document portion may be executed once. Analysis after the original content and markup may refer to the stored data of the document type and document portion corresponding to the current document.

Further, when the document type is determined, the document management service 105 may read the document type attribute from the data repository 170. Document type attributes may include characteristic values, such as stored keywords, phrases, formats, and any other characteristics associated with a particular document type. For example, if the document type is a Ricoh-specific product technical specification, the document type characteristic values are from the product acronym, technical specification format, and other technical specification documents used to train machine learning models. It may contain a determined product-specific phrase.

Once the document portion within the original content has been determined, the document management service 105 may determine the correlation between the particular markup and the document portion. In one embodiment, the content correlation service 134 may identify a match between a particular markup and a document portion of the original content. The content correlation service 134 may use a machine learning model configured to determine the match between the transcription of a particular markup and a document portion. For example, the mark 335 in FIG. 3B may be interpreted as "do we want to show the 22" model? "And may be correlated with a portion of the content identified as" 22'-84' IFDPs. " The correlation may be based on the highlighted markup portion and the transformed characters that specify the model 22 "identified within the document portion that specifies the model 22.

Correlating the particular markup to the document portion, the document management service 105 may generate annotations for the particular markup. Referring to FIG. 2, at block 214, the document management system 100 may generate annotations for a particular markup, including character transcription of the markup and a first user ID, and associate the annotations with the identified document portion. Annotations may be expressed as visible comments, such as balloon comments, including transcription, or as proposed edits, which can be expressed as tracked changes. For example, if the markup type is determined to be a comment, the annotation may be expressed as a visibility comment. However, if the markup is a proposed edit, the annotation may be expressed in the balloon comment as a tracked change or proposed edit based on the confidence score determined by the markup recognition service 132.

FIG. 3C shows an exemplary embodiment of an electronic document displayed with annotations representing the identified markup. Annotation 335A may represent an annotation generated for mark 335. Note 335A includes the converted characters of the particular markup and the ID of the first user. In another example, annotation 330A may represent the annotation generated for mark 330. Note 330A may include characters explaining that the mark 330 represents the proposed edit to remove the duplicate word "this". If the mark 330 has a high confidence score, the annotation 330A may be colored differently to indicate that the annotation 330A represents an edit that has already been applied. However, if Mark 330 has a low confidence score, Note 330A may appear as a pending change that requires user approval. In one embodiment, the confidence threshold may be used to determine whether the changes associated with the proposed edit should be applied to the content. For example, if the confidence score is based on a 100-point value scale, the confidence threshold may be set to 85. Here, confidence values above 85 are automatically applied and confidence values below 85 are not automatically applied, but are displayed in the annotation with an approve or reject button.

In one embodiment, the content correlation service 134 may be further configured to identify whether the translated characters of a particular markup include ready-to-execute instructions. The ready-to-execute command may be any kind of command for external content that can be automated by the document management service 105. For example, the converted characters may include a command for "insert a picture of printer model 92XX". The content correlation service 134 may identify this command as a command for reading a file, link, or other object. The content correlation service 134 may send a request to execute the command to the document management service 105. The document management service 105 may receive and execute the command. For example, if the command is based on the converted text "insert a picture of printer model 92XX", the document management service 105 may use an image from the image's internal repository or publicly available. The image of the 92XX printer may be read from. In another example, the document management service 105 may read a link to an image search result of a 92XX printer from a publicly available search engine.

The document management service 105 may send the result of the executed command, which may be a link to an image or a page of search results, to the annotation generation service 120. The annotation generation service 120 may be configured to receive the result of the executed command and insert the result into the annotation generated for a particular markup. For example, if the result is an internal or external image, the annotation generation service 120 may insert a link to the internal or external image. If the result is a link to an image search result, the annotation generation service 120 may insert the search result link into the annotation generated for a particular markup.

In one embodiment, when generating annotations for a particular markup, the document management system 100 may store the association between the generated annotations and the identified document portion in the data repository 170.

Upon generating the annotation, the document management system 100 may proceed to decision block 210 to determine if there is additional markup that needs to be analyzed. If there are additional markups to be analyzed, the document management system 100 may repeat blocks 212 and 214 to generate annotations for each of the markups. If decision block 210 determines that there is no additional markup to be analyzed, document management system 100 may proceed to block 216.

In one embodiment, the document management system 100 may repeat blocks 202-214 for each physical document, including markup, received from the user. For example, if a second user named John Smith also provides a physical document containing the markup, the document management system 100 receives and scans the marked-up document from John Smith and the John Smith user from the physical document. You may identify the code, identify one or more markups generated by John Smith, and generate one or more annotations associated with the markup provided by John Smith.

At block 216, the document management system 100 may display the original content in a display window in electronic form, along with one or more generated annotations. In one embodiment, the document management service 105 communicates with the document management service 105 to generate a graphical user interface that displays the original content with one or more generated annotations in a display window on the client computing device. You can do it. The graphical user interface may include, but is not limited to, overlaying the original content of the document, the generated annotations, and the markup associated with the annotations on the display of the original content.

FIG. 3C illustrates an exemplary embodiment of a graphical user interface used to display the original content of an electronic document, identified markup overlaid on top of the original content, and generated annotations. Shown. The document window 350 may present display elements configured to display the original content of the electronic document, superimposed markup, and other revisions related to the markup and / or other proposed edits. .. The reviewer drop-down list 360 may present a drop-down list of available reviewers that provided the suggested edits for the electronic document. If a particular reviewer, such as Jane Doe, is selected from the Reviewer drop-down list 360, the graphical user interface may display the markup and annotations associated with the reviewer Jane Doe. In one embodiment, the reviewer drop-down list 360 may include an "all reviewers" selection that may display markup and annotations from each reviewer who provided the edits proposed to be selected. When displaying annotations from multiple reviewers, the document presentation service 150 may configure reviewer-based annotations and associated markup color codes in a graphical user interface. For example, if all reviewer selections display suggested edits from Jane Doe and John Smith, the graphical user interface displayed highlights the markup and annotations associated with Jane Doe in red and John. Markups and annotations associated with Smith may be highlighted in green.

In one embodiment, the annotation presenting the edited edit proposed by the reviewer may include a control button to approve or reject the proposed edit. For example, annotation 330A includes a check button 331. When the check button 331 is selected, the proposed edit is applied to remove the duplicate word "this" from the electronic document. In another embodiment, Note 340A includes a check button 341 for applying the proposed edit and a reject button 342 for rejecting the proposed edit. Selecting the check button 341 causes the document management service 105 to change the spelling of "holigram" to the correct spelling of "hologram". The selection of the reject button 342 causes the document management service 105 to keep the original spelling of "holigram" in the electronic document without applying the proposed edits.

A user browsing the graphical user interface may approve or reject the proposed edits from the reviewer by approving or rejecting each of the displayed annotations. In one embodiment, the document management service 105 may store the changes applied to the electronic document as a new version of the electronic document in the data repository 170. A new version of an electronic document may contain one or more changes. For example, a new version of an electronic document may include all changes approved and / or rejected during a single review session of the graphical user interface by the user. In another example, each saved version of an electronic document may contain a single approved modification. In another embodiment, the document management service 105 may selectively apply each approved or rejected annotation to an electronic document in order to generate a particular version of the electronic document containing the particular changes. It may be configured to be stored as.

B. Processing Marked-Up Electronic Documents In one embodiment, document management system 100 may be used to collect and integrate document markups from different electronic documents. The document management system 100 may identify content proposals and generate annotations for the content in the reviewed electronic document. For example, a document can be used by multiple users with different document editing programs such as Microsoft Word for Windows, Microsoft Word for Mac, Open Office, Google Docs, PDF documents with comments, and any other document editing program. May be reviewed by. Each of the document editing programs may manage comments and edit them in different ways. As a result, viewing content suggestions from different editing programs within a single display can be a challenge.

The document management system 100 may receive and manage the original content from the original electronic document that has not been edited or marked up. The document management system 100 may be configured to receive and manage later received electronic documents, including the original content and markups for the original content. FIG. 4 is a flow diagram showing an approach of generating annotations based on suggestions received from a markup electronic document and displaying the document with the generated annotations. Process 400 may be executed by a single program or a plurality of programs. The operation of the processing as shown in FIG. 4 may be implemented by using the processor executable instruction stored in the computer memory. For the purpose of providing a clearer example, the operation of FIG. 4 is described as being performed by the service of the document management system 100. For clarity purposes, process 400 is described in terms of a single entity.

At block 402, the document management system 100 may receive an electronic markup document containing the original content of the review and electronic markup provided by the first user. In one embodiment, the content receiving service 110 may receive the electronic markup document from a reviewer such as a first user. Electronic markup documents can be uploaded directly to a designated website, from an email sent to a specific email account associated with the document management system 100, or from Box, Google Drive, Drop Box or any other. It may be received from a network location associated with a network storage service, such as a network storage service.

In one embodiment, the document management service 105 may associate a particular user with a received electronic markup document by identifying the particular identifier associated with the particular user. When the electronic markup document is emailed to the document management system 100, the particular identifier may represent the user's email address. In another example, a particular user may edit an electronic document with a particular file name associated with the particular user. For example, the user Jane Doe may edit and send the electronic markup document name "document1_userJaneDoe.doc". In yet another example, the electronic document sent to the review user may include user code embedded in the document metadata or in the file structure of the document. The embedded user code may then be extracted by the document management service 105 to identify the review user.

At block 404, the document management system 100 may identify one or more markups that represent content proposals proposed by the first user. In one embodiment, the document management service 105 is in electronic markup by first identifying the original content in the electronic markup document by comparing the electronic markup document with the original content from the original electronic document. Markups may be identified. The document management service 105 may identify markup in an electronic markup document by identifying content that is separate from the original content that was identified.

In one embodiment, the document management service 105 may use the markup recognition service 132 to analyze each of the identified markups to determine the meaning of the markup. The markup recognition service 132 may be configured to identify objects in various electronic document formats that present inserted comments and tracked changes. For example, an electronic markup document in Microsoft Word format may include a data object that presents a comment and an object that is edited and presents characters as a tracked change.

In one embodiment, the markup recognition service 132 may be configured to identify in-line character changes that may include comments or edits. In-line text changes may present edits that are not specified within the data object that presents comments or tracked changes. For example, the first user may insert a comment in a line formatted to include an opening brace before the comment and a closing brace after the comment. The markup recognition service 132 may implement a trained machine learning model to recognize comments within a line based on the special characters used to mark the comments. In another embodiment, the markup recognition service 132 may be configured to identify unmarked edits by comparing the content from the electronic markup document with the content of the original electronic document. Modified content, such as added or deleted characters, may be marked as a content suggestion.

At decision block 406, the document management system 100 may determine if there is any remaining markup to analyze in order to generate annotations. In one embodiment, the document management service 105 may be configured to iteratively analyze each of the identified markups. If the document management service 105 determines that there is remaining markup to analyze, the document management service 105 may proceed to block 408. However, if there is no remaining markup to analyze, the document management service 105 may proceed to block 412.

At block 408, the document management system 100 identifies a document portion corresponding to a particular markup from the original electronic document. In one embodiment, the document management system 100 implements one or more machine learning models to identify the document type of the original electronic document and to sections, phrases, specific keywords, and any other content-specific parts. The corresponding particular document portion within the original electronic document may be identified and the correlation between the particular markup and the particular document portion of the original electronic document may be determined.

In one embodiment, the content recognition service 131 may analyze the original electronic document from the document to determine the document type. Each document type may have its own term, phrase, and format. These may be used to help determine how a particular markup is interpreted and correlates with the document portion. In one embodiment, the content recognition service 131 may implement a machine learning model configured to determine the document type of the electronic document and the specific document portion of the original electronic document. The machine learning model may receive the original content from the electronic document as input and may output the document type of the electronic document and the identified document portion from the original content.

Once the document portion within the original electronic document has been determined, the document management system 100 may determine the correlation between the particular markup and the document portion. In one embodiment, the content correlation service 134 may identify a match between the electronic markup and the document portion of the original electronic document.

At block 410, the document management system 100 may generate an annotation about the particular markup, including the markup and the first user ID of the user who generated the markup. The annotation may be expressed as a visibility comment, such as a callout comment containing a transcription of the markup, or a proposed edit that can be represented as a change tracked by a callout change that describes the proposed change. For example, if the markup type is determined to be a comment, the annotation may be expressed as a visibility comment. However, if the markup is a proposed edit, the annotation is expressed in the balloon comment as a tracked change and / or a proposed edit based on the confidence score determined by the markup recognition service 132. Good.

FIG. 5 shows an exemplary embodiment of an electronic document displayed with annotations representing the identified markup. Document 505 may represent an electronic document reviewed during a document review meeting. The annotation 510 may represent a comment-type annotation corresponding to the document portion corresponding to the image in the document 505. Annotation 510 may include the text "Change the picture to the alternative" and may include a link from the electronic markup document to another photo provided in the electronic comment. It's fine. Note 510 displays a first user ID such as "Jane Doe".

After generating the annotation, the document management system 100 may return to decision block 406. At decision block 406, the document management system 100 may determine if there is any remaining markup to analyze in order to generate annotations. If there is remaining markup to analyze, the document management system 100 may proceed to blocks 408 and 410 to identify the document portion corresponding to the current markup and generate annotations about the current markup.

In one embodiment, blocks 402-410 may be repeated for each electronic document received by the content receiving service 110. For example, after a document review meeting, multiple reviewers may submit their proposed markup to the content receiving service 110 in the form of an electronic document. The document management service 105 may manage each of the received electronic documents by iteratively processing each of the electronic documents when they are received.

Once all of the identified markup has been analyzed in decision block 406, the document management system 100 proceeds to block 412 and displays the original electronic document in electronic form in the display window with the generated annotations. You can do it. In one embodiment, the document management service 105 communicates with the document presentation service 150 to generate a graphical user interface that displays the original content with one or more generated annotations in a display window on the client computing device. You can do it. The graphical user interface identifies, but is not limited to, the original content of the electronic document, the generated annotations associated with the document portion of the particular document displayed, and the applied proposed edits associated with the annotations. It may include overlaying and displaying the display of the original content of the document. In one embodiment, the document management service 105 may store each copy of one or more reviewed documents in the data repository 170.

C. Processing Media Content Items, Including Proposals Many professional and academic institutions may hold document review meetings to discuss and propose changes to one or more documents. Document review meetings may include formal meetings between reviewers, ad hoc meetings between reviewers, telephone and video conferencing between reviewers. For example, if reviewer Jane Doe initiates a phone call with John Smith to discuss Document A, the phone call may be considered as a document review meeting. During the document review meeting, the reviewer may verbally discuss changes or suggestions for changes to the document. Reviewers may generate notes on whiteboards, blackboards, interactive projectors, and other interactive devices. Document proposals may be stored by capturing the proposal as a media content item, whether made orally or in writing on a whiteboard or other device. Media content items include, but are not limited to, recorded audio from document review meetings, recorded videos from document review meetings, photos of notes from document review meetings, and projectors, interactive whiteboard appliances (Interactive Whiteboard Appliance:). It may represent a variety of different types of media content, including IWB), or screenshots captured from any other media device.

The IWB may be configured to capture user input on the IWB display in the form of a series of stylus pen inputs. For example, when a stylus pen is used to generate a mark on an IWB display, the IWB may generate a set of data objects. Here, each object includes a time stamp and (x, y) coordinates corresponding to the position on the IWB display where the IWB has detected the stylus pen. The set of data objects may correspond to the path of the stylus pen stroke. In another example, the path of the stylus pen stroke may be represented by a Bezier curve point. Bezier curves represent parametric curves used in computer graphics to model paths. In one embodiment, the set of data objects corresponding to the generated marks may be packaged in a media content item that may also include a screenshot of the IWB display along with the coordinates of the screenshot. For example, a media content item representing captured content from an IWB is a set of screenshots of an IWB display with corresponding pixel coordinates in the screenshot and corresponding marks generated from an input device such as a stylus pen. The object may be included.

In one embodiment, the document management system 100 is used to assist in capturing proposals at a document review meeting by receiving media content items that include content proposals for one or more documents under review. You can. For example, a document review meeting may be recorded using an audio recorder or video decoder. The record may represent a media content item for a review meeting that may be received by the document management system 100. The document management system 100 may then analyze the received media content item to determine one or more content suggestions for the document under review. One or more content suggestions may be used to generate annotations and associate the generated annotations with the document portion of the document under review.

FIG. 6 is a flow diagram showing an approach of generating annotations based on proposed changes from a media content item and generating an updated electronic document containing the corresponding proposed changes. Process 600 may be executed by a single program or a plurality of programs. The operation of the processing as shown in FIG. 6 may be implemented by using the processor executable instruction stored in the computer memory. For the purpose of providing a clearer example, the operation of FIG. 6 is described as being performed by the service of the document management system 100. For clarity purposes, process 600 is described in terms of a single entity.

At block 602, the document management system 100 may receive one or more media content items that represent content captured from the discussion of one or more electronic documents by one or more users. In one embodiment, the content receiving service 110 may receive media content items containing captured content from a document review meeting of one or more documents. For example, a media content item may include recorded audio, video, and captured screenshots from a document review meeting. The captured screenshot is a digital image captured by a digital camera, or a PDF (Portable Document File), PNG (Portable Network Graphics) file, PEG file, or a memo or markup image generated during a document review meeting. It may represent a digital image file like any other file that may contain. For example, the reviewer may use an IWB device to generate notes and other document markups for the document displayed within the IWB display screen. The IWB may generate a PDF copy of the document under review, including the written notes generated by the stylus on the IWB. At the end of the meeting, the IWB may upload the generated PDF file to the content receiving service 110.

At block 604, the document management system 100 may identify from one or more media content items a portion of media content that represents a document portion within one or more documents. The media content portion may represent a subset of media content items, such as an audio clip, a video clip, a copy of an image, or a subset of any other media content item. In one embodiment, the document management system 100 may use the conversation recognition service 133 to analyze each of the media content items to identify the media content portion that represents the document portion in the reviewed document. For example, conversation recognition service 133 may identify the user's audio clip speaking about editing a particular word, phrase, or chapter in the document being reviewed.

In one embodiment, the conversation recognition service 133 may implement a machine learning model configured to identify media content portions that indicate content suggestions for either comments or proposed edits. The machine learning model may receive media content items, including audio, as input. The machine learning model may analyze the detected conversation in the media content item and convert the conversation into text. The letters may then be analyzed for specific words or phrases that may indicate the proposed edits of the reviewed document. For example, remarks such as "the word ... is duplicated", "we need more examples", "the introduction is too short", "the word ... is misspelled", "could you update figure X" It may be shown that a particular media content portion is relevant to a suggestion or edit of a document. The machine learning model may then assign content suggestion types, such as comments or suggested edits, based on the translated characters of the conversation. The machine learning model may also identify statements that may indicate a position in the document, such as "on the second page," "the final paragraph," "the sentence starting with," and so on. The output of the machine learning model may include a set of media content parts identified as possible proposed edits for the reviewed document.

In one embodiment, the machine learning model may be further configured to analyze the conversation to determine speaker identification information. Machine learning models may be trained with employee conversation clips to determine speaker identification information. For example, a machine learning model may analyze frequency, pitch, and other audio qualities to determine if the audio clip of a conversation matches the voice quality of a known employee or other participant.

In one embodiment, if the received media content item contains media content that includes an image file that includes a video, screenshot, or markup image, the document management system 100 proposes using the markup recognition service 132. The mark corresponding to the edited edit may be identified. The markup recognition service 132 may analyze a mark, such as a handwritten mark, to determine the meaning of the mark. As mentioned above, the markup recognition service 132 may implement a machine learning model to determine the meaning of the mark and the mark from the media content item. In the video example, the document management system 100 uses both the markup recognition service 132 and the conversation recognition service 133 to make proposed edits from the marks captured by the video and the media content portion captured by the audio of the video. May be identified.

With reference to FIG. 6, in the determination block 606, the document management system 100 may determine whether or not there is a media content portion to be analyzed. If there is a media content portion to be analyzed to generate the annotation, the document management system 100 may select the media content portion from the media content portion and proceed to block 608. However, if there is no additional media content portion to be analyzed, the document management system 100 may proceed to block 612 to generate one or more updated electronic documents containing the generated annotations.

At block 608, the document management system 100 may identify the document portion corresponding to the media content portion from the electronic document portion of one or more electronic documents. In one embodiment, the document management system 100 implements one or more machine learning models to identify each document type of one or more documents, sections, phrases, specific keywords, and any other content. A particular document portion within one or more documents that corresponds to the unique portion may be identified and the correlation between the media content portion and the document portion may be determined.

In one embodiment, the document management service 105 reviews which documents have been reviewed based on the document file name selected by the user, such as the organizer of the document review meeting, or by analyzing received media content items. It may be determined by identifying the file name of the document. In one embodiment, the content recognition service 131 may analyze the content of one or more reviewed documents to determine the document type and the document portion of each reviewed document. The content recognition service 131 may implement a machine learning model configured to determine the document type of each document using the content of the document as input. Machine learning models may be configured to identify document parts from a document that can correlate content suggestions from media content items to their position in the document.

In one embodiment, the document management service 105 determines the correlation between the media content portion and the document portion identified from one or more electronic documents between the document portion and the media content portion, the content correlation service 134. May be sent to. The media content portion transmitted to the content correlation service 134 may be in the form of a data object that includes the converted characters from the conversation recognition service 133 and the media content portion in its original format. The document part is the electronic document associated with each document part, the document type associated with the electronic document, and any other that can be used to assist in finding a match between the media content part and the document part. It may include additional characteristic data such as metadata that defines document-specific information.

In one embodiment, the content correlation service 134 may use a machine learning model configured to determine the match between the transformed characters and the document portion. For example, if the translated text of the media content part contains "Please reword the first sentence of the abstract", the machine learning model is the first sentence of the abstract part of the electronic document. It may be determined whether or not there is a matching document portion corresponding to. The machine learning model may identify keywords within the media content portion, such as "abstract," and determine that the abstract keywords refer to chapters within the patent application document type. The machine learning model may then use summary keywords to filter document parts based on matching document types. In addition, if multiple document types have a summary chapter, the machine learning model may analyze each document portion associated with a matching document type. The content correlation service 134 may transmit a reference to the media content portion and a matching document portion to the document management service 105. The corresponding document part may be a reference to the document part and / or a positional position in the electronic document corresponding to the matching document part.

At block 610, the document management system 100 may generate annotations representing the media content portion and associate the annotation with the document portion corresponding to the media content portion. In one embodiment, the annotation generation service 120 includes: character transcription of the media content portion, confidence score associated with the character transcription, annotation type specifying whether the annotation is a comment or a proposed edit, a link to the media content portion. Annotations may be generated for the media content portion containing any of. The annotation type may be based on the proposed type determined for the media content portion. For example, if the proposal type is a comment, the generated annotation may be a balloon comment. However, if the proposal type is the proposed edit, the annotation may be displayed as the proposed edit.

FIG. 7 shows an exemplary embodiment of an electronic document displayed with annotations representing identified media content portions. Document 705 may represent a document reviewed during a document review meeting. Annotation 710 may represent a comment-type annotation corresponding to the document portion "AI Ricoh Interactive Assistance Service" in document 705. Note 710 is a media content item that includes the text transcription of "Check with marketing if RAIS should be demonstrated this year" and the media content portion that references the text transcription. May include a link to. In one embodiment, the link to the media content item may, when selected by the user, initiate a queued media player program at the location of the media content portion. For example, when the user selects a link in annotation 710, the media player program may be launched and queued to the media content portion.

Annotation 715 represents an editorial annotation proposed to replace the word "exciting" with the word "amazing". The transcription may display “Replace:“ exciting ”with“ amazing ”. If the associated confidence score is higher than the specific confidence score threshold, Note 715 is proposed. The edits may be applied to the document portion within the document 705, as shown in FIG.

In one embodiment, the annotation generation service 120 may update the annotation with conference-specific information that can be used to identify when the proposal was made and / or discussed. For example, the annotation generation service 120 may update the annotations to include details of the document review meeting, such as meeting name, meeting date and time, meeting location, and meeting attendees. Each of the conference characteristics may be used to filter the annotations that can be displayed within the graphical display of the document.

In one embodiment, the conversation recognition service 133 may associate a particular user with the identified media content portion when the user's voice is identified by the conversation recognition service 133 when analyzing the media content item. The user identification information may be based on the history and / or training media content item used to train the machine learning model within the conversation recognition service 133. When the conversation recognition service 133 associates a particular user with a conversation in the media content portion, the annotation generation service 120 may update the generated annotation to include the user ID of the user identified as the speaker. However, if the user is not identified, the annotation generation service 120 updates the generated annotation to include the global user ID, the company ID, or the department ID of the particular company and / or the department that held the document review meeting. Good.

In one embodiment, a user, such as a document owner or document reviewer, may update the annotation characteristics to assign a user ID to the annotation based on the media content portion when reviewing the annotation. For example, if James Smith is the user who made the proposal during the meeting, user Jane Doe may update annotation 710 to assign James Smith's user ID to annotation 710. In another embodiment, if multiple users engage in discussions related to annotation 710, user Jane Doe may update annotation 710 to include each of the users who participated in the discussion.

Upon generating the annotation, the document management system 100 may proceed to decision block 606 to determine if there is an additional media content portion that needs to be analyzed. If there is an additional media content portion to be analyzed, the document management system 100 may repeat blocks 608 and 610 to generate annotations for each of the media content portions. If there is no additional media content portion to be analyzed in decision block 606, document management system 100 may proceed to block 612.

At block 612, the document management system 100 may display one or more documents in a display window in electronic form, along with the generated annotations. In one embodiment, the document management service 105 communicates with the document presentation service 150 to generate a graphical user interface that displays the original content with one or more generated annotations in a display window on the client computing device. You can do it. The graphical user interface provides, but is not limited to, the original content of the document, the generated annotations associated with the document portion of the particular document displayed, and the applied proposed edits associated with the annotations. It may include displaying on top of the display of the original content of.

In one embodiment, the document management service 105 may store each copy of one or more reviewed documents in the data repository 170. A stored copy of a document may contain each of the generated annotations associated with each corresponding document.

D. Updating Annotations with Additional Proposals In one embodiment, the document management system 100 proposes proposals from media content items, electronic documents, and physical documents, including markups that reference document parts of the original content that already have existing annotations. You may receive it. For example, the specific document portion may represent the first sentence of the introductory portion, and the document management system 100 may previously associate a comment, including a comment, with the specific document portion. The document management system 100 may then receive a media content item containing another proposed comment about the same document portion that refers to the first sentence of the introductory portion. The document management system 100 may determine whether a comment from a media content item is related to an existing annotation. If relevant, the document management system 100 may update the annotations to include comments from media content items.

As described with FIG. 4, the document management system 100 receives one or more marked-up electronic documents and generates annotations based on the suggestions received from the one or more marked-up electronic documents. You can. The document management system 100 may analyze one or more markup electronic documents and generate annotations for each of the proposed markups in the one or more markup electronic documents. The document management system 100 may store the generated annotations in the data repository 170 as part of an updated electronic document. In one embodiment, the generated annotations may be stored as separate annotation objects in the data repository 170.

FIG. 8 is a flow diagram showing an approach of updating an existing annotation based on a proposed change from a media content item and generating an updated electronic document containing the corresponding proposed change. Process 800 may be executed by a single program or a plurality of programs. The operation of the processing as shown in FIG. 8 may be implemented by using the processor executable instruction stored in the computer memory. For the purpose of providing a clearer example, the operation of FIG. 8 is described as being performed by the service of the document management system 100. For clarity purposes, process 800 is described in terms of a single entity.

At block 802, the document management system 100 may receive one or more media content items that represent content captured from discussions of electronic documents by one or more users. In one embodiment, the content receiving service 110 may receive a media content item that represents a record of a document review meeting of electronic documents. Examples of media content items include recorded audio, video, and captured screenshots from document review meetings. The content receiving service 110 can be uploaded directly to a designated website, from an email sent to a specific email account associated with the document management system 100, or from Box, Google Drive, Drop Box or any other. Media content items may be received from network locations associated with network storage services, such as network storage services.

In another embodiment, the content receiving service 110 may receive a file other than the media content item, including the proposed changes to the electronic document. For example, the content receiving service 110 may include additional marked-up electronic documents, scanned versions of marked-up physical documents, and any other type that may include markup corresponding to proposed changes to the electronic documents. You may receive the file.

At block 804, the document management system 100 may identify the media content portion corresponding to the content proposal for the electronic document from the received media content item. The media content portion may represent a subset of media content items, such as an audio clip, a video clip, a copy of an image, or a subset of any other media content item. In one embodiment, the document management system 100 may use the conversation recognition service 133 to analyze each of the media content items to identify the media content portion that represents the document portion in the reviewed document. For example, conversation recognition service 133 may identify the user's audio clip speaking about editing a particular word, phrase, or chapter in the document being reviewed.

In one embodiment, the conversation recognition service 133 may implement a machine learning model configured to identify a media content portion that indicates a proposed change to an electronic document. The machine learning model may receive media content items, including audio, as input. The machine learning model may analyze the detected conversation in the media content item and convert the conversation into text. The letters may then be analyzed for specific words or phrases that may indicate content suggestions for the reviewed document. The machine learning model may also identify statements that may indicate a position in the document, such as "on the second page," "the final paragraph," "the sentence starting with," and so on. The output of the machine learning model may include a set of media content parts identified as possible proposed edits for the reviewed document.

In one embodiment, the machine learning model implemented by the conversation recognition service 133 may be configured to assign a confidence score to the character transcription of a set of identified media content portions. Here, the confidence score represents the value of the accuracy of the conversion. Determining the confidence score depends on the clarity of the conversation within the media content portion and how well the machine learning model can match the translated words and phrases into the spoken conversation within the media content portion. May be based.

In one embodiment, if the received media content item contains media content that includes an image file that includes a video, screenshot, or markup image, the document management system 100 proposes using the markup recognition service 132. The mark corresponding to the edited edit may be identified. The markup recognition service 132 may analyze a mark, such as a handwritten mark, to determine the meaning of the mark. As mentioned above, the markup recognition service 132 may implement a machine learning model to determine the meaning of the mark and the mark from the media content item. In the video example, the document management system 100 uses both the markup recognition service 132 and the conversation recognition service 133 to make proposed edits from the marks captured by the video and the media content portion captured by the audio of the video. May be identified. In another embodiment, if the received file contains a marked-up electronic document or a marked-up physical document, the document management system 100 uses the markup recognition service 132 to mark the proposed edits. May be identified.

In decision block 806, the document management system 100 may determine if there is a media content portion to be analyzed. If there is a media content portion to be analyzed to generate the annotation, the document management system 100 selects the media content portion from the media content portion, proceeds to block 808, and analyzes the selected media content portion. Good. However, if there is no additional media content portion to be analyzed, the document management system 100 may proceed to block 814 to generate an updated electronic document containing the updated annotations.

At block 808, the document management system 100 may identify the document portion corresponding to the media content portion from the document portion of the electronic document. In one embodiment, the document management system 100 implements one or more machine learning models to identify the document type of a document and correspond to sections, phrases, specific keywords, and any other content-specific parts. A particular document portion within a document may be identified and the correlation between the media content portion and the document portion may be determined.

In one embodiment, similar to block 6608 of FIG. 6, the document management system 100 may identify the document portion corresponding to the media content portion from the document portion of the electronic document. In one embodiment, the document management system 100 implements one or more machine learning models to identify each document type of one or more documents, sections, phrases, specific keywords, and any other content. A particular document portion within one or more documents that corresponds to the unique portion may be identified and the correlation between the media content portion and the document portion may be determined.

In one embodiment, the document management service 105 transmits the media content portion and the document portion identified from the electronic document to the content correlation service 134 to determine the correlation between the document portion and the media content portion. Good. The content correlation service 134 may use a machine learning model configured to determine the match between the transformed characters and the document portion. The content correlation service 134 may transmit a reference to the media content portion and a matching document portion to the document management service 105. The corresponding document part may be a reference to the document part and / or a positional position in the electronic document corresponding to the matching document part.

At block 810, the document management system 100 may determine that the document portion identified at block 808 is associated with an existing annotation. In one embodiment, the document management service 105 may access the existing stored annotations from the data repository 170 to determine whether the existing annotations are associated with the identified document portion. For example, if the identified document portion corresponds to the term "AI Ricoh Interactive Assistance Service" in the first paragraph of the document "exampleDoc.doc", then the document management service will address the existing stored annotations from the data repository 170. You may access it to determine if there are any existing annotations about the "AI Ricoh Interactive Assistance Service". If an existing annotation exists in the data repository 170, the document management service 105 may identify the annotation and proceed to block 812 to update the existing annotation.

In one embodiment, the document management service 105 analyzes the document portion associated with an existing annotation to determine whether the associated document portion of the existing annotation corresponds to the identified document portion of the media content portion. You may decide. As mentioned above, the document portion may represent words, phrases, sentences, images, paragraphs, and chapters. If the document portion associated with an existing annotation corresponds to a particular paragraph and the identified document portion of the media content portion corresponds to a word or phrase within the particular paragraph, the document management service 105 has overlapped the document portion. However, it may be determined that the two document parts represent separate parts of the document. The document management service 105 may then send a request to the annotation generation service 120 to generate a separate annotation for the corresponding media content portion. In one embodiment, the document management service 105 is configured to determine whether the amount of overlap between two document parts exceeds a threshold in order to determine that the document parts correspond to the same part. Good. For example, the threshold may be set to 60% overlap. Here, the two document parts must overlap by 60% or more in order to be regarded as the same document part. In the example above, the document part associated with the existing annotation corresponds to the complete paragraph, the document part associated with the media content part corresponds to the sentence within the paragraph, and the sentence constitutes less than 60% of the paragraph. In that case, the document management service 105 may determine that the threshold is not met and the existing annotation is not considered to overlap the media content portion. In one embodiment, the threshold for determining the overlap of document parts may be set to any overlap value between 0 and 100.

At block 812, the document management system 100 updates existing annotations based on the media content portion. In one embodiment, the document management service 105 may update existing annotations based on the media content portion. The update may include adding to the annotation object one of the following: a text transcription of the media content portion, a confidence score associated with the text transcription, and a link to the media content portion.

In one embodiment, the document management service 105 may determine whether the existing annotation type and the determined proposal type for the media content portion are the same. If the existing annotation type and the proposed type of the media content portion are the same, the document management service 105 may send a request to the annotation generation service 120 to update the existing annotation. However, if the existing annotation type and the proposed type of the media content portion are not the same, the document management service 105 may send a request to the annotation generation service 120 to generate a new annotation for the media content portion. The document management service 105 may ensure that the different proposal types are represented by unique annotations in order to clearly identify the different types of proposals.

In one embodiment, the document management service 105 may determine whether the user ID associated with the existing annotation type is the same as the user ID associated with the media content portion. If the user ID associated with the existing annotation type is the same as the user ID associated with the media content portion, the document management service 105 sends a request to the annotation generation service 120 to update the existing annotation. Good. However, if the user ID associated with the existing annotation type is not the same as the user ID associated with the media content portion, the document management service 105 will generate a new annotation for the media content portion in the annotation generation service 120. You may send a request to do so. The document management service 105 may request the generation of unique annotations for each user ID for the purpose of clearly displaying each user's suggestions. In addition, when displaying annotations in the display window, the user may filter the annotations based on the user ID. Therefore, having a unique user ID for each annotation may allow filtering by the user.

In another embodiment, if the user ID of the existing annotation and the user ID of the media content portion are different, the document management service 105 causes the annotation generation service 120 to update the existing annotation to include the media content portion. You may request and specify that the media content portion is associated with a different user ID. Annotations with content from multiple user IDs may be so marked or shown in the display window. Referring to FIG. 7, annotation 720 is an exemplary annotation that includes content from a plurality of users. Note 720 is associated with the document portion corresponding to the term "IFPD's". Note 720 includes two comments. The first comment is a conversation identified as a "Ricoh Reviewer" corresponding to a Ricoh employee attending a document review meeting and captured by a media content item. The second comment comes from an electronic document identified and marked up as "Jane Doe".

At block 814, the document management system 100 may display, in electronic form, an electronic document with updated annotations, existing annotations, and newly generated annotations in a display window. In one embodiment, the document management service 105 communicates with the document presentation service 150 to display an electronic document with one or more updated generated annotations in a display window on a client computing device. You may generate an interface. The graphical user interface may include, but is not limited to, displaying the original content of the electronic document, the associated annotations, and the markup associated with the annotations overlaid on the display of the original content.

With reference to FIG. 7, the reviewer drop-down list 730 may present a drop-down list of available reviewers who have provided suggestions for electronic documents. If a particular reviewer, such as Jane Doe, is selected from the reviewer drop-down list 730, the graphical user interface may display the markup and annotations associated with the reviewer Jane Doe. In one embodiment, the document presentation service 150 may be configured to display a particular annotation associated with the plurality of users when one of the plurality of users is selected from the reviewer drop-down list 730. For example, if reviewer Jane Doe is selected from the reviewer drop-down list 730, document presentation service 150 may display annotation 720 even if annotation 720 includes suggestions from Jane Doe and Ricoh Reviewer.

In another embodiment, the document presentation service 150 may be configured to display only annotation comments from a particular user selected from the reviewer drop-down list 730. For example, when Jane Doe is selected from the reviewer drop-down list 730, the document presentation service 150 includes "What does IFPD stand for? Maybe expand?" Only comments associated with Jane Doe may be displayed. In yet another embodiment, the document presentation service 150 may be configured to display annotation comments and public user comments from selected particular users. Public user comments may include comments associated with a general user ID, such as "Ricoh Reviewer," or any other group user ID configured to be published.

E. Generation of Third Party Annotation Request The electronic document under review may be managed by a system other than the document management system 100. For example, electronic documents may be managed by an external document management system, such as Google's G-Suite or any commercially available or privately managed document management system. The document management system 100 may be configured to identify content proposals from various inputs, determine the type of proposal, and generate annotation generation requests that may be sent to an external document management system that manages the electronic document under review. ..

FIG. 9 is a flow diagram showing an approach of generating an annotation request based on a proposal received from the content input. Process 900 may be executed by a single program or a plurality of programs. The operation of the processing as shown in FIG. 9 may be implemented by using the processor executable instruction stored in the computer memory. For the purpose of providing a clearer example, the operation of FIG. 9 is described as being performed by the service of the document management system 100. For clarity purposes, process 900 is described in terms of a single entity.

At block 902, the document management system 100 may receive content input representing the proposed content for the content in the electronic document managed by the third party management system. In one embodiment, the content receiving service 110 may receive content input, including suggestions for content in an electronic document. Content input may represent any type of media, including suggestions for content within an electronic document, including, but not limited to, marked-up physical documents, marked-up electronic documents, audio, video, or one or more. Includes media content items in the form of screenshots of, or any other file that contains proposed changes to the content. Content receiving service 110 may receive content input by direct upload to a designated website, a link to a network location associated with a network storage service, or any other method.

In one embodiment, the document management service 105 may determine the content of the electronic document from the received content input. For example, if the received content input is a marked-up electronic document, the document management service 105 determines the original content of the electronic document based on characters that do not contain markups, comments, or tracked changes. Good. In another embodiment, the document management service 105 may determine the original content of the electronic document by communicating with a third party document management system and reading the original version of the electronic document. For example, the document management service 105 may determine the name of the original electronic document from the file name given to the content input. In another example, the document management service 105 may send a request for the original electronic document to a third party document management system. The third party document management system may respond by copying the original electronic document or by linking to download the original electronic document.

In the example where the content input represents a content media item, the document management service 105 may extract the name of the original electronic document from the captured content in the content input. For example, if the content input is an audio file, the document management service 105 may analyze the audio content for spoken words that may represent the filename of the electronic document under review. The document management service 105 may then send a request for the original electronic document to a third party document management system.

At block 904, the document management system 100 may identify one or more content proposals that represent the proposed changes to the electronic document. In one embodiment, when the content input is a marked-up electronic document, the document management service 105 first compares the electronic markup document with the original content from the original electronic document that read the electronic markup document. Content proposals may be identified as markups in the marked-up electronic document by identifying the original content in the document. The document management service 105 may identify markup in an electronic markup document by identifying content that is separate from the original content that was identified.

In one embodiment, the document management service 105 may use the markup recognition service 132 to analyze each of the identified markups to determine the meaning of the markup. The markup recognition service 132 may implement a machine learning model to identify the meaning of each markup.

In one embodiment, when the content input is a media content item, the document management service 105 may identify the content proposal that refers to the document portion in the electronic document from the content input. Content suggestions may represent parts of media content, such as audio clips, video clips, or crops of screenshots that reference parts of an electronic document. In one embodiment, the document management service 105 may use the conversation recognition service 133 to analyze the content input to the content proposal representing the document portion of the reviewed document. The conversation recognition service 133 may implement a machine learning model configured to identify content suggestions that indicate proposed changes to the electronic document.

In one embodiment, the document management service 105 may determine the type of proposal from the meaning of the identified content proposal. For example, when the content input is a marked-up electronic document, the document management service 105 may use the markup recognition service 132 to determine the proposal type based on the meaning of the identified markup. In another example, when the content input is a content item, the document management service 105 uses the conversation recognition service 133 to determine the converted characters for the content proposal and to determine the proposal type based on the meaning of the converted characters. You may decide.

In one embodiment, the document management service 105 may determine the user ID of the user associated with the identified content proposal. For example, if the content input is a marked-up electronic document, the document management service 105 may provide metadata associated with the marked-up electronic document, such as the creator characteristics of the electronic document, or the marked-up electronic document. The user may be determined from the user name associated with the specific comment embedded in. When the content input is a media content item, the document management service 105 may use the conversation recognition service 133 to determine the user associated with the conversation corresponding to the particular content proposal.

Once one or more content proposals have been identified, the document management system 100 may iteratively analyze each content proposal to determine the corresponding document portion from the original content of the electronic document. At decision block 906, the document management system 100 may determine if there are any remaining content suggestions for analysis. If there are remaining content proposals to be analyzed, the document management system 100 may select the content proposals and proceed to block 908 to analyze the selected content proposals. However, if there are no further content proposals to be analyzed, the document management system 100 may proceed to block 912 and send one or more generated document proposal requests to a third party management system.

At block 908, the document management system 100 may identify the document portion corresponding to the selected content proposal from the document portion of the electronic document. In one embodiment, the document management system 100 implements one or more machine learning models to identify the document type of an electronic document and correspond to sections, phrases, specific keywords, and any other content-specific parts. , A particular document portion within an electronic document may be identified and the correlation between the media content portion and the document portion may be determined.

Once the document portion within the original electronic document has been determined, the document management system 100 may determine the correlation between the selected content proposal and the document portion. In one embodiment, the content correlation service 134 may identify a match between the selected content proposal and the document portion within the document portion identified from the document of the electronic document.

At block 910, the document management system 100 may generate a document proposal request for the selected content proposal. In one embodiment, the annotation generation service 120 provides character transcription of the content proposal, the type of proposal, the user ID determined from the content proposal, the electronic document position corresponding to the document portion of the electronic document, and the electronic link to the content input. You may generate a third-party compatible document proposal request, including. For example, if the selected content proposal is an audio clip from a comment media content item for a product described in an electronic document, the generated document proposal request is transcribed, converted from the audio clip. Proposal types such as comments determined from audio clips, user IDs that identify the user who made the proposal, document location corresponding to the document portion of the described product, and queued media content to a particular audio clip. It may include an electronic link to the item.

Upon generating the document proposal request, the document management system 100 may proceed to decision block 906. At decision block 906, the document management system 100 may determine if there are any remaining content suggestions to be analyzed. If there are remaining content proposals to be analyzed, the document management system 100 may repeat blocks 908 and 910 until all content proposals have been analyzed. If there are no remaining content suggestions to be analyzed, document management system 100 may proceed to block 912.

At block 912, the document management system 100 may send the generated document proposal request to a third party management system. In one embodiment, the document management service 105 may transmit the generated document proposal request to a third party management system via the network 160. In one embodiment, when the content input is a media content item, the document management service 105 may transmit the content input corresponding to the document proposal request together with the document proposal request. For example, if the document proposal request corresponds to an audio clip from a media content item, the document management service 105 may send the media content item along with metadata that queues the media content item to the corresponding audio clip. The third party management system may receive the document proposal request along with the corresponding media content item and generate annotations specific to the third party management system. For example, if the third-party management system is Google G-Suite, Google G-Suite may generate comments for the received document proposal request. If the request for proposal includes a media content item reference, the corresponding media content item may be stored in Google G-Suite and the generated comment may have an electronic link to the stored media content item. .. As a result, the user reviewing the electronic document can select the electronic sink and listen to the corresponding audio clip.

F. Meeting Summary Document Generation Many professional and academic institutions may hold document review meetings to discuss and propose changes to one or more documents. Document review meetings may include formal meetings between reviewers, ad hoc meetings between reviewers, telephone or video conferencing between reviewers. During a document review meeting, many different topics that may cover content from multiple different documents may be discussed. The summary meeting document may include different agendas covered during the document review meeting. For example, one such agenda may be to review figures related to multiple documents. It is useful to collect notes for each agenda, even when the agenda spans multiple documents. However, relying on one or more registrars has drawbacks as it can overlook some details of the discussion.

In one embodiment, the document management system 100 may be configured to receive one or more media content items of discussion captured during a document review meeting. The document management system 100 may analyze each of the media content items and generate a meeting summary document containing the agenda from the document review meeting.

FIG. 10 is a flow diagram showing an approach to generate a meeting summary document based on the media content items of the discussion captured during the document review meeting. Process 1000 may be executed by a single program or a plurality of programs. The operation of the processing as shown in FIG. 10 may be implemented by using the processor executable instruction stored in the computer memory. For the purpose of providing a clearer example, the operation of FIG. 10 is described as being performed by the service of the document management system 100. For clarity purposes, Process 1000 is described in terms of a single entity.

At block 1002, the document management system 100 may receive one or more media content items that represent captured content from a review meeting for one or more electronic documents. In one embodiment, the content receiving service 110 may receive media content items, including recorded audio, video, and captured screenshots of a document review meeting of one or more documents.

At block 1004, the document management system 100 may identify one or more agenda items discussed during the review meeting from one or more media content items. The agenda may represent keywords or phrases that describe the product, milestones, places, people, or even any other subject related to the document being reviewed that may be discussed during the review meeting. In one embodiment, the document management service 105 may use the conversation recognition service 133 to analyze each of the media content items to identify specific keywords and phrases that may represent the agenda. The conversation recognition service 133 may identify the agenda by implementing one or more machine learning models and analyzing media content items. Machine learning models may be trained with input datasets containing keywords and phrases that target the agenda to be identified.

In another embodiment, a user, such as a meeting organizer, may provide the document management service 105 with a set of agenda items discussed during the review meeting. For example, the meeting organizer may provide a list of scheduled agenda items for the review meeting before or after the review meeting. In another embodiment, the document management service 105 may receive a list of proposed agendas from the user and provide the list of proposed agendas, along with media content items, to the conversation recognition service 133. Good. The conversation recognition service 133 may be configured to use the proposed agenda list as a guideline and / or a starting point for identifying the agenda from the media content item.

In yet another embodiment, the document management service 105 may determine one or more electronic documents that are the subject of the review meeting and provide the one or more electronic documents to the content recognition service 131 for additional agenda. May be decided. Content recognition service 131 may analyze one or more electronic documents and determine possible agendas based on chapters, paragraphs, keywords, and titles within the electronic document.

At block 1006, the document management system 100 may identify from one or more media content items the media content portion that corresponds to the content proposal for one or more electronic documents discussed during the review meeting. In one embodiment, the document management system 100 may use the conversation recognition service 133 to analyze each of the media content items to identify the media content portion that represents the document portion in the reviewed document. For example, conversation recognition service 133 may identify the user's audio clip speaking about editing a particular word, phrase, or chapter in the document being reviewed.

In one embodiment, the conversation recognition service 133 may implement a machine learning model configured to identify media content portions that indicate content suggestions for either comments or proposed edits. The machine learning model may receive media content items, including audio, as input. The machine learning model may analyze the detected conversation in the media content item and convert the conversation into text. The letters may then be analyzed for specific words or phrases that may indicate the proposed edits of the reviewed document. For example, remarks such as "the word ... is duplicated", "we need more examples", "the introduction is too short", "the word ... is misspelled", "could you update figure X" It may be shown that a particular media content portion is relevant to a suggestion or edit of a document. The machine learning model may then assign content suggestion types, such as comments or suggested edits, based on the translated characters of the conversation. The machine learning model may also identify statements that may indicate a position in the document, such as "on the second page," "the final paragraph," "the sentence starting with," and so on. The output of the machine learning model may include a set of media content portions identified as possible content suggestions for the reviewed document.

In decision block 1008, the document management system 100 may determine if there is a media content portion to be analyzed. If there is a media content portion to be analyzed, the document management system 100 may select the media content portion from the media content portion and proceed to block 1010. However, if there is no additional media content portion to be analyzed, the document management system 100 may proceed to block 1014.

At block 1010, the document management system 100 may associate the agenda of one or more agendas with the media content portion. In one embodiment, the document management service 105 may use the conversation recognition service 133 to determine which agenda is associated with the selected media content portion. The conversation recognition service 133 may be configured to determine the association between the identified agenda and the media content portion by analyzing the keywords in the agenda and media content portion. The conversation recognition service 133 may implement a machine learning model to determine the association between the media content portion and the agenda. For example, a machine learning model may be trained to use specific keywords present within a piece of media content to determine associated agendas that may be associated with a particular keyword. In another embodiment, the conversation recognition service 133 analyzes the time stamps associated with the agenda and partial media content to determine if the media content portion corresponds to the time during the review meeting corresponding to the particular agenda. You can do it. Upon determining one or more agenda items associated with the media content portion, the document management service 105 may store in the data object of the media content portion a property that identifies the associated agenda item.

At block 1012, the document management system 100 may identify the document portion corresponding to the media content portion from one or more electronic documents. In one embodiment, the document management system 100 implements one or more machine learning models to identify each document type of one or more documents, sections, phrases, specific keywords, and any other content. A particular document portion within one or more documents that corresponds to the unique portion may be identified and the correlation between the media content portion and the document portion may be determined.

In one embodiment, the document management service 105 reviews which documents have been reviewed based on the document file name provided by the user, such as the organizer of the document review meeting, or by analyzing the received media content items. It may be determined by identifying the filename of the document that has been reviewed or by receiving a list of reviewed documents from the user. In one embodiment, the content recognition service 131 may analyze the content of one or more reviewed documents to determine the document type and the document portion of each reviewed document. The content recognition service 131 may implement a machine learning model configured to determine the document type of each document using the content of the document as input. Machine learning models may be configured to identify document parts from a document that can correlate content suggestions from media content items to their position in the document.

In one embodiment, the document management service 105 determines the correlation between the media content portion and the document portion identified from one or more electronic documents between the document portion and the media content portion, the content correlation service 134. May be sent to. The media content portion transmitted to the content correlation service 134 may be in the form of a data object that includes the converted characters from the conversation recognition service 133 and the media content portion in its original format. The document part is the electronic document associated with each document part, the document type associated with the electronic document, and any other that can be used to assist in finding a match between the media content part and the document part. It may include additional characteristic data such as metadata that defines document-specific information.

In one embodiment, the content correlation service 134 may use a machine learning model configured to determine the match between the transformed characters and the document portion. For example, if the translated text of the media content part contains "Please reword the first sentence of the abstract", the machine learning model is the first sentence of the abstract part of the electronic document. It may be determined whether or not there is a matching document portion corresponding to. The machine learning model may identify keywords within the media content portion, such as "abstract," and determine that the abstract keywords refer to chapters within the patent application document type. The content correlation service 134 may transmit a reference to the media content portion and a matching document portion to the document management service 105. The corresponding document part may be a reference to the document part and / or a positional position in the electronic document corresponding to the matching document part.

Upon identifying the document portion corresponding to the media content portion, the document management system 100 may proceed to determination block 1008 to determine if there is an additional media content portion that needs to be analyzed. If there is no additional media content portion to be analyzed, the document management system 100 may repeat blocks 1010 and 1012. In decision block 1008, if there is no additional media content portion to be analyzed, document management system 100 may proceed to block 1014.

At block 1014, the document management system 100 may aggregate the media content portions to generate a topic set for the media content portion based on their associated agenda. In one embodiment, the document management service 105 may generate a topic set for the media content portion based on the matching of the associated agenda. For example, if five media content parts are associated with the agenda "Review figures and update", the document management service 105 will use the agenda "Review figures and update". ) May be generated and the five media content parts may be included in the media content part topic set.

At block 1016, the document management system 100 may generate a conference summary document. In one embodiment, the document management service 105 may generate a meeting summary document containing a list of agenda items. FIG. 11 shows an exemplary embodiment of a display window displaying the generated conference summary document and media content item player. Display window 1105 represents a graphical user display that includes conference description 1115, first area 1110, and second area 1120. The meeting description 1115 may display general details of the review meeting, including the date and time of the meeting, meeting name, meeting location, and participants. The conference description 1115 schematically displayed in the display window 1105 displays the date of the conference.

The first zone 1110 may include a media content item player configured to play one or more media content items. The media content item player may be configured to have playback controls for rewinding, pausing, playing, fast-forwarding, and resizing buttons for resizing the playback window. The media content item player may be configured to play any type of media content item, including, but not limited to, audio files, video files, interactive video files, and any other playable media.

The second zone 1120 may include a display of a generated meeting summary document containing a list of one or more agenda items. Document topics 1130 may present configurable topic headings that may explain the types of topics listed. In one embodiment, each of the listed agenda items may be displayed as a listed item or as a selectable electronic link. Agenda 1135 presents the agenda "review figures and update". Agenda 1150 presents the agenda "Confirm product name with marketing". It is displayed as a selectable electronic link. If the user selects an electronic link for agenda 1150, the document management system 100 may generate and display a graphical user interface that includes an electronic link to the media content portion associated with agenda 1150.

In one embodiment, the media content portion associated with the agenda may be displayed in a sublist below the agenda. For example, a sublist of the three media content portions is displayed directly below Agenda 1135. In one embodiment, the associated media content portion may be displayed as a selectable electronic link. When the electronic link is selected, the selected media content portion may be regenerated in the media content item player in the first area 1110 of the display window 1105. For example, if the user selects the media content portion 1140 titled "Fig. 1-update numbers", the document management service 105 will play the media content item containing the media content portion 1140. May occur in the media content item player. The reproduction of the media content item may be queued to reproduce the portion corresponding to the media content portion 1140.

In one embodiment, the sublist displayed below each document topic may also include a list of second electronic links corresponding to the electronic document containing the document portion associated with the media content portion. For example, the sublist is the second electronic link 1145 corresponding to the electronic document "Doc-1" that includes the document portion associated with the media content portion 1140 "Fig. 1-update numbers". May be displayed. In one embodiment, the second electronic link 1145 may display a character indicating the electronic document name and a page number including the corresponding document portion. In another embodiment, when the display window 1105 displays a summary document of a meeting discussing a single document, the display text of the second electronic link in the summary document displays the page number corresponding to the document portion. Just need it.

In one embodiment, the document management service 105 may receive a selection request from the user to select the second electronic link 1145. Upon receiving the selection request for the second electronic link 1145, the document management service 105 may open the corresponding electronic document and display a document page containing the specific document portion corresponding to the second electronic link. For example, the user may select the second electronic link 1145. The second electronic link 1145 causes the document management service 105 to open "Doc-1" and page 2 of the electronic document including the document portion corresponding to "Fig. 1-update numbers". May be displayed.

In one embodiment, the document management service 105 may receive a request to play the specific media content item within the media content item player in the first area 1110. The document management service 105 may highlight a particular agenda during the playback of the particular media content item when the particular agenda is discussed during the playback of the particular media content item. The document management service 105 may be configured to identify the agenda based on the time stamp associated with each agenda. For example, the document management service 105 may play a particular media content item, and during playback, the document management service 105 matches the time stamp of the playing media content item with the time stamp associated with agenda 1135. The document management service 105 may then highlight the agenda 1135 within the second zone 1120.

In one embodiment, the document management service 105 will display a particular media content portion and / or a displayed document portion when the time stamp of the playing media content item matches the time stamp associated with the media content portion. It may be configured to highlight the electronic link of. In another example, the document management service 105 may be configured to open the corresponding electronic document and display the corresponding document portion in order to synchronize the display of the document portion with the playback of the media content item. For example, when a media content item is played and the current time stamp of the media content item corresponds to the document portion of the second electronic link 1145, the document management service 105 opens "Doc-1" and is playing the media. Page 2 of "Doc-1" may be displayed in order to display the document portion corresponding to.

Referring to FIG. 10, at block 1018, the document management system 100 may present the meeting summary document in the display window. In one embodiment, the document management service 105 may communicate with the document presentation service 150 to generate a graphical user interface that displays the conference summary document in a display window on the client computing device.

IV. Implementation Mechanism According to one embodiment of the present invention, the techniques described herein are implemented by one or more special purpose computer devices. A purpose-built computer device may be built into hardware to perform this technology, and one or more application-specific integrated circuits (ASICs) or field programmable gates that are permanently programmed to perform this technology. It may have a digital electronic device such as an array (FPGA) and may have one or more general purpose hardware processors programmed with firmware, memory, or other storage or combination according to program instructions to perform this technique. You can do it. Such purpose-built computer devices may combine custom hard-wired logic, ASICs or FPGAs with custom programming to achieve this technique. The special purpose computer device may be a desktop computer system, a portable computer system, a handheld device, a network device, or any other device incorporating hard-wired logic and / or program logic for carrying out this technique.

FIG. 12 is a block diagram showing an exemplary computer system 1200 in which embodiments of the present invention may be implemented. The computer system 1200 has a bus 1202 or another communication mechanism for communicating information, and a processor 1204 connected to the bus 1202 for processing information. The computer system 1200 also has main memory 1206, such as random access memory (RAM) or other dynamic storage element, which is connected to bus 1202 and stores information and instructions to be executed by processor 1204. The main memory 1206 may be used to store temporary variables or other intermediate information during the execution of information to be executed by processor 1204. The computer system 1200 is connected to bus 1202 and further comprises a read-only memory (ROM) 1208 or other static storage device that stores static information and instructions for processor 1204. A storage device 1210, such as a magnetic disk or optical disk, is provided and connected to bus 1202 for storing information and instructions.

The computer system 1200 may be connected via a bus 1202 to a display 1212 that displays information to the computer user, such as a cathode ray tube (CRT). Bus 1202 is illustrated as a single bus, but bus 1202 may have one or more buses. For example, bus 1202 is a control bus in which processor 1204 controls other devices in computer system 1200, an address bus in which processor 1204 specifies a memory location for instructions for execution, or data or data between components of computer system 1200. It may have any other type of bus for transferring signals, but is not limited to these.

Input device 1214 has alphanumeric characters and other keys, is connected to bus 1202, and transmits information and command selection to processor 1204. Another type of user input device is a cursor control 1216, such as a mouse, trackball or cursor direction key, which transmits direction information and command selection to processor 1204 and controls the movement of the cursor on the display 1212. This input device usually has two degrees of freedom in two axes, i.e., a first axis (eg X) and a second axis (eg Y), allowing the device to specify a position in a plane.

The computer system 1200 combines the techniques described herein with customized hardwired logic, one or more ASICs or FPGAs, a computer system to make the computer system 1200 a purpose-built machine or program firmware and / or It may be implemented using program logic or computer software. According to one embodiment of the invention, these techniques are performed by the computer system 1200 in response to processor 1204 executing one or more sequences of one or more instructions contained in main memory 1206. .. Such instructions may be read into main storage 1206 from another computer-readable medium such as storage device 1210. Execution of the instruction sequence contained in main memory 1206 causes processor 1204 to perform the processing steps described herein. In an alternative embodiment, a hard-wired circuit may be used in place of or in combination with the software instructions that implement the invention. As such, embodiments of the present invention are not limited to any particular combination of hardware circuits and software.

As used herein, the term "computer-readable medium" refers to any medium involved in providing data that causes a computer to operate in a particular way. In embodiments implemented using computer system 1200, various computer readable media are involved, for example, in providing instructions to processor 1204. Such media can take many forms, including but not limited to non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks such as storage device 1210. Volatile media include dynamic memory such as main memory 1206. Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic medium, CD-ROM, any other optical medium, RAM, PROM and EPROM, FLASH-EPROM, Includes any other memory chip or memory cartridge, or any other computer-readable medium.

Various forms of computer-readable media may be involved in transmitting one or more sequences of one or more instructions to processor 1204 for execution. For example, instructions may initially be carried on a remote computer's magnetic disk. The remote computer can load the instruction into its dynamic memory and send the instruction over the telephone line using a modem. A modem local to computer system 1200 can receive data over a telephone line and use an infrared transmitter to convert that data into an infrared signal. The infrared detector receives the data transmitted by the infrared signal and a suitable circuit can put that data on bus 1202. Bus 1202 transmits data to main memory 1206. The instructions received by the main memory 1206 that the processor 1204 reads and executes the instructions from the main memory 1206 may optionally be stored in the storage device 1210 before or after the instructions are executed by the processor 1204.

The computer system 1200 also includes a communication interface 1218 connected to bus 1202. Communication interface 1218 provides two-way data communication connected to network link 1220 connected to local network 1222. For example, the communication interface 1218 may be an ISDN (Integrated Services Digital Network) card or modem that provides a data communication connection to the corresponding type of telephone line. As another example, the communication interface 1218 may be a LAN (Local Area Network) card that provides a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, the communication interface 1218 transmits and receives electrical, electromagnetic or optical signals that carry a digital data stream that represents a variety of information.

Network link 1220 typically provides data communication to other data devices through one or more networks. For example, network link 1220 can provide a connection to a host computer 1224 or to a data device operated by an ISP (Internet Service Provider) 1226 through a local network 1222. ISP1226 also provides data communication services through a worldwide packet data communication network now commonly referred to as the "Internet" 1228. Both the local network 1222 and the Internet 1228 use electrical, electromagnetic or optical signals that carry digital data streams.

The computer system 1200 can send messages and receive data, including program code, through networks, network links 1220 and communication interfaces 1218. In the Internet example, the server 1230 may send the code requested for the application program through the Internet 1228, ISP1226, local network 1222 and communication interface 1218. The received code may be executed by processor 1204 when received by processor 1204 and / or stored in storage device 1210 or other non-volatile memory for later execution.

In the above specification, embodiments of the present invention have been described with reference to a number of specific details that may vary from embodiment to embodiment. Thus, the only indicator of the invention, and of what is intended to be an invention by the applicant, is a set of patents derived from the present application, including any future amendments in a particular form from which the claims are derived. It is the scope of claims. Therefore, limits, elements, properties, features, advantages, or attributes not explicitly stated in the claims should never limit the scope of such claims. Therefore, this specification and the accompanying drawings should be construed as exemplary rather than limiting.

100 Document management system 105 Document management service 110 Content reception service 115 User ID management service 120 Annotation generation service 130 Intellectual content recognition model 131 Content recognition service 132 Markup recognition service 133 Conversation recognition service 134 Content correlation service 140 Multi-function printing service 150 Document presentation service 160 Network 170 Data repository

Claims (20)

It ’s a device,
With one or more processors
Includes one or more memories that store instructions, said memory when processed by the one or more processors.
Steps to receive one or more media content items representing captured content from one or more electronic document review meetings, and
A step of identifying one or more agenda items discussed during the review meeting from the one or more media content items.
A step of identifying from the one or more media content items the media content portion corresponding to the content proposal for the one or more electronic documents discussed during the review meeting.
For each media content part of the media content part
The step of associating the agenda of the one or more agendas with the media content portion,
A step of identifying a document portion corresponding to the media content portion from the one or more electronic documents,
Based on the associated agenda of the media content part, the step of aggregating the media content part to generate a topic set of the media content part, and
A step of generating a meeting summary document containing a list of one or more agenda items, wherein each agenda item listed contains a corresponding topic set of media content portions.
In the display window, the step of presenting the conference summary document and
The equipment that gives rise to. The device according to claim 1, wherein the corresponding topic set of the media content portion includes a list of first electronic links corresponding to each media content portion in the corresponding topic set of the media content portion. Each first electronic link in the list of first electronic links corresponding to each media content portion corresponds to the electronic document associated with the document portion associated with the media content portion corresponding to the first electronic link. 2 The device according to claim 2, further comprising an electronic link. The one or more memories store additional instructions, and when the additional instructions are processed by the one or more processors,
A step of receiving a selection request for a specific second electronic link corresponding to a specific electronic document related to a specific document part, and
A step of opening the specific electronic document and displaying the specific document portion in response to receiving the selection request for the specific second electronic link corresponding to the specific electronic document.
When the specific electronic document is opened, the media content item player displayed in the first area of the display window is used to reproduce the specific media content item including the specific media content portion, and the reproduction is the specific. The specific media content item is queued for playback of the media content portion, and the specific media content item is associated with the specific document portion.
The device according to claim 3. The one or more memories store additional instructions, and when the additional instructions are processed by the one or more processors,
A step of receiving a selection request for a specific first electronic link corresponding to a specific media content part, and
In response to receiving the selection request for the specific first electronic link corresponding to the specific media content portion, the media content item player displayed in the first area of the display window is used to identify the specific. A step of playing a specific media content item that includes a media content portion, wherein the playback is queued to play the specific media content portion.
2. The device according to claim 2. The one or more memories store additional instructions, and when the additional instructions are processed by the one or more processors,
The step of receiving a request to play a specific media content item among the one or more media content items, and
A step of playing the specific media content item using the media content item player in the first area of the display window, and
When the specific media content portion in the specific media content item is played, the step of highlighting the agenda including the specific media content portion in the second area of the display window.
2. The device according to claim 2. The step of identifying the one or more agenda items to be discussed during the review meeting from the one or more media content items is
A step of identifying one or more agenda items based on spoken phrases and keywords in the one or more media content items using a machine learning model, wherein the machine learning model is the identified agenda item. The device of claim 1, comprising a step, trained using an input dataset of a media content item having. When one or more non-transitory computer-readable media storing instructions, the instructions are processed by one or more processors,
Steps to receive one or more media content items representing captured content from one or more electronic document review meetings, and
A step of identifying one or more agenda items discussed during the review meeting from the one or more media content items.
A step of identifying from the one or more media content items the media content portion corresponding to the content proposal for the one or more electronic documents discussed during the review meeting.
For each media content part of the media content part
The step of associating the agenda of the one or more agendas with the media content portion,
A step of identifying a document portion corresponding to the media content portion from the one or more electronic documents,
Based on the associated agenda of the media content part, the step of aggregating the media content part to generate a topic set of the media content part, and
A step of generating a meeting summary document containing a list of one or more agenda items, wherein each agenda item listed contains a corresponding topic set of media content portions.
In the display window, the step of presenting the conference summary document and
One or more non-transitory computer-readable media that give rise to. The one or more non-temporary items according to claim 8, wherein the corresponding topic set of the media content portion includes a list of first electronic links corresponding to each media content portion in the corresponding topic set of the media content portion. Computer-readable medium. Each first electronic link in the list of first electronic links corresponding to each media content portion corresponds to the electronic document associated with the document portion associated with the media content portion corresponding to the first electronic link. 2. One or more non-temporary computer-readable media according to claim 9, further comprising an electronic link. Further including additional instructions, the additional instructions are processed by the one or more processors.
A step of receiving a selection request for a specific second electronic link corresponding to a specific electronic document related to a specific document part, and
Upon receiving the selection request for the specific second electronic link corresponding to the specific electronic document, the step of opening the specific electronic document and displaying the specific document portion, and
When the specific electronic document is opened, the media content item player displayed in the first area of the display window is used to reproduce the specific media content item including the specific media content portion, and the reproduction is the specific. The specific media content item is queued for playback of the media content portion, and the specific media content item is associated with the specific document portion.
One or more non-transitory computer-readable media according to claim 10. Further including additional instructions, when the additional instructions are processed by the one or more processors,
A step of receiving a selection request for a specific first electronic link corresponding to a specific media content part, and
Upon receiving the selection request for the specific first electronic link corresponding to the specific media content portion, the media content item player displayed in the first area of the display window is used to include the specific media content portion. A step of playing a specific media content item, the playback being queued to play the particular media content portion, and
One or more non-transitory computer-readable media according to claim 9. Further including additional instructions, when the additional instructions are processed by the one or more processors,
The step of receiving a request to play a specific media content item among the one or more media content items, and
A step of playing the specific media content item using the media content item player in the first area of the display window, and
When the specific media content portion in the specific media content item is played, the step of highlighting the agenda including the specific media content portion in the second area of the display window.
One or more non-transitory computer-readable media according to claim 9. The step of identifying the one or more agenda items discussed during the review meeting from the one or more media content items is
A step of identifying one or more agenda items based on spoken phrases and keywords in the one or more media content items using a machine learning model, wherein the machine learning model is the identified agenda item. One or more non-transitory computer-readable media according to claim 8, comprising steps, trained using an input dataset of a media content item having. A method implemented by a computer
Steps to receive one or more media content items representing captured content from one or more electronic document review meetings, and
A step of identifying one or more agenda items discussed during the review meeting from the one or more media content items.
A step of identifying from the one or more media content items the media content portion corresponding to the content proposal for the one or more electronic documents discussed during the review meeting.
For each media content part of the media content part
The step of associating the agenda of the one or more agendas with the media content portion,
A step of identifying a document portion corresponding to the media content portion from the one or more electronic documents,
Based on the associated agenda of the media content part, the step of aggregating the media content part to generate a topic set of the media content part, and
A step of generating a meeting summary document containing a list of one or more agenda items, wherein each agenda item listed contains a corresponding topic set of media content portions.
In the display window, the step of presenting the conference summary document and
Methods performed by a computer, including. The computer-implemented method of claim 15, wherein the corresponding topic set of the media content portion comprises a list of first electronic links corresponding to each media content portion in the corresponding topic set of the media content portion. .. Each first electronic link in the list of first electronic links corresponding to each media content portion corresponds to the electronic document associated with the document portion associated with the media content portion corresponding to the first electronic link. 2. The method performed by the computer according to claim 16, further comprising an electronic link. A step of receiving a selection request for a specific second electronic link corresponding to a specific electronic document related to a specific document part, and
Upon receiving the selection request for the specific second electronic link corresponding to the specific electronic document, the step of opening the specific electronic document and displaying the specific document portion, and
When the specific electronic document is opened, the media content item player displayed in the first area of the display window is used to reproduce the specific media content item including the specific media content portion, and the reproduction is the specific. The specific media content item is queued for playback of the media content portion, and the specific media content item is associated with the specific document portion.
The method implemented by the computer according to claim 17, further comprising. A step of receiving a selection request for a specific first electronic link corresponding to a specific media content part, and
Upon receiving the selection request for the specific first electronic link corresponding to the specific media content portion, the media content item player displayed in the first area of the display window is used to include the specific media content portion. 16. The method performed by a computer according to claim 16, wherein the step of replaying the specific media content item, wherein the replay is queued to replay the particular media content portion. The step of receiving a request to play a specific media content item among the one or more media content items, and
A step of playing the specific media content item using the media content item player in the first area of the display window, and
When the specific media content portion in the specific media content item is played, the step of highlighting the agenda including the specific media content portion in the second area of the display window.
16. The method performed by a computer according to claim 16.

Images