KR20220133894A - Systems and methods for analysis and determination of relationships from various data sources - Google Patents

Abstract

The present invention relates to computer-implemented systems and methods for analyzing data from a variety of data sources. Embodiments of systems and methods further provide for generating a response to specific questions based on the analyzed data, the generating comprising: retrieving related documents associated with the analyzed data; determine from which of the retrieved related documents which information should be reported; and providing a response based on the graph schema and decisions associated with the relevant documents.

Description

Systems and methods for analysis and determination of relationships from various data sources

[0001] This application claims the benefit of filing date of U.S. Provisional Patent Application Serial No. 62/572,266, filed on October 13, 2017, filed on October 12, 2018, and is incorporated herein by reference in its entirety. It is a continuation-in-part of Patent Application Serial No. 16/159,088 and claims the benefit of the filing date.

The present invention relates to systems and methods for analyzing data from various data sources and generating answers to specific questions based on the analyzed data.

[0003] The digitization of labor continues as advances in machine learning, natural language processing, data analytics, mobile computing and cloud computing are used in various combinations to replace certain processes and functions. Solutions can be designed, tested and implemented at a relatively low cost, allowing basic process automation to be implemented without significant IT investments. Enhanced process automation includes more advanced technologies that can use data to support machine learning elements. Machine learning tools can be used to discover naturally-occurring patterns in data and predict outcomes. And natural language processing tools are used to analyze text in context and extract desired information.

However, since these digital tools are generally found in a variety of formats and coding languages, they are difficult to integrate and often not customized. As a result, these systems provide automated solutions to specific questions requiring analysis and processing of various types of input data, e.g., structured data, semi-structured data, unstructured data, images and speech. I will not be able to provide answers or answers. For example, these systems currently cannot efficiently address questions such as "Which of these 500 contracts do not comply with the new banking regulations XYZ?"

Therefore, it is possible to overcome the aforementioned shortcomings of known systems and apply automatic and custom analysis to analyze documents, communications, text files, websites and other structured and unstructured input files to ask specific questions. It would be desirable to have a system and method capable of generating output in the form of answers to and other supporting information.

According to one embodiment, the present invention relates to a computer-implemented system and method for analyzing data from various data sources. The method includes: receiving data from various data sources as inputs; converting received data from each of the various data sources into a common data structure; identifying keywords in the received data; generating sentence or word embeddings based on the document corpus; receiving a selection of one or more labels based on the generated sentence or word embeddings; adding the selected one or more labels to the model; training a model on a common data structure based on the configuration file; and generating a result in response to a user query based on the model, the generating comprising: retrieving related documents from the received data; determining from which of the retrieved related documents which information is to be reported; and providing a result based on the graph schema and the decision associated with the related documents.

[0007] The present invention also relates to a computer-implemented system for analyzing data from various data sources.

[0008] An exemplary document management workflow seamlessly integrates key tasks for document ingestion, forecasting, consolidation, and analysis. The workflow allows users to build a knowledge base by answering specific questions about documents (eg, contracts) and modeling relationships with other documents. In particular, each step (eg, collection, prediction, consolidation and analysis) is integrated into an end-to-end workflow that can be configured with minimal effort or changes required by the user. Each step builds on the previous steps to enable analysis and extraction of information from the documents. In this regard, other document management frameworks typically require a significant amount of "glue code" (eg, code tailored for a particular project) to bring the entire workflow together. On the other hand, the present invention allows users to configure each step without having to rewrite code, thus making the exemplary process easy to reuse in various projects.

Further, in accordance with an embodiment, the example workflow may address various types of document analysis problems, eg, by mapping a process to a specific problem/use case. Problems can arise in a variety of domains such as clause/compliance, procurement contracts, commercial leaks, contract risk analysis, etc. The example framework is also flexible, allowing users to customize business logic rules, post-processing and quality assessment tasks, and tailor them to the business use case and the specific needs of a specific user. . In other words, the exemplary process is suited to document analysis problems rather than attempting to fit the problem into a standard, inflexible framework. In addition, each part of the example workflow (eg, document processing, feature creation, model architecture, quality assessment, post-processing, and contract consolidation) may be associated with a default configuration, which addresses many specific questions. can cover However, these default configurations can be easily modified to handle new or unique questions. Additionally, Lume data structures enable unified learning models by persisting data and metadata throughout the exemplary process. Also, because the process is fully integrated, documents (eg, contracts) and their resulting documents can be processed to extract knowledge of the content within the document and answer specific questions. In addition, the example process may resolve information across multiple documents using a graph-based reasoning framework. For example, the business logic layer may allow subject matter experts to specify how to combine document families. In addition, the graph-based inference framework can specify the handling of conflicting clauses. Further, inferences may also be made at the document family level or at the individual document level.

[0010] The present invention also provides interchangeable model architectures that can be switched to find the optimal model framework for extracting a specific clause with minimal human interaction. A framework-specific language may be included in default configurations or custom configurations. Framework-specific features may also be made available through the knowledge-base. Also, very effective default options for certain problems minimize configurations by the user. In addition, interchangeable model architectures provide support for sequence labeling, classification and deep learning models that can be replaced via custom configuration files that can be used by non-experts.

[0011] Further, according to an embodiment, using the present invention, subject expertise may be encoded into the overall solution. For example, the present invention can digitize the completion of complex manual tasks to improve machine learning output. Also, post processing can be applied to organize or reformat high confidence answers based on client specifications. In addition, post processing can also utilize subject matter expertise to generate downstream answers to questions that rely on multiple pieces of information from a document. In addition, quality evaluation steps are added to ensure that high-confidence answers comply with the customer's specifications.

[0012] Further, according to an embodiment, the present invention also provides for the development of rich, high-quality training and testing datasets. For example, the present invention provides curated subject expertise in the labeling of data. In addition, the present invention utilizes search, text similarity and clustering techniques to obtain representative diverse labeled datasets that are more efficient and effective in generating high-performing models. In addition, datasets can also be integrated with framework-specific knowledge base information. In addition, the present invention also provides for the creation of custom word embeddings to better represent that particular domain. In addition, at least one of an integrated learning model or an active learning model may be utilized to label specific document information. Finally, specific models and results of the exemplary framework may be stored in a third-party storage device.

[0013] These and other advantages will be more fully described in the detailed description that follows.

[0014] To facilitate a more complete understanding of the present invention, reference is now made to the accompanying drawings. The drawings are not to be construed as limiting the invention, but merely to illustrate different aspects and embodiments of the invention.
1 is a functional block diagram of an analysis system according to an exemplary embodiment of the present invention.
2 is a diagram of an architecture of an analysis system according to an exemplary embodiment of the present invention;
3 is a representation of a standard data format for a converted file referred to herein as Lume, according to an exemplary embodiment of the present invention;
4A is a diagram depicting an example of a Lume structure and exemplary levels in accordance with an exemplary embodiment of the present invention.
4B illustrates a larger view of a document with metadata depicted in FIG. 4A .
[0020] Figure 5 is a diagram depicting the process of creating a Lume from a Microsoft Word document in accordance with an exemplary embodiment of the present invention.
6 is a diagram depicting a process of creating a dataset from a directory of Microsoft Word and text files in accordance with an exemplary embodiment of the present invention.
7 is a flowchart for an analysis system according to an exemplary embodiment of the present invention.
8 illustrates an example of a document to be collected and analyzed by an analysis system according to an exemplary embodiment of the present invention.
9 is an example of an expression presented as expression strings shown in a table according to an exemplary embodiment of the present invention.
10 is an example of output from an intelligent domain engine in the form of predictive answers according to an exemplary embodiment of the present invention.
11 is an example of output from an intelligent domain engine in the form of support and justification for answers according to an exemplary embodiment of the present invention.
12 is a system diagram of an analysis system according to an exemplary embodiment of the present invention.
13 is a flowchart for an analysis system according to an exemplary embodiment of the present invention.
[0029] Figure 14 is a flow chart of the annotating step depicted in Figure 13 in accordance with an exemplary embodiment of the present invention.
[0030] FIG. 15A is an architecture diagram for the active learning phase depicted in FIG. 13 according to an exemplary embodiment of the present invention.
[0031] Figure 15b is a workflow diagram for the active learning step depicted in Figure 13 in accordance with an exemplary embodiment of the present invention.
[0032] Figure 16 is a diagram of the machine learning phase depicted in Figure 13 in accordance with an exemplary embodiment of the present invention.
[0033] Figure 17 is a diagram of the consolidation phase depicted in Figure 13 in accordance with an exemplary embodiment of the present invention.
18 is a diagram depicting graph schemas for representing multiple documents in accordance with an exemplary embodiment of the present invention.

[0035] Exemplary embodiments of the invention will now be described to illustrate various features of the invention. The embodiments described herein are not intended to limit the scope of the invention, but rather to provide examples of components, use and operation of the invention.

According to one embodiment, the present invention relates to an automated system and method for the analysis of structured and unstructured data. An analytics system (sometimes referred to herein as a “system”) may include a portfolio of artificial intelligence capabilities, including artificial intelligence domain expertise and related technology components. The system may include basic capabilities such as document collection and optical character recognition (OCR), for example, the ability to take documents and convert them into machine readable formats to perform analysis. According to a preferred embodiment, the system also includes machine learning components that are not explicitly programmed (supervised and unsupervised) and provide the ability for the system to learn; deep learning components that model high-level abstractions of data; and natural language processing (NLP) and generation, eg, the ability to understand human speech or text and generate text or speech.

[0037] The system may also include structured data (eg, transactional system data and data organized into columns and rows, such as Microsoft Excel files); semi-structured data (eg, text that is not stored in a recognized data structure but still contains formatting or tabs of some type, such as forms); unstructured data (eg, text not stored in recognized data structures such as contracts, tweets, and policy documents); and images and voice (eg, photographs or other visual depictions of physical objects and human voice data), and may be designed to collect and process various types of input data.

The system may be deployed for collecting, understanding and analyzing the documents, communications and websites that make up the rapidly growing body of structured and unstructured data. According to one embodiment, the system: (a) reads report cards, tax returns, communications, financial reports and similar documents and input files, (b) extracts information and captures the information into structured files , (c) evaluate information in the context of policies, rules, regulations, and/or business objectives, and (d) design to answer questions, generate insights, and identify patterns and anomalies in information can be The system captures and stores subject expertise; collect, mine, and classify documents using natural language processing (NLP); integrating advanced machine learning and artificial intelligence methods; Collaborative and iterative improvements with advisory and customer stakeholders are available.

Examples of questions the system may answer include, for example, which documents comply with certain policies or regulations, which assets are most at risk, which rights warrant intervention, which customers are likely to suffer losses This could include which documents are highest/lowest, which clients will have increasing/decreasing wallets and market share, and which documents are undergoing a change in trend or meaning. Examples of policies or rules that the system can analyze include, for example, new regulations, accounting standards, profitability targets, identification of growth versus dilution projects, credit risk assessment, asset selection, portfolio rebalancing or Payment results may be included. Examples of documents that the system can analyze include, for example, legal contracts, loan documents, securities prospectus, company financial documents, derivative checks and masters, insurance policies, insurance claim notes, customer service report cards, may include email exchanges.

1 is a functional block diagram of a system for automated analysis of structured and unstructured data in accordance with an exemplary embodiment of the present invention. As shown in Figure 1, the system integrates various data sources, domain knowledge, and human interaction, in addition to algorithms for collecting and structuring content. The system comprises a scanning component 10 for collecting a plurality of documents 5 , such as contracts, loan documents and/or text files, and for extracting relevant data 6 . During the collection process, the system incorporates OCR technology to convert images (eg, PDF images) into searchable characters and incorporates NLP pre-processing to convert the scanned images into raw documents (11). and essential content 12 . In addition, appropriate aggregation approaches will be used to transform and preserve document metadata and formatting information. In many instances, the input unstructured data will reside in multiple documents that together form a corpus 15 of documents stored in the dataset.

The example of FIG. 1 depicts a “prescriptive rule set” implemented in a specific business context. An example of a set of regulatory rules could be new or amended financial rules, and a financial institution or company needs to ensure that contracts comply with the new rules. Manual review of contracts to assess compliance with new regulations is an alternative, but this approach can also entail significant time commitments and significant costs for experts to review contracts. Alternatively, the system reads contracts, extracts information and captures information into structured files, evaluates information in the context of revised regulations and/or business objectives, answers questions, and generates insights and the system can be configured to identify patterns and anomalies in contracts. Accordingly, exemplary embodiments of the present invention can automate the analysis of complex documents, which can provide advantages of enabling 100% coverage over existing sampling approaches, thereby reducing the development time and costs required to generate insights. It allows humans to achieve and manage precise consistency, leverage the knowledge and expertise of subject matter experts (SMEs), and automatically generate audit logs that describe how data is processed.

Referring to FIG. 1 , a set of prescriptive rules is used by subject matter experts in manual review and translated into relevant semantics 21 and decision strategy 22 in machine review. Semantics 21 includes domain knowledge realized as an ontology or knowledge base composed of entities, relationships and facts. The decision strategy 22 consists of business rules that are applied to the relevant semantics 21 to answer specific questions. This includes document-level assessments (e.g., conformance or non-compliance), feature-level extractions (end dates, key entities), inferred facts (e.g., inferred using extracted facts and ontology). , or hazard identification (eg, identifying parts of the document that require further scrutiny). The machine learning review 25a analyzes the directional features 26a, such as specified contract terms, dates, entities and facts, and automates it using an intelligent domain engine (sometimes referred to herein as an “IDE”). Initiate the document analysis evaluation 27a. The machine learning review 25a supports the machine compliance decision 28a by providing a confidence score. In parallel with this, a manual review 25b of selected documents performed, for example, by a subject matter expert, analyzes the directional features 26b and evaluates the document analysis on a sample of contracts 27b and a manual compliance decision 28b. to start Parallel manual and machine evaluations are used to determine an accuracy and confidence score (29), which is used as feedback (30) for manual and machine reviews. The feedback 30 may allow for improvement of the machine review, so that each iteration can provide improved accuracy and a corresponding increase in confidence score in the automated analysis. Active learning methods are used to reduce the number of iterations required to achieve a given accuracy.

[0043] Referring to FIG. 2, an architecture of a system in accordance with an exemplary embodiment of the present invention is depicted. As already mentioned, the system can support information extraction and data analysis for both structured and unstructured data. The input data 210 may take the form of various files or information of various types and formats, such as documents, text, video, audio, tables, and databases. As shown in FIG. 2 , the data to be analyzed may be input to the core document management system 220 .

According to a preferred embodiment of the present invention, the input data 210 is converted into a common data format 230 referred to as “Lume” in FIG. 2 . Lume may preferably be a common format for all components and data storage. As shown in FIG. 2 , the core document management system includes a document conversion system 240 (to convert documents to Lume format 230 ) and a document and corpus repository 220 . The document conversion system provides utilities for extracting document data and metadata and storing them in a format 240 used to perform natural language processing. The standardized Lume format 230 facilitates data processing and analysis of Lumes because multiple components can be easily applied to Lumes and utilize upstream information for enhanced processing. In one application, a processing workflow can be chained together to identify sentences, tokens, and other document structures; entity identification; Annotate a classification or ontology; and intelligent domain engine 251 may utilize this information to generate derived and inferred features. Each of these components utilizes Lume 240 as input and Lume 240 as output, and metadata may be further inserted into the Lume. Other examples of components include, for example, different engines, natural language processing (NLP) components 255, indexing components, and other types of components (eg, optical character recognition (OCR) ( 252 ), machine learning 253 , and image processing 254 ).

Components 250 read Lumes 240 and create Lume elements. Lume elements are then placed in a stand-off annotation format (database 220, parent class definition of base data format 230) and specific instances of formats in application specific data formats (240). described by them). As an example, NLP component 255 processes Lume 240 and represents human language specific constructs in the underlying data, including word tokens, part-of-speech, semantic role labels, named entities, joint reference phrases, etc. Add additional Lume elements to bet. These elements may be indexed to provide users with the ability to quickly search for a set (or individual) Lume 240 or Lume elements via a query language.

[0046] The Lume technique will be further described below with reference to FIGS. 3-6.

FIG. 2 also illustrates that multiple machine learning (ML) components 253 may be integrated into a system. For example, a system may include an ML transformation component, a classification component, a clustering component, and a deep learning component. The ML transformation component transforms basic Lume representations into machine-readable vectors for fast analysis processing. The classification component maps a given set of inputs to a set of learned outputs (categories or numbers) based on initial training and construction. The clustering component generates groups of vectors based on the predetermined similarity metric. A deep learning component is a specific type of machine learning component 253 that utilizes a multi-layer network representation of nodes and connections to learn outputs (categorical or numeric).

2 illustrates that a system may include a number of user interfaces 270 that allow various types of users to interact with the system. The IDE manager 273 allows users to modify, delete, and add representations to the system. The model manager 274 allows the user to select machine learning models to run in the pipeline. The search interface 272 (ie, data search) allows users to find data loaded into the platform. Document and Corpus Annotator 271 (ie, Commenting Manager) and editors can group Lumes into corpuses for users to manually create and modify annotations in Lume and to train and test the system. Visual workflow interfaces 275 (ie, a workbench) provide a visual capability for building workflows and can be used to create histograms and other statistical views of data stored in the platform.

3 illustrates the properties and characteristics of a Lume according to an exemplary embodiment of the present invention. As shown in Fig. 3, "name" is a character string composed of a non-qualified name of the document. "Data" is a string or binary representation of a document (eg, serialized data representing the original data). "Elements" are arrays of Lume elements.

As shown in FIG. 3 , each Lume element includes an element ID and an element type. According to a preferred embodiment of the present invention, only the element ID and element type are needed to define and create a Lume element. The element ID is a string containing the element's unique identifier. The element type is a string identifying the type of the Lume element. Examples of types of Lume elements include parts of speech (POS) such as nouns, verbs, and adjectives; and a named-entity-recognition (NER) such as a person, place, or organization. Also, file path and file type information may be stored as elements. The file path is a string that constitutes the full source file path of the document. The file type is a string constituting the file type of the original document.

[0051] Although not required, the Lume element may also include one or more attributes. A property is an object made up of key-value pairs. An example of key-value pairs may be, for example, {"name":"Wilbur", "age":27}. This creates a simpler yet more powerful format that allows for developer flexibility. According to an exemplary embodiment of the present invention, only the element ID and type is needed because it gives developers the flexibility to store information about a Lume in an element while also ensuring that it is accessible by ID or type. This flexibility allows users to decide how to store relationships and hierarchies between elements according to their domain expertise. For example, elements may contain information necessary for complex language constructs, store relationships between elements, or reference other elements.

[0052] According to an exemplary embodiment of the present invention, Lume elements are used to store a stand-off annotation format. That is, elements are not embedded in text, but are stored as comments separately from the document text. According to this embodiment, the system can restore the original data without modification.

According to a preferred embodiment, Lume elements are not stored in a hierarchical relationship to other Lume elements, and document data and metadata are stored in a non-hierarchical manner. Most known formats (except Lume) are hierarchical and difficult to manipulate and convert. Lume's non-hierarchical format allows easy access to all elements of document data or its metadata at the document level or text level. Also, editing, appending, or parsing a data structure may be done through operations on elements without resolving conflicts, managing hierarchies, or requiring other operations that may or may not be required by the application. According to this embodiment, since it is a stand-off annotation format, the system can preserve an exact copy of the original data and support overlapping annotations. It also allows for annotation in multiple formats such as audio, image and video.

[0054] Lume technology may provide a universal format for document data and metadata. Once a Lume is created, it can be used in each tool in the natural language processing pipeline without having to write format transformations to incorporate the tools into the pipeline. This is because the underlying conventions needed to pass data and metadata are established in the Lume format. The system provides utilities for extracting document data and metadata from multiple formats, including plain text and Microsoft Word. Format-specific parsers convert data and metadata from these formats into Lume, and correspondingly rewrite the modified Lume to the format. The system may use Lume technology to store information related to families of words in preparation for natural language processing such as preprocessing and stemming. The system may also use Lume technology to store information related to the relationships of documents, and graph structures.

[0055] According to an exemplary embodiment of the present invention, the system includes components other than Lume and Lume elements. In particular, the system may be configured to include a dataset, a Lume data frame, an Ignite component, and an element index. A dataset is a collection of Lume objects with a unique identifier. Datasets are generally used to specify training and testing sets for machine learning and can also be used to perform bulk operations on many documents. A Lume data frame is a special matrix representation of a Lume. Many machine learning and math components within the system can utilize this optimized format. The system may also include Ignite components that read Lume (or Lume corpus) data and return Lume (or Lume corpus) data, typically by processing existing Lume elements or original source data and adding new Lume element objects. have. Element indexes are representations commonly utilized in Ignite for computer object representation of sets or elements and for efficiency in retrieving Lume data and metadata. For example, some components may be optimized to work with character offsets, so an index to character offsets may speed up operations on those components.

According to an exemplary embodiment of the present invention, the main functions of the system include data representation, data modeling, discovery and configuration, and service interoperability, which are described as follows.

[0057] Data Representation: Lume is a general data format used to store and communicate analyzes in a system. Lume takes a stand-off approach to data representation, for example, analysis results are stored as annotations independent of the original data. According to one embodiment, Lume is implemented in Python and has computer-object representations as Python objects and is serialized to "JSON" (JavaScript Object Notation) for inter-process communication. Lume can be designed for use with web-based specifications such as JSON, Swagger (YAML), RESTful, and will interface with the Python ecosystem, but can also implement and support components written in Java and other languages. .

[0058] Data modeling: Lume can be designed to be simple and enforce only basic requirements for system users. Interpretations and business logic are left to system users rather than requiring declarative representations of both data and processes. The system can be designed to leave the modeling informal and leave details for implementations to processing components. This allows Lume to keep a very simple specification and be extended for specific applications without interfering with other applications. For example, if Lume search is important, it is integrated with an indexing module on top of the Lume structure. When working with the Document Object Model (DOM) is important, the DOM parser stores additional information in the Lume in the form of Lume elements and attributes and uses this information to transform it back into the DOM model.

[0059] Discovery and Configuration: Lume may also have additional design features related to analysis process sources. System workflows may require source information to facilitate repeatability and discovery of components. This source information is stored in Lume and can be enforced through source-enforcement workflows. For example, this may provide a look at each of the output Lumes to ensure that the correct processing steps have been completed. In the validation stage, you can provide a means to trace the origin of the Lume element that generated correct or incorrect metadata. In addition, it can also be tracked to ensure that all inputs are received as outputs.

[0060] Service Interoperability. Services provided by the system may require Swagger (YAML Markup Language) specifications according to an embodiment of the present invention. There may be many assumptions regarding the business logic utilized to implement the system components, the order of actions, and other interpretations of data. Identifying interoperable components may be accomplished through analysis of example workflows rather than input and output specifications. In the system, components can simply operate on Lume and return the correct error code in case of an error and log appropriate logging information.

4A illustrates an example of a Lume structure and initial conversion of different types of files to Lume. As shown in FIG. 4A , dataset 410 refers to a body of different types of files or documents. These documents may initially be in different formats such as Adobe Portable Document Format (PDF), unstructured text files, Microsoft Word files, and HTML files.

4A also illustrates an example of elements defined for Lume. For example, the first element 411 may correspond to a research director including contact information; The second element may correspond to a protocol manager including contact information 412 ; A third element may correspond to a contract research organization (CRO) comprising contact information 413 , a fourth element may correspond to a research and development company 414 , and a fifth element 415 may correspond to confidentiality of documents. Able to respond to maintenance notices. 4B illustrates a larger view of a document with metadata depicted in FIG. 4A .

Also shown in FIG. 4A are example levels of element types. For example, the system may provide functionality that enables a user to identify individual paragraphs, tokens or entities, each of which may be extracted from a Lume.

5 provides further details of an example of creating a Lume in a Microsoft Word document. As shown in Fig. 5, the first step, that is, step 501 is to initialize the original document. Initialization involves storing the original data in a Lume object. The second step, namely step 502, is to parse the document into elements in Lume format. This step may include a loop 502a in which elements are created corresponding to metadata from the source document. This is done by document specific components that collect a specific format. Specifically, during ingestion (i) the original file is opened, (ii) the DOCX format is decompressed into an XML file, and then (iii) the XML file is read into a data structure for parsing. Parsing separates the document's data from the metadata, then stores the data in the Lume's "data" field, and stores the metadata in Lume elements. Then, this is output to LumeText. Examples of stored metadata are author, page, paragraph, and font information.

[0065] When the process shown in FIG. 5 is finished, the input document is converted into a Lume, and desired elements are created and stored.

6 illustrates an example of applying the function of FIG. 5 to a corpus of documents. The first step in FIG. 6 , ie step 601 , includes initializing the dataset. Subsequent steps in FIG. 6 involve applying the processes shown in FIG. 5 to each document in the dataset. As the Lumes of the dataset are converted to Lume format in step 602, the results are stored in the dataset. The transformation includes creation of a Lume data structure (ie, loop 602b), transformation of format-specific metadata into Lume elements (ie, step 602a), and creation of semantic annotations, natural language processing, domain-specific features. , or any additional annotations needed, such as vectorization for a quantitative fingerprint. More specifically, in step 601 , dataset documents are identified in the URI, and then Lumes containing file data are passed to 602 . Next, at 602b, the Lume is passed to the appropriate parser, which creates the appropriate data structure for parsing. At 602a, the parsing works through the document and parses the data and metadata in the Lume's "data" field into Lume elements. Then, this is output to LumeText.

7 is a process diagram illustrating an example of a process for analyzing structured and unstructured data in accordance with an exemplary embodiment of the present invention. In step 710, documents such as text, Microsoft Word and/or Adobe PDF documents are collected into the system. The documents are then converted to Lume format as described above in step 712 . The OCR process may be used in step 714 to convert the image file to characters. In step 716, the documents are collected into a dataset. At step 718 the system identifies and annotates structural Lume elements (see, eg, FIG. 6 ). Once the documents are converted to Lume format and Lume elements are created, natural language processing (NLP) routines or components may be applied to the Lume format information in step 720 .

In step 722, the user of the system creates and inputs an ontology comprising a list of entities. According to an example, the ontology may describe people and what kind of company they are employed in. For example, an ontology can be useful for extracting people and businesses from documents in a platform. Alternatively, an ontology may describe the various products of a company, the categories to which the product belongs, and any dependencies between the products. Step 724 includes entity analysis and semantic annotation. Entity analysis determines whether the entities referenced in the data are actually the same real-world entities. This analysis is accomplished using extracted data, ontology and additional machine learning models. Semantic annotations relate the syntax of data to formally-defined concepts defined in ontology. In the business employee example above, occurrences of the words "John Doe" are identified and associated with employee John Doe in the ontology. This will allow downstream components to take advantage of additional information about John Doe, such as his position and capabilities in the company.

In step 726 , the system user creates representations to be applied to documents stored in the dataset. Representations may be, for example, comma-separated-value (CSV) files specifying patterns to search for or other distinguishing features of documents. Expressions may include the expertise and know-how of subject matter experts. For example, the expression may identify various specific words and relationships between words or patterns that identify specific contractual clauses or clauses in a tax document. These expressions are used to search for and identify specific aspects, clauses, or other identifying features of a document. Expressions can also utilize machine learning operators, pre-trained sequence labeling components, or algorithmic parsers that act as one of the operators in the IDE.

At step 728 , the representations are input into the IDE that reads the representations and applies them to the dataset. According to one embodiment, the output may include predicted answers and support and justification for answers. The IDE will be further described below with respect to FIGS. 8-12.

At step 730 , the output of the IDE may be utilized to fabricate additional features. This utilizes previously created Lume elements and creates new Lume elements corresponding to additional features. Feature engineering can be abstractly thought of as indicator functions for sets of Lume elements for generating features associated with specific signals for learning and inference tasks. In the general case, feature engineering may generate additional categorical or descriptive text features needed for sequence labeling, or sequence learning tasks. For example, engineering may prepare features for custom entity tagging, identify relationships, or target a subset of elements for downstream learning.

In step 732 , machine learning algorithms or routines are applied to generate results from the Lume elements generated upstream. Machine learning can also be replaced by sequence labeling, or Bayesian network analysis. It generates machine-learning scores, or probabilistic information about the accuracy of previous annotations, relationships between elements, or with new annotations or classification metadata. The results are analyzed in step 734 where the results are provided to an analyst for review either via a UI for examining the annotations or a workbench for performing further analysis on the results. In step 736, one or more iterations are performed to improve prediction accuracy. The steps of applying the representations 728 , engineering the features 730 , applying machine learning 732 , and reviewing the results 734 may be repeated to improve accuracy. Once the accuracy has improved to achieve the desired level, the results may be stored in a database at step 738 . Note that entity analysis and semantic analysis 724 , engineer features 730 , and machine learning 734 will also be utilized within the intelligent domain engine, but are separate for large processing pipelines.

According to an exemplary embodiment of the present invention, the IDE includes natural language processing, custom built annotation components, and a platform for utilizing manually encoded representations to systematically classify and analyze a corpus of documents. . An IDE can provide a platform for combining a company's cognitive/AI capabilities with industry domain knowledge. Each document classification may be represented by a set of expressions that may include features to be utilized, patterns of features to be identified, and reference location or range information for focusing the classification task. Representations can be constructed and manipulated with Lume elements and data contained in Lume. The IDE is designed to systematically evaluate the representations for each document in the corpus, producing specified results as well as annotated text to support classification decisions. It is noted that in this example the IDE is utilized for natural language processing and text mining, but the IDE framework applies to all Lume formats such as images, audio and video.

[0074] An IDE can provide a number of advantages. For example, the IDE can output annotated text to support classification decisions in addition to answering specific questions. Annotations can be used to audit results and provide transparency. Also, training an accurate machine learning model usually requires a large number of labeled documents. Using an IDE to integrate domain knowledge with machine learning can significantly reduce the number of documents required to train an accurate model utilizing expert-derived features. This is because machine learning problems involving unstructured data are generally over-determined, and the ability to select accurate and interpretable features typically requires more data than is available. For example, in documents, there may be tens of thousands of features, including a dictionary of words, spelling features, document structures, syntactic features, semantic features. Further, in accordance with an exemplary embodiment of the present invention, representations may be generated using a domain specific language that may be coded in codeless environments, such as via a spreadsheet (CSV or XLSX) or IDE user interface; Individuals such as subject matter experts (SMEs) entering expressions do not require computer coding skills. Thereby, the SME can create domain-related features that can be utilized in the machine training process. The IDE UI allows users to edit, delete and add representations to the system and launch the IDE to visualize the elements created. Also, representations may be designed to be interchangeable. They can be created for reuse in use cases across an industry or problem set. Additionally, the IDE can be designed to utilize the Lume format for storing and working with documents. This design allows annotations and metadata as inputs to representations in addition to text features present in the document.

[0075] According to an exemplary embodiment of the present invention, a process for creating and using a representation may include: (1) manually reviewing documents, (2) capturing patterns through representations and utilizing machine learning or statistical extraction. By generating custom built code that has an IDE, (3) loading representations into the IDE and running the IDE, (4) building confusion matrices and accuracy statistics (i.e., by comparing the current results to a set of invisible documents, this is how well the representations work) generate an estimate of whether it will generalize, determine whether the system meets performance requirements), (5) iterate and refine previous steps, and (6) provide support and justification for predicted answers and answers It includes generating output such as sections that

According to one particular example, an IDE may be used to analyze documents such as investment management agreements or other legal documents to automatically determine answers to legal questions. For purposes of illustration, it is assumed in this particular example that a company has eight legal questions to answer in relation to its 500 investment management agreements. An exemplary question may be "Does the contract require notification in connection with identified personnel changes?" 8 depicts an example of investment management contract sections related to legal questions.

9 illustrates examples of representations according to an embodiment of the present invention. As shown in FIG. 9 , the representations may be detailed in a table format (eg, CSV) rather than a code. In the example of FIG. 9 , each representation has a “name” that may be useful when referring to other representations. The name can also be used by the output file to create features. Each expression may also include a “range” that focuses and limits the expressions to be applied. The scope itself is evaluated as an expression, and the results are used to constrain the scope of the parent expression. For example, a range expression may refer to Lume elements (either predefined in a conversion to Lume format or generated by another expression), or it may be the result of an operator that identifies the appropriate clause in the contract. The expression also includes a "string" field in which the expression is contained. The string field has a predetermined syntax. The string field can specify patterns to look for in documents or logical operations. 9 shows examples of a character string field.

[0078] The expression may also include a “condition” field that is used to determine whether a particular expression should be evaluated. This is useful for implementing control logic that activates or deactivates representations for computational efficiency, or activates or deactivates certain types of processing.

An expression may be used to search for patterns in documents, and the expression may encapsulate these patterns. Examples of such patterns include various ways of expressing notification requirements and staff changes. For example, many words for "HR", such as "core", "investment team", "professional", "senior", "senior", "portfolio manager", "portfolio manager" , "investment managers", "key decision makers", "key employees" and "investment managers". Case sensitivity is important in some cases. For example, "investment manager" may refer to an employee; An “investment manager” may refer to a client's investment organization. In some cases the order of words (representing a subject-object relationship) is important. For example, the investment manager notifying the client is not the same as the client notifying the investment manager. All these types of patterns can be encapsulated in representations. Subject matter experts (SMEs) may encapsulate in representations the know-how of analyzing certain types of specialized document types.

[0080] Figure 10 illustrates an example of one form of output from the IDE: predicted answers. It contains answers to each question for each document. For example, as shown in FIG. 10 , the output may include the filename of the input file, a table listing answers to four questions providing decisions about the features of the contract. Depending on the embodiment, there may be more questions or features to be output in the IDE.

[0081] Figure 11 illustrates another form of example coming out of the IDE: support and justification for answers. 11 , the user interface displays the actual contract language used by the IDE to support and justify a given answer. The actual contract language is presented so that the user can evaluate whether the IDE is correct or not. The system may utilize the information stored in the Lume element to highlight certain words of text that specifically form the basis for the answer provided by the IDE. In this way, the IDE allows a human user to easily verify that an answer is correct. It also enables the user to understand any errors and the ability to modify the expression to correct these errors.

12 is a system diagram of a system according to an exemplary embodiment of the present invention. 12 , the system may include a server 120 and an associated database 122 along with software and data used to run the system. The system may also include a scanner 126 used to scan and collect original documents into the system. Server 120 and database 122 may be used to store collected documents as well as IDE, Lumes and Lume elements, and other software and data used by the system. A user 125, such as a subject matter expert (eg, a tax professional), can access the server 120, scanner 126 and database (eg, via personal computing device 124 , such as a laptop computer, desktop computer, or tablet computer) 122) can be accessed and used.

[0083] The system may also be configured to allow one or more clients or other users to access the system. For example, as shown in FIG. 12 , client 135 may use personal computing device 134 and corporate server 130 to access server 120 via network 110 . The client also sends client-specific data (eg, a set of contracts to be analyzed) stored in the client database 132 to the system in dataset documents to be analyzed by the server 120 and stored in the database 122 . can be integrated The server 120 shown in FIG. 12 includes other documents, spreadsheets, pdf files, text files, audio files, video from other clients or users, generally represented as servers 140 and 150 . files, and other structured and unstructured data.

Also shown in FIG. 12 is a network 110 . Network 110 may include, for example, the Internet, an intranet, a local area network (LAN), a wide area network (WAN), an Ethernet connection, a WiFi network, a Global System for Mobile Communication (GSM) link, a cellular telephone network, a Global Positioning (GPS) network. System) link, a satellite communication network, or any one or more of other networks. Other computing devices, such as servers, desktop computers, laptop computers, and mobile computers, may for example be operated by different individuals or groups, and may transmit data such as contracts or insurance policies to network 110 . It can be transmitted to the server 120 and the database 122 through. In addition, cloud-based architectures with containerized or microservices-based architectures can also be used to deploy the system.

13 is a flowchart for an analysis system according to an exemplary embodiment of the present invention. As depicted in the figure, the flow diagram 1300 includes a document collection step 1310, a preprocessing step 1320, an annotating step 1330, an ML framework step 1340, a post-processing step 1350, and a multi-document consolidation step 1360 . As a result of these steps, flowchart 1300 may provide extracted document knowledge.

According to an embodiment, during step 1310, data from various data sources, for example machine-readable and/or non-machine-readable PDFs, Word documents, Excel spreadsheets, images, HTML, etc. is collected (i.e. input). In particular, raw data from various data sources is converted and stored in the same Lume data structure, providing consistency across different data types.

Also, according to an embodiment, during the preprocessing step 1320 , a number of operations are performed to enrich the downstream modeling steps. For example, if necessary, optical character recognition (OCR) may be performed to convert text from machine-readable PDFs or images to machine-readable text. Additionally, additional Lume elements can be added to incorporate image-related features that can also be utilized downstream. In addition, natural language processing operations on the document text are also performed. For example, words and sentences of document text may be tokenized and/or titled. In addition, optional information such as speech tagging or part of named-entity-recognition (NER) may also be included during this step to enrich the information available for subsequent modeling. Custom word embeddings may also be added to token elements, where word embeddings are added to word elements and/or sentence elements retrained and tokenized for a domain-specific document set. According to an embodiment, word embeddings may be retrained with multiple documents, for example more than 50 documents. Further, according to an embodiment, the added word embeddings may simplify annotation and smooth OCR errors in feature creation and modeling. Also, in situations where documents are compiled into a single file (e.g., master service agreements and corresponding multiple revisions typically stored in a single PDF), it may be necessary to split the file into component documents. . In these cases, a heuristic or trained model is utilized to partition the documents into constituent parts. According to an embodiment, document segmentation is useful in cases where consolidation logic is applied to sets of document families. In these situations each document needs to be analyzed and considered separately in order to properly apply the logic to the set of documents. For example, assuming a master service agreement has three modifications, information across these related documents, eg payment terms for a contract, can be consulted after preprocessing and model prediction are executed. . However, only one of the documents, for example the most recent revision, may contain the most relevant information, such as payment terms for a contract. Thus, contract consolidation can be used to apply logic across a set of documents and extract the most relevant information.

Further, according to an embodiment, during annotating step 1330 , human knowledge and expertise may be incorporated into process 1300 , where SMEs may label specific information in a document. This information may label specific phrases and/or text to extract, or specific clauses or paragraphs with specific types, eg Type A, Type B, etc. According to an embodiment, this SME knowledge may be integrated in a variety of ways, for example in a web or Excel-based user interface. These annotations can then be added directly to the Lume data structure.

14 is a flow chart of annotating step 1330 . After preprocessing is complete, the Lume data structure can be annotated. Lume's data includes not only the text of the documents, but also elements that describe the words, sentences, etc. Subsequently, Lume's information is utilized during the annotation phase. In particular, annotations are added as elements that directly reference data (eg text) contained in a Lume. As depicted in the figure, at step 1331 keywords/phrases and representative examples of the document language are identified. According to an embodiment, identification may be performed with the SME via a user interface. In addition, identified keywords/phrases and representative examples may be provided to the knowledge base 1334 . In addition, the identified keyword/phrases and representative examples may also be used to compute embeddings of example sentences as depicted in step 1332 . Then, at step 1333 , custom word embeddings are trained based on the calculated embeddings and SME knowledge, which may also be provided to the knowledge base 1334 . Also, as depicted in the figure, active learning steps may also be performed.

[0090] During active learning, strategies are created to identify and streamline the data annotation and training set creation process. According to an embodiment, active learning utilizes word embeddings, sentence embeddings and keywords to find possible candidates of text in a wider dataset. In particular, a set of logical keyword searches as well as some examples of target text (eg, example sentences of where the target information appears) are input for analysis. For example, when searching for candidates to annotate contract terms, keywords may include language such as “condition”, “period”, “years” or “months”. Also, sentence embeddings such as "[t]he Agreement will last for a term of 10 years" can be used to find similar contextual languages. This particular active learning strategy narrows the search for similar but inaccurate annotations with high probability. The user can then review these results and use these candidate annotations to add labels directly to the Lume dataset of documents. Further, according to an embodiment, this active learning strategy is also useful for balancing the training set with rare information, such as rare fields. Also, through active learning, various annotations can be created and representative datasets can be developed in a streamlined manner and stored in Lume along with other metadata. In this way, annotations can be utilized with complementary information stored in Lume.

According to an embodiment, as depicted in the figure, a specific active learning strategy (eg, increasing data diversification, improving model information, etc.) may be applied. For example, the similarity of sentence embeddings can be compared to the mean. Then, at step 1336, the user may review the results of the strategy, for example, by confirming or rejecting certain labels. The results are then incorporated into the Lume metadata. In addition, the user can also refine searches or annotations, or add new data as needed. The identified labels are then added to the model, as depicted by step 1337 .

According to an embodiment, the exemplary framework combines both implicit knowledge transfer and explicit knowledge transfer in a complementary manner. For example, implicit knowledge transfer, such as feature engineering in the form of IDE representations, is used to support explicit knowledge transfer, ie annotation through active learning. That is, the IDE representations can be used to provide the active learning algorithm with the ability to supply candidates for the SME to label/review. Further, in accordance with an embodiment, in the process of reviewing candidates, features engineered based on the SME's observations are also updated/improved. This cycle (eg, IDE representation features ("explicit") -> review of candidates ("implicit") -> improvement of features based on observations ("explicit") -> review of more candidates ("implicit")) iterates until the model meets the expected performance.

15A and 15B illustrate the interaction between components in the active learning phase depicted in FIG. 13 . According to an embodiment, the active learning phase includes a user interface 1410 , an active learning application programming interface (API) 1420 , a database 1430 , a module management module 1440 , an Ignite platform 1450 , and A local platform 1460 may be utilized. The API 1420 communicates with the model management module 1440 , which allows the user to run any number of experiments on a given dataset (eg, changing hyperparameters or feature sets). The API 1420 also tracks performance metrics for specific settings of the experiment. In addition, API 1420 may also be an Ignite platform (eg, a cloud server running Ignite software to execute workflows) or a local platform (eg, a local server running Ignite software to execute workflows). or a personal computing device) to interpret the instructions for active learning. For example, if an SME is attempting to create a model that predicts a “supplier name” in multiple contracts, the SME, eg, via user interface 1410 , may determine that the supplier name is typically “by,” “between”. ," "agreement," "inc.," etc. may indicate to the model that it can be located somewhere around words. According to an embodiment, the SME may provide this information to the model in the form of IDE representations. The active learning strategy then uses API 1420 to select annotation candidates that are best suited for description of IDE representations, eg, auto-annotations (“auto-annotations”). These candidates may be reviewed by the SME using the user interface 1410, thus providing the model with implicit knowledge of the “supplier name”. For example, an initial model, eg, Model 1 of 15B, can be trained on the reviewed examples (candidates manually identified by the user) and additional auto-annotated examples of the active learning strategy. The model performance may then be evaluated on the test set. According to an embodiment, manually-reviewed examples may be maintained for future training; Auto-annotated examples do not propagate through additional model iterations. During this candidate review process, the SME may refine the IDE representations based on the observed results (eg, remove the word "by" and add the word "company"). Once these refinements are complete, a Model 2 active learning strategy may be constructed in the IDE representation enhancements that may be provided by the SME via the user interface 1410 . Users can then manually review examples of this updated active learning strategy. As in the first iteration, the new model will be trained from both manually-reviewed annotations (provided by the SME via user interface 1410), and auto-annotations (provided directly by the active learning prediction framework). will be. This results in a new model version (eg, from model 1 to model 2 in FIG. 15B ), which is then utilized within the active learning prediction framework to generate new candidates for review based on these improvements. The cycle continues until the model has enough implicit and explicit knowledge to make predictions at an acceptable performance level.

According to an embodiment, after the SME annotations are incorporated into the Lume data structure, model training may begin with the ML framework 1340 . According to an embodiment, the ML framework 1340 consists of several components that work together to train or apply algorithms to Lume data structures. For example, the information extraction component 1349 acts as an interactive layer with the machine learning component 1346 . Further, according to an embodiment, users may create a configuration file 1341 that may be interpreted by the information extraction component 1349 prior to sending instructions to the machine learning component 1346 . According to an embodiment, the instructions in the configuration file 1341 include the task type (eg, training, validation, prediction, etc.), the algorithm type and the package (eg, a regression algorithm such as sklearn logistic regression). , recursive algorithms such as keras LSTM, etc.) and features (eg, custom features, word embeddings, etc.). The machine learning component 1346 operates on information passed from the configuration file 1341 by executing training or prediction as commanded and/or sending instructions to a regression or recursive algorithm. The machine learning component 1346 may also apply any labeling techniques that may be needed, such as BIO labeling, sliding windows, etc., as well as save or load the trained models. According to an embodiment, regression and recursive algorithms receive data inputs from machine learning component 1346 , perform training or prediction as commanded via configuration file 1341 , and produce results (eg, training model or prediction) back to the machine learning component 1346 . Also, according to an embodiment, the process builder 1345 may enable all of the above tasks by acting as an API for building and translating instructions that may be provided in YAML format. For example, if the user wants to use different modeling packages for training and prediction, the user can provide the package and model type names of the YAML construct to the framework 1347 of the process builder 1345 . The user may also customize any default modeling algorithm using module 1348 . Also, using the ML framework 1340, minimal changes to the YAML file, if any, are needed to change the inclusion/exclusion of feature engineering and model training. Also, differences in behaviors across models are isolated in a configuration YAML file and not mixed with a common code base. This allows the code base to remain "stable", while at the same time providing users with a target at any point and at any scope (eg, fine-grained and/or coarse modifications) to the workflow behavior of a particular model instance. It still allows the flexibility to make modifications. Also, because these modifications reside within configuration file 1341 (not code), they can be safely delivered to the platform without the need to install additional code in the distribution. For example, the user may modify the model input to ignore punctuation, stop words, add additional features such as word embeddings, and determine if a word is capitalized. These changes can be made by modifying the configuration YAML file instead of changing the source code. The configuration file can then take the referenced features and create a feature matrix from the training dataset.

[0095] Figure 16 is a diagram of the machine learning phase depicted in Figure 13 in accordance with an exemplary embodiment of the present invention. According to an embodiment, the training of the model as well as predictions from the existing model is performed using the same configuration file, eg, configuration file 1341 . As depicted in the figure, during training mode, target truth tables may be extracted from a training dataset, such as LumeDataset, and then provided to an initialized model. In addition, features may also be extracted from the training dataset and then provided to the initialized model. The selected model architecture, eg, a third party modeling package 1440 (eg, CykitLearn, Keras, etc.), then executes a model training step, which is then stored in the database 1430 . Then, during prediction mode, the trained model can be loaded from database 1430 and run on features extracted from the testing dataset as well as feature matrices established from a configuration file to predict results from the testing dataset. . According to an embodiment, the training dataset is data specifically used to develop a model but not used to test model performance; Conversely, the testing dataset is used to test model performance, but not to train the model. However, both datasets must be labeled.

According to an embodiment, many of the questions that may be asked about documents involve explicit extraction of raw information from the text itself. However, for machine-unreadable documents where spelling errors are common and/or formatting does not match, additional processing is required. For example, dates may be recorded in documents in many different ways (eg 4/5/2010, 4.5.10, 5 April 2010, 5th April 2010, etc.) - The information must still be formatted accordingly when reported for analysis. Therefore, post-processing is required. In this regard, during the post-processing step 1350 , the user may customize certain tasks and functions to perform on the model results. In addition, the post-processing step 1350 may also be used to impose certain business logic and conditions on the results of the models. For example, certain business logic may be imposed if one field may depend on another field - if the model predicts that the contract should not have auto-renew, there should be no consequences for the auto-renew period. do. Accordingly, by the post-processing step 1350, data may be provided in a format required by the user. Also, business logic may be imposed on various model predictions if the results contain interdependent fields.

Further, according to an embodiment, during the consolidation phase 1360 , relevant documents are entered and then business logic is applied to the graph consolidation engine 1361 (Fig. 17). For example, in the case of a master service agreement with multiple modifications, the information about the contract term should be derived from the most recent modification. According to an embodiment, this logic may be coded in the graph consolidation engine 1361 by the user. Further, the consultation operations may be implemented by the graph database 1370 (eg, JanusGraph) to model relationships between documents. For example, as depicted in FIG. 17 , multiple documents 1362 and 1363 (or versions of the same document, ie, “document 1”) are updated or conflicting facts (eg, facts). A and B) together with the graph consolidation engine 1361 may be input. For example, with respect to document 1362, fact A = “true” and fact B = “1”. On the other hand, in document 1363, fact A = “false” and fact B = “2”. In this regard, to resolve conflicts between documents 1362 and 1363 , graph consolidation engine 1361 uses other model outputs found in the document, which can be retrieved from graph database 1370 . The graph consolidation engine 1361 can then provide a consolidated output 1364 that reflects the current true facts for document 1 .

18 is a diagram depicting graph schemas for representing multiple documents in accordance with an exemplary embodiment of the present invention. For example, as depicted in the figure, document 1366 (i.e., Doc 1, Doc 2, Doc 3, and Doc 4) can be either graph schema 1367 (i.e. graph schema A) or graph schema 1368 ( That is, it can be expressed as graph schema B). According to an embodiment, graph schemas 1367 and 1368 may be based on custom models for business cases defined by SMEs. Graph schemas 1367 and 1368 can be created via a configuration file, where the SME can specify information in the document that can be used to determine connections between documents 1366 of the graph. This graph model can then be loaded into a graph database, and all data loaded into the graph conforms to this graph model. Also, graph edges can be established automatically and dynamically based on the processed models. In this regard, using graph schema 1367, documents 1366 are linked by a shared document ID, eg, “contract family 1”. Also, using the graph schema 1368, Lumes are linked to the document root through their associated client names.

Also, according to an embodiment, the example framework may answer questions about a document family using graph query fitting to a dynamic schema. For example, assuming the question is "find renewal period, prioritize revisions by newest", the exemplary framework would convert the query to a graph query, perform a traversal of the graph, find only revisions, and "model effective date" sort them by Results are returned to the user with a full description of how the consolidation was performed, but no understanding of the underlying graph model is required. For example result: "Find X revisions, the date may be. They have the following renewal periods: Y. Best answer is Z". Also, if the question is "the lowest price is a valid price," the exemplary framework converts the query to a graph query and performs a traversal of the graph to find any document with a price, followed by the lowest price." So, the result is: "Find X documents with price. The values are […] ]lim. lowest values are Y".

[00100] Also, as depicted in FIG. 13 , an exemplary framework, eg, flow 1300 , also implements QA checks after every step of flow 1300 to enforce high quality and consistency quality assessment. Includes (QA) components. These checks are based on (i) whether certain Lume elements were created and added to the Lume data structure as expected, (ii) whether all Lumes were successfully passed from step to step, and (iii) whether the correct attribute keys and numbers were assigned to each step. Whether or not it is included may be included. In addition, users can also configure and add their own custom quality assessment checks as needed.

[00101] It will be appreciated by those skilled in the art that the various embodiments described herein are capable of wide utility and applicability. Accordingly, while various embodiments are described in detail herein in connection with exemplary embodiments, it is to be understood that the present disclosure is intended to provide a possible disclosure that is and is illustrative and exemplified of various embodiments. Accordingly, this disclosure is not intended to be construed as limiting the embodiments or to exclude any other such embodiments, adaptations, variations, modifications and equivalent arrangements.

[00102] The foregoing description provides examples of different configurations and features of an embodiment of the invention. Although certain nomenclature and application/hardware types are described, other names and application/hardware usage are possible and the nomenclature is provided as non-limiting examples only. Further, while specific embodiments are described, it should be appreciated that the features and functions of each embodiment may be combined in any combination as they come within the ability of those of ordinary skill in the art. The drawings provide additional illustrative details regarding various embodiments.

[00103] Various exemplary methods are provided herein by way of example. The described methods may be practiced or performed by one or a combination of various systems and modules.

[00104] Use of the term computer system in this disclosure may refer to a single computer or multiple computers. In various embodiments, multiple computers may be networked. Networking may be any type of network including, but not limited to, wired and wireless networks, local area networks, wide area networks, and the Internet.

According to exemplary embodiments, system software is one or more computer program products, eg, a computer program encoded in a computer-readable medium for executing by or controlling operation of a data processing apparatus. It may be implemented in one or more modules of instructions. Implementations may include single or distributed processing of algorithms. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, or a combination of one or more thereof. The term “processor” includes all apparatus, devices and machines for processing data, including, for example, a programmable processor, computer, or multiple processors or computers. The device may include, in addition to hardware, software code that creates an execution environment for a corresponding computer program, for example code constituting processor firmware, protocol stack, database management system, operating system, or one or more combinations thereof.

[00106] A computer program (also known as a program, software, software application, script or code) may be written in any form of programming language, including compiled or interpreted language, and may be a stand-alone program or module, component, subroutine or may be distributed in any form, including other units suitable for use in a computing environment. A program may contain other programs or data (e.g., one or more scripts stored in a markup language document), a single file or multiple control files dedicated to that program (e.g., one or more modules, subprograms or portions of code) may be stored in the portion of the file holding the files). The computer program may be distributed for execution on one computer or multiple computers located at one site or distributed over multiple sites and interconnected by a communication network.

A computer may include all apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may include, in addition to hardware, code that creates an execution environment for a corresponding computer program, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[00108] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. Processes and logic flows may also be performed by a special purpose logic circuit, for example an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) and the apparatus may also be implemented as said special purpose logic circuit.

[00109] A computer-readable medium suitable for storing computer program instructions and data includes, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and all forms of non-volatile memory, media and memory devices, including CD ROM and DVD-ROM disks. The processor and memory may be supplemented or integrated by special purpose logic circuitry.

Although embodiments have been particularly shown and described within a framework for performing an analysis, it will be appreciated that variations and modifications may be effected by those skilled in the art without departing from the scope of the various embodiments. Furthermore, those skilled in the art will recognize that such processes and systems need not be limited to the specific embodiments described herein. Other embodiments, combinations of the present embodiments, and their uses and advantages will be apparent to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. Specifications and examples are to be considered illustrative.

Claims (19)

A computer-implemented method for analyzing data from various data sources, comprising:
receiving data from the various data sources as inputs;
converting the received data from each of the various data sources into a common data structure;
identifying keywords in the received data;
generating sentence or word embeddings based on the identified keywords;
receiving a selection of one or more labels based on the generated sentence or word embeddings;
adding the selected one or more labels to a model;
training the model on the common data structure based on a configuration file; and
and generating a result in response to a user question based on the model, wherein the generating comprises:
retrieving related documents from the received data;
determining from which of the retrieved related documents which information is to be reported; and
and providing the result based on the determination and a graph schema associated with the related documents. According to claim 1,
wherein the various data sources include at least one of a machine-readable document, a machine non-readable document, a spreadsheet, an image, a Hypertext Markup Language file. According to claim 1,
and partitioning the received data into component documents, wherein the received data is partitioned based on one of a heuristic model and a trained model. According to claim 1,
tokenizing at least one of word elements and sentence elements of the received data; and
and adding default word embeddings to at least one of the tokenized word elements and sentence elements. According to claim 1,
wherein the configuration file includes instructions regarding at least one of a task type, an algorithm type, and a feature. 6. The method of claim 5,
(i) the task type is one of training, validation, and prediction, (ii) the algorithm type is one of a regression algorithm and a recursive algorithm, and (iii) the features include word embeddings. According to claim 1,
The computer-implemented method further comprising the step of performing at least one quality assessment check. According to claim 1,
receiving, via a user interface, at least one representation;
providing the at least one representation to the model;
selecting, using an application programming interface, annotation candidates associated with the at least one representation; and
and training the model based on the selected annotation candidates. According to claim 1,
During training, target truth labels and features are extracted from a training dataset and then provided to the model. A computer-implemented system for analyzing data from various data sources, comprising:
A processor comprising:
receive data from the various data sources as inputs;
transform the received data from each of the various data sources into a common data structure;
identify keywords in the received data;
generate word or sentence embeddings based on the identified keywords;
receive a selection of one or more labels based on the generated word or sentence embeddings;
add the selected one or more labels to a model;
train the model on the common data structure based on a configuration file;
to generate results in response to user questions based on the above model;
constituted, and the generating is
retrieving related documents from the received data;
determining from which of the retrieved related documents which information should be reported;
and providing the result based on the determination and a graph schema associated with the related documents. 11. The method of claim 10,
wherein the various data sources include at least one of a machine-readable document, a machine non-readable document, a spreadsheet, an image, a hypertext markup language file. 11. The method of claim 10,
wherein the processor is further configured to partition the received data into component documents, wherein the received data is partitioned based on one of a heuristic model and a trained model. 11. The method of claim 10,
The processor is
tokenize at least one of word elements and sentence elements of the received data; and
add default word embeddings to at least one of the tokenized word elements and sentence elements;
further configured, a computer-implemented system. 11. The method of claim 10,
wherein the configuration file includes instructions regarding at least one of a task type, an algorithm type, and features. 15. The method of claim 14,
(i) the task type is one of training, validation, and prediction, (ii) the algorithm type is one of a regression algorithm and a recursive algorithm, and (iii) the features include word embeddings. . 11. The method of claim 10,
and the processor is further configured to perform at least one quality assessment check. 11. The method of claim 10,
The processor is
receive, via the user interface, at least one representation;
provide the at least one representation to the model;
using an application programming interface to select annotation candidates associated with the at least one representation;
to train the model based on the selected annotation candidates.
further configured, a computer-implemented system. 11. The method of claim 10,
During training, target truth tables and features are extracted from a training dataset and then provided to the model. A computer-implemented system for analyzing data from various data sources, comprising:
application programming interface; and
including a processor;
The processor is configured to generate a result in response to a user question based on the machine learning model, wherein the generating comprises:
retrieving related documents from the received data;
determining from which of the retrieved related documents which information should be reported; and
providing the result based on the determination and a graph schema associated with the related document;
wherein the machine learning model is trained on annotation candidates provided by the application programming interface.

Images