JP2022538705A - Regular expression generation for negative examples with context - Google Patents

Abstract

Techniques for generated regular expressions are disclosed. In some embodiments, a regular expression generator can receive input data that includes one or more character sequences. A regular expression generator can convert a character sequence into a set of regular expression codes and/or span data structures. A regular expression generator can identify the longest common subsequence shared by a set of regular expression codes and/or spans and can generate a regular expression based on the longest common subsequence. Negative examples can be used to generate regular expressions. To generate regular expressions, context from negative examples can be determined.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application is related to U.S. patent application Ser. 327, entitled "AUTOMATED GENERATION OF REGULAR EXPRESSIONS," filed June 13, 2018 under 35 U.S.C. 119(e); Claiming priority to U.S. Provisional Patent Application No. 62/684,498 and under 35 U.S.C. No. 62/749,001 entitled "Regular Expression Generation". 119(e), filed Jun. 24, 2019, entitled "AUTOMATED GENERATION OF REGULAR EXPRESSIONS." 62/865,797 is claimed. their entire contents are incorporated herein by reference for all purposes.

Background Big data analytics systems can be used for predictive analytics, user behavior analytics, and other advanced data analytics. However, before any data analysis can be effectively performed to provide useful results, the initial dataset may need to be formatted into a clean and curated dataset. This data onboarding often presents a challenge to cloud-based data repositories and other big data systems where data from a variety of different data sources and/or data streams can be compiled into a single data repository. . Such data may include structured data in multiple different formats, semi-structured data according to different data models, and even unstructured data. Such data repositories often contain data representations in a variety of different formats and structures, and may contain duplicate and erroneous data. When these data repositories are analyzed for reporting, predictive modeling, and other analytical tasks, low signal-to-noise ratios in initial data sets can lead to inaccurate or unhelpful results.

Many current solutions to data formatting and preprocessing problems involve manual and ad-hoc processes to cleanse and curate data in order to manipulate it into a common format before performing data analysis. While these manual processes can be effective for certain smaller data sets, such processes are inefficient and inefficient when attempting to preprocess and format large data sets. can be practical.

Overview Aspects described herein provide various techniques for generating regular expressions. As used herein, a "regular expression" may refer to a sequence of characters that define a pattern that may be used to search for matches within longer input text strings. In some embodiments, a regular expression may be constructed using a symbolic wildcard matching language, where the pattern defined by the regular expression is matched against a string of characters and/or given as input. It may be used to extract information from character strings. Various embodiments described herein use a regular expression generator, implemented as a data processing system, to receive and display input text data and generate specific character subsets of the input text via a client user interface. , and then generate one or more regular expressions based on the selected character subset. After generating one or more regular expressions, a regular expression engine can be used to match patterns of the regular expressions against one or more data sets. In various embodiments, data matching the regular expression may be extracted, reformatted, modified, or the like. In some cases, additional columns, tables, or other data sets may be created based on the data matching the regular expression.

According to some aspects described herein, a regular expression generator implemented via a data processing system provides a determined longest length shared by different sets of one or more regular expression codes. A regular expression can be generated based on the common subsequence (LCS). Regular expression codes (which may also be referred to as category codes) may include, for example, L for letters of the English alphabet, N for numbers, Z for spaces, P for punctuation, and S for other symbols. Each set of one or more regular expression codes may be converted from a different sequence of one or more characters received as input data via the user interface. Regular expression codes that are excluded from the LCS may be expressed as optional and/or alternative. In some embodiments, a regular expression code may be associated with a minimum number of occurrences of the regular expression code. Additionally or alternatively, the regular expression code may be associated with a maximum number of occurrences of the regular expression code. For example, a set of category codes may include L<0,1> to indicate that a particular portion of the LCS contains a character at most once, if any. As explained in more detail below, generalizing the input data as an Intermediate Regular Expression Code (IREC) can offer various technical advantages, including using very little input data, which is , allows near-instantaneous generation of regular expressions that are immune to false-positive or false-negative matches in unseen data.

According to additional aspects described herein, regular expressions may be generated based on input data that includes sequences of three or more characters. If three or more character sequences are identified as input data, a regular expression generator that identifies the LCS of character sequences can provide an exponential increase in runtime. In order to identify the LCS of all character sequences in a fully functional manner, the regular expression generator may run the LCS algorithm on each distinct combination of two character sequences. Based on the results of the LCS algorithm, a fully connected graph may be generated, each graph node representing a different sequence of characters, and the length of each graph edge corresponding to the LCS of the node defining the graph edge. The order in which character sequences are selected may then be determined by performing a depth-first traversal of the minimum spanning tree over the fully connected graph.

A further aspect described herein relates to generating a regular expression based on input that includes both example positive character sequences and example negative character sequences. Positive examples may refer to sequences of characters that match the regular expression to be generated, and negative examples may refer to sequences of characters that do not match the regular expression to be generated. In some embodiments, when both positive and negative examples are received, the regular expression generator identifies a discriminator, the shortest subsequence of one or more characters that distinguishes positive from negative examples. You may The selected discriminator may be the shortest sequence (e.g., expressed as a category code) and may be either positive or negative, thus matching positive examples and not matching negative examples. become. The discriminator may then be hard-coded into the regular expression generated by the regular expression generator. In some cases, the shortest subsequence may be included in the negative example prefix or suffix portion.

A further aspect described herein relates to one or more user interfaces through which input data may be provided to generate regular expressions. In some embodiments, the user interface may be displayed on a client device communicatively coupled to the regular expression generation server. The user interface may be generated programmatically by a server, by a client device, or by a combination of software components running on the server and client. Input data received via the user interface may correspond to user selection of one or more character sequences that may represent positive or negative examples. In some cases, the user interface may support input data including selection of a first character sequence within a second character sequence. For example, a user can highlight one or more characters in a larger, previously highlighted sequence of characters, and a second user selection is for a first, larger user selection. Can provide context. This allows the input data to be provided to the regular expression generator with higher specificity, providing a "context" to the regular expression generator that produces a regular expression that avoids false positives. make it possible. A regular expression generator may generate and display a regular expression in response to a user selecting a character sequence via the user interface. For example, when a user highlights a first sequence of characters, the regular expression generator generates regular expressions that match the first sequence of characters, as well as other similar sequences of characters (e.g., the user's intent for matching sequences). ) can be generated and displayed. When the user highlights the second sequence of characters, the regular expression generator may generate an updated regular expression that includes both the first sequence of characters and the second sequence of characters. Then, when the user highlights a third sequence of characters (eg, within either the first sequence or the second sequence), the regular expression generator may update the regular expression again, and so on.

According to additional aspects described herein, a regular expression may be generated based on the longest common subsequence from one or more input sequence examples, but with characters present in only some of the examples. can also be handled. To handle characters that are only present in some input examples, a span may be defined over which both the minimum and maximum number of occurrences of the regular expression code are tracked. The minimum number of occurrences may be set to zero if spans may not exist for all given input examples. These minimum and maximum numbers can then be mapped to a regular expression multiplicity syntax. The longest common subsequence (LCS) algorithm can be run on spans of characters derived from input examples, including "optional" spans (e.g. minimum length of zero) that do not appear in all input examples. good. Consecutive spans may be merged during execution of the LCS algorithm, as described below. In such cases, when additional optional spans carried together result in successive occurrences, the LCS algorithm may be recursively performed on those optional spans as well. good.

A further aspect described herein is that the LCS algorithm performed by the regular expression generator is run multiple times to properly match all given positive examples, all and/or generate multiple correct regular expressions from which the most desirable or optimal regular expression can be selected, combinatoric search Regarding. In some embodiments, the LCS algorithm may generally be run from right to left on the input examples to generate the regular expression. However, for comparison purposes and to find alternative regular expressions, the LCS algorithm may be run separately on the input examples in reverse (eg, left-to-right direction). For example, exemplary character sequences received as user input may be reversed before they are passed through the LCS algorithm, and then the results from the LCS algorithm are reversed (including the original text fragment). You can put it back. Further, in some embodiments, the LCS algorithm is generated by the regular expression generator multiple times, positional at the beginning of the line, fixed at the end of the line, both in normal character sequence order and in reverse order. , may be executed without positioning at the beginning or end of a line. Therefore, in some cases, the LCS algorithm may be run at least these six times, and the shortest successful regular expression may be selected from these runs.

1 is a block diagram illustrating components of an exemplary distributed system for generating regular expressions in which various embodiments may be implemented; FIG. [0014] Figure 4 is a flowchart illustrating a process for generating regular expressions based on input received via a user interface, according to one or more embodiments described herein; FIG. 4 is a flowchart illustrating a process for generating a regular expression using the longest common subsequence (LCS) algorithm on a set of regular expression codes, according to one or more embodiments described herein. For generating a regular expression based on two example character sequences using the longest common subsequence (LCS) algorithm on a set of regular expression codes according to one or more embodiments described herein: 1 is an exemplary diagram; FIG. FIG. 4 is a flowchart illustrating a process for generating regular expressions using the longest common subsequence (LCS) algorithm over a larger set of regular expression codes, according to one or more embodiments described herein. For generating a regular expression based on five example character sequences using the longest common subsequence (LCS) algorithm on a set of regular expression codes according to one or more embodiments described herein: 1 is an exemplary diagram; FIG. FIG. 10 is a flow chart illustrating a process for determining the order of execution for the longest common subsequence (LCS) algorithm on a larger set of regular expression codes, according to one or more embodiments described herein; FIG. FIG. 11 illustrates a total join graph used to determine the order of execution for the longest common subsequence (LCS) algorithm on a larger set of regular expression codes, according to one or more embodiments described herein; FIG. Minimum Spanning of All Connective Graphs Used to Determine Execution Order for the Longest Common Subsequence (LCS) Algorithm on a Larger Set of Regular Expression Codes, According to One or More Embodiments Described herein Shows a tree representation. 4 is a flowchart illustrating a process for generating regular expressions based on example positive and negative character sequences, according to one or more embodiments described herein. 5 is an example user interface screen showing the generation of regular expressions based on example positive character sequences in accordance with one or more embodiments described herein; 4 is an exemplary user interface screen showing the generation of regular expressions based on example positive and negative character sequences in accordance with one or more embodiments described herein; [0014] Figure 4 is a flowchart illustrating a process for generating regular expressions based on user data selections received within a user interface, according to one or more embodiments described herein; FIG. 4 is a flowchart illustrating a process of generating regular expressions and extracting data based on capturing groups via user data selections received within a user interface, according to one or more embodiments described herein; . 5 is an exemplary user interface screen showing a tabular data display in accordance with one or more embodiments described herein; 5 is an exemplary user interface screen showing generation of regular expressions and capture groups based on selection of data from a tabular display in accordance with one or more embodiments described herein; 5 is an exemplary user interface screen showing generation of regular expressions and capture groups based on selection of data from a tabular display in accordance with one or more embodiments described herein; 4 is an exemplary user interface screen showing generation of a regular expression based on selection of positive and negative examples from a tabular display in accordance with one or more embodiments described herein; 4 is an exemplary user interface screen showing generation of a regular expression based on selection of positive and negative examples from a tabular display in accordance with one or more embodiments described herein; 5 is another exemplary user interface screen showing generation of regular expressions and capturing groups based on selection of data from a tabular display in accordance with one or more embodiments described herein; FIG. 10 is a flowchart illustrating a process for generating regular expressions, including optional spans, using the longest common subsequence (LCS) algorithm, according to one or more embodiments described herein. FIG. 4 is an exemplary diagram for generating regular expressions, including optional spans, using the longest common subsequence (LCS) algorithm according to one or more embodiments described herein; [0014] Figure 4 is a flowchart illustrating a process for generating regular expressions based on a combinatorial execution of the longest common subsequence (LCS) algorithm, according to one or more embodiments described herein; 1 is a block diagram illustrating components of an exemplary distributed system in which various embodiments of the invention may be implemented; FIG. 1 is a block diagram illustrating components of a system environment in which services provided by embodiments of the present invention may be provided as cloud services; FIG. 1 is a block diagram of an exemplary computer system upon which embodiments of the invention may be implemented; FIG. 4 illustrates a regular expression generator, according to some example embodiments; 4 illustrates a user interface for implementing a split command, according to some example embodiments; 4 illustrates a user interface for implementing a split command, according to some example embodiments; 4 illustrates a user interface displaying the results of a split command on a dataset, according to some example embodiments; 4 illustrates a flow chart of a method for executing a split command, according to some exemplary embodiments; 4 illustrates a user interface for implementing a delete command, according to some exemplary embodiments; 4 illustrates a user interface displaying the results of a delete command on a dataset, according to some example embodiments; 4 depicts a flowchart of a method for executing a delete command, according to some exemplary embodiments; 4 illustrates a user interface for implementing obfuscation commands, according to some example embodiments; 4 illustrates a user interface displaying results of an obfuscation command on a dataset, according to some example embodiments; 4 illustrates a flowchart of a method for executing obfuscation commands, according to some example embodiments; 4 illustrates a user interface for implementing replacement commands, according to some exemplary embodiments; 4 illustrates a user interface for implementing replacement commands, according to some exemplary embodiments; 4 illustrates a user interface displaying the results of a replace command in a dataset, according to some example embodiments; 4 depicts a flow chart of a method for executing a replacement command, according to some exemplary embodiments; 4 illustrates a user interface for implementing row filtering commands, according to some example embodiments; 4 illustrates a user interface for implementing row filtering commands, according to some example embodiments; 4 illustrates a user interface displaying results of a row filtering command on a dataset, according to some example embodiments; 4 illustrates a flowchart of a method for executing row filtering commands, according to some exemplary embodiments; 4 illustrates a user interface displaying a view of a dataset in single-level mode, according to some example embodiments; 4 illustrates a user interface displaying highlighted data in a nested full control mode, according to some example embodiments; 4 illustrates a user interface displaying highlighted data in a nested full control mode, according to some example embodiments; 4 illustrates a user interface for providing an example, according to some example embodiments; 4 illustrates a user interface displaying updated generated regex, according to some example embodiments; 4 illustrates a user interface displaying alternative data highlighting, according to some example embodiments; 4 illustrates a user interface displaying updated generated regex, according to some example embodiments; 4 is a flowchart of a method for performing multiple highlighting, according to some example embodiments; 4 illustrates a flowchart of a method for providing negative examples, according to some exemplary embodiments; 4 depicts a flow chart of a method for determining context from negative examples, according to some example embodiments. FIG. 4 illustrates a diagram for generating regular expressions based on span highlighting alignment, according to some example embodiments; 4 illustrates a flow chart of a method for performing span highlight alignment, according to some exemplary embodiments; 4 illustrates a flowchart of a method for tracking spans, according to some example embodiments; 4 illustrates a user interface displaying punctuation spans and symbol spans, according to some example embodiments;

DETAILED DESCRIPTION In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the invention. However, it will be apparent to one skilled in the art that embodiments of the invention may be practiced without some of these specific details. In other instances, well-known structures and devices are shown in block diagram form.

The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It is to be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the invention as set forth in the claims.

Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Additionally, it is noted that individual embodiments can be described as processes depicted as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. Additionally, the order of operations may be rearranged. A process is terminated when its operations are completed, but may include additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, or the like. Where a process corresponds to a function, its termination may correspond to the function returning to its calling or main function.

The term "computer-readable medium" means such media as portable or fixed storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and/or data. Including, but not limited to, non-transitory media. Code segments or computer-executable instructions may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and/or receiving information, data, arguments, parameters or memory content. Information, arguments, parameters, data, etc. may be passed, transferred, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, and the like.

Furthermore, embodiments may be implemented in hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. If implemented in software, firmware, middleware, or microcode, the program code or code segments that perform the required tasks may be stored on the machine-readable medium. A processor may perform the necessary tasks.

Described herein are various techniques (e.g., methods, systems, techniques executable by one or more processors) for generating regular expressions that correspond to patterns identified in one or more example input data. (such as a non-transitory computer readable storage memory for storing instructions for In certain embodiments, in response to receiving a selection of input data, one or more patterns in the input data are automatically identified, and a regular expression (or "regex" for short) is generated to represent the identified pattern. ”) can be automatically and efficiently generated. Such patterns can be based on sequences of characters (eg, sequences of letters, numbers, spaces, punctuation marks, symbols, etc.). Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

In some embodiments, regular expressions may be constructed using a symbolic wildcard matching language to match character strings and/or extract information from character strings provided as input. .

Input data received by the regular expression generation system may include, for example, one or more "positive" data examples and/or one or more "negative" data examples. As used herein, a positive example may refer to a character sequence received as input and to be matched by a regular expression generated based on that input. A negative example, on the other hand, may refer to an input character sequence that would not be matched by a regular expression generated based on that input.

Several technical advantages may be realized within the various embodiments and examples described herein. For example, some techniques described in this disclosure may improve the speed and efficiency of the regular expression generation process (eg, a Regex solution may be generated in less than a second and the user interface is suitable for interactive real-time use). obtain). The various techniques described herein may also be deterministic, may not require training data, may generate solutions without requiring any initial regular expression input, It may be fully automated (eg generate regular expressions to the extent that it requires any human intervention). Moreover, the various techniques described herein need not be limited with respect to the types of data input that can be effectively processed, and such techniques improve the human readability of the resulting regular expressions. can.

Some embodiments described herein include one or more implementations of the longest common subsequence (LCS) algorithm. The LCS algorithm can be used in some situations as a diff engine (eg, the engine behind the Unix "diff" utility) configured to determine and show differences between two text files. In some embodiments, input data (eg, strings and other character sequences) may be converted into abstract tokens, which may then be provided as input to the LCS algorithm. Such abstract tokens may, for example, be tokens based on regular expression codes representing regular expression character classes (eg, Loogle codes or other character class codes). A variety of different examples of such code are possible and may be referred to herein as "regular expression code" or "intermediate regular expression code" (IREC). For example, the input character sequence "May 3" may be converted to the IREC code "LLLZN", after which the tokenized string may be subjected to the LCS algorithm along with other tokenized strings. . In some embodiments, IRECs (eg, regular expression codes) that the input character sequences do not have in common appear as optional (eg, optional spans) in the final generated regular expression. good. In certain embodiments, the regular expression code may be a category code based on the Unicode category codes shown at https://www.regular-expressions.info/unicode.html#category, or http://unicode May be a general category property code from .org/reports/tr18/#General_Category_Property. For example, code L may represent a letter, code N may represent a number, code Z may represent a space, code S may represent a symbol, code P may represent a punctuation mark. good etc.

Additionally, these different categories may be disjoint or mutually exclusive. That is, in this example, categories L, N, Z, P and S may be made disjoint so that there is no overlap between members of the categories.

Additional technical advantages may be realized in various embodiments, including more efficient generation of regular expressions based on the use of regular expression codes (eg, category codes), spans, and the like. By using such codes, computational resources need not be wasted if the LCS algorithm successfully identifies all or substantially all of the characters in the input string as being different. Further technical advantages provided by various embodiments herein are improved readability of generated regular expressions, and support for both positive and negative examples as input data, and various advantageous Including providing user interface features (eg, allowing the user to highlight larger character sequences or text fragments within data cells for extraction).

I. General Overview Various embodiments disclosed herein relate to the generation of regular expressions. In some embodiments, a data processing system configured as a regular expression generator generates regular expressions by identifying the longest common subsequence (LCS) shared by different sets of regular expression codes (e.g., category codes). can be generated. Each set of regular expression codes can be converted from a sequence of characters received as input data via the user interface. Among the technical advantages described here are the ability to efficiently generate regular expressions using very little input data by abstracting the input data as intermediate code (e.g. regular expression code, spans, etc.) can do.

FIG. 1 is a block diagram illustrating components of an exemplary distributed system for generating regular expressions in which various embodiments may be implemented. As shown in this example, the client device 120 communicates with the regular expression generation server 110 (or regular expression generator) and interacts with the user interface to retrieve and display tabular data and input data through the user interface. A regular expression can be generated based on the selection (eg, example). In some embodiments, client device 120 may access regular expressions via client web browser 121 and/or client-side regular expression application 122 (eg, a client-side application that receives/consumes regular expressions generated by server 110). may communicate with the generator 110; Within regular expression generator 110 , requests from client device 120 may be received over various communication networks at a network interface and processed by an application programming interface (API), such as REST API 112 . The user interface data model generator 114 component along with the regular expression generator 110 can provide server-side programming components and logic to generate and render various user interface features described herein. Such features include a user retrieving and displaying tabular data from the data repository 130, selecting example input data to initiate generation of regular expressions, modifying and/or modifying data based on the generated regular expressions. Or it may contain functions that allow it to be extracted. In this example, the regular expression generator component 116 converts the input character sequence into regular expression codes and/or spans, runs an algorithm (eg, LCS algorithm) on the input data, and generates/ It can be implemented to generate regular expressions, including simplification. The regular expression generated by the regular expression generator 116 may be sent by the REST service 112 to the client device 120, where the Javascript code on the client browser 121 (or corresponding client-side application component 122) then: Regular expressions can be applied to all cells within a spreadsheet column rendered within the browser. In other cases, to identify matched/unmatched data on the server side, a separate regular expression engine component can be implemented on the server side to store the generated regular expressions in a table displayed on the user interface. Comparisons may be made within formal data and/or other data stored in data repository 130 . In various embodiments, matching/non-matching data may be automatically selected (eg, highlighted) within the user interface, selected for extraction, modification, deletion, and the like. Any data extracted or modified via the user interface based on the generation of regular expressions may be stored in one or more data repositories 130 . Additionally, in some embodiments, the generated regular expressions (and/or corresponding inputs to the LCS algorithm) may be stored in the regular expression library 135 for future retrieval and use. In some embodiments, the generated regular expressions need not actually be stored in a 'library', but may be incorporated into a 'conversion script'. For example, ET. S. As described in more detail in US Pat. No. 10,210,246 (herein incorporated by reference for all purposes), such a conversion script is used to convert received data. or may include a program, code, or instructions that may be executable by multiple processing units. Other possible examples of conversion scripts may include "rename column", "capitalize string data", or "guess gender from first name and create new column with gender", and the like.

FIG. 2 is a flowchart illustrating a process 200 for generating regular expressions based on input received via a user interface, according to one or more embodiments described herein. At step 201, the regular expression generator 110 may receive a request from the client device 120 to access the regular expression generator user interface and view specific data through the user interface. The request in step 201 may be received via REST API 112 and/or a web server, authentication server, etc., and the user's request may be parsed and authenticated. For example, a user within a business or organization may be able to analyze and/or modify transaction data, customer data, performance data, forecast data, and/or any other category of data that may be stored in the organization's data repository 130. , can access the regular expression generator 110 . At step 202, the regular expression generator 110 can retrieve and display the requested data via a user interface that supports the generation of regular expressions based on selected input data. Various embodiments and examples of such user interfaces are described in detail below.

At step 203 , the user may select one or more input character sequences from data displayed in the user interface provided by regular expression generator 110 . In some embodiments, data may be displayed in a tabular format within the user interface, including labeled columns having specific data types and/or categories of data. In such cases, the selection of input data in step 203 may correspond to the user selecting a data cell or selecting (eg, highlighting) an individual text fragment within the data cell. However, in other embodiments, the regular expression generator 110 may support searching and displaying semi-structured and unstructured data via a user interface, allowing users to create semi-structured or unstructured data Input data for regular expression generation may be selected by selecting a character sequence from . A user selecting an input character sequence from displayed tabular data, as described in the examples below, is just one example of a use case. In another example, a user (e.g., perhaps a software developer or power user trying to build a regular expression for the Linux command-line tools grep, sed, or awk, etc.) can extract examples from a spreadsheet. You can type it from scratch in the example instead of picking it up.

At step 204 , regular expression generator 110 may generate one or more regular expressions based on the input data selected by the user at step 203 . At step 205, the regular expression generator 110 updates the user interface, for example, to display the generated regular expression and/or highlight matching/non-matching data within the displayed data. You may At step 206, which may be optional in some embodiments, the user interface may support functionality that allows the user to modify the underlying data based on the generated regular expression. For example, the user interface features allow users to filter, modify, delete, or extract specific data fields from tabular data based on whether those fields match regular expressions. may be supported. Filtering or modifying the data can include modifying the underlying data stored in repository 130, and in some cases presenting the extracted data to repository 130 as new columns and/or new tables. can be stored.

Although these steps provide a general and high-level overview of exemplary user interactions with the user interface of regular expression generator 110, other embodiments support various additional features and functionality. good too. For example, in some embodiments, regular expression codes (or category codes) may be associated with a minimum number of occurrences of the code. Additionally or alternatively, a regular expression code may be associated with a maximum number of occurrences of the code. As an example, the set of regular expression codes may include the code L<0,1> to indicate that a particular portion of the LCS contains a character at least 0 times, and at most 1 time.

Additionally, in some embodiments, input data may include more than two character sequences. In such embodiments, various techniques may be used to determine the order for performing the LCS algorithm on sequences of three or more characters, and the resulting regular expression generated in a well-functioning manner. In this way, the exponential increase in runtime caused by input character sequences of three or more can be avoided. Note that the regular expression generator 110 may alternatively run the LCS algorithm on two character sequences at a time and determine the order for selecting character sequence pairs based on the graph. For example, the fully connected graph indicates that a first run of the LCS algorithm (eg LCS1) should be run for sequence 1 and sequence 3, then a second run of the LCS algorithm (eg LCS2 ) may indicate that LCS1 and sequence 2 should be executed, and so on. The graph may be a fully connected graph, with nodes representing character sequences and edges connecting nodes to represent the length of the LCS shared by the connected nodes. Each node in the graph may be connected to every other node in the graph, and the order for selecting character sequences may be determined by performing a depth-first traversal of the minimum spanning tree over the graph. .

In further embodiments, input data may be provided via the user interface in a number of different ways. For example, input data may indicate a first user selection of one or more characters within a second user selection of a set of characters. For example, a user may highlight a character within a previously highlighted set of characters. Thus, the second user selection may provide context for the first user selection, which may allow input data to be provided to regular expression generator 110 with greater specificity. In some embodiments, the regular expression generator 110 can generate and display regular expressions in near real-time in response to each user selection. For example, if the user highlights a first range of characters, regular expression generator 110 may display a regular expression representing the first range of characters. Then, when the user highlights a second range of characters within the first range of characters, the regular expression generator 110 may update the displayed regular expression.

Further, in some embodiments, the regular expression generator 110 can generate regular expressions based on input containing both positive and negative examples. As noted above, positive examples may refer to sequences of characters that should be included in the regular expression, and negative examples may refer to sequences of characters that should not be included in the regular expression. In such cases, regular expression generator 110 can identify the shortest subsequence of one or more characters that distinguishes positive from negative examples at a particular position. The shortest subsequence may then be hard-coded within the regular expression generated by regular expression generator 110 . In various examples, the shortest subsequence may be included in the prefix/suffix portion, or the midspan within the negative example.

Further examples of automatically generating regular expressions according to certain embodiments are described below. These examples may correspond to various specific possible implementations of the general technique of FIG. 2, and software executed by one or more processing units (eg, processors, cores) of the respective system (eg, code, instructions, programs, etc.), hardware, or a combination thereof. The software may be stored on non-transitory storage media (eg, on memory devices). Further examples described below are intended to be illustrative and non-limiting. Although these examples show various processing steps occurring in a particular order or order, this is not meant to be limiting. In some alternative embodiments, the steps may be performed in some different order, or some steps may be performed in parallel.

In some examples, user input received via a user interface (eg, step 203) includes one or more "positive examples" matched by the regular expression output and zero or more not matched by the regular expression output. It can include more "negative cases". Optionally, one or more of the positive examples can be highlighted to select a particular range (or subsequence) of characters. Optionally, at step 204, positive examples received via the user interface may be converted to spans of regular expression codes (eg, character category codes such as Unicode category codes). A sequence of spans may be generated for each positive example. In some embodiments, each vertex corresponds to one of the sequences of spans, and the edge weights from the LCS algorithm run on those two sequences of spans correspond to the endpoints of the edges. A graph may be generated that is equal to the length of the output. A minimum spanning tree can be determined for a graph. For example, in some embodiments, Prim's algorithm may be used to obtain the minimum spanning tree. A depth-first traversal may be performed on the minimum spanning tree to determine the traversal order, and then the LCS algorithm may be performed on the first two elements of the traversal. Then, one by one, each additional element of the traverse may be merged into the current LCS output in order by running the LCS algorithm again on the output of the previous LCS iteration and the next current traversal element. . The final output of the LCS algorithm, which can be a sequence of spans, may then be converted to regular expressions. This transformation may be a one-to-one transformation in some embodiments, but certain optional embodiments described herein may not correspond to a one-to-one transformation. Finally, the resulting regular expression may be tested against all positive and negative examples received via the user interface at step 203 . If any of the tests fail, the above process may be repeated using all positive examples and any negative examples that fail.

II. Regular Expression Generation Using the Longest Common Subsequence Algorithm on Regular Expression Codes As noted above, some aspects described herein provide a longest common subsequence algorithm shared by different sets of regular expression codes corresponding to input data. It relates to the generation of regular expressions based on subsequence (LCS) computation.

FIG. 3 is a flowchart illustrating a process 300 for generating regular expressions using the LCS algorithm on a set of regular expression codes, according to one or more embodiments described herein. At step 301, regular expression generator 110 may receive one or more character sequences as input data. As noted above, in some examples the input data may correspond to positive case data selected from within tabular data displayed in the user interface, although in some embodiments the user interface may optionally Yes, and it should be appreciated that the input data may correspond to any character sequence received over any other communication channel (eg, non-user interface) in various examples.

At step 302, each character sequence received at step 301 may be converted to a corresponding regular expression code. In various embodiments, the regular expression code may be a Loogle code, a Unicode category code, or any other character class code representing a regular expression character class. For example, one input character sequence "May 3" may be converted to the Loogle code "LLLZN". In some embodiments, the regular expression code may be a category code based on the Unicode category codes shown at https://www.regular-expressions.info/unicode.html#category. For example, code L may represent a letter, code N may represent a number, code Z may represent a space, code S may represent a symbol, code P may represent a punctuation mark. good, etc.

At step 303, the longest common subsequence among the set of regular expression codes generated at step 302 can be determined. In some embodiments, the LCS algorithm may be run using two sets of regular expression codes as input. Various different characteristics of the implementation of the LCS algorithm (eg, processing direction, positioning, blank pushing, coalescing of low-density spans, alignment on common tokens, etc.) may be used in different embodiments. At step 304, a regular expression can be generated based on the output of the LCS algorithm. Optionally, step 304 may include capturing the output of the LCS algorithm in regular expression code and converting the regular expression code to a regular expression. At step 305, the regular expression may be simplified and output, for example, by displaying the regular expression to the user via a user interface.

FIG. 4 is an exemplary diagram for generating a regular expression using the Longest Common Subsequence (LCS) algorithm on a set of regular expression codes based on two example character sequences. FIG. 4 thus shows an example of applying the process described above in FIG. As shown in FIG. 4, the regular expression in this example is generated based on two input strings "iPhone 5" and "iPhone X". Each sequence in this example can be converted to a respective set of regular expression codes. Therefore, iPhone 5 may be converted to "LLLLLLZN" and iPhone X may be converted to "LLLLLLZL". As shown in FIG. 4, these category codes are then provided as input to the LCS algorithm, which determines that both sets of IRECs (or category codes) contain six Ls and one Z, I judge. Z-category codes excluded from the LCS may be expressed as optional and/or alternative.

The curly brackets containing the digit 6 indicate 6 instances of the letter and the question mark indicates that the last digit/letter is optional. Finally, a simplification process can be performed by the regex generator, during which the regex is simplified by inserting the common text fragment "iPhone" back into the final regex, resulting in a wider regex

replace part.
As shown in this example, an input string received by regular expression generator 110 may be converted into a "regular expression code" representing a regular expression broad category (which may also be referred to as a "category code"). Well, the LCS algorithm may be run on those regular expression codes. In some embodiments, Unicode category codes may be used for regular expression codes. For example, the input text string is Regex Unicode Broad Category

can be converted into a code representing This approach illustrated by FIGS. 3 and 4 may be referred to as the indirect approach. However, in other embodiments a direct approach may be used in which the LCS algorithm is run directly on the character sequence received as input.

In some embodiments, the indirect approach may offer additional technical advantages in that it does not require large amounts of training data and produces effective regular expressions with a relatively smaller number of input examples. obtain. This is because the indirect approach uses heuristics to reduce uncertainty in regular expression generation and to eliminate potential false positives and false negatives. For example, in generating a regular expression based on the input strings "May 3" and "Apr 11", a direct approach would require at least one instance per month to generate a valid regular expression that matches the date pattern. may require Relying only on those two examples, a direct approach may generate a regex of "[AM][ap][yr] [13]1?". In contrast, the indirect approach is based on the Unicode broad category,

may generate a more efficient regular expression for Additionally, as noted above, one of the technical advantages described herein is the ability to efficiently generate regular expressions using very little input data, possibly even from a single example. including doing For example, regarding the generation of a regular expression from a single instance 'am', one heuristic is to generate 'am' for the regular expression

can be determined to generate Both are probably correct, but programmed heuristics may use user preferences and/or criteria to determine how to best generate the regular expression (e.g., whether it should also match "pm"). It can be realized.

In addition, the indirect approach also allows the generated regular expression

can be simplified to make it more human readable. This may be beneficial in some embodiments, for example, when outputting to non-sophisticated regular expression users who may not be familiar with the Unicode representation for regular expressions.

Further, in some embodiments, instead of treating each character independently when performing the LCS algorithm, sequential and equal regular expression codes may be converted into span data structures (which may also be referred to as spans). In some cases, a span may include a representation of a single regular expression code (eg, Unicode broad category code) along with a repeat count range (eg, minimum and/or maximum number). Conversion from regular expression code to spans can facilitate a number of different additional features described below, such as recognizing alternations (e.g. separation), and also the generated To further simplify the regular expression, merging of adjacent optional spans can also be facilitated.

As mentioned above, the LCS algorithm remembers underlying text fragments in the input character sequence that can potentially be inserted back into the final regular expression, such as the string "iPhone" in FIG. , can be configured to hold By keeping track of the text fragment that originally resulted in the assigned category code for that span, such an embodiment allows literal text (e.g. am and pm) to be included directly in the generated regular expression. , can reduce false positives and make the regular expression output more human readable.

In some embodiments, certain heuristics can be used to determine when string literals should be output in the generated regular expression as opposed to broadly matching regular expression code. As mentioned above, in some cases it may be desirable for the regular expression to require an exact match against the string fragment. Therefore, in some embodiments, some heuristic may be used to determine whether the correct string fragment is output to the regular expression. For example, one heuristic is that if only one particular string fragment (e.g., "pm") has ever been encountered for a particular span, and if there are more than one instance of that span, then the exact It may be determined that the string fragment is output to the regular expression. For punctuation spans, the heuristic may lower the threshold to only one occurrence for that span (eg, based on the assumption that punctuation is less likely to change). for example,

For both , the threshold may be lowered to only one occurrence to output the literal string within the regular expression. The heuristics may be modified and/or adjusted to be somewhat restrictive based on the threshold number of examples required to output the correct string fragment to the regular expression. Note that if the heuristic errs in being overly restrictive (by outputting literal string fragments into the regular expression too easily), the user may compensate by introducing additional positive examples. want to be Similarly, if the heuristic error errs in being overly permissive (by outputting too often broad regular expression code), the user may compensate by introducing additional negative examples. Heuristics can be initially programmed (and subsequently adjusted) with appropriate amounts of limits based on previous user interaction and experimentation.

As described above with reference to FIG. 4, the simplification process may be performed by regular expression generator 110, during which the regular expression may be simplified using various techniques. In some cases, simplification may involve replacing long regular expression repeat codes (eg, using curly braces) with regular expression short codes (eg, *, +, and ?). For example, if the span comes from an example string fragment that represents repetitions between, say, 0 and 3 (minimum and maximum), then the regular expression generator 110 may, to prevent false negatives and improve readability: You can simplify the expression by outputting * instead of {0,3} in the regular expression. Further, instead of frequently using the {a,b} syntax, the regular expression generator 110 may instead use the + syntax if a>2 or b>4. This makes the generated regular expression more readable and increases the chances of avoiding false negatives.

In some cases, a bank of predefined character ranges may be used. For example, the regular expression Unicode character category for characters is

, which may not be well known or readily readable compared to older character ranges such as [A-Z]. Thus, for English input, the regular expression generator 110 converts [A-Z], [a-z], and [A-Za-z] to

may be made if a given example can be successfully matched with one of those substitutions.

Additionally, regular expression generator 110 may be configured in some embodiments to default to a maximum tolerance for whitespace in regular expressions. As described herein, the regular expression generator 110 may, for example, prefer literal string fragments to extensive matching code in some cases, and apply minimum and maximum bounds to repeated output within curly braces. can be configured to generate more specific regular expressions. However, for whitespace, the regular expression generator 110 may be configured to default to a maximum tolerance in some embodiments. That is, for any white space (eg, even a single space that appears between words), the regular expression generator 110 converts it to the regular expression category code

, and repeats may be specified to be +, which means one or more. Thus, such an embodiment can anticipate and match extra white space, or even tabs and carriage returns that can be matched exactly (eg, as in completely unstructured text).

In some cases, repetitions within the generated regular expression may expand into literally repeating regular expression code.

For example, the regular expression generator 110 may generate Unicode category codes that are two characters in length (e.g.

) and the number of repetitions is 4 or less, or the Unicode category code is 3 characters long (e.g.

) and the repetitions may be expanded if the number of repetitions is 3 or less, or if the length of the Unicode category code is greater than 3 characters and the number of repetitions is 2 or less. Additionally, in some embodiments, if a literal character needs to be output (eg, a regular expression "special character"), the regular expression generator 110 may be configured to escape it.

In various additional embodiments, regular expression generator 110 detects various types of paired parentheses (e.g., { and }) and replaces negated character classes (e.g., [^}]*) with can be configured to generate a regular expression with For example, within text highlighted (or otherwise selected) by the user, or within positive/negative examples or other input to the system, an open or closed bracket is detected and matched with the corresponding bracket. may be connected or paired. In some examples,

Several different types of brackets may be detected in the input text, including some or all of the . So, as an example, for HTML tags you might generate <[^>]*> instead of something like <[a-z]+(?: [a-z]+=[a-z]+>. When detecting and pairing parentheses of sorts, the regular expression generator 110 may be configured to properly detect and pair parentheses nested within parentheses or highlighted (or Otherwise, the regular expression generator 110 may be configured to ignore parentheses that overlap the (user-selected) portion, and in some cases, the regular expression generator 110 may add a new negation between parentheses if the content is unique. i.e., if the content is always all the same (e.g., continuing the HTML example, if all the tags identified are <h1>), the literal is negated may be output in the generated regular expression instead of the bracketed character class (e.g. just output <h1> instead of <[^>]*>).

In an exemplary embodiment, collapsing within parentheses may be allowed when the highlighting is just contained by the parentheses. This allows extraction of parenthetical annotations. For example, to extract Windows NT from (Windows NT), the generated regex would be

It will be more common and result in fewer false negatives.
III. Regular Expression Generation Using the Longest Common Subsequence Algorithm on Regular Expression Code Combinations Further aspects described herein are based on input data comprising three or more strings (e.g., three or more distinct character sequences) Concerning regular expression generation. If more than two strings are identified as input data, the regular expression generator 110 may use a performance optimization feature that determines the optimal order for the sequence of LCS algorithm executions. As described below, performance optimization functions for three or more strings can be based on the size of the vertices corresponding to each string and the LCS output between each string and all other strings edge length/ constructing a graph with the weights. A minimum spanning tree may then be derived using those edge weights, and a depth-first traversal may be performed to determine the order of the input strings. Finally, using the determined input string order, a series of LCS algorithms may be performed.

FIG. 5 is a flowchart illustrating a process 500 for generating regular expressions using the longest common subsequence (LCS) algorithm for larger sets of regular expression codes (eg, sequences of three or more characters). is. Accordingly, steps 502-505 in this example may correspond to step 303 described above in FIG. However, since this example relates to generating regular expressions based on more than two input character sequences, the LCS algorithm may be run multiple times. For example, to avoid an exponential increase in runtime for more than two input strings, the LCS algorithm may be run multiple times, each run for only two input strings. For example, regular expression generator 110 may perform a first run of the LCS algorithm on two strings (e.g., two input character sequences or two transformed regular expression codes), then a first A second execution of the LCS algorithm may be performed on the output of the LCS algorithm and the third string, and then a third execution of the LCS algorithm on the output of the second LCS algorithm and the fourth string. and so on.

To improve and/or optimize the performance of such embodiments, it may be desirable to determine the optimal order of input strings (e.g., input character sequences or regular expression codes) to run a sequence of LCS algorithms. . For example, good order for fetching input strings can affect the readability of generated regular expressions, such as by minimizing the number of optional spans. To keep the generated regex concise, we note that the additional strings that are LCS'd to the current regex already have some similarity to the current regex (the intermediate result from LCSing already seen strings). preferable.

Thus, in step 501, multiple (eg, three or more) input character sequences are converted into regular expression codes. At step 502, the LCS algorithm is used to determine the order in which to process the regular expression code. The determination of the order in step 502 is described below with reference to FIG. In step 503, either the first two regular expression codes in the determined order are selected (for the first iteration of step 503) or the next regular expression code in the determined order is selected (following step 503). ) is selected for iterations of In step 504, the LCS algorithm is run on two input strings corresponding to the regular expression code format. In the first iteration of step 504, the LCS algorithm is performed on the first two regular expression codes in the determined order, and on subsequent iterations of step 504, the LCS algorithm is performed on the next It is run on the regular expression code and the output of the previous LCS algorithm (which may be in the same regular expression code format). At step 505, regular expression generator 110 determines whether there are additional regular expression codes in the determined order that have not yet been provided as input to the LCS algorithm. If there are such additional regular expression codes in the determined order, processing returns to step 503 for another execution of the LCS algorithm. Otherwise, in step 506 a regular expression is generated based on the output of the last run of the LCS algorithm.

FIG. 6 is an exemplary diagram for generating a regular expression based on five example input character sequences. In this example, each input character sequence is converted to a regular expression code, and then the LCS algorithm is repeatedly performed based on the determined order of the regular expression codes. FIG. 6 thus shows an example of applying the process described above in FIG. In this example, the determined order for the five regular expression codes is Code #1 through Code #5, and each code is input to the LCS algorithm in the determined order to produce the regular expression output. The final regular expression output (Reg Ex#4) corresponds to the final regular expression generated based on all five of the input character sequences.

FIG. 7 is a flowchart illustrating a process for determining execution order for the longest common subsequence (LCS) algorithm on a larger set (eg, three or more) of regular expression codes. Thus, as shown in this example, steps 701-704 may correspond to the order determination in step 502 above. In step 701, the LCS algorithm may be run on each distinct pair of regular expression codes corresponding to the input data, and the resulting output LCS may be stored for each run. Thus, for k input data, this is all (k(k−1))/2 possible pairings of strings run through the LCS algorithm, or in some embodiments k(k− 1). For example, if k=3 input character sequences are received, the LCS algorithm may be executed three times in step 701; if k=4 input character sequences are received, the LCS algorithm is It may be run 6 times; if k=5 input character sequences are received, the LCS algorithm may be run 10 times in step 701, and so on. In step 702, a fully connected graph may be constructed from k nodes where the edge weights of (k(k−1))/2 edges represent a string with the length of the raw LCS output between the two nodes. . At step 703 , a minimum spanning tree may be derived from the total connectivity graph at step 702 . At step 704, a depth-first traversal may be performed on the minimal spanning tree. The output of this traversal may correspond to the order in which the regular expression code is entered into the sequence of LCS algorithm executions.

8A and 8B, FIG. 5 shows an example of a fully connected graph generated based on a received k=5 input character sequence, and FIG. 8B shows a minimum spanning graph for the fully connected graph. A tree representation is shown.

In some embodiments, the approaches described in FIGS. 5-8B may provide additional technical advantages in terms of performance. For example, certain conventional implementations of the LCS algorithm can exhibit O(n ² ) runtime performance, where n is the length of the string. Extending such an implementation to k strings instead of just 2 can yield exponential run-time performance O(n ^k ), since the LCS algorithm can be asked to search a k-dimensional space. It is from. Such conventional implementations of LCS algorithms may not perform well or be well suited for real-time online user experiences.

As mentioned above, the LCS algorithm may be run (k(k−1))/2 times, where sometimes the overlap is exactly the same as seen before, because the LCS algorithm may be the case when raw input examples from the user are converted to Regex category codes. Thus, in some cases memorization may be implemented and a cache may be used to map a previously seen LCS problem to a previously worked LCS solution.

Certain exemplary embodiments also provide an effective method of controlling cache size when the cache table is too large. Part of the cache may be deleted. For example, a hashmap table that can be truncated can be used. Hashmap tables can be arbitrarily truncated.

IV. Generating Regular Expressions Based on Positive and Negative Pattern Matching Examples A further aspect described herein relates to generating regular expressions based on input data corresponding to both positive and negative examples. As noted above, positive examples may refer to an input data character sequence that is specified as a string of examples to match the regular expression generated by the regular expression generator. Negative examples, on the other hand, may refer to an input data character sequence that is specified as a string of examples that should not match the regular expression generated by the regular expression generator. As described below, in some embodiments, the regular expression generator 110 may be configured to identify a position and the shortest subsequence of characters that distinguish positive examples from negative examples at that position. The shortest subsequence can then be hard-coded into the generated regular expression such that positive examples will be matched by the regular expression and negative examples will be excluded (eg, not matched) by the regular expression.

FIG. 9 is a flowchart illustrating a process 900 for generating regular expressions based on positive and negative example character sequences. At step 901, regular expression generator 110 may receive one or more input data character sequences corresponding to positive examples. At step 902, regular expression generator 110 may generate a regular expression based on the received positive examples. Accordingly, steps 901-902 may include some or all of the steps performed in Figures 3 or 5 discussed above to generate a regular expression based on an input data character sequence.

At step 903, regular expression generator 110 may receive one additional input data character sequence corresponding to a negative example. Therefore, negative examples are specified specifically so that they do not match the regular expression generated in step 902 . In some embodiments, the negative examples received in step 903 may first be tested against the regular expression generated in step 902, and if it is determined that the negative examples do not match the regular expression, No further action is taken. However, in this example it may be assumed that at least one of the negative examples received in step 903 matches the regular expression generated in step 902 . Accordingly, step 904 may determine the disambiguation position within the regular expression generated in step 902 . In some embodiments, the disambiguation position may be selected as either a prefix position (eg, at the beginning of the regular expression) or a suffix position (eg, at the end of the regular expression). For example, the regular expression generator 110 determines the first number of characters that would be needed in the prefix to distinguish positive from negative examples and the number of characters that would be needed in the suffix to distinguish positive from negative examples. A second number of characters that will be played may be determined. Regular expression generator 110 may then select a suffix or prefix based on the minimum number of replacement characters required. In some cases, using prefixes as disambiguating locations may be preferred (eg, weighted) for readability. In yet another example, the disambiguation positions may be midspan positions that do not correspond to regular expression prefixes or suffixes.

In step 905, the regular expression generator 110 generates a replacement sequence of custom character classes (or, more specifically, a custom character class) that, when inserted into the regular expression at the determined position, can distinguish positive from negative examples. sequence of the "bracket" regular expression character class). In some embodiments, in step 905, regular expression generator 110 retrieves text fragments corresponding to disambiguation positions (or replacement positions) from each of the positive and negative examples, and then replaces those text fragments with can be used to determine discriminators that are used as permutation sequences to distinguish positive from negative examples. Further, the discriminator replacement sequence determined in step 905 can include multiple different replacement sequences for the custom character class, which are replaced either at the same position or at different positions within the regular expression. can be done.

As noted above, in some cases the determination of permutation sequences in step 905 may be performed in conjunction with the determination of disambiguation positions (or permutation positions) in step 904 . For example, regular expression generator 110 may determine one or more replacement sequences that may distinguish positive from negative examples at the first possible replacement position. Regular expression generator 110 may also determine one or more other permutation sequences that may distinguish positive from negative examples at a second different possible permutation position. In this example, when choosing between different possible replacement positions and corresponding replacement sequences, the regular expression generator 110 applies a heuristic formula to determine the size of characters at the replacement positions as well as the number and number of corresponding replacement sequences. The selection can be performed based on one or more of: size. Finally, at step 906, the regular expression may be modified by inserting one or more determined replacement sequences at the determined positions to replace previous portions of the regular expression. Optionally, following modification of the regular expression in step 906, the positive and/or negative examples are tested against the modified regular expression such that positive examples match the regular expression and negative examples match the regular expression. You may check that it does not match with

10A and 10B are exemplary user interface screens showing the generation of regular expressions based on example positive and negative character sequences. Accordingly, the examples shown in FIGS. 10A and 10B may correspond to user interfaces displayed during execution of the process of FIG. 9 described above. In FIG. 10A, a user provides three positive examples 1001 of a data input character sequence, and regular expression generator 110 generates regular expressions 1002 that match each of the positive examples. Next, in FIG. 10B, the user provides one negative example 1004 and the regular expression generator 110 generates a modified regular expression based on both the current set of positive examples 1003 and the current set of negative examples 1004. 1005 is generated.

As noted above, in some embodiments, when both positive and negative examples are received, the regular expression generator 110 generates a discriminator, i.e., one or more A shortest subsequence of characters may be identified. The selected discriminator may be the shortest sequence (e.g., represented by a category code) and may be either positive or negative, with positive examples matching and negative examples not matching. . In some cases, the discriminator may correspond to a replacement subsequence that can then be hard-coded into the regular expression at step 905 . As an example, in "[AL][a-z]+", [AL] is equivalent to "Alley," "Avenue," and "Lane," assuming it applies to street suffixes. )” are positive discriminators that will match (or allow them) but will not match (or disallow) all others. As another example, in "[BC][o][a-z]+", [BC][o] is two character classes that would match "Boulevard" and "Court". is a positive discriminator consisting of the sequence of As yet another example, in "[^A][a-z]+", [LA] may be a negative discriminator that would disallow "Alley" and "Avenue". In some cases, the algorithm may generate negative lookbehind to discriminate correctly. For example, (?<!Av)[A-Za-z]+ would exclude "Avenue" but allow "Alley".

As another example, if a user supplies positive examples "202-456-7800" and "313-678-8900" and negative examples "404-765-9876" and "515-987-6570", an embodiment Now, the regular expression generator 110 generates the regular expression

may be generated. That is, the replacement character subsequence is such that for regular expression suffixes, telephone numbers ending in 00 distinguish positive from negative examples (assuming, for example, that the objective is a regular expression that matches business telephone numbers). can be identified based on the determination that Although this is an example of negative examples by suffix (more specifically, an example of addressing negative examples by using positive suffixes), various other embodiments include prefix, suffix, or mid Substitutions at any of the span positions may be supported. In the example of substitution at mid-span positions, characters that are offset into the span can be tracked and split at the mid-span point.

To determine whether to use a prefix or a suffix, some embodiments use a heuristic in which the minimum score is selected over all k _a and prefix/suffix combinations:

k _a = number of characters considered to disambiguate the affix (prefix or suffix)
|F _p | = number of unique text fragments from positive examples required to disambiguate the affix
|F _n | = number of unique text fragments from negative examples required to disambiguate the affix
|E _p | = number of (complete) positive examples provided by the user
|E _n | = number of (complete) negative examples provided by the user In the example above, the heuristic is a shorter disambiguation than a longer disambiguating text fragment (thus multiplication by e.g. k _a ) Designed to like text fragments. The heuristic is also designed to favor prefixes over suffixes (hence, eg, a 0.1 penalty for suffixes) to improve readability. Finally, the heuristic disambiguates the longer prefix or suffix disambiguation (e.g. replacement) by using a larger number of string fragments (thus, e.g. the number of string fragments to be replaced squared). designed to be preferred over

As noted above, some embodiments may also support negative midspan examples, as well as negative lookbehind and negative lookahead examples.

Once the prefix/suffix and k (the number of characters to disambiguate) are determined, the regular expression generator 110 also determines how to express the disambiguation in the generated regular expression. can judge. The generated regular expression may be permissive for affixes (eg, prefixes or suffixes) that look like positive examples, or exclude affixes that look like negative examples.

If usePermissive is greater than zero, what appear to be positive examples are passed by generating a regular expression that allows characters taken from positive examples, one for each character position. In other cases, the regular expression generator 110 takes an approach that disallows what appears to be a negative example by generating, one at a time (for each character position), a regular expression that disallows characters taken from negative examples. may be taken.

As another example, the regular expression generated for a positive example at 8am and a negative example at 9pm is

may be. This uses caret syntax. In some cases, regular expression generator 110 may be configured to prefer shorter regular expressions, which may not only be more readable to the user, but may also be more likely to be correct. The principle is that characters that appear frequently in the future are more likely to appear again in the future, so emphasis should be placed on characters that appear frequently. If there are fewer unique characters |F _p | (less unique, since those that occur appear more often), this is rewarded in the heuristic by having it in the denominator.

Referring again to the usePermissive example heuristic above, determining one unique positive affix is not a great feature if there is only one positive example from the user. Therefore, a low |E _p | in this heuristic is penalized by having it in the numerator (ie, a high |E _p | is rewarded in this heuristic).

Additionally, in some embodiments, negative examples may be based on lookbehind and/or lookahead. For example, the user provides a positive example of "323-1234" and a negative example of "202-754-9876", in which case it uses the Regex lookbehind syntax (?<) to exclude phone numbers with area codes. !).

In some cases, negative examples may be based on an optional span. For example, a user may provide "ab" and "a2b" positive examples and "a3b" negative examples. In this case, an exemplary implementation may fail because it may attempt to discriminate based only on the required span and the digit "2" is in the optional span. In this example, failure may refer to a situation in which the generated regular expression matches all of the positive examples (correctly) and one or more of the negative examples (incorrectly). In such cases, the user can be warned of the failure, and via the user interface to manually repair the generated regular expression and/or remove some of the negative examples. can be offered a choice.

V. User Interface for Regular Expression Generation Additional aspects described herein include several different features and functions within the graphical user interface related to regular expression generation. As described below, some of these features are various choices for user selection and highlighting of positive and negative cases, color-coding of positive and negative cases, and may contain multiple overlapping/nested highlighting of .

FIG. 11 is a flowchart illustrating a process 1100 for generating regular expressions based on user data selections received within a user interface. The exemplary process of FIG. 11 may correspond to any of the aforementioned examples of generating regular expressions based on input data character sequences. However, FIG. 11 describes processing related to user interfaces that may be generated and displayed on client device 120 . At step 1101, in response to a request from a user via the user interface, the regular expression generator 110 retrieves data (eg, from the data repository 130) and renders/displays the data in tabular form within the graphical user interface. You may It should be appreciated that while this example uses tabular data, other examples may not use and display tabular data. For example, in some cases, a user can type raw data directly (rather than selecting data from a user interface). Furthermore, when data is presented through a user interface, it need not be in tabular form, but rather unstructured data (e.g. documents) or semi-structured (e.g. unformatted data such as tweets or posts). /spreadsheet of unstructured data items). In various examples, tabular data corresponds to transaction data, customer data, performance data, forecast data, and/or any other category of data that may be stored in data repository 130 for a business or other organization. obtain. At step 1102, a user selection of input data may be received via a user interface. The selected input data may correspond to, for example, an entire data cell selected by a user, or a subsequence of characters within a data cell. At step 1103 , regular expression generator 110 may generate a regular expression based on the input data (eg, data cell or portion thereof) received at step 1102 . At step 1104, the user interface may be updated in response to generating the regular expression. In some cases, the user interface may simply be updated to display the generated regular expressions to the user, while in other cases the user interface may be updated in various other ways as described below. may be As shown in this example, the user may select multiple different input data character sequences via the user interface, and in response to each new input data received, the regular expression generator 110: An updated regular expression may be generated that includes both the first and second (positive) examples of the character sequence. Then, when the user highlights a third sequence of characters (e.g., outside both character sequences, or within the first or second character sequences), the regular expression generator 110 updates the regular expression again. and so on. In some embodiments, the regular expression generator 110 may run the algorithm in real-time (or near real-time) such that an entirely new regular expression is generated for each new keystroke or each new emphasis made by the user. It can be generated in response to the displayed section. Further, if the user partially highlights (or otherwise selects) over a previous highlight, the old highlight may be deleted and a new highlight added.

Thus, as shown in FIG. 11, in response to user selection of character sequences via a user interface, regular expression generator 110 may generate and display regular expressions. For example, when a user highlights a first sequence of characters, the regular expression generator may generate and display a regular expression representing the first sequence of characters. When the user highlights the second sequence of characters, the regular expression generator may generate an updated regular expression that includes both the first sequence of characters and the second sequence of characters. Then, when the user highlights a third sequence of characters (eg, within either the first sequence or the second sequence), the regular expression generator may update the regular expression again, and so on.

FIG. 12 is another flowchart illustrating a process 1200 for generating regular expressions and extracting data based on capturing groups via user data selections received within a user interface. In step 1201, regular expression generator 110 may retrieve data (eg, from data repository 130) and render/display the data in tabular form within a graphical user interface, as described above in step 1101. At step 1202, the regular expression generator 110 may receive user highlighting selections for text fragments within a particular data cell. At step 1203, the regular expression generator 110 can generate a regular expression based on the positive examples of the selected data cell, and at step 1204, the regular expression capture based on the highlighted text fragment within the cell. Groups can be created. At step 1205, the regular expression generator 110 may determine one or more additional cells within the displayed tabular data that match the generated regular expression, and at step 1206, the generated regular expression Corresponding text fragments in additional cells that match the expression can be extracted.

Thus, in addition to supplying positive examples, the user may select text fragments within any of the selected positive examples (eg, via mouse text highlighting). In response, the regular expression generator 110 extracts the text fragment from the example and extracts the corresponding fragment from all other matches in the text to which the regular expression has been applied. You may create an expression capture group. Extracting text fragments from matching data cells can also include deletions and modifications, and in some cases can be used to create new columns of data from existing columns of semi-structured or unstructured text. .

Using the example in which the user selects positive data examples, if the user highlights a year, the regular expression generator 110 generates the regular expression

can be generated. As shown in this example, the regular expression generator 110 puts parentheses around the year, and also replaces the old parentheses around the month and day (used for alternation) with the ?:regex syntax " Converting to "non-capturing" group. In some embodiments, the extract/capture groups may be required to be on span boundaries, and in such embodiments the regular expression generator 110 may take the highlighted character range as input, It can be extended to encompass the near locator span boundary. However, in other examples midspan extraction/capture may be supported by the user interface.

In some embodiments, the user interface may support input data from a user including selection of a first character sequence within a second character sequence. For example, a user may highlight one or more characters within a larger previously highlighted character sequence, and a second user selection provides context for the larger first user selection. may provide. Such embodiments may allow input data to be provided to the regular expression generator 110 with greater specificity.

Further, in some examples, actions can be initiated and dialogs can be opened in response to user selections within the user interface (eg, highlighting text). In some cases, the dialog may be a non-modal dialog, such as a floating toolbox window that does not interfere with user interaction with the main screen. Dialogs can also change in appearance and/or functionality depending on what primary action the user is performing. Thus, in such cases, the user does not need to search for additional menu items after highlighting the selected text to initiate modification, extraction, etc. of the capturing group text fragment.

Further, in certain embodiments, the user interface provided for generating regular expressions may include three highlighting modes: nested automatic, nested manual, and single level. Nested auto is also known as nested (auto-outer). Nested manual is also known as nested (full-control). In these examples, single level may refer to one level of highlighting (or other forms of text selection), which may cause regular expressions to be generated without capturing groups. A nested manual highlighting mode allows the identification of two levels of highlighting (or other forms of text selection). This causes the generation of regular expressions with capturing groups. Nested auto-highlighting is similar to nested manual highlighting, except that outer highlighting (or other forms of text selection) can be automatically set to be the content of the entire text (e.g., an entire spreadsheet cell). It may be the same as highlight mode.

In some cases, the default mode of operation may be for the entire cell to be identified as the highlighted region, and the user may further highlight one or more additional subsequences within the highlighted cell. You may In other modes, the user may be allowed to manually specify both highlights within the data cells of the tabular data display. In still other modes, the user may be allowed to manually specify outer highlighting without inner highlighting. These other modes may be more suitable for columns of data consisting of semi-structured data, for example tweets or other long strings such as browser "user agent" strings. "Semi-structured" data refers to data that may be displayed in tabular form within a user interface, but where the columns within the table consist of unstructured text.

In some such embodiments, a user's inner and outer selections (eg, highlighting) via the user interface may be distinguished by color coding. For example, the outer highlighting of positive examples may be shown with a first text/background color combination and the inner highlighting of positive examples may be shown with a different, contrasting text/background color combination. .

As described above, the user can specify the selection of capture groups via the selection of character subsequences. A GUI may be used to facilitate user selection via highlighting (or other display). An example is shown in FIG. 13, where an exemplary user interface screen is shown with a tabular data display. In this example, FIG. 13 shows highlighting within a column value caused, for example, by a user dragging the mouse across one or more desired elements of the column value. Note that the "cell" on which user highlighting is performed may show a color change to indicate selection of the column value. This color change may be interpreted as automated highlighting in response to user highlighting.

14 and 15 are exemplary user interface screens showing the generation of regular expressions and capture groups based on selection of data from a tabular display. In these examples, FIGS. 14 and 15 show additional user interface windows that are automatically displayed upon detection of user highlighting 1401 within the tabular data display. A window is dynamically (and nearly instantaneously) generated in response to the selection of a field 1402 for displaying positive examples, a field for displaying negative examples, and a positive example from a tabular data display. and a field for displaying regular expressions. In these examples, user highlighting within column values 1401 may be equivalent to user highlighting within automated highlighting. User highlighting of the area code thus populates the positive example field 1402 with not only the user-highlighted area code 1401, but also the remainder of the telephone number.

However, it should be appreciated that user highlighting is not limited to performance within auto-highlighting. For example, user highlights may alternatively be performed within other user highlights. As another example, user highlighting may alternatively be performed without inner highlighting (eg, further highlighting within the highlighted text). These alternatives are particularly suitable for semi-structured data such as columns of data containing "tweets" or other long strings (eg browser "user agent" strings).

Additionally, once the corresponding regular expression is generated, other column values 1402 that match the regular expression can be identified based on additional automated highlighting. In the examples shown in FIGS. 14 and 15, additional automated highlighting shows those other column value elements that match the generated regular expression capturing groups. Additional automated highlighting may be performed using colors different from those used for user highlighting.

As shown in FIG. 15, additional user highlighting is shown to indicate another example user selection. Additional user highlighting may be performed in a manner similar to that described above. Accordingly, the user interface of FIG. 15 shows another population of examples in field 1502 for displaying positive examples. This may occur in response to detecting additional user highlights. Additionally, the generated regular expression 1503 may be dynamically and nearly instantaneously updated so that it matches all of the positive examples 1502 . In response to generating the updated regular expression, automated highlighting of other column values 1504 that match the updated regular expression may also be updated. Dynamic color coding may also be used in some implementations. For example, matches may be color-coded using a first color (e.g., blue), positive examples may be color-coded using a second color (e.g., green), and negative The examples may be color coded using a third color (eg red). Within a tabular data view (such as a spreadsheet) or other infinite scrolling data view (e.g. of semi-structured or unstructured data), when the view is scrolled down to the actual additional data, it becomes newly visible. data may be color coded.

16A and 16B are example user interface screen shots showing the generation of regular expressions based on selection of positive and negative examples from a tabular display. 16A-16B, individual examples from positive example field 1602 may be removed from positive example field 1603 and/or moved to negative example field 1603. FIG. Within the user interface, this may be done, for example, by the user clicking (eg, right-clicking) one of the examples to select it. The selection can cause the user interface to display a menu 1602 containing delete and modify options. Then clicking on an option will perform the corresponding function.

In the example shown in FIGS. 16A and 16B, the result of user selection of the change option is to move the selected example to negative example field 1603 and update regular expression 1601 to regular expression 1604, which is: It can be generated dynamically and almost instantaneously (eg, between 30 ms and 9000 ms, or 100 ms median in some embodiments). In response to generating updated regular expression 1604, automated highlighting of other column values that match the updated regular expression may also be updated within the tabular data display. In addition, automated highlighting may be performed for some or all of the negative examples, including any column value corresponding to the negative example, which is a different color than any of the colors used above. , or in other aspects may be distinguished in the user interface using other visual techniques.

In some embodiments, designating a negative example via the user interface first designates the example as a positive example and then converts it to a negative example, as shown in FIGS. 16A and 16B. need not be required. Rather, negative examples can be designated in various ways. For example, a user selects a column value (e.g., one of the other column values for which automated highlighting has been performed to indicate a match with the generated regular expression) via the user interface. (eg, right-click), which causes the display of a menu with options (eg, "create new counterexample") to designate the selected column value as a negative example.

Thus, using the example shown in FIGS. 16A and 16B, in response to generating updated regular expression 1604, the automated highlighting of other column values matching the updated regular expression is also updated. obtain. In these examples, the updated regular expressions specify phone numbers ending in "9".

Referring briefly to FIGS. 14 and 15, when the "Extract" button is clicked or otherwise selected by the user, highlighting in all cells matching the current regular expression 1403 or 1503 Operations may be initiated to extract the rendered text fragment. Although not shown in FIGS. 14 and 15, in some embodiments the user interface may provide other selectable buttons in addition to or instead of the "Extract" button. For example, a "replace" button may be presented as an option to replace the user-highlighted element with a user-specified element. Additionally or alternatively, one or more "delete" buttons may be presented as options that effectively replace the user-highlighted element with nothing. For example, one or both of a "delete fragment" operation and/or a "delete line" operation may be implemented, which may result in deleting either a user-highlighted text fragment or an entire line, respectively. Become. Additional operations that may be implemented in various embodiments are "keep rows" operations, "split" operations (e.g., highlighting commas and then extracting the comma-separated components into separate new columns). , and “obfuscation” operations (eg, replacing highlighted text/capturing groups with “#” or other sequences of symbols).

In this example, in response to selecting the "extract" button, an extract operation may be added to a list of transform scripts to be executed by downstream operations. In some embodiments, a list of transformation scripts may be displayed in a portion of the user interface for review/modification by the user. Alternatively, the extraction operation may be performed in-place to generate a new column containing the content of the Regex capture group (eg, elements corresponding to user-highlighted portions of positive examples). In the examples shown in FIGS. 14 and 15, a new column of area codes and/or a new table may be generated in response to selection of the "Extract" button.

FIG. 17 is another exemplary user interface screen showing generation of regular expressions and capture groups based on selection of data from a tabular display in accordance with one or more embodiments described herein.

A. Multiple Highlighting In some embodiments, the user interface may also support multiple highlighting per example. 42, 43, 44, 45, 46, 47, and 48 illustrate user interfaces 4200, 4300, 4400, 4500 for implementing multiple highlighting, according to some example embodiments. , 4600, 4700, and 4800 are shown. Different highlighting methods are described below. In an exemplary embodiment, inner highlighting and outer highlighting can be distinguished by using different color codes.

FIG. 42 shows a user interface 4200 displaying a view of a dataset in single level mode, according to some example embodiments. FIG. 42 shows how highlighting appears on the user interface while in single-level highlighting mode. In FIG. 42, the user highlights version number 4235 (eg, "5.0") of application 4236 (eg, Mozilla). In response to the user highlighting version number 4235, the user interface data model generator displays version numbers 4237 (“6.1”), 4238 (“1.9”), 4239 (“2.2”), and 4240 (“ 3.6").

Based on the highlighting, Regex By Example dialog box 4230 appears. The Regex by Example dialog box 4230 includes the specified example 4233 specified by the user. In this example, the version number "5.0" is specified by the user. Also, example Regex dialog box 4230 indicates that highlight mode 4250 is single level 4251 .

FIG. 43 shows a user interface 4300 displaying highlighted data in nested full control mode, according to some example embodiments. In the example shown in FIG. 43, “Nested (full control)” 4351 is selected from highlight mode panel 4350 of dialog box 4330 . Also, as shown in FIG. 43, the user displays the outer highlight as software name 4236 (eg, "Mozilla") and version number 4235 (eg, "5.0"). The Regex by Example dialog box 4330 shows that the highlight mode 4350 is in nested (full control) 4351 . Additionally, the dataset is currently in multi-highlight mode 4360 . The highlighting performed in FIG. 43 performs outer highlighting in a nested fully automatic mode.

In an exemplary embodiment, the currently running outer highlight, the ongoing outer highlight, can be handled in a first class manner, with its own urgency color, such as gold. In an exemplary embodiment, highlights can appear in different colors that can be used to indicate the urgency, priority, and/or importance of the highlight.

FIG. 44 shows a user interface 4400 displaying highlighted data in nested full control mode, according to some example embodiments. As shown in Figure 44, multi-highlight mode 4460 is selected. The user is viewing the outer highlight as software name 4236 and version number 4235 . The software name 4236 (eg, Mozilla) and version number 4235 (eg, 5.0) selected for outer highlighting may be highlighted in a first color. After selecting the outer highlight, the user can identify the inner highlight. For example, the user can select only version number 4235 (eg, "5.0") for inner highlighting. The inner highlighting may be highlighted with a second color different from the first color. Since the inner highlighting and the outer highlighting appear in different colors, the outer highlighting and the inner highlighting can be easily distinguished. The example shown in FIG. 44 is in nested full control mode highlighting. Thus, highlight mode panel 4450 indicates that the dataset is in nested full control mode 4451 .

In an exemplary embodiment, inner highlighting and outer highlighting can be distinguished by using different colors or color codes. For example, the outer highlighting of positive examples may be shown in black text on a light green background, and the inner highlighting of positive examples may be shown in light green text on a dark green background.

In FIG. 44, the user has selected version number “5.0” identified as element 4235 . If the user selects version number "5.0" (element 4235), the user interface data model generator can automatically select version numbers "3.6" (element 4240) and "5.1" (element 4241). Since the user interface data model generator emphasized the version number in addition to the version number selected by the user, if the additional version number identified by the user interface data model generator is not desired by the user: The user can give additional examples. Alternatively, if the user consents to additional highlighting performed by the user interface data model generator, the user can proceed to apply commands to the highlighted data.

The Regex by Example dialog box 4430 displays the specified example 4433 . Further, example Regex dialog box 4430 indicates that highlighting mode 4450 is nested (full control) 4451 . Regex by Example dialog box 4430 also indicates that it is in multi-highlight mode 4460 . A generated regular expression 4432 is also identified in the Regex by Example dialog box 4430 .

As shown in Figures 45, 46 and 47, the user can provide further examples. FIG. 45 shows a user interface 4500 in which two positive examples are provided, according to some example embodiments. FIG. 46 shows a user interface 4600 in which two positive examples are provided, according to some example embodiments. FIG. 47 shows a user interface 4500 in which three positive examples are provided, according to some example embodiments.

As shown in FIG. 45, the user can indicate the outer highlighting as software name 4536 and version number 4535 to give positive examples. The software name 4536 (eg, Windows NT) and version number 4535 (eg, 6.1) selected for outer highlighting may be highlighted in the first color. After selecting the outer highlight, the user can identify the inner highlight. For example, the user can select version number 4535 (eg, "6.1") as the inner highlight. The inner highlighting may be highlighted with a second color different than the first color. A regular expression 4532 can be generated based on the highlighting.

The highlighted example provided by the user is identified in the specified example 4533 within the Regex by Example dialog box 4530 .

FIG. 46 shows a user interface 4600 displaying updated generated Regex, according to some example embodiments. As shown in Figure 46, the generated Regex 4632 is updated based on two specified examples 4633 (eg, "Mozilla 5.0" and "Windows NT 6.1"). As additional outer and inner highlighting is provided by the user in the specified example 4633, the generated regex 4632 will change accordingly. The generated regex 4632 appears in the regex by example dialog box 4630 .

FIG. 47 shows a user interface 4700 displaying alternative data highlighting, according to some example embodiments. A version number with an underscore can be given as an example for generating regex, as shown in FIG.

As shown in FIG. 47, the user can indicate the outer highlighting as software name 4735 and version number 4735 to give another positive example. The software name 4735 (eg, MacOS X) and version number 4736 (eg, 10_6_8) selected for outer highlighting may be highlighted in a first color. After selecting the outer highlight, the user can identify the inner highlight. For example, the user can select version number 4736 (eg, 10_6_8) as the inner highlight. The inner highlighting may be highlighted with a second color different than the first color.

The highlighted example provided by the user is identified in the specified example 4733 within the Regex by Example dialog box 4730 . As shown in Figure 47, the generated Regex 4732 is updated based on three specified examples 4733 (eg, "Mozilla 5.0", "Windows NT 6.1" and "MacOS X 10_6_8"). As additional outer and inner highlighting is provided by the user in the specified example 4733, the generated regex 4732 will change accordingly.

FIG. 48 shows a user interface 4800 displaying updated generated Regex, according to some example embodiments. As shown in Figure 48, in response to the user selecting a version number with an underscore, the generated regex 4832 is updated to include the underscore information.

The generated regex 4832 is updated based on three specified examples 4833 (eg, "Mozilla 5.0", "WindowsNT 6.1" and "MacOS X 10_6_8"). As additional outer and inner highlighting is provided by the user in the specified example 4833, the generated regex 4832 will change accordingly. The generated regex 4832 is displayed in the regex by example dialog box 4830 .

FIG. 49 is a flowchart of a method 4900 for performing multiple highlighting, according to some example embodiments.

At step 4910, the user may select data (eg, data fragments) within the dataset. That is, the user can select outer highlighting (eg, "Mozilla 5.0") and inner highlighting (eg, "5.0"). The user can perform outer highlighting and inner highlighting on the first data record 4320 .

At step 4920, after the user has performed the initial highlighting, the user interface data model generator generates the Data (eg, data fragments) can be automatically highlighted.

At step 4930, a regular expression may be generated. A regular expression is generated based on the highlighting provided by the user at step 4910 and based on additional highlighting performed by the user interface data model generator at step 4920 .

If the user is not satisfied with the highlighted results, the user can highlight additional data to provide additional examples. Accordingly, steps 4910, 4920 and 4930 can be repeated until the user is satisfied with the final highlighting on the dataset. Alternatively, the user can modify the generated regular expression (eg, 4832) to produce the desired highlighting.

In an exemplary embodiment, for multiple highlights in nested (auto-outer) mode, a search is performed against the minimum number of surrounding spans required. If the user interface is in nested (auto-outer) mode with multiple highlights, the backend algorithm can enter a special mode. The reason for the special mode is that multiple highlights per example tend to mean that the example string is longer than usual. Since the LCS algorithm has an exponential runtime, this would take too long to run the full length of outer highlighting. In a special mode, the algorithm starts with an inner highlight and gradually grows the outer highlight around the inner highlight until a satisfactory regex is produced.

Exemplary code for executing special modes can include some or all of the following code:

Multiple highlighting allows users to easily view data (e.g. data fragments) to which any of the commands (e.g. extract, split, delete, obfuscate, replace, and filter lines) can be applied. can be selected to

For example per-multiple highlighting capabilities, individual highlights may be clickable instead of whole lines. In some cases, checkboxes (or other techniques within the user interface) may be used to switch between per-example single highlighting mode and per-example multiple highlighting mode. For example, in a mode that supports multiple highlighting per example, the user may allow multiple different highlighting within a single data example (e.g., spreadsheet cell) to provide multiple different examples for which a regular expression may be generated. The displayed portion can be selected. In multiple highlighting per example, both positive and negative "examples" can be identified and shown as color codes in one example list. For example, in multi-highlight mode, clicking on a highlight makes only that one sub-example negative. The functionality of example-by-multiple highlighting is illustrated in FIGS. 42-49.

As shown in this series of examples, the user sequentially selects the outer highlighted region followed by the inner highlighted region to generate/update the corresponding regular expression. Additionally, as shown in these figures, the user can specify multiple different inner/outer highlighting examples within a single data cell, and the generated regular expressions can be highlighted via the user interface. It may be updated with each new example displayed (or otherwise selected by the user).

In some embodiments, the special multi-highlight nested auto-outer search mode may be limited to a predetermined number of spans (eg, 10 spans) in diameter. In such a case, the condition for the final do-while loop could be like while (!result.isSuccess && !reachedMaximal && numLookAheadAndBehindSpans < 10).

Multiple highlighting may be used when implementing a particular command.
B. Commands As noted above, the user interface may support additional types of commands in addition to extraction commands in various embodiments. Elements of the user interface data model generator can be configured to implement those commands.

FIG. 24 shows a regular expression generation server 2400, according to some example embodiments. A regular expression generation server 2400 corresponds to the regular expression generator server 111 shown in FIG. The regular expression generation server 2400 can include multiple processors and memory. Regular expression generation server 2400 can include regular expression generator 2410 and user interface data model generator 2420 . Regular expression generator 2410 may also be referred to as a Regex generator. Regular expression generator 2410 may correspond to regular expression generator 116 of FIG. User interface data model generator 2420 may correspond to user interface data model generator 114 of FIG.

The regular expression generator 2420 can perform several different commands including extract, split, delete, obfuscate, replace, and filter lines. Thus, regular expression generator 2420 can include extraction data generator 2411 , split data generator 2412 , removal data generator 2413 , obfuscation data generator 2414 , replacement data generator 2415 , and line filtering data generator 2416 . can. Each component of the regular expression creation server 2400 will be described in more detail below.

Although extract, split, delete, obfuscate, replace, and line filtering commands are described, exemplary embodiments may be configured to perform additional commands. Regular expression generator 2410 may also include command generators other than those shown in FIG.

1. Extraction As described above, in an extraction command, a regular expression pattern match may be identified and the matching data extracted into a newly created column. The extraction function is described above with respect to FIGS. 14 and 15. FIG.

2. Split Figures 25A, 25B, and 26 show examples of split commands implemented via the user interface. Figures 25A and 25B show user interface 2500 and user interface 2501 for implementing a split command, according to some example embodiments. FIG. 26 shows the results of a split command on a dataset, according to some exemplary embodiments.

A split command can result in the creation of multiple different columns based on the presence of a particular regular expression used as a delimiter. For example, if a comma (",") is given as the pattern to be matched, the comma is used as the delimiter during the extraction operation. Each data to be extracted that contains a single comma is split at the comma and extracted into two different columns (ie, the data before the comma and the data after the comma). If the data item to be split contains two commas, the data is split into three parts based on the commas and extracted into three new columns. Thus, the split command can perform delimiter-based extraction (to one or more new columns) rather than pattern-match-based extraction (where matches would be put into a single column). .

In other examples, the split command uses braces or brackets, asterisks, "<" or ">" or hyphen and dash characters in the pattern (or delimiter) to be matched for certain types of content. and so on. Additional symbols may be used as delimiters.

A dataset is displayed on a user interface 2500 as shown in FIG. 25A. A dataset can be a spreadsheet. The dataset can include a column 2510 containing phone numbers. A phone number can include a first hyphen 2511 and a second hyphen 2512 . If a split command is implemented on the phone number column 2510, the phone number can be split as desired by the user.

FIG. 27 shows a flowchart of a method 2700 for executing split commands, according to some example embodiments. In the example of FIG. 27, a hyphen character is used as a delimiter for dividing data.

At step 2710 , the user can select the first record 2520 hyphen in the “phone number” column 2510 . A hyphen may be selected by the user highlighting the hyphen on the display of the device, for example with a mouse or gesture, when the user is on the interactive user interface. In the example shown in FIG. 25A, the user has highlighted the first hyphen 2511 within the first record 2520 .

At step 2720, after the user highlights the first hyphen in the first record 2520, the user interface data model generator displays the , all of the first hyphens in the phone number string corresponding to the selection made by the user can be automatically highlighted. That is, based on the highlighting made by the user for the first record 2520, the user interface data model generator determines what highlighting for the remaining records (eg, 2-25) in the phone number column. It can decide whether to display. Although twenty-five records are shown in the figure, exemplary embodiments may apply the split command to more or less than twenty-five records. Moreover, a dataset can include a large number of records, such as thousands or thousands of records. A regular expression can determine what additional highlighting to perform in the same record or in additional records based on the initial highlighting made by the user.

At step 2730, after the highlighting is done, a regular expression can be generated based on the highlighting. FIG. 25A shows a generated regex 2532 generated based on the highlighting performed by the user and the automatic highlighting performed by the regular expression generator. The generated regular expression is updated based on the highlighting changes made by the user.

At step 2740, the "Regex by Example" dialog box 2530 automatically appears on the screen. In the example shown in FIG. 25A, the "Regex by Example" dialog box 2530 appears after all the records (eg, records 1-25) in the phone number column 2510 have been highlighted, but the "Regex by Example" dialog box 2530 appears. Dialog box 2530 may also appear after initial input (eg, highlighting of the first record by the user).

A specified example 2533 is shown in FIG. 25A to allow the user to easily highlight selected data or modify the highlighting made to the selected data. The user can modify the highlighting by changing the highlighting in the data in the "phone number" column 2510, changing the highlighting in the specified example 2533, or modifying the generated regex 2532. can be done. Changes made to the “phone number” column 2510 are reflected in the regular expression 2532 that is generated.

In one exemplary embodiment, the minimum spanning tree can be used on examples rather than on highlights. Therefore, if there are multiple highlightings for each example given and the highlightings overlap, only a single copy of the longest common subsequence algorithm needs to be run for each example. If the highlights for a given example are non-overlapping, the longest common subsequence algorithm can be run for each of the highlights.

In an exemplary embodiment, the longest span list is selected for the minimum spanning tree (MST) vertices and fed first to the LCS queue. The LCS queue contains spans for which the longest common subsequence algorithm is applied. One example representing multiple highlights is contained in the array passed to graphLcs(), with multiple highlights per example, the longest such example being an ordered list of spans passed to the LCS queue It is chosen to be the vertex in the graph of the minimum spanning tree to be determined.

At step 2750, a selection of a "split" command is received. For example, the user can select the “Split” button 2531 on the example Regex dialog box 2530 . As shown in Regex By Example dialog box 2530, the multi-highlight checkbox 2534 and highlight mode drop-down list are disabled when split is selected.

In an exemplary embodiment, the checkbox is animated to draw the user's attention when the UI automatically checks the checkbox based on the user performing a second highlight. may Certain exemplary embodiments activate checkboxes based on state changes caused by user interactions elsewhere in the UI. A checkbox is activated when the server system automatically causes the checkbox to change its state based on user interaction elsewhere on the screen.

After the "split" command selection is received, at step 2760, the split data generator 2412 of the regular expression generator 2410 can automatically split the phone number based on the highlighted delimiters. The example shown in Figure 25B occurs after the user selects the "Split" command 2531 in Figure 25A.

As shown in Figure 25B, the split command changes the highlight setting to multi-highlight. In FIG. 25A, the multi-highlight mode 2534 has not been selected prior to selecting the split command. In FIG. 25B, multiple highlight mode 2534 is selected after selection of the split command. In some embodiments, when the "Split" command is selected, the user interface can enable a multi-highlight checkbox and a highlight mode dropdown list. For example, referring to FIG. 25B, when the user clicks on the "split" command, the highlighting mode may automatically change to single level, and then the multi-highlighting textbox may be enabled.

At step 2770, the split command results may be displayed. As shown in user interface 2600 shown in FIG. 26, the phone numbers from phone number column 2510 are now split into three columns 2610, 2620 and 2630. FIG. Column 2610 contains the portion of the telephone number that appears before the first hyphen, column 2620 contains the portion of the telephone number that occurs between the first and second hyphen, and column 2630 contains the portion of the telephone number that appears before the first hyphen. Contains the part of the phone number that appears after the second hyphen. Since there are two hyphens acting as delimiters, the phone number is split into three columns. For some numbers, a column may contain the area code with the local code because the area code appears before the first hyphen. For phone numbers, the fields will be populated based on the information in the phone number. If the phone number has only one hyphen, only two columns of information will appear for that number.

Three additional columns are generated in the spreadsheet view as shown in FIG. Because the phone number is split, the user can more easily identify the desired information. For example, users can more easily identify area codes in data records. Additionally, the user can perform additional actions on each of columns 2610, 2620 and 2630. FIG. Therefore, the user can use the data more easily. For example, the user may identify the customer's primary area code.

When a split command is executed, the executed split command (eg, phone string split) can be added to the transform script for the data set. A transformation script indicates commands that are applied to columns of data. Transformation scripts can be used to apply the same commands to different data sets.

Exemplary embodiments thus provide a quick and efficient user interface for partitioning data within a dataset. Additionally, the regular expression generator can be configured to identify settings that are more likely to produce the results desired by the user.

Although the flow diagrams are illustrated with specific steps, the order of the steps can be changed. For example, a regular expression can be generated based on the initial highlighting performed by the user.

3. Delete FIG. 28 shows a user interface 2800 for implementing a delete command, according to some example embodiments. FIG. 29 shows a user interface 2900 displaying the results of a delete command on a dataset, according to some example embodiments. In the Delete command, the user highlights (or otherwise selects) an example text, generates a regular expression that corresponds to the pattern, and then retrieves data from a spreadsheet or other data view that matches the pattern. May be deleted. The delete function replaces the data with an empty string (eg the delete command can be implemented by converting the generated regex into three capturing groups).

The dataset is displayed in a user interface 2800 as shown in FIG. The dataset may include columns 2810 containing street addresses. In the example shown in Figure 28, the user wants to remove the street number 2811 from the address. If a delete command is implemented in the "street_address" column 2810 of record 2820, the street address information can be deleted as desired by the user. In FIG. 28, the user has selected some positive examples to identify the street number in the "Street Address" column of the spreadsheet. These examples appear as designated examples 2833 . The user then clicks the "delete" button to initiate the delete operation. FIG. 29 shows the resulting spreadsheet with the street numbers removed. So the delete command is essentially replaced with (empty string). This can be accomplished by converting the generated regex into three capturing groups, as discussed below with respect to the "replace" command.

FIG. 30 depicts a flowchart of a method 3000 for executing a delete command, according to some example embodiments.

At step 3010, the user can select a portion of the address information for deletion. In the example shown in FIG. 28, for the first record 2820, the street number portion 2811 of the address information is selected. The street number portion 2811 of the address information may be selected by the user on the device's display by highlighting the street number portion, for example with a mouse or gesture, when the user is on the interactive user interface.

At step 3020, after the user has highlighted the street number portion 2811 in the first record 2820, the user interface data model generator displays, for each of the data records (eg, records 2-25 shown in FIG. 28), " All of the street number portions within the "street_address" column 2810 can be automatically highlighted. That is, based on the highlighting performed on the first record 2820, the user interface data model generator will determine what the remaining records (eg, 2-25) in the “street_address” column 2810 are It can be determined whether highlighting should be performed. Although twenty-five records are shown in the figure, illustrative embodiments may apply the delete command to more or less than twenty-five records. A regular expression can determine what additional highlighting to do based on the initial highlighting by the user.

At step 3030, a regular expression is generated. A generated regular expression 2832 is shown in FIG. The generated regular expressions allow the user to easily highlight which data to select. For the delete command, the generated regular expression allows the user to easily highlight which data to delete. The generated regular expression corresponds to the highlighting done by the user and the regular expression generator. The user can modify the highlighting by changing the highlighting in the data in the "street_address" column 2810 or by changing the highlighting in the regular expression 2832. Changes to the 'street_address' column 2810 are reflected in the generated regular expression 2832 . The generated regular expression is updated based on the highlighting changes made by the user.

At step 3040, after all street numbers in the "street_address" column 2810 have been highlighted, the "Regex by Example" dialog box 2830 can be automatically displayed on the screen. In the example shown in FIG. 28, the "Regex by Example" dialog box 2830 appears after all the records (eg, records 1-25) in the "street_address" column 2810 have been highlighted, but the "Regex by Example" dialog box 2830 appears. dialog box 2830 can also appear after initial entry (eg, highlighting the first record). Specified example 2833 identifies an example provided by the user.

At step 3050, selection of a delete command may be accepted. For example, the user can select the “Delete” button 2831 in the “Regex by Example” dialog box 2830 .

After receiving the delete command selection, at step 3060, the delete data generator 2413 of the regular expression generator 2410 can automatically delete the street number based on the highlighting. In the example shown in FIG. 28, all of the "street_address" records contain highlighting, so the delete command is applied to each of the records shown in FIG.

At step 3070, the deletion results can be displayed. As shown in the user interface 2900 displayed in Figure 29, the street number from the street address in the "street address" column 2810 has been removed. Street numbers are removed in place.

After the delete command is executed, the executed delete command (eg, delete street address column) can be added to the transformation script for the dataset. A transformation script indicates commands that are applied to columns of data. Transformation scripts can be used to apply the same commands to different data sets.

Exemplary embodiments thus provide a quick and efficient user interface for deleting data in a dataset.

4. Obfuscation FIG. 31 illustrates a user interface 3100 for implementing obfuscation commands, according to some example embodiments. FIG. 32 illustrates a user interface 3200 showing results of obfuscation commands on a dataset, according to some example embodiments.

The obfuscate command allows the user to highlight (or otherwise select) an example text, generate a regular expression that corresponds to the pattern, and then retrieve data from a spreadsheet or other data view that matches the pattern. May be obfuscated (e.g., obfuscated rather than deleted). Data can be obfuscated so that all of the data in the record is not visible. For example, a user may wish to obfuscate information for security or privacy reasons.

In FIG. 31, the user has selected the middle two digits 3112 in the "ssn" column 3110 of the spreadsheet. The user then clicks the "Obfuscate" button to initiate the obfuscation operation. Figure 32 shows the resulting spreadsheet after replacing the middle two digits of the entire "ssn" column with two pound signs.

The dataset is displayed in a user interface 3100 as shown in FIG. The dataset includes, for example, a column 3110 directed to social security numbers. Although a social security number is used to indicate obfuscation commands, any of the data within the columns of data can be obfuscated as desired by the user. For example, a user may wish to obfuscate any kind of sensitive information, such as credit card information or bank account information.

The social security number can be divided into first field 3111 , second field 3112 and third field 3113 . A first field 3111 may precede the first dash, a second field 3112 may lie between the dashes, and a third field 3113 may follow the second dash of the social security number. In the example shown in FIG. 31, the user has edited the second field 3112 of the social security number (eg, the digits between the first and second dashes) so that all of the social security number is not visible. I would like to obfuscate. Although the second field is obfuscated, the user can choose to obfuscate any or all of the fields within the social security number.

FIG. 33 shows a flowchart of a method 3300 for executing obfuscated commands, according to some example embodiments.

At step 3310, selection of the second field 3112 in the Social Security Number column 3110 is accepted. Selected fields 3112 are selected for the first record 3120 . A second field 3112 may be selected by using, for example, a mouse or gestures on the device's display when the user is on the interactive user interface. A second field selected by the user may be highlighted with a first color. Selections made by the user are provided as named examples 3133 in dialog box 3130 .

At step 3320 , all of the social security numbers in the social security number column can be highlighted to correspond to the highlighting performed at step 3320 . That is, all second fields within the social security numbers in column 3110 are highlighted in a first color, and all first and third fields 3111 and 3113 within the social security numbers in column 3110 are It will be highlighted in the second color. Fields are highlighted in different colors so that the user can easily distinguish between modified and unmodified fields.

The user interface data model generator can highlight all records (eg, 2-25) to correspond to the highlighting performed by the user on the first record 3120 . That is, based on the highlighting performed for the first record 3120, the user interface data model generator determines what highlighting for the remaining records (eg, 2-25) in the Social Security Number column. should be executed. Although 25 records are shown in the figure, exemplary embodiments may apply obfuscation commands to more or less than 25 records.

At step 3030, a regular expression may be generated. A generated regular expression 3132 is shown in an example dialog box 3130 in FIG. The generated regular expressions allow the user to easily highlight which data should be obfuscated. The user can modify the highlighting by changing the highlighting in the data in the "ssn" column 3110 or by changing the highlighting in the generated regular expression 3132. Changes to the 'ssn' column 3110 are reflected in the generated regular expression 3132 . That is, the generated regular expression is updated based on the highlighting changes made by the user to the data set.

At step 3340, after the social security number has been highlighted, a "Regex by Example" dialog box 3130 may automatically appear on the screen. In the example shown in FIG. 31, the "Regex by Example" dialog box 3130 appears after all the records (eg, records 1-25) in the Social Security Number column 3110 have been highlighted, but the "Regex by Example" dialog box 3130 appears. dialog box 3130 can also appear after initial entry (eg, highlighting the first record). The "Regex by Example" dialog box 3130 includes the specified example 3133 entered by the user. The “Regex by Example” dialog box 3130 also includes a generated regular expression 3132 .

At step 3350, selection of the "obfuscate" command is accepted. For example, the user can select the “Obfuscate” button 3131 in the “Regex by Example” dialog box 3130 . As shown in FIG. 31, the user can select the Obfuscate button 3131.

At step 3360, after the obfuscation command is received, the obfuscation generator 2414 of the regular expression generation server 2400 automatically obfuscates all of the second field of the social security number in the "ssn" column 3110. be able to.

At step 3370, the obfuscation results can be displayed. FIG. 32 shows a user interface 3200 displaying results of obfuscation commands on a dataset, according to some example embodiments. As shown in Figure 32, the second field 3112 selected by the user is replaced with "##" instead of the highlighted field to be obfuscated. Therefore, the second field 3112 of the social security number in the social security column 3110 has all been replaced with "##". That is, the second field 3112 for all social security numbers is obfuscated.

After the obfuscation command is executed, the executed obfuscation command (eg, obfuscate_column_ssn) can be added to the transformation script 3230 for the dataset. A transformation script indicates commands that are applied to columns of data. Transformation scripts can be used to apply the same commands to different data sets.

Accordingly, example embodiments can provide a quick and efficient user interface for obfuscating data in a dataset and maintain the privacy of user information.

5. Substitutions FIG. 34 illustrates a user interface 3400 for implementing replacement commands, and FIG. 35 illustrates a user interface 3500 for implementing replacement commands, according to some exemplary embodiments. In the Replace command, the user highlights (or otherwise selects) an example text to generate a regular expression that corresponds to the pattern, and then replaces the text that matches the pattern with other selected text. can do.

The replace command involves dynamically displaying a two-column table of before and after examples within the dialog. To utilize the content of a capturing group in a substitution expression, users can use the standard Regex substitution syntax $1 from the Java® and Javascript Regex APIs.

FIG. 37 depicts a flowchart of a method 3700 for executing replacement commands, according to some example embodiments.

At step 3710, the user can select the field within the "street_address" column 2810 that the user wishes to replace with other data. In the example shown in FIG. 34, the user wishes to replace the street address "Drive" with "Dr.". As shown in Figure 34, the user can select "Drive" from the fifth record 3420 of the dataset. The data to be replaced may be selected by the user, when on the interactive user interface, by highlighting the data to be replaced, for example with a mouse or gestures on the display of the device.

At step 3720, after the user highlights the data to be replaced in the fifth record (e.g., “Drive”), the user interface data model generator changes the highlighting performed in the fifth record 3420 to All corresponding fields in the rest of the record can be automatically highlighted. The user interface data model generator can automatically highlight street addresses containing "Drive". As shown in FIG. 34, records 7, 13, 16, 18, and 20 contain "Drive", so in records 7, 13, 16, 18, and 20, "Drive" is automatically Highlighted by the User Interface Data Model Generator. "Drive" may be highlighted in the first color in records 5, 7, 13, 16, 18, and 20; The rest of the street address containing "Drive" can be highlighted in a second color different from the first color. Therefore, the portion to be replaced can be easily identified.

At step 3730, a regular expression 3432 may be generated. As shown in FIG. 35, a generated regular expression 3532 of three parts is shown. The three-part generated regular expression allows the user to easily replace the 'Drive' part of the data, the part before the 'Drive', or the part after the 'Drive'. In this example, the user replaces the "Drive" portion of the data with "Dr" and the user can immediately see the results in the preview spreadsheet in the pop-up "Regex by Example" dialog box 3530.

In Figure 34, the user has selected a sufficient number of examples 3433 (positive and negative) from the "street_address" column of the spreadsheet to generate a regular expression corresponding to addresses ending in "Drive". Address 3434 is a positive example and address 3435 is a negative example.

At step 3740, after all records have been highlighted, the "Regex by Example" dialog box 3430 automatically appears on the screen. In the example shown in Figure 34, the "Regex by Example" dialog box 3430 appears after all records containing data to be replaced have been highlighted, but the "Regex by Example" dialog box 3430 It can also appear after the initial entry (eg highlighting of "Drive" in the fifth record). The “regex by example” dialog box 3430 can include a specified example 3433 and a generated regular expression 3432 . Address 3434 is a positive example and address 3435 is a negative example.

At step 3750, selection of the "replace" command is accepted. For example, the user can select a "Replace" button 3431 on the "Regex by Example" dialog box 3430 to initiate a replace command.

At step 3760, panel 3538 and "replace with" field 3537 may be displayed to assist the user in determining what information is being replaced. As shown in FIG. 35, within panel 3538, the user can see the initial address and how it will appear when replaced. For example, the word "Drive" in the address appears as "Dr." Panel 3538 can assist the user in providing the user with a preview of how the data will appear if the replace command is applied.

As shown in FIG. 35, Regex 3532 is transformed into a triple capturing group. The generated regex is "^(.*?)(D[a-z]+)()$". So the user can easily see what data appears before highlighting (^(.*?)), on highlighting ((D[a-z]+)), and after highlighting (()). be able to. In the example shown in FIG. 35, no data appears after highlighting (()). The data on the highlight is the data selected by the user. In this example, the data on the highlight is "Drive" which is the data the user has selected for replacement.

The user can also identify what the selected word will be replaced with. A "replace with" field 3537 identifies the word (eg, "Dr") that the selected word (eg, "Drive") should be replaced with. The user can amend the replacement word in the “replace with” field 3537 . The replacement word selected by the user in the "replace with" field 3537 is applied to the highlighted record. If the user agrees with the replacement, the user selects the create button 3536 and the replacement data generator 2415 can make the replacement.

As indicated above, the replace command may also include dynamically displaying a two-column table within the dialog showing the before and after examples (e.g., the preview spreadsheet in the pop-up "Regex by Example" dialog box). . To utilize the content of a capturing group in a substitution expression, users can use the standard Regex substitution syntax $1 from the Java and Javascript Regex APIs. Additionally, if the user selects the replace command and the user is in a mode that generates capture groups (i.e. nested-auto-outer), there are a total of three generated regex displayed in the dialog, i.e. highlighted We get two additional capture groups: before the highlight, above the highlight, and after the highlight. The user can then refer to these as $1, $2 and $3 within the replacement expression. Additionally, if there is no capturing group, the system can wrap the entire generated regex into a capturing group, and the user can refer to the original text using $1.

As disclosed in the exemplary embodiment, highlighting modes can include nested automatic, nested manual, and single level. Nested auto is also known as nested (auto outer). Nested manual is also known as nested (full control).

At step 3770, substitutions may be made. After confirming the changes by clicking the "Create" button 3536, the data is replaced.

At step 3780, the replacement results can be displayed. FIG. 36 shows a user interface 3600 displaying the results of a replace command on a dataset, according to some example embodiments. As shown in the user interface 3600 shown in Figure 36, the record containing "Drive" is now replaced with "Dr.". As a result, records containing "Drive" in the "street_address" column 2810 are replaced with "Dr.".

After the replacement commands are executed, the performed replacement commands (eg, street address string replacement) can be added to the transformation script for the data set. A transformation script indicates commands that are applied to columns of data. Transformation scripts can be used to apply the same commands to different data sets.

Exemplary embodiments thus provide a quick and efficient user interface for replacing data in a dataset.

6. Filtering Rows FIGS. 38 and 39 illustrate user interfaces 3800 and 3900 for implementing row filtering commands, according to some example embodiments. FIG. 40 illustrates a user interface 4000 displaying results of row filtering commands on a dataset, according to some example embodiments.

In a filtering operation, the user highlights (or otherwise selects) an example text to generate a regular expression that corresponds to the pattern, and then filters the data to include data that matches the pattern (or excluded). Examples of "row filtering" operations are shown in FIGS.

FIG. 41 depicts a flowchart of a method 4100 for executing row filtering commands, according to some example embodiments. In the example shown in Figure 41, the user wishes to filter the records in the dataset to identify records that have "Avenue" in the street address.

At step 4110, the user can select the data that will be used to filter the records. In the example shown in FIG. 38, the user has selected 'Avenue' from the 'street address' column 2810 . Data can be selected by the user by highlighting the data to be used for filtering. For example, when the user is on an interactive user interface, data can be selected with a mouse or via gestures on the device's display.

At step 4120, after the user highlights data to be used for filtering, the user interface data model generator can automatically highlight corresponding data in other records. For example, the user interface data model generator may highlight the word "Avenue" in records 8, 13, and 19. That is, based on the highlighting performed for the first record 3820, the user interface data model generator determines what highlighting for the remaining records (eg, 2-25) in the "street_address" column 2810. It can be determined whether display should be performed.

At step 4130, a regular expression may be generated based on the highlighting selected by the user. Figure 39 shows a dialog box 3930 containing the generated regular expression.

FIG. 39 shows the generated regular expression 3932 so that the user can easily identify the data to be used as the basis for filtering. In FIG. 39, three examples are highlighted by the user and displayed in designated example 3933 . Thus, in the specified example 3933, three addresses appear. The user has selected enough examples 3933 (positive 3934 and negative 3935) from the "street_address" column of the spreadsheet to generate a regular expression corresponding to addresses ending in "Avenue". Three examples are shown, but the user can use one or more examples based on the results desired by the user.

Positive and negative examples are displayed in different colors or in any manner in which positive and negative examples are displayed differently within dialog box 3930 and within data sets displayed on the user interface. can be In the designated example 3933 shown in FIG. 39, two addresses containing "street" are selected by the user from the "street address" column 2810 as negative examples. That is, they are examples of data that the user does not want to appear in the record. However, if additional examples are given by the user, a more accurate regex representation to be applied to the filtering can be generated.

The user can modify the highlighting by changing the highlighting in the data in the "street_address" column 2810 or by changing the highlighting in the regular expression 3932. Changes to the 'street_address' column 2810 are reflected in the generated regular expression 3932 . In another example, a user may use similar techniques to define regular expressions corresponding to patterns that should be filtered out (rather than retained).

At step 4140, after the highlighting is done, the "Regex by Example" dialog box 3930 automatically appears on the screen. In the example shown in FIG. 39, the “Regex by Example” dialog box 3930 appears after all the records (eg, records 1-25) in the “street_address” column 2810 have been highlighted, but the “Regex by Example” dialog box 3930 appears. dialog box 3930 can also appear after initial entry (eg, highlighting the first record).

At step 4150, selection of the "filter rows" command is accepted. For example, the user can select the “Filter Rows” button 3931 in the “Regex by Example” dialog box 3930 . The user clicks the "Filter Rows" button 3931 in the user interface to initiate a filter option that filters out all other types of addresses leaving only "Avenue" addresses, as shown in FIG. be able to.

At step 4160, a confirmation regarding the row filtering command is received. Specifically, it accepts a selection of whether to retain the record identified by example 3933 (hold button 3937) or delete the record identified by example 3933 (delete button 3938). After selecting the "Keep" button 3934 or the "Delete" button 3935, the user can select the Create button 3936 to initiate filtering.

At step 4170, the row filtering data generator 2416 of the regular expression generator 2410 can automatically filter the records according to the filtering criteria entered by the user.

At step 4180, the results of row filtering are displayed. As shown in the user interface 4000 shown in FIG. 40, the records have been filtered to show only records with "Avenue" in the address. In this example, the spreadsheet contains over 25 records. For illustration, the first 25 records with "Avenue" in "street_address" are shown.

When a row filtering command is executed, the executed row filtering command (eg, street address column row filtering) can be added to the transform script for the dataset. A transformation script indicates commands that are applied to columns of data. Transformation scripts can be used to apply the same commands to different data sets.

Exemplary embodiments thus provide a quick and efficient user interface for filtering rows of data in a dataset.

VI. Negative Example Context In an exemplary embodiment, for a negative example in a nested manual, a search can be performed for the context corresponding to the positive example.

When the user interface is in nested manual mode, an example with an outer highlight of "Windows NT 6.1" and an inner highlight of "6.1" may be received. For example, the UI may highlight "Windows NT 6.0" in response to an initial example provided by the user. The UI, via the regular expression generator, could highlight "Windows NT 6.0" because "Windows NT 6.0" would match "Windows NT 6.1" given first as an example. for it will be judged. The user may then select "Windows NT 6.0" to indicate that "Windows NT 6.0" is not the desired match for "Windows NT 6.1". That is, the user can give negative examples by selecting "Windows NT 6.1". If the user clicks on "Windows NT 6.0" to create a negative example, the UI may only send "6.0" as the negative example. Thus, according to an exemplary embodiment, the UI can search for a match "Windows NT" context before executing the LCS. An exemplary embodiment searches for context based on the given negative examples.

FIG. 50 shows a flowchart of a method 5000 for providing negative examples, according to some example embodiments.

At step 5010, an initial selection may be accepted. A regular expression is generated based on the initial selection.

For example, the user can select "Windows NT 6.1" outer highlighting and "Windows NT 6.1" "6.1" inner highlighting. Outer highlights and inner highlights can be created by highlighting data. For outer highlighting, the user can highlight "Windows NT 6.1", and for inner highlighting, the user can highlight "6.1" in the already highlighted "Windows NT 6.1". . The outer highlighting corresponds to the application name and the inner highlighting to the version number. The outer highlighting can appear in a different color than the inner highlighting. Highlighting can be performed by, for example, using a mouse or gestures on the device's display when the user is on the interactive user interface.

At step 5020, after the user highlights the desired data fragment (e.g. inner and outer highlighting), the user interface data model generator automatically highlights the corresponding data in the same record or the rest of the records. can do. For example, the user interface data model generator can highlight all instances of "6.1" within a record. However, the regular expression generator can also highlight all instances of "Windows NT 6.0" from the records in the dataset. Therefore, the user can provide negative examples to ensure high precision in the results.

At step 5030, the user can provide negative examples. Negative examples may be provided because the user does not want all applications and version numbers automatically highlighted by the UI. For example, the user may select "Windows NT 6.0" to give negative examples. The UI can then identify all records that contain a version number of "6.0". However, this can result in identifying applications other than "Windows NT" with a "6.0" version number.

At step 5040, the UI determines the context from the negative examples. Instead of highlighting all version numbers, including the "6.0" version number, the UI determines the context from the example given by the user. That is, the UI not only identifies the version number "6.0" from the data records, but also identifies the application name "Windows NT" from the records when identifying records containing negative examples. Therefore, instead of simply searching for records containing "6.0" when identifying records containing negative examples, the UI will search for "Windows NT 6.0" when identifying records containing negative examples.

Determining context from negative examples is described in more detail below with respect to FIG.

At step 5050, the highlighting on the dataset is updated according to the identified context. The UI highlights words containing "Windows NT 6.0" instead of just "6.0", thereby making negative examples more accurate.

The generated regular expression can be updated based on the context identified from the negative examples.
Thus, exemplary embodiments provide a more accurate method of giving negative examples. Context is determined prior to running the LCS algorithm to determine the longest common subsequence shared by different sets of one or more regular expression codes. By judging the context from the examples given, more precise regular expressions can be generated.

FIG. 51 shows a flowchart of a method 5100 for determining context from negative examples, according to some example embodiments. FIG. 51 describes step 5040 of FIG. 50 in more detail.

In step 5110, given a negative example, it is determined whether there are multiple highlights and whether those multiple highlights are nested (inner + outer highlighting). As shown in FIG. 51, the context of a negative example is not determined unless a negative example is provided (eg, step 5030 of FIG. 50).

At step 5120, context is obtained from the data to the left of the negative example. Specifically, context is derived from the data to the left of where the negative highlighting is embedded. The embedded location of the negative highlighting can be determined prior to obtaining the context. The code to perform this step is val eLookBehindStart = r.s.slice(0, r.highlightSpans.min).foldRight(es.slice(0, highlightSpanRange.start).reverse)((sElem,esLookBehind) => esLookBehind. Can include dropWhile(_.cc == sElem.cc)).length.

In one exemplary embodiment, the furthest left span is used. eLookBehindStart gets the span number of the leftmost span (from the left edge of negative highlighting) from the left walk.

Step 5120 may be an iterative process. For example, filtering may be performed for each span to the left of where negative highlighting is embedded. Fragment highlighting in the data set can be removed for each span of fragments that do not match the span of the negative example. At every iteration, the list of spans under consideration will be shortened.

At step 5130, context is obtained from the data to the right of the negative example. Specifically, it takes context from the data to the right of where the negative highlighting is embedded. The code for this step is val eLookAheadEnd = r.s.slice(r.highlightSpans.max+1, r.s.length).foldLeft(es.slice(highlightSpanRange.end, es.length))((esLookAhead,sElem) => esLookAhead.dropWhile (_.cc == sElem.cc)).length can be included.

In one exemplary embodiment, the furthest right span is used. eLookAheadEnd gets the span number of the rightmost span (from the right edge of negative highlighting) from the right walk.

Step 5130 may be an iterative process. For example, filtering can be performed on each span of the fragment to the right of where the negative highlighting is embedded. Fragment highlighting of the dataset can be removed for each fragment that does not match the span of the negative example. At every iteration, the list of spans under consideration will be shortened.

At step 5140, fragment filtering is performed. Highlighted fragments that do not correspond to the left and right contexts of negative examples (eg, the contexts identified in steps 5120 and 5130) are removed.

Context is used on the left and right sides of negative examples, but for example, if only left context data or only right context data are present, only left or right context may be used to identify context for negative examples. good. Also, although the left context is identified before the right context in the example described in FIG. 51, the right context may be identified before the left context. Further, although the above examples are described in terms of determining context with respect to negative examples, it is also possible to determine context with respect to positive examples.

Exemplary code for retrieving context corresponding to negative examples in nested manual mode can include some or all of the following code:

VII. Regular Expression Generation Using Longest Common Subsequence Algorithm Over Span A further aspect described herein relates to the generation of a regular expression based on the LCS algorithm from one or more data input character sequences, wherein: Instrument 110 can also handle characters that are present in only some of the examples. To handle characters that are only present in some input examples, spans can be defined in which both the minimum and maximum number of occurrences of the regular expression code are tracked. For example, for the "9pm" and "9 pm" character sequence inputs, there is an optional space between the digits and the "pm" text. In such cases, the minimum number of occurrences is set to zero if there may not be a constant span (e.g. a single space between '9' and 'pm') in all of the given input examples. may These minimum and maximum numbers can then be mapped to a regular expression multiplicity syntax. A Longest Common Subsequence (LCS) algorithm may be run on spans of characters derived from input examples, including "optional" spans (e.g. minimum length of zero) that do not appear in all input examples. . Consecutive spans may be merged during execution of the LCS algorithm, as described below. In such cases, when additional optional spans carried together result in consecutive occurrences, the LCS algorithm may be recursively performed on those optional spans as well. . That is, although the execution of the LCS algorithm is recursive in nature, in these cases the entire LCS algorithm may be executed recursively (eg, the recursive LCS algorithm is executed recursively). Among other technical advantages, this may enable shorter, cleaner, and more readable regular expression generation. For example, (am | am) (i.e., with an optional whitespace before am) might be produced without recursively running the LCS algorithm, whereas recursively running the LCS algorithm yields Regular expressions can result in shorter, cleaner ( ?am) being produced.

FIG. 18 is a flowchart illustrating a process 1800 for generating regular expressions with optional spans using the longest common subsequence (LCS) algorithm, according to one or more embodiments described herein. is. At step 1801, regular expression generator 110 may receive as input data one or more character sequences corresponding to positive regular expression examples. At step 1802, regular expression generator 110 may convert the character sequence into a regular expression code. Accordingly, steps 1801 and 1802 may be similar or identical to the previous corresponding examples described above. Then, at step 1802, the regular expression code may be further transformed into a span data structure (or span). As noted above, each span may include a data structure that stores a character class code (eg, Regex code) and repeat count range (eg, minimum count and/or maximum count). At step 1804, regular expression generator 110 may run the LCS algorithm to provide the set of spans as input to the algorithm. The output of the LCS algorithm in this example may include an output set of spans including at least one span with a minimum repeat count range equal to zero corresponding to an optional span in the output of the LCS algorithm. Finally, at step 1805, regular expression generator 110 may generate a regular expression based on the output of the LCS algorithm's output, including the optional span.

In some embodiments, the regular expression generator 110 may use single span alternations to improve readability and achieve an appropriate level of restriction. For illustration purposes, after the LCS algorithm has been executed in step 1804, its output (i.e., the intermediate output before final conversion to a regular expression) is a sequence of spans each having a minimum and maximum iteration count. can be done. A simple direct conversion from a list of spans to a regular expression can result in multiple regular expression codes marked as optional (eg, using question mark '?' modifier codes). In contrast, in some cases it may be desirable to generate a regular expression containing alternations, which can be expressed as vertical bar-separated alternatives enclosed in brackets (https:// (see www.regular-expressions.info/altemation.html). For example, a direct conversion of a span over time to a regular expression is

can be However, in this example, the regular expression generator 110 converts the regular expression into a more readable

can be configured to convert to To perform that transformation, the regular expression generator 110 may keep track of the original string fragments that underlie the span, and if all original string fragments appear in at least two given positive examples, , that a list of their original string fragments should be output as alternatives separated by vertical bars (e.g., rather than multiple optional regular expression codes).

FIG. 19 is an exemplary diagram showing the generation of a regular expression using the longest common subsequence (LCS) algorithm, the generated regular expression including an optional span. In this example, the two input data character sequences are "8am" and "9 pm". The input data character sequence is first converted to regular expression code (step 1802) and then to spans (step 1803), as described above. A span may be provided as an input to the LCS algorithm (step 1804), and the LCS output includes an optional span Z ^{<0, 1>} , an optional single space between numbers and two-letter text sequences. Show what is possible. That is, the superscript notation in this example may include two numbers, a minimum repeat count range (eg, 0) and a maximum repeat count range (eg, 1), applied to the preceding code (eg, Z=blank). Finally, a regular expression may be generated based on the output span of the LCS algorithm, and an optional span may be converted to the corresponding regular expression code "pZ ^* ".

In some embodiments, the optional rendering and use of whitespace by the regular expression generator 110 during execution of the LCS algorithm may provide additional technical advantages in terms of performance and readability. For example, when generating regular expressions, it is sometimes desirable to be able to handle both characters that are common among all given examples and characters that are present only in some of those examples. .

In some embodiments, both the minimum number of occurrences of category codes and the maximum number of occurrences of category codes may be tracked for each span data structure. If there are no spans at all in one or more of the given examples, the minimum is set to zero. As another example, to generate a regular expression for dealing with spelled months, use the minimum and maximum numbers, then the regular expression multiplicity syntax with curly braces (e.g. [A-Za-z ]{3,9}).

In some embodiments, regular expression generator 110 may track the minimum and maximum number of occurrences for each span, but may handle additional implementation details. For example, as a result of the combination of handling optional spans and performing LCS on spans of characters, regular expression generator 110 detects and merges consecutive spans through execution of the LCS algorithm. can be configured as In addition, any additional optional spans that are carried together may occasionally appear consecutively, and it may be desirable to recursively run the LCS algorithm on them as well. For example, in some cases, regular expression generator 110 modifies the LCS algorithm to favor (or weight) fewer transitions between optional and required sequence elements (e.g., spans) and/or Expand. For example, grouping optional spans together can minimize the number of grouping parentheses that must be used within the regular expression, thus improving human readability of the generated regular expression. can do. In some cases, if the resulting lengths are equal even after considering the optional span, the regular expression generator 110 chooses an alternative with fewer transitions between the optional span and the required span. You may indicate your preference. For example, in some cases a standard LCS algorithm may be implemented to prefer the selection of longer sequences at its decision points. However, at decision points where the alternatives are of equal length, configuration preferences may be programmed into regular expression generator 110 . One such configuration preference may be, for example, favoring shorter sequences (when optional spans are considered). Thus, the customized LCS within this configuration can simultaneously optimize longer sequences (of desired span) and shorter sequences (of desired span plus optional span).

In some embodiments, the implementation of the LCS algorithm by regular expression generator 110 may be configured to select shorter outputs. As mentioned above, the LCS algorithm can generally be used to find the longest common subsequence. For generation of regular expressions with spans, the LCS algorithm can be used to identify the longest sequence of required sequence elements (in our case the required span). However, in embodiments involving optional spans, the LCS algorithm is designed to retain the original goal of maximizing the number of common sequence elements (e.g., spans) while simultaneously minimizing the number of optional spans. may be configured. That is, in the original LCS algorithm, either consume sequence elements (e.g. spans) from the first example string or Any choice can be made as to whether to consume In such cases, the regex generator 110 may implement a modified version of the LCS algorithm that can choose the one that results in a shorter overall regex length when the optional span is also considered. . The shorter the regular expression in the final result, the better the readability.

Additionally, in some embodiments, the implementation of the LCS algorithm by regular expression generator 110 may be configured to prefer a larger number of required spans. That is, given the choice, rule expression generator 110 may select LCS output with fewer optional spans to improve human readability.

In some embodiments, the generated regular expression does not start the regular expression at an optional span, but at a required span (which can also serve as a mental anchor for human readers). can be more readable. Thus, in some cases, if the resulting alternatives have an equal number of transitions, the alternative with the earlier non-optional span may be selected. Additionally, the LCS algorithm performed by the regular expression generator 110 is configured in some embodiments to push all whitespace (including optional spans corresponding to whitespace) to the right within the regular expression. may be Pushing all whitespace to the right can increase the chances that spans of whitespace can be merged together, which can simplify the resulting regular expression and improve readability. Thus, if during the execution of the LCS algorithm it is determined that two sets of substrings have the same LCS, instead of arbitrarily choosing one of the two sets of substrings, readability A set may be selected that facilitates the improvement of Additionally, in some embodiments, the LCS algorithm may be configured to prefer a higher number of required spans and/or a lower number of optional spans to improve readability.

As noted above, in some cases negative examples may be based on an optional span. For example, a user may provide 'ab' and 'a2b' positive examples and 'a3b' negative examples. In this case, an exemplary implementation may fail because it may attempt to discriminate based only on the required span and the digit "2" is in the optional span. In such cases, the user can be warned of the failure, and via the user interface to manually repair the generated regular expression and/or remove some of the negative examples. can be offered a choice.

In some embodiments, there may be isSuccess returned as part of the JSON coming back from the REST service. In some embodiments, the generated regex may be a different color (eg, red) when isSuccess=false.

As mentioned above, the regular expression generator 110 can use single span alternations in some cases. Further, in some embodiments, regular expression generator 110 may be configured to perform multi-span alternation. That is, sometimes some spans in a row are

can form an alternation such as the date represented as . To detect these, the regular expression generator 110 may first discover which spans from the output of the LCS algorithm are actually used in all given input positive examples. These common spans, together with the phantom span before the first span and the phantom span after the last span, may constitute the anchor point. Between every pair of consecutive anchor points (which also have at least one non-anchor span in between) there is a "bridge" that must be crossed. Then, for each bridge, the regular expression generator 110 (a) determines the string fragments covered by the bridge span for each input positive example; The string fragments may be converted to regular expressions, then (c) remove duplicates from the above and put them into regular expression alternation syntax.

Instead of the alternatives in alternation appearing in arbitrary order, they can also be sorted alphabetically. By having a more deterministic output, the generated regex is more sensitive to small changes in the input example, such as during typing when the regex is generated in real-time with each character the user types. may not change much. In addition, some advanced Regex interpreters can explicitly use greedy versus non-greedy strategies when encountering alternations, depending on whether a greedy quantifier is present. However, other Regex interpreters, such as those found in standard Java and Javascript APIs, may simply try alternatives within an alternation in the order in which they appear. To compensate for this commonly found weakness, Regex generation may output longer alternatives within the alternation first. Alphabetical order may also be used as a secondary sorting criterion to break any ties.

In some embodiments, the regular expression generator can be configured to collapse spans that appear in a small number of positive examples to non-greedy wildcards .*?. That is, the example input may be a little more verbose. The regex generator uses the wildcard .*? when detecting significant variance in the number of spans and original fragment content, rather than trying to match every word and their specific word length. may In some embodiments, a span must meet one of the following three conditions in order for it to be considered for participation in wildcard .*? such hiding, etc. not. (1) the span may (exactly) contribute to less than one-third of the total number of examples entered, (2) the span is a SPACE span, and (3) the original text for the span All of the fragments occur infrequently and have "unique punctuation" in the solution. "Rarely" can mean (strictly) less than two-thirds of the number of instances. "Unique punctuation" means the presence of a SYMBOL or PUNCTUATION span anywhere in the solution, or the solution is located with a ^ at the beginning of the text or a $ at the end of the text. means to be positioned by

In some cases, eol wildcards may be prevented from touching hidden wildcards inside highlighting. Recall that if ^ or $ are used, the wildcard appears at the opposite end. For example, ^([A-Z]+).*? matches the first word. Indiscriminate hiding of wildcards would not give the Regex interpreter anything concrete to match against, if you end up with something like ^(.*?).*? There is To solve this, ^ is used so that if the last span in the capturing group is part of hiding up to the wildcard .*?, it will be used for the negative character class bordering the wildcard. , one additional span after the capturing group is used in the output before the end-of-line wildcard is emitted. Similarly, if $ is used and the first span in the capturing group is part of hiding up to the wildcard, then one additional span before the capturing group is explicitly used in the output. .

In some embodiments, symbols and punctuation marks may no longer be spannable, eg, to allow a highlighting endpoint to come between two punctuation marks. In such examples, each punctuation mark and each symbol may take up its own PUNCTUATION or SYMBOL span. In other cases, "A.,S" may result in three spans, LETTER (with original fragment A), PUNCTUATION with original fragment (.,), and LETTER (with original fragment S). . However, in these embodiments there may be four spans. In this example, it can be helpful if the comma acts as a delimiter in the comma-separated list, especially if there are multiple user highlights per example. In this example, the dots (periods) may be redundant and should not be in the same span as the commas.

Additionally, for spans of type ALPHANUMERIC, the regular expression generator can keep track of how many character and number spans have been replaced. Replace multiple LETTER and NUMBER spans with a single ALPHANUMERIC span, since many aspects of the algorithm, such as determining the start and end of inner highlighting in the overall solution, keep track of position by span indices. These indices and positions are then discarded. Therefore, it may be necessary to maintain a mapping from pre-substitution indices to post-substitution indices. Moreover, these need to be tracked on an example-by-example basis, because generally each example contains all indices from the global solution (including optional spans for spans not represented in 100% of the examples). because it is not. A class member numPreSubstitutedSpans has been added to the Span class with the following types and default values:
numPreSubstitutedSpans:Map[ID,Int]=orig.flatMap(_.fromExamples).distinct.map((_,1)).toMap
In some embodiments, a commonly available Regex API can be used to find the end offset of a capture group. For example, when using substitution commands in nested auto-outer mode, as described in connection with substitution commands that may be implemented in some user interfaces, additional capturing groups follow the regular capturing groups (three in all). for one, like the previous one). The user interface can perform this substitution, and similarly employ techniques for finding the closing parenthesis of capturing groups to avoid having to implement a full parser in the user interface. This can be more complicated than finding the left parenthesis of a capturing group, because there is no easy Because there is no way. Distinguishing the left parenthesis may be possible in some cases by looking for ?: using a lookahead ad. In some embodiments, the search may employ techniques that require examples of something known to match a regular expression. By comparing the captured groups concatenated together with the original example text, it may be verified that the regular expression is completely covered by the capturing groups without gaps. The code for this technique is shown below:

VIII. Regular Expression Generation Using the Combinatoric Longest Common Subsequence Algorithm A further aspect described herein is that the LCS algorithm performed by the regular expression generator 110 is run multiple times to produce a "correct" regular expression (e.g., A regular expression that properly matches every given positive example and properly excludes every given negative example), and/or from which the most desirable or optimal regular expression can be selected. It relates to combinatorial search that can generate regular expressions. For example, during combinatoric search, the entire LCS algorithm and regular expression generation process may be run multiple times, including different combinations/permutations of text processing directions, different positioning, and other different properties of the LCS algorithm.

FIG. 20 is a flowchart illustrating a process 2000 for generating regular expressions based on combinatorial execution of the longest common subsequence (LCS) algorithm. At step 2001, the regular expression generator 110 may receive an input data character sequence corresponding to positive examples. At step 2002, the regular expression generator 110 may iterate over various different combinations of execution techniques for the LCS algorithm. As shown in this example, during each iteration of step 2002, regular expression generator 110 generates the following LCS algorithm execution parameters (or properties): position specifier (i.e. no position specifier, at start of line positioning, positioning at the end of a line), processing direction (i.e., right-to-left order, left-to-right order), whitespace-pushing (i.e., whitespace-pushing or not), and hiding spans (collapse ) (ie, span hiding or not), can be selected. In step 2003, the LCS algorithm is run on the input data character sequence (or on the regular expression code if the input character sequence was first transformed) and the LCS algorithm is applied to the parameters/characteristics selected in step 2002. constructed based on In step 2004, the output of the LCS algorithm may be stored by the regular expression generator 110 and may include data such as whether the LCS was successfully identified by the algorithm and the length of the corresponding regular expression. In step 2005, the process may iterate until the LCS algorithm has been run with all possible combinations of combinatorial search parameters/properties. Finally, at step 2006, a particular output from one of the LCSs is selected as the optimal output (eg, based on success and regular expression length) and a regular expression is generated based on the selected LCS algorithm output. obtain.

In various embodiments, combinatorial searches such as those described above with reference to FIG. 20 may be performed for a variety of different combinations of parameters/characteristics. For example, in some embodiments, the LCS algorithm uses the caret symbol ^ to locate the regular expression to the beginning of the text and/or the dollar symbol $ to locate the regular expression to the end of the text. may be used. In some cases, such positioning can result in the generation of shorter regular expressions. A locator can be particularly useful when a user wishes to find a particular pattern at the beginning and/or end of a string. For example, the user may want the product name at the beginning. To avoid confusing the LCS algorithm with the varying number of words describing the product name, the caret can be used to position Regex to the beginning of the string, as shown in the image below. .

Further, in some embodiments, the LCS algorithm may be run with input data that is either forward or backward (or similarly, the LCS algorithm receives input data in normal order and then may be configured to reverse the order before running the algorithm). Thus, in some embodiments, a combinatorial search of LCS algorithms that may be performed on input character sequences or code pairs may be as follows.

1. 1. Normal (right to left) order, no positioning to start or end. 3. In normal (right-to-left) order, position relative to the beginning of the line using the caret ^. 4. Position to end of line using normal (right to left) order, dollar $. 4. Reverse (left to right) order, no positioning to beginning or end. 5. In reverse (left to right) order, use caret ^ to position to start of line. In this example using dollar $ to position to the end of the line in reverse (left to right) order, of the 6 runs of LCS, the shortest resulting regular expression may be selected. (Step 2006).

In some embodiments, the combinatorial search of the LCS algorithm may iterate over the greedy quantifier '?' and the non-greedy quantifier '??'. For example, by default, one question mark is emitted if an optional span is present, e.g., for first and last names with optional middle initials

is. If no satisfying regular expression is found when using greedy quantifiers, combinatoric search replaces all question mark quantifiers with double question mark quantifiers (e.g.

You can try to replace it with A double question mark corresponds to a non-greedy quantifier, which can instruct downstream regular expression matchers to enter backtracking mode in order to find a match.

Additionally, in some embodiments, the combinatorial search of the LCS algorithm may also iterate on whether to favor whitespace on the right. For example, as noted above, in some embodiments that push whitespace to the right, whitespace spans are merged together if, for example, the LCS algorithm faces an arbitrary choice of otherwise equal alternatives. , a strategy may be used in hopes of resulting in a lower number of overall spans. This feature adds another option to the combinatoric search, namely to either push blanks to the right or to run according to the conventional LCS approach which leaves the decision arbitrary.

Additionally, in some embodiments, the combinatorial search of the LCS algorithm may also iterate for scan/non-scan for literals common to all examples by performing LCS on the original string. . In such embodiments, the LCS algorithm may be configured to identify and align common words. As used herein, "common words" may refer to words that appear in all positive examples. Once a common word is identified, its span type may be converted from LETTER to WORD, and subsequent runs through the LCS algorithm may then naturally align with it.

Therefore, in the example below, the combinatoric search may iterate over several parameters/characteristics to reach 96 times the full LCS algorithm is run. The various parameters/properties to be repeated in this example are:
Locator (3) (value = ^, $, or neither)
・Push blank (2) (value = Yes or No)
Coalescence of low cardinality spans into wildcards (2) (value = Yes or No)
・Greedy quantifier? (2) (value = Yes or No)
Alignment (2) on common tokens of the LCS algorithm (value = Yes or No)

As mentioned above, in this example the full LCS algorithm is run 96 times (eg 3*2*2*2*2*2=96).

However, in other embodiments, the regular expression generator 110 may provide performance enhancements whereby only the first three properties of the above list (locator, push whitespace, and low Coalescence of cardinality spans into wildcards) may participate in the combinatoric search. This can result in a much smaller number of complete LCS algorithms being executed (eg 3*2*2=12 times). In such an embodiment, the last three properties in the above list (Greedy quantifier, alignment on common tokens of the LCS algorithm, and

) do not participate in the combinatoric search, but these properties can be tested individually and sequentially at the end. A technical advantage may be realized in such an embodiment, because splitting the search space in this way still yields a satisfactory regular expression, This is because it can be the result that is found.

To illustrate, the following example of combinatoric search may offer performance advantages over the previous example. In this example, a combinatoric search can be performed based on the following parameters/properties to be iterated:
・Position specification (3): BEGINNING_OF_LINE_MODE (beginning of line mode), END_OF_LINE_MODE (end of line mode), NO_EOL_MODE (no end of line mode)
order/direction (2): right-to-left (normal) LCS vs. left-to-right (reverse) LCS
Push (2): Whether or not to try to push spaces to the right in the LCS algorithm Compress to wildcards (2): Try to compress long sequences of spans that occur only occasionally to wildcards . ^* ? Do or not The combinatorics in this example can result in running the complete algorithm 3*2*2*2=24 times. The regular expression generator 110 may then take the best of the 24 results of the LCS algorithm, where "best" means (a) the LCS algorithm was successful and (b) the shortest regular expression was generated. The regular expression generator 110 can then perform the following three additional tasks.
1. A sequence of letters and numbers uninterrupted by spaces, punctuation, or symbols

Attempts to compress up to a new span type called ALPHANUMERIC, corresponding to the generated regex of . This can be useful for hex numbers found in IPv6 addresses from clickstream logs.
2. Try to use the non-greedy quantifier ?? instead of the greedy quantifier ?.
3. Try to align on literals.

A. Span Highlighting Alignment A positive or negative example can contain multiple highlights. Each example given can include multiple highlights (eg, an outer highlight and an inner highlight). Accordingly, exemplary embodiments provide a method for efficient and accurate processing when multiple examples each have their own highlighting.

Each example is decomposed before highlighting, on highlighting, and after highlighting. The full algorithm is run on each set, each set including pre-highlight, on-highlight, and post-highlight. User highlighting on multiple examples will be supported even if there is a large degree of variability between the examples given, avoiding highlighting failures.

FIG. 52 shows a diagram 5200 for generating regular expressions based on span highlight alignment, according to some example embodiments.

FIG. 52 shows the execution of highlight alignment for two data examples “Jane Doe” 5210 and “David Williams Jr.” 5220 . As shown in Figure 52, the last name of each entry (eg, Doe and Williams) is highlighted. For example, a user may select "Doe" as an example in a first record and "Williams" as an example in another data record. Two examples are given for ease of explanation. However, highlight alignment can be performed for multiple data instances.

In the example shown in FIG. 52, the Regex is generated in parts (eg, three parts). There may be more or fewer portions depending on the information in the data cells of the data record. Regex can be generated for all "before highlight", "on highlight", and "after highlight" fragments. A first regex 5231 is generated for the "before-highlight" fragment, a second regex 5232 is generated for the "above-highlight" fragment, and a third regex 5233 is generated for the "after-highlight" fragment. generated by

Spans of generated regex (ie, intermediate results before final regex generation) are concatenated together. The generated Regex spans for the portions (eg, pre-highlight, over-highlight, and post-highlight fragments) are concatenated together.

FIG. 53 shows a flowchart of a method 5300 for performing span highlight alignment, according to some example embodiments. The steps of FIG. 53 may be performed by a regular expression generator such as the regular expression generators shown in FIGS.

At step 5310, an initial highlight selection can be accepted. For example, a user can select a text fragment from the first record in the column of data. In the example shown in Figure 52, the user has highlighted last name 5210 "Doe" in the name column. The selected fragment can be identified as the "highlighted" fragment. Fragments preceding the selected fragment can be identified as "pre-highlight" fragments. Fragments after the selected fragment can be identified as "after-highlight" fragments.

At step 5320, a second highlight selection can be accepted. For example, the user can select a text fragment from a second record within the same data column (eg, name column) as the first highlight selection. In the example shown in Figure 52, the last name 5220 "Williams" in the name column is highlighted. In the second example shown in Figure 52, the user has also highlighted a surname. The selected fragment can be identified as the "highlighted" fragment. Fragments preceding the selected fragment can be identified as "pre-highlight" fragments. Fragments after the selected fragment can be identified as "after-highlight" fragments. A user may wish to provide multiple examples, so the user may provide additional highlighted selections. However, the examples given may not line up explicitly.

At step 5330, fragment alignment is performed. Fragments occurring before highlighting are aligned together, fragments occurring after highlighting are aligned together, and fragments occurring after highlighting are aligned together. Thus, in the example shown in Figure 52, fragment 5211 containing "Jane" and fragment 5221 containing "David" are aligned because they occur before the highlighted fragment. Fragment 5210 "Doe" and fragment 5220 "Williams" are aligned together because they occur on the highlight. Fragments 5215 and 5222, which do not contain characters, are aligned as they occur after highlighting.

At step 5340, a regex for the "before highlight" fragment is generated. For example, a first Regex 5231 can be created. A first regex 5231 may be generated by a regex generator based on the previous data of the highlighted fragment.

At step 5350, a regex for the "highlighted" fragment is generated. For example, a second Regex 5231 can be created. A second regex 5232 may be generated by the regex generator based on the data in the highlighted fragment.

At step 5360, a regex for the "after-highlight" fragment is generated. For example, a third Regex 5231 can be created. A third Regex 5233 may be generated by the Regex generator based on the data after the highlighted fragment.

Regex is generated in three portions 5231, 5232, 5233 as shown in FIG. A first regex 5231 is generated for the "before highlight" fragment, a second regex 5232 is generated for the "on highlight" fragment, and then a third regex 5232 for the "after highlight" fragment. Regex 4233 is generated. Although the order of generating the first, second, and third regexes has been described, the order of generating the regexes can vary. Further, although three portions are described, more than three portions of the example data may be highlighted.

At step 5370, the spans for the three regexes are concatenated. That is, instead of concatenating the three resulting regexes together, spans (ie, intermediate results before final regex generation) are concatenated together.

At step 5380, the longest common subsequence for the three concatenated spans is determined. LCS may be determined for the output of running the LCS algorithm on the pre-highlight span, the above-highlight span, and the post-highlight span.

Accordingly, the exemplary embodiment provides a more accurate method of determining the longest common subsequence, as the longest common subsequence is determined based on similarly located data.

Certain exemplary embodiments also provide alphanumeric spans. Determining the start and end of inner highlighting, keeping track of position by span index, and discarding index and position by replacing multiple character and numeric spans with a single alphanumeric span be able to. Accordingly, the exemplary embodiment provides a mapping from pre-substitution indices to post-substitution indices. A mapping is generated that associates indices of alphanumeric spans with indices of one or more digits and one or more letters.

Furthermore, the mapping is tracked on an example-by-example basis since each example may not contain all of the indices from the global solution. A number of pre-permuted spans can be identified.

FIG. 54 shows a flowchart of a method 5400 of tracking spans, according to some example embodiments. Specifically, exemplary embodiments provide a method for determining how many character spans and number spans of an alphanumeric span are to be replaced.

At step 5410, one or more character spans and one or more numeric spans are replaced with a single alphanumeric span. For example, an HTML hexadecimal color code can be replaced with a single alphanumeric span instead of three different spans, such as a letter span and a numeric span. An HTML hexadecimal color code such as <span style="color:#FF030A">BUY NOW!</span> replaces three different spans (e.g., a character span, a number span, and another character span) with a single One alphanumeric span FF030A is used.

Some example data fragments may include hexadecimal "digits" AF. This hex digit can appear anywhere in the hex code. Thus, alphanumeric spans can be used to easily identify matches for such example data fragments.

At step 5420, example data fragments are identified that include alphanumeric spans that replace one or more character spans and one or more numeric spans.

At step 5430, a mapping is generated that identifies the pre-replacement index and the post-replacement index for the example data fragment. A mapping is generated for each example data fragment.

At step 5440, a number of pre-permuted spans may be identified.
FIG. 55 shows a user interface 5500 displaying punctuation spans and symbol spans, according to some example embodiments. As shown in FIG. 55, the user has provided a specified example 5533 . User-specified examples include examples 5534 and 5535 . Regex 5532 is generated based on specified example 5533 .

As shown in FIG. 55, the symbols and punctuation marks are no longer spanable so that the highlight endpoint can come between two punctuation marks.

Each punctuation mark and each symbol can have its own punctuation span or symbol span. For example, an example containing "A.,S" would contain four spans "A" "." "," and "S".

Exemplary embodiments provide improved Regex production aesthetics. Specifically, fragments can be associated with the example that gave rise to them. Fragments are associated back to the instance that spawned them. In an exemplary embodiment, the example that gave rise to the fragment can be added to a list of example numbers.

Additionally, certain exemplary embodiments provide an even more detailed method of associating a fragment with the example that spawned it. Examples can be tied to specific text fragments rather than just span objects as a whole. In this way, instead of just looking at the example number in a list of example numbers, the user can see the example itself, making it easier for the user to determine which example is associated with the fragment. be able to.

For example, for the following two example inputs, Example 1: "a#s#" and Example 2: "a#", the solution span is [(LETTER, [("a",[1,2])]) , (SYMBOL, [("#",[1,2])]), (LETTER, ["s",[1]]), (SYMBOL, ["#",[1]])] can be done. By having an example number available for each individual fragment, the algorithm searches for the start and/or end of highlighting (or beyond highlighting to establish context) when processing a particular example. can walk back and forth (left or right) when Each example can be decomposed into spans and an LCS merge of those spans into its own set of spans is performed.

For two other example inputs, namely Example 1: "8pm" and Example 2: "9am", the solution span is [(NUMBER, [("8",[1]),("9",[2 ])]), (LETTER, [("pm",[1]),("am",[2])]), (PUNCTUATION, [(".",[1,2])])] can contain. A fragment is therefore associated with the particular instance that generated it.

In another example, a Regex generated aesthetic may include that spans must come from the same set of examples to be merged. To facilitate matching, in certain exemplary embodiments spans may not be merged unless they come from the same set of examples.

Accordingly, embodiments of the present invention provide various mechanisms for generating accurate regular expressions. Regular expressions can be applied to data to get the data results desired by the user. Users can more easily and efficiently obtain desired information without extensive searching or data manipulation.

IX. Hardware Overview FIG. 21 shows a simplified diagram of a distributed system 2100 for implementing certain embodiments. In the illustrated embodiment, distributed system 2100 includes one or more client computing devices 2102 , 2104 , 2106 , and 2108 coupled to server 2112 via one or more communications networks 2110 . Client computing devices 2102, 2104, 2106, and 2108 may be configured to run one or more applications.

In various embodiments, server 2112 may be adapted to run one or more services or software applications that enable automated generation of regular expressions described in this disclosure. For example, in certain embodiments, the server 2112 can receive user input data transmitted from a client device, which is received by the client device via a user interface displayed at the client device. be. Server 2112 can then convert the user input data into regular expressions that are sent to the client device for display via the user interface.

In particular embodiments, server 2112 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services are web-based, such as a Software as a Service (SaaS) model, to users of client computing devices 2102, 2104, 2106 and/or 2108. It can be provided as a service or cloud service. A user operating a client computing device 2102, 2104, 2106 and/or 2108 may utilize one or more client applications to interact with the server 2112 to take advantage of the services provided by these components.

In the configuration shown in FIG. 21, server 2112 may include one or more components 2118 , 2120 and 2122 that implement the functions performed by server 2112 . These components may include software components that may be executed by one or more processors, hardware components, or combinations thereof. It should be appreciated that a wide variety of system configurations are possible that may differ from distributed system 2100 . Accordingly, the embodiment shown in FIG. 21 is an example of a distributed system for implementing an embodiment system and is not intended to be limiting.

A user may use a client computing device 2102, 2104, 2106 and/or 2108 to run one or more applications, which may generate regular expressions according to the teachings of this disclosure. A client device may provide an interface that allows a user of the client device to interact with the client device. The client device may also output information to the user through this interface. Although FIG. 21 shows only four client computing devices, any number of client computing devices may be supported.

Client devices can be various types of computing devices, such as portable handheld devices, general purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices. can include a coding system. These computing devices run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX or UNIX-like operating systems, Linux® Or Linux-based operating systems, for example, various mobile operating systems (for example, Microsoft Windows Mobile (registered trademark), iOS (registered trademark), Windows Phone (registered trademark), Android (registered trademark), BlackBerry (registered trademark), Palm OS ( Google Chrome® OS), including Google Chrome® OS). Portable handheld devices may include cellular phones, smart phones (eg, iPhones), tablets (eg, iPads), personal digital assistants (PDAs), and the like. Wearable devices may include Google Glass(R) head-mounted displays and other devices. Gaming systems include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox gaming consoles with and without Kinect gesture input devices, Sony PlayStation systems, Nintendo ) may include various game systems provided by ). A client device may be capable of executing a wide variety of applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use various communication protocols. .

Network 2110 may be any type of network known to those skilled in the art capable of supporting data communication using any of a variety of available protocols, such as TCP/ Including but not limited to IP (Transmission Control Protocol/Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, and the like. Solely by way of example, network 2110 can be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet. , public switched telephone network (PSTN), infrared networks, wireless networks (e.g., wireless networks operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol suites, Bluetooth and/or any other wireless protocols), and/or any combination of these and/or other networks.

Server 2112 may be one or more general purpose computers, dedicated server computers (examples include PC (personal computer) servers, UNIX servers, midrange servers, mainframe computers, rackmount servers, etc.), server farms , server clusters, or any other suitable configuration and/or combination. Server 2112 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization. For example, one or more flexible pools of logical storage that can be virtualized to maintain virtual storage for servers. In various embodiments, server 2112 may be adapted to run one or more services or software applications that provide the functionality described in the above disclosure.

A computing system within server 2112 may run one or more operating systems, including any of the operating systems described above, as well as commercially available server operating systems. Server 2112 may also be used for various other server applications including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, JAVA servers, database servers, etc. and/or middle-tier applications. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), and the like. .

In some implementations, server 2112 may include one or more applications for parsing and consolidating data feeds and/or event updates received from users of client computing devices 2102, 2104, 2106 and 2108. . As an example, data feeds and/or event updates can be real-time related to sensor data applications, financial stock exchanges, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, etc. may include, but are not limited to, Twitter feeds, Facebook updates or real-time updates received from one or more third party sources and continuous data streams, which may include events of not. Server 2112 may also include one or more applications for displaying data feeds and/or real-time events via one or more display devices of client computing devices 2102 , 2104 , 2106 and 2108 .

Distributed system 2100 may also include one or more data repositories 2114,2116. In certain embodiments, these data repositories can be used to store data and other information. For example, one or more of the data repositories 2114, 2116 can be used to store information such as new data strings that match system-generated regular expressions. Data repositories 2114, 2116 may exist in various locations. For example, the data repositories used by server 2112 may be local to server 2112 or remote from server 2112 and communicate with server 2112 via network-based or dedicated connections. . Data repositories 2114, 2116 may be of different types. In particular embodiments, the data repository used by server 2112 may be a database, eg, a relational database such as those provided by Oracle Corporation® and other manufacturers. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved to and from the database in response to commands in SQL format.

In particular embodiments, one or more of data repositories 2114, 2116 may be used by applications to store application data. Data repositories used by applications may be of various types, for example, key-value store repositories, object store repositories, or general-purpose storage repositories supported by file systems.

In certain embodiments, functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 22 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services, in accordance with certain examples. In the example shown in FIG. 22, cloud infrastructure system 2202 may provide one or more cloud services that users may request using one or more client computing devices 2204, 2206 and 2208. Cloud infrastructure system 2202 may include one or more computers and/or servers, which may include those described above with respect to server 2112 . The computers in cloud infrastructure system 2202 may be organized as general purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and/or combination.

Network 2210 may facilitate communication and exchange of data between clients 2204 , 2206 , and 2208 and cloud infrastructure system 2202 . Network 2210 may include one or more networks. The networks may be of the same type or of different types. Network 2210 may support one or more communication protocols, including wired and/or wireless protocols, to facilitate communication.

The example shown in FIG. 22 is only one example of a cloud infrastructure system and is not intended to be limiting. Note that in some other examples, cloud infrastructure system 2202 may have more or fewer components than those shown in FIG. 22, may combine two or more components, or , may have different configurations or arrangements of components. For example, although FIG. 22 shows three client computing devices, in alternate embodiments any number of client computing devices may be supported.

The term cloud service is generally used to refer to a service made available to users on demand by a service provider's system (eg, cloud infrastructure system 2202) over a communication network such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. As such, Customer may utilize cloud services offered by cloud service providers without purchasing separate licenses, support, or hardware and software resources for the services. For example, a cloud service provider's system may host an application and a user may order and use the application on demand over the Internet without purchasing infrastructure resources to run the application. Cloud services are designed to provide easy and scalable access to applications, resources and services. Several providers offer cloud services. For example, several cloud services such as middleware services, database services, Java cloud services, etc. are provided by Oracle Corporation of Redwood Shores, California.

In certain embodiments, the cloud infrastructure system 2202 can be implemented in various models such as software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS) models, including hybrid service models. A model may be used to provide one or more cloud services. Cloud infrastructure system 2202 may include a suite of applications, middleware, databases, and other resources that enable provisioning of various cloud services.

The SaaS model allows applications or software to be delivered to customers as a service over a communication network, such as the Internet, without the customer having to purchase the hardware or software for the underlying application. For example, a SaaS model may be used to allow customers to access on-demand applications hosted by cloud infrastructure system 2202 . Examples of SaaS services provided by Oracle Corporation(R) include services for human resource/capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, etc.

The IaaS model is commonly used to provide flexible computing and storage capabilities by offering infrastructure resources (eg, servers, storage, hardware and networking resources) as cloud services to customers. Various IaaS services are provided by Oracle Corporation (registered trademark).

The PaaS model is commonly used to provide platform and environmental resources as a service that allow customers to develop, run, and manage applications and services without having to procure, build, or manage environmental resources. be done. Examples of PaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, various application development solution services, and the like.

Cloud services are generally provided in an on-demand, self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by cloud infrastructure system 2202 via a subscription order. Cloud infrastructure system 2202 then performs processing to provide the services requested in the customer's subscription order. Cloud infrastructure system 2202 may be configured to provide one or more cloud services.

Cloud infrastructure system 2202 may provide cloud services via various deployment models. In the public cloud model, cloud infrastructure system 2202 may be owned by a third party cloud service provider and cloud services are provided to general public customers. This customer may be an individual or a business. In a private cloud model, cloud infrastructure system 2202 may function within an organization (eg, within a corporate organization) and services are provided to customers within this organization. For example, the customers may be various departments of the company, such as the human resources department, the payroll department, or individuals within the company. In a community cloud model, the cloud infrastructure system 2202 and the services provided may be shared by various organizations within the relevant community. Various other models may be used, such as hybrid models of the above models.

Client computing devices 2204, 2206, and 2208 may be of different types (eg, devices 2102, 2104, 2106, and 2108 shown in FIG. 21) and may be capable of operating one or more client applications. . A user may interact with cloud infrastructure system 2202 , such as requesting a service provided by cloud infrastructure system 2202 , by using a client device.

In some embodiments, the processing performed by cloud infrastructure system 2202 to provide management-related services may include big data analytics. This analysis may involve using, analyzing, and processing large data sets to detect and visualize various trends, behaviors, relationships, etc. within this data. This analysis may be performed by one or more processors, possibly by processing the data in parallel, performing simulations using the data, or the like. For example, big data analysis may be performed by cloud infrastructure system 2202 to determine regular expressions in an automated manner. The data used for this analysis may be structured data (e.g. data stored in a database or structured according to a structured model) and/or unstructured data (e.g. data blobs (binary large object: Binary Large Objects)).

As shown in the example of FIG. 22, cloud infrastructure system 2202 may include infrastructure resources 2230 that are utilized to facilitate provisioning of various cloud services provided by cloud infrastructure system 2202 . Infrastructure resources 2230 may include, for example, processing resources, storage or memory resources, networking resources, and the like.

In particular embodiments, resources are defined as sets of resources or resource modules (such as resource modules) to facilitate efficient provisioning of these resources to support various cloud services provided by cloud infrastructure system 2202 to different customers. (also referred to as a "pod"). Each resource module or pod may contain a pre-integrated and optimized combination of one or more types of resources. In certain embodiments, different pods may be pre-provisioned for different types of cloud services. For example, a first podset may be provisioned for a database service, and a second podset may be provisioned for a Java service, etc., which may contain a different combination of resources than the pods in the first podset. good too. For some services, resources allocated for provisioning these services may be shared between services.

The cloud infrastructure system 2202 itself may internally use services 2232 that are shared by different components of the cloud infrastructure system 2202 and that facilitate provisioning of services by the cloud infrastructure system 2202 . These internal shared services include Security Identity Services, Integration Services, Enterprise Repository Services, Enterprise Manager Services, Virus Scanning and Whitelisting Services, High Availability, Backup Recovery Services, Cloud Support Enabling Services, Email Services, and Notifications. services, file transfer services, etc., but are not limited to these.

Cloud infrastructure system 2202 may include multiple subsystems. These subsystems can be implemented in software or hardware, or a combination thereof. As shown in FIG. 22, subsystems may include a user interface subsystem 2212 that allows users or customers of cloud infrastructure system 2202 to interact with cloud infrastructure system 2202 . The user interface subsystem 2212 includes a variety of different interfaces, such as a web interface 2214, an online store interface 2216 where cloud services provided by the cloud infrastructure system 2202 are advertised and available for consumer purchase, and other interfaces 2218. obtain. For example, a customer may use a client device to request (service request 2234) one or more services that cloud infrastructure system 2202 provides using one or more of interfaces 2214, 2216, and 2218. good. For example, a customer may access an online store, browse cloud services provided by cloud infrastructure system 2202, and order a subscription for one or more services offered by cloud infrastructure system 2202 that the customer wishes to subscribe to. can do This service request may include information identifying the customer and one or more services that the customer wishes to offer. For example, a customer may place a subscription order for automatic regular expression generation related services provided by cloud infrastructure system 2202 .

In certain embodiments, such as the example shown in FIG. 22, cloud infrastructure system 2202 may include an order management subsystem (OMS) 2220 configured to process new orders. As part of this process, OMS 2220 creates an account for the customer if not already created and provides billing and/or account information that will be used to bill the customer for providing the requested service to the customer. is received from the customer, the customer information is verified, after verification, the order is reserved for the customer, and the order is prepared for provisioning by coordinating various workflows.

Once properly validated, OMS 2220 may invoke order provisioning subsystem (OPS) 2224 configured to provision resources for this order, including processing, memory, and networking resources. Provisioning may include allocating resources for an order and configuring resources to facilitate services requested by customer orders. The manner in which resources are provisioned for an order and the types of resources provisioned may depend on the type of cloud service ordered by the customer. For example, according to one workflow, OPS 2224 may be configured to determine the particular cloud service being requested and identify the number of pods that would have been pre-configured for this particular cloud service. . The number of pods allocated for an order may depend on the size/quantity/level/extent of service requested. For example, the number of pods to allocate may be determined based on the number of users the service should support, the time period for which the service is requested, and the like. The assigned pods may then be customized to the specific customer requesting them to provide the requested service.

Cloud infrastructure system 2202 may send a response or notification 2244 to the requesting customer to indicate when the requested service will be available. In some examples, the customer may be sent information (eg, a link) that allows the customer to begin using and taking advantage of the requested service. In certain embodiments, for customers requesting automatic regular expression generation related services, the response may include instructions that, when executed, cause the display of a user interface.

Cloud infrastructure system 2202 may provide services to multiple customers. For each customer, cloud infrastructure system 2202 manages information related to one or more subscription orders received from the customer, maintains customer data related to the orders, and provides requested services to the customer. play a role. Cloud infrastructure system 2202 may also collect usage statistics regarding customer usage of subscribed services. For example, statistics may be gathered about the amount of storage used, the amount of data transferred, the number of users, the amount of system uptime and system downtime, and the like. This usage information may be used to bill the customer. Billing may be done monthly, for example.

Cloud infrastructure system 2202 may provide services to multiple customers in parallel. Cloud infrastructure system 2202 may store information about these customers, possibly including copyright information. In certain embodiments, cloud infrastructure system 2202 is configured to manage customer information and segregate managed information such that information about one customer is not accessed from information about another customer. Also includes Identity Management Subsystem (IMS) 2228 . IMS 2228 may be configured to provide various security-related services, such as identity services, information access management, authentication and authorization services, services for managing customer identities and roles and related capabilities, and the like.

FIG. 23 shows an example computer system 2300 . In some embodiments, computer system 2300 may be used to implement any of the systems described above. As shown in FIG. 23, computer system 2300 includes various subsystems including processing subsystem 2304 that communicates with several other subsystems via bus subsystem 2302 . These other subsystems may include processing acceleration unit 2306 , I/O subsystem 2308 , storage subsystem 2318 , and communication subsystem 2324 . Storage subsystem 2318 may include non-transitory computer-readable storage media, including storage media 2322 and system memory 2310 .

Bus subsystem 2302 provides a mechanism for causing the various components and subsystems of computer system 2300 to communicate with each other as intended. Although bus subsystem 2302 is shown schematically as a single bus, alternative examples of bus subsystems may utilize multiple buses. Bus subsystem 2302 may be any of several types of bus structures including memory buses or memory controllers, peripheral buses, local buses, etc., using any of a variety of bus architectures. For example, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association ( Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured according to the IEEE P1386.1 standard.

Processing subsystem 2304 controls the operation of computer system 2300 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). A processor may include a single-core or multi-core processor. The processing resources of computer system 2300 can be organized into one or more processing units 2332, 2334, and so on. A processing unit may include one or more processors, one or more cores from the same or different processors, combinations of cores and processors, or other combinations of cores and processors. In some embodiments, processing subsystem 2304 may include one or more dedicated co-processors such as graphics processors, digital signal processors (DSPs), and the like. In some embodiments, some or all of the processing units of processing subsystem 2304 may use customized circuitry, such as an application specific integrated circuit (ASIC) or field programmable gate array (FPGA).

In some embodiments, the processing units within processing subsystem 2304 may execute instructions stored in system memory 2310 or computer readable storage media 2322 . In various examples, a processing unit may execute various programs or code instructions and maintain multiple programs or processes executing simultaneously. At any given moment, some or all of the program code to be executed may reside in system memory 2310 and/or computer readable storage medium 2322, potentially including one or more storage devices. . Through appropriate programming, processing subsystem 2304 can provide the various functions described above. In examples where computer system 2300 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

In certain embodiments, to accelerate the overall processing performed by computer system 2300, to perform customized processing, or to offload some of the processing performed by processing subsystem 2304. A process acceleration unit 2306 may optionally be provided for this purpose.

The I/O subsystem 2308 may include devices and mechanisms for inputting information into the computer system 2300 and/or for outputting information from or through the computer system 2300 . . In general, use of the term “input device” is intended to include all possible types of devices and mechanisms for entering information into computer system 2300 . User interface input devices include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens built into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice command recognition systems. Accompanying voice input devices, microphones, and other types of input devices may also be included. A user interface input device receives input using a Microsoft Kinect® motion sensor, a Microsoft Xbox® 360 game controller, gestures and voice commands that allow a user to control and interact with the input device Motion sensing and/or gesture recognition devices may also be included, such as devices that provide an interface for. A user interface input device detects eye movements from a user (e.g., "blinking" while taking a picture and/or making a menu selection) and recognizes the eye gestures as an input device (e.g., Google Glass). It may also include an eye gesture recognition device such as the Google Glass(R) blink detector that translates as an input to the (trademark)). User interface input devices may also include voice recognition sensing devices that allow a user to interact with a voice recognition system (eg, Siri® navigator) via voice commands.

Other examples of user interface input devices are three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphics tablets, as well as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, bar It may also include, but is not limited to, audio/visual devices such as code readers 3D scanners, 3D printers, laser range finders, and eye tracking devices. User interface input devices may also include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, and the like.

In general, use of the term output device is intended to include all possible types of devices and mechanisms for outputting information from computer system 2300 to a user or other computer. User interface output devices may include display subsystems, indicator lights, non-visual displays such as audio output devices, and the like. The display subsystem may be a flat panel device, such as one that uses a cathode ray tube (CRT), liquid crystal display (LCD) or plasma display, a planning device, a touch screen, or the like. For example, user interface output devices include various display devices that visually convey text, graphics and audio/visual information such as monitors, printers, speakers, headphones, automotive navigation systems, plotters, audio output devices and modems. but not limited to them.

Storage subsystem 2318 provides a repository or data store for storing information and data used by computer system 2300 . Storage subsystem 2318 provides tangible, non-transitory computer-readable storage media for storing the basic programming and data structures that provide the functionality of some examples. Software (eg, programs, code modules, instructions) that provide the described functionality when executed by processing subsystem 2304 may be stored in storage subsystem 2318 . Software may be executed by one or more processing units of processing subsystem 2304 . Storage subsystem 2318 may also provide a repository for storing data used in accordance with the teachings of this disclosure.

Storage subsystem 2318 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 23, storage subsystem 2318 includes system memory 2310 and computer readable storage media 2322 . System memory 2310 includes volatile primary random access memory (RAM) for storing instructions and data during program execution, and nonvolatile read only memory (ROM) or flash memory in which fixed instructions are stored. memory. In some implementations, a basic input/output system (BIOS), containing the basic routines that help transfer information between elements within computer system 2300, such as during start-up, is typically may be stored in ROM. RAM typically contains the data and/or program modules currently being manipulated and executed by processing subsystem 2304 . In some implementations, system memory 2310 may include multiple different types of memory such as static random access memory (SRAM), dynamic random access memory (DRAM), and the like.

By way of example, and without limitation, as shown in FIG. 23, system memory 2310 may include running applications, such as web browsers, middle-tier applications, relational database management systems (RDBMS), etc. Programs 2312, program data 2314, and operating system 2316 may be loaded. By way of example, operating system 2316 may include Microsoft Windows®, Apple Macintosh® and/or Linux operating systems, various commercially available UNIX or UNIX-like operating systems (including various GNU/Linux operating system, including but not limited to Google Chrome® OS, etc.) and/or iOS®, Windows® Phone, Android® OS, BlackBerry® OS , various versions of mobile operating systems such as the Palm® OS operating system, and the like.

Computer readable storage media 2322 may store programming and data structures that provide some example functions. Computer readable storage media 2322 may provide storage of computer readable instructions, data structures, program modules and other data for computer system 2300 . Software (programs, code modules, instructions) that provide the described functionality when executed by processing subsystem 2304 may be stored in storage subsystem 2318 . By way of example, computer readable storage medium 2322 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD ROM, a DVD, an optical disk drive such as a Blu-Ray disk, or other optical media. . Computer readable storage media 2322 may include Zip drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital video tapes, etc., including but not limited to Not limited. The computer readable storage medium 2322 includes flash memory based SSDs, enterprise flash drives, solid state drives (SSDs) based on non-volatile memory such as solid state ROM, volatile memory such as solid state RAM, dynamic RAM, static RAM. SSDs based on static memory, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

In particular embodiments, the storage subsystem 2318 may also include a computer readable storage media reader 2320 that is further connectable to computer readable storage media 2322 . Reader 2320 may be configured to receive and read data from memory devices such as disks, flash drives, and the like.

In particular embodiments, computer system 2300 may support virtualization techniques including, but not limited to, virtualization of processing and memory resources. For example, computer system 2300 may provide support for running one or more virtual machines. In particular embodiments, computer system 2300 may execute programs such as hypervisors that facilitate configuration and management of virtual machines. Each virtual machine may be assigned memory, computing (eg, processors, cores), I/O, and networking resources. Each virtual machine typically runs independently of other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from operating systems run by other virtual machines run by computer system 2300 . Thus, potentially multiple operating systems can be run by computer system 2300 at the same time.

Communications subsystem 2324 provides an interface to other computer systems and networks. Communications subsystem 2324 serves as an interface for sending and receiving data between other systems and computer system 2300 . For example, communications subsystem 2324 enables computer system 2300 to establish communications channels over the Internet to one or more client devices to send and receive information to and from the one or more client devices. can be

Communication subsystem 2324 may support both wired and/or wireless communication protocols. In some embodiments, the communication subsystem 2324 uses (e.g., cellular telephony, advanced data network technologies such as 3G, 4G or EDGE (high speed data rate for global evolution), WiFi (IEEE 802.XX family of standards, or other radio frequency (RF) transceiver components, global positioning system (GPS) receiver components, and/or Other components may be included, hi some embodiments, communications subsystem 2324 may provide a wired network connection (eg, Ethernet) in addition to or instead of a wireless interface.

Communication subsystem 2324 may receive and transmit data in various formats. In some embodiments, the communications subsystem 2324 receives input communications in the form of structured and/or unstructured data feeds 2326, event streams 2328, event updates 2330, among other forms. may For example, the communication subsystem 2324 may communicate with social media networks and/or web feeds such as Twitter feeds, Facebook updates, Rich Site Summary (RSS) feeds, and/or one or more third parties. It may be configured to receive (or send) data feeds 2326 in real time from users of other communication services, such as real time updates from information sources.

In certain embodiments, the communications subsystem 2324 may be configured to receive data in the form of a continuous data stream, which is continuous or infinite in nature with no definite end. An event stream 2328 and/or event updates 2330 of possible real-time events may be included. Examples of applications that generate continuous data include, for example, sensor data applications, financial stock exchanges, network performance measurement tools (e.g. network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, etc. can.

Communications subsystem 2324 may be configured to communicate data from computer system 2300 to other computer systems or networks. This data is communicated to one or more streaming data source computers coupled to computer system 2300 in a variety of different formats, such as structured and/or unstructured data feeds 2326, event streams 2328, event updates 2330, and the like. may be communicated to one or more databases to obtain.

Computer system 2300 includes handheld portable devices (eg, iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (eg, Google Glass® head-mounted displays), personal computers, workstations , mainframe, kiosk, server rack, or other data processing system. Due to the ever-changing nature of computers and networks, the description of computer system 2300 shown in FIG. 23 is intended only as a specific example. Many other configurations having more or fewer components than the system shown in FIG. 23 are possible. A person of ordinary skill in the relevant art will recognize other aspects and/or ways to implement the various examples based on the disclosure and teachings herein.

Although specific examples have been described, many variations, modifications, alternative constructions, and equivalents are possible. Examples are not limited to operation within a particular data processing environment, but are free to operate within multiple data processing environments. Furthermore, it should be apparent to those skilled in the art that while the examples have been described using a particular series of transactions and steps, this is not intended to be limiting. Although some flowcharts describe the operations as a sequential process, many of these operations may be performed in parallel or concurrently. Additionally, the order of operations may be respecified. A process may have additional steps not included in the figure. Various features and aspects of the above examples may be used individually or together.

Furthermore, although specific examples have been described using specific combinations of hardware and software, it should be understood that other combinations of hardware and software are possible. Certain examples may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination.

Where a device, system, component or module is described as being configured to perform a particular action or function, such configuration is, for example, by designing electronic circuitry to perform the action By programming a programmable electronic circuit (such as a microprocessor) to perform actions, e.g., programmed to execute code or instructions stored in a non-transitory memory medium, or any combination thereof computer instructions or code, or by executing a processor or core, or the like. Processes can communicate using a variety of techniques including, but not limited to, conventional techniques for inter-process communication, different pairs of processes may use different techniques, and processes in the same pair may communicate using different techniques. may use different techniques at different times.

Specific details are provided in this disclosure to provide a thorough understanding of the examples. However, the example may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail in order to avoid obscuring the examples. This specification provides illustrative examples only and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples will provide those skilled in the art with an enabling description for implementing various examples. Various changes are possible within the function and configuration of the elements.

The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that additions, subtractions, deletions and other modifications and changes may be made thereto without departing from the broader spirit and scope of the claims. Thus, although specific examples have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

Although the foregoing specification describes aspects of the disclosure with reference to specific examples thereof, those skilled in the art will appreciate that the disclosure is not so limited. Various features and aspects of the above disclosure may be used individually or together. Further, examples may be utilized in a variety of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

In the above description, the methods are described in a particular order for purposes of illustration. It should be appreciated that in alternate examples, the methods may be performed in a different order than the order described. The methods described above may also be performed by hardware components or machine-executable instructions which, when used, may be executed by a machine, such as a general-purpose or special-purpose processor or logic circuit programmed with such instructions. It should also be understood that the method may be embodied in a sequence of machine-executable instructions that may cause it to be performed. These machine-executable instructions are stored on one or more machine-readable media such as a CD-ROM or other type of optical disk, floppy disk, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory. It can be stored on a medium or other type of machine-readable medium suitable for storing electronic instructions. Alternatively, these methods may be performed by a combination of hardware and software.

When a component is described as being configured to perform a particular operation, such configuration includes, for example, designing electronic circuitry or other hardware to perform a particular operation; It may be accomplished by programming a programmable electronic circuit (eg, a microprocessor or other suitable electronic circuit) to perform the operations, or any combination thereof.

Although illustrative examples of the present application have been described in detail herein, the concepts of the present invention may be variously embodied and employed in other forms, and the claims may be subject to any claim if limited by the prior art. is intended to be construed to include such variations.

When a component is described as being "configured to" perform a particular operation, such configuration means, for example, designing electronic circuitry or other hardware to perform the particular operation; It may be accomplished by programming a programmable electronic circuit (eg, a microprocessor or other suitable electronic circuit) to perform a particular operation, or any combination thereof.

Claims (20)

A method of generating a regular expression comprising:
A regular expression generator comprising one or more processors receives a first selection including one or more positive character sequences, each of said one or more positive character sequences comprising said Corresponding to positive examples to be matched by the regular expression generated by the regular expression generator, the method further comprises:
The method includes the regular expression generator generating a first regular expression, the first regular expression matching the positive examples, the method further comprising:
the regular expression generator receiving a second selection including one or more negative character sequences, each of the one or more negative character sequences generated by the regular expression generator; corresponding to negative examples that should not be matched by said regular expression, said method further comprising:
Determining the context of the one or more negative character sequences corresponding to the negative example in response to receiving the second selection;
and updating the first regular expression based on the determined context of the one or more negative character sequences. 2. The method of claim 1, wherein receiving the first selection comprises receiving a selection of the one or more positive character sequences in a first data cell of a data set via a user interface. . Further comprising the regular expression generator automatically selecting character sequences within a plurality of data cells within the data set that correspond to a first selection that includes the one or more positive character sequences. 3. The method of claim 2. 4. The method of claim 3, wherein receiving the second selection comprises receiving a selection of the one or more negative character sequences in a second data cell of the data set via the user interface. the method of. Further automatically selecting, by the regular expression generator, character sequences within the plurality of data cells in the data set that correspond to a second selection that includes the one or more negative character sequences. 5. The method of claim 4, comprising: 4. The first selection is highlighted with a first highlighting format and the second selection is highlighted with a second highlighting format different from the first highlighting format. The method described in . Determining the context of the one or more negative character sequences corresponding to the negative examples includes:
identifying embedded highlight locations of the second selection;
determining context from data to the left of the embedded highlight location of the second selection;
and determining context from data to the right of the embedded highlight location of the highlighted second selection. Determining the context of the one or more negative character sequences corresponding to the negative examples further comprises:
the one automatically selected based on the determined context from the data to the left of the embedded highlight position and the determined context from the data to the right of the embedded highlight position; or filtering character sequences within the plurality of data cells in the data set corresponding to the first selection including a plurality of negative character sequences;
removing the filtered character sequences corresponding to the selected one or more negative character sequences from the selected character sequences in the plurality of data cells in the data set; 8. The method of claim 7. determining the context from data to the left of the embedded highlight location includes identifying a first span to the left of the embedded highlight location;
Filtering the character sequences in the plurality of data cells in the data set corresponding to the selected one or more negative character sequences comprises: 9. The method of claim 8, further comprising identifying spans in the character sequences in the plurality of data cells corresponding to which do not match the first span to the left of the embedded highlight position. determining context from data to the left of the embedded highlight location further includes identifying a second span to the left of the embedded highlight;
Filtering the character sequences in the plurality of data cells in the data set corresponding to the selected one or more negative character sequences comprises: 10. The method of claim 9, further comprising identifying spans in the character sequences in the plurality of data cells corresponding to which do not match the second span to the left of the embedded highlight position. determining the context from data to the right of the embedded highlight location includes identifying a first span to the right of the embedded highlight location;
Filtering the character sequences in the plurality of data cells in the data set corresponding to a second selection containing the one or more negative character sequences comprises: and further comprising identifying a span that does not match the first span to the right of the embedded highlight position in the sequence of characters in the plurality of data cells corresponding to a second selection comprising 7. The method according to 7. A regular expression generator server computer,
a processor;
memory;
a computer readable medium coupled to the processor, the computer readable medium storing instructions executable by the processor to implement a method, the method comprising:
A regular expression generator comprising one or more processors receives a first selection including one or more positive character sequences, each of said one or more positive character sequences comprising said Corresponding to positive examples to be matched by the regular expression generated by the regular expression generator, the method further comprises:
The method includes the regular expression generator generating a first regular expression, the first regular expression matching the positive examples, the method further comprising:
the regular expression generator receiving a second selection including one or more negative character sequences, each of the one or more negative character sequences generated by the regular expression generator; corresponding to negative examples that should not be matched by said regular expression, said method further comprising:
Determining the context of the one or more negative character sequences corresponding to the negative example in response to receiving the second selection;
and updating the first regular expression based on the determined context of the one or more negative character sequences. 13. The regularization of claim 12, wherein receiving the first selection comprises receiving a selection of the one or more positive character sequences in a first data cell of a data set via a user interface. A representation generator server computer. 14. The first selection is highlighted with a first highlighting format and the second selection is highlighted with a second highlighting format different from the first highlighting format. The method described in . Determining the context of the one or more negative character sequences corresponding to the negative examples includes:
identifying embedded highlight locations of the second selection;
determining context from data to the left of the embedded highlight location of the second selection;
and determining context from data to the right of the embedded highlight location of the highlighted second selection. Determining the context of the one or more negative character sequences corresponding to the negative examples further comprises:
the one automatically selected based on the determined context from the data to the left of the embedded highlight position and the determined context from the data to the right of the embedded highlight position; or filtering character sequences within a plurality of data cells in the data set corresponding to the second selection including a plurality of negative character sequences;
removing from the selected character sequences in the plurality of data cells in the data set the filtered character sequences corresponding to the second selection including the one or more negative character sequences. 16. The method of claim 15, comprising: A non-transitory computer-readable medium containing instructions configured to cause one or more processors to perform an action, the action comprising:
A regular expression generator comprising one or more processors receives a first selection including one or more positive character sequences, each of said one or more positive character sequences comprising said Corresponding to positive examples to be matched by the regular expression generated by the regular expression generator, the operation further comprises:
The regular expression generator generating a first regular expression, the first regular expression matching the positive examples, the operation further comprising:
the regular expression generator receiving a second selection including one or more negative character sequences, each of the one or more negative character sequences generated by the regular expression generator; corresponding to negative examples that should not be matched by said regular expression, said action further:
Determining the context of the one or more negative character sequences corresponding to the negative example in response to receiving the second selection;
and updating the first regular expression based on the determined context of the one or more negative character sequences. 18. The reading of claim 17, wherein receiving the first selection comprises receiving a selection of the one or more positive character sequences in a first data cell of a data set via a user interface. possible medium. Determining the context of the one or more negative character sequences corresponding to the negative examples includes:
identifying embedded highlight locations of the second selection;
determining context from data to the left of the embedded highlight location of the second selection;
and determining context from data to the right of the embedded highlight position of the highlighted second selection. Determining the context of the one or more negative character sequences corresponding to the negative examples further comprises:
the one automatically selected based on the determined context from the data to the left of the embedded highlight position and the determined context from the data to the right of the embedded highlight position; or filtering character sequences within a plurality of data cells in the data set corresponding to the second selection including a plurality of negative character sequences;
removing from the selected character sequences in the plurality of data cells in the data set the filtered character sequences corresponding to the second selection including the one or more negative character sequences. 20. The readable medium of claim 19, comprising:

Images