CN112214393A - Generating defect descriptions for software products - Google Patents
Generating defect descriptions for software products Download PDFInfo
- Publication number
- CN112214393A CN112214393A CN202010639448.3A CN202010639448A CN112214393A CN 112214393 A CN112214393 A CN 112214393A CN 202010639448 A CN202010639448 A CN 202010639448A CN 112214393 A CN112214393 A CN 112214393A
- Authority
- CN
- China
- Prior art keywords
- verb
- noun
- terms
- computer
- keywords
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000007547 defect Effects 0.000 title claims abstract description 40
- 238000000034 method Methods 0.000 claims abstract description 42
- 238000004590 computer program Methods 0.000 claims abstract description 8
- 230000004044 response Effects 0.000 claims abstract description 3
- 238000004883 computer application Methods 0.000 claims 3
- 238000010586 diagram Methods 0.000 description 29
- 230000006870 function Effects 0.000 description 29
- 238000012545 processing Methods 0.000 description 17
- 230000009471 action Effects 0.000 description 13
- 230000008569 process Effects 0.000 description 11
- 230000003068 static effect Effects 0.000 description 8
- 239000011159 matrix material Substances 0.000 description 7
- 230000007704 transition Effects 0.000 description 7
- 238000007726 management method Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000008520 organization Effects 0.000 description 3
- 241000288105 Grus Species 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 241000282412 Homo Species 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009172 bursting Effects 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000012417 linear regression Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013439 planning Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000010845 search algorithm Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000013179 statistical model Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3692—Test management for test results analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/73—Program documentation
Abstract
The invention relates to generating defect descriptions for software products. The invention provides a method, a computer system and a computer program product for generating a defect description. According to the method of the present invention, one or more keywords are received from a user, the keywords being used to describe defects encountered while using a software product. One or more terms matching at least one keyword and a path corresponding to the one or more terms are retrieved in an operation graph, wherein the operation graph includes all possible operational terms and paths of the software product describing a software product based user interface of the software product. And in response to retrieving the one or more terms and corresponding paths that match the at least one keyword, generating a defect description based on the one or more matched terms, the corresponding paths, and the received keywords.
Description
Technical Field
The present invention relates generally to defect descriptions, and more particularly to generating defect descriptions for software products.
Background
Defects or errors may be encountered during the development, testing, or even support phase of a software product. For example, the support may receive feedback from the user, QA testers may discover software product errors at test time, and developers may discover errors during development.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described herein in the detailed description. This summary is not intended to identify key factors or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
In one illustrative embodiment of the present invention, a computer-implemented method is provided in which one or more keywords are received from a user, the keywords being used to describe defects encountered while using a software product. One or more terms matching at least one keyword and a path corresponding to the one or more terms are retrieved in an operation graph, wherein the operation graph includes all possible operational terms and paths of the software product describing a software product based user interface of the software product. Further, in response to retrieving the one or more terms and corresponding paths that match the at least one keyword, a defect description is generated based on the one or more matched terms, corresponding paths, and received keywords.
Other embodiments and aspects, including but not limited to computer systems and computer program products, are described in detail herein and are considered a part of the claimed invention.
These and other features and advantages of the present invention will be described and/or will become apparent to those of ordinary skill in the art in view of the following detailed description of the exemplary embodiments of the present invention.
Drawings
These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. The various features of the drawings are not to scale, since they are shown for clarity in order to assist those skilled in the art in understanding the invention in conjunction with the detailed description. In the drawings:
FIG. 1 illustrates a cloud computing node according to an embodiment of the present invention;
FIG. 2 illustrates a cloud computing environment according to an embodiment of the present invention;
FIG. 3 illustrates a model layer according to an embodiment of the invention;
FIG. 4 shows an example of an operational diagram according to an embodiment of the present invention.
FIG. 5 shows a flowchart of an example method according to an embodiment of the invention.
FIG. 6A illustrates an example of an enhanced Hidden Markov Model (HMM) in which one or more verb-to name word pairs are generated in accordance with an embodiment of the present invention.
FIG. 6B illustrates an example of an enhanced HMM where one or more noun-verb pairs are generated, according to an embodiment of the present invention.
FIG. 7 illustrates an enhanced Encoder-Decoder (Encoder-Decoder) model in which sentences are generated based on verb-to verb pairs or noun-verb pairs according to an embodiment of the present invention.
Fig. 8 shows an example of a feedback process of generating a similarity according to an embodiment of the present invention.
Detailed Description
Detailed embodiments of the claimed structures and methods are disclosed herein; however, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods that may be embodied in various forms. This invention may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.
It is to be understood in advance that although this disclosure includes detailed descriptions regarding cloud computing, implementation of the teachings recited herein is not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in connection with any other type of computing environment, whether now known or later developed.
Cloud computing is a service delivery model for convenient, on-demand network access to a shared pool of configurable computing resources. Configurable computing resources are resources that can be deployed and released quickly with minimal administrative cost or interaction with a service provider, such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services. Such a cloud model may include at least five features, at least three service models, and at least four deployment models.
Is characterized by comprising the following steps:
self-service on demand: consumers of the cloud are able to unilaterally automatically deploy computing capabilities such as server time and network storage on demand without human interaction with the service provider.
Wide network access: computing power may be acquired over a network through standard mechanisms that facilitate the use of the cloud through heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, Personal Digital Assistants (PDAs)).
Resource pool: the provider's computing resources are relegated to a resource pool and serve multiple consumers through a multi-tenant (multi-tenant) model, where different physical and virtual resources are dynamically allocated and reallocated as needed. Typically, the customer has no control or even knowledge of the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center), and thus has location independence.
Quick elasticity: computing power can be deployed quickly, flexibly (and sometimes automatically) to enable rapid expansion, and quickly released to shrink quickly. The computing power available for deployment tends to appear unlimited to consumers and can be available in any amount at any time.
Measurable service: cloud systems automatically control and optimize resource utility by utilizing some level of abstraction of metering capabilities appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled and reported, providing transparency for both service providers and consumers.
The service model is as follows:
software as a service (SaaS): the capability provided to the consumer is to use the provider's applications running on the cloud infrastructure. Applications may be accessed from various client devices through a thin client interface (e.g., web-based email) such as a web browser. The consumer does not manage nor control the underlying cloud infrastructure including networks, servers, operating systems, storage, or even individual application capabilities, except for limited user-specific application configuration settings.
Platform as a service (PaaS): the ability provided to the consumer is to deploy consumer-created or acquired applications on the cloud infrastructure, which are created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but has control over the applications that are deployed, and possibly also the application hosting environment configuration.
Infrastructure as a service (IaaS): the capabilities provided to the consumer are the processing, storage, network, and other underlying computing resources in which the consumer can deploy and run any software, including operating systems and applications. The consumer does not manage nor control the underlying cloud infrastructure, but has control over the operating system, storage, and applications deployed thereto, and may have limited control over selected network components (e.g., host firewalls).
The deployment model is as follows:
private cloud: the cloud infrastructure operates solely for an organization. The cloud infrastructure may be managed by the organization or a third party and may exist inside or outside the organization.
Community cloud: the cloud infrastructure is shared by several organizations and supports a specific community of common interest relationships, such as mission missions, security requirements, policy and compliance considerations. A community cloud may be managed by multiple organizations or third parties within a community and may exist within or outside of the community.
Public cloud: the cloud infrastructure is offered to the public or large industry groups and owned by organizations that sell cloud services.
Mixing cloud: the cloud infrastructure consists of two or more clouds (private, community, or public) of deployment models that remain unique entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting traffic sharing technology for load balancing between clouds).
Cloud computing environments are service-oriented with features focused on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that contains a network of interconnected nodes.
The cloud computing node 10 has a computer system/server 12 or a mobile electronic device (e.g., a communications device) that is operable with numerous other general purpose or special purpose computing system environments or configurations. As is well known, examples of computing systems, environments, and/or configurations that may be suitable for operation with computer system/server 12 include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set top boxes, programmable consumer electronics, network pcs, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above systems, and the like.
Computer system/server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system/server 12 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.
As shown in fig. 1, the computer system/server 12 in the cloud computing node 10 is in the form of a general purpose computing device. The components of computer system/server 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
Computer system/server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system/server 12 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. The computer system/server 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 1, and commonly referred to as a "hard drive"). Although not shown in FIG. 1, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The computer system/server 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with the computer system/server 12, and/or with any devices (e.g., network card, modem, etc.) that enable the computer system/server 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, the computer system/server 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet) via the network adapter 20. As shown, network adapter 20 communicates with the other modules of computer system/server 12 via bus 18. It should be understood that although not shown in the figures, other hardware and/or software modules may operate with the computer system/server 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Referring now to FIG. 2, an exemplary cloud computing environment 50 is shown. As shown, cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as Personal Digital Assistants (PDAs) or mobile phones 54A, desktops 54B, laptops 54C, and/or automotive computer systems 54N may communicate. The cloud computing nodes 10 may communicate with each other. Cloud computing nodes 10 may be physically or virtually grouped (not shown) in one or more networks including, but not limited to, private, community, public, or hybrid clouds, or a combination thereof, as described above. In this way, cloud consumers can request infrastructure as a service (IaaS), platform as a service (PaaS), and/or software as a service (SaaS) provided by the cloud computing environment 50 without maintaining resources on the local computing devices. It should be appreciated that the types of computing devices 54A-N shown in fig. 2 are merely illustrative and that cloud computing node 10, as well as cloud computing environment 50, may communicate with any type of computing device over any type of network and/or network addressable connection (e.g., using a web browser).
Referring now to FIG. 3, a set of functional abstraction layers provided by cloud computing environment 50 (FIG. 2) is shown. It should be understood at the outset that the components, layers, and functions illustrated in FIG. 3 are illustrative only and that embodiments of the present invention are not limited thereto. As described, the following layers and corresponding functions are provided:
the hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a host computer 61; a RISC (reduced instruction set computer) architecture based server 62; a server 63; a blade server 64; a storage device 65; networks and network components 66. Examples of software components include: web application server software 67 and database software 68.
The virtual layer 70 provides an abstraction layer that can provide examples of the following virtual entities: virtual server 71, virtual storage 72, virtual network 73 (including a virtual private network), virtual applications and operating system 74, and virtual client 75.
In one example, the management layer 80 may provide the following functions: the resource provisioning function 81: providing dynamic acquisition of computing resources and other resources for performing tasks in a cloud computing environment; metering and pricing function 82: cost tracking of resource usage and billing and invoicing therefor is performed within a cloud computing environment. In one example, the resource may include an application software license. The safety function is as follows: identity authentication is provided for cloud consumers and tasks, and protection is provided for data and other resources. User portal function 83: access to the cloud computing environment is provided for consumers and system administrators. Service level management function 84: allocation and management of cloud computing resources is provided to meet the requisite level of service. Service Level Agreement (SLA) planning and fulfillment function 85: the future demand for cloud computing resources predicted according to the SLA is prearranged and provisioned.
It should be noted that the processing of the defect description in accordance with embodiments of the present invention may be implemented by the computer system/server 12 of FIG. 1.
As previously described, defects and/or errors may be encountered during the development, testing, and even support phases of a software product. Therefore, there is a need to generate and present information regarding software product defects/errors in a standardized and efficient manner. Existing ways of generating defect information include manual or automatic methods. Of course, the way in which defects are manually recorded, such as filled in Excel form or by template management errors, is time consuming and labor intensive. Preferably, the computer is caused to automatically describe the defect, for example, a method in which the computer automatically logs an error. By using this method, accurate information about the error can be obtained, but since the log of the defect is based on the code level, this information can be difficult to understand and only assists in the error repair process. Other automatic error reporting tools exist that can automatically detect errors and generate error reports. However, these types of tools only detect application exceptions or errors that terminate a program during a testing phase, and cannot automatically detect functional errors. With the development of artificial intelligence technology, there are also methods that can automatically accomplish the detection or generate text describing defects based on images or speech showing problems. All of these existing approaches lack a solution for utilizing a user interface and generating defect descriptions based on only a few keywords entered by the user.
Defect description generation and presentation in embodiments of the present invention may include reproducing the operational steps that occur before a user encounters a defect. Embodiments of the present invention utilize a user interface of a software product and, in one embodiment, may use an interactive user interface to generate defect description information based on received user input (i.e., keywords). Embodiments of the present invention may automatically generate the operational steps prior to the occurrence of a defect based on a predefined operational diagram of a software product having a structured user interface or desktop.
The operational view of the software product is predefined based on the user interface of the software product having a structured user interface or desktop. It is well known that for software products having structured graphical user interfaces, windows, icons or buttons and other graphics displayed in the user interface may represent different actions or functions, and a user may select any one by using a pointing device such as a mouse to perform various operations. In general, the user can start from any button in the home page to any other button in any page to complete various operations. Operations may occur sequentially/serially or in parallel. In practice, an operation is a series of processes that include clicking a button or the like in order to perform a function or action for a different purpose. Terms may be used to refer to names, functions, operations, etc. of the operations, and paths may be used to refer to the order or process of the operations. All possible operations of a software product may be represented or represented in terms of terms and paths, which may form a graph referred to herein as an operational diagram. These terms may also be referred to as nodes in the operational graph. It is noted that the operational diagram may be a visual presentation of various operations based on the user interface of the software product.
Referring now to FIG. 4, FIG. 4 illustrates an example of an operational diagram according to an embodiment of the present invention. The operational diagram shown in FIG. 4 is predefined or created for a software product. Any software product having an interactive user interface is suitable for the method of the present invention to create an operational diagram. In the created operation chart, "home page", "primary title", "primary (primary) subtitle", "secondary title", "secondary subtitle", "action level", and "result" represent different hierarchical structures. In other words, "home" represents the first page, "first-level heading" represents the second page, "main subtitle" represents the third page, and so on, with a logical order between them. The nodes under the respective hierarchies are terms indicating functions or operations for different purposes of the operation, and the like. Parallel nodes may be executed in parallel, and nodes from left to right may be executed sequentially from left to right in the direction of the arrow. For example, if the user were to perform the action "Select Variables," he/she could start with the "welome" interface in the home page, then click on "File," Open dataset, "" Analyze, "" Descriptive Statistics, "" Frequencies, "" Variables, "in that order, and then perform the action" Select Variables. Thus, "File," "Open dataset," "Analyze," "Descriptive Statistics," "Frequencies," "Variables" are related terms, and the process or sequence forms the corresponding path to which these terms relate.
The embodiment of the invention is based on the predefined operation diagram and can provide the user with the capability of finding defects related to functions or actions based on the user interface, for example, the button cannot work, the file cannot be imported and the like.
Hereinafter, an embodiment of the present invention regarding how to generate the user interface-based defect description information will be described in detail with reference to fig. 5, 6, 7, and 8.
Referring now to FIG. 5, shown is a flow diagram of an exemplary method in accordance with an embodiment of the present invention.
The method starts at step S510 and proceeds to step S530. In step S510, one or more keywords from a user describing defects encountered while using the software product are received. The user may be a developer, tester, supporter, end user, etc. The keywords from the user may be obtained in any form, for example, in the form of a questionnaire. The questionnaire may be designed based on the action graph such that at least one expected keyword/answer to the question from the user in the questionnaire matches the terms in the action graph. The user may fill in or select keywords in the questionnaire based on the defects/problems encountered by the user. The content of the questions and the format of the questionnaire are not limited. For example, questions may include: "what are you using? "," under which interface you are? "and/or" under what function and tab (tab) do you are? "and" what problem did you encounter? "and the like. In terms of format, any suitable means may be used, such as requiring the user to input an answer, or providing the user with terms in an action map, and requiring the user to make a selection. The following is an example of a questionnaire and keywords entered/entered by the user. This example is specific to a certain software product whose operational diagram is shown in fig. 4 and will be used throughout the process of the present invention.
a. What data set you are using now?
Employee dataset.csv
b. What functions and tabs you are now under? Or under which interface you are now?
Frequencies,Statistics
c. What problem did you encounter?
Name option result
Up button can’t work<None>
For the above example, assume that the keyword "Employee dataset. csv, queries, statics, Up button can't work" is received from the user. Then, the method proceeds to step S520, where one or more terms matching the at least one keyword and a path corresponding to the one or more terms are retrieved in the operation diagram. The operational diagram includes all possible operational terms and paths of the software product that describe the user interface of the software product. The operational diagram may be predefined in advance and may be ready for later use, rather than being defined each time. The operational diagrams have been discussed previously, and FIG. 4 provides an example of an operational diagram. In the above example, the keywords "Employee dataset. csv, Frequencies, statics, Up button can't work" are used for the search in FIG. 4. As can be seen by the dashed arrow lines in the operation diagram, the terms "dataset, Analyze, Descriptive statics, and Frequencies" as well as the path and related terms corresponding to the terms are determined. For example, it may be determined that the path starts at a welcome interface and ends at the user's current interface, there may be one or more intermediate interfaces between the welcome interface and the current interface, and one or more determined or related terms are nodes in the path. It is noted that due to differences in expression of the same things, the related terms may be the same or similar to the keywords received from the user, e.g., "dataset" is similar in expression to "employee dataset. All keywords do not need to be searched out in the operation graph. Typically, keywords are found that are the same or similar to the nodes in the operational diagram regarding the application's name, function, tabs, and/or interfaces. The problem description part such as "Up button can't work" may not be found, and the complete information about the problem mainly depends on the input of the user.
Searching and determining relevant terms and corresponding paths in the operational graph may be accomplished using, for example, the a-star algorithm. The a-algorithm is a search algorithm for shortest path search and graph traversal to find paths between multiple "nodes".
The method then proceeds to step S530, wherein responsive to the received one or more terms/keywords matching the at least one related term and the corresponding path, defect description information is generated based on the one or more related and matched terms, the corresponding path, and the received keyword. As can be seen in FIG. 4, the search result in the operation chart is "dataset, Analyze, Descriptive Statistics, Frequencies" along the path shown by the dashed arrow line. The terms "dataset, Analyze, Descriptive Statistics, Frequencies" represent different operational steps in sequence. Thus, the sequential terms in the operational figures may reproduce the operational steps that occur before a defect is encountered.
In addition to the reproduction operation steps, the defect description includes problems. Questions may be derived from keywords of the questionnaire. For the above example, the question is "Up button, can't work". If the question is not available from the questionnaire, it can be inferred. Since problems typically occur in the last step by default, the user can infer that "statics" is a problem, depending on the order or path of operation. So far, a simple defect description including a reproduction step and a problem is generated.
Further, to more automatically generate and approximate the defect description to human natural language, according to an embodiment of the present invention, verb-to-name word pairs are generated using an enhanced Hidden Markov Model (HMM), as shown in FIG. 6A. Similarly, as shown in FIG. 6B, noun-verb pairs may also be generated with an enhanced HMM. According to another embodiment of the present invention, as shown in FIG. 7, a verb-to-noun pair or a noun-verb pair is input to an enhanced coder-Decoder (Encoder-Decoder) model to automatically generate a sentence describing a defect, which is closer to the natural language of humans.
FIGS. 6A and 6B provide an illustration of verb-to-noun pairs and noun-verb pairs generated with an enhanced Hidden Markov Model (HMM), according to one embodiment. HMMs are statistical models used to describe markov processes with hidden unknown parameters. The model is used to determine hidden sequences from observable sequences. The embodiment of the invention uses an enhanced HMM model, and the enhancement mainly focuses on two aspects. In one aspect, the enhanced HMM model is used to retrieve verbs that best match nouns or nouns that best match verbs. On the other hand, the enhanced HMM model includes an inverse phase, indicated by dashed arrow 606 in FIG. 6A, to order verb-noun pairs in the order of operations associated with respective ones of the verb-noun pairs, i.e., to generate a logical order/sequence between the generated verb-noun pairs. For example, "Select-Analyze" occurs after "Open-Employee dataset. csv," and "Select-Analyze" occurs after "Select-describe statics," and so on. The existing application program of the HMM algorithm only has positive phase, and is easy to fall into logic errors.
Referring now to FIG. 6A, the input to the enhanced HMM model is a series of nouns, including n1,n2,n3…nn. The nouns are from keywords entered by the user and terms searched out in the operation diagram. For example, in the aforementioned examples, the nouns include "Frequencies, statics, dataset, Analyze, Descriptive statics". Verbs can be found from keywords/terms entered by the user and the action graph, especially from the "action level" in the action graph. In particular, for the aforementioned example, the verbs are found in the terms "can't work, open, select, choose, enable, etc.". In FIG. 6A, the observable sequence is n1,n2,n3…nnAnd the hidden sequence is v corresponding to a noun1,v2,v3…vn。
To find verbs that match nouns to generate verb-to-noun pairs, it may be determined that a transition between verbs and verbs is required. This transition is based on two probability matrices. One is a positive transition probability matrix 604 that represents the probability that a verb will be transitioned to the next verb. The other is a reverse transition probability matrix 606, which represents the probability that the verb infers the previous verb. The transition between nouns and verbs is computed from the transmit transition matrix 608, 608 indicating the probability that a verb corresponds to a possible noun.
The above-mentioned values of the probability are calculated by the following mathematical formula, in which a "C-inverse state-transition matrix is defined: the "with" inverse state "portion is an improvement over existing HMM models and corresponds to 606 shown in FIG. 6A.
Definition of
V-the set of all possible verb states,
n-the set of all possible noun observations.
V＝v1，v2，v3，...，vM，N＝n1，n2，n3，...，nK，
M-number of possible verb states
K-number of possible noun states
I-state sequence. It has a length of T
O-Observation sequences corresponding to I
I＝(i1，i2，i3，…，iT) O＝(o1，o2，o3，...，oT)
A-state-transition matrix:
A＝[aij]M×M
i＝1，2，3，...，M j＝1，2，3，...，M
transition to state v at time t +1 under the condition of state vi at time tjProbability of (2)
aij＝P(it+1＝vj|it＝vi)
C-inverse state-transition matrix:
C＝[cij]M×M
i＝1，2，3，...，M j＝1，2，3，...，M
at time t, in state viIs transitioned to state v at time t-1jProbability of (2)
cij＝P(it-1＝vj|it＝vi)
B-observe the probability matrix:
B＝[bj(u)]M×K
at time t, in state vjConditional generation noun observation nuProbability of (2)
bj(u)＝P(ot＝nu|it＝vj)
Pi-initial state probability vector:
π＝(πi)
πi＝P(i1＝vi)
HMM model λ ═ (A, B, C,. pi.)
Enhanced viterbi algorithm:
input HMM model λ ═ (A, B, C, π) and observation model
O＝(o1，o2，o3，...，oT)
Outputting the optimal path
Positive phase:
(1) initialization:
δ1(i)＝πibi(o1)
ψ1(i)＝0
(2) recursion for T2, 3, …, T:
(3) termination conditions were as follows:
(4) best path backtracking, for T-1, T-2, …,1:
(5) obtaining a positive optimal path:
reverse phase:
(1) initialization:
δ′T(i)＝δT(i)
(2) recursion: for T-1, T-2, …,1:
(3) comparing and outputting:
(4) and finally, outputting:
similarly, referring now to FIG. 6B, the enhanced HMM model can also be used to find nouns that match verbs to generate noun-verb pairs. The difference is that in this case the observable sequence is v1、v2、v3…vnAnd the hidden sequence is n1、n2、n3…nn. Meanwhile, the order in which nouns are generated does not need to be guaranteed, and thus, the reverse process shown in fig. 6A is not needed in fig. 6B.
Further, the user may input a fixed pair in addition to the generated verb-to-verb pair and/or noun-to-verb pair. For example, the user enters a noun-verb pair: "Up button can't work". In this case, there is no need to generate a verb for "Up button" or a noun for "can't work".
For the aforementioned example, the verb-noun pairs generated by the enhanced HMM model (also including the fixed pairs) are as follows:
a.Open-Employee dataset.csv
b.Select-Analyze
c.Select-Descriptive Statistics
d.Select-Frequencies
e.Choose-Statistics
f.Up button-can’t work
it should be appreciated that while the enhanced HMM model is used to generate verb-verb pairs or noun-verb pairs in accordance with embodiments of the present invention, other suitable models or algorithms may be applied to embodiments of the present invention.
Referring now to FIG. 7, an enhanced coder-Decoder (Encode-Decoder) model is shown in which a sentence is generated based on a resulting verb-to-noun pair or noun-to-verb pair, according to an embodiment of the present invention. One application of the coder-decoder model is in the field of machine translation, such as inputting chinese into the coder and then outputting the translated english to the decoder. In one embodiment of the method of the present invention, the enhanced coder-decoder model is used to generate a sentence having verb-to-noun pairs or noun-verb pairs generated in the enhanced HMM model shown in FIG. 6A or 6B. On the one hand, in the existing coder-decoder model, the length of the input is uncertain. In the existing enhanced encoder-decoder model, only verb-noun pairs, noun-verb pairs and symbols are input "<cos>", the operation speed is relatively fast. Meanwhile, the verb-noun pair or the noun-verb pair is expanded into a sentence conforming to the natural language of the human. On the other hand, C701 shown in fig. 7 is a vector connecting an encoder and a decoder, whereas in the existing encoder-decoder model, C701 is h1To h3In the current enhanced coder-decoder model, except for receiving h1To h3In addition, C701 receives a pre-training score called similarity 703, which is pre-trained and provided by the feedback as shown in fig. 8.
As shown in fig. 7, the encoder 701 receives 3 inputs: verb v1N, noun1And an end identifier<eos>. After using three gated cyclic units (GRUs), the GRUs are combined with the similarity 703 to form vector C705. C705 contains the information in three inputs. Decoder 707 receives 3 inputs: verb v1An identifier<BOS>And vector C705. If the first word of the generated sentence is fixed as verb v1That is, if a verb-noun pair is entered, the first word of each generated sentence is the corresponding verb, and the generated sentence will reproduce the action step. Similarly, if the first word is fixed as a noun n1That is, if a noun-verb pair is input, then a question may be generatedThe statement of description. Thereafter, if the output statement does not contain a noun n in the input verb-noun pair1Then the nearest noun n in the parsing can be determined and replaced1The term (1). Furthermore, to describe defects directly in terms of software products, the generated words are replaced by terms in the operational diagram to which they are most related.
Further, according to one embodiment, upon inputting verb-verb pairs or noun-verb pairs generated in the enhanced HMM model as shown in FIGS. 6A and 6B, a sentence describing a defect may be automatically generated as shown below, where sentences 1-5 are operational steps for the reproduction and sentence 6 is a problem encountered:
1.Open-Employee dataset.csv——Open the‘Employee dataset.csv’dataset.
2.Select-Analyze——Select Analyze title.
3.Select-Descriptive Statistics——Select the Descriptive Statistics subtitle.
4.Select-Frequencies——Select Frequencies function.
5.Choose-Statistics——Choose Statistics tab.
6.Up button-can’t work——The problem is that the‘Up’
button can’t work.
FIG. 8 illustrates an example of the feedback process illustrated in FIG. 7 to generate a similarity, according to an embodiment of the present invention. For the above example, the generated statements and actual statements are as follows:
the generated statement:
1.Open the‘Employee dataset.csv’dataset.
2.Select Analyze title.
3.Select the Descriptive Statistics subtitle.
4.Select Frequencies function.
5.Choose Statistics tab.
6.The problem is that the‘Up’button can’t work.
the actual statement:
1.Open"Employee dataset.csv"
2.Select Analyze title
3.Select the Descriptive Statistics subtitle
4.Select Frequencies
5.In Statistics tab,the‘Up’button can’t work.
the above example is only one case. In fact, there may be other situations for training. In the training process, the generated sentence and the actual sentence (i.e., actual data for each case) are input into feedback to output a value of "similarity", which represents the similarity between the generated sentence and the actual sentence. After feeding back the "similarity" to the enhanced encoder-decoder model shown in fig. 7, the enhanced encoder-decoder model updates its parameters based on the "similarity" to output a sentence close to the natural language.
The main purpose of the feedback is to compare the difference between the generated sentence and the actual sentence. Calculating the difference includes two parts. The first part is to calculate the literal distance using Levenshtein distance, Jacorwinkler (Jarowinkler) score, and the simulated hash (SimHash) algorithm. The second part is to calculate semantic similarity. After building the embedding of the two sentences, semantic similarity is calculated using word2 vec. Based on the overall similarity weights 802, the average weights 804 are then returned to the enhanced encoder-decoder model as the similarities. Four weights w1、w2、w3And w4A non-linear regression model is constructed for training and is not discussed in detail herein.
It should be appreciated that while the enhanced coder-decoder model is used to generate statements according to embodiments of the present invention, other suitable models or algorithms now known or to be developed may also be applied to embodiments of the present invention.
The present invention may be a system, method and/or computer program product in any combination of possible technical details. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement various aspects of the present invention.
The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
Computer program instructions for carrying out operations of the present invention may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine related instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein were chosen in order to best explain the principles of the embodiments, the practical application, or technical improvements to the techniques in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (9)
1. A computer-implemented method, comprising:
receiving, by one or more processors, one or more keywords describing at least one flaw associated with a computer application;
searching, by one or more processors, an operational graph to search for one or more terms that match the one or more keywords, wherein the one or more terms describe operation of the computer application based on a user interface associated with the computer application;
responsive to determining that the one or more keywords match the one or more terms associated with the operational graph, determining, by one or more processors, an operational path that corresponds to the one or more terms and includes a sequence of operational steps that precede the at least one defect; and
in response to determining the match and the corresponding operational path, generating and presenting, by one or more processors, defect description information based on the match, the corresponding operational path, and the one or more keywords.
2. The method of claim 1, wherein generating the defect description information comprises at least one of:
determining a matching verb for each noun of the one or more terms and the one or more keywords to generate one or more verb-noun pairs; and
determining a matching noun for each verb in the one or more terms and the one or more keywords to generate one or more noun-verb pairs.
3. The method of claim 2, wherein determining a matching verb for each noun further comprises:
ordering the verb-noun pairs in an order associated with a sequence of operational steps associated with respective ones of the verb-noun pairs.
4. The method of claim 3, further comprising:
generating a sentence describing the at least one flaw based on at least one of the verb-verb pair and the noun-verb pair using an enhanced coder-decoder model.
5. The method of claim 4, further comprising:
generating a similarity between the generated sentence and the real sentence for feedback into the enhanced coder-decoder model.
6. The method of claim 1, wherein the search is performed using an a-x algorithm.
7. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to perform the steps of the method of any of claims 1 to 6.
8. A system, comprising:
a processor; and
a memory coupled to the processor, wherein the memory includes instructions that, when executed by the processor, cause the processor to perform the steps of the method of any of claims 1-6.
9. A computer system comprising means for performing the steps of the method of any one of claims 1 to 6.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US16/508378 | 2019-07-11 | ||
| US16/508,378 US11182155B2 (en) | 2019-07-11 | 2019-07-11 | Defect description generation for a software product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112214393A true CN112214393A (en) | 2021-01-12 |
Family
ID=74059587
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010639448.3A Pending CN112214393A (en) | 2019-07-11 | 2020-07-06 | Generating defect descriptions for software products |
Country Status (2)
| Country | Link |
|---|---|
| US (1) | US11182155B2 (en) |
| CN (1) | CN112214393A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11182155B2 (en) | 2019-07-11 | 2021-11-23 | International Business Machines Corporation | Defect description generation for a software product |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100042970A1 (en) * | 2008-08-13 | 2010-02-18 | Electronic Data System Corporation, a corporation of Delaware | Generating loadrunner scripts |
| US20110219360A1 (en) * | 2010-03-05 | 2011-09-08 | Microsoft Corporation | Software debugging recommendations |
| US20130014093A1 (en) * | 2010-03-29 | 2013-01-10 | Soft4Soft Co., Ltd. | Code inspection executing system for performing a code inspection of abap source codes |
| US20140101208A1 (en) * | 2012-10-09 | 2014-04-10 | Industrial Technology Research Institute | Keyword-based user interface in electronic device |
| CN107430612A (en) * | 2015-02-12 | 2017-12-01 | 微软技术许可有限责任公司 | Search document of the description to the solution of computational problem |
| CN108121829A (en) * | 2018-01-12 | 2018-06-05 | 扬州大学 | The domain knowledge collection of illustrative plates automated construction method of software-oriented defect |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| NO316480B1 (en) * | 2001-11-15 | 2004-01-26 | Forinnova As | Method and system for textual examination and discovery |
| US20110083123A1 (en) * | 2009-10-05 | 2011-04-07 | Microsoft Corporation | Automatically localizing root error through log analysis |
| JP6019968B2 (en) | 2012-09-10 | 2016-11-02 | 株式会社リコー | Report creation system, report creation apparatus and program |
| CN103207951B (en) * | 2013-04-22 | 2015-02-25 | 腾讯科技（深圳）有限公司 | Way finding method and device |
| US10395323B2 (en) * | 2015-11-06 | 2019-08-27 | International Business Machines Corporation | Defect management |
| CN106649223A (en) | 2016-12-23 | 2017-05-10 | 北京文因互联科技有限公司 | Financial report automatic generation method based on natural language processing |
| US11182155B2 (en) | 2019-07-11 | 2021-11-23 | International Business Machines Corporation | Defect description generation for a software product |
-
2019
- 2019-07-11 US US16/508,378 patent/US11182155B2/en active Active
-
2020
- 2020-07-06 CN CN202010639448.3A patent/CN112214393A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100042970A1 (en) * | 2008-08-13 | 2010-02-18 | Electronic Data System Corporation, a corporation of Delaware | Generating loadrunner scripts |
| US20110219360A1 (en) * | 2010-03-05 | 2011-09-08 | Microsoft Corporation | Software debugging recommendations |
| US20130014093A1 (en) * | 2010-03-29 | 2013-01-10 | Soft4Soft Co., Ltd. | Code inspection executing system for performing a code inspection of abap source codes |
| US20140101208A1 (en) * | 2012-10-09 | 2014-04-10 | Industrial Technology Research Institute | Keyword-based user interface in electronic device |
| CN107430612A (en) * | 2015-02-12 | 2017-12-01 | 微软技术许可有限责任公司 | Search document of the description to the solution of computational problem |
| CN108121829A (en) * | 2018-01-12 | 2018-06-05 | 扬州大学 | The domain knowledge collection of illustrative plates automated construction method of software-oriented defect |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11182155B2 (en) | 2019-07-11 | 2021-11-23 | International Business Machines Corporation | Defect description generation for a software product |
Also Published As
| Publication number | Publication date |
|---|---|
| US20210011713A1 (en) | 2021-01-14 |
| US11182155B2 (en) | 2021-11-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11501187B2 (en) | Opinion snippet detection for aspect-based sentiment analysis | |
| US10776082B2 (en) | Programming environment augment with automated dialog system assistance | |
| US10621166B2 (en) | Interactive dialog in natural language using an ontology | |
| US11645470B2 (en) | Automated testing of dialog systems | |
| US20190050443A1 (en) | Method and system for improving training data understanding in natural language processing | |
| US10740374B2 (en) | Log-aided automatic query expansion based on model mapping | |
| US10133732B2 (en) | Interactive location sensitive network response | |
| US11144607B2 (en) | Network search mapping and execution | |
| US20230095180A1 (en) | Question answering information completion using machine reading comprehension-based process | |
| US11176019B2 (en) | Automated breakpoint creation | |
| US20180129544A1 (en) | Suggesting Application Programming Interfaces Based on Feature and Context Analysis | |
| US10216802B2 (en) | Presenting answers from concept-based representation of a topic oriented pipeline | |
| US10380257B2 (en) | Generating answers from concept-based representation of a topic oriented pipeline | |
| US20220215047A1 (en) | Context-based text searching | |
| CN112214393A (en) | Generating defect descriptions for software products | |
| WO2019180515A1 (en) | Query recognition resiliency determination in virtual agent systems | |
| US11481212B2 (en) | Automatic identification of reference data | |
| US20220075936A1 (en) | Mining multi-party collaboration platforms to create triaging trees and playbooks | |
| US10585986B1 (en) | Entity structured representation and variant generation | |
| US11811626B1 (en) | Ticket knowledge graph enhancement | |
| US20230153634A1 (en) | Composite feature engineering | |
| US11138383B2 (en) | Extracting meaning representation from text | |
| US11971887B2 (en) | Identifying and replacing logically neutral phrases in natural language queries for query processing | |
| US20230367619A1 (en) | Bootstrapping dynamic orchestration workflow | |
| US20230267342A1 (en) | Iterative answer and supplemental information extraction for machine reading comprehension |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |